Neural Network is not getting better So i started building my own neural network framework in node.js, just to understand the concept of neural networks better.

Currently i implemented Neurons, Connections between these Neurons and Layers.

Given this network:

N3s output for example, would be Sigmoid((N1 * W1 + N2 * W4) + B) where B is a randomly generated Bias between -1 and 1.

To train the network, a dataset which contains the inputs an their desired output is used. The network will be fed the inputs (Every input getting assigned to one Neuron in the first layer, for example [1, 0] would result in N1 beeing 1 and N2 beeing 0). Then for each Connection, the weight will be increased or decreased by 1% if the resulting output of the network would be closed to the desired one.

For example if 1 is the desired output and the current output would be 0.5, the output with the weight increased would be 0.37 and the output with the weight decreased would be 0.46. The weight would be increased.

Same thing is done for every Neurons Bias.

My Problem:

If i train above network with the dataset:

input: 0, 0
desired Output: 0

input: 0, 1
desired Output: 1

input: 1, 0
desired Output: 1

input: 1, 1
desired Output: 0

for 10000 iterations, it will give me something similar to the following output:

[ 0, 0 ] => [ 0.13500189926566186 ]
[ 0, 1 ] => [ 0.14749124528600177 ]
[ 1, 0 ] => [ 0.14238677719564685 ]
[ 1, 1 ] => [ 0.15412548826116046 ]

I am kinda new to Neural Networks, so sorry if this is a dumb question.

Your training algorithm seems wrong. You should either implement Back Propagation, or tweak your own training algo towards that.

Now your training brings all the weights to an average value, that's why your output seems so close after 10000 iterations. What BP does is it modifies the weights that matter more.

Alternatively, if you don't want to use BP, you can play with evolution instead. It will give you the same result, just slower.

edit: can you post your training algo's code somewhere?
Also, you don't really need different objects (like Neurons, Connections, etc.), you only need matrix algebra for an efficient solution.


On which algorithm for boosting is the method xgbLinear of the xgboost/caret- package based on? In caret package of R, there is a method 'xgblinear'. What is the working algorithm behind this method.

I am not sure if this is helpful or not but here is a link to an academic paper detailing both the algorithm behind XGBoost and some use cases.

http://dmlc.cs.washington.edu/data/pdf/XGBoostArxiv.pdf


Shouldn't an autoencoder with #(neurons in hidden layer) = #(neurons in input layer) be "perfect"? I'm exploring autoencoders for the first time. I'm using the Matlab neural networks toolbox. I have created a synthetic dataset consisting of points in 2D space plus some noise. My idea was to visualize the features of the hidden layer of the autoencoder to get the feeling about how it works. After training the autoencoder, I apply it to the same training data set to check the output.

If I consider one neuron in the hidden layer, then I get the output data all clustered on a straight line. Which makes sense to me, as with one degree of freedom (plus bias), there is not much else the autoencoder can do other than to find the line that minimizes the distance to the data points.

However, if I have two neurons in the hidden layer, I would expect the autoencoder to be perfect, since we have two degrees of freedom to match all the points. Hence, with enough epochs, it should be able to reduce the error to almost zero, and reproduce the input data perfectly. To my surprise, this doesn't happen: the training quickly stalls and we see some imperfections when trying to reproduce the input data.

So where's the catch ? Shouldn't an autoencoder be able to trivially replicate the input to the output when it doesn't have to compress the input information ?

PS: in the picture below, "# hidden layers" actually means "# of neurons in the hidden layers"

I think a part of your problem could be related to the activation function - you're probably using tanh or relu for the hidden layer. If the middle layer did not have an activation function, the network would learn to simply pass the information forward - ideally, there would be a weight of 1 between every input and a corresponding hidden layer neuron and the other weights would be 0, completely replicating the information. But since you have activation functions, this replication wouldn't work, the activation would distort the result...

The activation function, which breaks the linearity of the problem is what makes NN so powerful, but it also causes these problems.. they are also the cause that NN have multople local minimum, as Lucas said, so the NN can get stuck in a "good enough" solution.

In short: remove the activaction function (make it linear) and it should work...


How do CNNs find different feature maps? Assume I have a CNN that in the first (conv) layer takes a 1-channel signal (the input) and gives a 2-channel output. Let's further assume that the rest of the net has symmetric architecture from the point-of-view of any of those channels. Moreover, let's assume all the weights are initialized by zero (or the same number).

How do we know that the feature maps (kernels of the channels) after training are not similar given the symmetry in such network? Specifically, what is unreasonable to have kernels with identical feature maps?

You shouldn't initialize all the weights with zero, use random initialization instead. Otherwise you can't break the symmetry and all the outputs in the network would be same.


How to split a CSV into a dataframe without newline Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same line?

#Loads the data in the textfile
df = sc.textFile("data.csv").map(lambda line: line.split(","))
df.take(1)

[['2',

  '20200118',

  '0',

  'AE',

  'ABV',

  'AUH',

  '20200416',

  '0',

You actually have a matrix, all items are grouped by row.

A very small example:

csv:
a,b
c,d

Will become:

[['a','b'],['c','d']]

The formatting of the output is indeed misleading.


xgboost classifier predicted negative probabilities I'm using XGBoost for a binary classification problem. There is no negative label, only 1 and 0.

I tunned the hyperparameters using Bayesian optimization then tried to train the final model with the optimized hyperparameters.

Mdl_XGB = xgb.train(OptimizedParams, dtrain)

scores_train = Mdl_XGB.predict(dtrain)

scores_test = Mdl_XGB.predict(dtest)

My problem is that the predicted scores for both train and test sets include both negative values and numbers greater than one. The scores are between -0.23 and 1.13.

Shouldn't these scores present the probability of belonging to class 1 (positive class)?

You have to set the option objective = binary:logistic to get probabilities between 0 and 1, otherwise you only get relative scores.


Mapping one embedding to another using Deep Learning I am trying to write a model that has the input vector of one embedding (say $E_1$) and predicts the corresponding vector in the second embedding $E_2$. Both are n-dimensional real dense vectors $\mathbb{R}^n$.

Concretely one is a skipgram word embedding, and the other is a node2vec graph embedding. I have approximately 30 000 training examples that provide a mapping between the two. Since they are both just real vectors, it seems like a trivial task to write a simple MLP that learns the non-linear transformation of one to the other (I actually don't really care about over fitting here, since the domain is closed).

However, I can't seem to get it to work properly.

In Keras, naively something like this should work:

in = Input(e1_dim)
hidden = Dense(some_value, activation="tanh")(in)
out = Dense(e2_dim)(hidden)

I have tried adding more hidden layers, but I think my problem lies with the fact that the input and output vectors are in the domain (-1,1), and so the choice of initializer, loss function, and activation function is critical.

I have tried setting the initializer to RandomUniform, but still no good results. For loss I have tried MAE en cosine_proximity, but both seem to produce terrible results. In particular cosine_proximity seems to not get above -0.5 which might be a sign. Any thoughts on the choice of architecture and loss function for mapping one embedding onto another (essentially a high-dimensional non-linear regression?)

You should take into account a few factors:

some_value is key here. You should probably choose a value greater than e1_dim and e2_dim. This may be a problem due to the large memory needs for high embedding dimensionalities.
if e2_dim is bounded to $(-1,+1)$, you may want to set a tanh activation in the last layer (i.e. after Dense(e2_dim)). With that in place, you may want to replace the activation of the hidden layer with a ReLU, to avoid squashing the gradients unnecessarily.
If your embeddings are directional, or they are in the unit ball, then you should be fine with cosine_proximity. Maybe a better option would be to use the von Mises-Fischer distance, but I think it's not straightforward. Otherwise, you may try World-Mover's Distance.

That being said, a lot of embedded space mapping works out there assume that the embedded spaces are approximately isomorphic and just go ahead with a linear transformation. You may want to have a look at the relevant bilingual embedding mapping literature; you may start with the work by Artetxe et al., 2016. I don't know if the mentioned assumption is met to any degree in your scenario (skipgram $\leftrightarrow$ node2vec), but you may assess this with the VF2 algorithm, like Artetxe et al. did.


A single column has many values per row, separated by a comma. How to create an individual column for each of these? As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make them all binary values?

i.e. code6254, code5854 etc...., where all these columns will be of binary value 0 or 1 depending on whether that row has the code or not? Thanks in advance :)

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this:

import pandas as pd
df = pd.DataFrame({'code': ['6254', '5854, 5676, 7265, 6051', '5815']})

At the start, your dataframe looks like this:

                     code
0                    6254
1  5854, 5676, 7265, 6051
2                    5815

A possible solution would be to do this:

df['code'].str.get_dummies(sep=', ').add_prefix('code')

which gives you this:

   code5676  code5815  code5854  code6051  code6254  code7265
0          0          0          0          0          1          0
1          1          0          1          1          0          1
2          0          1          0          0          0          0


Methodology for parallelising linked data? If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any particular piece of data currently being worked on, is there a methodology to split this out into chunks without reducing the validity of the data?

For example, assume I have a grid of crime across the whole of a country. I wish to treat this grid as a heat map of crime and therefore "smear" the heat from a crime to nearby points.

However, the time to calculate this is too long if I try to do it across the whole of the country. Therefore, I want to split this grid out into manageable chunks.

But if I do this then high crime areas on the edges of the chunks on this grid will not be "smeared" into nearby areas. I don't want to lose this data validity.

What, if any, is the methodology to solve this linking of data in parallelisation?

I'd look into applying some kind of sliding window to you data so you can process one subset at the time. By adjusting the window overlapping with previous chunks you can control how much context have on each batch.

The implementation details will depend on what kind of data structure you use, but since you mentioned a grid in which you want to smear the data, a convolution+pooling approach might do the trick. It's very easy to run in parallel and can be super efficiently on modern GPUs.

Also, you example suggests that the result of your chunk processing would be some kind of subsampling. If the data is too large you could subsample it first. There are many ways to do it depending on which criteria is acceptable for discarding data points so you'll have to dig into it to figure out which method is best for your use case


Recursive Feature Elimination (RFE) with Logistic Regression and little correlation between the features and the target (SKLearn) I have little experience in the field of Machine Learning, so I apologise in advance for my ignorance and lack of intuitions.

I'm trying to build a classifying algorithm to identify if a subject is healthy or ill. I have the following data:

17 subjects.
Each subject has 1140 features, ranging from -1 to 1.
Each subject has a label from 0 to 6. 0 meaning very ill and 6 completely healthy.
I only have data of subjects with labels from 0 to 2.
The features (X) have low correlation with the labels (Y).

The way I obtained the features is irrelevant to this question, so I'm not going to go through it.

This is how the data looks:

Each colour represents a subject. As I have mentioned, I only have data of subjects with labels from 0 to 2, that is, Y=0, Y=1 or Y=2. And the features (X) are then 17 Subjects x 1140 features.

What I want to do is the following:

Fit the data into a Logistic Regression.
Use the Recursive Feature Elimination algorithm in order to fit the data into the classification function and know how many features I need to select so that its accuracy is high.
Use Stratified Cross Validation to enhance the accuracy.

This is how I've implemented the algorithm in Python.

skf = StratifiedShuffleSplit(n_splits=50, test_size=0.3) # Cross-validation 50 times
estimator = LogisticRegression(C=1) # The estimator used is Logistic Regression
selector = RFECV(estimator, step=1, cv=skf, scoring="accuracy") # Run RFE
selector = selector.fit(X, Y) # Fit the data
print(selector.grid_scores_) # Print accuracy

And I obtained an accuracy around 60% for all the features, as shown below:

So, my question is. Do you think I implemented the code in the right way? Do you have any suggestion to enhance the accuracy? Maybe adjust a bit the parameters?

Thanks a lot. I hope I made myself clear.

Do you think I implemented the code in the right way?

Code is correct, but he is minimal possible.

  The features (X) have low correlation with the labels (Y).

It's could be the biggest problem, features have to have correlation with labels.

  Do you have any suggestion to enhance the accuracy?

Make transformation of labels(X). Preparing of labels its one most important parts in Machine Learning. Please read data transformation or feature extraction.
After this try different algorithms like xgboost, adaboost, random forest etc.
Perhaps you should use regularization L1 or L2.
Try early stopping.
After this step use cross-validation.
Choose suitable metrics for yours problem.


What came first? Backpropagation or Sigmoid? Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.

When did the sigmoid function become so popular in NNs?

I imagine the sigmoid function would have made its way into ANN's "since the beginning" since it is used in computational neuroscience to approximate the observed activation function of biological neurons. See for example this 1972 paper.


Displaying network error as a single value I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked with usually display the error/loss after each batch as a single decimal value, and I'd like to implement this functionality but I'm not sure how.

I understand squared error is given by $(y - \hat{y})^2$, and that for an output layer with $m$ neurons, you should have an error vector of size $m$. However, how is the error vector displayed as one value?

You can simply sum the absolute m error to have 1 overall error value of the neural network. You can also compute mean square error. Whatever error you choose will not change much, in both cases the smaller the error the better your neural network. Even if by convention we prefer to take the mean squared error.


What tool to find expected and hidden patterns in data? I have no background AT ALL in data science/stats/mathematics. However, I've always been interested in what data shows.

I have a huge dataset right now - daily attendance figures for a factory of ~300 for the past 10 years. I'm interested in finding out answers to questions like is there a pattern of leaves correlated with public holidays? For example, around which holiday (+/- 2 days) are workers most likely to take a leave? This is an expected pattern. Or, was there a significant increase (+10%) in on-time reporting after bonuses were issued. Maybe there are hidden patterns which an algorithm can find.

Is there a tool I can plug this data in which can help me find these patterns? Google tells me there's a tool http://www.i-programmer.info/news/84-database/3501-mine-finding-patterns-in-big-data.html but I'm not sure if this is the right direction for me.

I'd appreciate any advice!

You are unlikely to find a tool that will truly do this for you. Various bits of software may produce some nice graphs. However we're not at a stage where tools can generate interesting questions to ask, ask them, and then explain the results to you.

If this is commercial, your best bet is to hire someone with an appropriate background and set them some questions. Tell them what your biggest goals and costs are, explain the data to them, and see what happens.  So the answer is a contractor.


Binary Encoding of Ordinal Categories I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame.

I want to encode these ordinal categories in columns with this method which is proposed here

I didn't understand what I should do. Should I find the probability of each category in each column and substitute the amount of probability with the string of each category?

The paper you've linked describes a method for predicting an ordinal dependent variable, but it sounds like you're looking for a way to use ordinal independent variables.  That part makes up just the first part of the paper's method, and is relatively easier.  It's somewhat analogous with one-hot encoding: you have indicators $x\leq \alpha$ for each level $\alpha$ (except the maximum one).

For example, if the levels are Low, Medium, High, your new variables will be "Low" and "Low or Medium" (skipping "Low or Medium or High" since that's always true).


best similarity measure for images with different angles I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using some libraries like
from image_similarity_measures.quality_metrics import rmse, psnr, fsim
in_img1 = cv2.imread("./IMG_4835.jpg")
in_img2= cv2.imread("./IMG_4836.jpg")
out_rmse = rmse(in_img1, in_img2)
out_fsim = fsim(in_img1, in_img2)
print(out_rmse)
print(out_fsim)

but none of the metrics seem to provide good results in my case. According to the description of different metrics given here, most of them check for contract/signal to error reconstruction ratio etc etc. https://up42.com/blog/tech/image-similarity-measures
What similarity metric or image comparison method in general would suit well in my case?

A brute-force method is simply to try all rotation angles and decide if 2 images are a rotation of one another.
However, there are features (eg fourier coefficients) which are rotation-invariant. So comparing these rotation-invariant features is a similarity metric for determining if 2 images are a rotation of one another.
References:

Rotation Invariance in Images
Rotation invariant indexing of shapes and line drawings
Which Transformation, Or Similarity Metric, Is Rotation, Shift and Scale Invariant?


How to normalize technical skills of IT using machine learning? I have a huge collection of skills collected/scraped from various online sources. It was huge effort done by our team.
Now, the biggest challenge we are facing is trying to normalize the skills back to its root form to reduce the duplication.
Here's are few examples,
agile    (or) agile methodologies  (or) agile software
java 2.x (or) java 3.x (or) java

These kind of duplicates are very common is my datasets. So I'm looking for a way to normalize them.
I am not very sure of the way to solve this issue. Can anyone suggest me some very good ways we can solve this problem with some decent accuracy?
Thank you

In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a few API.
Regex, Levenshtein distance, and other old school metrics are great, but in general you are never certain as to what you want to consider "as similar" besides the fact they words have a few common concepts (such as agile methodologies vs scrum vs kanban).
However, if you have a good pretrained encoder, this similarities will be captured.
In the second and next iteration, you can now use the encoder to check if similarities between skills are flagged properly, using the for example cosine distance. You can then automatically label the similarities and eventually correct the labels. Once you have sufficient label, you can now retrain the encoders on your own (smaller) dataset.


What are possible approaches to deal with unseparable data even after PCA? Greetings data scientists,

I am dealing with a complex classification/prediction problem and I am finding very difficult to separate the classes at all.
Even after PCA, my data (over the two PCs), look like this:

Given this dataset, what approach could be best to accurately predict the classes?

Background: I am trying to classifying modelled problems via the best approach/algorithm to solve them.

Many thanks,

Is your problem unsupervised? If not, the data does not need to be "visually" separable in two dimensions, you can use classifiers in multiple dimensions to create separations for you (linear or nonlinear, just depends on the classifier).

If your problem is unsupervised try other methods for clustering data such a K-Means or tree-based clustering (i.e. hierarchical).


Key pixels, key "features" detection in CNNs I am working on a dataset but I don't know what the labels mean.
I was wondering if using CNNs there was a way to understand which pixels where most significant for the network.
A little bit in the way that running an SVM would show which pixels are the most usefull for the classification.
Or, if there is any way to understand what my  Deep Learning algorithm is training on?

To give a little more detail I'm using VGG on the celebA dataset getting 85% accuracy on test-set. Would you know how to get back to the label's meaning and therefore maybe crop part of the image that is useless.

Class Activation Map is what I was looking for


how to change keras backend in windows? In ubuntu I can find

~/.keras/keras.json

and change the backend
but in windows, I can not find this file. How can I  change keras backend in windows.

You can create keras.json in the same way as on linux. The 'backend' key will have either 'tensorflow' or 'theano'. Place the file in %USERPROFILE%/.keras


Reducing Regression to Classification If a regression problem is reduced to classification, does minimizing the classification loss translate to minimizing regression error and hence better regression performance?

No, minimizing the classification loss does not translate to minimizing regression error and does not yield better regression performance.

This is a core tenant of Measure Theory. Regression measures outcome variables on ratio, interval, or ordinal scales. Classification measures outcome variables on nominal scales. You can always go from more complex scales (i.e., ratio, interval, or ordinal) to simpler scales (i.e., nominal). There are no guarantees about going back. This is technically called a non-injective mapping.

One way to think of it is that you are picking a downsampling scheme to map the data from regression to classification, and there are many possible mappings back from classification to regression. There are no guarantees on which mapping was used. Once downsampled, the data has to stay in the lower sampling space.


Anomaly detection using clustering of highly correlated Categorical data My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.

I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.

Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.


Conventions for dimensions of input and weight matrices in neural networks? Im currently learning neural networks and I see conflicting decsriptions of the dimensions of weight and input matrices on the internet. I just wanted to know if there is some convention which more people use than the other.
I currently define my input matrix X with the dimensions of:
(m x n)
Where m is the number of samples and n is the number of features.
And I define my weight matrices with the dimensions:
(a x b)
Where a is the number of neurons in the layer and b is the number of neurons in the last layer.
Is that conventional or should I change something?

I would not say there is such a convention for it per se (if anyone has anything to comment on this, I would also like to know).
I think to make it clearer how the layer's input x interacts with the weights W, it might better to define the dimensions as the following:

x: (m x n)
W: (n x k)
bias term b: (k)

m remains as the number of examples. n represents number of input features and k represents number of neutrons in the layer.
As we know, we compute the output of the layer y as Wx + b. Therefore, the resulting output matrix will be  (m x k)


(Newbie) Decision Tree Randomness I'm starting at Data Science and, to get something going, I just ran the code from Siraj Raval's Intro to Data Science video. He implements a simple Decision Tree Classifier but I couldn't help but notice that, given the same training set, the classifier doesn't always yield the same prediction (nor the same fit apparently); which I happen to find terribly weird, since, from what I've learned, a Decision Tree is supposed to be deterministic.

The only thing I can think of that could be causing the randomness would be that the branches are being chosen at random at some point because 2 options might be identically valued. I would say this could be corrected with a little bit more training data, but even if I add 5 more people, nothing changes. Does anybody have an explanation for what's going on?

Following is the code (in Python) from the video in a for loop to count how many predictions for male and female the Decision Tree has yielded.

from sklearn import tree
from sklearn.svm import SVC

n_male_pred_tree = 0
n_female_pred_tree = 0

n_male_pred_svm = 0
n_female_pred_svm = 0

for i in range (1,1000):
    # This loop tests the consistency of the CLF
    # The Decision Tree is not very consistent (It's 50-50)
    X = [[181,80,44],[177,70,43],[160,60,38],[154,54,37],
         [166,65,40],[190,90,47],[175,64,39],[177,70,40],
         [159,55,37],[171,75,42],[181,85,43]]

    Y = ['male','female','female','female',
         'male','male','male','female',
         'male','female','male']

    tree_clf = tree.DecisionTreeClassifier()
    svm_clf = SVC()

    tree_clf.fit(X,Y)
    svm_clf.fit(X,Y)

    tree_prediction = tree_clf.predict([[190,70,43]])
    svm_prediction = svm_clf.predict([[190,70,43]])

    if tree_prediction == 'male':
        n_male_pred_tree += 1
    else:
        n_female_pred_tree += 1

    if svm_prediction == 'male':
        n_male_pred_svm += 1
    else:
        n_female_pred_svm += 1

print(f"MALE pred Tree: {n_male_pred_tree}")
print(f"FEMALE pred for Tree: {n_female_pred_tree}")
print("\n")
print(f"MALE pred for SVM: {n_male_pred_svm}")
print(f"FEMALE pred for SVM: {n_female_pred_svm}")

From sklearn:

  The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

If you manually set the random_state variable when you create your tree object, you'll find that it does become deterministic.

In simpler terms, the data you are feeding it is a little small, and there are several splits that have the same information gain, so the split that is chosen is subject to random factors.


Lower Variance vs. Higher Validation Scores So I'm trying to compare between two models, say model(1) has training accuracy of 90% and validation accuracy of 86%, while model(2) has training accuracy of 87% and validation accuracy of 85%.
Now, model(1) has a better validation score, but with high variance, and model(2) has a lower variance, but a slightly worse validation score.
Which one should I pick? assuming these are the best results we'll ever get.
I'm new in this, but my intuition is pushing me towards picking the more stable model with lower variance, but I would like to get feedback from more experienced professionals.

It seems like both models are on par in terms of performance. It will be interesting to see what would happen if you combine predictions from both models via a simple average. In many cases an ensemble of models has shown to yield better performance than any single model.
So maybe it turns out you want to keep the best of both worlds by averaging.


Unable to use readline() for function() input I recently started learning R and was trying to make a script that would give specific values of a probability distribution. Ideally, in this script, there would be a list of probability distributions with an associated number. Users would be able to enter a number and it would print out a message while giving them a bunch of variables relevant to the specific probability distribution to enter.

This is my code:

DistriName <- readline(prompt = "Enter Distribution: ")
DistriName <- as.numeric(DistriName)
function(DistriName) {
  if(DistriName = 1) {
    print("You have selected the binomial distribution")
  } else {
    print("I am not familiar with the distribution. Sorry.")
  }
}

For some reason, it keeps giving me this error:

Error: unexpected '}' in "  }"
Error: unexpected '}' in "}"

I have checked multiple times for any missing opening brackets, but could not find anything.

Try

if (DistriName == 1)

(two == means "is equal“ in a logical way)


Best ML model for predicting yearly data with many blocks? I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect.

I would like to use a machine learning model for the current dataset that I have gathered. The dataset has about 8000 blocks (or entities), and each of these blocks has 7 features. Each feature has around 10 yearly values, which are counts (ex: 2005=1, 2006=2 ... 2014=0, 2015=3). The long stretch goal would be able to predict the 11th, and even 12th, yearly value for each feature.

I would really love to apply machine learning to this dataset. I've looked into the following because they seem to be the best solution to my problem: linear regression models, LSTM neural networks, and time series forecasting, although I am still unsure of how my dataset can fit into these certain models.

Please let me know if I can offer any more information!

Your data consists of time series. Recurrent neural networks, RNNs, are applied successfully to problems like this data. Its structure is appropriate for you. Especially, you can use LSTM which is very powerful for time series data. Also there are a lot of papers in arxiv about its capabilities, pros and cons.

I think, linear regression can fail for your task.

I suggest you to read this and this articles to understand how LSTM works and what you can do with it. Then you can decide is it appropriate for your problem or not.


Which cloud platform to maximize my impact as a data scientist? I am looking to pick up the knowledge/software skills to move towards becoming an end to end deep learning engineer.  By this I mean handling the following on my own:

preprocess big data at low latency
design & train deep learning models on massive data
deploy models to serve predictions at massive scale
stream/preprocess incoming data to update models in real time

Which cloud platform would you choose to do this?

GCP:  Allows me to do the above with the minimum effort (serverless model hosting, model versioning etc), however, it ties me to tensorflow (I'm an MXNet fan).  Looks like I need to pick up apache beam for distributed data preprocessing...
AWS: Maximum flexibility but seems far less clean.  Appears to be more suited to a team of 5 experts wanting to achieve the above.

What software would you choose?

I'm essentially looking to pickup the minimum amount of things to have the maximum impact.
Currently spend most of my time using python + MXNet + EC2 and am comfortable with (2)

It has been my experience that transitioning from local modeling to large-scale distributed programming is a lot more work than most data scientists realize, and leaves little room for anything BUT data engineering, similar to what @Emre said above.

If you're developing an infrastructure yourself (say, spark) using GCP or AWS VMs, installing and maintaining that is a LOT of work.  This is doubly true if you're running a multi-tenant system and/or supporting production jobs.  You will constantly be trying to solve the 'why didn't my job run?' or 'why does it take my job 14 days to run?' problems.

If you're using the built-in data science infrastructure to those systems (Redshift, Athena, ElasticSearch, etc), you can save some time but it is still EXTREMELY non-trivial to manage.  There is a reason why, year after year, Data Scientists' favorite tools always include Databricks or similar -- because managing these things is a pain, and requires a completely different skillset than actual modeling.

All that being said, I have two suggestions.  First, AWS is more mature and has more community answers that help overcome hurdles than GCP does.  You will run into issues with its IAM systems (which become a necessary evil of responsible big data engineering), and the quirkiness of its various products (lambda, for instance, will only run < 3m scripts), but overall everything you're trying to do has already been done and documented by someone else.  It is the better choice, IMO.

However, I'd urge you to take a closer look at a managed platform like Qubole or Databricks (I work for neither).  I set up and maintained a Qubole / AWS environment for over a year.  I learned a ton about data engineering, architecture, elastic spark infrastructure and all the nuances / pitfalls / limitations of distributed computing that they don't tell you about in the rote documentation, but was still able to maintain a functioning system.  My use case didn't include deep learning, but they could have, by me turning a few knobs.

Then, once you have taken your lessons from these environments that work, you can architect, deploy & support your own big data infrastructure to your heart's content (which will now be your full time job).  Hope that helps.


I want to detect similar meaning in sentence "in my final year web based system" When students submit a topic for his/her final year research, the system should be able to detect that this topic was carried on by some student or already exists.

How can I be able to do it? If its a model, how can embed python in php database?

The simple option is to implement it as a text search, preferably enriched with tf-idf weights and/or ignoring stop words: in theory all the possible topics (documents) are encoded as tf-idf vectors. When a query is provided it's also encoded as a tf-idf vector, then compared against all the documents vectors, typically with cosine similarity. The results are then ordered by the similarity score.

In practice various efficiency techniques can be used (the most obvious one is not to represent the vectors as sparse vectors).

There are obviously more advanced techniques, but it can get pretty complex so I would start with this simple option first and then consider improvements later.


Multi-image superresolution using CNNs I'm trying to write a program that can take multiple low-resolution images as inputs and output a high-resolution image.

My understanding is that for single-image superresolution, Convolutional Neural Networks work great. I can just take a network with just three convolution layers as described here. I can then train the network with a huge dataset of low-res images and their corresponding high-res images (with mean-squared error cost function), and it should theoretically work.

If I want to achieve greater accuracy, though, I can gain more information about a potential high-resolution image by looking at multiple low-resolution images.

So my question is: How would I modify this neural-network based algorithm to be able to take multiple images (of the same object) as input, and output a more accurate high-resolution image as a result? I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.

(Sorry if anything I said is incorrect; I know virtually nothing about this field, so any advice helps!)

I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.

I agree with your analysis that this problem presents different challenges than single-image super-resolution.

In the single image case, we know exactly where everything should be in the output image, because it is just a higher resolution version of the input.

So why not try and extend this idea to your case? Here is how you might do that:

Concatenate each low-res image and a low-res version of the output image as each training and test example
For prediction concatenate each low-res image and duplicate one of them to specify the desired orientation of the output

So if you have n 3-channel h x w input images in each training example, your input shape should be (h, w, 3*(n+1))

This should give the network enough information to learn the correct transformation.


Mathematical formulation of Support Vector Machines? I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused.

Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one on each side of hyperplane $\pi$.
So the equation of the margin maximizing plane $\pi$ can be written as,
$$\pi:\;W^TX+b = 0$$

If $y\in$ $(1,-1)$ then,

$$\pi^+:\; W^TX + b=+1$$
$$\pi^-:\; W^TX + b=-1$$

Here $\pi^+$ and $\pi^-$ are parallel to plane $\pi$ and they are also parallel to each other. Now the objective would be to find a hyperplane $\pi$ which maximizes the distance between $\pi^+$ and $\pi^-$.

  Here $\pi^+$ and $\pi^-$ are the hyperplanes passing through positive and negative support vectors respectively

According to wikipedia about SVM I've found that distance/margin  between $\pi^+$ and $\pi^-$ can be written as,
$$\hookrightarrow\frac{2}{||w||}$$
Now if I put together everything this is the constraint optimization problem we want to solve,
$$\text{find}\;w_*,b_* = \underbrace{argmax}_{w,b}\frac{2}{||w||} \rightarrow\text{margin}$$
$$\hookrightarrow \text{s.t}\;\;y_i(w^Tx\;+\;b)\;\ge 1\;\;\;\forall\;x_i$$

Before proceeding to my doubts please do confirm if my understanding above is correct? If you find any mistakes please do correct me.

How to derive margin between $\pi^+$ and $\pi^-$ to be $\frac{2}{||w||}?$ I did find a similar question asked here but I couldn't understand the formulations used there? If possible can anyone explain it in the formulation I used above?
How can $y_i(w^Tx+b)\ge1\;\;\forall\;x_i$?

Your understandings are right.

  deriving the margin to be $\frac{2}{|w|}$

we know that $w \cdot x +b = 1$

If we move from point z  in  $w \cdot x +b = 1$ to  the $w \cdot x +b = 0$ we land in a point $\lambda$. This line that we have passed or this margin between the two lines $w \cdot x +b = 1$ and $w \cdot x +b = 0$ is the margin between them which we call $\gamma$

For calculating the margin, we know that we have moved from z, in  opposite direction of w to point $\lambda$. Hence this margin $\gamma$ would be equal to $z - margin \cdot \frac{w}{|w|} = z - \gamma \cdot \frac{w}{|w|} =$ (we have moved in the opposite direction of w, we just want the direction so we normalize w to be a unit vector $\frac{w}{|w|}$)

Since this $\lambda$ point lies in the decision boundary we know that it should suit in  line $w \cdot x + b = 0$
Hence we set is in this line in place of x:

$$w \cdot x + b = 0$$
$$w \cdot (z - \gamma \cdot \frac{w}{|w|}) + b = 0$$
$$w \cdot z + b  - w \cdot \gamma \cdot \frac{w}{|w|}) = 0$$
$$w \cdot z + b  = w \cdot \gamma \cdot \frac{w}{|w|}$$
we know that $w \cdot z +b = 1$ (z is the point on $w \cdot x +b = 1)$
$$1  = w \cdot \gamma \cdot \frac{w}{|w|}$$
$$\gamma= \frac{1}{w} \cdot \frac{|w|}{w} $$
we also know that $w \cdot w = |w|^2$, hence:
$$\gamma= \frac{1}{|w|}$$
Why is in your formula 2 instead of 1? because I have calculated the margin between the middle line and the upper, not the whole margin.

  How can $y_i(w^Tx+b)\ge1\;\;\forall\;x_i$?

We want to classify the points in the +1 part as +1 and the points in the -1 part as -1, since $(w^Tx_i+b)$ is the predicted value and $y_i$ is the actual value for each point, if it is classified correctly, then the predicted and actual values should be same so their production $y_i(w^Tx+b)$ should be positive (the term >= 0 is substituded by >= 1 because it is a stronger condition)

The transpose is in order to be able to calculate the dot product. I just wanted to show the logic of dot product hence, didn't write transpose

For calculating the total distance between lines $w \cdot x + b = -1$ and $w \cdot x + b = 1$:

Either you can multiply the calculated margin by 2 Or if you want to directly find it, you can consider a point $\alpha$ in line $w \cdot x + b = -1$. then we know that the distance between these two lines is twice the value of $\gamma$, hence if we want to move from the point z to $\alpha$, the total margin (passed length) would be:
$$z - 2 \cdot \gamma \cdot \frac{w}{|w|}$$ then we can calculate the margin from here.

derived from ML course of UCSD by Prof. Sanjoy Dasgupta


Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding For solving a prediction problem I'm willing to use the Factorization Machines, a model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing interactions between features in this new space.

I was told that performing the hashing trick to convert categorical features to 1-of-k binary features (using sklearn’s DictVectorizer, which returns sparse matrix) can destroy feature interaction and I should try regular one-hot encoding instead.

Can anyone explain why?

I decided to expand a bit on my comment and make a full answer.

So the reason why somebody may say that performing the hashing trick can destroy interactions is because it may map different features to the same bucket. But usually it's not a big deal.

Note that DictVectorizer doesn't perform the hashing trick:

  When feature values are strings, this transformer will do a binary one-hot (aka one-of-K) coding: one boolean-valued feature is constructed for each of the possible string values that the feature can take on. For instance, a feature “f” that can take on the values “ham” and “spam” will become two features in the output, one signifying “f=ham”, the other “f=spam”.

To do it, you need to use a different vectorizer: HashingVectorizer


Doing a cox regression, how do I analyze a continuous variable in which a lower result is worse? I have this variable that is clinically worse the lower it is. Instead of interpreting, for example, a hazard ratio of 0.9 as "per unit increase in variable x, there is a x% reduction in risk for mortality", I want to get a >1 HR (for ex 1.05) and say "per unit decrease in variable x there is a 5% higher risk for mortality".
How do I handle this using SPSS?

With a simple linear regression you have a dependent and independent variable, with an outcome model of:
y = - mx + c

If I understand your problem, y would be mortality, and x would be your clinical variable (i.e. when x increases y decreases). And it would be the same if you want to do your other example, because there is an inverse relationship between the two (i.e. when you decrease x, y increases).
It sounds like your hazard ratio is m (i.e. the relationship between the clinical variable and the mortality)
m = (y - c )/x

So to getting a HR>1 will depend on your data.


Principle Component Analysis on multiple functions I am working on a research project that uses data of geometrical structure complexities of different land coverage types (primarily cities, pastures and natural structures). My supervisor has instructed me to use PCA for the reduction of dimensions, but I have trouble understanding how it would work on my data.

That data is a collection of a hundred 2D plots of which the x-axis runs from 0 to 255 (in steps of 1) and the y-axis from 1 to 2 (in non-integer steps). The individual plots are not linear, but to some extent have the same shapes.

My problem is that for as far as I know, PCA won't work here because for every x-value there are multiple y-values, if I plot all of the individual datasets into one big graph. Also, isn't it a problem that the individual plots are nonlinear?

So my question is: would PCA work on this 'multivalued and nonlinear' dataset? If so, where could I look for an explanation on how exactly it would work?

Variations of PCA can still be applicable.

If the data is not linear, use nonlinear PCA.

It is not an issue there a multiple "y"s for every "x". PCA is unsupervised, there is no notion of targets. In PCA, there are only dimensions.

Typically, the dimensions are standardized so each dimension can be weighted independent of scale.

One caveat - PCA is domain agnostic. Given that you have physical land data, you can use domain-specific methods such as hierarchical spatial indexing.


How to use BERT in seq2seq model? I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need <SOS>, <EOS> tokens which are not trained with BERT. How should I get them ? Can I use <CLS> token as <SOS> and <SEP> as <EOS> ? Or I have to create these two embeddings as trainable Variables and concatenate them to the decoder input / labels ?

In principle, it is possible to reuse the special tokens as you describe.
However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality.
Another option would be to reuse just the embeddings instead of the whole model.


How to use Graph Neural Network to predict relationships between nodes with pytorch_geometric? Let's say I have a partly connected graph that represents members of many unrelated communities. I would like to predict the possible friendships between members of the same community: on an sliding scale between 0 to 10 how likey would they like each other? I have some characteristics of them whether they are christian, or like sports, and also some geographical features, the distance between them.

The connections could be whether or not they are friends on a social media platform. In the networks, they are not necessarily connected with edges.

I am using pytorch_geometric to build a graph for each community and add edges for connections on the social media platform. One edge for each direction, so the graph is bi-directional. Then I create Data() instances.

Data(x=x, edge_index=edge_index)

Where x is an array with node features and edge_index

x = array([[ 0,  4,  6,  0,  0,  1],
   [ 1,  4,  6,  0,  0,  1],
   [ 2,  4,  6,  0,  0,  1],
   [ 3,  4,  6,  0,  1,  0],
   [ 4,  4,  6,  0,  1,  0],
   ...])

edge_index = [[0, 1],
 [0, 9],
 [0, 10],
 [0, 11],
 [1, 2],
 [1, 7],
 [1, 12],
 [2, 3],
 [2, 6],
 [2, 13],
 [3, 4],
 ...]

Not sure what is the best route from here to train on and predict relationships. What is generally used in this case? There are a few options mentioned in the documentation: EdgeConv, DynamicEdgeConv, GCNCon. I am not sure what to try first. Is there anything available that is made for this kind of problems or do I have to setup my own MessagePassing class?

Data() accepts an argument y to train on nodes. Can I actually use pytorch_geometric for this kind of problem or do I have to go back to pytorch?

Seems the easiest way to do this in pytorch geometric is to use an autoencoder model. In the examples folder there is an autoencoder.py which demonstrates its use. The gist of it is that it takes in a single graph and tries to predict the links between the nodes (see recon_loss) from an encoded latent space that it learns. The example is of one large graph, for my purposes I had multiple graphs which meant each one got their edges split and trained separately.


How to cluster n-grams? I just wanted to know how to cluster n-grams based on their semantics. Like clustering together n-grams that are semantically similar by leveraging the distributional hypothesis suggesting that similar words appear in similar contexts.

Distributional semantics models are based on representing an instance with the words that represent its meaning. Typically if one is interested in representing the meaning of a word $w_i$ then the words that represent its meaning are the ones which appear close to it, e.g. withing a window of N words preceding/following it.

Example where $w_i$ is the target word with a window +/- 2:

$w_1,w_2,...,w_{i-3},\mathbf{w_{i-2},w_{i-1}},w_{i},\mathbf{w_{i+1},w_{i+2}},w_{i+3},...,w_N$

So the "meaning" of $w_i$ in this instance would be represented by the four words in the context window: $w_{i-2},w_{i-1},w_{i+1},w_{i+2}$. By collecting occurrences of $w_i$ everywhere in the document one obtains a set of contexts that can be used in different ways. For example one could build a context vector $a_1,...a_{|V|}$ over the vocabulary $V$ where each cell $a_k$ contains the frequency of $w_k \in V$ in the context of the target word. By applying the same process with different words, each word can be represented with its context vector and then various operations can be made on these vectors: similarity measures, clustering, etc.

Generally clustering based on words semantics is done with specific methods such as Latent Semantics Analysis.

In theory the method can be applied to any unit (e.g. n-grams), but more complex units require more data, more memory and more computing power.


Best methods to solve class imbalance problem and why? I have a data set where I need to detect fraud. 99% are not fraud and 1% are.

What methods can be used to solve problems where classes are imbalanced?

There are 2 Techniques:

Oversampling: There are many techniques under this, ROSE and SMOTE are the most famous techniques used for oversampling. In ROSE it just increases the minority classes. In SMOTE it synthetically generates more number of rare minority classes for balancing. Most of the Scenarios SMOTE gives better results than ROSE but you should try both. Other than that there is just another techniques which just duplicates the records to the make it equal n number. This Link, is for implemenation of SMOTE in Python.
UnderSampling: There are many techniques under this too, but this Link-1, Link-2 gives you better idea about undersampling. Generally I don't prefer Undersampling, as you would loose some infomation.

The reason why you need to use these techniques are, if we don't use then the model accuracy will be very high it will be able to predict with 99% accuracy correctly the cases which are not fraud, which we don't want. if it can predict with same accuracy to find out fraud, then that is a great insight.
This can be achived only by using either of the above techniques.

Do have a look and Let me know if you have any additional questions.


How to draw a hyperplane using the weights calculated In the simple example where I have n input neurons, I can consider this to be a point in a n-dimensional space.

If the output layer is just one neuron with value 0 or 1, if I get convergence, the neural net should define a hyperplane dividing the two classes of points - those mapped to 0 and those mapped to 1. How do I calculate this hyperplane using the (n,1) matrix of weights I calculated?

I think a better to way to think about a neural net as you have it written is that it is a function.

$$f_{nn} : \mathbb{R}^n \rightarrow \{0, 1\}$$

which is generally a non-linear decision boundary. You did not indicate your network structure and activation function. If your data is linearly separable then a single layer perceptron will define weights for you in the input layer, plus a bias input node for the intercept.

If you specifically want a separating hyperplane you could consider support vector machines. See here.

https://stackoverflow.com/a/8021219/431895


CNN implementation low accuracy on MINST data I'm trying to implement VGG11 (Model A of Table 1 from this article) on the MINST dataset but I'm getting ~10% train & test accuracy (as bad as random guessing). I had to resize the MINST data from 28x28 to 32x32 to fit the CNN architecture. This is what I did:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPooling2D, Flatten
from keras import optimizers, utils
from PIL import Image, ImageFilter
import numpy as np
import tensorflow as tf

# Preprocessing

x_size = 6000 # Changed to reduce training time
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train_ = np.ndarray((x_size, 32, 32))
x_test_ = np.ndarray((x_test.shape[0], 32, 32))

# Resizing inputs to 32x32
for i in [0, x_size-1]:
    im = Image.fromarray(x_train[i], mode=None)
    im = im.resize((32, 32))
    x_train_[i] = np.array(im)
for i in [0,x_test.shape[0]-1]:
    im = Image.fromarray(x_test[i], mode=None)
    im = im.resize((32, 32))
    x_test_[i] = np.array(im)

x_train_ = x_train_.reshape(x_train_.shape[0], 32, 32, 1)
x_test_ = x_test_.reshape(x_test_.shape[0], 32, 32, 1)

y_train = utils.to_categorical(y_train,10)
y_test = utils.to_categorical(y_test,10)
y_train_ = y_train[:x_size]

# Model A (VGG11) of Table 1: ConvNet configurations from paper arXiv:1409.1556v6

model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same', input_shape=(32, 32, 1), data_format='channels_last'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(128, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(256, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(256, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dense(4096, activation='relu'))
model.add(Dense(1000, activation='relu'))
model.add(Dense(10, activation='softmax'))

# Model compilation

model.compile(loss='categorical_crossentropy', optimizer=optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True, clipnorm=1.), metrics=['accuracy'])

# Model fitting

model.fit(x_train_, y_train_, epochs=1, batch_size=32)

# Model evaluation

score = model.evaluate(x_train_, y_train_)
print('Train loss after 1 epoch:', score[0])
print('Train accuracy after 1 epoch:', score[1])

I've tried normalizing the input, changing training sizes, increasing epochs, changing FC/filter size, and changing optimizers (and learning rate). Train accuracy is as low from both the evaluation report and TensorFlow's History report. I'm expecting >95% accuracy. What am I doing wrong?

Try by adding dropout to the network to avoid overfitting.
read the docs for more information
https://keras.io/layers/core/

and try these things as well

since the targets are integers,its better to use sparse_categorical_crossentropy than categorical_crossentropy and optimizer as Adam

model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.Adam(lr=0.001), metrics=['accuracy'])

and try by using sigmoid activation function for output layer

model.add(Dense(10, activation='sigmoid'))


Visualising feature selection results for multiple classifiers and feature subset sizes I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:
fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)

fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)

fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)

I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:
def compareAlgorithms(X_train, y_train, score):
    # Compare Algorithms
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))

    # evaluate each model in turn
    results = []
    names = []
    scoring = score

    print(score, ":")

    for name, model in models:
        skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
        #kfold = model_selection.KFold(n_splits=5, random_state=seed)
        cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
        results.append(cv_results)
        names.append(name)
        msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
        print(msg)

    return results, names

As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:

I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:
def plot(results,names, score):
    import matplotlib.pyplot as plt
    # plot for algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    plt.plot(results)
    ax.set_xticklabels(names)
    plt.show()

Which results in this plot:

The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.
Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?
Many thanks.

You will have to study and understand Matplotlib and its tweaks.
This code will do the basic work. You can extend it. Also, please go through the references that are added at the end.
import matplotlib.pyplot as plt

data = {'AUC':{'RF':[0.7,0.2,0.5,0.9,0.4], 'LR':[0.9,0.25,0.35,0.99,0.55], 'SVM':[0.3,0.5,0.8,0.6,0.7] } }
x = ['S1','S2','S3','S4','S5']

plt.plot(x, data['AUC']['RF'], marker='^', linestyle='solid')
plt.plot(x,data['AUC']['LR'], marker='o', color='r',linestyle='dashed')
plt.plot(x,data['AUC']['SVM'], marker='s', color='b',linestyle='dashdot')

$\hspace{3cm}$
References -
Matplotlib Official tutorial
Python Data Science Handbook by Jake VanderPlas
Matplotlib Lines
Matplotlib Markers
Matplotlib Colors


can someone give me an idea how to extract information for text such as invoice bills using natural language processing suppose i have text file with invoice details in it . I want to extract only some information based on my certain condition such as Mobile - 25,000 and quantity - 1 i want to extract only mobile and quantity based information and store it in text file.Can someone give me an idea how to do this.

You might be looking for regex. With this you can parse the text based on your specified search patterns.

Some info:
https://en.wikipedia.org/wiki/Regular_expression

With python you can follow this documentation in order to build your regex expressions:
https://docs.python.org/3.8/library/re.html


Too much inputs = overfitting? First question : can I mix different sorts of inputs types for example, height and age (of course my inputs are normalized)? in general, can we mix different types of inputs in a neural network ?

Second question : can too much different inputs cause overfitting ?

I am using 120 inputs neurons and 20,000 train data and I am overfitting at 53% accuracy (bad)...

Thank you.

Yes, you can mix any different sort of inputs when the scales of the features are similar, which is achieved by normalising the feature vectors.

I assume you mean too many features when you say 'too much input'

If you mean the size (number of training examples) of input data, size of input data is not directly related to overfitting. Overfitting depends on model complexity. It happens when model tries to fit to the noise of the input data and hence becomes too specific that it can't generalize well to new training data.

Any model that is "sufficiently" complex (for eg. one that contains many hidden layers, large number of neurons in each layer, whose weights are not regularized) can easily converge to give very little loss on training data (unless it converges to a different sub-optimal local minima), but will give poor accuracy on test data. But in general, on the contrary, lack of enough data often  leads to overfitting because the model tries to learn based on very few specimens which are less diverse. It's like showing a child a samples of balls containing only white and orange table tennis balls, and asking him/her to identify a blue colored ball.

Too many features can lead to overfitting because it can increase model complexity. There is greater chance of redundancy in features and of features that are not at all related to prediction.

For eg. if you're predicting quality of a tennis ball, the feature chosen as colour of the ball is irrelevant, but the network will learn from training examples and there is a chance that since people like yellow colored balls to play with, they play more often with them and those balls don't last long.


Is there a quick way to speed up ICP in python using a cached KD-tree I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor lookup. Each point cloud has roughly 500 points each. I am looking at using a cached KD tree but I am unsure about how to turn a KD tree into a cached KD tree. Is there an implementation in Scipy? SVD in python seems fast enough for my needs at the moment, but I would also be open to any known faster SVD solvers. My current code for the ICP is:

>tree=spatial.KDTree(TimeStep1)
>qHat=TimeStep2
> for i in range(10):
>         Phat=tree.query(qHat,eps=.15,distance_upper_bound=2)
>         H = np.dot(qHat.T, TimeStep1[Phat[1]-1,:])
>         U, S, Vt = np.linalg.svd(H)
>         R = np.dot(Vt.T, U.T)
>         if np.linalg.det(R) < 0:
>             Vt[2,:] *= -1
>             R = np.dot(Vt.T, U.T)
>             Yes.append(1)
>         Eu=rotationMatrixToEulerAngles(R)
>         T=TimeStep1[Phat[1]-1,:].T-np.dot(R,qHat.T)
>         T=TimeStep1[Phat[1]-1,:]-qHat
>         T=T.mean(axis=0)
>         qHat=np.dot(R,TimeStep2.T).T+T
>         qHat=TimeStep2+T.shape

Try using spatial.cKDTree, in my ICP implementation switching from KDTree to cKDTree 30 iterations went from ~160 seconds to ~50 seconds


What is the the cost of combining categorical variables? I have 2 categorical variables e.g. state and city. Missing are only in city. As opposed to throwing out all observations with missing values for city or throwing out city all together I was considering making a variable location that is a concat of the two columns e.g. state:california city:LA becomes California_LA & state: california city: Missing becomes california_None.

I am building a NN and I want to know what the cost of doing this is. Will I be required to do more computation because the unique values in the location variable will be much greater than state or city?

It depends on whether you retain the original columns or not.  You are not providing any additional information to the NN either way, so it's just a matter of how many features to compute in each batch.  That said, for the same reason, you might as well just use a value (e.g. -1 or 0) for the missing cities.  Depending on your implementation language, this is probably quicker easier to implement in terms of data engineering, and will amount to the same thing in terms of results.  Obviously the NN will need numerical values, so effectively you could just label encode the city feature and assign missing values to one of the labels.

There are some very minor considerations, such as if you combine state and city, you will end up with a larger label set for the single column, which, if you normalise to an appropriate range (e.g. 0 to 1), which NN's often benefit from and which can help to avoid exploding gradients, you will have a finer graduation than a column each for city and state.  Depending on the size and complexity of your data, this could have an impact on the NN's performance (prediction-wise) but it is likely to be a very minor effect compared to all of the other decisions you'll need to make in the process.


Project structure - many projects share same large dataset I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format.  I want these to exist separately from each other and I usually try to use the cookie cutter data science model for project structure, and keep all my data in a data folder in the root of the project.

But because this dataset is big, I don't want to have ten copies of it in the root of these ten projects.  I also don't want to put them in one big project sharing it because I feel like they don't belong together.

What's the best way to structure multiple different projects that all share the same large dataset?

A database is the best option to share data across projects.

Another option is version control. Check the csv into version control. It could be git, GitHub, or data specific version control system.


PCA with Catagorical Variable in R Is there any Package for PCA for data having Categorical variable ?

No, there aren't.  I wish there were.  It would make life a lot easier.

A categorical variable is just that:  a category.  It might be represented by a number but it cannot be treated like one.  For example, if you have a list of cars with their colors, and the colors are denoted this way:

1 = red
2 = blue
3 = green

you can't really work with them as numbers.  You can't say that blue is the average color, for instance, because the average of 1 + 2 + 3 is 2.

What you can do is create dummy variables, usually a 0 and 1, which then describe the category.  In our example:

0 = not red
1 = red

You can then create dummies for the other colors along similar lines:

0 = not blue
1 = blue

and

0 = not green
1 = green

This can be done across different columns to make things easier.  It's all part of the mungification of data, getting it into shape so you can run an analysis.


Binary Classification on small dataset < 200 samples I have a dataset consisting of 181 samples(classes are not balanced there are 41 data points with 1 label and rest 140 are with label 0) and 10 features and one target variable. The 10 features are numeric and continuous in nature. I have to perform binary classification. I have done the following work:-

I have performed 3 Fold cross validation and got following accuracy results using various models:-

LinearSVC:
0.873
DecisionTreeClassifier:
0.840
Gaussian Naive Bayes:
0.845
Logistic Regression:
0.867
Gradient Boosting Classifier
0.867
Support vector classifier rbf:
0.818
Random forest:
0.867
K-nearest-neighbors:
0.823

Please guide me how could I choose the best model for this size of dataset and make sure my model is not overfitting ? I am thinking of applying random under sampling to handle the unbalanced data.

This post might be of interest. Basically by selecting the model with the best crossvalidation score, you already account for overfitting.

Also, you should separate your dataset into two parts. For the first one (validation) you run the crossvalidation on to select a model, in your case LinearSVC. For the second one (testing) you run crossvalidation again, but this time only with LinearSVC to get unbiased estimates of the accuracy.


Can R + Hadoop overcome R's memory constraints in any case? I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.

Can R + Hadoop overcome R's memory constraints in any case?

The answer might be clear as the accepted answer to
this question implies but for me it isn't. To be more precise:

Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?

I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?

To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.

Similar to that R+ Hadoop also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.

Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing multithreading to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.

Edit:  As requested more elaboration on the last point

Simply put when looking at just R as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.

And to answer your question whether R can perform the same tasks as R+Hadoop provided if there are no time constraints

The answer is theoretically Yes practically No!. Intuitively we might think R can handle such tasks by consuming more time. But that isn't always the case. It depends more on the task.

For example, if R were doing a simple frequency Count of the number occurrences in an array of a few million records you can obviously say that even with a few threads it might be slower than R+Hadoop and finish the task after a certain amount of time. But if the task were to say something like Eigen Value Decomposition of a Matrix, such a transformation requires a lot of calculation at the element level of a matrix. Assuming that the size of that matrix is a few million x few million, this type of calculation will surely cause a memory error using just any normal application as it will bottleneck all the data it has to hold temporarily into the RAM. This bottlenecking problem might cause the whole system to fail at times but the most simple case is that of a memory error.

Using R+Hadoop solves such problems as it lets you distribute the same strategy over multiple computers with their corresponding cores. Hence it lets you handle loads much more easily than running it on the same application, this manner you can also prevent the errors which I mentioned above. This is the power of distributed computing.


Quantifying binary classification uncertainty I am designing a binary classifier and would like some ideas for how best to quantify uncertainty in the final model classification. The design process consists of first training the model, then performing ROC analysis, and finally choosing a threshold such that the probability of false alarm is ~0.01. So, when an unknown sample is passed into this classifier, it outputs a number between 0 and 1, and if the number is above the threshold, the sample is classified into class 1 (C1), or into C0 if the number is below the threshold.
My question is about how to design a confidence score around the final classification. Say my threshold is 0.9. So, if the model outputs a score of 0.91, then my sample is classified as C1, but if the output score is 0.89, then the sample is classified as C0. Intuitively, given each of these scores, I should have lower confidence in the labels than if the model output was 0.99 or 0.01, respectively. One possibility is to simply compute a normalized [0,1] distance from the score to the threshold, like: $|\hat{y} - T|/(1 - T)$ for predictions above the threshold (i.e., samples labeled as C1), and $|\hat{y} - T|/T$ for predictions below the threshold (where $\hat{y}$ is the model output and $T$ is the threshold). One issue with this is that it's not symmetric. That is, for a threshold of 0.9, a model output of 0.91 has an "certainty score" (i.e. the computation shown above) of 0.1, while a model output of 0.89 has an certainty score of 0.011.
Another possibility that occurs to me is the probability of missed detection or the probability of false alarm for instances where $\hat{y}<T$ (sample classified as C0) or $\hat{y}>T$ (sample classified as C1), respectively. So, for a model output of 0.89, I would compute $FNR = 1- sensitivity$, while for a model output of 0.91, I would compute $FPR = 1-specificity$, and use these as measures of uncertainty in the final classification.
Are there other suggested methods for computing this kind of classifier uncertainty?

First things first: be sure to use a validation set in order to determine the optimal threshold, do not use the final test set.
Another general point: it depends what you mean by "false alarm", but in general you should not assume that it is possible for the classifier to keep the error rate under a predefined value (unless you have already done an experiment which shows this, of course). If by "false alarm" you mean only one type of error then it is usually possible to keep one type of error under a threshold at the expense of the other type of error. For instance if the goal is to minimize false positive errors, then one can let the classifier make as many false negative errors as necessary in order to keep the false positive rate under the threshold.
Now about quantifying uncertainty:

The first approach that you propose makes sense, but as you said it's not symmetric and also it wouldn't allow to convey uncertainty in any practical way. Ideally you'd want to be able to say something like "there's an X% chance that the prediction is correct" but you cannot obtain that from a difference with the threshold.
As far as I understand your second approach goes in the right direction but I'm not sure if the details are clear. So what you could do is to determine experimentally the rate of error (either FPR or TPR depending on which side of the threshold) for different intervals of predicted $\hat{y}$. For example if $T=0.9$ you can calculate the FNR for every interval $[0.0,0.1]\ [0.1,0.2]\ ...\ [0.8,0.9]$. This way for any predicted $\hat{y}$ you can provide the corresponding error rate.


Hindsight Experience Replay: what the reward w.r.t. to sample goal means Referring to the paper on Hindsight Experience Replay

Is it right that sampled goals which are visited states should be followed by a positive (or non-negative) rewards in order to allow an agent learn?

On page 5 of the paper, a "Algorithm 1 Hindsight Experience Replay (HER)" scheme reads in particular:

for t = 0, T-1 do
    r_t := r(s_t, a_t, g)
    Store the transition (s_t || g, a_t, r_t, s_(t+1) || g) in R

    Sample a set of additional goals for replay G := S(current episode)
    for g' ∈ G do
        r' := r(s_t, a_t, g')
        Store the transition (St||g', a_t, r', s_(t+1)||g') in R
    end for
end for

where:

  g  : current goal
  R  : replay buffer

  All other symbols with a dash indicate that they were sampled in addition to the actual current goal within the current episode.

It means (as long as I understand) that for the sampled goals (g') the reward is now a function of action taken in state given the sampled goal. It is not very clear whether the agent will learn the task in case the reward is still the old function (which is non-positive for all states that are different from the final goal).

As an example, in a grid-world an agent gets -1 reward while not in the final cell of its destination, but with the new goals introduced, the agent's reward with respect to its current state is not r, but r' (reward after reaching goal).

Illustrating example (grid-world):

...  the reward is now a function of action taken in state given the sampled goal.

I believe the action taken is that from the original goal, not from the newly sampled goal (as you say you understand). Otherwise I think you have everything more or less correct.

We see in the first block of the algorithm, that each action $a_t$, given the current goal, g, results in the reward, $r_t$ (as usual). This is stored, along with the new state $s_{t+1}$ concatenated with the current goal (shown by the || symbol). This is highlighted as being standard experience replay.

In the second block, using the sampled (virtual) goals $g'$, we receive a virtual reward for our performance using the same action as previously $a_t$. This is repeated for some number of simulated goals, selected by a sampling strategy, of which several are discussed in Section 4.5.

I myself was wondering how many replays are sampled, as it seems that the key there is to sample enough, so that the buffer itself sees the right balance of additional goals (to reduce the reward density), but not so much that the virtual HER recordings from the second for-loop do not outnumber the real performed goal-action pairs from the first loop. In the paper (Section 4.5), this seems to be around the $k=8$ mark, where $k$ is the ratio of sampled/virtual goals to the original goals.

So I believe the sampled goals that are indeed visited states from the original goal, would indeed receive a non-negative reward.

I think the following is a key statement to help explain the intuition:

  Notice that the goal being pursued influences the agent's actions but not the environment dynamics and therefore we can replay each trajectory with an arbitrary goal assuming we have an off-policy RL algorithm like DQN ...

This is very true in life. Imagine you try to throw a frisbee straight across a field to a friend. It doesn't make it, instead flying off to the right. Although you failed, you could learn that the wind is perhaps blowing left to right. If that had just so happened to be the task at hand, you would've received some positive reward for it!
The authors sample many additional goals, which in my analogy, may be the flight dynamics of that particular frisbee, the air density/humidity etc.

The main contribution of this paper, is a method to increase the density of the reward function i.e. to reduce how sparse the reward is for the model while training. Sampling these additional goals after each attempt (failed or otherwise) gives the framework the opportunity to teach the model something in each episode.

In the grid-based example, if for example the agent doesn't reach the final goal (as its original goal), it records -1 to the replay buffer. Then other goals are sampled from the possible next steps according to a sampling strategy, $\mathbb{S}$. For If you were close to the goal, it would make sense that sampling from future states selected at random from the same episode - after the transition - that you would likely end up at the goal. It is important here to realise that the goal has changed, which allows reward to be received. I point this out because the goal usually doesn't change in grid-based games; however, the experiments in the paper were performed on a robotic arm with 7-DOF in continuous space (only the reward was discrete).

EDIT

Below is a sketch of an example path, where we reach the final goal after 10 transitions (blue arrows). I set $k = 4$, so in each of the states $s_t$, we also have 4 randomly selected goals. We then take the corresponding action $a_t$ for the current state, which is the blue arrow. If the randomly sampled goal, $g'$, happens to be the same as $s_{t+1}$, we get a non-negative reward - these are the orange arrows. Otherwise, a negative reward is returned: the green arrows.

This is an example of the random sampling strategy, as my sampled goals $G$, are states that have been encountered in the whole training procesdure (not just the current episode), even though you cannot see it in my sketch.

So here we see there are 4 sampled goals, whch do indeed return non-negative reward. That was chance. The authors do say that:

  In the simplest version of our algorithm we repla each trakectory with the goal $m(s_T)$, i.e. the goal which is achieved in the final state of the episode.

In that case, it would mean $k=1$ and is always simply where the episode ended. This would mean negative rewards in the HER portion of the algorithm for all time steps exluding the final, $t=T$, where we would reach the sampled goal.

That would indeed equate to the model having learned from an otherwise failed episode. In every single episode!


Why do I get different coefficients from Logistic regression in Python and SPSS I am a bit confused in regards to the model coefficients calculated by SPSS and sklearn's LogisticRegression.
I am getting different coefficients and intercepts for both methods.
in Python, I am running the following code:
import numpy as np
from sklearn.linear_model import LogisticRegression

vals = [0.01, 0.04, 0.07, 0.08, 0.08, 0.09, 0.10, 0.15, 0.20, 1.85, 1.93, 1.97 ,2.02, 2.09, 2.12, 2.13, 2.21, 2.25]
labels = [0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1]

X = np.array(vals).reshape(-1, 1)
y = np.array(labels)

solvers = ['liblinear', 'newton-cg', 'lbfgs', 'sag', 'saga']

for i in solvers:
    model = LogisticRegression(solver=i, random_state=0).fit(X, y)
    print(f'intercept: {round(model.intercept_[0],3)} coefficient: {round(model.coef_[0][0],3)} ... solver: {i}')

which outputs:
>> intercept: -1.012 coefficient: 1.239 ... solver: liblinear
>> intercept: -1.609 coefficient: 1.498 ... solver: newton-cg
>> intercept: -1.609 coefficient: 1.498 ... solver: lbfgs
>> intercept: -1.609 coefficient: 1.496 ... solver: sag
>> intercept: -1.609 coefficient: 1.498 ... solver: saga

(In reality, my dataset is much larger of course)
In SPSS if I use Analysie > Regression > Binary Logistic at default settings. I am getting a different coefficient and intercept.

Is there a transformation on the coefs that is done by skleanr but not by SPSS or is the Logistic regression method used by sklearn so much different from the one in SPSS. When I calculate the log_odds and the the predicted probabilities for the equations derived from either python and SPSS, I am getting quite similar results, however still a little different.
Does anyone know why this difference occurs or whether there is a common method for the two that would yield similar results?
thank you in advance.

The LogisticRegression class from sklearn by default uses an L2 penalty as a regularization mechanism, which I assume is not used in the SPSS implementation of the logistic regression. This has been a discussion point before but the main reason for this is that sklearn is focused on prediction and getting the best prediction results (where regularisation is used to prevent overfitting) instead of fitting the model to the data and finding which factors affect the outcome. You can try setting the penalty argument to 'none' to see if that gives you the same results as in SPSS.


How to tinker with CNN architectures? I was thinking of creating a CNN. Now it is known CNN takes long times to train so it is advisable to stick to known architectures and hyper-parameters.

My question is: I want to tinker with  the CNN architecture (since it is a specialised task). One approach would be to create a CNN and check on small data-sets, but then I would have no way of knowing whether the Fully Connected layer at the end is over-fitting the data while the convolutional layers do nothing (since large FC layers can easily over-fit data). Cross Validation is a good way to check it, but it might not be satisfactory (since my opinion is that a CNN can be replaced with a Fully Connected NN if the data-set is small enough and there is little variation in the future data-sets).

So what are some ways to actually tinker with CNN and get a good estimate for future datasets in a reasonable training time? Am I wrong in my previous assumptions? A detailed answer would be nice!

CNN Architectures: These are proven configurations of the combination of CNN/FC/POOL etc layers, that seem to work in commonly defined problems.

Now from your question what I understand is you want to "Tinker" with the architecture and not to come up with an entirely new configuration from scratch. One thing that can help you in this scenario is transfer learning.

Transfer Learning: It is a process of using pre-learnt weights from a similar task on altogether different data, to your current problem and data.

In transfer learning what we do is we take some layers usually 80-90 % of the layers keep the weights the for these layers and delete the weights for remaining layers, and initialise these layers with some kind of initialisation batch norm or something like that. And train this newly created model with the data you have.

This have proven to be effective in various problems.

You can come up with innovative ways to make changes to arbitrary layers of this network train them and check for error metrics. If you follow this approach whatever you are claiming in your question can be achieved.


Can an entire data frame be used as a prediction variable? I am attempting to use XGBoost in R to train a model that predicts a fixed number of target variables using all data from previous dates, as well as the two categorical variables (Cat1 and Cat2) for the current date as predictors. The original data is in this format:
╔═════════╦═════════╦══════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Date     ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
╠═════════╬═════════╬══════════╬══════╬══════╬══════╬══════╣
║ 1       ║ 2       ║ 01/01/20 ║ A    ║ B    ║ 3    ║ 4    ║
║ 5       ║ 6       ║ 02/01/20 ║ C    ║ D    ║ 7    ║ 8    ║
║ 8       ║ 7       ║ 03/01/20 ║ A    ║ D    ║ 6    ║ 5    ║
║ 4       ║ 3       ║ 04/01/20 ║ C    ║ B    ║ 2    ║ 1    ║
║         ║         ║          ║      ║      ║      ║      ║
╚═════════╩═════════╩══════════╩══════╩══════╩══════╩══════╝

And I conceptualise the training data looking like this for each row of the data frame, where the Train_DataFrame column contains all data from previous dates:
╔═════════╦═════════╦══════╦══════╦═════════════════╦═════════╦══════╦══════╦══════╦══════╗
║ Target1 ║ Target2 ║ Cat1 ║ Cat2 ║ Train_DataFrame ║         ║      ║      ║      ║      ║
╠═════════╬═════════╬══════╬══════╬═════════════════╬═════════╬══════╬══════╬══════╬══════╣
║       4 ║       3 ║ C    ║ B    ║ Target1         ║ Target2 ║ Cat1 ║ Cat2 ║ Var1 ║ Var2 ║
║         ║         ║      ║      ║ 1               ║ 2       ║ A    ║ B    ║ 3    ║ 4    ║
║         ║         ║      ║      ║ 5               ║ 6       ║ C    ║ D    ║ 7    ║ 8    ║
║         ║         ║      ║      ║ 8               ║ 7       ║ A    ║ D    ║ 6    ║ 5    ║
║         ║         ║      ║      ║                 ║         ║      ║      ║      ║      ║
╚═════════╩═════════╩══════╩══════╩═════════════════╩═════════╩══════╩══════╩══════╩══════╝

First question; is it possible to pass an entire data frame as a variable to a model?
If so, then how can this be done in a memory efficient manner, so as to avoid data duplication? I know I could have the Train_DataFrame column contain lists of each "past" data, however, this would lead to data duplication and inefficient memory usage. Is there a way to have this column contain a sub-setting function to pass to the original data frame, for example?
Or is there a better approach to the problem that I am potentially missing?
Thanks in advance.

First question; is it possible to pass an entire data frame as a variable to a model?

No, each feature must be a single value. In other words you could provide the data frame as a vector containing all the values, assuming the size is fixed: each column would correspond to a specific cell in the original data frame.

But I think an even better option in your case is to look into methods which take (chronological) sequences into account. Conditional Random Fields might be a good option, assuming you need to predict the target variables for the whole sequence?


Definition of a "lexicon" in Named Entity Recognition I am writing a paper on Named Entity Recognition and I mention that in the literature were proposed a lot of methods which make use of lexicons. However, I am struggling to find a definition of a "lexicon" which I can cite. As far as I understand, a lexicon is a list of lemmatas (word parts), whole words or word combinations where these are classified to an entity. So, for example if in a sentence i have "New York is located in" and in my lexicon I have the pattern " X is a located in" -> CITY my system will look up the pattern and see that New York belongs to the entity CITY.
Could someone explain to me if I am wrong? If so, I would like to get a definition of a "lexicon", "gazzetteer" and "dictionary", because I get the feeling that some of these are sometimes used interchangeably in papers.

As far as I know the word "lexicon" is mostly used for a simple list of words (or terms), I would say it's quite rare to use it for describing a list of patterns/rules. "gazetteer" and "dictionary" would be a bit more general in my opinion, for instance one can have a "dictionary of rules" which associates specific patterns with actions. But overall I agree with your impression that these terms are often used interchangeably, I'm not aware of any standard definition.


How to use GridSearch for LinearSVC / Random Forest with time series data I have a question related on how to use the GridSearch to find the best models for my problem with time series data.
Every 3 rows is 1 one row in the original dataset. To make my time series problem a supervised one, I parsed like the one below. This was resolved from one of my previous question.

id
Age
gender
m1
m2
m3
Label

1
20
M
12.4
34
12
0

2
20
M
13
324
34
0

3
20
M
34
232
12
0

4
45
F
1.3
32
19
1

5
45
F
14
132
19
1

6
45
f
94
232
19
1

My question is: How can I use GridSearch for example to find my best machine learning model configuration using time series data? As far as I understand, using cross validation wouldn't work in this case because of the time series nature of the dataset.
I'm not sure how to proceed with this.

So it´s a classification problem with a grid-search, without cross-validation. Yes, don´t use cv in time series data. There is an option, in which you can use cv, when you slowly start with less data and put more and more data during the process. But it´s complex.
For the grid-search are 2 opportunities. Either use GridSearchCV and define cv as none, or you use ParameterGrid().
For my interest I used this method:
https://stackoverflow.com/questions/44636370/scikit-learn-gridsearchcv-without-cross-validation-unsupervised-learning/44682305#44682305
in which is GridSearchCV defined as none.
import pandas as pd
test = pd.DataFrame({"id":[1,2,3,4,5,6,7,8,9], "age":[20,30,32,40,55,32,20,41,38], "gender":[0,1,0,1,0,0,1,1,0],
            "m1":[12.4, 30,9.4,14,19,20,34,31,16], 'm2':[34,36,22,16,22,27,42,65,13], 'label':[0,0,1,1,0,1,1,1,0]})
test.head()

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score

X = test.drop('label', axis=1)
y = test.label

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)

rf_params = {'n_estimators': [100, 200],
             'max_features': ['auto', 'sqrt'],
             'max_depth': [10, 50],
             'min_samples_split': [2, 20]}

cv=[(slice(None), slice(None))]

rf_clf = GridSearchCV(RandomForestClassifier(random_state=42),rf_params, n_jobs=-1, verbose=2, cv=cv)
rf_clf.fit(X_train, y_train)

#best parameters of model
print(rf_clf.best_params_)

#make predictions
rf_pred = rf_clf.predict(X_test)
print('Accuracy', accuracy_score(rf_pred, y_test))

The GridSearchCV part shows me:
GridSearchCV(cv=[(slice(None, None, None), slice(None, None, None))],
estimator=RandomForestClassifier(random_state=42), n_jobs=-1,
param_grid={'max_depth': [10, 50],
'max_features': ['auto', 'sqrt'],
'min_samples_split': [2, 20],
'n_estimators': [100, 200]},
verbose=2)
So this method works.
Here I used random forest, because in my own experience, random forest is in most cases very good. In big datasets, the SVC takes too much time.
PS: Before I forget, I changed the gender into numbers. You can use one-hot encoding for that or catboost, which can do this automatically. But with catboost you get different results in comparison with rf or other algorithms. So I prefer to transfer gender into numbers.