Hypertune xgboost to dealing with imbalanced dataset My training data has extremely class imbalanced {0:872525,1:3335} with 100 features. I use xgboost to build classification model with bayessian optimisation to hypertune the model in range
{learning rate:(0.001,0.1), min_split_loss:(0.10), max_depth:(3,70), min_child_weight:(1:20), max_delta_step:(1,20), subsample:(0:1), colsample_bytree:(0.5,1), lambda:(0,10), alpha:(0,10), scale_pos_weight:(1,262), n_estimator:(1,20)}.
I also use binary:logistics as the objective model and roc_auc as the metrics with booster gbtree.
The cross validation score is 82.5%.
However, when I implemented the model to the testing data I got the score only
Roc_auc: 75.2%, pr_auc: 15%, log_loss: 0.046, and confusion matrix: [[19300 7],[103 14]].
I need helping to find the best way to increase the true possitive with tolerance false positive until 3 times actual positive.

Given an imbalanced dataset and focusing on increasing your true positive rate, it is quite relevant to use the right evaluation metric (the one used to validate the model being trained on each evaluation round).
In this case, I recommend you use Precision-Recall AUC instead of ROC AUC, so you force your model to focus on the minority class.
A nice post about it can be found here
Another points to take into account could be:

increase the range of possible number of tree estimators in your hyperparameter tuning process
set the scale_pos_weight at about your (majority class samples number)/(minority class samples number)


Binary classification algorithm I have a retail customer transactional data set with features such as customer ID, product, date, number of products bought, ZIP code of customers, amount of the transaction. There already exists a business rule-based segmentation to segment customers if they are interested in coke or not based on their buying.  I have to do a binary classification to add more customers into the coke segment. Any suggestions on the kind of algorithms and approach that would suit this data set best?

Possible scenarios:

You can predict "Likely to Buy Probabilities": Create features that covers before purchase (e.g: product click, product view, purchases from same category etc). After, train a classifier and use prediction probabilities  (predict_proba_) to obtain scores. Then sort them pick x% and report them to business unit. They can offer discounts, campaigns to converge customers fast.
Try to find segment transitions. For example coke buyers move to lets say pie segments faster than vegetable segment. Identify those transitions and apply advertisement on cross categories.
If you have transaction data you should try to use apriori based algorithms and collaborative filtering methods for recommendations.
Additionally you can do RFM analysis and compare RFM segments and already created segments to find best customers of yours.

Hope this is helpful!


Why does logistic function use e rather than 2? sigmoid function could be used as activation function in machine learning.

$${\displaystyle S(x)={\frac {1}{1+e^{-x}}}={\frac {e^{x}}{e^{x}+1}}.}$$

If substitute e with 2,

def sigmoid2(z):
    return 1/(1+2**(-z))
x = np.arange(-9,9,dtype=float)
y = sigmoid2(x)
plt.scatter(x,y)

the plot looks similar.

Why does the logistic function use $e$ rather than 2?

Since you are going to minimize later on the log likelihood, there is actually no big difference between $\log 2^x=x * \log2$ and $\log e^x=x$. You see the difference is simply a constant.
Nevertheless one could argue to use $2^x$ instead of $e^x$ und also use $\log_2$ instead of $\log$ when it comes to the optimizing step. In fact it is possible to use $2^x$ and also many other functions, which show some desired properties.
Which are:

$\lim\limits_{x \rightarrow \infty}{f(x)}=1$
$\lim\limits_{x \rightarrow -\infty}{f(x)}=0$
$f(x) = -f(-x) + 1$, (symmetric in $(0, 0.5)$

Here is an example of suitable functions from wikipedia.


how to cluster inseparable data Suppose I have a dataset containing two very similar classes of data. By similar, I mean that the 'distance' between these two classes is very small. For example, one instance in Class 1 is the sum of one instance in Class 2 and some noise. If the SNR is high, we can say these two instances are similar. Because of the similarity, the dataset is inseparable. I am wondering if there exists any effective clustering algorithm that can work. Thank you very much！

As you point out, the problem is not on the clustering algorithm, but on the features. So the question comes to the particular data you might be dealing with.
As an example, say you want to cluster different kind of animals. It is in general much easier to tell an elephant from a horse apart. But if you want to distinguish between races of horses, it gets much harder. But the bottleneck lies (mostly) on the features.


Do I classify all types if they are mutually exclusive I am trying to classify an image that can represents 3 states. Up, down or Middle.

If the image does NOT represent Up or Down, then it is by default Middle.

Should I train my CNN with a dataset including all three, or just Up and Down?

Which would this make classification more or less accurate?

You won't be able to train your model correctly if you set only two classes. Because for all the samples the model will only predict Up or Down. This will cause even the Middle classes to be predicted as Up or Down, which will reduce your model's accuracy. Hence you need to keep all three classes as labels.


Precision Vs Recall Curve analysis I have the following averaged − curves with 4 models. Which one is the best?

It depends.

The problem you are trying to solve decides which among the curves is the best

For example
If you are trying to solve a problem that is like identifying cancer where the cost of false positive is extremely high. The first blue plot (model 3) is the best. However, you are compromising on recall or number of positive examples captured. It means that not all patients who have cancer will be identified as having cancer by model 3 but who ever the model says has cancer is most likely to have it amongst all the models plotted.

If your objective is just a simple marketing campaign where you would want most responders to be targetted but you are fine contacting people who are not likely to respond, model 2 which has the maximum recall.

When targetting a non responder is also expensive like a direct mail campaign, model 4 is the best because it balances both precision and recall.

Having said this response rate and sample size also need to be taken into account before arriving at which model is good.
Because if accuracy of 94% and 87% are because of just 2 accurate samples (very low response) accuracy alone might not make sense.


Hyperparameter search for LSTM-RNN using Keras (Python) From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge."

So this is more a general question about tuning the hyperparameters of a LSTM-RNN on Keras. I would like to know about an approach to finding the best parameters for your RNN.

I began with the IMDB example on Keras' Github.

the main model looks like this:

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
                                                      test_split=0.2)

max_features = 20000
maxlen = 100  # cut texts after this number of words (among top max_features most common words)
batch_size = 32

model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(LSTM(128))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          class_mode="binary")

print("Train...")
model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=3,
      validation_data=(X_test, y_test), show_accuracy=True)
score, acc = model.evaluate(X_test, y_test,
                        batch_size=batch_size,
                        show_accuracy=True)

print('Test accuracy:', acc)
Test accuracy:81.54321846

81.5 is a fair score and more importantly it means that the model, even though not fully optimized, it works.

My data is Time Series and the task is binary prediction, the same as the example. And now my problem looks like this:

#Training Data
train = genfromtxt(os.getcwd() + "/Data/trainMatrix.csv", delimiter=',', skip_header=1)
validation = genfromtxt(os.getcwd() + "/Data/validationMatrix.csv", delimiter=',', skip_header=1)

#Targets
miniTrainTargets = [int(x) for x in genfromtxt(os.getcwd() + "/Data/trainTarget.csv", delimiter=',', skip_header=1)]
validationTargets = [int(x) for x in genfromtxt(os.getcwd() + "/Data/validationTarget.csv", delimiter=',', skip_header=1)]

#LSTM
model = Sequential()
model.add(Embedding(train.shape[0], 64, input_length=train.shape[1]))
model.add(LSTM(64))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          class_mode="binary")

model.fit(train, miniTrainTargets, batch_size=batch_size, nb_epoch=5,
      validation_data=(validation, validationTargets), show_accuracy=True)
valid_preds = model.predict_proba(validation, verbose=0)
roc = metrics.roc_auc_score(validationTargets, valid_preds)
print("ROC:", roc)
ROC:0.5006526

The model is basically the same as the IMDB one. Though the result means it's not learning anything. However, when I use a vanilla MLP-NN I don't have the same problem, the model learns and the score increases. I tried increasing the number of epochs and increasing-decreasing the number of LTSM units but the score won't increase.

So I would like to know a standard approach to tuning the network because in theory the algorithm should perform better than a multilayer perceptron network specially for this time series data.

An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This representation conversion is learned automatically with the embedding layer in Keras (see the documentation).

However, it seems that your data does not need any such embedding layer to perform a conversion. Having an unnecessary embedding layer is likely why you cannot get your LSTM to work properly. If that is the case then you should simply remove the embedding layer.

The first layer in your network should then have the input_shape argument added with information on the dimensions of your data (see examples). Note that you can add this argument to any layer - it will not be present in the documentation for any specific layer.

By the way, hyperparameters are often tuned using random search or Bayesian optimization. I would use RMSProp and focus on tuning batch size (sizes like 32, 64, 128, 256 and 512), gradient clipping (on the interval 0.1-10) and dropout (on the interval of 0.1-0.6). The specifics of course depend on your data and model architecture.


Change values of a particular column to value_count() Suppose i have the following data set. I need to replace the value of a particular column by the value_count(). I saw few posts where it is done for the entire data set. I need to do it for a particular column.

data = pd.DataFrame({'Item 1': ['apple', 'potato', 'cheese', 'banana', 'cheese', 'banana', 'cheese', 'potato', 'egg'],
          'Item 2': [1,5,2,7,8,4,9,0,3],
          'Item 3': ['a','b','a','c','d','c','e','e','e']})

How do i replace the values of column 'Item 1' by the value_counts()?

I tried the following code.

data.apply(lambda x: x.map(x.value_counts()))

But this is applying to whole data. I need to do it for one column.

The resulting value should be of the form -

'Cheese':3

'Potato':2

Try this:

# Do value_count outside of loop to avoid doing it multiple times
vc_item1 = data['Item 1'].value_counts()

data['Item 1'].apply(lambda x: vc_item1[x])

But if understand your comment correctly you want both key and value in each cell:

vc = data['Item 1'].value_counts()

data['Item 1'].apply(lambda x: str(x) + ':' + str(vc[(x)]))


How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline? As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and still don't) have the hardware to fully train models from scratch.

Lately, I've been dealing with more and more images, and the need to feed data on the fly during training (fine-tuning) has become important. My current setup is essentially the legacy train.py script for training and feed_dict method for inference.

I've been reading a lot on Estimators API and their usual pipeline, but I just cannot find tutorials or help on how to use pretrained models like I have with that pipeline; all examples work on image classification and do not mix dataset creation and model training, which is a real puzzle.

*Tensorflow r1.14

Step 1: Dataset creation

So, given my data in the following format:

data = [
  [img_path_1, boxes_1],
  [img_path_2, boxes_2],
  ...,
  [img_path_N, boxes_N]
]

Where each boxes_i is in the format:

[
  [label_A, xa, ya, xb, xb],
  [label_B, xa, ya, xb, xb],
  [label_A, xa, ya, xb, xb],
  ...,
  [label_D, xa, ya, xb, xb],
]

According to some tutorial and tweaked for object detection instead of classification, it seems like I would need to process images like this (further preprocessing and batching functions left out for simplicity):

dataset = tf.data.Dataset.from_tensor_slices(data)

def load_image_and_annotations(path_and_boxes):
    path, boxes = path_and_boxes
    image_string = tf.read_file(path)

    image = tf.image.decode_jpeg(image_string, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    return (image, boxes)

dataset = dataset.map(load_image_and_annotations, num_parallel_calls=...)

and then make use of the tf.data.Iterator object to traverse the dataset as needed.

Step 2: Training

Here is mostly where I'm unsure of how to proceed. The pre-made estimators and custom estimators tutorials seem inappropriate for my task since I already have a frozen model that I wish to convert to an estimator (instead of fully building one).

How should I proceed to transition from my old setup to this new pipeline ?

After a lot of research, I came to the conclusion that this is doable, but would require a lot of work.

Essentially, the steps to achieve such a transformation are as follows:

Use this version of the model exporter (or any other solution) to re-export the savedmodel with its variables (by default, the Model Zoo does not give the variables, only an inference-ready model).
Load the model using the SavedModel API, i.e. loaded = tf.saved_model.load(...).
Extract the inference function: inference_func = loaded.signatures['serving_default'].
Define your loss function. This part is the most complex because the existing functions are part of the Object Detection API which is pretty much outdated. Tensorflow is already working on the upgrade of the API but at the time of writing this, they still expect a couple more months to finish it.
Set up your Dataset.
Train your model.

The best option is probably to wait for the new release of the official OD API.


The data in our relational DBMS is getting big, is it the time to move to NoSQL? We created a social network application for eLearning purposes. It's an experimental project that we are researching on in our lab. It has been used in some case studies for a while and the data in our relational DBMS (SQL Server 2008) is getting big. It's a few gigabytes now and the tables are highly connected to each other. The performance is still fine, but when should we consider other options? Is it the matter of performance?

A few gigabytes is not very "big". It's more like the normal size of an enterprise DB. As long as you go over PK when joining tables it should work out really well, even in the future (as long as you don't get TB's of data a day).

Most professionals working in a big data environment consider > ~5TB as the beginning of the term big data. But even then it's not always the best way to just install the next best nosql database. You should always think about the task that you want to archive with the data (aggregate,read,search,mine,..) to find the best tools for you problem.

i.e. if you do alot of searches in you database it would probably be better to run a solr instance/cluster and denormalize your data from a DBMS like Postgres or your SQL Server from time to time and put it into solr instead of just moving the data from sql to nosql in term of persistence and performance.


How can I imporve accuracy for text classification and mapping using SVM? I am working on a problem where I need to predict the text corresponding to another text in my training data file. For example: if I have value like the software in one of my columns and another corresponding column holds a value adobe pdf for it then my algorithm should be able to predict the same for my test data as well. For example, if my test data has Tableau then the predicted category should be software corresponding to it. here are data options like Software, Tax, etc and corresponding to them there are subcategories. My job is to predict the right sub-category against the primary category in my test data.

What should I do to increase my accuracy? Is n_ngram affecting the accuracy a lot? I could see some but not a lot.

from sklearn.featureextraction.text import CountVectorizer from sklearn.featureextraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import LinearSVC
#preparing the final pipeline using the selected parameters
model = Pipeline([('vectorizer', CountVectorizer(ngramrange=(1,4))), ('tfidf', TfidfTransformer(useidf=True)),
('clf', OneVsRestClassifier(LinearSVC(class_weight="balanced")))])

I have already used SVM and linear SVC for doing the classification but my accuracy is only 78%. My training data set is in German with no stop words as these are categories and not plain long text.

A general good approach for high dimensional text data would be to use a word embedding and neural networks. If you want to stick to the models and approach you are using, I suggest you don't over-engineer your dataset and first try with simple bag of word approaches and while cross-validating try different n-grams, pos-tagging but I suspect the very high dimensional feature space is bringing your accuracy score down.


Does it make sense to use TF-IDF to extract most important tokens from a corpus? I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus.

My understanding of TF-IDF is that it is calculated per token per document, so the calculated weights are relative to a given document in the corpus. Is there a way to use TF-IDF to recover the most significant terms in the entire corpus, or is this the wrong approach? If the latter, what would be a more appropriate NLP approach?

Provided limited information & context you have provided, I would suggest you to look for feature selection when each dimension belongs to a word. Feature selection will give you most important words. Most important words in the sense, words deciding the decision surface of the model.


Convert bin model to pickle I trained a word2vec model using Gensim library which is of type .bin
Q1: can we convert this trained model in bin format to pickle?
Q2: would it speed up the execution time?

Pickle is not a format but a protocol to serialize a python object to binary format.
In the documentation of gensim, you can read they actually use pickle for serializing to binary format.
You can of course use pickle library instead of the .save method in gensim, but it will be exactly the same, since gensim already use pickle. I would recommend you to use .save method however, because gensim likely has better way to handle the hierarchical structure and dependencies of your model.


What does the activation of a neuron mean? In a neural network, each neuron will have it's activation.
But what the activation mean?

Does it just mean nothing but a temporal value to produce the final result or it's has something to with our understanding to the problem?

For example, considering a neural network to recognize a handwritten character from a picture, I wonder if it's possible that a neuron's activation represents the how the pixel matches the specific value in a small part of the picture?

The activation of a neuron is mathematically nothing but a function of its input. Consider a neural network with one hidden layer and one input vector $\mathbf{x}$. The input $a_j$ of neuron $j$ can be written as:

$a_j = w_j^T \mathbf{x} + b_j$.

The activation of neuron $j$ is then a transformation $g: \mathbf{R} \rightarrow \mathbf{R}$ of the input. For exmaple, one can use the sigmoid activiation function

$sig(a_j) = \dfrac{1}{1 + \exp(-a_j)} $

The weights $w$ are chosen to minimize a loss function and not for the sake of their interpretation. Nevertheless, one can try to interpret the activation of a neuron as internal representation of the input. One of the nicest examples I've seen comes from the Rummelhart et al (1986), Figure 4. In that paper, two family trees were fed into a neural network. The family trees represented an Italian and an English speaking family comprising three generations. Among other things, when activating one name of the tree, the neuronal activities represented whether the person was from the English or Italian tree and which generation the person embodied.


How to yield better AUC score? I have a dataset with 5K records and 60 features focused on binary classification. Class proportion is 33:67

Currently I am trying to increase the performance of my model which is stuck at F1-score of 89% (majority) and 75% (minority) class and AUC of 80%.

I tried Gridsearchcv and feature engineering. Though I don't explicity call out the best parameters on Gridsearch below, I guess when I fit, it takes the best parameters only. But nothing seems to help.

Does this mean my data has issues? When I mean issue, I am not talking about missing values. I mean the way the data was extracted. Can it be data entry issues?

This is what I tried for gridsearchcv. Am I doing it right?

import xgboost as xgb
parameters_xgb = {
    'learning_rate': (0.1,0.01,0.05,0.5,0.3,1),
    'n_estimators': (100,200,500,1000),
    'max_depth':(5,10,20),}
xg_clf = xgb.XGBClassifier()
xgb_clf_gv = GridSearchCV(xg_clf,parameters_xgb,cv=5)  # using cross validation with best hyperparameters
xgb_clf_op = xgb_clf_gv.fit(X_train_std,y_train)
y_pred = xgb_clf_op.predict(X_test_std)
cm = confusion_matrix(y_test, y_pred)
print(cm)
print("Accuracy is ", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))

I also tried catboost and gb. The AUC is only around 80-82% throughout in test data.

I would not necessarily call it data issues. There is always some threshold that you just can not surpass, depending on the dataset ofcourse. Generally feature engineering and understanding the data will yield much greater increases than just hyp.par. optimization, which as you can see from the picture, yields often marginal increases (there is a case where its worst than default parameters)


Determine if a piece of text is a job ad I am interested in learning more about the field of ML and more specifically in text classification.

Given a piece of text, is there any technique or algorithm to determine if that piece of text refers to a job ad or not?

Thank you.

For text classification in your case, it can be solved by training a binary classes classification network.
Main steps as follows:

1. Training data preparation

Two classes data are needed, one class contain job ad text, and the other does not contain job ad.

2. Building the network

Building a binary classification network with Keras, Caffe or any other DL library you like.

3. Train the network

Training the network on the data prepared above.

4. Text class prediction

Use the trained network to predict the text class you want to deal with.

It’s a easy classification task for neural network: )


Problem in input shape Keras-LSTM I want to make a predictor using Keras LSTM model. I have a sequence of places visited. The task is to predict the last destination.
I went through different examples but it seems I am not able to shape the input properly.
I am stuck on how to prepare in my program my data to give them to the LSTM model.
Here is a minimum code related to my problem.
input_csv ='input.csv'
max_features = 6
df = pd.read_csv(input_csv)

df.head()
#Cafe =0.1, Park =0.2, Shop =0.3, Home=0.4, Movie = 0.5, School=0.6

For example, Person A  visited
cafe (0.1) ->   park(0.2) ->    cafe (0.1) ->   park(0.2)   and finally ended at school(0.6). The desired task is to predict y (end) based on input X (place1,place2,place3,place4).
X = df.iloc[:,1:5].to_numpy()
X_train = X.reshape(6, 1, 4) # X.reshape(samples, timesteps, features)
X_train.shape
#(6, 1, 4)

y = df.iloc[:,-1].to_numpy()
y.shape
#(6,)

#Building model
model = Sequential()
model.add(LSTM(6, dropout=0.2, recurrent_dropout=0.2, input_shape=(None, 1)))
model.add(Dense(max_features, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Train
model.fit(X_train,y)
pred = model.predict(X_train)
predict_classes = np.argmax(pred,axis=1)

ValueError: Error when checking input: expected lstm_4_input to have shape (None, 1) but got array with shape (1, 4)
I would appreciate if someone could help me in clearing my confusion or pointing to some explanations. Thank You.

Few points that i want to mention :

First of all, Here i think the input u are preparing where I see doubt. The objective is to predict end location. The input shape u prepare is a doubt for me because your sequence_length should be "4" and you have an initial hidden dimension if "1". If I am not wrong then your final share should be (batch_size,sequence_length,hiiden_dimension) = (6,4,1). Because this is what dimension shape LSTM is expecting always.
Second is change your loss function because u want to do predict from more than one class as mentioned like Cafe =0.1, Park =0.2, Shop =0.3, Home=0.4, Movie = 0.5, School=0.6. Chnage your loss function to categorical_crossentropy in fit() function. Also change activation='sigmoid' to activation='softmax'.

I think these changes u will have to do according to your use case. Please feedback my answer.


Memory efficient structure for membership checking without false positive The initial task can be described like this:
I have a requirement to deduplicate HUGE list(potentially billions of items) without storing the original items - it's simply unaffordable

All I need to know is answer to the question "Has my system ever seen this element before?"

The most close data structure I was able to find so far is a bloom filter, but it has false positives which better to avoid in my task as it results in data loss

For example providing I account to store at least 2^32 items, with positive error rate of just 1%(which means 1% of all urls won't be visited) I would need at least

n = 4,294,967,296, p = 0.01 (1 in 100) → m = 41,167,512,262 (4.79GB), k = 7

4.79GB of memory...

The task itself is a high scale web crawler, so I need to keep track of already visited urls(or sha1 hashes of this urls)

Any help is welcome

Thanks!

For web crawler scale why not use a distributed database like Apache Cassandra? Lookups on indexes are efficient and no false positives.


What preprocessing steps to be followed before image comparison? 1
down vote
favorite

For example I am trying to find the similarity between two images using skimage - SSIM. The code block will be as follows

from skimage.measure import compare_ssim as ssim
from skimage import io
from skimage.transform import resize

a = io.imread("http://ecx.images-amazon.com/images/I/51PV4Dd8wAL._AC_UL246_SR190,246_.jpg",as_grey=False,dtype="float64")
b = io.imread("http://ecx.images-amazon.com/images/I/914ZHE6JolL._UY500_.jpg",as_grey=False,dtype="float64")

a = resize(a,b.shape)
ssim(a, b,multichannel=True)

I got the similarity score as follows

0.21009350738786017

Although both the images are same where one images in slightly oriented , I am getting very really low scores for this comparisons. Am I missing any preprocessing steps here before I compute compare images. If so what are the things I should consider before comparing images.

Thanks In Advance !!

There is not any fault! This is one of pitfalls of SSIM. It is very sensitive to the geometrical degradations such as orientation!


Online news classification I am performing an online news classification. The idea is to recognize group of news of the same topic.
My algorithm has these steps:

1) I go through a group of feeds from news sites and I recognize news links.

2) For each new link, I extract the content using dragnet, and then I tokenize it.

3) I find the vector representation of all the old news and the last one using TfidfVectorizer from sklearn.

4) I find the nearest neighbor in my dataset computing euclidean distance from the last news vector representation and all the vector representations of the old news.

I have a problem using TfidfVectorizer because it weights more the special words that only appear in a few news, like Apple, and news that talk about Aple are grouped together even when they deal with different topics.

So, Is there a common approach more efficient than the one I am using?

The usual rule is to ask only one question per post.  I will answer your first question.

Build a dictionary, once, in advance.  With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, and it's likely to contain all the words you need; the likelihood that you're missing some important word is pretty low.  Now, each time you see an article, throw away any words not in the dictionary.  That way, you won't need to vectorize the old news over and over every time; you can do it once and be done.  That should improve performance.

Another variation: if you see a new word on some day, you can add that word to the dictionary; at the end of the day retokenize everything you saw that day.


Tensor Decomposition in TensorFlow for multinomial time series dimensionality reduction I'm doing unsupervised learning (clustering and DR) on multinomial time series. I need to reduce dimensions for my data, which is sparse and has a lot of dimensions. I realized that some form of tensor decomposition may be the best way to go about this. What I have is a set of samples each with a set of features in time. Each time length can different, but there are the same features for each sample.

Want to end up with is a set of samples with a single feature in time. With that, I can do clustering using distance determined by dynamic time warping.

Does this entire scheme sound reasonable?

TensorFlow has a feature tf.qr (https://www.tensorflow.org/api_docs/python/tf/qr) which can do tensor decomposition, but I don't understand the output or math behind it well enough to utilize it. I have tried running it with what seems like success, but I don't understand what is stored in the q and r output tensors. If someone could explain this to me and if this even applies to what I am trying to do, I would really appreciate that.

In addition, I tried using scikit tensor's cp decomposition (https://github.com/mnick/scikit-tensor/blob/master/sktensor/cp.py) which  returns a single tensor. I also do not understand what is stored in this output or if it does something similar to TensorFlow's tf.qr function or related to what I want to do.

As you can probably tell by now, I am quite lost in the math. I've tried to find explanations of how these functions work, but I can't without jumping into multilinear algebra which I don't understand. If someone could point me in the right direction, I would greatly appreciate it.

EDIT:

if s1 and s2 are two multidimensional time series, then this should computer the distance between them. I tried it in python and it does in fact output a number, but I don't know if I am understanding this correctly. It behaves in the same way DTW would with one dimension but takes the sum of the MSE of all features instead of just one of them when computing the distance at a given point in time. Also, this DTW function has a time window which is set to 10 but of course can change...

def MDTWDistance(s1, s2, window=10, num_columns=1):

    DTW={}
    w = max(window, abs(len(s1)-len(s2)))

    for i in range(-1,len(s1)):
        for j in range(-1,len(s2)):
            DTW[(i, j)] = float('inf')
    DTW[(-1, -1)] = 0

    for i in range(len(s1)):
        for j in range(max(0, i-w), min(len(s2), i+w)):
            dist= mdist(s1.loc[i], s2.loc[j], num_columns)
            print i, j, dist
            DTW[(i, j)] = dist + min(DTW[(i-1, j)],DTW[(i, j-1)], DTW[(i-1, j-1)])

    return np.sqrt(DTW[len(s1)-1, len(s2)-1])

With...

def mdist(a, b, num_col):

    dist = 0
    for col in range(num_col):
        dist = dist + (a[col]-b[col])**2
        print a[col], b[col]
    return dist

Is this doing what I think it does/what the papers say is appropriate?

"Each time length can different, but there are the same features for each sample." You can make them all the same length, see Myth 1 of
http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf

"Want to end up with is a set of samples with a single feature in time" This might work, if the multinomial time series are "Dependent" See fig 1 of
http://www.cs.ucr.edu/~eamonn/Multi-Dimensional_DTW_Journal.pdf
Otherwise, this is a very very bad idea.

"With that, I can do clustering using distance determined by dynamic time warping." Sure, but you can do clustering with multinomial time series under DTW. See [a].

--

[a] Nurjahan Begum, Liudmila Ulanova, Jun Wang, Eamonn Keogh (2015). Accelerating Dynamic Time Warping Clustering with a Novel Admissible Pruning Strategy  SIGKDD 2015


How to use cluster analysis with grouped data so one cluster may only have not more than one item from each group? I need to group items by their approximity to each other in multi (<5) dimensional space. The items also have a categorical feature. I need to form groups (clusters) such that none of the records from the same category would appear in thr same cluster.  Is there a class of clustering algorithms that can do that?

My way of thinking is to use custom distances that would measure two records in the same category further then ones in different categories. That works to some extent, but it does not guarantee satisfaction of the given requirement.

A simple one dimension (x) + category (c) example:

    x       c
0   0.80    0
1   0.90    1
2   0.10    0
3   0.30    1
4   0.20    0

The goal is to group records into two clusters [0, 1];and  [2 or 4, 3]; then record 4 or 2 respectively should remain outside of the second cluster because a record with c=0 is already present in the cluster.

Any suggestions?

You can write your own algorithm.  I drafted something up quickly. It can be significantly optimized.

Let's make some random data

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = 9
x = np.random.rand(n,2)
y = np.zeros((n,))
y[n//3:2*n//3] = 1
y[2*n//3::] = 2

plt.scatter(x[:,0], x[:,1], c=y)
plt.show()

Now let's get the interdistance of each of these points.

dists = np.asarray([np.linalg.norm(i-j) for i in x for j in x]).reshape(n,n)

plt.imshow(dists)
plt.show()

We will make a list which will hold the radius of each circle for each point. For each point we will iterate over the other points by their relative nearness. If an associated label is not yet seen, add it to the temporary list. Otherwise, we end the function and take the average of the last distance and the current illegal point.

radii = []

for row in dists:
    labels = []
    dis = []
    for i in np.argsort(row):
        if y[i] not in labels:
            labels.append(y[i])
            dis = row[i]
        else:
            dis = (dis + row[i])/2
            break
    radii.append(dis)

Now we can plot these circles by

fig, ax = plt.subplots(figsize=(10,10))

for ix, i in enumerate(radii):
    circle = plt.Circle((x[ix, 0], x[ix, 1]), i, color='b', fill=False, alpha = 0.5)
    ax.add_artist(circle)

plt.scatter(x[:,0], x[:,1], c=y)
plt.xlim([-0.2,1.4])
plt.ylim([-0.2,1.4])
plt.show()

If we increase the number of datapoints $n$ we get


What does it mean when the shape of input images is (600,64,64,3)? While attempting an assignment, I found that shape of the input image was (600,64,64,3).
I thought 3 stood for the number of channels but it's listed as the 4th dimension.
What does this mean?
This is in reference to convolutional neural networks.

600 can be  the number of  images of shape 64x64x3 and not  only one image.


Before doing anything more advanced, I would check the correlation of new value / old value pairs. If there seems to be a linear relationship, or some kind of correlation, that would be easy to solve. If that is not the case and the "new" value for each column depends on every other column in a nontrivial way, you may have to use more complicated methods.

Such a problem is called as a Multivariate Regression Analysis, and if you have a large enough dataset, you can try using using a neural network. I would have the old values for each variable as the inputs, and the new ones as outputs. A fully connected neural network - with size depending on the size of your dataset - could be a good first step. But to be honest, I don't have much experience with such models and don't know any tutorials available online. Another way might be to train separate regression models for each output. In such a setting, you can look for correlations with different inputs and perhaps eliminate any useless ones.

By the way, what are the last two columns title NA? If they carry information, I would add them to the input layer.


Interpreting a curve val_loss and loss in keras after training a model I am having trouble understanding the curve val_loss and loss in keras after training my model.

Can anyone help me understand it? Also, is my model overfitting or underfitting?

The loss curve shows what the model is trying to reduce. The training procedure tried to achieve the lowest loss possible. The loss is calculated using the number of training examples that the models gets right, versus the ones it gets wrong. Or how close it gets to the right answer for regression problems.

The loss curves are going smoothly down, meaning your model improves as it is training, which is good. Your test loss is slightly higher than your training loss, meaning your model is slightly overfitting the training data, but that’s inevitable, it doesn’t seem problematic. All seems ok from this plot.

Now your model is getting an accuracy of 30% or so. Unless you tell us what the model is doing and how you define accuracy, there’s no way of knowing if that’s ok or not.


Query regarding (.output_shape) parameters used in CNN model I am applying CNN model on my dataset for predictions. After reshaping the dimensions, the input_shape of my model1 becomes:

model1.input_shape:
(None, 1, 3, 4)

then i apply CNN ist input layer defined below:

model1.add(Convolution2D(128, (2,2), border_mode= 'valid' , input_shape=(1,  3, 4), activation= 'relu'))

here above 1 is number of channel, 3,4 represents my nodes means 12 input nodes or features, now wehn i check the output_shape of model1, it is:

model1.output_shape:
 (None, 128, 2, 3)

here 128 are number of neurons i specified in my input layer. My question is what does the elements 2,3 shows here?

The output shape in a 2D convolutional layer depends on :

The input size W, H
The filter size F
The padding P
The stride S
The number of filters or depth D

Here, in your model, W=3, H=4, F=2, P=0 (default padding in Keras), S=1 (default stride in Keras), D=128 as you defined it.

The transformation is defined by the mathematical convolution operation, giving you an output shape as:

$\frac{W−F+2P}{S} + 1$

This output shape shows how your input reacts (in terms of size) when convolved with your 128 filters one by one.


Naive Bayes Should generate prediction given missing features (scikit learn) Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction given that there are some features missing in the test data.

I know that it is common practice to impute missing data, but why do this when Naïve Bayes should be able to make a prediction given that there are some features missing?

Can this be implemented in sci-kit learn? I tried a test set with less features, and got a ValueError as the shapes are not aligned.

So theoretically this is possible, but is it possible in scikit learn?

Your question is sensible. The way in which posterior probability is calculated in the classical Naive Bayes classifier (in sklearn) is like  summation of the conditional probabilities of the all the features in the dataset. Even though the features are treated as conditionally independent, to learn the classification probability all the features are always used in this setup. Once the model has been learned you still all those features to calculate the posterior for a new observation. The conditional independence is just an assumption that is taken to make the statistics and math obey the rules and work.
But slightly modifying the way in which the posterior is calculated you can use Bayesian approach to make predictions even with the absence of certain features. Using Bayesian approach to make predictions in the absence of certain features is still an ongoing work. You may want to have a look at this paper in which Bayesian approach is applied to astronomy to do classification with missing values.
Naive Bayes can be easily implemented in python as it is a simple calculation. The sklearn Naive Bayes does not support predictions with missing values and can be tricky to implement to what you are thinking. Maybe writing your own piece of code would be better :)


What is the meaning of Face Recognition in wild and in static? What is meant by when someone says face recognition on wild dataset and on static dataset?

In the wild refers to uncontrolled imagery, as opposed to images taken in lab environments or posed pictures such as driver's license/passport pictures or mug shots.
static in this context refers to still imagery, as opposed to temporal data (e.g. video).
With regard to facial expression recognition (not sure if you're interested in this related area), you could read this paper which describes a popular static database for images in the wild, which has good background information: Static Facial Expression Analysis in Tough Conditions: Data, Evaluation Protocol and Benchmark


Tree complexity in xgboost According to xgboost paper, regularization is given by:
$$\Omega(f) = \gamma T + \lambda || w||^2$$
where $\gamma$ is the complexity of a tree (i.e., number of leaves in the tree).
The parameter gamma in xgboost library, on the other hand, controls the minimum split at a node in order to proceed. Hence, is the $\gamma$ in the equation above used by xgboost software package? I could not find any reference to it.

In the paragraph following equation (1):

$T$ is the number of leaves in the tree.

$\gamma$ is a hyperparameter that affects how much regularization occurs on the size (number of leaves) of the tree.
Now it turns out that you can interpret $\gamma$ (at least roughly, see note at bottom) as ([source]):

Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be.

You can see that from equation (2), the regularized objective:
$$\mathcal{L}(\phi) = \sum_i l(\hat{y}_i, y_i) + \sum_k \Omega(f_k),\\
\text{where }\Omega(f)=\gamma T + \frac12 \lambda \|w\|^2.$$
By making the split, you increase $T$ by one, so the penalty increases by $\gamma$, and so your base loss term $l$ needs to decrease by at least $\gamma$ for this to be an overall improvement.  Note: Of course, this ignores what happens to the leaf weights $w$ in splitting one node into two.


Handling Categorical Features on NGBoost Recently I have been doing some research on NGBoost, but I could not see any parameter for categorical features. Is there any parameter that I missed?

__init__(self, Dist=<class 'ngboost.distns.normal.Normal'>, Score=<class 'ngboost.scores.MLE'>, Base=DecisionTreeRegressor(ccp_alpha=0.0, criterion='friedman_mse', max_depth=3,
 |                        max_features=None, max_leaf_nodes=None,
 |                        min_impurity_decrease=0.0, min_impurity_split=None,
 |                        min_samples_leaf=1, min_samples_split=2,
 |                        min_weight_fraction_leaf=0.0, presort='deprecated',
 |                        random_state=None, splitter='best'), natural_gradient=True, n_estimators=500, learning_rate=0.01, minibatch_frac=1.0, verbose=True, verbose_eval=100, tol=0.0001)

https://github.com/stanfordmlgroup/ngboost

It does not support at the time (it will come just as xgboost did not have to have it)

Given thats its a boosting method in the first place one can ask whats the history of xbgoost and subsequent cat and lgboost. XGBoost implementation of gradientboosting did not handle categorical features because it did not have to, it was sufficient enough as it was. What made xgboost special was the use of Hessian information. When other implementations (e.g. gbm in sklearn in Python) used just gradients, XGBoost used Hessian information when boosting. Which in turn was super faster.

TL;DR handling categorical features is not just a matter of convinience, it is also important for speed. This feature will come subsequently (if we contribute!) for time its all about representing uncertainty.


Cost sensitive classification with individual cost I'm currently sitting on a problem, where i'm uncertain if there is not a much simpler solution.

I'm trying to train a DNN with a dataset for a classification task that should be cost sensitive. Classic literature on this kind of task use a cost weight that is constant for any kind of misclassification. My problem needs to use one of the dimensions of the input as the cost of misclassification for that single classification.

My solution would be to use TensorFlow and add the parameter i later need to a collection and then write a custom loss function where i grab the values from the collection for my cost sensitive loss.

So my question would be, does anybody know of any simpler solution, open source implementation etc.?

There are two types of classification costs: per class, and per instance.

In keras, for instance cost, we assign a cost to each training sample by feeding sample_weight to .fit. For example, if we have four training samples in rows 1, 2, 3, and 4, with misclassification costs 2.5, 1.5, 1.0, and 1.0, we feed sample_weight=[2.5, 1.5, 1.0, 1.0].

For class cost, if there are three classes 0, 1, and 2, with misclassification costs 1.0, 3.0, 1.0, we feed class_weight=[1.0, 3.0, 1.0].

Here is a step-by-step classification example in Keras.

You should feed your weights in this line:

model.fit(train_images, train_labels, epochs=5)


What is a XML dataset? What are xml datasets? Is it possible to convert them to csv files?
I'm working on a Java program and I sometimes download datasets wich are in a binary format, are those xml?

Thank you.

XML is a markup language similar to html. One uses tags with attributes to build data structures. For example,

<sampleXML>
  <Menu>
    <Food>
      <item1>Spaghetti Bolognese</item1>
      <item2>Spaghetti Carbonara</item2>
    </Food>
    <Drinks>
      <item1 class = 'drinks'>Sprite</item1>
    </Drinks>
  </Menu>
</sampleXML>

As you can see XML employs tags such as <Food></Food> and attributes such as class = 'drinks' which is exactly what HTML has. To access the XML data in java you got couple choices. You can read it in as a string and parse it using built in DOM parser. Or you can use JAXB to map XML directly to Java objects.

Surely you can convert XML to a csv file. There are free websites online for that. Just google "XML to csv converter."

Binary files are not XML, but can be. This needs little explanation. The letters and words you see here are ASCII characters. This is human readable text. Each ASCII character has a binary representation. For example, j in binary is 1101010. A binary file is any file in computer language (0 an 1). A binary file can also be a combination of text and binary. You can convert binary to ASCII and those files you download may indeed be XML I described above. To convert binary to ASCII just google it.


Recurrent neural network producing same predictions I am trying to train a recurrent neural network that I built in keras on timeseries data to predict number of sales for next 10 days. For this, I've created my dataset as -

var(t) -> var(t+1)
var(t+1) -> var(t+2)
var(t+2) -> var(t+3)
var(t+3) -> var(t+4) and so on

I did Min-Max scaling on this data and the RNN code is as follows -

model = Sequential()
model.add(LSTM(20, input_shape=(1, look_back),activation='tanh',bias_initializer='ones'))
model.add(Dense(1, activation='linear',bias_initializer='ones'))
opt=adam(lr=0.1)
model.compile(loss='mean_squared_error',optimizer=opt)
model.fit(xtrain, ytrain, epochs=100, batch_size=1, verbose=2)

But the plot I am getting is when I did predictions on xtrain (green = ytrain, blue =  ypred) -

The rnn isn't learning anything at all. Its producing same results for each dataset. I've tried adding hidden layers, increasing number of neurons, changing parameters (learning rate, momentum), optimizers (sgd, adam, adagrad, rmsprop), lstm activation fxn (tanh, softsign). I got little fluctuations in some cases in the graph. But the output is mostly constant.
Also, I've only 200 datasets.

Can someone please guide me what I am doing wrong here. What else I can try. Will small sized data not work using RNN at all ? If so, is there any other way to solve this problem (except ARIMA model) ?

EDIT - Increased batch size to 100 and epochs to 1000. Received some better results. Also, I did mean normalization [(x-mean)/std_dev] instead of MinMax scaling.

Try increasing your batch size. If your batch size is small, the gradients are a rough approximation of the true gradients.


join row based on column list I would like to make a join between on 3 columns and also on the rows without using pandas.
Anyone have an idea?
listOfToJoin= ['ColA','ColB','ColC']

columns = ['ColA','ColB','ColC','ColD','ColE']
Rows = [[234, 343, 49, 54609, 499],
        [23, 56, 765, 54609, 432]]

the result should be :
final_Columns = ['ColA_ColB_ColC','ColD','ColE']
final_Rows = [[234_343_49, 54609, 499],
              [23_56_765, 54609, 432]]
```

for i in range(len(Rows))):
    row = [Rows[i][0] + '_' + Rows[i][1] + '_' + Rows[i][2]]
    row.extend(Rows[i][3:])
    Rows[i] = row


what are effects of working with categorical dataset I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over sample and created a model. I also tried similar approach with undersampling. Both the method with logistic regression performs mediocre. I want to know how having too many categorical variables impacts the model and possible efficient way to approach the problem

feature1:1-3
feature2:0-1
feature3:0-3
feature4: 1-4
feature5: 0-2
feature6: 0-5
feature7: 1-4
feature8: continuous( max 10)
feature9 continuous( max 10)
class: 0-1

You need to distinguish a few problems here. How well can your data classify some outcome, what features should be used, how to deal with imbalanced classes.

Categorical features are not a problem per se. In the current stage, feature selection might be an issue. Thus, use logit with lasso or ridge to shrink features which are not too helpful (happens automatically). Also dummy/one-hot encoding would be worth a try (jointly with lasso).

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifier.html


allocation to customers My data set contains the pickup latitude, pickup longitude of the customer and drivers current location latitude and longitude.
My aim is to distribute a set of  M or M-x (in the case where shortage of drivers)
for N customers. The driver has to be chosen with less distance and less ETA.

Any solution in data science with python or R?

customer_id  pickuplocationaddress  pickuplatd  pickuplong  driverlatd  driverlong

This problem is an instance of the more general assignment problem. In the assignment problem you have:

  The assigment problem has a number of agents and a number of tasks. Any agent can be assigned to perform any task, incurring some cost that may vary depending on the agent-task assignment. It is required to perform all tasks by assigning exactly one agent to each task and exactly one task to each agent in such a way that the total cost of the assignment is minimized.

In your description the agents are drivers and the tasks are "driving to the pickup location", the cost can be either the eta or the distance of the driver's location to the pickup location.

There are a number of algorithms for solving this problem, one of the most famous is the Hungarian algorithm also known as Kuhn-Munkres algorithm.  There is an implementation of the algorithm in the scipy library (this is in Python), under the name of linear_sum_assignment, the documentation is here.


Text embeddings and data splitting I have created some document embeddings which were then used further in text classification tasks. After revisiting my code I was unsure about the workflow I used to train the document embeddings.

At the moment I am creating the document embeddings based on the complete corpus available at the time of training. After the training is done, I evaluate the model by looking whether it creates useful similarities between the document embeddings. Those embeddings are used then in machine learning models and that's where the embeddings will be split into train, test and validation sets.

Now my question is: Where is the right time to split the data? Should I do it before creating the document embeddings to prevent data leakage? I have used the mentioned approach because I viewed the creation of the document embeddings as a preprocessing step, so the computer can work with textual data. However, after I have put some thought into it, I think it's the wrong approach. I wanted to hear from more experienced NLP practitioners how they approach this task. Sorry for this very basic question.

Thanks.

TL;DR

If you are training the document-embedding model, then split the data before you convert the text into embeddings.

If you are using a pre-trained document-embedding model, then it won't matter and it is pre-processing step that it doesn't matter when you execute it.

Pipeline when training your own document-embedding model

Split your text data into train/validate/test sets.
Use your train set to train the document-embedding model.
Use your trained document-embedding model to convert train and validation sets to train your other model (e.g. classification model).
Test your final model by using your trained document-embedding model to convert the test set and test the trained final (classification) model.


Rank terms in a bag -of-words model I have a set of documents where I need to extract important keywords in the document and then rank those keywords.

The ranking should be done based on relevance and/or other metrics. Are there any popular algorithms/methods/libraries that I can use for this task?

It would help if you clarified what "relevance" and "important" mean, but you should take a look at Term Frequency-Inverse Document Frequency.

tf-idf weights words by the frequency of appearance within a document but penalizes words that appear across many documents. The concept is that words appearing many times in a single document are likely noteworthy, but they could also just be common words that appear frequently in all documents.

tf-idf has implementations in most popular langauges including Python and R.
The Python Implementation is in Sci-kit learn and is called TfidfVectorizer. It takes a list of filenames or a single document as input. You can also specifiy the stop words to apply to the documents and allows for tokenization. You probably also would want to apply a stemmer to your documents which can be done using Sci-kit learn.
If you prefer to work in R, you would want to use the tm package. This package has numerous functions for text mining.


Why convolute if Max Pooling is just going to downsample the image anyway? The idea of applying filters to do something like identify edges, is a pretty cool idea.

For example, you can take an image of a 7. With some filters, you can end up with transformed images that emphasize different characteristics of the original image. The original 7:

can be experienced by the network as:

Notice how each image has extracted a different edge of the original 7.

This is all great, but then, say the next layer in your network is a Max Pooling layer.

My question is, generally, doesn't this seem a little bit like overkill? We just were very careful and deliberate with identifying edges using filters -- now, we no longer care about any of that, since we've blasted the hell out of the pixel values! Please correct me if I'm wrong, but we went from 25 X 25 to 2 X 2! Why not just go straight to Max Pooling then, won't we end up with basically the same thing?

As an extension the my question, I can't help but wonder what would happen if, coincidentally, each of the 4 squares all just happen to have a pixel with the same max value. Surely this isn't a rare case, right? Suddenly all your training images look the exact same.

Max pooling doesn't down-sample the image. It down-samples the features (such as edges) that you have just extracted. Which means you get more approximately where those edges or other features are. Often this is just what the network needs for generalisation - in order to classify it doesn't need to know there is a vertical edge running from 10,5 to 10,20, but that there is an approximately vertical edge about 1/3 from left edge about 2/3 height of the image.

These rougher categories of features inherently cover more variations in the input image for very little cost, and the reduction in size of the feature map is a nice side effect too, making the network faster.

For this to work well, you still need to extract features to start with, which max pooling does not do, so the convolutional layer is necessary. You should find you can down-sample the original image (to 14x14) instead of using the first max-pooling layer, and you will still get pretty reasonable accuracy. How much pooling to do, and where to add those layers is yet another hyper-parameter problem when building a deep neural network.


String similarity algorithms for string containment (rather than string equality) I saw that there are a lot of string similarity algorithms to whether two strings are the same. I  have a slightly different problem - I get two strings "a" and "b" and I need a similarity algorithm to whether "b" contains "a" ("b" is likely to contain "a" with some "error"). I haven't found any algorithms to solve this problem and I'd be happy to hear about some (if they exist).

My first thought was that it's difficult to formally define this concept:

"b" is likely to contain "a" with some "error"

On the one hand, there is the idea that a is a substring of b. This question should have a boolean answer: either it does contain it, or it doesn't.
On the other hand there is the idea of approximate matching: b should contain a substring c which is "similar enough" to a. In general the question of similarity between two strings is answered with a numerical value, usually a real number between 0 and 1.

As far as I know the only way to solve this issue is to consider that there is a threshold on the similarity score between a and c, where c is any substring of b. This way the answer becomes boolean.
However there might be a way around this by considering the substring operation as part of the similarity calculation. In particular the Levenshtein edit distance can account for insertions/deletions of characters, which is what a substring is w.r.t the string which contains it.
More interestingly, it is possible to assign a different cost to any particular edit operation in the Levenshtein distance. So it's probably possible to define a variant of Levenshtein where insertions at the beginning or at the end would have cost 0, thus making the final distance $x$ between a and b equivalent to "b contains a substring c which has distance $x$ against a".
The way I would try to implement this is:

Calculate the regular edit distance, keeping the matrix used for the calculation
From the matrix, count how many insertions were done but only at the beginning and at the end, then substract this value from the distance.

Note that there might be flaws in my idea, I didn't try it :D


Variance as criteria for feature selection I'm working on an unsupervised clustering problem. I read multiple times that a variable with higher variance can be chosen over a variable with a lower variance. For example, scikit-learn implements a function that removes features with a variance lower than a threshold. (sklearn.feature_selection.VarianceThreshold)

However, isn't the variance entirely dependent on scale/measurement unit? If I standardize my features, the variance is 1 for all of them.

You are correct that variance depends on the scale and typically it is not a good proxy for how informative a feature will be in terms of the response. The exception is zero variance features. A feature with zero variance has the exact same value for all observations and is therefore completely uninformative. Sklearns VarianceThreshold function defaults to removing only the features with exactly zero variance.

Another group of non-informative features is the near-zero-variance feature. However, typically, classifying a feature as a near-zero-variance feature does not actually rely on computing the feature's variance at all. See, for example, the NearZeroVar function documentation from R's caret package:

  "nearZeroVar diagnoses predictors that have one unique value (i.e. are zero variance predictors) or predictors that have both of the following characteristics: they have very few unique values relative to the number of samples and the ratio of the frequency of the most common value to the frequency of the second most common value is large."

You can see that the algorithm is not scale-dependent and does not rely on variance.


Using Random Forest Regression correctly I have run Random forest regression with python and I have the fear that I haven't done it correct.

I have original image that has 3 different bands (each pixels has 3 values) and I want to try to see if I can predict the first value using the two others.

For that I used Random Forest regression, I have created train and test and fit the model:

rf.fit(X_train,y_train)
rf_pred=rf.predict(X_test)

then after I checked the prediction on the test and saw it was good, I wanted to use the same model in order to predict all the values of all the pixels in the image, so I did that:

pred_all=rf.predict(data)

*data includes all the pixels of the image.

My question : is this the right way to do this? can I just predict all the  pixel values just by using the rf.predict after fit it with the train and test sets? Or am I missing here some step that should be taken?

It seems good. (Understanding that you are not getting any error)

You are fitting your model in some data, then evaluating on other (basic validation).

Now you have to find data that have the same format and similar distribution and predict there.

You can have a look at the random forest documentation to see what else you can do

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html


Error when using MAX in Apache Pig (Hadoop) I am trying to calculate maximum values for different groups in a relation in Pig. The relation has three columns patientid, featureid and featurevalue (all int).
I group the relation based on featureid and want to calculate the max feature value of each group, heres the code:

grpd = GROUP features BY featureid;
DUMP grpd;
temp = FOREACH grpd GENERATE $0 as featureid, MAX($1.featurevalue) as val;

Its  giving me  Invalid scalar projection: grpd Exception. I read on different forums that MAX takes in a "bag" format for such functions, but when I take the dump of grpd, it shows me a bag format. Here's a small part of the output from the dump:

(5662,{(22579,5662,1)})
(5663,{(28331,5663,1),(2624,5663,1)})
(5664,{(27591,5664,1)})
(5665,{(30217,5665,1),(31526,5665,1)})
(5666,{(27783,5666,1),(30983,5666,1),(32424,5666,1),(28064,5666,1),(28932,5666,1)})
(5667,{(31257,5667,1),(27281,5667,1)})
(5669,{(31041,5669,1)})

Whats the issue ?

The issue was with column addressing, heres the correct working code:

grpd = GROUP features BY featureid;
temp = FOREACH grpd GENERATE group as featureid, MAX(features.featurevalue) as val;


Can overfitting occur even with validation loss still dropping? I have a convolutional + LSTM model in Keras, similar to this (ref 1), that I am using for a Kaggle contest. Architecture is shown below. I have trained it on my labeled set of 11000 samples (two classes, initial prevalence is ~9:1, so I upsampled the 1's to about a 1/1 ratio) for 50 epochs with 20% validation split.I was getting blatant overfitting for a while but I thought it got it under control with noise and dropout layers.

Model looked like it was training wonderfully, at the end scored 91% on the entirety of the training set, but upon testing on the test data set, absolute garbage.

Notice: the validation accuracy is higher than the training accuracy. This is the opposite of "typical" overfitting.

My intuition is, given the small-ish validation split, the model is still managing to fit too strongly to the input set and losing generalization. The other clue is that val_acc is greater than acc, that seems fishy. Is that the most likely scenario here?

If this is overfitting, would increasing the validation split mitigate this at all, or am I going to run into the same issue, since on average, each sample will see half the total epochs still?

The model:

Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
convolution1d_19 (Convolution1D) (None, None, 64)      8256        convolution1d_input_16[0][0]
____________________________________________________________________________________________________
maxpooling1d_18 (MaxPooling1D)   (None, None, 64)      0           convolution1d_19[0][0]
____________________________________________________________________________________________________
batchnormalization_8 (BatchNormal(None, None, 64)      128         maxpooling1d_18[0][0]
____________________________________________________________________________________________________
gaussiannoise_5 (GaussianNoise)  (None, None, 64)      0           batchnormalization_8[0][0]
____________________________________________________________________________________________________
lstm_16 (LSTM)                   (None, 64)            33024       gaussiannoise_5[0][0]
____________________________________________________________________________________________________
dropout_9 (Dropout)              (None, 64)            0           lstm_16[0][0]
____________________________________________________________________________________________________
batchnormalization_9 (BatchNormal(None, 64)            128         dropout_9[0][0]
____________________________________________________________________________________________________
dense_23 (Dense)                 (None, 64)            4160        batchnormalization_9[0][0]
____________________________________________________________________________________________________
dropout_10 (Dropout)             (None, 64)            0           dense_23[0][0]
____________________________________________________________________________________________________
dense_24 (Dense)                 (None, 2)             130         dropout_10[0][0]
====================================================================================================
Total params: 45826

Here is the call to fit the model (class weight is typically around 1:1 since I upsampled the input):

class_weight= {0:1./(1-ones_rate), 1:1./ones_rate} # automatically balance based on class occurence
m2.fit(X_train, y_train, nb_epoch=50, batch_size=64, shuffle=True, class_weight=class_weight, validation_split=0.2 )

SE has some silly rule that I can post no more than 2 links until my score is higher, so here is the example in case you are interested:
Ref 1: machinelearningmastery DOT com SLASH sequence-classification-lstm-recurrent-neural-networks-python-keras

I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of what you sampled is used to validate your network , this means that you train and validate using balanced data set. In the testing you used imbalanced database. This means that your validation and testing sets are not equivalent.
In such case you may have high validation accuracy and low testing accuracy.
Please find this reference that talks mainly about data imbalance problem for DNN, you can check how they sample to do the training, validation and testing


How to build a model on a dataset having 40% missing values in most of the variables? I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category) and addresses of the companies.
What may go wrong if I build a model on other 60% records(Non-missing values)? Can you suggest any measures/actions to take before spending time on building the model?

Couple of things I planned to do to ensure 60% records represent the population:

Distribution of companies by industry category/state for 60% records(Non-missing values) vs 100% records

Target variable distribution of 60% records(non-missing) vs 100% records

Generally speaking, you should investigate the process by which your values are missing and try to deal with it. I assume you checked that :

There is no meaningfull way to fill those missing values. Sometimes, typically with companies data that often represent amount of money, missing values means 0$. For exemple values missing by block may correspond to a tax form they didn't fill because they don't pay it and most features could be set to 0.
Similarly you don't have enough data to fill the blanks. Sometimes you can do missing value imputation based on what information you have. Specifically with companies data, you might be able to calculate financial ratios based on other metrics. Basically, you might want to check that your missing values do not correspond to some calculation that hasn't been made (then either correct that calculation or do it yourself in your pre-processing). There are more advanced imputations techniques, but I don't think they would be usefull when values are missing by blocks.

If you checked that, you can proceed to the next step : modeling. How you build your model depends on what is its purpose, what you want to do with it.  Here, you should ask yourself some practical questions about those 40% records with missing values.
The first one is : do you actually need to output something for those instances with a lot of missing values ?
In some cases, notably when the output is public (or can be contested individually) it might just be best to output nothing or something along the line of 'we do not have enough data' for those instances. Practically that means removing them from your data set and building your models on your other instances.
On the contrary, when you look for performance (and not explainability to someone) you might want to use that pattern as an info. The best way to do so would be to introduce some feature that count the proportion of missing value for that instance. It might help a general model (see below) deal with that info.
If you need to output something, because you need some sort of an average, or because management asked for it, then consider the second question : should you make multiple models ?
One approach is to use just one model. Some models, mainly trees, can handle missing values. More advanced methods like XGBoost (eXtreme Gradient Boosted trees) are even considered state of the art for tabular data like companies data. Generally speaking this works (and might works very well with XGBoost). However, having done that myself, you'll end up with something quite hard to explain and overall you'll obfuscate that a significant chunk of your population is missing data. More specifically with data missing in blocks like yours, you'll probably end up with a very early split in your tree and actually have two models inside of one, with possible bizare interactions.
The alternative is to devise two models: one, very performant (say XGBoost), on your data with few missing values and another, way simpler on your other data. This will help you deal with instances with lot of missing values quite efficiently. For companies data with lots of missing data, you might just want to build a simple table of average of the outcome by your main descriptors (size, industry, geographical zone). This will leave you with most of your time to deal with the more complex model and its explanation (which is probably why you are paid for and is way more fun).


How to create these two plots from two arrays with numbers? I have two numpy arrays, one with the data for x axis and another one with the data for the y axis. The position i in each array matches the other array.
Some values in the x axis array are repeated. I would like each of them to appear in a separate area, as in the following examples (note that 5 and 10 appear twice in the x axis):

Here is the code I tried:
dataFrame = pd.DataFrame({'xx': x_data, 'yy': y_data})
plt.plot('xx', 'yy', data=dataFrame, marker='o')

But this code is creating a chart where there is more than one y axis value for each x axis value (possible due to the repetitions).
The dataFrame is like this:
     xx   yy
     5    1
     10   2
     15   1
     0    0
     5    3
     20   4
      4   3
     10   4

Thanks in advance.

Changed the values for better representation
# Gathering Dataframe
import pandas as pd
data = [[1, 5], [1, 6], [2, 5], [0,0], [5,8], [20,4]]
df = pd.DataFrame(data, columns = ['xx', 'yy'])
df

#Drawing the plot (Scatter Plot)
from matplotlib.pyplot import *

plott = df.plot.scatter(x='xx', y='yy', title= "Scatter plot between two variables X
and Y");

plott

# (line plot connecting the scatter plots)
plt.plot( 'xx', 'yy', data=df, linestyle='-', marker='o')
plt.show()


Theano/Lasagne/Nolearn Neural Network Image Input I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype.
All standard examples like MNIST numbers classification run well, but problems appear when I try to work with my own images.

I want to use 3-channel images, not grayscale.
And there is the code where I'm trying to get arrays from images:

 img = Image.open(item)
 img = ImageOps.fit(img, (256, 256), Image.ANTIALIAS)
 img = np.asarray(img, dtype = 'float64') / 255.
 img = img.transpose(2,0,1).reshape(3, 256, 256)
 X.append(img)

Here is the code of NN and its fitting:

X, y = simple_load("new")

X = np.array(X)
y = np.array(y)

net1 = NeuralNet(
    layers=[  # three layers: one hidden layer
        ('input', layers.InputLayer),
        ('hidden', layers.DenseLayer),
        ('output', layers.DenseLayer),
        ],
    # layer parameters:
    input_shape=(None, 65536),  # 96x96 input pixels per batch
    hidden_num_units=100,  # number of units in hidden layer
    output_nonlinearity=None,  # output layer uses identity function
    output_num_units=len(y),  # 30 target values

    # optimization method:
    update=nesterov_momentum,
    update_learning_rate=0.01,
    update_momentum=0.9,

    regression=True,  # flag to indicate we're dealing with regression problem

       max_epochs=400,  # we want to train this many epochs
        verbose=1,
        )

  net1.fit(X, y)

I recieve exceptions like this one:

Traceback (most recent call last):
  File "las_mnist.py", line 39, in <module>
    net1.fit(X[i], y[i])
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 266, in fit
    self.train_loop(X, y)
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 273, in train_loop
    X, y, self.eval_size)
  File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 377, in train_test_split
    kf = KFold(y.shape[0], round(1. / eval_size))
IndexError: tuple index out of range

So, in which format do you "feed" your networks with image data?
Thanks for answers or any tips!

Just out of curiousity: why use 3-channel images? I work in CV as well and from what I've seen, it's standard to grayscale the images. Especially with MNIST, which is not colored to begin with, it doesn't seem that using color has any benefit.

I also believe that there's an intuitive reason to use grayscale - often times color is a confounding variable. If you were to look at a red "1" and and blue "1", you'd say "Hey! Those are both ones! They're just different colors". However, computers are much stupider than you or I. Even with intelligent algorithms, a computer may still end up in the situation where it's looking at a blue "1" and say to itself, "Oh crap, what is that?!? Why is it blue? I've never seen anything like this before!" Keep in mind how large of a difference there is from the color (255, 0, 0) and (0, 255, 255) and how slight the differences in images can be (especially outside the comfortable playground of MNIST).

At any rate, most of the NoLearn examples I've seen use data shapes like:

X = X.reshape(-1, 1, size, size)

If you grayscale your images you can coerce it into this shape. Unfortunately, I'm not sure of how you'd cram your colored data into noLearn and get the results you're looking for.


How to tell how much information I lose when I simplify the graph data structure with respect to unsimplified graph? I have the following problem: I have some sort of data (that I can't publish here, but they are in the form of points with XYZ coordinates) and I can represent them as a collection of graphs i.e. $Q = \{G_1, G_2 ... G_t\}$, where for every node there is an associated set of features, e.g. node $u_i$ has feature vector $\mathcal{F}_i$ and the features are changing between graphs (but graph structure does not). The resulting graphs are big in size with this approach. Therefore I decided to make the graphs smaller, by truncating some of the nodes and edges. And I would like to calculate how much information I lose when I simplify the graphs with respect to the not simplified graphs or original data. I would like to get something like "This graph explains 77% variance in the data" And the truncated graphs "This graph explains 55% variance in the data".

The question is then fallowing:
How to tell how much information I lose when I simplify the graph data structure.

Edit: Also the feature vector can be replaced with weighted edges. I think it can make the problem a bit simpler to solve.

Graph comparisons can be tricky.

One option is to take an Information Theory approach, something like "An information-theoretic, all-scales approach to comparing networks"


Is it possible to ensure that all classes are represented in the output of a scikit-learn decision tree? I am working with an ordinal classification problem with six ordered classes and I want to compare a neural network classifier with a baseline classifier that is as simple and parameter-free as possible.

In my case, I want the baseline to use only the most important single feature $X$ that the neural network uses. For this particular problem, it makes sense to look for a set of class thresholds $\{t_i\}$ that lets me classify as:

      X <= t_0 -> class 0
t_0 < X <= t_1 -> class 1
t_1 < X <= t_2 -> class 2
t_2 < X <= t_3 -> class 3
t_3 < X <= t_4 -> class 4
t_4 < X        -> class 5

I have looked at a few options, and decision trees seem to fit the bill. They are very simple, and do not require choosing any free parameters (unlike e.g. $k$-nearest neighbours which requires choosing a value for $k$). If I create the tree with the argument max_leaf_nodes=6 it is quite easy to extract the thresholds $\{t_i\}$ from the resulting decision tree after fitting. (If there is another method I could use that would fit the bill as well or better to achieve this, please let me know in the comments!)

I have divided my data into six folds while ensuring that the class distribution is very similar in all folds. For this baseline I balance the classes by oversampling, as the classes are originally somewhat unbalanced. (The ratio between samples in the most common class and the rarest class is around 5:1).

For five out of my six folds, the decision tree method works very well when I use that fold for testing and the rest for training. However, for the last combination of folds used for training, I get a decision tree where class 2 is not represented in the output. Instead, two of the leaf nodes represent class 1:

Is there any way to force the decision tree to build itself in such a way that the six leaf nodes of the tree represent the six different classes?

It's expected behavior that the tree will not define rules for all classes if the models doesn't see a need. If you just want to compare a neural net's class predictions, you can use tree.predict_proba(...) to get all classification probabilities. Use the outputs of the neural net after softmax and before binarize.

For the fold that had little knowledge of class 2, the probability would be 0 or close to 0 given how the tree was built. If you want to force the tree to represent class 2 and assuming that it is present in all folds of your cross validation, you can change the learning params to default.

Another, but not great way, it to upsample your data to a large degree. This is bad because it would invalidate the purpose of CV, but you could at least see a small representation of all classes and your mean classification scores will remain similar.

If there are plenty of each class, you can stratify the classes across folds.

Also, if the problem is that you have insufficient samples of a minority class, you should consider up sampling the minority during training. https://towardsdatascience.com/dealing-with-imbalanced-classes-in-machine-learning-d43d6fa19d2


Is it possible to use Recurrent NN (LSTM) for classification? I have a dataset C of 50,000 (binary) samples each of 128 features. The class label is also binary either 1 or -1. For instance, a sample would look like this [1,0,0,0,1,0, .... , 0,1] [-1]. My goal is to classify the samples based on the binary classes( i.e., 1 or -1). I thought to try using Recurrent NN (LSTM) to generate a good model for classification. For that, my question is: Is it possible to use Recurrent NN (LSTM) for such a classification problem?

tr_C, ts_C, tr_r, ts_r = train_test_split(C, r, train_size=.8)
batch_size = 1000

print('>>> Build STATEFUL model... (%s, %s)' % (tr_C.shape[1], tr_C.shape[2]))
model = Sequential()
model.add(LSTM(129, batch_input_shape=(batch_size, C.shape[1], C.shape[2]), return_sequences=False, stateful=True))
model.add(Dense(1, activation='relu'))

print('>>> Training...')
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(tr_C, tr_r,
          batch_size=batch_size, epochs=1, shuffle=True,
          validation_data=(ts_C, ts_r))

Thank you

Yes, it is possible. You will first use a LSTM layer and then use a Dense/Linear layer on top of it to get the final output. Make sure that your linear layer has number of outputs = number of classes.


Unable to transform the variable to normal distribution I have a variable called 'list_price' in my data ,which is not normally distributed.
Here is the screenshot of how the distribution looks like before transformation

Tried transforming it with the following transformations:
1.log
2.Sqrt
3.reciprocal
4.boxcox
None has resulted in normal distribution.Most of the transformations gave a skewed distribution after transformation.
Would like to use this variable in modelling.
How would we proceed further in such scenarios?
Should we go with original distribution itself?
Need some guidance on this

Based on your image, your data looks like it is uniformly distributed. You might try using more than 3 bins to look at the distribution more closely.
A few points related to your questions:

For uniformly distributed data, you won't be able to transform this data into a valid normal distribution
This distribution is fine for use in models, no need to change it.
Here is a good resource explaining assumptions about linear regression, this information also applies to many other algorithms


Is machine learning successful in solving combinatorial optimisation problems in NP-hard? Discuss problem of scheduling using machine learning Chess is Exp-complete. I assume Alpha GO is also Exp complete. Correct me if I am wrong. Now deep reinforcement learning has been successful in tackling this problem. What about other more practical problems like scheduling, bin-packing, and routing?
In particular, what is the state of the art for scheduling applications using ML techniques? How should one start in this direction?

You don't have to get to EXP-complete in order to get a hard problem.
NP-Complete is bad enough...

Cryptographic assumptions (e.g., the existence of one way functions) are also a good way to create hard problems

There is research in the area lead of plenty of results like Cryptographic Limitations on Learning Boolean Formulae and Finite Automata - DFA is not PAC learnable in polynomial time.

However, there is an important difference between these negative results and writing a great chess program.
The negative results means that for a given problem no algorithm will solve ALL instances (let say, in polynomial time).
That doesn't mean that you won't be able to play better the human champion on some/most/all instances.

Note that this complexity observation is indifference of the technique. Machine learning is just another (useful) way to cope with these problems.

As for machine learning ability to cope with these problems, you can find NP-Complete problems all around in machine learning (SVM, Neural Networks, Decision trees). That didn't prevent them from finding many useful models.


TimeDistributed with different input / output sequence length I'm looking into using TimeDistributed in my LSTM to see if it would improve the accuracy of my model.  I'll be honest, I'm still not 100% sure what the specific use case for TimeDistributed, I just thought I'd give it a try and see if it improved my accuracy. Aside from blind trial/error, why would someone use TimeDistributed?

Now, my next question is probably tied to my lack of understanding of exactly what TimeDistributed's specific use case is, but I understand that the output from TimeDistributed is a sequence, and that sequence is the same length as the input sequence because you use return_sequences=True on the final layer of the model (all of them in my case, as it's stateful).

In my case I'm inputting a sequence which is 100 length, and I'm then forecasting a sequence of length 3 from that input sequence, but I get this error:

  Error when checking target: expected time_distributed_1 to have shape
  (100, 3) but got array with shape (3, 1)

Am I right in assuming that you can only use TimeDistributed if your output sequence is the same length as the input sequence?

Many thanks

I have looked for an answer to this question for quite a while, and couldn't really find a clear yes/no answer.

From what I understand, TimeDistributed really only works for similar input/output lengths.
According to what Jason Brownlee writes in the comments section of his TimeDistributed tutorial, he would treat the different input/output lengths as a seq2seq/encoder-decoder problem (which he describes here).

If you find a better answer, or find a way to make TimeDistributed work with different output sizes, please let me know.

*Update (copying my comment from below, with better formatting):

Actually, I have found a different solution which does use TimeDistributed: You can trim the output sequence you get from TimeDistributed using Cropping1D layer. So that my model now looks like this:

model.add(LSTM(hidden_size, return_sequences=True, input_shape=(num_input_timepoints, 1) ,dropout=0.05)
model.add(TimeDistributed(Dense(1)))
model.add(Cropping1D(cropping=(num_input_timepoints - num_output_timepoints,0))) # cropping the end

Which, in your case, would trim the first 97 time points and would leave you with the last 3. That would fit the shape of your target data.


Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used the scikit-learn built-in feature importance), but with Deep Neural Networks I get very different results (used permutation feature importance). I also ran PCA, and PCA gave me similar results to DNN! Is this normal? Could both of them be accurate still? I have quite a large amount of initial features.

Few factors that can cause such differences -

A common issue with the default RF FE approach.
It's not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories
Read

Correlated Features
 When you have correlated features,
RF Tree-based FI approach will divide the importance depending upon how the Tree used the two correlated Features
Permutation based method will assign negligible importance to both as skipping anyone will have no impact on the result

Feature Interaction
 If we have two Features with Interaction,
Tree based approach will catch the Interaction in successive splits and accordingly divide the importance into two Features
Permutation based method will assign high importance to both the Features as dip will be high when anyone is skipped(Assuming additive Interaction)
In this case, this is a question to ponder how the FI should be divided when two Features are interacting. Ideally, a third feature(e.g. X1*X2) should be created and should get all the FI.

You should try -

Identity correlated Features using respective tests e.g.Pearson, Crammers, Spearman etc.
Use Permutation approach for all cases. Sklearn has got the option

#https://scikit-learn.org/stable/modules/permutation_importance.html
from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=30)

Then if needed, look for any interaction effect (See suggested reading)

Suggested reading
Beware Default Random Forest Importances
Discovering-interaction-effects-in-ensemble
Chapter#7 - Feat Engg
Interpretable-ml-book


Trying to figure out how to set weights for convolutional networks I am working on CNN, and I have some doubts. Let's assume I only want one feature map, just to make things easier. And let's suppose my image is grayscale, to make things even easier. So, let's say my image is (32,32) --grayscale, hence just a channel and we don't need to write it explicitly, and my filter is (3,3) --again, one feature map, so I won't bother writing 1.
I understand this will map to a (30,30) layer.

How many parameters will I have? If I understand it correctly, I will have 9 weights and one bias, so a total of 10, because we map each (3,3) subregion using the same weights. Back-propagation will determine the best values for those weights and that will give me one feature map, or a filter.

So far, so good. What I don't understand is how does the training work? I need to keep the same weights and bias when moving across the image (that's why I only have 10 parameters), but won't those change when I do back-propagation? How can I apply back-propagation and keep the same values for the weights regardless of the subregion they are applied to?

You are right there are just 10 params in your example.

For determining gradients, you just add up all the deltas from backpropagation in each location - i.e. you run backpropagation 30x30 = 900 times, for each position the 3x3 kernel is used, for every example in your batch (or just for one example if you are running most simple onine stochastic gradient descent), and for each position you add those delta values into a suitably-sized buffer (10 values for weight deltas, or 9 values for previous layer activation deltas). You will end up with one set of summed deltas matching your single 3x3 filter (plus a delta bias term). You then apply the summed version to update the weights of your single filter + bias.

Note this is a general rule you can apply whenever multiple gradient sources from backpropagation can be applied to any parameter - they just add. This occurs in RNNs too, or in any structure where you can set an objective function for non-output neurons.