Python dataframe iterate over rows Dataset

**Col1**                                 **Col2**      **Col3**
dog                                        Z             st02
dog,cat                                    Z             st02
dog,bat,cat                                Z             st02
bat,cat,elephant                           Y             st02
dog,bat,cat,elephant                       Y             st02
tiger                                      Z             st01
pigeon                                     Z             st01
pigeon,parrot                              Z             st01
dove,parrot                                Z             st01
pigeon,parrot                              Z             st01
pigeon,parrot,dove                         Z             st01
lion,leopard,cheetah                       Z             st01
tiger,lion,leopard,cheetah                 Z             st01
dog,tiger,cheetah                          Y             st01
dog,tiger,leopard,cheetah                  Y             st01
eagle,jaguar,Kangaroo,zebra                Z             st02
cheetah,eagle,jaguar,Kangaroo,zebra        Z             st02

The expected output is:

**Col1**                                 **Col2**       **Col3**
dog,bat,cat                                Z              st02
dog,bat,cat,elephant                       Y              st02
tiger,lion,leopard,cheetah                 Z              st01
dog,tiger,leopard,cheetah                  Y              st01
cheetah,eagle,jaguar,Kangaroo,zebra        Z              st02
pigeon,parrot,dove                         Z              st01

In order to extract the above rows as output, I tried tracing the patterns and using the below logic:

data = pd.read_excel("data.xlsx")
data['Col4'] = data['Col1'].str.count(',')
v1 = []
v2 = []
v1.append(0)
v2.append(0)
for i in range(0,data.shape[0]-1):
    x = data['Col2'][i]
    y = data['Col2'][i+1]
    t1 = data['Col3'][i]
    t2 = data['Col3'][i+1]
    g1 = (x == y) & (t1==t2)
    d1 = data['Col1'][i]
    d2 = data['Col1'][i+1]
    c1 = data['Col4'][i]
    c2 = data['Col4'][i+1]
    flag = 0
    if(all(x in d2 for x in d1)):
      flag = 1
    g2 = (flag == 1)&(c2>c1)
    v1.append(g1)
    v2.append(g2)
    data['new_cond1'] = v1
    data['new_cond2'] = v2
    data['Final_flag'] = (data['new_cond1']==True)&(data['new_cond2']==True)
    data_output = data[data['Final_flag']==True]

But I didn't end up getting the expected output, rather few additional rows are also present in output. Could someone please help me extracting the rows mentioned in expected output.

From the dataset, I am trying to extract 1) Rows which has maximum number of animals separated by commas (or consider birds wherever pigeon/parrot/dove is mentioned). 2) Need not be the case that there should be only one maximum number of animals per Col2 or Col3, there might be even more than one Example as in case of row no. 1 and row no. 5 with same value in Col 2 and Col 3. This is because category of animals is different in row no.1 and row no. 5. Hope it's clear.

Thanks in advance!

It seems like you want to reduce the records in the data if Col1 is a subset of another record in Col1. First you can sort the column alphabetically, and then use the function below.
import pandas as pd
df = pd.DataFrame()

df['Col1']  = ['X','X,Y','Z','Z,W']

def test(x):
    return df.Col1.apply(lambda y: y in x).sum()>1

mask = df.Col1.apply(lambda x: test(x))

df[mask]

Apply is the way to iterate through records in pandas. You can nest apply functions to efficiently solve your task.
Note: if there are duplicates in your dataset, it will also mask those. You will need to deduplicate the dataset first with df.drop_duplcates(subset='Col1')
OR
if you want to keep the variability in the other two columns, then you can pass use the apply function of the whole df, with axis=0. See the docs.


Propensity model with Only Positive Data Is it possible to build a propensity model (i.e., the likelihood that a user will buy an item) using only positive values.
For example, I have a bunch of data about Customers (people that bought stuff) and Users (people that haven't bought stuff yet)
I want to get the likelihood that a User becomes a Customer.
It seems that the only way to do so is to train a model using the data of Customers, therefore using only Positive values.

The model built using only positive value will be biased towards positive value and will always predict for the customers who bought any items before. But customers who didn't buy anything,  the model won't be able to predict anything.


select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, colsample_byTree, min_child_weight, and subsample). I did a literature review to specify the range and interval of values for each parameter. Using those ranges and intervals, the parameter space generated around 62,500 parameter combinations. I am using R caret::train function to generate the best hyperparameter combination for my dataset. However, the amount of simulations (62,500) is too much. I read about the Latin hypercube sampling (LHS) and I think that is what I need to reduce the number of simulations by applying initial selection of hyperparameters using LHS. But I am having trouble implementing the approach in my dataset. My goal is to generate a manageable number of hyperparameter combinations (i.e., ~500) using LHS, and then use caret::train function to select best parameters. I would like to ask for help in implementing LHS using my parameter space.
nrounds <- seq(from = 200, to = 1000, by = 200)
maxdepth <- seq(from = 2, to = 10, by = 2)
eta <- c(0.01, 0.05, 0.1, 0.2, 0.3)
gamma <- seq(from = 0, to = 0.4, by = 0.1)
colsample_bytree <- seq(from = 0.4, to = 1, by = 0.2)
min_child_weight <- seq(from = 1, to = 5, by = 1)
subsample <- seq(from = 0.6, to = 1, by = 0.1)
dataGrid <- expand.grid(nrounds, maxdepth, eta, gamma, colsample_bytree, min_child_weight, subsample)

dials from Tidymodels has a grid_latin_hypercube function you can use for this https://dials.tidymodels.org/reference/grid_max_entropy.html


Classification of jumbled images I want to be able to create a model that would be able to classify an image that has been split into 9 parts and jumbled around.

I did see a paper on it but it is quite old (7-8 years old). Could anyone point me towards any resources? Is building a CNN the best approach?

Any help is appreciated.

This 2019 paper states that all the current image classification models are actually doing the same thing, i.e. classifying jumbled image patches. The reason is that they are weak at peaking up long-range dependencies (relations between pixels far away). As a result, breaking these dependencies by shuffling the image patches does not affect them substantially. Here is a post that explains the findings and the BagNet method that suits your need.


On the choice of LSTM input/output dimension for a spatio-temporal problem I am using LSTM neural networks from (R)Keras for a matter of spatio-temporal interpolation. I manage to get the network to output predictions but the results are not outstanding (very little improvement on validation loss). I am wondering about the shapes of training data and labels.

Say I have 50 dates of measurements of the variable of interest $y$, accompanied by about 100 covariates $x$ (spatial coordinates, temperatures...). Each date has 24 measurements of $y$, so nsamples=50*24=1200. If I set the timestep hyperparameter of LSTM to e.g. 3, and use a moving window of step 1, I have therefore an input table $X$ of shape (1200, 3, 100).

On the other hand, should the labels table $Y$ be of dimension (1200, 3) or (1200, 1) ? More precisely, which of the following describes the problem the best:
$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t-2} ; Y_{n,t-1} ; Y_{n,t})
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow (Y_{n,t} ; Y_{n,t} ; Y_{n,t})
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t}
$$

$$
(X_{n,t-2} ; X_{n,t-1} ; X_{n,t}) \rightarrow Y_{n,t+1}
$$

Or are they all plausible ways of addressing slightly different problems? As I said, I'm trying to spatially interpolate $Y$ for the 50 dates of measurements, as well as predicting $Y$ for the year(s) to come.  So I expect one is more relevant than the others but I have no clue on which one.

I hope this is understandable as I clearly miss some technical vocabulary here.

On time-series models

All models that you have mentioned are correct and practical depending on the problem (the index $n$ is not required). The second one however produces redundant results which is a waste of computation. Even
$$
(X_{t} ; X_{t+1} ; X_{t+2}) \rightarrow (Y_{t-1})
$$
is correct, if you are fitting on an archive and want to predict a year given the covariates from the next three years.

But only the last model
$$
(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+1})
$$
is a forecasting model. So in general, if you want to interpolate into the next $k$-th year from now $t$, you should use:
$$
(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+k})
$$

or

$$
(X_{t-2} ; X_{t-1} ; X_{t}) \rightarrow (Y_{t+1},...,Y_{t+k})
$$
Even a better model that takes advantage of known $Y$'s in the past would be:

$$
(X_{t-2}|Y_{t-2} ; X_{t-1}|Y_{t-1} ; X_{t}|Y_{t}) \rightarrow (Y_{t+k})
$$

where $|$ denotes vector concatenation to produce a 100 + 1 dimensional vector for each known year.

As a personal opinion, for the time-series prediction task, 24 data points per year is very small compared to the dimension of $X$, which is 100. 1200 samples for $X \rightarrow Y$ regression (ignoring the time) is more practical; if selecting 10 from 100 covariates is possible even better.

Because of the small data set, I would suggest:

$(Y_{t-m} ;...; Y_{t-1} ; Y_{t}) \rightarrow (Y_{t+k})$ for time series prediction, and
$X \rightarrow Y$ regression for estimating the relation between X and Y.

Relation to LSTM and RNN

If we use LSTM/RNN to model time-series, they would be stateful. That is, when input $X_{t-2}$ is fed to an LSTM, it keeps an internal state (hidden state) to be combined with the next input $X_{t-1}$ and so on. Regarding the input/output dimension, here is an RNN animation from a post on medium by Raimi Karim that shows an arbitrary step among 3 steps of feeding $(X_{t-2} ; X_{t-1} ; X_{t})$ to the network:

As you see, dimension and number of inputs are independent of output. We can feed 5 inputs $X_{t-4}$ to $X_{t}$, each 100 dimension (100d) and receive a 1d output by setting the dimension of hidden states to 1d, or setting it to 10d and use an extra dense layer at the end to convert 10d to 1d, or receive a 50d output, or a 150d (three 50d) output, etc.

Word "stateful" in Keras (source)

LSTM and RNN are stateful by definition, this [badly named] variable in Keras means

  If stateful=True, the last state for each sample at index i in a batch will be
  used as initial state for the sample of index i in the following
  batch. Fabien Chollet

For example, if each batch has 24 samples indexed from 0 to 23 (each sample could have the form $(X_{t-2}, X_{t-1}, X_{t}, Y_{t+1})$), then the last hidden state $h$ from 8th sample will be used as the initial hidden state for 8th sample in the next batch. Except for special cases that there is a temporal order between batches and their samples, this must be set to False.


How to use ndcg metric for binary relevance I am working on a ranking problem to predict the right single document based on the user query and use the NDCG metric to measure the model.

Given the details :

Queries ( Q ), Result Document ( D ), Relevance score.
But the relevance score is a binary ( 0 or 1 ) i.e out of document lists, only one document is marked as relevance score =1.

Data set example:

 query, docs,relevance
{
[1, doc2,0],[1, doc3,0],[1, doc4,0 ],[1, doc6,1],[1, doc9,0]
[2, doc3,0],[2, doc5,1],[2, doc10,0],[2, doc11,0],[2, doc1,0]
}

My questions:
1. Is it possible to use the NDCG metric for binary relevance problems?
2. If so, please share some reading notes or suggestions.

Thanks

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts.
A similar measure often used with binary relevance scores is the mean average precision defined as:
$\text{MAP}=\frac{\sum_{q=1}^Q\text{AveP}(q)}{Q}$,
where $Q$ is the number of queries.
A comprehensive explanation of both nDCG and MAP is available here


Bagging or Random Subspace Method for Random Forest? I am reading a lot about the Random Forest regressor. I read about bagging (Bootstrap and aggregation) and random subspace. But I am not sure if the random forest regressor is just using bagging or bagging and random subspace method. Because in some articles, the random subspace method seems like an alternative to the bagging method.

Most modern implementations do both, at least optionally.

sklearn has max_features and bootstrap.
ranger has mtry and replace/sample.fraction.
xgboost's random forest has colsample_bynode and subsample.
h2o has mtries/col_sample_rate_per_tree and sample_rate (and a couple modifiers).


Analyzing mobile usage. What kind of approach should I apply? I need to analyse a dataset about mobile phone usage (#calls, #sms, #internetConnections) per each cell and hour in the different days.

[date] [CDR/Position] [#calls] [#sms] [#internetConnections]

My purpose is detecting similarities in the data (Monday-Tuesday is similar... or Monday night is different...). After this, I'd like to find the reason they are similar/dissimilar.

What can I apply?

There are two straight forward (vanilla) ways without going for any fancy featurization:

Clustering:

Run a clustering algorithm. Something like k-means should work well with this kind of a dataset. While doing this, I would not feed the day_of_week information into the clustering algorithm.

I would suggest running k-means (after normalizing each of the columns). Choose a small number of clusters that is easy to investigate (or you could use the number of clusters that maximizes the BIC).

Investigate the clusters to understand membership by day_of_week in each of these clusters.

Multi-class Classification:

Treat the day_of_week as the response that you would like to predict. Build a decision tree of a fixed depth to predict the day_of_week given the columns. By examining this tree, you can easily tell, which decisions led to a set of leaves being labeled Sunday vs the set of decisions that led to a set of leaves being labeled Monday. These decisions will also help you understand the similarities between different days.


do the results change if you remove duplicate rows and you sum their weights? Assuming that we have a dataset with the respondents on each row (N respondents) and their respective characteristics as columns (C characteristics). Each respondent has also a weight.

In case of high number of respondents, is it a good idea to remove the duplicate respondents and sum their weights ? Will this lead to different results ?

So my initial data would look like this

> dt
   id weight v1 v2
1:  1     10  2  4
2:  2     11  2  4
3:  3     12  2  4
4:  4     13  3  5
5:  5     14  3  5
6:  6     15  3  5

And since respondents 1,2,3 are the same, and respondents 4,5,6 are the same i would end up with this

> dt
   id weight v1 v2
1:  1     33  2  4
2:  2     42  3  5

With weighted linear regression, it is exactly the same, as the expression for the loss function is a sum of weights multiplied by errors in the prediction. This works, of course, for other methods with loss functions, such as logistic regression and neural networks. This is due to the fact that the loss function is linear with respect to the weights. As you save memory, it is totally recommendable.

With other methods, you should check if the criteria for choosing parameters or method is linear or not with respect to the weights. If not, you should not do it (to me it doesn't make sense that methods are not linear with respect to weights, but there might be a case where this happens).


How do I calculate distance of test data point from centroids in KMeans scikit-learn? I am using Kmeans clustering on my data
After training the model, I want to calculate the distance between Test Data points and CLuster centers.
How do I do it?
My code is like below:

model = KMeans(clusters=2, random_state=42)

model.fit(X_train)

# get centroids
centroids = model.cluster_centers_

But I'm not sure how to use those centroids to calculate the distances for new data point

you can try below approach
centroids is a matrix  with all cluster centers
centroids=[[20,40,60,80],[60,120,180,240],[100,200,300,400]]

TestData_vector=[130,170,250,300] #you new test data as a vector

import numpy as np
from sklearn.metrics.pairwise import euclidean_distances
euc_res=euclidean_distances(np.array(centroids), np.array([TestData_vector]))

# normalize the result
normlaized_res= (1/euc_res)/((1/euc_res).sum())
#convert to list and sort it
normlaized_res_list=normlaized_res.tolist()
sorted_res=sorted(normlaized_res,reverse=True)
#get the nearset cluster
nearest_cluster=[]
for i in sorted_res[:10] :
    nearest_cluster.append(normlaized_res_list.index(i))


Overfitting while fine-tuning pre-trained transformer Pretrained transformers (GPT2, Bert, XLNET) is popular and useful because of their transfer learning capabilities.
Just to remind: The goal of Transfer learning is is to transfer knowledge gained from one domain/task and use that transfer/use that knowledge to solve some related tasks. This is done by training a model on a huge amount of labelled data (which we already have and probably easy to get), then remove last few layers and fine-tune the model for the new related task with task-related dataset.
I took a recent pretrained transformer published by Google called XLNET, and just add classification layer of the top of that and fine-tune the whole network. (Which is the main intention of this kind of model, correct me if I am wrong )
Problem is, the model is hugely overfitting. I have 1200 example to train and each has 350 words on average.
To overcome the overfitting, I set the dropout of each layer of the transformer from 0.1 to 0.5. This did not work. So I  decreased the number of trainable parameters (since the transformer has a huge number of parameters), by freezing first 10 layers (11 layers + 1 classification layer in total). Even that does not work. So I counted the number of trainable parameters in the last layer. There are 7680000 parameters which are very high compared to my dataset (around 1200*350= 420000 words). So, this high number of tunable parameters is the most possible reason for overfitting.
Here is the loss graph:

My questions are:
Do you see any flaw in my analysis?
Is there anything I can do to decrease overfitting? (tried with low learning rate and large batch size)
If my analysis is correct, then the claim that "fine-tune pre-trained transformers with small dataset" is bit misleading and datasets should not be so small. Am I correct?

What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so your model continues to improve its ability to generalize to unseen data.
Once your validation loss starts creeping upward, then you have begun to overfit.
See chapter 5 of Jeremy Howard's Deep Learning for Coders with fastai and PyTorch for more details. https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527


Estimating box size from the contents I'm currently on week 4 of my Coursera course on ML, so I have much to learn about data science. However, I got the opportunity to apply what I've learned at work, and I'd like some guidance. Our company ships random objects to customers in boxes. We'd like to be able to estimate how big boxes will be, given the random objects inside.

Here's an example of the input data:

box # | contents                          | box size
----- | --------------------------------- | ---------
1     | a widget, a doodad, and a trinket | 20x12x8
2     | 3 widgets                         | 12x12x12

However, our list of items has a long tail. I did a count of total items shipped by object type, ordered by count descending. Here's the result:

rank | object count | object type
---- | ------------ | -----------
1    | 500,000      | doodad
2    | 350,000      | trinket
3    | 300,000      | widget
---  | snip         | ---
50   | 6,000        | whatyoumacallits
---  | snip         | ---
300  | 5            | quarts of blinker fluid

Etc. By item number 340, the count is 1, and there are 360 distinct items. I think one way to approach this at first would be to only consider the top 50 items, and try to do a simple polynomial regression with 50 features to estimate L, W, and H (assuming each variable is less than the previous one).

It won't be 100% accurate, but it will be better than wild guesses. But is there a better way to do this? Any advice is much appreciated.

This is known as "3D Bin Packing Problem" in literature.

https://en.wikipedia.org/wiki/Bin_packing_problem

Since this is NP-Hard; Some of the approaches  are :

Heuristics : https://www.researchgate.net/publication/226249396_A_New_Heuristic_Algorithm_for_the_3D_Bin_Packing_Problem
Deep Reinforcement Learning : https://arxiv.org/abs/1708.05930
Ensemble : https://medium.com/@alitech_2017/alibabas-ai-solution-for-the-3d-bin-packing-problem-3ce66d730ecc


How to visualise GIST features of an image I am currently working on a image classification application using deep learning algorithms (either by using GIST features or CNN). I need help in understanding the below queries.

I have extracted the GIST features of an image (Reference Link). These extracted features will be given as input to deep learning algorithm to classify the images.
Is there a way to visualize the extracted features on top of the image?

CNN or GIST, Which is better for image classification? Is GIST outdated when compared to CNN?

Thank you,
KK

Given that code is trivial for both (GIST + Network and Raw Pixel + Network), you can try three approaches for a given project.

GIST + Dense layers (GIST is not space-distributed)
Raw Pixels + CNN + Dense Layers
Raw pixels + CNN + Dense + input layer 2 (GIST) + Dense

For some projects, GIST can help since it is an abstract feature that CNN might or might not learn.
EDIT: This paper compares GIST and CNN
Regarding:

Is there a way to visualize the extracted features on top of the image?

This can be done with an attention layer in approach 3 (CNN + GIST).
CNN provides spacial distribution (Required for visualization) and dense layer that merges CNN's output with GIST can be used with an attention layer.
Paper for visualization


How to use neural networks with large and variable number of inputs? I'm new to machine learning, but I have an interesting problem. I have a large sample of people and visited sites. Some people have indicated gender, age, and other parameters. Now I want to restore these parameters to each user.

Which way do I look for? Which algorithm is suitable to solve this problem? I'm familiar with Neural Networks (supervised learning), but it seems they don't fit.

There exist many possibilities for populating empty gaps on data.

Most repeated value: Fill the gaps with the most common value.
Create a distribution: Make the histogram and drop values according to that distribution.
Create a new label: Since you do not have information, do not assume any value and create another label/category to indicate that value is empty.
Create a classifier: Make a relation among the variable with empty gaps and the rest of the data and create a simple classifier. With this, populate the rest of the data.

There exist many others, but these are the most common strategies. My suggestion is not to populate and to keep unknown what is unknown.


Autocomplete with deep learning I got interested in autocompletion using deep learning and tutorials that I found where conditioned always on specific number of characters (given 40 characters predict the next character or the whole complete word). But in real world the autocomplete is done after entering a first letter already.

I was checking out how autocompletion works in Whatsapp I had following observations:
 #1. Autocomplete starts working already after a first letter.
 #2. Before I would enter a space, Whatsapp suggests the continuation of a word, example: entering 'L' would result in suggesting 'L', 'Like', 'Last'.
 #3. If I enter a space after a word, autocomplete would suggest me whole next word, example: entering 'Last ' would result in suggesting 'time', 'night', 'week'.
 #4. If I enter 'I am ru' it will suggest 'running', 'run'. (Pay attention: grammatically, run is wrong)

One could use RNNs/LSTMs with one-hot encoding or word-embedding to do such things.

However, I wondered what the train dataset should look like:

is it char-based model with different n-grams words, example:
1.1 char-based 1-gram ('l', 'i', 'k', 'e') word would cover the cases #1 and #2 because I would be able to sample the next character.
1.2. char-based with 2-gram word ('l', 'i', 'k', 'e', ' ', 's','u','n') would cover #3 as I would be able to predict the whole complete next word (character-based) and it also preserves the dependence on the previous word.

By using 1.1 and 1.2 I could cover cases #1, #2 and #3 and model 2-words dependence.

is it word-based model with different n-grams, example:
2.1 word-based with 1 grams, eg. "like sun" -> sequence: "like ", predict: "sun". This would allow me to model #3
2.2  word-based with 2 grams, eg. "like sun and" -> sequence: "like sun ", predict: "and". This would allow me to model a bit longer dependence.

By using 1.1 and 2.1 I would cover cases #1, #2, #3.

Questions:
I wonder what would be better to do: use 1.1 and 1.2 or 1.1 and 2.1.

Also, let me know if my way of thinking is correct or if I am completely wrong somewhere or if I miss something or if you think there are some easier way of doing it. Would be glad about any suggestions.

This can be easily worked out using a standard Seq2Seq word-level model without any modifications in the training process. During inference, in the decoder, just mask the vocabulary based on the partial word input which has been provided.

By masking, I mean to assign a very high score (inf) to words which don't start with the partial word input, thus constraining the decoder to predict words which only start with the partial word input.


What plt.subplots() doing here? Below is the code I am trying to execute.

x = np.random.normal(size=1000)
fig, ax = plt.subplots()
H = ax.hist(x, bins=50, alpha=0.5, histtype='stepfilled')

Can anybody elaborate what fig, ax = plt.subplots() doing here ?

Thank you.

As you can read from here

plt.subplots() is a function that returns a tuple containing a figure
and axes object(s). Thus when using fig, ax = plt.subplots() you
unpack this tuple into the variables fig and ax. Having fig is useful
if you want to change figure-level attributes or save the figure as an
image file later (e.g. with fig.savefig('yourfilename.png'). You
certainly don't have to use the returned figure object but many people
do use it later so it's common to see. Also, all axes objects (the
objects that have plotting methods), have a parent figure object.

The parameter bins is used to set the number of ranges to be used to accumulate the data in those ranges. I don't know where else you have question.


Regression for non linear data I have a non-linear dataset. How can I do regression in orange? Orange does not have polynomial regression widget.

You have to add higher order polynomial terms to your dataset yourself. Accordingly, you should add new columns to your dataset which consists of higher order polynomials with the desired combination of the existing features. It means that if you have already a CSV file which contains two columns for different samples, you have to add your desired extra feature columns which are the polynomial combination of the existing features. Then use simple regression.


question about splitting data set by attributes and labels Why the dataset need to use X as attributes and y as label?
WHy not use a X to have it all?

You could indeed use an X to have it all. However your downstream models expect the features and labels (I.e. Xs and ys) to be referenced via different object pointers in order to train and evaluate the model.
E.g
model.fit(X,y)


How handle the add of a new feature to the dataset? Let's say that you trained a model (eg. Random Forest) on a dataset with ten features (or columns).

Now you add one or more features to the dataset. You need the information brought from these new features so you want to enrich your model.

What happens to the data collected till now? How can I use it? Those data have just ten features. How can I use them to train the new model?

What you ask for is known as Transfer Learning in the Machine Learning framework, so you might want to look more into that direction.

An interesting publication regarding Transfer Learning in Decision Trees is this.


How do I use NLTK Text object with Re library? I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error;

  TypeError: expected string or bytes-like object

I have coded the following;

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

""" importing data set """
from nltk.corpus import PlaintextCorpusReader
corpus_root = './'
wordlists = PlaintextCorpusReader(corpus_root, 'evidence1.txt')
wordlists.fileids()
wordlists.words('evidence1.txt')
stringx = wordlists.words(wordlists.fileids()[0])
print (stringx)

""" cleaning the texts """
from nltk.text import Text
text = Text(stringx)
print(text)

import re
lowerx = re.sub('[^a-zA-Z0-9]','',text)

I think I need right kind of object to pass on to re.sub

You are passing the variable text (where you stored the nltk.Text object) to re.sub (which expects a string str object, not a nltk.Text object).

Contrary to what that class's name may suggest, Text is not a string-like object itself, it simply stores strings. That may be what misled you.

Those strings are probably what you need when calling the re-related functionality below. You can access them normally and in a pythonic via standard enumeration methods, e.g. for-looping over that instance, for word in theText.... See the code below for a full example.

import re
import random
from nltk.text import Text

# Initialize a dummy text with integers as its words.
# With random proability, we add a non-alphanumeric
# character to the word so that we can test the regular
# expression in the original example.
text = Text([
    str(random.randrange(1, 100)) + (
        '' if random.random() >= 0.5 else
        random.choice(['!', ',', '?', ':', ';'])
    )
    for _ in range(20)
])

# The tokens in the `Text` object can be accessed
# via enumeration (the class implements a Python iterator
# underlyingly)
print(text)
print(text.tokens[:10])
print(list(text)[:10])
print([x for x in text][:10])

# To avoid the exception, we comment the original line
# We can't pass the variable `text` to `re.sub` because
# it expects a string and `text` stores an instance of
# NLTK's `Text` class.
#lowerx = re.sub('[^a-zA-Z0-9]','',text)

# Test for the expected behavior:
for token in text:
    lowertoken = re.sub('[^a-zA-Z0-9]', '', token)
    print('<input="%s"  output="%s">' % (token, lowertoken))


how can I generate a Bernoulli block mixture model in matlab? I am trying to write the code of a Bernoulli block mixture model in matlab, but am facing an error every time I run the function. In particular, I'm having a problem with how to relate the distribution parameter $\alpha$ to the latent variables $Z$ and $W$.

In other words, \mathbf{Z} follows a multinomial distribution of parameter $\pi$, such that $\dim(\pi)=(1, g)$, where $g$ is the number of clusters of rows and $\sum \pi=1$.

$W$ follows a multinomial distribution of parameter $\rho$ such that $\dim(\rho)=(1,m)$, where $m$ is the number of column clusters and $\sum \rho=1$.

$\mathbf{Z}$ is a matrix of $\dim(N,g)$, and $\mathbf{W}$ is a matrix of $\dim(d,m)$, where $N$ is the number of observations, $p$ is the number of variables, $g$ is the number of row clusters, and $m$ is the number of column clusters.

$X$ follows a bernoulli distribution of parameter $\alpha_{ZW}$, where the notation $\alpha_{ZW}$ denotes the values of $\alpha$ depends on $\mathbf{Z}$ and $\mathbf{W}$, and $\alpha$ is a $\dim(g,m)$ matrix. Please find attached a graphical representation of the model.

How do I write the code of $\alpha_{ZW}$ in order to get generate the model?

First, $x_{ij}$ is not a bernoulli random variable. In the text, $x_{ij}$ is described as real valued. Also, the dimensions of $z$ and $w$ are incorrect.

The text indicates that we're drawing a matrix of data call it $X (n\times d)$ such that $X_{i,j}=x_{ij}$.

A particular entry has distribution $x_{ij} \sim f(\alpha_{z_i, w_j})$. Where $\alpha$ is some given matrix of parameters. The maximum row index of $\alpha$ is $g$. The maximum column index of $\alpha$ is $m$. So $\alpha$ is $g \times m$.

All that remains is to pin down the indices $z$ and $w$. These are vectors rather than matrices since they have a categorical distribution, for example $z_i$ should be an integer between $1$ and $g$.

Putting this together we need to:

Draw $z$, a vector of length $n$, by $z_i \sim \text{Cat}(\pi), i=1,...,n$
Draw $w$, a vector of length $d$, by $w_j \sim \text{Cat}(\rho), i=1,...,d$
Draw $X$, an $(n\times d)$ matrix, by $x_{ij} \sim f(\alpha_{z_i, w_j})$


Duplicating data from training set Does a model trained on a larger training set created by duplicating records perform better on the test data than the one trained on data without duplicates?

Duplicating training data for classes with fewer samples can actually improve your accuracy and that's what we call Oversampling.
But if you have balanced data and trying to duplicate then it doesn't differ from doing one more training iteration/epoch.


Searching interactions with RandomForest and/or GBM I'm trying to explain a count variable and a continious variable > 0 with GLM, using R.
In order to improve the quality of the regression, I want to add some interactions that can be useful for the model.
As I'm a newbie in machine learning, I want to know if RF and GBM can help me to determine useful interactions.
I saw that interact.gbm can assess the relative strength of interaction effects in non-linear models. The question is :
Will it be "mathematically" correct to add variables with important strength of interaction in order to reduce MSE/Deviance ?

Thank you !

Several times it does happen that interactions among variables improve the bias of the model. This is especially true when the effect of one independent variable on the target depends on the values of other independent variables.
I don't think there's anything mathematically incorrect in doing this.

E.g: let's say you are trying to predict revenue as a function of advertising. In this example it's reasonable to assume that the effect on revenue of one extra unit of advertising on Television  would depend on the existing level of advertising on Facebook (say). The true data generating function (if you had access to it) might be something like: $$Revenue=Ad_{TV}^{\beta_{1}}Ad_{Print}^{\beta_{1}}...$$ where $Ad_{TV}$ is the # advertisements shown on TV etc.
If you use provide the model the opportunity to handle such interaction terms you will be closer to modeling the true data generating function.

However adding more feature increases the complexity (capacity) of the model and you might have to use regularization wisely to prevent over-fitting


Spark MLLib - how to re-use TF-IDF model I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm.
Here's the process:

Datasets:

Full sample data (labeled)

Training (labeled)

Test (labeled)

Unseen (non-labeled)

This is my current workflow:

Fit IDF model (idf-1) on full Sample data

Apply(Transform) idf-1 on full sample data

Split data set into Training and Test data

Fit ML model on Training data

Apply(Transform) model on Test data

Apply(Transform) idf-1 on Unseen data

Apply(Transform) model on Unseen data

I read somewhere that I should split my data into training and test BEFORE fitting IDF model; Fit IDF only on training data and then use the same transformer to transform training and test data.

Why would you do that? What exactly do IDF learn during the fitting process that it can reuse to transform any new dataset. Perhaps, idea is to keep same value for |D| and DF|t, D| while use new TF|t, D| ?

Also, how often I will Fit (not transform) IDF model against new unseen data? let say my model is ready for prediction. I made n prediction using same IDF and Classifier model. After that I want to retrain model as I have new data now. Should I also retrain IDF then?

tf-idf will learn a vocabulary, idf, and some will also learn stop words (based on min_df, max_df, max_features). Read over sklearn's TfidfVectorizer and you can see the attributes that the fit method will set.

When you expose a trained tf-idf to new data it will transform that data into a vector of the same size as your original data using the vocabulary to construct Term_Counts which are then converted into your tf vector. The value in this is that you can use another model to predict an outcome based on the tf_idf as each new document will have the same size tf_idf vector as the documents you used to train the model. Otherwise you couldn't use it to make a prediction! For example with a naive bayes classifying tfidf:

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(X_train)
nb = MultinomialNB()
nb.fit(X, y_train)

# When you receive a new document
X = tfidf.transform(new_doc)
prediction = nb.predict_proba(X)

And I don't think you would want to refit the model. If you want some kind of continuous real-time update consider implementing a bayesian update


Performance of model in production varying greatly from train-test data I was wondering if anyone has any advice on where to start digging for this problem. I have a model which has gone through development and all train/cv/test data sets now perform above 95% both for accuracy and F-Score. The total development data set is around 60k samples, with a 2/3 split for positive and negative samples.  These samples are based on extracts for the months of Jan to Nov of last year. Final test results were:

Precision: 0.9751 Recall: 0.9320 Accuracy 0.9693 F score 0.9531

However, the first runs in production showed a very high precision:95%+ but a very low recall:~50%. Accuracy = 48%, FScore = 68%.

Any thoughts from the group on this, where to look, potential causes.  We will run this over the next couple of months, as we may have exceptional variations due to the Xmas period, but we were surprised. Any help appreciated. Thanks

This happened to me in the past.

The first question I would ask myself will be, how am I splitting train, validation and test sets? The second one would be, are my production data in the same domain as of those in train/validation/test?

Sometimes a simple split of data might still result in testing data leaking into the validation and even test data, this is the case of domains where we want to study the behaviour of users, normally you will not want the same users in train/test/validation sets (that is, ideally a user should remain within the same data set).

Apart from that, looks like you might just be overfitting the data.


Are there any very good APIs for matching similar images? I am working with Python. I have 3000 thousands images of front-faced watches like the following ones: Watch1, Watch2, Watch3.

I want to find an API which receives this collection of photos or even others taken under less ideal conditions (different background colour, darker lightning etc) and which finds/matches the most similar watches with one each other.

By similarity I mean that I expect that a round in shape, brown watch with thin lace will be only matched with watches of round shape, of dark colour and with thin lace (from the same collection of photos).

I am aware of APIs of this kind from Google, Amazon, Microsoft, TinEye, Clarifai, Indico etc but I am not sure that they will perform so well in a so specialised application. For example, these APIs are useful for matching car images with car images and not with food images but matching among  the same kind of objects (e.g. watches) based on a very high level of detail (shape, colour, thickness etc) is significantly more demanding.

For instance, this is an application on a specific kind of objects like clothes with Indico: https://indico.io/blog/clothing-similarity-how-a-program-is-more-fashionable-than-me/. However, if you notice it, the results are not that good and essentially they could be retrieved to a great extent even by simply applying PCA and KNN to these images.

Therefore, my question is: Is there any API which can match similar images based on a high level of detail?

Never used this, but http://matelabs.in has something close... pretrained models on jewelry. You can probably extend those models. The new google dataset may also have labeled versions of the stuff you need. I cannot find the link I saw origionally, but it has bounding boxes for 20K classes down to 'teacup' 'teacup handle', 'tea', so I would not write google off before testing either.

Since this is DataScience SE, I will give the more DS answer as well. The thing you are looking for sounds like a multi-category classifier. You can probably find good resources and pretrained models here, and on the end, sklearn has apis for common classifiers.

The general idea is run your image through a Convnet, minus the classification part. You then have 1K-8K 'features' which a second algorithm set of classifiers, [decision forest](https://pdfs.semanticscholar.org/.../99dff11dcb3a48dee07a19052b07fdd2e7fe.pdf
), logical compisitional model, embedding, etc... gives a score for each binary attribute.


Memory issue when trying to initiate zero tensor with pytorch I am facing a memory issue while trying to initialize a torch.zeros -
torch.zeros((2000,2000,3200), device=device)
Getting the following error:
RuntimeError: CUDA out of memory. Tried to allocate 47.69 GiB (GPU 0; 8.00 GiB total capacity; 1.50 KiB already allocated; 6.16 GiB free; 2.00 MiB reserved in total by PyTorch)

My question is: why does a zero tensor need that big memory? Or am I doing any mistake?
P.S. I was checking with getsizeof in another system - the size of this tensor is showing as 72bytes only.

The reason the tensor takes up so much memory is because by default the tensor will store the values with the type torch.float32. This data type will use 4kb for each value in the tensor (check using .element_size()), which will give a total of ~48GB after multiplying with the number of zero values in your tensor (4 * 2000 * 2000 * 3200 = 47.68GB). What you could try to do is change the datatype from torch.float32 to something like torch.int8, which only uses 1kb for each value, reducing the memory needed by 75% to 12GB. But given that you only seem to have 8GB available this will also not solve your problem. The only solution therefore would simply be to use a tensor will fewer values.


How is the Fully Connected Layer of a cnn network trained? When training a cnn network, only the filters are updated, but how are the Fully Connected Layer weights updated?

Their weights are also trained as well as the weights of the CNN filters.
So, the gradient of the loss is computed with respect to these weights also, and at each gradient step, both of the dense (fully connected) and CNN layers' weights are updated.


Seaborn barchart for frequency of data I'm trying to plot a bar chart to represent the frequency of two variables (Dead and Alive) in my test and training data. I want it to look like the second example described in the documentation (minus the error bars).

Currently i have
yfreq = pd.DataFrame(data = [Counter(Ytrain),Counter(Ytest)],
                     index = ['Train','Test'])
yfreq.columns = ['Dead', 'Alive']
print(yfreq)
>>>        Dead  Alive
    Train   384    239
    Test    165    103

And then for the graph
sns.barplot(data = yfreq)

Which correctly outputs a graph but does not split the data into test and training data.

I know it has something to do with the hue kwarg but i can't work out how too do it.
Thanks in advance

This should do the trick. You have to melt your data frame to use x,y and hue in your seaborn barplot.

yfreq['type'] = yfreq.index
yfreq = yfreq.melt(id_vars = 'type')
sns.barplot(x = 'variable', y = 'value', hue = 'type', data = yfreq)


How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets? I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(specifically STALE FILE HANDLEs for now in GPFS file system). I am categorizing the log files and clustering based on their instances per time interval. Since some messages are more predominant over the others in any given time frame than the others, i don’t want the clustering to bias towards the one with maximum variance.

Its unclear what the OP is asking (so this response is somewhat general), but the table below illustrates common contexts and the transformations that are typical:

sales, revenue, income, price --> log(x)

distance --> 1/x, 1/x^2, log(x)

market share, preference share --> (e^x)/(1+e^x)

right-tailed dist --> sqrt(x), log(x) caution log(x<=0)

left-tailed dist --> x^2

You can also use John Tukey's three-point method as discussed in this post.  When specific transformations don't work, use Box-Cox transformation. Use package car to lambda <- coef(powerTransform()) to compute lambda and then call bcPower() to transform. Consider Box-Cox transformations on all variables with skewed distributions before computing correlations or creating scatterplots.


Which approach for user classification on chat text (classifier, representation, features)? I'm trying to train a classifier to classify text from a chat between 2 users so later on I can predict who of the two users is more likely to say X sentence/word. To get there I mined the text from the chat log and ended up with two arrays of words, UserA_words and UserB_words.

Which classifier should I use fot this purpose and what structure the training data should have? I've researched for the bag of words structure but dont know exactly how to train a classifier with data in that format.

To clarify this last point, for now I have the data in a dict like {"hello":34, "how":12} and so on, being the terms word:frequency of each user. As far as I know, there is no way to use this two dicts as a classifier fit input. So, how do I transform this 2 dicts into an array that I can use to train a classifier (let's say I want to use a gaussian Naive Bayes just for the sake of the example)

You're asking what ML representation you should use for user-classification of chat text. User-classification is not the usual text-processing task.

It's not strictly necessary to semantically understand what the user is saying, only how they're saying it; so we look for telltale features indicative of a specific user. And we don't necessarily need to use, or solely rely on, the usual text-processing representations like bag-of-words, word-counts, TFIDF and word-vector.

Here are some features which are predictive of the user:

character length, word length, sentence length of each comment
typing speed (esp. if you have timestamps in seconds)
ratio of punctuation (e.g. 17 punctuation symbols in 80 chars = 17/80)
ratio of capitalization
ratio of numerals
ratio of whitespace
character n-grams (and notice these can pick up e.g. l0ser, f##k, :-) )
use of Unicode (emojis, symbols e.g. stars)
ratio of specific punctuation (e.g. how many '.', '!', '?', '*', '#' )
word-counts, esp. anything statistically anomalous, foreign, slang, insults
anything else you can think of that seems predictive for these two users, e.g. number of misspelled words per sentence (may be actual typos, or come from predictive swiping on a cellphone)


migrating to python from R: specific questions I have been using R and RStudio for prototyping and model building and due to some persisting problems (which would only be applicable to the environment that I am using in) we have decided to use python. I am looking to know:

is there a python development environment like "RStudio Server"?
how easy/hard to enable multi-threading/multi-core processing in
Python for decision trees/gradient boosting?
can data persist in python dev environment server? Meaning: Can I save the dev environment with code and data frames and come back the
next
    day to access/pick up where I left?
Can you connect to Oracle database from Python?
Is there a way I can import .RData to python development
    environment?

Why ask these questions here?

While looking up on search engines, I am mostly getting unreliable results, training insitutes promotions, outdated blogs and whitepapers from industry giants. I need a reliable answer.

1) Anaconda Spyder, maybe

2) sklearn, random forest has an option to select the number of jobs, and it will take care of parallelizing

3) i don't think so, but you can pickle objects and load them up. you can probably do something like create a variable_name - value dictionary and just pickle that

4) never tried

5) according to https://stackoverflow.com/questions/21288133/loading-rdata-files-into-python, I don't think so


is it better to correlate and encode or encode and correlate? I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting different features

Correlation is for continuous variables, example Pearson co-relation. Using Pearson correlation for categorical or ordinal variables are not recommended. If you are encoding the data, i can imply that it is a categorical variable.
For categorical variables you can find association between the variables (instead of correlation) using Chi-squared test of independence. This test is quite lenient , in the sense, even for a weak association it will show a low p-value. To further validate the result of Chi-squared test of independence , you can calculate Crammers V-test which will give a specific value between 0 and 1 . The analysis gets easy with Crammers V-test. Crammers V-test uses Chi distribution.
Whether you calculate association before or after the encoding (for categorical variable) it will not make any difference because it works on the frequency table rather than the actual values of the column


Why don't convolutional computer vision networks use horizontally - symmetric filters? If, for example, I have a neural network for classifying dog breeds, and I feed it an image of some dog, inherently it shouldn't matter whether I feed it the original image or the image, mirrored horizontally.
I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.

But,
by my research modern networks don't use filters like that. Well... Why? what disadvantages does this architecture have that I can't find a single mention of such idea?

I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.

If two mirrored images are in the same class - e.g. they both show a dog or a cat - that is not the same as having all the components in the image (lines, textures, shapes) responding well to symmetric filters. In general this is not the case. Even for symmetric looking shapes such as faces, it is only true at a certain scale and specific pose.

  what disadvantages does this architecture have that I can't find a single mention of such idea?

It will work poorly, because the lines, textures and shapes being detected as components of the image are rarely horizontally symmetrical.

CNNs cannot easily take advantage of mirroring or rotational invariance - where the class or detection does not vary under mirror or rotation transformations. One thing you can do to improve generalisation for transformations is data augmentation, i.e. performing the non-class-changing transformations of your training images, perhaps randomly on demand during the training process. This does work, but has the opposite impact that you were hoping for in terms of efficiency.

The recently-published CapsNet by Geoffrey Hinton's group may be able to take advantage of more types of transformation invariance. However, this is still in early stages of research, and not clear whether it offers a practical advantage in your case. There are implementations in various frameworks such as Keras, that you could try if you are interested.


What is a 'hidden state' in BERT output? I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.

I understand what RNN's have a 'hidden state' that gets passed to each time step, which is a representation of previous inputs. But I've read that BERT isn't a RNN - it's a CNN with attention.

But you can output the hidden state for each layer of a BERT model.  How is it that BERT has hidden states if it's not a RNN?

BERT is a transformer.

A transformer is made of several similar layers, stacked on top of each others.
Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n.

The hidden state you mention is simply the output of each layer.

You might want to quickly look into this explanation of the Transformer architecture :
https://jalammar.github.io/illustrated-transformer/

Note that BERT use only Encoders, no Decoders.


how to train custom word2vec embeddings to find related articles? I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when  we search on website. For this i decided to train my own embedding.
I found two methods for this:

One is to train network to find next word( i.e inputs=[the quick,the quick brown,the quick brown fox] and outputs=[brown, fox,lazy]
Other method is to train with nearest words(i.e [brown,fox],[brown,quick],[brown,quick]).

Which method should i use and after training how should i convert the sentence to a single vector to apply cosine similarity means sentence- the quick brown fox will return 4 vectors how should i convert it to feed for cosine similarity(which takes only one vector) with another sentence.

I find your question bit convoluted, so I will answer with the following bullet points:

Train your own word embeddings: There are many implementations out there, gensim is one.
Find related articles: On that point, without being an expert, I would suggest to do some research on Topic Modelling. There are also a lot of libraries you can use.
Word embeddings to sentence embeddings: This process is not as straightforward, the semantics change just by adding words together. You can use Word Mover's Distance or numerous other which train in a supervised way sentence embeddings or unsupervised.


Public dataset for news articles with their associated categories I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc.

I want to use such dataset for topic detection of various sentences or paragraphs. I was planning to train a classifier with such a dataset and use it for predictions. However, I couldn't find any. Are there any such known datasets available?

Here is a massive dataset of news with categories which I created for exactly such a reason.
Includes all the headlines published by Times of India from 2001-2019 with categories.
Contains ~3 million entries.


Why do we need a gain ratio I'm learning about decision trees, and I feel like up till now I've understood them and the math behind them pretty well except for one thing: the gain ratio.

As I understand, the gain ratio is implemented in order to punish for features that may take on A LOT of possible values.

If a feature takes on a lot of possible values, it becomes plausible that if we split on that feature there may be values that only point to a single class, but simply because there are only 1 or 2 data points with that value for that feature anyways.

In other words, the only reason that we would get low entropy for splitting on that feature is because the feature could take on a lot of values, and therefore a lot of those values pointed specifically to a single label. So our decision tree algorithm would end up splitting up on something like "ID #", and wrongly calculate that we just had a HUGE information gain.

However, this only seems like a problem because "ID#" is a feature we shouldn't be splitting on to begin with. I mean, if we had another feature that also took on a lot of possible values, but each of those values actually DID imply some label for that datapoint, then wouldn't applying the gain ratio mean that we are actually messing up our decision tree by punishing what was actually a very good split with tons of information gain?

Isn't it better to just identify which feature will have nothing to do with our labeling BEFORE we feed in the training data to the algorithm?

IDK, I just don't see why the gain ratio would really be useful...

I had the same doubt when I was doing my Master's Degree. First of all, you don't include something as random as 'IDs'. This is where data preprocessing comes in. Let's take a dataset which has users and their movie genre preferences based on variables like gender, group of age, rating, blah, blah. With the help of information gain, you split at 'Gender' (assuming it has the highest information gain) and now the variables 'Group of Age' and 'Rating' could be equally important and with the help of gain ratio, it will penalise a variable with more distinct values which will help us decide the split at the next level.


install_tensorflow() keeps overwriting virtualenv, making it impossible to install tensorflow and keras I would like to use the Keras & Tensorflow package for R in RStudio.

Everytime I use the command install_keras() or install_tensorflow(), I get the following error:

Collecting keras
Could not fetch URL https://pypi.python.org/simple/keras/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement keras (from versions: )
No matching distribution found for keras

I discovered that every time I tried to install Keras within R, it created a virtualenv in the following directory (basically overwriting it):

/users/username/.virtualenvs/

Where it has Python version Python 2.7.10 (default, Jul 14 2015, 19:46:27). When I directly activate the virtualenv and try pip install keras, I receive the same error. I figured out on Stack Exchange that I could solve this issue by using pip install requests[security]. This seemed to fix the problem but nevertheless, R won't use it.

How can I fix this problem, realising that every time I use install_keras(), I overwrite the virtualenv and the same problem occurs? I even tried using the tensorflow URL directly, but I get the same error when it tries to install other important packages.

Important information:

MacOS: 10.10.5
R: 3.3.3

I'm new to this, so my apologies if I don't use the terminology correctly. I did manage to make Tensorflow & Keras work in Anaconda and using Jupyter, but I can't seem to make it work in R in that environment as well.

Please check that which pip you are using to install keras. It is possible that your pip may be pip3( will find libraries for python 3.x), while pip2 aka pip (will find libraries for python 2.x).

Keras is supported in python 2.7:

Tensorflow also, they promote use of python 3.x, you can checkout this tow download 2.7 compatible whl. tensorflow 2.7

You can check the pip version by executing below cammand from terminal.

pip --version

To find location of python/ pip use:

which pip
which python

I would personally suggest use python 3.6 and above. As for python 2.x and python 3.4, support is going to be end soon.


Measure degree of heteroscedasticity I analyzed my time series using Breusch Pagan test and observed the presence of heteroscedasticity in it. After box-cox transformation, I again tested the time series using Breusch Pagan test. The time series still shows heteroscedasticity. However, I do not know how to test if the heteroscedasticity has been rectified even partially or not. Can anyone please guide on how to measure the degree of heteroscedasticity because the test I am using in R, gives Pvalue and I am only able to figure out if heteroscedasticity is present or not.

Since you did not specify your exact problem, first some general remarks. Heteroscedasticity does not affect estimated coefficients, but the (estimated) standard deviation of coefficients, p-values, and confidence intervals in linear models. Heteroscedasticity means that the model's variance is contingent on features (explanatory variables).

The Breusch-Pagan (BP) test is suitable for linear models. Say you have a linear model

$$ y = \beta X + u. $$.

You can estimate the model using $\hat{\beta} = (X'X)^{-1} X'y$ and obtain the residuals $\hat{u}=y-X\hat{\beta}$. The BP test checks if residuals $\hat{u}$ are contingent on explanatory variables $X$. To do so, an auxiliary regression is used:

$$ \hat{u}^2=\gamma X + e.$$

The general idea is that $\hat{u}^2$ should be unrelated to $X$, so that coefficients $\gamma$ (apart of the intercept) should be statistically not different from zero.

This conjecture could be used to drill up problems with heteroscedasticity. In your case you are interested in how strong heteroscedasticity is. Looking at the vector $\gamma$ may be one option. A second option would be to simply look at the outcome of the BP test (the $\tilde{\chi}^2$ and related p-value).

R code (regression and BP test):

library(MASS)
data(Boston)

# Regressions
reg1 = lm(medv~indus+lstat, data=Boston)
reg2 = lm(medv~indus+crim, data=Boston)

# BP test
library(lmtest)
bptest(reg1)
bptest(reg2)
# Null hypothesis of BP test = homoscedasticity (= variance does not depend on auxiliary regressors)
# If the p-value is "small" (<0.05), the null hypothesis is rejected
# reg1 -> heteroscedasticity
# reg2 -> no heteroscedasticity

Result:

> bptest(reg1)

    studentized Breusch-Pagan test

data:  reg1
BP = 20.542, df = 2, p-value = 3.462e-05

> bptest(reg2)

    studentized Breusch-Pagan test

data:  reg2
BP = 0.053288, df = 2, p-value = 0.9737

For reg1, the Null is rejected, for reg2, the Null is not rejected. This suggests that errors are not homoscedastic for reg1. Let us try to drill the problem up a bit.

Auxiliary regression:

# 1) Get residuals
Boston$u1 = residuals(reg1)^2
Boston$u2 = residuals(reg2)^2

# 2) Aux regression
aux1 = lm(u1~indus+lstat, data=Boston)
summary(aux1)
aux2 = lm(u2~indus+crim, data=Boston)
summary(aux2)

Result:

Regression 1

Call:
lm(formula = u1 ~ indus + lstat, data = Boston)

Residuals:
   Min     1Q Median     3Q    Max
-64.78 -36.94 -22.64  -2.13 581.64

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  62.0549     7.6859   8.074 5.06e-15 ***
indus         1.0020     0.6539   1.532    0.126
lstat        -2.7603     0.6282  -4.394 1.36e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 80.36 on 503 degrees of freedom
Multiple R-squared:  0.0406,    Adjusted R-squared:  0.03678
F-statistic: 10.64 on 2 and 503 DF,  p-value: 2.974e-05

Regression 2

Call:
lm(formula = u2 ~ indus + crim, data = Boston)

Residuals:
   Min     1Q Median     3Q    Max
-61.72 -56.21 -42.16 -16.11 995.65

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 61.49061   13.08357   4.700 3.37e-06 ***
indus        0.01188    1.08488   0.011    0.991
crim        -0.18560    0.86527  -0.214    0.830
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 152.8 on 503 degrees of freedom
Multiple R-squared:  0.0001053, Adjusted R-squared:  -0.00387
F-statistic: 0.02649 on 2 and 503 DF,  p-value: 0.9739

It seems that lstat in regression 1 causes heteroscedasticity in this model since the coefficient is statistically different from zero. In regression 2 all coefficients (except intercept) are not statistically different from zero.

Now, if you try to mitigate heteroscedasticity, you could use the results from the auxiliary regression(s) to see if some (linear) transformation of the data helped to make the variance more harmonic (note that the interpretation of $\gamma$ will change when you make linear transformations, but p-values will give you some idea).


Recognition human in images through HOG descriptor and SVM classifier performs poorly I'm using a HOG descriptor, coupled with a SVM classifier, to recognise humans in pictures. I'm using the Python wrappers for OpenCV.
I've used the excellent tutorial at pymagesearch, which explains what the algorithm does and furnishes hints on how to set the parameters of the detectMultiScale method.

Specifically, I do

# initialize the HOG descriptor
hog = cv2.HOGDescriptor()

# Set the support vector machine to be pre-trained for people detection
hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())

# Detect people in the image
(rects, weights) = hog.detectMultiScale(image,
                                        winStride=(4, 4),
                                        padding=(8, 8),
                                        scale=1.05)

The parameters are chosen according to a fine tuning on both accuracy and performance, following the explanations in the tutorial itself.

My problem is that this method, which seems like the currently best method to recognise humans in a picture according to literature (the original paper is dated 2005) seems to perform pretty poorly on my images.
I have images containing clothes, both with a model and without it and I'm trying this approach to recognise those with the model. On a subset of 300 images which I manually scanned to tag them for containing the model or not, the method fails 30% of the times.

These are some images as examples.

Here it detected a missing human:

Here it did not get the full human:

Here it did not recognise it at all:

I understand that the detector works for upright humans. Should they also be full-figure?
My images encompass half-figure, figures with no head or no feet.

Before this, I have tried a Haar-feature based cascade classifier to recognise the face in an image and the accuracy on the same set of image has been 90%, so I was trying to improve on this. Also, I'm interested in understanding why things are not working here.

You are using the training set that opencv is giving you which it doesn't correspond to the kind of images you are using. The data you are using comes from getDefaultPeopleDetector and the kind of images that the default detector uses are pictures of many people, not a female model from a fashion ecommerce.

If you want to distinguish between models and garments you can try to train your own classifier with HOG or other features.

Another path you can take is to detect if there is a face or not. You could use haar cascades for that.


How clustering is used in data management? Can anyone please tell me how is clustering is used in data management?

I was recently asked this question and was left totally stumped.

This is a wide open question, so I am not sure which aspects you are most interested in.  Here are some Wikipedia articles you can get started with.

Metadata Harvesting

Metadata Discovery


NLP: What are some popular packages for phrase tokenization? I'm trying to tokenize some sentences into phrases. For instance, given

  I think you're cute and I want to know more about you

The tokens can be something like

  I think you're cute

and

  I want to know more about you

Similarly, given input

  Today was great, but the weather could have been better.

Tokens:

  Today was great

and

  the weather could have been better

Can NLTK or similar packages achieve this?

Any advice appreciated.

Spacy can do this. Spacy's semantic parser is based on Language models trained on large corpus of text.

This parser can break sentence into lower level components such as words / phrases.

More details and examples :

https://spacy.io/usage/linguistic-features

Example with the first sentence from  questions: https://explosion.ai/demos/displacy?text=I%20think%20you%27re%20cute%20and%20I%20want%20to%20know%20more%20about%20you&model=en_core_web_sm&cpu=0&cph=0


when my RF log_loss socre is different then submit score? I train the RF and I sumbit to the data competition
but my train score is so much different then their submit socre
is there way to decrease this gap?
Here is my code with RF

%%time
# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
def under_over_test(X,y):
    RFC_METRIC = 'gini'  #metric used for RandomForrestClassifier
    #RFC_METRIC= 'multi_logloss'
    NUM_ESTIMATORS = 100 #number of estimators used for RandomForrestClassifier
    NO_JOBS = 2018 #number of parallel jobs used for RandomForrestClassifier
    RANDOM_STATE = 42
    MAX_ROUNDS = 2000 #lgb iterations
    EARLY_STOP = 50 #lgb early stop
    OPT_ROUNDS = 1000  #To be adjusted based on best validation rounds
    VERBOSE_EVAL = 50 #Print out metric result
    clf = RandomForestClassifier(n_jobs=NO_JOBS,
                             random_state=RANDOM_STATE,
                             criterion=RFC_METRIC,
                             n_estimators=NUM_ESTIMATORS,
                             verbose=True
                            )

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    clf.fit(X_train, y_train)
    clf_probs = clf.predict_proba(X_test)
    score_origin = log_loss(y_test, clf_probs)

In a Kaggle-like competition, the data your submission is evaluated on is not part of what is given to you. It may always produce a different result than what you estimate in your training process.

But, if it's very different, then you have likely over-fit and your model is high variance, so, your test error is probably higher than the training error here. In particular, you don't seem to be using early stopping. Your submission error is also then going to be higher than training error.

You mean your test error is lower than submission error, right? they may be quite different than your test error due to the high model variance.

The submission error on average might be higher than your test error, even when done properly, because one tends to 'overfit' the hyperparameter tuning in any validation process. (But here I don't see hyperparam tuning even?)

A more correct process would be, at least, to try to find the best tuning params with some cross-validation process, then refit the model on all the data you have using those params.


Remove indiferent respondents in survey data I have data for a product rating survey, which requires the respondents to rate a product in five levels: Very Bad, Bad, Regular, Good and Very Good. This survey was applied to several communities of clients.

After taking a look on the data, I observed that a lot of communities have an overwhelming "Regular" preference for the product. Later I found out that "Regular", was the default response for the survey question. Apparently, in some communities user's just didn't bother and left all the questions in the default value.

Is there a proper way of identifying the communities that are indifferent to the survey, so they can be excluded? I was thinking in a simple threshold-based approach (If the community has more than 50% of non-default responses, it is not indifferent), but I was wondering if there is a more principled way.

I'd recommend converting your responses to the Likert Scale and going from there.

It's a widely researched area so there should be plenty of research on how to best deal with 'so-so' responses.


How to treat a categorical variable which can have multiple values in regression? I am working on a regression problem and has a categorical feature 'CATEGORY' which can take as many as 1600 categories.

On top of that, each observation can have any number of categories in that column. For eg.
CATEGORY column of row 1 has 'a' (just one category 'a') but CATEGORY column of row 2 has 'a','b','x' (three categories 'a','b','x') and so on.

What is the most optimal way to deal with such feature for regression problem other than using one-hot encoder or hash vectorizer?

In this kind of case, I would look into mean/median target encoding. Basically, you take the average/median target/response value of y (based on your training set only) per each category in your feature. This new statistic per group is now your encoding. For your validation/testing data, you simply join back the "learned" statistics from your training set as a lookup table.

It tends to work extremely well for features with high cardinality. Also, suppose you do use one-hot encoding. How sparse are some of the resulting columns? If you have multiple sparse columns (can be detected easily by looking for low variance features or by simply counting how many 0's to 1's there are) then likely, these categories can be grouped.

Finally, there are algorithms that have specifically been made to handle high cardinality situations internally; in fact, CatBoost (whose motivation in its creation was in part to create a gradient boosting framework that directly supported categorical variables) to my belief heavily popularized mean/median target encoding (what I explain above) and gives you many different options with respect to encoding categorical features.


How to group and order columns in r GameID        Drive        down    yrdln        PlayType       sp
2009091000      1           1        22           run          0
2009091000      1           2        25           pass         0
2009091000      1           3        32           sack         0
2009091000      2           1        42           pass         0
2009091000      2           2        44           run          0
2009091000      2           3        43           pass         0
2009091000      2           4        33           Field Goal   1
2009091001      1           1        5            pass         0
2009091001      1           2        10           pass         1

I am to trying to determine the probability of scoring either a touchdown and/or a Field Goal based on a given starting yard line. The information given above is very similar to the data frame I am currently using, however, there are a few hundred thousand data entries, with thousands of games and multiple drives for each game.
I have been trying to find the first and last row for each drive for each GameID and then determine if a Field Goal or Touchdown was scored at the end of the drive. Ultimately I would like the data organized in a way where each row has a game ID, drive, 1st play yrdln, last play yrdln, Field Goal (yes or no), and sp (scoring play). Or hopefully something close to this.
I don't really know where to begin with something like this and have been failing for a good part of a day trying, so any and all help is greatly appreciated. If anything needs further clarification, please let me know!

The R code below should help you clean your data.
Please note that I made some assumptions -- see the comments.

library(dplyr)
df = read.delim(['Path to your data'], sep = ",")

# I am making the assumption that record can be uniquely identified by the gameid, drive and down variable

df_drive = df %>% group_by(GameID) %>% mutate(max_drive = max(Drive), min_drive = min(Drive)) %>% select(GameID,Drive, down, max_drive, min_drive)
df_down = df %>% group_by(GameID, Drive) %>% mutate(max_down = max(down), min_down = min(down)) %>% select(GameID, Drive,down, max_down, min_down)

# Max and Min List
min_max = inner_join(df_drive,df_down,by=c("GameID","Drive","down"))

# Determine first and last row for each GameID
keep = min_max %>% mutate(idx = ifelse(Drive==min_drive & down == min_down, 'first', ifelse(Drive==max_drive & down == max_down, 'last', 'other'))) %>% filter(idx %in% c('first','last'))

games = inner_join(df, keep, by=c("GameID","Drive","down"))
games %>% filter(idx == 'first') %>% mutate(field_goal = ifelse(PlayType=='Field Goal','yes','no')) %>% select(GameID,yrdln, field_goal)

# I am making the assumption that sp and down will come from the last record.
first_play = games %>% filter(idx == 'first') %>% select(GameID,yrdln) %>% rename(first_play_yrdln = yrdln)
last_play = games %>% filter(idx == 'last') %>% mutate(field_goal = ifelse(PlayType=='Field Goal','yes','no')) %>% select(GameID,Drive,yrdln, field_goal, sp)  %>% rename(last_play_yrdln = yrdln)

df_final = inner_join(first_play, last_play , by=c("GameID")) %>% select(GameID, Drive, first_play_yrdln, last_play_yrdln, field_goal, sp)
df_final

You should get:

      GameID Drive first_play_yrdln last_play_yrdln field_goal sp
1 2009091000     2               22              33        yes  1
2 2009091001     1                5              10         no  1


Issue with backpropagation using a 2 layer network and softmax I have a simple neural network with one hidden layer and softmax as the activation function for the output layer. The hidden layer uses various activation functions since I am testing and implementing as many of them as I can.

For training and testing for the moment I am using the MNIST dataset of handwritten digits so my input data is a matrix that in each row has a different image and in each column a pixel of that image that has been reshaped as a vector.

When I use a sigmoid activation function for both layers the computed gradients and analytical gradients seem to agree but when I try something else like tanh or softplus for the hidden layer and softmax for the output layer there are big differences as can be seen from the data below (Left->Numerical Gradient, Right->Analytical Gradient)

(1)sigmoid (2)softmax

  -9.4049e-04  -6.4143e-04
  -6.2623e-05  -2.5895e-05
   1.0676e-03   6.9474e-04
  -2.0473e-03  -1.3471e-03
   2.9846e-03   1.9716e-03
   4.0945e-05   2.7627e-05
  -2.5102e-05  -1.7017e-05
   8.8054e-06   6.0967e-06
   7.8509e-06   5.0682e-06
  -2.4561e-05  -1.6270e-05
   5.6108e-05   3.8449e-05
   2.0690e-05   1.2590e-05
  -9.7665e-05  -6.3771e-05
   1.7235e-04   1.1345e-04
  -2.4335e-04  -1.6071e-04

(1)tanh (2)softmax

  -3.9826e-03  -2.7402e-03
   4.6667e-05   1.1115e-04
   3.9368e-03   2.5504e-03
  -7.7824e-03  -5.1228e-03
   1.1451e-02   7.5781e-03
   1.5897e-04   1.0734e-04
  -9.6886e-05  -6.5701e-05
   3.3560e-05   2.3153e-05
   3.3344e-05   2.1786e-05
  -1.0282e-04  -6.8409e-05
   2.1185e-04   1.4774e-04
   9.0293e-05   5.3752e-05
  -4.0012e-04  -2.6047e-04
   6.9648e-04   4.5839e-04
  -9.7518e-04  -6.4468e-04

(1)sigmoid (2)sigmoid

-9.2783e-03  -9.2783e-03
   8.8991e-03   8.8991e-03
  -8.3601e-03  -8.3601e-03
   7.6281e-03   7.6281e-03
  -6.7480e-03  -6.7480e-03
  -3.0498e-06  -3.0498e-06
   1.4287e-05   1.4287e-05
  -2.5938e-05  -2.5938e-05
   3.6988e-05   3.6988e-05
  -4.6876e-05  -4.6876e-05
  -1.7506e-04  -1.7506e-04
   2.3315e-04   2.3315e-04
  -2.8747e-04  -2.8747e-04
   3.3532e-04   3.3532e-04
  -3.7622e-04  -3.7622e-04
  -9.6266e-05  -9.6266e-05

The way I implement backpropagation is as follows:

Variables->

Theta1, Theta2: tables with the various weights for the first and second layer.

m: size of my training set

y: a vector with the correct category for every input sample

Y: a matrix with the one hot encoding for the category for every input sample

X: a matrix with input data, each row is a different training sample

% Feedforward
a1 = [ones(m, 1) X];
z2 = a1*Theta1';
a2 = [ones(m, 1) activation(z2, activation_type)];
z3 = a2*Theta2';
a3 = activation(z3, 'softmax');
h = a3;

% Calculate J
J = sum(sum((-Y).*log(h) - (1-Y).*log(1-h), 2))/m + lambda*p/(2*m); # sigmoid
%J = -(sum(sum((Y).*log(h))) + lambda*p/(2*m)); # softmax

% Calculate sigmas
sigma3 = a3.-Y;
sigma2 = (sigma3*Theta2).*activationGradient([ones(m, 1) z2], 'sigmoid');
sigma2 = sigma2(:, 2:end);

% Accumulate gradients
delta_1 = (sigma2'*a1);
delta_2 = (sigma3'*a2);

The first cost calculation J was computed for sigmoid and the one below it for softmax (see comments) so I switch between the two.

Have I missed something during backpropagation, why is it working as expected with sigmoids but not as expected with sofmax?

I think it might be a relatively trivial bug in your cost function for softmax:

J = -(sum(sum((Y).*log(h))) + lambda*p/(2*m))

should be

J = -sum(sum((Y).*log(h)))/m + lambda*p/(2*m)

I.e. for softmax only, you have effectively subtracted the regularisation term from the cost function instead of adding it. Also, you forgot to divide the error term by the number of examples in the batch (and you are taking this average when calculating the gradients)

Your back propagation calculations look correct to me if you correct this miscalculation for J.


Data visualization in the terminal? I've gotten to love using the terminal for processing data. At work, I'm using Windows Subsystem for Linux (WSL). For school and personal, I'm using Linux terminal on a Google Cloud instance.

I love how quick I can move files and code around in the terminal and using Vim editor. I love the minimalism of text directories, etc. The major downside is I haven't been able to use charts, like matplot or seaborne, because I'm in a terminal.

Is there a solution, other than copying files and code over to something like Jupyter? I can write modules to plot basic bar charts in the terminal, but I obviously need more than that.

Matplotlib and Seaborn are graphical plotting libraries, there is no straightforward way to display data visualizations in the terminal.

The best option is to run Jupyter Notebook with --no-browser flag on the remote instance. Then, paste the URL from the remote instance in your local computer browser. You make have make sure the ports and permissions are correctly set. Connecting to a remote Jupyter server from a local computer allows plotting on the remote instance.


Why am I getting misaligned values while plotting heatmaps using seaborn? While plotting heatmaps using seaborn, the numbers are getting printed on the boundary of the heatmap (Please see the image attached). I have already plotted heatmaps many times using the same code. It used to work well but since a few days, I'm facing this issue of misalignment.

Code:

#Defining a custom function
def get_confusion_matrix(clf,X_te,y_test):
y_pred = clf.predict(X_te)
df_cm = pd.DataFrame(confusion_matrix(y_test, y_pred), range(2),range(2))
df_cm.columns = ['Predicted NO','Predicted YES']
df_cm = df_cm.rename({0: 'Actual NO', 1: 'Actual YES'})
sns.set(font_scale=1.4)#for label size
sns.heatmap(df_cm, annot=True,annot_kws={"size": 16}, fmt='d',cmap="YlGnBu", linewidths=5)

#Using the function to plot the heatmap
get_confusion_matrix(model_s1,X_tr,y_train_1)

Kindly help.

There seems to be an issue with the matlpotlib package.
3.1.1 update has caused issues with heatmaps.
Downgrading to v3.1.0 fixes this issue.
Use the following code to downgrade.

pip install matplotlib==3.1.0

Refer this github discussion for details


Time series binary classification Which deep learning architecture and algorithms do you most recommend for time series classification problem?
Of course LSTM, I am looking for state of the art papers.

https://paperswithcode.com/task/time-series-classification/latest -> You can find all the state-of-the-art papers along with code for Time Series Classification.
https://arxiv.org/pdf/1809.04356.pdf -> This paper provides an in-depth review of some deep learning architectures for Time Series Classification.
Hope it helps!


Duplicated features for gradient descent Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens?

I guess that we can not find a unique solution because that's the case for the close form in linear regression, but I do not see how to show that intuitively, or even if it is true or not.

In 'Efficient Backprop' by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), they explain why correlated variables are bad (§ 4.3 normalizing the inputs).

Duplicated data is a special case of linear dependence, which is a special case of correlation. Say you have duplicated variables $X1 = X2$, so the network output is constant over the weight lines $w_2 = c - w_1$, where c is a constant. It means that the gradient of the error is 0 along those lines : moving along those lines has no effect on the learning. In that sense the solution won't be unique.

It's bad because you could be solving a problem of lesser dimension. Removing one of the input will leave you with a network of lesser dimension.

Other than the dimension problem, for simple learner I don't think it will affect the learning process that much. For more complex learning processes (learning rate depending on time / on variables), it might get more complex.


How to handle different input sizes of an NN when One-Hot-Encoding a categorical input? let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of the categories. ["Man", "Woman", "Diverse"] has no order to it so having one input that represents them all within one dimension makes little sense.

When using cross validation, the dataset often gets split into a lot smaller subsets. These subsets may not hold all categories. When using OHE of sklearn, the input set is used to determine the dimensionality. This can lead to unpredictable column counts of the networks data input. It can also lead to different categories taking different positions in the NN.

How would one process this input to feed to the NN without hard-coding all possible categories and still be able to handle varying numbers of categories in the input set?

Two intuitive ideas that don't work:

determine the dataset size (after OHE) and set the input size of the NN based off of that: Doesn't work because each CV subset would potentially have a different model and each category position in the dimensions doesn't map to other subsets
0 pad to an arbitrarily high (but likely never reached) input size: dirty, because the ordering is still not ensured and if one category is missing, all inputs may be shifted by 1+ positions

Would autoencoders help?

A solution is set a maximum length for the number of categories and fill the future categories in order to see them in your data. Hence, that part of the categories which are not seen in the sampled data would be set to zero.


Help improving my "read_excel" execution time in python. My code reads slowly My first question here so please bare with me.

I'm trying to feed my neural network with training data read in from an excel file. It works perfectly fine when i have less than 50 rows in the sheet. But when i try with the real excel file containing almost 4.000 rows it suddenly takes forever. Although 4.000 is a lot i'm pretty sure my way of doing it is still very inefficient.

as you can see in the code below i'm using the read_excel over and over again in the loop.
I feel like there should be a way to only read the whole column 1 time and then work with it from there.

My goal is to read in 5 rows as the 1st input starting from row 0. then reading 5 rows in again but starting from row 1 and 5 rows again starting from row 3
So it's like a window of 5 rows that is read and then moving the window by 1.
The output should allways be the 1 row after the window.

**Example:** if row 1-20 contains numbers 1-20 then:
input1 = [1,2,3,4,5] and output1 = 6
input2 = [2,3,4,5,6] and output2 = 7
...
input15 = [15,16,17,18,19] and output15 = 20

notice how inputs are lists and outputs are just numbers. So when i append those to the final input & output lists i end up with inputs being a list of lists and out being list of outputs

My code

from pandas import read_excel

# initialize final input & output lists. The contents of the temporary input & output lists
# are gonna be appended to these final lists
training_input = []
training_output = []

# excel relevant info
my_sheet = 'Junaid'
file_name = '../Documents/Junaid1.xlsx'

# initialize counters
loop_count = 0
row_counter = 0

for x in range(25):

    # load the excel file containing inputs & outputs
    # using parameters skiprows, nrows (number of rows) and index col
    df = read_excel(file_name, sheet_name = my_sheet, skiprows=row_counter, nrows=6, index_col=0)

    # initialize temporary input & output lists
    input_temp = []
    output_temp = []

    for y in df.index:
        # append the first 5 rows of the 6 to input list
        if loop_count < 5:
            input_temp.append(df.index[loop_count])
            loop_count += 1
        else:
            # append the 6th data to output list
            training_output.append(df.index[loop_count])

    training_input.append(input_temp)
    row_counter += 1
    loop_count = 0

Well yes it would be slow because you are opening and closing the file for every iteration of the for loop. A general rule in programming is that if the file is not constantly changing, then only open and read it a single time. Also, there are large sections of your code that can be shaved off if you simply use list comprehension

Here, I have rewritten your code to only open the file and read it once, then it creates the two lists using list comprehension and slicing.

from pandas import read_excel

# excel relevant info
my_sheet = 'Junaid'
file_name = '../Documents/Junaid1.xlsx'

df = read_excel(file_name, sheet_name = my_sheet, index_col=0, header=None)

training_input = [df.index[i:i+5].tolist() for i in range(len(df)-5)]
training_output = [df.index[i].tolist() for i in range(5, len(df))]

Also, there seems to be a bug in your code since the excel file you described in your question does not have a header (i.e. the very first row contains data), thus your code skips the very first row of values. To fix that you should pass the parameter "header=None" to the pandas function to tell it that there is no header index. You can read more about that here.


How to calculate perplexity of language model? In one of the lecture on language modeling about calculating the perplexity of a model by Dan Jurafsky in his course on Natural Language Processing, in slide number 33 he give the formula for perplexity as

Then, in the next slide number 34, he presents a following scenario:

"If a system has to recognize
• Operator (1 in 4)
• Sales (1 in 4)
• Technical Support (1 in 4)
• 30,000 names (1 in 120,000 each)"

Perplexity in this case is 53.

Can anyone explain how the answer 53 came ?

I believe he meant: you need to identify/predict a sequence of 4 consecutive things. First: an operator, then a sales person, then a technical support person, and finally one name out of 30,000 names. One and only one sequence is correct.

The probability of the correct sequence:

$$(1/4)*(1/4)*(1/4)*(1/120,000) = 0.0000001302083333$$

If you get the 4th root, that gives you the geometric mean (in some sense that's the average per step for four steps)

$$(0.0000001302083333)^.25 = 0.01899589214 ≈ (1/53)$$

So:

$$(1/53)*(1/53)*(1/53)*(1/53) ≈ (1/4)*(1/4)*(1/4)*(1/120,000)$$

It was however, not clear in the slides or the explanation.


Where the following propagation model arise in practice This is in continuation to a previos question. I would like to know where in practice will the following M-ary tree-structured social network propagation model arise. I am looking for some concrete examples in social media (twitter and youtube etc.). Eventually, I want to do some statistical analysis on such social networks.

Imagine each node in your graph as a user, and each edge as an action such as 'share'. It may also be a bidirectional relationship 'share' and 'view'.

Some social scientists and engineers estimate the probability that a message is 'shared' and 'viewed' given that a particular user decides to share it. This process is called "information diffusion", "information propagation", "cascading", etc.

If you are interested in the details of how these calculations are performed, check out these papers:

http://cs.stanford.edu/people/jure/pubs/netrate-netsci14.pdf

http://cs.stanford.edu/people/jure/pubs/cascades-www14.pdf

http://cs.stanford.edu/people/jure/pubs/infopath-wsdm13.pdf

Similar topics:
http://snap.stanford.edu/papers.html


What annotators are used in Cohen Kappa for classification problems? I am working on a classification problem using algorithms such as Logistic Regression, Support Vector Machines, Decision Trees, Random Forests and Naive Bayes. My data consists of binary class classification i.e. Fire(1) or No Fire(0). Due to the imbalance in data Cohen Kappa was recommended for evaluation of model performance.

I am using the scikit-learn sklearn.metrics.cohen_kappa_score to compute the cohen kappa score. To compute the value it takes the following inputs

from sklearn.metrics import cohen_kappa_score
cohen_score = cohen_kappa_score(y_test, predictions)
print(cohen_score)

So it takes the y_test and predictions made by the specific model the same inputs used for confusion matrix and classification report.

However, Cohen Kappa is suppose used to measure the inter-rater agreement between observers or annotators. If we were measuring the quality of a document based on scores by 2 reviewers. Those 2 reviewers would be the annotator. But in the case of a classification problem. When I compute the score in scikit learn as mentioned above. What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?

What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?

The two annotators are exactly the two parameters you fed the method with: cohen_kappa_score(y_test, predictions), i.e. the truth/labels and your predictions. And these are then measured in terms of their "inter-rater" agreement.


Using DNN as the objective function for a multi-objective optimization algorithm When creating a multi-objective optimisation/MCDM algorithm such as NSGA-ii, does it make sense to use a deep neural network trained on a supervised tabular regression prediction task, in place of a simple equation for the objective function?

Is possible or advantageous to replace a nonlinear equation with model.predict() function in Keras to be able to model more complex objective functions?

I am using pymoo with nsga-ii

This nonlinear equation will again be approximated with the net. There is no point in introducing this much computation complexity, if it is not learned by than than it wont be learned. Ocam rasor


How can I show the relations between travel destinations? I'm trying to do a project about email marketing. I'm working on a tourism company and I want to make a best destination suggestion for the clients. But I need to see the relations between destinations.

Example: How many people visited Dublin and then visited London?

My question: How can I best analyse this relation between the cities, given data about traveler itineraries?

I want to send email offers to clients who went to London and didn't go to Dublin (assuming a strong relation between London and Dublin).

You can try Graph databases (Neo4j/Orient DB etc). Store location and connection between the location as nodes and edges. Then do analysis over graph data. Based on your need, you can use additional attributes (like count) and assign weights for edges etc. Neo4j supports collaborative filtering also.


what is the largest network used for image recognition/segmentation? What is the largest network (in number of params and layers) considered in the literature for image recognition/segmentation task? I am in particular interested in ResNet architectures.
Any recommendation for literature is appreciated.
For natural language processing, the largest models are of order of billions, such as Megatron-LM, or DeepSeed with Zero.
Is this also the case for image-classification?

There are many models proposed for image segmentation but I've used the U-net architecture which is basic enough for educational reading.
Some more examples can be found here