What should be the labels for subword tokens in BERT for NER task? For any NER task, we need a sequence of words and their corresponding labels.
To extract features for these words from BERT, they need to be tokenized into subwords.

For example, the word 'infrequent' (with label B-count) will be tokenized into ['in', '##fr', '##e', '##quent']. How will its label be represented?

According to the BERT paper, "We use the representation of the first sub-token as the input to the token-level classifier over the NER label set".

So I assume, for the subwords ['in', '##fr', '##e', '##quent'] , the label for the first subword will either be this ['B-count', 'B-count', 'B-count', 'B-count'] where we propagate the word label to all the subwords. Or should it be ['B-count', 'X', 'X', 'X'] where we leave the original label on the first token of the word, then use the label “X” for subwords of that word.

Any help will be appreciated.

Method 2 is the correct one.

Leave the actual label of the word only in the first sub-token, and the other sub-tokens will have a dummy label (which in this case is 'X'). The important thing is that when calculating the loss (e.g., CELoss) and metrics (e.g., F1), this 'X' labels on the sub-tokens are not taken into account.

This is also the reason why we don't use method 1 is that otherwise, we would be introducing more labels of the type [B-count] and affecting the support number for such a class (which would make a test set no longer comparable with other models that do not increase the number of labels for such class).


Feature selection for Support Vector Machines My question is three-fold

In the context of "Kernelized" support vector machines

Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and the main motive behind introducing kernels to a SVM is to increase the dimensionality of the problem, in such a case reducing the dimensions by parameter reduction seems counter-intuitive
If the answer to the 1st question is "NO", then, On what conditions would the answer change that one should keep in mind ?
Are there any good methods that have been tried to bring about feature reduction for SVMs in scikit-learn library of python - I have tried the SelectFpr method and am looking for people with experiences with different methods.

Personally, I like to divide feature selection in two:

unsupervised feature selection
supervised feature selection

Unsupervised feature selection are things like clustering or PCA where you select the least redundant range of features (or create features with little redundancy). Supervised feature selection are things like Lasso where you select the features with most predictive power.

I personally usually prefer what I call supervised feature selection. So, when using a linear regression, I would select features based on Lasso. Similar methods exist to induce sparseness in neural networks.

But indeed, I don't see how I would go about doing that in a method using kernels, so you are probably better off using what I call unsupervised feature selection.

EDIT: you also asked about regularization. I see regularization as helping mostly because we work with finite samples and so the training and testing distribution will always differ somewhat, and you want your model to not overfit. I am not sure it removes the need to avoid selecting features (if you indeed have too many). I think that selecting features (or creating a smaller subset of them) helps by making the features you do have more robust and avoid the model to learn from spurious correlations. So, regularization does help, but not sure that it is a complete alternative. But I haven't thought thoroughly enough about this.


Classifying Data in Orange: What's the difference between 'Features', 'Target Variables', and 'Meta Attributes' I've been looking for a more visual way to calculate statics on my spectroscopy data.  Python's Orange Canvas seems like it might be a great alternative to Pipeline Pilot, but I'm having trouble getting started with more simplistic data analyses.

Most notably, when I import data, I'm unclear as to what the difference between a 'Target Variable', 'Feature', and 'Meta Attribute' is.

How do I decide what type of data I have?

Meta variables are meta data, data about data, not used for statistical inference.
Features or variables or attributes are the measured inputs of the problem domain, the independent variables. The target variable is the dependent variable or the measure we're trying to model or forecast. Not all problems can be or need to be formulated in such a way. Orange was traditionally designed to accommodate machine learning workflows, hence the naming.


Why are my Decision Tree Leafs not pure? I'm making a using DecisionTreeClassifier from SKlearn (v0.21.3) with its default settings, using Python. I do not want regularize it in any way, I want it to overfit as much as possible.

When drawing the tree out I saw that some of the leafs were not pure. Is this normal? Was the tree not able to separate the samples?

   ...
   model = DecisionTreeClassifier(criterion="entropy")
   model = modell.fit(X, y)
   ...

With default settings the DecisionTreeClassifier does not have any restrictions in terms of complexity as described in the Scikit Documentation.

Therefore, it will stop further branching the tree if either a given node is pure (all examples have the same classification) or there are no further attributes to branch on.

So if your final tree contains leaves which are not pure (for the data set it has been trained on) the algorithm did not have any attributes to further split on.

In case you applied any kind of randomization on your data like a random split of training and test data this might turn out differently when splitting again and getting a different training set.


Time series forecast: when should I use the additive or the multiplicative method? In order to proceed with a time series forecasting, the data set has to be stationary. Stationarity can be determined using a number of packages, the most famous (as far as I could understand) is statsmodel.

What I was not able to pick to date is when I have to use one method vs the other (additive or multiplicative).

Any simple explanation?

You say you have managed to make your data stationary, so I would probably say an additive model is your best starting point. You could simply plot your stationary data and check that the variance doesn't increase with the nominal values. High variance will mean higher errors for a linear regression.

When we have a basic regression model, like the following:

$$
y = \beta_1x_1 + \beta_2x_2 + \epsilon
$$

the residual error $\epsilon$ is hypothesized to be constant (assuming the model itself is accurate). $\epsilon$ should not get larger when there are larger values for the covariates $x_1$ $x_2$. In other words, you expect homoscedasticity: that the error term is the same across all values of the model covariates. This is what you will easily spot on the plot of your processed (stationary) data.

If you made you time series stationary by taking the logarithms (a.k.a differencing), then an additive model of the log-ed variables would almost correspond to a multiplicative model.

Just to be clear, if you still seem to have heteroscedasticity with $\epsilon$ varying greatly, this might imply that your model itself is ill-formed e.g. that an important factor is missing.


Which target variable should I use? I have a problem where I want an LSTM to predict the resistance of a body. This value can also be calculated if we know the drag coefficient and the speed of that body. In my case, at inference time, the speed is known, meaning that I can do the following:

predict the drag coefficient, and then calculate the resistance accordingly
predict the resistance directly

Which one should I use as my learning target?

Interesting question!

Short (but maybe naive) answer

Experiment with both options and see which performs best!

Longer answer

predict the drag coefficient, and then calculate the resistance accordingly

If you do this, your network will try to optimize something different than your actual goal (which is the resistance). This means that your model will not "care" if the resistance you eventually calculate is any good, which can result in strange results.

predict the resistance directly

This would be better from a machine learning perspective as your model's goal will be the same as yours, however, you will lose the advantage that you have by knowing how the resistance is calculated.

Solution A

Predict both and then have a final step to decide what your final resistance will be. With LSTM, this is definitely possible, your target will just become 2 numbers instead of 1.

Solution B

The best solution, in my opinion, would be to have the LSTM output a single number (which would act as the drag coefficient), and then, add a layer which calculates the resistance using the known formula so that you can backpropagate on the entire thing, and you get the best of both worlds. In PyTorch this can be done rather elegantly. The big caveat is that the formula to calculate the resistance needs to be differentiable.


Early-Stopping for logistic regression. Theano I am trying to understand the code for the logistic regression on the official documentation, but I am struggling to understand the logic behind this code:

# early-stopping parameters
patience = 5000  # look as this many examples regardless
patience_increase = 2     # wait this much longer when a new best is
                              # found
improvement_threshold = 0.995  # a relative improvement of this much is
                               # considered significant
validation_frequency = min(n_train_batches, patience/2)
                              # go through this many
                              # minibatches before checking the network
                              # on the validation set; in this case we
                              # check every epoch

best_params = None
best_validation_loss = numpy.inf
test_score = 0.
start_time = time.clock()

done_looping = False
epoch = 0
while (epoch < n_epochs) and (not done_looping):
    # Report "1" for first epoch, "n_epochs" for last epoch
    epoch = epoch + 1
    for minibatch_index in xrange(n_train_batches):

        d_loss_wrt_params = ... # compute gradient
        params -= learning_rate * d_loss_wrt_params # gradient descent

        # iteration number. We want it to start at 0.
        iter = (epoch - 1) * n_train_batches + minibatch_index
        # note that if we do `iter % validation_frequency` it will be
        # true for iter = 0 which we do not want. We want it true for
        # iter = validation_frequency - 1.
        if (iter + 1) % validation_frequency == 0:

            this_validation_loss = ... # compute zero-one loss on validation set

            if this_validation_loss < best_validation_loss:

                # improve patience if loss improvement is good enough
                if this_validation_loss < best_validation_loss * improvement_threshold:

                    patience = max(patience, iter * patience_increase)
                best_params = copy.deepcopy(params)
                best_validation_loss = this_validation_loss

        if patience <= iter:
            done_looping = True
            break

Could any one, explain to me, what do the variables: patience, patience_increase, improvement_threshold, validation_frequency, iter,  represent ?

What this condition do ?

if (iter + 1) % validation_frequency == 0:

Patience is the number of training batches to do before you stop. iter is the number of training batches you've seen. Each iteration, you decide whether or not your validation is lower than your previous best. The improvement is only stored if the score is lower than improvement_threshold * validation_score.

It appears that patience_increase is a multiplier.  Every time you have a new lowest score, you up the total number or training batches to iter*patience_increase, but not below the current value of patience.

validation_frequency is just the number of batches between times you check the validation score.


Did anybody ever use mean pooling and publish it? I found a couple of sources which mention mean pooling for convolutional neural networks (CNNs) - including all lectures I had about CNNs so far - but I could not find any paper with at least 10 citations which uses mean pooling.

Do you know of a paper which uses mean pooling?

I just found an answer myself:

  Boureau, Y-Lan, Jean Ponce, and Yann LeCun. "A theoretical analysis of feature pooling in visual recognition." In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 111-118. 2010.

See also: shortscience summary


Constrained k-means algorithms in R (must-link constraints) I currently face an unsupervised learning task that is to be approaches using clustering. More specifically, it is a segementation task and hence there is some prior knowledge about a) the number of clusters and b) the rough content of each segment.
From literature this seems like a prime example, where "constrained k-means" would come into play and I'm really eager to try it out. Both of the above mentionend pieces of prior knowledge can be incorporated in "constrained k-means" as a) is represented by k and b) can be expressed in "must-link constraints". (For a good overview over constrained k-means see Wagstaff, Cardie, Rogers, Schrödl, & others (2001).)

My problem now is that I can't find a suitable implementation in R on CRAN.
Programming it myself in R would be ok, based on the pseudocode, but my programming is most certainly not efficient enough, since we are talking about a somewhat large dataset (75.000 x 30).

So, I turn to this community hoping that someone has found an efficient implementation of "constrained k-means" in R or is willing to provide code that is somewhat efficient.

Thank you very much,

Clemens

Update 23.06.2016:
Thanks for your replies so far. In order to make the problem a little more succint please see a sample data file here. (Note: this is not an excerpt of my original data as it is highly sensitive, but an artificial dataset with the same characteristics (less rows though))

Two examples of simple constraints:

all observations that have either a 1 in v305 or a 1 in v306 must be linked (must-link-constraint)
all observations that have a 1 in v258 and v262 must be linked (must-link-constraint)

Currently looking at your suggestions, Rahul. Thanks very much.

For the given problem I would apply constrained Hierarchical clustering in R. That is more efficient in this case. If you like to apply Hierarchical clustering the package is "rioja" and the function that you can use is chclust().
This is not constrained K-means implementation in R but this should solve your actual problem.

There are two types of constrained in chclust() function, you can set those with the help of method parameter in function. This will add the constraint on the basis of sum of square.You can read in detail about these constraint with simple search.

This chclust function will produce the constraint object and you can check the proper number of group by using other functions in "rioja" package.

A practical implementation of same can be checked on below link:
https://gist.github.com/benmarwick/7097120


Should I make single machine learning model for predicting price of house or 6 different models, given 6 datasets for different cities? I am currently working on "Housing Prices in Metropolitan Areas of India" (kaggle dataset), with 6 different csv files for 6 different cities.
All have same columns (40).
Working on a real estate website project, I am confused, should I make separate model for each csv or merge all csv to form one model (after data cleaning) to predict house prices.

Both options would work. It would depend on which method you want to use, what you want to do afterwards, or computing capacity.
E.g. developing a model for each city might allow you to contrast and compare the outcomes more simply. But you can also do multivariate predictions to predict for each city even once you have combined your dataset.
Also if your dataset is very large it might make sense to treat the cities independently to make the computations slightly cheaper.


R error condition has length > 1 Hi i am getting this error:

  Error in if (x < min) { : missing value where TRUE/FALSE needed
  In addition: Warning message:
  In if (x < min) { :
    the condition has length > 1 and only the first element will be used

I have seen this error in others post but his solutions don't fix it,
I think this may be a different case

My code:

min <- data[1,2]

for(i in data){

  if(data[i,2] < min){

    min <- data[i, 2]

  }

}

The column that I am using(the second) all are numeric values

Thanks!

If you change your for loop to for(i in 1:nrow(data)){... then it will work.

I would not call your variable min because that is the name of the function which you should be using instead such as:

tmp=min(data[,2])


How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task) First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library. I was thinking of averaging all of the Word Piece embeddings for each document so that each document has a unique vector. I would then use those vectors for clustering. Please feel free to comment if you think this is not a good idea, or if I am missing something or not understanding something.
The issue that I see with this is that you are only using the first N tokens which is specified by max_length in Hugging Face library. What if the first N tokens are not the best representation for that document? Wouldn't it be better to randomly choose N tokens, or better yet randomly choose N tokens 10 times?
Furthermore, I realize that using the WordPiece tokenizer is a replacement for lemmatization so the standard NLP pre-processing is supposed to be simpler. However, since we are already only using the first N tokens, and if we are not getting rid of stop words then useless stop words will be in the first N tokens. As far as I have seen, in the examples for Hugging Face, no one really does more preprocessing before the tokenization.
[See example below of the tokenized (from Hugging Face), first 64 tokens of a document]
Therefore, I am asking a few questions here  (feel free to answer only one or provide references to papers or resources that I can read):

Why are the first N tokens chosen, instead of at random? 1a) is there anything out there that randomly chooses N tokens perhaps multiple times?
Similar to question 1, is there any better way to choose tokens? Perhaps using TF-IDF on the tokens to at least rule out certain useless tokens?
Do people generally use more preprocessing before using the Word Piece tokenizer?
To what extent does the choice of max_length affect performance?
Why is there a limit of 512 max length in Hugging Face library? Why not just use the length of the longest document?
Is it a good idea to average the WordPiece embeddings to get a matrix (if you want to do clustering)?
Is it a good idea to use BERT embeddings to get features for documents that can be clustered in order to find similar groups of documents? Or is there some other way that is better?

original:
'Trump tries to smooth things over with GOP insiders. Hollywood, Florida (CNN) Donald Trump\'s new delegate guru told Republican Party insiders at a posh resort here on Thursday that the billionaire front-runner is recalibrating the part "that he\'s been playing" and is ready
tokenized:
['[CLS]',
 'trump',
 'tries',
 'to',
 'smooth',
 'things',
 'over',
 'with',
 'go',
 '##p',
 'insider',
 '##s',
 '.',
 'hollywood',
 ',',
 'florida',
 '(',
 'cnn',
 ')',
 'donald',
 'trump',
 "'",
 's',
 'new',
 'delegate',
 'guru',
 'told',
 'republican',
 'party',
 'insider',
 '##s',
 'at',
 'a',
 'po',
 '##sh',
 'resort',
 'here',
 'on',
 'thursday',
 'that',
 'the',
 'billionaire',
 'front',
 '-',
 'runner',
 'is',
 'rec',
 '##ali',
 '##bra',
 '##ting',
 'the',
 'part',
 '"',
 'that',
 'he',
 "'",
 's',
 'been',
 'playing',
 '"',
 'and',
 'is',
 'ready',
 '[SEP]']

Here are the answers:

In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not really the order). For example: In bag of words: "I ate ice-cream" and "ice-cream ate I" are same, while this is not true for the models that treat entire sentence as ordered sequence. Thus, you cannot pick N random words in a random order.
Choosing tokens is model dependent. You can always preprocess to remove stop words and other contents such as symbols, numbers, etc if it acts as noise than the information.
I would like to clarify that lemmatizing and word-piece tokenization is not the same. For example, in lemmatization "playing" and "played" are lemmatized to "play". But in case of word-piece tokenization it's likely split into "play"+"##ing" or "play"+"ed", depending on the vocabulary. Thus, there is more information preserved.
max_length should be optimally chosen such that most of you sentences are fully considered. (i.e, most of the sentences should be shorter than max_length after tokenization). There are some models which considers complete sequence length. Example: Universal Sentence Encoder(USE), Transformer-XL, etc. However, note that you can also use higher batch size with smaller max_length, which makes the training/fine-tuning faster and sometime produces better results.
The pretrained model is trained with MAX_LEN of   512. It's a model's limitation.
In specific to BERT,as claimed by the paper, for classification embeddings of [CLS] token is sufficient. Since, its attention based model, the [CLS] token would capture the composition of the entire sentence, thus sufficient. However, you can also average the embeddings of all the tokens. I have tried both, in most of my works, the of average of all word-piece tokens has yielded higher performance. Also, some work's even suggests you to take average of embeddings from the last 4 layers. It is merely a design choice.
Using sentence embeddings are generally okay. But, you need to verify with the literature. There can always be a better technique. Also, there are models specific to sentence embeddings (USE is one such model), you can check them  out.


Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process? I am new to RL and I am trying to understand how to find solutions of an MDP.
This is what I understand so far -> since the nature of our environment is stochastic, at a state 's' if we take an action 'a' we do not know which state we would end up in. So we define the following terms :

T(s, a, s') or P(s'/s, a) - transition probability. This is the probability that starting at state 's' and taking action 'a' we end up in state s'

V(s) - value of a state, which tells us how good or bad this state 's' is.

R(s) - reward of being in a state 's', this is also known as the immediate reward of being in state 's'

Q(s) - state action value function. This tells us that given we are at a state 's' and we take a particular action 'a' what is the expected utility of the state s' we might end up in

Now coming to the part that confuses me.
From what  I understand so far, the difference between v(s) and q(s) is that v(s) gives us the utility of a state assuming we do not know which action to take while q(s) is the value of that state given we take a particular action 'a'.

Coming to $v^\pi(s)$ = same as what I defined v(s) to be, so the value of a state
$v^*(s)$ = value of a state given we take the optimal action 'a' so this is like following max (q(s)) over all actions

I read online how to mathematically define these terms and I am not sure if I correctly understand them. Also, I don't really understand the difference between $v^*(s) $ and $v^\pi(s)$
I found some conflicting ideas online but since I am just starting out I am not sure what is lacking in my understanding:

$
V^\pi(s) = R(s) + \gamma*max\sum P(s'/s,a)*V^\pi(s')
$

$V^*(s) = max\ Q*(s,a)$ and here we can substitute for Q*(s,a) as $Q^*(s,a) = \sum T(s,a,s')*[R(s) + \gamma V^*(s')]$ so we get $V^*(s) = max  \sum T(s,a,s')*[R(s) + \gamma V^*(s')]$

$V(s) = \gamma*max\sum P(s'/s,a)*V(s')$

As you can see there are different equations which are defining the same V(s) quantity? So what exactly is the clear definition of V(s)? Are these equations just dependent on the case of RL we are dealing with?
I don't understand how to distinguish between these, any suggestions/links/readings are much appreciated! Thanks!

Your confusion seems to come from mixing up between some policy $\pi$ and an optimal policy $\pi^*$. Your summary is generally correct, but missing these extra details.
Let me try go through it again. Starting with the MDP definitions:

First of all, we have the transition probabilities $T(s,a,s') = P(s'|s,a)$ which are conditional probabilities of arriving at state $s'$, given that we've taken action $a$ in state $s$
The (expected) reward is generally associated with the state-action pair $R(s,a)$ - there is some slight variations about it in literature, but that's a subject for a different discussion.
Then we have the policy $\pi(a,s) = P(a|s)$ - the probability of taking an action $a$ in state $s$ for an agent following the policy.

Given all three things above: $T$, $R$ and $\pi$ (plus a discount factor $\gamma$) you can define your value functions as average discounted rewards collected by the agent that follows $\pi$.

$V^\pi(s)$ is the average reward that the agent following $\pi$ will collect, when starting from the state $s$.
$Q^\pi(s,a)$ is the average reward that the agent following $\pi$ will collect, when starting from the state $s$ and taking an action $a$.

Quite often authors drop the index $\pi$ and assume implicitly that we are dealing with some policy $\pi$, but one should keep in mind that some (maybe unspecified) policy is always in the context when we are talking about value functions $V$ or $Q$.
These value functions satisfy the following recursive relationships for any policy $\pi$ (note that there's no $\max_a$ in these):
$$V^\pi(s) = \sum_a\pi(a,s)\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)V^\pi(s')\right]$$
$$Q^\pi(s,a) = R(s,a) + \gamma \sum_{s',a'}T(s',s,a)\pi(a'|s')Q^\pi(s',a')$$
Now, some policies maximize the expected reward - these policies are called "optimal" and standardly denoted with a star: $\pi^*(a,s)$ - optimal policy. The  index $\pi$ for the value functions is usually replaced with a star as well: $V^*$ and $Q^*$. The optimal value functions satisfy Bellman equations (the ones that have $\max_a$ in them):
$$V^*(s) = \max_a\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)V^*(s')\right]$$
$$Q^*(s,a) = R(s,a) + \gamma \max_{a'}\sum_{s'}T(s',s,a)Q^*(s',a')$$
Hope this clarifies it.


Deep Q network disregards input, giving identical output no matter the input state I've creating a very simple game,
The board is an array of size 6.
0 is "empty cell" , 5 is "Goal", 8 is "player location"
[8 0 0 5 0 0] for example means the agent needs to move 2 "right" to win.

hitting a "5" ends the game with reward +1
hitting a wall ends the game with reward -1
other moves have reward 0.

the "state" is simply the board, 6-long vector that can be 0\8\5

When i built a regular Q-Table based agent the game converged (to a good result) very fast.

The problem is that the DQN (Deep Q network) agent never converged. i sampled the network after a few thousand iterations and saw that the network for all inputs always has the same result.

The actual result keeps changing, each replay of the memory, but at each point if i ask to predict, no matter the input, the output will be "LEFT: X, RIGHT: Y".
(X and Y are numeric values). the network ignores the input. any suggestions why?

Appendix:
Hyperparams:
batch-size = 64
memory = 100 (tried 1000 as well(
gamma = 0.99  # discount rate
epsilon_min = 0.05
epsilon_decay = 0.95
epsilon_start = 1.0
learning rate = 0.01 / 0.1 / 1.0

the network is ( i tried several):

        # Neural Net for Deep-Q learning Model
    model = K.models.Sequential()
    model.add(K.layers.Dense(8, input_dim=self.state_size, activation='relu'))
    model.add(K.layers.Dense(8, activation='relu'))
    model.add(K.layers.Dense(self.action_size, activation='linear'))

    model.compile(loss='mse',
                  optimizer=K.optimizers.Adam(lr=self.learning_rate))

The value function you want to learn should be relatively simple, but I would expect optimal action values to be quite similar between moving left or right, at least when that is away from a wall. Recall that action values measure the expected reward for taking the selected action, then following the policy from that point on.

So in state [0 0 8 5 0 0], the action value under an optimal policy for "move right" should be +1, but the action value for "move left" is not much different, at 0.9801 . . . because under an optimal policy the agent will move right twice afterwards when following the policy.

That means the neural network will have to learn fine detail differences between estimates. You have made this harder than it needs to be if you are using the representation as "raw" input to the neural network. Neural networks learn badly when the input number range is high. In addition your learning rates seem quite high.

My suggestions:

Scale the inputs. You should have code that turns the state representation into neural network features. I'd suggest the neural network will function nicely with +1 for goal and -1 for agent, if you must work with this state representation.
Try lower learning rates. A rate of $0.01$ is high for the Adam optimiser.
Reduce discount factor $\gamma$ - that will make a clearer difference in this short episode-length environment between returns for faster and slower routes.


Predict_proba on a binary classification problem I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the samples used for inference.
My understanding is that predict_proba for most classification algorithms isn't accurate and needs to be calibrated. Is there a common approach to get objectively accurate class probabilities ?
Algorithms names , techniques and some code if possible. Thanks!

Note : my classes are imbalanced 80/20.

sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using Platt's scaling (sigmoid) and one using isotonic regression.

Another method is by using a Venn-Abers predictor, which is not implemented in sklearn, however you can find a custom implementation here.

As for imbalanced datasets, make sure to exploit the class_weights argument of an sklearn classifier, to increase the weight of under-represented classes.


Library to calculate Distance Metrics on GPU Suggestions of widely used and fast GPU libraries that calculate the Euclidean distance of a vector from a matrix of vectors (of the same dimension) ?

Preferably one that works on NVIDIA GPUs.

Need it as a part of comparing solutions that evaluate Radial Basis Function Neural Networks.

If you are using python, use CUDA tool-kit with cudamat library. You will be able to perform matrix operations on GPU using cudamat.


Can Micro-Average Roc Auc Score be larger than Class Roc Auc Scores I'm working with an imbalanced data set. There are 11567 negative and 3737 positive samples in train data. There are 2892 negative and 935 positive samples in validation data. It is a binary classification problem and I used Micro and Macro averaged ROC for evaluation of my model. However, I noticed that Micro averaged Roc-Auc score is higher than Class specific Roc-Auc scores. That doesn't make sense to me.

As you can see on the plot, micro averaged roc-auc score is higher for all points. If it is possible can you explain the reason behind that ? I used that sklearn-link and I converted it for binary classification (y-true -> one hot representation). I also added my code on below.
xgboost_model = XGBClassifier(n_estimators= 450,max_depth= 5,min_child_weight=2)
xgboost_model.fit(X_train,y_train)

yy_true,yy_pred = yy_val, xgboost_model.predict_proba(Xx_val)# .predict_proba gives probability for each class

# Compute ROC curve and ROC area for each class
y_test = flat(yy_true) # Convert labels to one hot encoded version
y_score = yy_pred

n_classes=2
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

# Compute micro-average ROC curve and ROC area
fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel())
roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])

# First aggregate all false positive rates
all_fpr = np.unique(np.concatenate([fpr[i] for i in range(n_classes)]))

# Then interpolate all ROC curves at this points
mean_tpr = np.zeros_like(all_fpr)
for i in range(n_classes):
    mean_tpr += interp(all_fpr, fpr[i], tpr[i])

# Finally average it and compute AUC
mean_tpr /= n_classes

fpr["macro"] = all_fpr
tpr["macro"] = mean_tpr
roc_auc["macro"] = auc(fpr["macro"], tpr["macro"])

# Plot all ROC curves
plt.figure()
plt.plot(fpr["micro"], tpr["micro"],
         label='micro-average ROC curve (area = {0:0.2f})'
               ''.format(roc_auc["micro"]),
         color='deeppink', linestyle=':', linewidth=2)

plt.plot(fpr["macro"], tpr["macro"],
         label='macro-average ROC curve (area = {0:0.2f})'
               ''.format(roc_auc["macro"]),
         color='navy', linestyle=':', linewidth=2)

colors = cycle(['aqua', 'darkorange', 'cornflowerblue'])
for i, color in zip(range(n_classes), colors):
    plt.plot(fpr[i], tpr[i], color=color, lw=lw,
             label='ROC curve of class {0} (area = {1:0.2f})'
             ''.format(i, roc_auc[i]))

plt.plot([0, 1], [0, 1], 'k--', lw=lw)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve for nonSampling Training Data')
plt.legend(loc="lower right")
plt.savefig('nonsample.png', format='png', dpi=600)

plt.show()

In a binary problem there's no reason to average the ROC (or any other metric). Normally micro and macro performances are used to obtain a single performance value based on individual binary classification measures in the case of multiclass classification.
So here what happens is this:

The ROC curves for the two classes are the mirror of each other (using the top-left bottom-right diagonal as symmetry axis), because they represent exactly the same points but with the positive and negative classes swapped.
The macro-average curve is the average of both curves, which doesn't make a lot of sense since both already have the same shape. This is why all the AUC values are identical for macro, class 0 and class 1.
The micro-average ROC is the weighted average, so it's made mostly of the majority class (around 75%): since most points in the majority class are correctly predicted as the majority class, the performance looks much better. This is related to the fact that the micro-average F1-score is equal to accuracy, although in the ROC I'm not sure how the points are weighted exactly.

In my opinion, this is a good illustration of why ROC curves should be used very carefully (or not at all) in a multiclass setting. ROC curves are meant for binary (soft) classification, they are useful and interpretable in this context but not necessarily in another context.
In general it also illustrates that it's not because something can be calculated that the resulting value makes sense ;)


How to apply the gradient of softmax in backprop I recently did a homework where I had to learn a model for the MNIST 10-digit classification. The HW had some scaffolding code and I was supposed to work in the context of this code.

My homework works / passes tests but now I'm trying to do it all from scratch (my own nn framework, no hw scaffolding code) and I'm stuck applying the grandient of softmax in the backprop step, and even think what the hw scaffolding code does might not be correct.

The hw has me use what they call 'a softmax loss' as the last node in the nn. Which means, for some reason they decided to join a softmax activation with the cross entropy loss all in one, instead of treating softmax as an activation function and cross entropy as a separate loss function.

The hw loss func then looks like this (minimally edited by me):

class SoftmaxLoss:
    """
    A batched softmax loss, used for classification problems.
    input[0] (the prediction) = np.array of dims batch_size x 10
    input[1] (the truth) = np.array of dims batch_size x 10
    """
    @staticmethod
    def softmax(input):
        exp = np.exp(input - np.max(input, axis=1, keepdims=True))
        return exp / np.sum(exp, axis=1, keepdims=True)

    @staticmethod
    def forward(inputs):
        softmax = SoftmaxLoss.softmax(inputs[0])
        labels = inputs[1]
        return np.mean(-np.sum(labels * np.log(softmax), axis=1))

    @staticmethod
    def backward(inputs, gradient):
        softmax = SoftmaxLoss.softmax(inputs[0])
        return [
            gradient * (softmax - inputs[1]) / inputs[0].shape[0],
            gradient * (-np.log(softmax)) / inputs[0].shape[0]
        ]

As you can see, on forward it does softmax(x) and then cross entropy loss.

But on backprop, it seems to only do the derivative of cross entropy and not of softmax. Softmax is left as such.

Shouldn't it also take the derivative of softmax with respect to the input to softmax?

Assuming that it should take the derivative of softmax, I'm not sure how this hw actually passes the tests...

Now, in my own implementation from scratch, I made softmax and cross entropy separate nodes, like so (p and t stand for predicted and truth):

class SoftMax(NetNode):
    def __init__(self, x):
        ex = np.exp(x.data - np.max(x.data, axis=1, keepdims=True))
        super().__init__(ex / np.sum(ex, axis=1, keepdims=True), x)

    def _back(self, x):
        g = self.data * (np.eye(self.data.shape[0]) - self.data)
        x.g += self.g * g
        super()._back()

class LCE(NetNode):
    def __init__(self, p, t):
        super().__init__(
            np.mean(-np.sum(t.data * np.log(p.data), axis=1)),
            p, t
        )

    def _back(self, p, t):
        p.g += self.g * (p.data - t.data) / t.data.shape[0]
        t.g += self.g * -np.log(p.data) / t.data.shape[0]
        super()._back()

As you can see, my cross entropy loss (LCE) has the same derivative as the one in the hw, because that is the derivative for the loss itself, without getting into the softmax yet.

But then, I would still have to do the derivative of softmax to chain it with the derivative of loss. This is where I get stuck.

For softmax defined as:

The derivative is usually defined as:

But I need a derivative that results in a tensor of the same size as the input to softmax, in this case, batch_size x 10. So I'm not sure how the above should be applied to only 10 components, since it implies that I would diferentiate for all inputs with respect to all outputs (all combinations) or  in matrix form.

After further working on this, I figured out that:

The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an activation function is also valid.
The homework implementation is indeed missing the derivative of softmax for the backprop pass.
The gradient of softmax with respect to its inputs is really the partial of each output with respect to each input:

So for the vector (gradient) form:

Which in my vectorized numpy code is simply:

self.data * (1. - self.data)

Where self.data is the softmax of the input, previously computed from the forward pass.


Query relating to Pandas Rows manipulation I have a query regarding Pandas data manipulation.
Let's say I have a dataframe, df with following structure.
A B C
1 1 7
5 3 3
3 3 2
7 5 2
5 NaN 2

We have 3 columns in the dataframe A, B & C.
B column consists of mean values wrt A.
For example,
Value of B in 3rd row (which is 3) is mean of first 3 rows of A (9/3)
Similarly, value of B in 4th row = (Sum of values in 2nd,3rd and 4th row of A)/3
Now, let's say I have many NaN values in B and there are no NaN values in A, how do I write a function or code to fill the NaN values as per the logic discussed above?
I tried using loc and iloc but I guess I made some mistake.

Thank you for the above answer!
That definitely works. However, I found a more efficient way in terms of computation using np.rolling
df['D'] = df['A'].rolling(min_periods=1, window=3).mean()
df['B'] = np.where(df['B'].isnull,df['D'],df['B'])

np.rolling helps to compute the cumulative sum of previous n values.
np.where helps to apply some output based on a condition: syntax: np.where(condition,
value if true, value if false).
Column D can be dropped once it is used.


Find parameters to maximise output score Not sure this is the right place to ask. Lets say there is a function f() where its implementation is unknown but it returns a score. I would like to get the highest possible score by modifying the input parameters. I also try to be better than brute force (finding all possible combination of input parameters, if that is even possible)
I do know that

f() runs an algorithm against a known dataset. The algorithm is tweaked by the input parameters.
f() needs 6 parameters
I know the type of each parameter (int, float, boolean)
I know the range of each parameter i.e [-1,5](int), [0,1](float, percentage, i.e 0.5 = 50%)
Some parameters have an open range i.e >1 (int)
Some are dependent of each other. i.e min and max parameter. That is min < max.

Any good pointers to algorithms that could help me solve that would be highly appreciated.

In this kind of optimization problem a genetic algorithm is often a good approach, assuming computing the value of  f() is not too costly.
The idea is to represent the 6 parameters as "genes". In the first generation their values are assigned randomly, then each "individual" in the generation (combination of parameters) is evaluated (i.e. calculate f), and the top performing "individuals" are selected. The next generation is obtained by cross-over and random mutation, and the process is repeated until f converges to a maximum.


Comparing two time series data to find deviations between them This is a use case that I have and I am trying to automate this. Any pointers would be helpful.

Use Case:

When we deploy any new version of a web service, we keep monitoring it (while deploying to live) to make sure it is not introducing any new errors. For this, we just visually compare with last weeks trend of errors (within the same time frame) and if they look similar, we approve the new build, or if the number of errors seems to increase, we decide to rollback.

What I am looking for is to automate this decision making. Basically, compare the error data during the push with last weeks (or any other time frame) and decide if the two error trends are similar and to what degree they are similar.

The data that I have is
x axis -> Time stamp
y axis -> # of errors at that time stamp.

I also have details such as number of requests at that timestamp, latency, etc

Your use-case looks solvable using dynamic time warping. DTW is an algorithm for comparing similarity between two temporal sequences.

Implementation of DTW are available in standard data-science programming language via libraries like this R implementation.


Train and predict on different data structures Is it possible to train a Machine Learning model with a set of the same data structure, but use another data structure as input in prediction/classification ? If so, how ?

Let me elaborate and explain my motivation :
I am a summer intern in a research lab (and will be starting my last year in Mathematical Engineering, so I am not too deep into ML yet, I will be specializing this year). I have followed the work of a previous intern on Chronicle Mining.
A Sequence is a list of timestamped events, e.g.
<({A}, 1), ({A, C}, 2), ({A, B}, 4), ({B, C}, 7)>
is a sequence in which the event A happens at time 1, the events A and C happen at time 2, the events A and B happen at time 4 and the events B and C happen at time 7.
A Chronicle is a data structure based on graphs. It translates time constraints between events, or in my case, sequences of events.

This tells us that a first A event must come between 1 and 3 units of time before the sequence B, C, but B, C must come between 1 and 3 units of time before a second A event and between -2 and 2 units of time before A, C (so A, C can actually come before B, C). A first A event must come between 1 and 4 units of time before A, C, and you get the rest.
So as you can see, it is a directed graph, there can be mutliple nodes with the same label (which are sequences of events), multiple vertices with the same label (which are time intervals) and there can be negative values within the intervals. In my case, events are partially ordered (in this example it is lexicographical order) to make some things simpler.
I have accomplished efficient extraction of these weird things from sequential databases, so let's say I have a lot of them. In my case, there is always a final state (like the middle A in the example), and it is always the same (all sequences end with the same event/sequence).
Now, my tutor would like me to "perform Machine Learning" on these graphs. But there is a catch : the prediction/classification input has not the same shape as the training input. He would like for me to feed the algorithm a simple sequence (not a chronicle) and output some results. Thus, I have come up with two problems :

Given a large set of chronicles and a partial sequence (without the common final state), predict the sequence (and intervals between each event) between the last event on the sequence and the common final state.
Given a large set of chronicles and a partial sequence (without the common final state), tell whether the next k-th step will be the common final state.

Both are similar, but one is a prediction problem and the second one is a classification problem. However, in both cases, I don't know where to start !
I have read papers on k-th steps sequence prediction with LSTM NN and other techniques, but what gets me is the fact that I apparently have to train my model with chronicles, but predict using sequences. Is that even possible ? Do I have to transform my chronicles in some kind of way ?

From what I understood from your problem, the chronicles are a way to model relationship between events and if so they are extracted from the sequential data (as a higher interpretation of data). I had a similar project involving Episode Mining in sequential database involving alarms.

The problem is whether you are trying to extract chronicles or use existent ones to predict something in a sequence (which is like using a pre-trained model).

Training a model mostly involves adjusting weights to inputs and applying (non-) linear functions on them in order to output the result in a certain format. (This is a very broad and abstract way of describing most ML algorithms)

So to answer your question, No you cannot do it in this broad way but you could try to make it so that the inputs are homogenous.

Building a  model that extract from the sequence a chronicle would help or doing the exact opposite, generating a sequence with the chronicles and then perform machine learning.

Use simple Apriori/FPGrowth to extract rules (old way of doing things)
Build a (Hidden) Markov Model to estimate each state probability, in order to predict outcome.
Use RNN to learn how a sequence behaves (LSTM or GRU) with the fact that it's time-dependent, the RNN will remember important states and forget unimportant ones (which is similar to chronicles but in a black box kind of way).
Create something hybrid or completely new.


Insights betwwen two columns/variables in Dataframe I have data in two columns one is range of old credit score (Input score range) and new credit score (cvsc100). How do i find insights from both of them ? where the old is range of values and other column is not(CVSC100)

I know how to calculate Pearson Correlation in Python of Dataframe of two column . but that should not be sufficient i believe. How should i proceed can you please advise

The first step would be to bin cvsc100 in the same bin you have for your input, that will surely help your comparisons.

The second step would be to build a frequency table with your input range in columns and the binned CVS100 in rows, and at the intersection the count of values. This would help to observe how CVSC100 is distributed compared to your input and understand the underlying process.

Assuming that the score you have can be written as score for period n and period n+1,  with some hypotheses on the stationarity of the underlying process and modulo some slight renormalisation you can get transfer probabilities and use this matrix for simulations to get the distribution of scores for period n+2,... and so on.

Without more data (explanatory variables or actual default values), you can't really get more usefull metrics.


i have to classify an sms into categories like educational, bank related,etc. Is this a problem of topic modelling or text classification? If this is a problem of text classification, is a similar dataset available or i have to make one on my own??
I have a dataset of emails for classification into spam/ham.

Topic modeling is usually used in the context of unsupervised learning, while classification is used for supervised learning. So it more depends on your dataset, does it have labels that you can learn from (classification) or does it need to figure out topics itself (topic modeling)?


Sklearn Linear Regression examples Could someone give an example of the application of Tf-idf with sparse data (lots of zeros) in sklearn? I am not quite sure where to insert the weight of Tf-idf and how to rightly obtain the weight. Without knowing this fully, I would not be able to well use the tool for prediction. Thank you.

There is an application of tf-idf on the sklearn website.

sklearn handles sparse matrices for you, so I wouldn't worry about it too much:

  Fortunately, most values in X will be zeros since for a given document less than a couple thousands of distinct words will be used. For this reason we say that bags of words are typically high-dimensional sparse datasets. We can save a lot of memory by only storing the non-zero parts of the feature vectors in memory.
  scipy.sparse matrices are data structures that do exactly this, and scikit-learn has built-in support for these structures.

Regarding your point about inserting the weight, I guess you have already performed tf-idf on your training corpus, but you don't know how to apply it to your test corpus? If so you could so as follows (taken from above link)

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data)

# Perform tf-idf
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

docs_new = ['God is love', 'OpenGL on the GPU is fast'] # New test documents
X_test_counts = count_vect.transform(docs_new) # Count vectorise the new documents
X_test_tfidf = tfidf_transformer.transform(X_new_counts) # transfrom the test counts


Stacked features not helping I am wondering why my stacked features do not help me to improve against my loss metric. Here's what I'm doing: I am adding new features which are simple the predictions originated from train, predict of other   models to the original train/test features. Every time I have tried this method, it has failed. I am curious  what the issue could be the with this. Can anyone give me some advice?

As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features to another model.

First such kind of construction used logistic regression as a final ensemble and and class probabilities from each base learner as input features. Now, what I have described is a technical layout, the intuition behind is the following: considering that there are no models which are good over all joint probability space of features, one can combine their results in order to get the best from each one. In other words we can state the we explore the richness of models (seen as function spaces) to get a combined thing. This strategy does not work always but often it works.

I think you do something wrong. I think is better to use original features only for base learners. Be careful to use scores or probabilities if possible, instead of final classifications from base learners, it gives more space for improvements. Often is better to stack learners from different families, not the same model with different parameters (better to use a gradient boost and random forest than two gradient boosts). All of those advices are not rules which cannot be broken, and even if you take them all there is no guarantee that there will be improvements.


What's an efficient way to plot many features for EDA in python? I'm trying to visualize data for exploratory data analysis motivated by visualizing  multiple scatterplots of features simultaneously, similar to this question. But I quickly run into problems when using a large number of features (~50) and rows (~50K). While I like using seaborn pairplots the generation of a large number of plot panes can get computational intractable for a large number of features and observations. Subsetting a very large table to a smaller number of features or observations does not seem complete.

My question is: What's an efficient way to plot many features for EDA in python? If there is not an efficient way then is there a defensible way to reduce the number of features or observations

I would suggest t-SNE (just google it). It helps you to have a general overview on what is gling on with your high-dimensional data. Please note that parameters of t-SNE are pretty sensitive so you need to put some effort to find a good embedding.


Question and Answer Chatbot for Customer Support I want to build a chatbot that serves as a first line customer support on a  retail website. I have a large log of chat sessions between customers and support professionals that I can use. I am wondering what is the best way to go about building this chatbot.

Here are some ideas I have looked into;

The first thing I did was to breakdown the chat logs into Q&A pairs. Treat each question as a document, compute a term-document matrix and use it to retrieve the most similar question when a customer asks sometihng. The idea was to then simply pick the answer given to the question by the human (with some modifications). However, this gives really abysmal results and does not match to previous questions very well. Even if I get this approach to work well, we would be limited to the questions that already been asked.
Another thing I thought about was to modify something like ALICE and write some AIML for customer support related QA. However, that would take a lot of very precise AIML writint to work well. This solution would not be able to scale to other languages, which is something I want to do.
Another idea I have is to try and cluster the answers given by the customer support staff (they are more likely to be aligned compared to the questions asked by customers). This would give me a sense of what questions are similar. Then I can use something like LSA on the questions and fall back to the first approach.
Yet another way is to build an ontology of the domain specific knowledge and given a question decide which category the question falls into. The questions from the chat logs can then be mapped to the ontology and I could train classifiers for mapping questions to different knowledge areas. Once I reach the leaf nodes, I can give back pre-defined answers. However, my reservation is that mapping chat logs to the ontology would be very tedious. Is there a good way to map the existing QA to the ontology?

This was intended as a comment.

I think this question is not receiving enough attention because it is too difficult to answer without trying several approaches first.

Your third idea is a nice one, but LSA is most likely not going to be able to help you choose clusters with the necessary granularity. For example, you may get a cluster of "broken things" but not to the specificity that is required to be helpful to the customer. A similar approach would be using Hierarchical Dirichlet Processes, but again, you may not get the clusters that you want.

I wouldn't be so eager to try your fourth option because it requires a lot of work and I'm not sure you're going to get great results unless your customer support deals with very well-defined questions and in that case, you're going to have to build your own ontology from scratch.

In your situation, I would be curious if I can get information about the inner workings of Watson (obviously, the purpose is only to have a general idea of some steps that are sensible enough to try.) Here is a descriptive image that could be helpful from Wikipedia:

The main idea is to segment questions into its components (using a regular chunking), then come up with several possibilities (hypothesis generation) of what the question might refer to (maybe this step can be implemented as a clustering). The third stage is rating the previous hypothesis based on available information. Finally, you choose the more likely hypothesis. There is a feedback involved in this process using sample questions and their answers. It would be interesting to see if the rating stage can be modeled in terms of a Bayesian or an incremental approach to allow for a feedback.


Semantic analysis score as input to LSTM model for improving stock price prediction accuracy I have created a univariate LSTM model that is predicting value of Open Price based on last 5 years opening price of a particular stock. I'm getting a decent accuracy. Now, I want to do sentiment analysis of the news data of the last 5 years of that stock and combine that score with my existing LSTM model in order to improve the predicted values of LSTM model. Does anyone has an idea whether this implementation is possible and if yes how can I proceed. Thanks in advance :)

It is possible. Your problem will then become multivariate, where on top of the opening price for a particular day, you would also add a sentiment score for that day.
LSTM can handle multiple inputs natively, so there shouldn't be much for you to change.


machine learning disasters I am writing a research paper and I am looking for reliable sources that provide information on disasters of machine learning. Especially in the field of autonomous driving. Have there been any incidents when something went really wrong? Any links to articles or research papers would be helpful.

I don't know, there might be disasters which happened due to ML but my guess is that it might be hard to really consider ML as the cause: ML is always deployed in a setting designed by humans for a very specific task, and the design must take into account the statistical nature of ML. So I would tend to interpret most problems as caused by a wrong design, not by ML itself.
If you are looking more generally into problems in the applications of ML, as far as I know the most common problems cited are:

Increasing existing bias, for instance face recognition being is more accurate with white than black faces.
Lack of transparency, especially in critical applications like medicine, military, law enforcement
Lack of a legal responsibility framework, e.g. who is at fault if a self-driving car causes an accident?


Why is there a difference of "ML" vs "MLLIB" in Apache Spark's documentation? I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.

  https://spark.apache.org/docs/2.2.0/mllib-feature-extraction.html#word2vec
  https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec

mllib seems to be for using RDD's. And ml seems to be using "DataFrames".

What is the difference? Shouldn't they both be using RDDs if this is spark under the hood?

What is a "DataFrame" here? As the documentation doesn't explain it.

An RDD is a Read-only partition collection of records. RDD is or was the fundamental data structure of Spark. It allows a programmer to perform in-memory computations on large clusters in a fault-tolerant manner. Thus, speed up the task.

A dataframe on the other hand organizes data into named columns. It is an immutable distributed collection of data. DataFrame in Spark allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction.

The two are very different and more details on the two can be found on this link https://data-flair.training/blogs/apache-spark-rdd-vs-dataframe-vs-dataset/

mllib was in the initial releases of spark as at that time spark was only working with RDDs. Now a lot of Spark coding is done around dataframes, which ml supports. Spark has also put mllib under maintenance.

  As of Spark 2.0, the RDD-based APIs in the spark.mllib package have
  entered maintenance mode. The primary Machine Learning API for Spark
  is now the DataFrame-based API in the spark.ml package.

  Q. What are the implications?

  MLlib will still support the RDD-based API in spark.mllib with bug
  fixes. MLlib will not add new features to the RDD-based API. In the
  Spark 2.x releases, MLlib will add features to the DataFrames-based
  API to reach feature parity with the RDD-based API. After reaching
  feature parity (roughly estimated for Spark 2.3), the RDD-based API
  will be deprecated. The RDD-based API is expected to be removed in
  Spark 3.0.

  Q. Why is MLlib switching to the DataFrame-based API?

  DataFrames provide a more user-friendly API than RDDs. The many
  benefits of DataFrames include Spark Datasources, SQL/DataFrame
  queries, Tungsten and Catalyst optimizations, and uniform APIs across
  languages. The DataFrame-based API for MLlib provides a uniform API
  across ML algorithms and across multiple languages. DataFrames
  facilitate practical ML Pipelines, particularly feature
  transformations. See the Pipelines guide for details.

  Q. What is “Spark ML”?

  “Spark ML” is not an official name but occasionally used to refer to
  the MLlib DataFrame-based API. This is majorly due to the
  org.apache.spark.ml Scala package name used by the DataFrame-based
  API, and the “Spark ML Pipelines” term we used initially to emphasize
  the pipeline concept.

  Q. Is MLlib deprecated?

  No. MLlib includes both the RDD-based API and the DataFrame-based API.
  The RDD-based API is now in maintenance mode. But neither API is
  deprecated, nor MLlib as a whole.

You will find more details on Sparks official release https://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api


making a picture of a neural network in tensorflow I'm new to Machine Learning and I have just gone through a tutorial explaining how to create a neural network in TensorFlow. I was wondering if it is possible to visualize the neural network I created. The output should be a picture like this
Thanks.
MRE:
import tensorflow as tf

ann = tf.keras.models.Sequential()
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))
tf.keras.utils.plot_model(ann, to_file='model.png', show_shapes=True,
                          show_layer_names=True, rankdir='TB',
                          expand_nested=False, dpi=96
                          )

This should do the trick:
tf.keras.utils.plot_model(
        model, to_file='model.png', show_shapes=False, show_layer_names=True,
        rankdir='TB', expand_nested=False, dpi=96
    )

This will generate an image like:

Example: https://www.tensorflow.org/guide/keras/functional


Adding to jrouquie's answer, the formula would be:

For each column,

rescaled_feature = (input_feature - shift)  * inv_scale


Accuracy after selftraining didn't change I used Decisiton Tree Classifier which I trained with 50 000 samples. I have also set with unlabeled samples, so I decided to use self training algorithm. Unlabeled set has 10 000 samples. I would like to ask if it is normal, that after retrainig model with these 10 000 unlabeled samples, accuracy didn't chaned as well as confusion matrix has same values? I expected some changes (better or worse prediction). Thank you in advance.

Well, that is a bit of a turn down but: your model has limitations.

If the 50.000 data forms a complete set for your problem that means that more data won't be needed or helpful.

What do I mean by complete set is: there are enough samples to form a full rank correlation matrix in your feature space. So from your samples you can get a set that can generate all other samples in your feature space by linear combination.

Also, while your data might represent everything a decision three needs to know for classificating your data in the generated feature space, there may be other feature spaces that benefit from the extra data (such as deeper trees or other models)

You might try helping you decision tree by providing a few normalizations for data and feature engineering


Naive Bayes Multinomial, independence assumption misunderstood This is embarrassing but I think I miss understand something.

In multinomial distribution, "while the trials are independent, their outcomes X are dependent because they must be summed to n." wiki
Naive Bayes assumption is that the features are independent (given the class): "assume that each feature $x_{i}$ is conditionally independent of every other feature" wiki

Thus, how can 1&2 co-exist in what's known as the Multinomial Naive-Bayes model?

Specifically, $p(x_i|c)=p_i^{x_i}(1-p_i)^{n-x_i}\binom{n}{k}$ and $p(x_1,...,x_n)=\frac{n!}{\prod_i{x_i}!}\prod{p_i^{x_i}}$ thus $p(x_1,...,x_n) \ne \prod_ip(x_i|c) $

Their outcomes X are dependent because they must be summed to n

If you are really want to model multiple experiments probabilistically, please do not forget the normalization constant that make the unnormalized quantity normalized by integrating/summing them to one.

Multinomial Naive Bayes model, as described by Manning et al (2008), estimates the conditional probability of a particular word/term/token given a class as the relative frequency of term t in documents belonging to class c:

Thus this variation takes into account the number of occurrences of term t in training documents from class c, including multiple occurrences Not multiple experiments.

see The Naive Bayes Text Classifier


Example of binary classifier with numerical features using deep learning I would like to get more understanding of deep learning. Browsing the web I find applications in speech recognition and hand-written digits. However I would be interested to get some guidance on how to apply this in the classical setting:

binary classifier
numerical features (each sample is a numerical vector of $K$ entries, no 2D pixels or such).

I am doing my own experiments choosing learning rates, number of hidden neurons and so on, but I would be happy to see an application by somebody more experienced.

The software that I use offers weight initialziation using Restricted Boltzmann Machines (RBMs). I wonder whether this is useful in this context and whether the other special techniques that one encounters in the literature (convolutional NN) are useful here to.

Could anybody share a blog post, a paper or personal experience?

I used Binary classification for sentiment analysis of texts. I converted sentences into vectors by taking appropriate vectorizer and classified using OneVsRest classifier.

On another approach, my words were converted into vectors and there, I used a CNN based approach to classify. Both when tested on my datasets were giving comparable results as of now.

If you have vectors, there are already really good approaches available for binary classification which you can try. On Binary Classification with Single–Layer
Convolutional Neural Networks is a good read for you for classification using CNNs for starters.

This is one of the first blogs I read to gain more knowledge about this and doesn't require much of pre-requisites to understand(I am assuming you know the basics about convolution and Neural Networks).


How to generate plot of reward and its variance? I am new to reinforcement learning and I would like know how to generate a learning curve plot such as that shown below (taken from this blog post), that illustrates the reward (return) and its variance (shaded region). I would like to use Matplotlib or any other Python plotting framework.

To estimate the variance, you probably need to run your algorithm multiple times and keep track of the return for each of these runs. From these multiple returns, you can estimate the variance.

Once you have the standard deviation (or variance) of the return, you can plot something like your plots using matplotlib's fill_between function.


First and foremost, you need to reformat your data into what's called a balanced panel structure. For each day in your training data, each user should have a record for that day associated with an indicator variable for whether or not they visited. If every record in your training data corresponds to a visit, you're not giving those classifiers much to work with. You're model needs to know what a user's status looks like on days when they don't visit to be able to discriminate between days when they will and won't visit. The indicator variable, denoting "this user visited today", is the target of your prediction. Not the date or time column, if that's what you were trying.

Additionally, you're going to need to do some feature engineering. This process comprises the bulk of time and creativity expended on most data science projects, and the amount of effort you will put in will make or break your project. For each record in the data set, you should calculate things like:

What is the day of the week/month/year?
how many times did the user visit on this day of the week in the last k days?
how many times did the user visit on weekdays/weekends in the last k days?
How many days has it been since the users last visit?
How many times has the user visited the library in the last k days?
how many times has the user visited the library and paid a fine in the last k days?
How many users visited in the last k days?
Is the library open this day (e.g. it could be a national holiday)?
Are the public schools currently in session? On holiday? In finals?
How many days since the beginning of the most recent semester?
How many days until the semester ends?

Better yet, if you're not limited to what you've shown us...

How many books are currently in the users possession? How many are late?
If the user visited today, how much would they owe in fines?
how many books did they check out in their last visit?
how many books have they checked out in the last k days?
how long has this person been a member at the library?
has their membership expired?
in the last k years, have they allowed their membership to expire?
have they recently updated their address?
How much have they donated in the last k years?
What fraction of books checked out in the last 5 years by this user have belonged to each dewey decimal class?

These are just some ideas off the top of my head. Don't let yourself be limited by my suggestions.

If you really only have the data you've shown us, a project you can try that would be more likely to give you good results would be to predict just the number of people who will visit the library on a particular day, rather than which actual people. If this sounds like it might be useful to you, check out Poisson models.


Question about reshaping array size for KNN Classifiers I keep trying to run a new set of data through my KNN Classifier but would recieve the message:
 ValueError: query data dimension must match training data dimension

It then used:
x_new = pd.read_csv('NewFeaturePractice.csv' , names = attributes)
x_new = x_new.values.reshape(52,84)

(which is the dimensions of the training data) but would then receive:
ValueError: cannot reshape array of size 672 into shape (52,84)

The second data set doesn't have the same amount of rows as the first meaning that even if I tried reshaping the array I would have several empty spaces. How can I run the code so that no matter the size of my new data set, I can still get results?

Seems you are making two mistakes -
Reshaping dim. is mismatched with the size i.e. 10 != 3X3
Dim in train and test are different. Trained of 5 features and testing on 4 features.
What I will suggest -
 You would be aware of your features count - let's say N
 then reshape using reshape(-1,N). It will automatically decide the first dim.
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
N=2
x.reshape(-1,N)


What are the criteria for updating bias values in back propagation? During back propagation, the algorithm can modify the weight values or bias values to reduce the loss.

How does the algorithm decide whether it has to modify the weight values or bias values to reduce the loss?

Does it modify the weight values in one pass and bias values in another pass?

Thanks!

Actually, weight values and bias values are updated simultaneously in each pass of backpropagation. That’s because the orientation of loss gradient vector is determined by the partial derivatives of all weights and biases with respect to the loss function. So if in each pass, you want to move in the correct direction towards the minimun of loss function, you must update both weights and biases at the same time and in the correct orientation.


Reshaping big dataset with MinMaxScaler giving error My data set is of shape (1249, 228). Most of the entries are zero and other are integers like 1,2,5,10,20 etc. I want to transform this set for the input into LSTM. But when I am applying MinMaxScaler. It is giving the following error:

load the dataset:

dataset1 = pd.read_csv('g:/hello.csv', engine='python')
dataset1= dataset1.drop('packages', axis=1)
dataset1 = dataset1.astype('float32')

normalizing the set

scaler = MinMaxScaler(feature_range=(0, 1))
dataset1 = scaler.fit_transform(dataset1)

  ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

How can I transform this data set according to the input in LSTM.

I assume you checked for NaN and Inf values manually.

According to the solutions posted here: https://stackoverflow.com/a/44869902/6204860 and
https://datascience.stackexchange.com/a/11933/52089

if dataset1 is a Pandas DataFrame try converting it to a matrix by running this:

dataset1 = dataset1.as_matrix().astype(np.float)
np.nan_to_num(dataset1)

and then run dataset1 = scaler.fit_transform(dataset1).

Let me know if this works :)


Can Keras be used to build clustering models The keras.wrappers.scikit_learn module can be used to build KerasClassifier model.
Can Keras be used to build clustering models?
If it can be, are there any examples for that?

Yes, you can use Keras to build clustering algorithms. Recalling that Keras is a high-level api for Tensorflow or Theano, and that you can build a K-means algorithm (very well known clustering technique) in Tensorflow, you should be able to build it in Keras. Example (in tensorflow):https://learningtensorflow.com/lesson6/.

Nevertheless, sklearn has in-built clustering algorithms like K-means or Gaussian Mixture Models, so at first, I think a better approach is using these sklearn funcionalities.


R, keras: How to get output of a hidden layer? I am using package Keras in R to do a neural network. How may I extract the output from a hidden layer? I found an example in python, but it is just I have no idea how to do that in R.

You can get the answer here.
Here it is:
model <- ...  # create the original model
layer_name <- 'my_layer'
intermediate_layer_model <- keras_model(inputs = model$input,
                                    outputs = get_layer(model, layer_name)$output)
intermediate_output <- predict(intermediate_layer_model, data)


Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one position for one office. I am storing this information into a preset "key" for what the offices are... so when there are two individuals for one position in one office it just duplicates the information except for where there was the NAME1/NAME2.  The first line will use NAME1 and the second line will use NAME2.
For example:
key1  ...    NAME1
key1  ...    NAME2
key2  ...    NAME3
key3  ...    NAME4

I'd like to make this end up as:
key1  ...    NAME1/NAME2
key2  ...    NAME3
key3  ...    NAME4

So that the duplicates are merged by adding a / ... This should be as simple as string concatenation. But I am struggling without making a bunch of loops to do this.

Without a reproducible exmple is difficult but I would go with something like this:
df = pd.DataFrame({"key_column":["key1","key1","key2","key3"],"name_column":["NAME1","NAME2","NAME3","NAME4"]})

df.groupby("key_column").name_column.apply(lambda x: "/".join(list(x)))

Result:
key_column
key1    NAME1/NAME2
key2          NAME3
key3          NAME4
Name: name_column, dtype: object


Decision Tree generating leaves for only one case I had asked a question regarding predictive analysis for marketing earlier. Prediction model for marketing to prospective customers (using pandas) Still have some doubts about it, but I have a doubt regarding the decision tree that I generated for the marketing data.
My aim is to predict if a lead will be won or lost, depending on how they were made aware of the product etc. I have a bool variable "Won", 0-sale had failed, 1-sale was made. Using a decision tree, I was able to generate a model, however, there are no leaves for cases that lead to Not Won.
Is this normal? Ive seen examples of the iris data set where all 3 features were represented in the tree, adn, hence am wondering if the approach Ive taken is correct.
In the dataset of 38000, there are roughly 1700 who have Won=1.

Im using pandas and my parameters for DecisionTreeClassifier are:
min_samples_split=2
min_samples_leaf=1
max_depth=3
I got then using grid search for parameter optimisation. I got a mean val of 0.95. If I use a bigger depth, the tree becomes too big to analyse.
Thanks
EDIT
Since Mark said to post some code, I am doing so
from sklearn.tree import DecisionTreeClassifier,export_graphviz
dt=DecisionTreeClassifier(min_samples_split=2,min_samples_leaf=1,max_depth=3)
dt.fit(x,y)
features=x.columns
#print(features)
with open("dt.dot", 'w') as f:
    export_graphviz(dt, out_file=f,feature_names=features,class_names=["Won","Lost"])
command = ["dot", "-Tpng", "dt.dot", "-o", "hello.png"]
try:
    subprocess.check_call(command)
except:
    exit("Could not run dot, ie graphviz, to produce visualization")

from sklearn.cross_validation import cross_val_score
scores=cross_val_score(dt,x,y,cv=10)
print(scores.mean())

This is the main training code, all previous lines are just munging
The cross validation score come to 0.95

Here is the csv snapshot
I use all values from "Won" onwards and am training for ""Won"
There was a single column X, which had many categorical values (20), "Won" being one of them.
Known' 'Recycled' 'Engaged' 'Prospect/MQL' 'Intern Transfer' 'MQL'
 'Working' 'Opportunity' nan 'Current Customer' 'Vendor' 'Disqualified'
 'Converted' 'SAL' 'SQL' 'engaged' 'working' 'Won' 'Web Registration'

'Inactive
However, they were all uniformly distributed ie. from 37000, all had almost the same number of observations. I used get_dummies to transform to numerical values, and dropped all the columns except "Won"
All the rest of the values value for things like designation, opp(money value scaled from 1-3) and other categories, which are all boolean

You are predicting binary class: 0 - sale failed, 1 - sale succeed. In the decision tree, the information value is sorted. So for example in the first node, you have 35011 predictions of 0 and 1785 predictions of class 1.

But then, in your code you have this: class_names=["Won","Lost"]). So you are telling your decision tree that name of class 0 is "Won" and name of class 1 is "Lost". I assume it should be inverted.

So in fact, the model always predict 0 - sale failed. Which seems reasonable as you said that out of 38000 samples, only 1700 are class 1.

Further, your dataset is highly unbalanced. So when cross validation gives accuracy of 95%, it says nothing, because only if your model gives all prediction 0, you will have accuracy of...95%.
For those cases, use models which can put weights to classes (SVM, Logistic regression or Random Forest classifier in scikit have this possibility). Think about undersample your dataset. Or add more of class 1 - dig in your database or do it artificially (SMOTE),

Plot confusion matrix and give CV different score measure - precision or recall, depends on your situation.

Last thing, don't expect that your decision tree with depth of 3 will perform well. It is nice to show to business, but it gives poor prediction.


How to select features for clustering to detect the number of different unique products in a search result? I am trying to use clustering to determine the number of products in a search of products. So far I am using kmeans clustering. I have run into a problem where I cannot determine good features to use. Here are my clustering results for one search.

As you can see, there is clearly three different groups showing up at the moment. However, when you look at the top terms of each cluster, there is a clear issue. A lot of the terms are repeating in each cluster.

There is another issue, when I try to have the model predict which clusters a few test items are in, which are clearly different products, it says that they are in the same group. Here are the test terms:

Adjustable DC-DC LM 2596 Converter Buck Step Down Regulator Power Module LW SZUS
10pcs Mini360 3A DC Voltage Step Down Power Converter Buck Module 3.3V 5V 9V 12V
5V Out, 6V to 12V In AMS1117-5.0 5.0V Step-Down Linear Voltage Regulator Module
10pcs Mini LM2596s 3A DC to DC Buck Converter Power Supply Step Down Module

I have thought about using the grammar in each title as a feature, but as there is an 80 character limit for the titles, there rarely is any grammar. I am at a loss for where to go next in terms of trying to pick more features that would get me better results. I am doing all of this in python using scikitlearn and nltk.

You cannot expect clustering to understand what a "product" is. So it will not make one cluster per product.

Also, the lines are very blurry: is the same mobile phone in different colors the same product, or not?

An unsupervised method cannot do this. You need to provide training data and use supervised learning.


Pretrained graph embeddings Is there any precedent for DL use of pretrained graph embeddings in a similar manner to word embeddings?

As far as I understand the topic embeddings are not portable between graphs since every graph has their own characteristics which you need to learn. At the field of word embeddings, it is different since domain stays the same (words).

In the field of graphs, you usually train the model on the same graph that will be embedded later. It is also important to train on the same graph because the labels of nodes are usually nodes IDs which are graph specific. In the field of words embeddings, the method recognizes words by their name (sequence of letters) while in graphs nodes are recognized by their IDs.

Hope that it explains a bit. If you want to understand graph embeddings, in general, I suggest reading this story.


Why is np.where not returning '1'? Only returns '0' This code should return a new column called orc_4 with the value of 1 if the value of the row in df['indictment_charges'] contains 2907.04 or 0 if not.
Instead it is returning all 0's
for index, item in enumerate(df.indictment_charges):
    s = '2907.04'
    if s in str(item):
        df['orc_4'] = np.where(item == s, 1, 0)

Why won't it return 1?
Example output for df.indictment_charges:
['2903.112907.022907.042907.04']

You are first checking if the item contains the string, but then in np.where you are checking if the values are equal (item == s), which is obviously different. In addition, you set the whole column equal to the value from np.where (and overwriting it after each row), which results in the whole column getting the value based on the final row of the dataframe.
To avoid looping over the rows (which is relatively quite slow) you can use pandas.Series.str.contains to check if a string contains a certain value like this:
df["orc_4"] = np.where(df["indictment_charges"].str.contains("2907.04"), 1, 0)


Train and predict on a varying number of inputs - time based events I have the problem where I am trying to build a model which takes in n events for a single user as input for prediction, the problem is that the number of events is not set, so for example:

A user may have performed a single event:
time - eventType - eventData

Yet another user for whom I need a prediction may have performed many of these time based events.

What architectures / models can I explore?

When dealing with sequence data it is usual to have sequences with different sizes. One answer for your problem might be to using padding in your events, in this way you keep a fixed number of events for all your data.
It happens in the same way with NLP applications, let's set number of events to be five.
So you have the following two rows in your data set:
UserID | time     | eventType | eventData
1        08:00:00      A           X
1        08:01:00      A           X
1        08:02:00      B           X
1        08:03:00      C           Y
1        08:03:00      A           Z
1        08:00:00      A           X
2        07:00:00      A           Z
2        07:01:00      B           X

So, you have for userId = 1 six events and for userId = 2 two events. Using paddings = 5, you will add 'Null'/Zero data for user 2 and you will truncate one event for user 1. Usually, you keep the last ones, so your data will become:
UserID | time     | eventType | eventData
1        08:01:00      A           X
1        08:02:00      B           X
1        08:03:00      C           Y
1        08:03:00      A           Z
1        08:00:00      A           X
2        07:00:00      Null        Null
2        07:00:00      Null        Null
2        07:00:00      Null        Null
2        07:00:00      A           Z
2        07:01:00      B           X

But what about the null values? You can use GradientBoosting algorithms that deals with null values by default. Or a missing imputation stretegy such as the most common value or a new class indicating null value.
Then, to finish, you can transform your data to have each user a row in your dataset:

Notice that your 'window' size is a hyperparameter and it is a good practice to define it via cross-validation.
I hope it helps.


Why is ReLU used as an activation function? Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.

Which I am able to understand intuitively for the activation functions like sigmoid.

I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation. However, I am not able to understand why is ReLU used as an activation function if its output is linear?

Doesn't the whole point of being the activation function get defeated if it won't introduce non-linearity?

In mathematics (linear algebra) a function is considered linear whenever a function$f: A  \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property:  $f(x) + f(y) = f(x+y)$.  By definition the ReLU is $max(0,x)$.  Therefore, if we split the domain from $(-\infty, 0]$ or $[0, \infty)$ then the function is linear.  However, it's easy to see that $f(-1) + f(1) \neq f(0)$.  Hence by definition ReLU is not linear.
Nevertheless, ReLU is so close to linear that this often confuses people and wonder how can it be used as a universal approximator.  In my experience, the best way to think about them is like Riemann sums.  You can approximate any continuous functions with lots of little rectangles.  ReLU activations can produced lots of little rectangles.  In fact, in practice, ReLU can make rather complicated shapes and approximate many complicated domains.
I also feel like clarifying another point.  As pointed out by a previous answer, neurons do not die in Sigmoid, but rather vanish.  The reason for this is because at maximum the derivative of the sigmoid function is .25.  Hence, after so many layers you end up multiplying these gradients and the product of very small numbers less than 1 tend to  go to zero very quickly.
Hence if you're building a deep learning network with a lot of layers, your sigmoid functions will essentially stagnant rather quickly and become more or less useless.
The key take away is the vanishing comes from multiplying the gradients not the gradients themselves.


Using categorical_crossentropy for binary classification Is it ok to use categorical_crossentropy for binary classification or is it better to use binary_crossentropy

Binary cross-entropy is a special case of categorical cross-entropy with just 2 classes. So theoretically it does not make a difference.

If $y_k$ is the true label and $\hat{y}_k$ is the predicted label of class $k$ (both one-hot encoded, i.e. $\sum_k y_k =1 \land \sum_k \hat{y_k} =1 \land y_k,\hat{y_k} \in \{0,1\}$) the multicategorical cross-entropy for $K$ categories is
$$-\sum_{k=1}^{K}y_k \log(\hat{y}_k)$$

which is equal to the binary cross-entropy for $K=2$ :
$$-\sum_{k=1}^{K}y_k \log(\hat{y}_k) = -y \log(\hat{y})- (1-y) \log(1-\hat{y})$$

Just in case there are any implementation differences, e.g. speed-wise, I would still just use binary cross-entropy for a binary classification problem instead of multicategory with $K=2$.


What's the issue with my code for visualizing linear regression in 3 dimensions with matplotlib? I am trying to use linear regression that takes two variables "Idade" and "LF" and tries to predict a third one, "DGAF". I'm trying to both do the scatterplot with the observations and the model prediction on the same graph. Bellow is the code I used (using Python).
X = df[["Idade","LF"]]
y = df["DGAF"].values.reshape(len(df["DGAF"]),1)

reg = LinearRegression()
reg.fit(X, y)

fig = plt.figure()
ax = Axes3D(fig)

ax.scatter(X['Idade'],X['LF'], y,s=10)

x_pred = np.linspace(0,100,1000)
y_pred = np.linspace(0,100,1000)
xx_pred, yy_pred = np.meshgrid(x_pred, y_pred)

model_viz = np.array([xx_pred.flatten(), yy_pred.flatten()]).T

ax.set_xlabel('Years since oil change')
ax.set_ylabel('LF')
ax.set_zlabel('DGA Score')

ax.plot(model_viz[0], model_viz[1], reg.predict(model_viz),color='red')
plt.show()

I get this error:
ValueError: input operand has more dimensions than allowed by the axis remapping

I am able to do the scatterplot, the issue seems to be with plotting the prediction. How can I solve this?

Following the suggestion given by @Ubikuity, I changed the ax.plot to ax.scatter, but there were still some issues with the dimensions. It was due to the transposition of the matrices, that was being done in the wrong place. The code below works correctly.
X = df[["Idade","LF"]]
y = df["DGAF"].values.reshape(len(df["DGAF"]),1)

reg = LinearRegression()
reg.fit(X, y)

fig = plt.figure()
ax = Axes3D(fig)

ax.scatter(X['Idade'],X['LF'], y,s=10)

x_pred = np.linspace(0,100,100)
y_pred = np.linspace(0,100,100)
xx_pred, yy_pred = np.meshgrid(x_pred, y_pred)

model_viz = np.array([xx_pred.flatten(), yy_pred.flatten()])

ax.set_xlabel('Years since oil change')
ax.set_ylabel('LF')
ax.set_zlabel('DGA Score')

ax.scatter(model_viz[0], model_viz[1], reg.predict(model_viz.T),color='red')
plt.show()
```


What machine learning model should be used to predict coincidence factor I have coincidence  factor for different sizes of groups and the associated attributes (e.g. building usage type and floor area) for each consumer in the group. I want to predict coincidence  factor for new groups using the attribute for each consumer in the new group and the number of consumers in the group. What machine learning method can solve the problem? :)

Defintions:
Coincidence factor is the peak of a system divided by the sum of peak loads of its individual components

From what you are saying I guess you have both categorical and continuous variables, One suggestion would be to check the distributions of your features to see if, more or less, the criteria for a linear regression are met.
If you have too many variables, which I still haven't picked up, you could even apply a PCA and then a regression.


Effects that Empty cells have in Unsupervised Machine Learning I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on.

The file itself has some cells that are not filled - they are just empty:

As shown in the picture above, some of the cells are empty. So the question follows - will these empty cells effect the outlier detection process which I would like to use?

Or should I instead work on a code that fills these empty cells with 0's

I have been told that that filling the empty cells has to be done so we don't have data sparsity, even though my csv files is thousands of rows long

Any advice regarding this is more than welcome.

Empty cells can be considered NaN of missing values. There are several ways to work with missing values. You can check this source:

Encode NAs as -1 or -9999. This works reasonably well for numerical features that are predominantly positive in value, and for tree-based models in general. This used to be a more common method in the past when the out-of-the box machine learning libraries and algorithms were not very adept at working with missing data.
Casewise deletion of missing data. Here you simply drop all cases/rows from the dataset that contain missing values. In the case of a very large dataset with very few missing values, this approach could potentially work really well. However, if the missing values are in cases that are also otherwise statistically distinct, this method may seriously skew the predictive model for which this data is used.
Another major problem with this approach is that it will be unable to process any future data that contains missing values. If your predictive model is designed for production, this could create serious issues in deployment.
Replace missing values with the mean/median value of the feature in which they occur. This works for numerical features. The choice of median/mean is often related to the form of distribution that the data has. For imbalanced data, the median may be more appropriate, while for symmetrical and more normally distributed data, the mean could be a better choice.
Label encode NAs as another level of a categorical variable. This works with tree-based models and other models if the feature can be numerically transformed (one-hot encoding, frequency encoding, etc.). This technique does not work well with logistic regression.
Run predictive models that impute the missing data. This should be done in conjunction with some kind of cross-validation scheme in order to avoid leakage. This can be very effective and can help with the final model.
Use the number of missing values in a given row to create a new engineered feature. As mentioned above, missing data can often have lots of useful signal in its own right, and this is a good way to encode that information.


Using class weights with validation data in Keras I'm using validation data for hyperparameter optimization and am trying to use class weights. For model.fit(), there is an option to pass class weights (the argument is class_weight) to the function for test data, but I'd like to do the same to validation data.

Another user had the same question a while back but no satisfactory answers were given:

Class weighting during validation in Keras

When passing validation data as an argument to model.fit(), the validation data can take the form of a tuple: (x_val, y_val, val_sample_weights) but in the Keras documentation I'm not seeing any way to use class weights instead of sample weights.

Does anyone have any idea on how to use class weights with validation data in Keras?

Here is some info that might be helpful:

From the Keras documentation, description of the 'class_weight' argument: "Dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to "pay more attention" to samples from an under-represented class."

Here is a notebook from Francois Chollet which shows an example of using class_weight: https://colab.research.google.com/drive/1xL2jSdY-MGlN60gGuSH_L30P7kxxwUfM#scrollTo=REP1KlrSEx-m

Thanks for any help!

Class weights make sense only in the context of a loss function. When you validate your model you are making predictions and comparing to ground truth using a metric - but in that phase you aren't propagating back any changes, so weights are useless.