Multinomial Naive bayes giving wrong result from sklearn.naive_bayes import GaussianNB,MultinomialNB
xx = [[1],[1],[1],[2],[2],[3]]
yy = [1,1,1,0,0,0]
clf = GaussianNB()
# clf = MultinomialNB()
clf.fit(xx,yy)
clf.predict(xx)

The expected result is [1,1,1,0,0,0]
but code output is [0,0,0,0,0,0].

There is only one feature for xx.

For a MultinomialNB, the modelled probability of the feature per class $p(x_i|y)$ is always the same ($\frac{3}{3}=1$ for class $1$ and $\frac{7}{7}=1$ for class $0$). The prediction is therefore the class prior $p(y_i)$ which is also the same here ($p(0) = 0.5$ and $p(1)=0.5$) so by default class $0$. Having one feature does not really makes sense because it gives no information to the model on what gives discriminative power toward the classes.

For a GaussianNB this is different because the modelled probability is given by $P(x_i \mid y) = \frac{1}{\sqrt{2\pi\sigma^2_y}} \exp\left(-\frac{(x_i - \mu_y)^2}{2\sigma^2_y}\right)$ so the mean and variance per class of the single feature give sufficient information.


is SST=SSE+SSR only in the context of linear regression? the problem of regression is to minimize the sum of squared errors, i.e. $\sum\limits_{i=1}^n (y_i - \hat{y}_i)^2 = 0$ .
But only in linear regression could you use the expression $\hat{y}_i = \beta_0 + \beta_1.x_i$ , then minimize the sum of squared errors w.r.t. $\beta_0$ and $\beta_1$ to obtain the following constraints:
$ \begin{align*} \sum\limits_{i=1}^n x_i(y_i - \hat{y}_i) &= 0 \\ \sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) &= 0 \end{align*} $
and then these constraints are used to prove that the quantity $\sum\limits_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) =0$, which yields SST=SSE+SSR.
So, my question is, if suppose regression wasn't linear, would:

SST=SSE+SSR still hold? If yes, why? If no, why?
If the answer to the above question is a no, then a follow-on question I would like to pose is: why is $R^2$ the go-to measure for assessing the regression performance of a model or a technique?

if suppose regression wasn't linear, would SST=SSE+SSR still hold? If yes, why? If no, why?

Just to be clear that with linear regression it is perfectly OK to model nonlinear associations such as $y = 2x + 3x^2 + 17log(x)$ simply by including the relevant nonlinear terms, because it would still be linear in the parameters. I guess you are aware of this, but just wanted to make sure. In those cases, SST=SSE+SSR will hold.
Now, the crux of the matter is that SST=SSE+SSR is actually a special case that only holds when the model is linear in the parameters.  When we are dealing with a nonlinear model such as logistic regression, or any Generalised Linear Model, the situation is quite different because we model the linear predictor's association with the response variable via a link function so that a simple sum of squared deviations does not meaningfully reflect the variability in the response because the variance of an individual response depends on its mean.
There is a very good further explanation of this on Cross Validated, by Ben, and so I will just post it here, for completeness:

The sums-of-squares in linear regression are special cases of the more general deviance values in the generalised linear model.  In the more general model there is a response distribution with mean linked to a linear function of the explanatory variables (with an intercept term).  The three deviance statistics in a GLM are defined as:
$$\begin{matrix}
\text{Null Deviance} \quad \quad \text{ } \text{ } & & \text{ } D_{TOT} = 2(\hat{\ell}_{S} - \hat{\ell}_0), \\[6pt]
\text{Explained Deviance} & & D_{REG} = 2(\hat{\ell}_{p} - \hat{\ell}_0), \\[6pt]
\text{Residual Deviance}^\dagger \text{ } & & \text{ } D_{RES} = 2(\hat{\ell}_{S} - \hat{\ell}_{p}). \\[6pt]
\end{matrix}$$
In these expressions the value $\hat{\ell}_S$ is the maximised log-likelihood under a saturated model (one parameter per data point), $\hat{\ell}_0$ is the maximised log-likelihood under a null model (intercept only), and $\hat{\ell}_{p}$ is the maximised log-likelihood under the model (intercept term and $p$ coefficients).
These deviance statistics play a role analogous to scaled versions of the sums-of-squares in linear regression.  It is easy to see that they satisfy the decomposition $D_{TOT} = D_{REG} + D_{RES}$, which is analogous to the decomposition of the sums-of-squares in linear regression.  In fact, in the case where you have a normal response distribution with a linear link function you get a linear regression model, and the deviance statistics reduce to the following:
$$\begin{equation} \begin{aligned}
D_{TOT} = \frac{1}{\sigma^2} \sum_{i=1}^n (y_i - \bar{y})^2 = \frac{1}{\sigma^2} \cdot SS_{TOT}, \\[6pt]
D_{REG} = \frac{1}{\sigma^2} \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 = \frac{1}{\sigma^2} \cdot SS_{REG}, \\[6pt]
D_{RES} = \frac{1}{\sigma^2} \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \frac{1}{\sigma^2} \cdot SS_{RES}. \\[6pt]
\end{aligned} \end{equation}$$
Now, the coefficient of variation in a linear regression model is a goodness-of-fit statistic that measures the proportion of the total variation in the response that is attributable to the explanatory variables.  A natural extension in the case of a GLM is to form the statistic:
$$R_{GLM}^2 = 1-\frac{D_{RES}}{D_{TOT}} = \frac{D_{REG}}{D_{TOT}}.$$
It is easily seen that this statistic reduces to the coefficient of variation in the special case of linear regression, since the scaling values cancel out.  In the broader context of a GLM the statistic has a natural interpretation that is analogous to its interpretation in linear regression: it gives the proportion of the null deviance that is explained by the explanatory variables in the model.
Now that we have seen how the sums-of-squares in linear regression extend to the deviances in a GLM, we can see that the regular coefficient of variation is inappropriate in the non-linear model, since it is specific to the case of a linear model with a normally distributed error term.  Nevertheless, we can see that although the standard coefficient of variation is inappropriate, it is possible to form an appropriate analogy using the deviance values, with an analogous interpretation.

$^\dagger$ The residual deviance is sometimes just called the deviance.

Now as to the other part of the question:

If the answer to the above question is a no, then a follow-on question I would like to pose is: why is R2 the go-to measure for assessing the regression performance of a model or a technique?

My  answer to this is that $R^2$ is NOT the go-to measure for assessing regression performance. On the contrary, $R^2$ is very poor. I would recommend reading this thread on CV
Is $R^2$ Useful or Dangerous
This has currently 10 answers, many of which are good, but please pay particular attention to the answer by whuber.


plot gridsearch csv results how? how can i plot my results from gridsearch csv?
clf = GridSearchCV(pipeline, parameters, cv=3,return_train_score=True)
clf.fit(x, y)
df = pd.DataFrame(clf.cv_results_)

i'm trying to get a similar plot to what is here: https://matthewbilyeu.com/blog/2019-02-05/validation-curve-plot-from-gridsearchcv-results , but this uses the grid search object and i have tried and failed at trying to get the same using just the gridsearch df (from above).
can anybody help in how i go about this?

Pass your clf object to the function provided in the website. It should work.


Unable to immediately continue Jupyer Notebook after closing then reopening the project When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by rerunning all the code. Is there a better way?

If you

run a jupyter server
open a new notebook with one cell: x=1
close the notebook
re-open the notebook
add another cell under the 1st: x+1

it would output 2, showing that the variable you had set is still in memory in the kernel

But, if you completely shutdown the jupyter server, or if you shutdown the kernel of your notebook, then you'll need to re-run the whole thing.

EDIT: JupyterLab is now in Beta and worth trying out as recommended here


Doing Supervised Learning where output should be a probability that adds to 1 I'm trying to do some supervised learning on a dataset to surface which input is the most probable true input
Example Dataset
As you can see, each INPUT_ID only has one True IS_TRUE_INPUT. The goal is to obtain a probability of each input being the true input for each INPUT_ID.
I'm stuck as this does not seem like a normal classification problem and the probability of each input for each INPUT_ID should add up to 1.
Thank you for your help!

IF YOU HAVE SMALL NUMBER OF INPU_ID
Start with simples approach which is multiclass classification. Take the combination of INPUT_ID and IS_TRUE_INPUT as labels. The rest are your features. Convert categorical features to numericals (e.g. one-hot). Let the model decide.
I don't know why you need the probability distribution of predictions per INPUT_ID. You enter a new input and model gives you a probability of being member of a class. If I did not understand your use-case please comment and I will update.
Example for lines 77 and 78:
line 77 --> input = feature_set, output = {is_i171:1, is_i120:0, ...}
line 78 --> input = feature_set, output = {is_i171:0, is_i120:0, ...}

Depending on the model you use, probabilistic output will be a probability distribution over is_i171, is_i120, etc. e.g. a softmax decision function in NNs.
IF THERE ARE SO MANY INPUT_ID
Embedd all data into a n-dimensional vector space. Calculate the mean vector of TRUE instances for each INPUT_ID. Foe a new query (input), calculate distances from each INPUT_ID mean vector and divide distances by sum of distances. Then you get a probability distribution of a new input being true for any of your input classes.


Which process step in KDD or CRISP-DM includes labeling of the data? KDD and CRISP-DM are both processes to structure your Data Mining procedure. Is data labeling not also a important part of Data Mining?

Data labeling is for example in unsupervised learning the target of the Data Mining process. So if I want to classify a data set that was labelled by me before, do I just do the process twice? In my opinion sometimes the labeling is quite trivial, so that doing the process twice would be quite unnecessary?

Is it possible to include the labeling into the data exploring or preprocessing phase? E.g. in CRISP-DM Preprocessing there is something like generating a new parameter. Can this Parameter be also a new target/label?

I know this question is quite process orientated and in Data Mining you are quite free but just assume that in this case you have to follow the process.

Data Labeling is a very trivial process as you have mentioned.

As far as I know, it falls under Data Understanding(Exploration Analysis). When you don't know anything about data then you do exploratory analysis to understand and derive some insights. If you don't know the target variable then the problem falls under Unsupervised Learning, as you have mentioned in the question that your problem is unsupervised. So, you don't know your Target Variable, you are trying to make new feature/dimension to get some good insight irrespective of the factor whether you derive your Target Variable or any other feature it falls under Data Preparation(any new derived variables), which ever we think are important for our Analysis


How to deal with situation where LSTM fails to learn (constantly makes the same incorrect prediction) I am trying to use LSTM neural networks in order to make a song composer. Basically this is based of a text generator (tries to predict the next character after looking at a sequence of characters) but instead of characters, it tried to predict notes.

Structure of the midi file that serves as the input (Y-axis is the pitch or note value while X-axis is time):

And this is the predicted note values:

I set an epoch of 50, but it seems that the LSTM's loss rate does not decrease, most of the time its loss rate does not improve.

I suspect this is because there is an overwhelming number of a particular note (in this case, note value 65) which makes the LSTM lazy during training phase and predict 65 each and every time.

I feel like this is a common problem among LSTMs and time-series based learning algorithms. How would I solve a problem like this? If what I mentioned is not the problem, then what is the problem and how do I solve that?

Try using a recurrent neural network and use a RProp minus trainer without weight backtracking (aka RProp-, cf. [Igel&Huesken, Neurocomputing 50, 2003]) and without ponderation, ie. all training samples have the same weight.


Machine Learning - Range of Hypothesis space and choiceof Hypothesis function type I am new to machine learning and seek your help in clarifying my elementary doubts. I did a fair amount of googling, but find most literature jumping directly into math.

What I know is that given a labelled training data, a ML algorithm chooses from a hypothesis space H a hypothesis function h.

As an example, assume that a feature vector in training data contains 3 features (x1 through x3)

Now the data from the training set is taken and plugged into a formula (function type). If x is the feature vector and w represents the coefficients of the formula, then output y = A pre-determined f(w,x).

My questions are:

1. Who decides the range of each of the coefficients?
2. Who decides the formula? Is the formula fixed for a ML algorithm?
3. What exactly is the hypothesis space? Is it range of w, or is it different formulas or both?

I acknowledge I have asked more than one question and against the rules, but it was convenient to logically group them in a single post.

I will answer your question as I understand them, the clarity you are looking for comes from the Maths behind ML.

There is no fix range of values of the coefficients or weights. Finding the value of these weights is the work of ML algorithms. although using regularization you can reduce the value of these weights but range depends on algorithm. no. of weights you can decide
Yes, formula is loosely fixed for a ML algorithm.
e.g. for linear regression in your case

h(w,x) = w0*x0 + w1*x1 + w2*x2 +w3*x3 (where x0 =1)

the weights will change after each iteration though. for neural network it is multiplication of a weight matrix with an input matrix for each layer.
The output of h(w,x) is the prediction of ML which you can compare against y.

All the ML algorithms available are your hypotheses space(H)

e.g., quadratic,

linear equation(the one above),

high degree polynomial,

complex neural network.


LSTM Validation MSE always lower than Train MSE I am trying to train a LSTM network to forecast time steps further.
I have a list of queries and the current question is based on one among them.

The validation loss (using mse) is always lower than the Train loss (mse), I know I am under fitting hence, generalizing pretty badly.

So atleast overfit for sometime, what changes can be brought in the network. Below is the code for the same.

learning_rate = 0.001
n_neurons = [150, 80, 60, 40, 25, 10]
dropout = 0.2

def fit_lstm(train, n_lag, n_seq, n_batch, nb_epoch, n_neurons, dropout=dropout, reset_state=False):
    # reshape training into [samples, timesteps, features]...
    X, y = train[:, :n_lag], train[:, n_lag:]
    X = X.reshape(X.shape[0], 1, X.shape[1])
    # design network..
    model = Sequential()
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                 dropout_U=dropout, stateful=True, return_sequences=True))
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                dropout_U=dropout, stateful=True, return_sequences=True))
    model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]),
                 stateful=True, return_sequences=False))
    model.add(Dense(y.shape[1], activation='tanh'))
    tic = time.time()
    model.compile(optimizer=adam, loss='mse') #, metrics=[mean_absolute_percentage_error]
    # fit network..
    loss, val_loss = list(), list()
    for i in range(nb_epoch):
        print('Running Epoch ==> %s' %i)
        history = model.fit(X, y, nb_epoch=1, batch_size=n_batch, validation_split=0.1,
                callbacks=[early_stop,reduce_lr], verbose=2, shuffle=False)
        loss.append(history.history['loss'])
        val_loss.append(history.history['val_loss'])
        model.reset_states() ## clears the state...
    toc = time.time()
    print('====='*10)
    print('Total computation time to train the Model : %0.2f' %((toc - tic) * 100) + ' secs')
    return model, loss, val_loss

Any help would be highly appreciable.

Thanks in advance.

One important thing to start with is to check that your targets are in [-1, 1] range because you have a 'tanh' as output function. You should also analyze the distribution of your targets (True class).

Then, one of the first steps that I would recommend is to try to overfit your model. Make a model 'complex enough' and overfit small proportion of your initial data set (20% maybe?).

If your model does not overfit, there are two possibilities:

Model architecture not suited for this task (maybe lstm) or model not complex enough
Your data is not adapted for prediction task.

If your model overfits (Good sign):

Try to increase the complexity of your model when increasing the size of the samples (training set). Do not use any dropout or regularization and increase the number of layers instead of the number of units/layer.


How to create a model to suggest similar words in realtime? I have a huge database of job titles, I want to build a system where if you enter something like "jav " then it should suggest next some similar job titles like (java developer, java engineer) etc..
How should one approach this problem? How can build something like this, the latency is biggest concern because it has to be real time. We have to integrate this in UI at the end.
Any suggestions how to proceed further?

Autocompletion algorithms usually build on either tree or hash structure for the sake of efficiency. One famous approach is the Ternary Search Tree. Use this source to have an initial idea. It is also a comparably space-efficient solution. You can find more complex systems build on the Ternary Search Tree.
Another good solution is Trie. Trie data structure is a good alternative of Ternary Search Trees, as an example implementation check this.
These both are handy at the front end of the application since they are simple but powerful approaches. Other than that, they have a good balance of time and space complexity.
Moreover, a fast-autocomplete library may help you. Since, in your case, you may have two names for the same job, it has synonym functionality.
In general, tree structures are what you are looking for. You can do this task even using Binary Trees. However, I would suggest you consider the case that in reality many times there appears spelling problems, so in that case, you either will not be able to suggest a word or the suggestion will be incorrect. For such problems, you can also have a spelling check algorithm that first checks if such a word exists, if not using spell check you can have proposed words and you can suggest the autocompletion based on those suggestions. E.g. instead of 'Java' one may type in 'Jsva' and your algorithm will find the correct word and make the suggestion accordingly. This is a one-of-the best algorithms that does auto-completion using ML techniques. You can also build a simple one using Levenshtein distance. (However, spelling error cases mostly discarded in the systems because of its being costly to fix them)


Stochastic Gradient Descent Batching I'm new to regression and we are doing a very simple exercise in a course. I'm taking to get a basic understanding of GD and SGD for a linear regression.
From my understanding, the only difference between GD and SGD is that instead of performing the algorithm on dataset size m as is processing in GD, SGD performs the operation on subsets of m.
My question is, for SGD does one simply perform the algorithm on the mini-batch, or is there some sort of summation of the results to come out with a final answer? Apologies if I'm not asking in the correct terms, I'm newer to some of the mathematical concepts involved.

In SGD you just feed an example to your model, compute the gradient of the loss function of that example and update the weights according to the gradient of the loss of that example.

In mini-batch gradient descent you feed a batch to your model, compute the gradient of the loss of that batch and update the weights according to the gradient of the loss of that batch.

In fact, SGD is mini-batch gradient descent with batch size equal to 1.


Sampling trying to keep as much multivariate variance as possible I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables).

The benefit might be that it will allow development of code around the sample, and really work with the edge cases in the data.

You can then later always take a representative sample.

So, I am wondering if people have tried to sample for maximum variance before and if there is a clever way to sample with as high possible variance (of course an approximation is just fine).

It depends on what you mean by sampling. Is it sampling between or within features?

For between features, scikit-learn has a built-in option for VarianceThreshold which removes features whose variance does not meet some threshold.


What's the difference between data classification and clustering (from a Data point of view) What are the differences and the similarities between data classification (using dedicated distance-based methods) and data clustering (which has certain defined methods such as k-means)
Is data classification a sub-topic of data clustering ?

Classification is a problem where your input data consists of elements with 2 parts:

Some data features that reflect the traits of an entity
A label that assigns the entity to a group or class.

With that kind of data, you can train a model that receives the data features (first part) and generates the label (second part). This kind of training, where you train a system to generate some output when it receives a specific input is called "supervised learning".
On the other hand, in Clustering, your dataset only has the data features, that is, your dataset does not have the labels. Clustering methods allow you to group the entities in classes without having any labels, normally by defining a priori how many groups you want, and then grouping the entities by their similarity. This kind of training, where there are no labels and you have to learn just from the entity data features is called "unsupervised learning"


What are named entity advantages over strings? I am unclear about the practical benefit of named entity recognition. Specifically, I don't get how this entity is any better than a string representation of the same object. Let's take the popular example of movies. I can create a model where each row is a movie. Any attribute you can have in a recommendation system can just as easily be a column in my movie table: release date, actors, genre, etc. If a user does a search for 'animation' - or even if they don't, but you know from that user's history that they saw The Lion King, any decent search engine could pull up Moana as a recommendation, so how is a named entity any better?

Or you pull a named entity out of a text document. Great, but now what? Isn't it still just a string, which can be assigned as an object with it's own unique dict of values, like genre and release date? I assume there are benefits over strings, or it wouldn't be such a big deal to so many people. I just don't know what those benefits are.

The comparison between named entity and string is irrelevant, because the concept of named entity doesn't belong to data types such as string or numeric values. Of course a named entity is usually represented as a string but so is a text document, a cryptographic key or a translation of a text to Vietnamese: what matters here is the semantics, not the technical representation.

The reason why named entities (NEs) are often offered special treatment is because they are particularly relevant for certain tasks and are notoriously difficult to detect and analyze:

The task of Named Entity Recognition (NER) consists in detecting named entities in a raw text document. It's not a trivial task, the problem is still an active area of research.
Even after having been identified, named entities are often difficult to disambiguate. The same entity may appear under many different variants for various reasons:

"John Smith", "J. Smith" and "John A. Smith" may or may not be the same person.
"The president" and "Donald Trump" may or may not be the same entity.
"Big Apple" and "New York" represent the same entity, but not "big apple" and "New York".
...

Of course when one is provided with the result of the NE recognition/disambiguation process, they can process NEs like any other piece of information... but that's because all the hard work about NEs has been done before.


what should i learn for deep learning i have studied the basic of machine learning ,algebra ,statics and probability.now i want start with deep learning for field of image recognition,classification.
i don't know where to start deep learning ,can anyone suggest me where to start and end like syllabus

Have a look at fast.ai (https://www.fast.ai), only heard the best about it. Supposedly,  it's completely free and easy to get into.

Other often mentioned resources are the course by Andrew Ng (https://www.coursera.org/specializations/deep-learning and/or https://www.coursera.org/learn/machine-learning) or the (also free) book by Ian Goodfellow (https://www.deeplearningbook.org).


How to deal with name strings in large data sets for ML? My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later.

Some word embedding techniques were used for longer text sequences preferably, not for single-word strings as in this case. So I think these techniques wouldn't be the way that will work correctly. Additionally Label encoding or Label binarization may not be suitable ways to work with names, beacause of many different values on the on side (Label binarization) and no direct comparison between names on the other side (Label encoding).

Are there other approaches to use or transform especially name information in order to work with ML algorithms?

You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace names by the mean on target variable, however, this is highly prone to overfitting and you should take care.

The following two videos will give an excellent explanation:

https://www.coursera.org/learn/competitive-data-science/lecture/b5Gxv/concept-of-mean-encoding
https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization

However, I would also consider taking out sensitive information such as name depending on your application, always think about if the features makes sense.

I hope this helps, any question let a comment.


Tool for clustering and cleansing data set I have a large-ish data set (400K records) composed of two fields (both strings). I am looking for a tool that will enable me to cluster the data e.g. around the first column, either using exact matches or some kind of string proximity function like Levenshtein distance. I would also like to be able to find all duplicate records and merge them into one.

OpenRefine looks ideal for my purposes but it is so slow when clustering my data or creating a text facet that it is unusable. Apparently this is a known issue.

I looked around but couldn't find another tool that would enable me to explore a data set of this size, cluster, eliminate dupes, look for anomalies, etc. Can anyone recommend something that might fit the bill?

What do you mean to cluster data around a column? You can use standard clustering models, such as k-Means clustering or Gaussian Mixture Models for an exploratory analysis of your dataset. However, I'm not sure this is what you're looking for, please let me know.
I often study a new dataset by employing some dimensionality reduction technique. The most common is PCA, but I don't recommend it since it can extract only latent factors that are linearly associated with your variables. You can use t-SNE models (available in sklearn), or, if you are familiar with Deep Learning, with Autoencoders for dimensionality reduction. Once your dataset is compressed/reduced, you can observe how different values-factors are distributed on it.
Dimensionality reduction is also very important in case your dataset suffers from high multicollinearity.
You can remove duplicates using pandas' drop_duplicates() function, explained here.

Hope this helps, otherwise let me know.


F1 score graph skewed The following code

DBSCAN_score = pd.read_csv("DBSCAN_scores.csv")
sigmoid_score = pd.read_csv("Sigmoid_scores.csv")

score = pd.concat([sigmoid_score.F1_score.fillna(0), DBSCAN_score.F1_score.fillna(0)], axis=1)
score.columns = ["Sigmoid", "DBSCAN"]
ax = plt.gca()
ax.set_title("F1 score comparison")
score.plot(ax=ax)
plt.show()

Normaly I would get the output bellow

Now I am getting the following result with the same code and the same files.
I'm struggling to make it look like the output above

The way I see it is because some reason your x limits is different, you could try:

plt.xlim(0,120)


Returning rows from a list of indexes in Python Pandas Context
I have a CSV containing two types of rows, an observation record and on the row below, an observation value which is related to the observation record above. The record line contains a four-letter code which denotes the type of observation. My goal is to create a new CSV containing only those observation records which match a particular list of codes, along with the associated observation values from the row below.
Example from file
OBSERV\LTRC,CL1,0,10.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,10,20.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,20,30.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,30,40.00;
OBVAL\14,,0.5,V;

Code so far
import pandas as pd
data = pd.read_csv(r"CSVFILEPATH")
df = pd.DataFrame(data)
df.set_index
newdf = df.loc[df[0].str.contains('LLRT|LLTX|LRRT|LTRC|LV10|LV3|LEDR|LTRV|LES2|LES1', regex = True)]
# this returns all observation record rows I care about but I still need the associated observation values.

keep_ind = [] #This list will contain all indexes to keep

observ_ind = ndf.index.values.tolist()#The list of observation record indexes to keep
keep_ind.append(observ_ind)#Added these to the keep_ind list

The Question
How do I take this list of indexes (keep_ind), append a new list which is the same list with 1 added to each item (to get all of the observation value rows beneath the records) and create a new dataframe which contains all of the rows at each of these indexes in this combined list?
So far, I've tried:
keep_ind.append(observ_ind + 1 for i in observ_ind)

But this gives the error:
generator object <genexpr> at 0x0000028057C33648>]

I want to adress two issues. One is how you get your CSV and the other is what probably doesn't work in your code.

how to get the associated values

What you try to do is a nice task for the shift() method

# you probably don't need the semicolons at the end of the line, right?
# if you want to get rid of them, you can do:
df= pd.read_csv(
    r"CSVFILEPATH",   # your file
    engine='python',  # use the python engine instead of C to use a regex as separator
    sep=r'[,;]',      # use ; as an alternative separator
    usecols=range(3), # exclude the last column (after the ;)
    names=range(3))   # assign names, if you like you can also assign a list of more verbose column names here (this just uses numbers)

# create a dataframe that is a version of the original
# which is just one row shifted to the top
df_shifted= df.shift(-1)

# concatenate it with the original data frame and assign unique column names
df_concat=  pd.concat([df, df_shifted], axis='columns')
df_concat.columns= range(6)

# apply your filter
newdf= df.loc[df_concat[0].str.contains('LLRT|LLTX|LRRT|LTRC|LV10|LV3|LEDR|LTRV|LES2|LES1', regex = True)]

This outputs:

In [44]: newdf
Out[44]:
             0    1     2         3    4    5
0  OBSERV\LTRC  CL1   0.0  OBVAL\14  NaN  0.0
2  OBSERV\LTRC  CL1  10.0  OBVAL\14  NaN  0.0
4  OBSERV\LTRC  CL1  20.0  OBVAL\14  NaN  0.0
6  OBSERV\LTRC  CL1  30.0  OBVAL\14  NaN  0.5

From the following test data you provided:

import io
import pandas as pd

raw=\
r"""OBSERV\LTRC,CL1,0,10.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,10,20.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,20,30.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,30,40.00;
OBVAL\14,,0.5,V;"""

df= pd.read_csv(io.StringIO(raw), engine='python', sep=r'[,;]', usecols=range(3), names=range(3))

Code, that probably doesn't do, what you intend to do

# the following line references a method, but doesn't call it:
df.set_index
# if you execute this line, it outputs:
df.set_index
Out[18]:
<bound method DataFrame.set_index of              0    1     2
0  OBSERV\LTRC  CL1   0.0
1     OBVAL\14  NaN   0.0
...
# look at the <bound method part, this is the __repr__ string
# of the object (bound methods are objects themselfes)
# if you execute this line inbetween a script, it has no effect
# at all (just maybe slows down execution a very tiny bit)
# because you don't do anything with the returned object

You wrote, you get the message generator object <genexpr> at 0x0000028057C33648>]. This is not an error method. Like the bound method message above, this is also the __repr__ string of an obect. In this cas a generator object. If you call append on a list, it treats the argument passed in append as one object. I guess you rather wanted to add the indexes increased by one to an existing list. That can be done with the following code:

# create a copy of the list to avoid funny results
new_indices=list(observ_ind)
# add the elements of returned by the generator object
# to the list (rather than the generator object itself
new_indices.extend(i + 1 for i in observ_ind)


How to best accommodate snapshots of data over time into a single dataset for training Say we have customers who acquire or not a product, and we have snapshots of the customer's profile monthly, with the information if at that given month they acquired or not (binary label).
I have two questions:
(1) What should be the best way to handle this dataset; concatenating the information as columns (for say, the last 3 snapshots so that the model can find a trend), with only the result of the last month? Or appending the snapshots as rows, and do some aggregated calculations (for example, averages for the same customer, how many times it acquired the product in the past x months, etc)
(2) If I go with the appending rows route, I guess all the aggregations have to consider only the previous rows based on the time on the snapshot (otherwise, I guess there would be a leakage of information), but how would I manage a cross validation scenario, should I take a specific caution here?

It would depend on the approach you want to take...
Based on the information you gave, I could imagine turning the data into a classification problem, whereby you cluster in the feature space of the various customer profile features. You could train the classifier on a "did buy"/"didn't buy" column.
The descriptive statistics you mentioned could also be valuable. I would recommend doing a mixed method approach and see which yield the most interesting results


How can I appropriately handle cleaning of gender data? I’m a data science student and I’ve begun working with an open mental health dataset. As part of this, I need to clean the data so that I can perform an analysis of it.
In this dataset, the gender field is a string that could have had anything entered into it. While cleaning most entries are fairly straightforward (“f”, “F”, “female”, “cis female” and “woman” can all be coded to “F”), what I was wondering about was how to properly handle trans or queer identities (e.g. an entry that says something like “trans female” or “queer/she/they”).
Should I create a new code for trans entries for each gender, or should I just code them as though they were members of the gender they identify as?
Should I just drop them from the dataset entirely, because they might distort it? I remember reading that trans individuals suffer from much higher rates of mental illness than cis individuals.
Are there any best practices that I should follow in this regard?

There are at least two general considerations to make:

Domain-related

If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit from a direct encoding. For example: if being trans is correlated with different psychological disorders then I'd include a direct feature for this. This way it is easier for your model to make a prediction since it does not need to combine two features in the first place (e.g. no need to combine "sex at birth" and "gender identification" to identify a transsexual person (which would not even accurate since "trans" is a much broader term than just sex at birth != gender identity)).

Moreover, I'd apply the same thinking to other feature engineering questions. Sex has predictive power for many tasks related to mental disorders, e.g. because mood disorders are more common among women and anti-social personality disorders are more common among men. However, whether these are rather related to the sex at birth or the gender a person identifies with is another question. So if your hypothesis is that in your task the gender a person identifies with is important then, again, it makes sense to include this in addition to the sex at birth.

Model-related

Different models are able to handle predictors differently. For example, tree-based models can more easily work with two separate attributes sex == female and trans == True to implicitly derive trans female == True. However, linear models like neural networks might benefit from having a combined binary feature female trans.


Approximation of long sequence of layers by one layer Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of the model.
Layers from i up to j, effectively perform some nonlinear transformation:
$$
\mathbb{R}^{d_i} \rightarrow \mathbb{R}^{d_j}
$$
Where $d_i$ and $d_j$ is the dimensionality of the input at $i_{th}$ and $j_{th}$ layers. Has there been any reseach on approximation of this nonlinear transform by some linear one - search of convolution layers with the activation function, which approximate the action of multiple layers as close as possible.
The loss function for this approximation can be simply an MSE error, where the inputs are the activations of $\mathbb{R}^{d_i} $ and outputs - activations $\mathbb{R}^{d_j}$. I see the possible problem, that mapping from one high dimensional space to another requires a lot od training data for reliable estimate, but, nevertheless I wonder, whether such an approach has been pursued in the literature?

The teacher/student model approach (at least as I understand it) could be used.  It is normally used to replace all layers, but there is nothing stopping you applying it to a subset of layers.
First you create training data, by running your training data through the (trained) network, and recording the inputs to layer i, as your new training data (x) and the outputs from layer j as your desired answer (y).
You now simply (!) want to design a model that takes x and produces y. Ideally using fewer weights than layers i to j currently use.
If those layers were a fully-connected NN, then this is the idea behind autoencoders. If a Transformer then try with one attention + FCNN layer, and see how low the error will go. If it flatlines at an unacceptable error rate, maybe you will need to try two layers. If a CNN, then I am less sure it will work; my hunch would be would have better success keeping the same number of layers and convolution sizes, and just using fewer features in each layer.
Anyway, when you have trained this student model, you can plug it in to your original model to replace layers i to j. (Then maybe run some fine-tuning training.)
Note: if your layers have residual connections, you will have to think carefully about how they handled. Taking the i-1th output and feeding it to the j+1th layer, when it previously expected the output of the jth layer, may go badly. Especially if the dimension of the data is different.


What methods are used to determine acceptable loss levels for autoencoders? I've been looking at autoencoders for disparate uses such as dimension reduction, blurring or sharpening images and data denoising.

What methods are used to determine acceptable loss levels for autoencoders?

Acceptable loss levels depend on the specific domain, dataset, and model.

Loss levels can be compared within model (e.g., training loss by epoch) or across models (e.g., autoencoders with different levels of compression).


How to run list comprehensions on GPU? Is there a way to run complex list comprehensions like the following on GPU?

[[x[index] if x[index]>len(x) else x[index]-1 for x in slice] if (len(slice)==1) else slice for slice,index in zip(slices,indices)]

To what degree is it Possible? Do I have to convert it to some kind of numpy comprehension (if so what part is speciffically possible/necessary)

The goal is performance optimization on large datalists/arrays.

There are several different methods.
One method to use Numba which is a just-in-time complier. Numba can compile Python to target CUDA. CUDA is a API for GPU computation.


Configure BRAT so that if no `.ann` file is found, then an empty `.ann` file is created I am annotating a new corpus with BRAT. I have a set of .txt files to annotate. Is it possible to configure BRAT so that if no .ann file is found, then an empty .ann file is created? Or am I supposed to myself provide the empty .ann files?

I run into the same problem with BRAT from time to time. The way I solved it for a folder with .txt files is to run this Bash one-liner directly in the shell when inside the directory with these files:

for f in *.txt; do touch $(basename $f .txt)".ann"; done

What is does is the following: With a for-loop, it loops over every .txt file, and creates an empty file with touch having the same basename, but a .ann extension instead of a .txt extension.


How to make use of POS tags as useful features for a NaiveBayesClassifier for sentiment analysis? I'm doing sentiment analysis on a twitter dataset (problem link). I have extracted the POS tags from the tweets and created tfidf vectors from the POS tags and used them as a feature (got accuracy of 65%). But I think, we can achieve a lot more with POS tags since they help to distinguish how a word is being used within the scope of a phrase. The model I'm training is MultnomialNB().
The problem I'm trying to solve is to find the sentiments of tweets like positive, negative or neutral.
Structure of datset:
Created pos tags:
I created tfidf vectors from the tweet and gave the inputs to my model:
tfidf_vectorizer1 = TfidfVectorizer(
    max_features=5000, min_df=2, max_df=0.9, ngram_range=(1,2))
train_pos = tfidf_vectorizer1.fit_transform(train_data['pos'])
test_pos = tfidf_vectorizer1.transform(test_data['pos'])

clf = MultinomialNB(alpha=0.1).fit(train_pos, train_labels)
predicted = clf.predict(test_pos)

With the above code I got 65% accuracy. Rather than creating TF-IDF vectors of POS and using them as modal inputs. I'm wondering is there any other way that we can use POS tags to increase the accuracy of the model?

There are so many ways you could go about this. For starters, you could use Conditional Random Fields (CRF). There is a sweet implementation in Python. In which you can set the POS features and more. There is a website from the same source you posted on how to use CRF for your purpose (I have not read it thoroughly). Spacy is another great resource to get all the features that you need fast. Nonetheless, for SOTA you will need some NN implementations.


Neural Network Multiple | Averging predictions I am training multiple neural networks with various parameters. I am trying to average their predictions, but I am not really sure what that means, I am confused about what to average exactly. Here is what I mean: For a single observation in binary classification for example, the final node will give p a value between 0 and 1 (or -1 and 1 if you're using hyperbolic tangent Activation Function), then this p will be rounded to 1 or 0 if it's > 0.5, depending on your decision boundary.

Now, here is what I don't understand, should average p1, p2 and p3 produced by the models before rounding, or I should round the values to True/False responses and then compute the average? and how does that work exactly ?

There are multiple ways you could do. All of them are categorized under Ensemble methods in machine learning.

Voting classifiers: which is the simplest way. You just take votes based on the label from all models and uses the majority label. That means, you should first round up all labels to 0 or 1 and then use the majority.

Weighted voting classifiers: similar to previous one, but some models have higher weights in voting.

For more information look here.


Deep Learning model for classifying skin diseases I have planned to create a deep learning model that classifies skin diseases(around 5 to 7 diseases). Please suggest me a good deep learning model to go with. I am planning to integrate this model as a mobile application. It goes like this. The user uploads a skin disease image and the machine is going to classify it accordingly. I need a good degree of accuracy.

If you are classifying images, you should obviously use CNN for good accuracy.
The dataset you are classifying also plays a great role in determining the hyperparameters, optimizers, loss function.
So you should start with a basic CNN model that you can train with optimizers such as SGD and try tuning the hyperparameters accordingly.
If you are asking about the transfer learning models you can use RezNet or GoogLeNet


How to cluster text-based software requirements I'm beginner in deep learning and I'd like to cluster text-based software requirements by themes (words similarities/frequency of words) using neural networks. Is there any example/tutorial/github code of unsupervised neural network that groups texts based on themes and words similarities?

Thank you very much for your answers!

I recommend using word2vec as feature vector of words and LSTM autoencoder to encode a sentence (or text) . After you get a vector for each sentence (or text), you can cluster your sentences (or texts) using a variety of clustering techniques like k-means or dbscan and represent them using t-sne or u-map. Start from here:
https://blog.myyellowroad.com/unsupervised-sentence-representation-with-deep-learning-104b90079a93


How do I find all the drastic changes in the stock and fetch news for those days of the stock? I am analyzing MindTree company's stocks as an internship project using python pandas and have been given the task to find all the drastic changes by looking at the produced plot for the stock and fetch news for those days of the stock. I am not sure what is considered a drastic change in the stock market. How do I find all the drastic changes in the stock? Also which news API should I use to find the news for those specific days?

I would say that it probably has a great deal to do with what you are planning on doing with the results of your model. For example, if you are looking for news that corresponds to a move of the stock price by more than 3% on that day, you would only want to look at those days. Alternatively, if you are looking for 'abnormally large' movements in the stock price, you might consider taking the mean and standard deviation of your day to day movements and only considering those days that are more than two or three standard deviations from the mean.

In terms of how to get the news for a particular day, here is a python tutorial that may be useful:
https://www.w3resource.com/python-exercises/basic/python-basic-1-exercise-8.php


Why ConLL is not in XML format I am new to NLP, I found a format named ConLL which seems a tab-separated file,
like

ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL

I found a dependency Treebank in this format for Persian. I would like to do some experiment on it. I am familiar with C# and C# has great libraries for XML document files.

I just wonder why such a corpus is not in XML format! I don't say a sophisticated one but they could put each line in a node of XML. Then at least, I would know what is the tag for each element.

However, I can convert it to XML.

One reason for sure is, that you can easialy open it in spreadsheet viewer.


How can I apply PCA to KNN? I want to know the do not want to how to use library

I will denote a $n\times p$ data matrix by $X$, where $n<p$. That is, each row of $X$ is one sample data with $p$ feature variables.

By using singular vector decomposition method, I can decompose $X$ into $A$, $B$, and $C$ such that $X=ABC$, where $A$ is a $n\times n$ matrix satisfying $A^TA=I_{[n\times n]}$, $B$ is a $n\times p$ matrix whose diagonal elements are all non-negative and real but non-diagonal elements are all zeros, and $C$ is a $p\times p$ matrix such that $P^TP = I_{[p\times p]}$.
Without the loss of generality, I assume that $B$ is obtained satisfying that the diagonal elements of $B$ is descending order, i.e., $b_{i,i} > b_{j,j}$ for $i<j$.

The score matrix, $S$, is obtained by $S=XC$, whose dimension is $n\times p$.

In order to reduce the dimension of variables, we do as follows:
Denoting the $i$-th column of $C$ by $c_i$, which is a loading vector with respect to $i$-th principal component.
Hence, deleting $c_{k+1}$ to $c_{p}$, I can obtain $C_k = [c_1~c_2~\cdots~c_k]$.
The dimension-reduced score matrix, $S$, is obtained by $S_k=XC_k$.

In this situation, how can I run KNN?
Actually, I already made my own KNN code, whose input is trainX, trainY, and testX, and whose output is evaluatedY.
I want to compare (i) basic KNN and (ii) dimension-reduction-applied KNN.

I obtained error rate of (i) basic KNN using 5-fold cross validation method.

yhat = myKNN(trainX, trainY, testX);
errRate = myErr(y, yhat);

To obtain error rate of (ii) dimension-reduction-applied KNN, is the following correct?

trainS= myDimensionReduction(trainX, k)
testS= myDimensionReduction(testX, k)
yhat = myKNN(trainS, trainY, testS);
errRate = myErr(y, yhat);

where myDimensionReduction is a function that converts a data matrix $(n\times p)$ into a reduced data matrix $(n\times k)$.

Everything looks good. However, what I doubt about is the myDimensionReduction function. When you apply the PCA on the test data, you have to multiply $X_{test}$ by the matrix $C$ that has been generated from the training data, not by the test data. This is to stress that the test data should not be used in any step of the training process. If this is what the myDimensionReduction function does, that is alright. Otherwise, if this function creates the matrix $C$, then it is not right.

I know you didn't ask for the library, but I think it makes sense in here: The whole philosophy of scikit-learn is to create objects that correspond to algorithms, use the training data with the fit method and apply the transform method to the test data.


Difference between MDS and other manifold learning algorithms From sklearn docs:

  Note that the purpose of the MDS is to find a low-dimensional representation of the data (here 2D) in which the distances respect well the distances in the original high-dimensional space, unlike other manifold-learning algorithms, it does not seeks an isotropic representation of the data in the low-dimensional space.

Can someone elaborate, in layman's terms, what the distinction is?

The images in the link you provide, of the severed sphere and its lower-dimensional representations, go some way towards explaining the difference.

The severed sphere is a set of points in a three-dimensional space, but we want a two-dimensional representation of it. The objective of manifold-learning is (shockingly) to find a manifold: a subset of that three-dimensional space which (a) closely fits all the points that make up the severed sphere, and (b) can be described with a two-dimensional coordinate system.

If you look at some of the other lower-dimensional representations of the severed sphere, it's like they're taking it and flattening it out into a rectangle so it'll fit in two dimensions. It's taking the severed sphere and figuring out a new coordinate system that maps as closely as possible onto all the points that make up the severed sphere.

The MDS lower-dimensional representation, though, is more like a shadow that the severed sphere casts on a wall. Rather than finding a new coordinate system that closely fits the sphere, it's just "forgetting" whichever of the dimensions it thinks it can most afford to lose while maintaining the same distance to and from all the points.

A good analogy would be maps of the earth. A good map of the earth makes a new coordinate system that fits a sphere onto a 2D surface. To do this it has to distort the relative distances between places, but you end up with effective 2D coordinates that relate well to places on the globe.

Instead of doing this, you could just take two photos of the earth from above the north and south pole and glue them back to back. You'd still have a 2D representation of the earth, but it doesn't work so well as a coordinate system.

This isn't to say that MDS is "bad". It's just doing something different. You probably wouldn't use MDS for dimensionality reduction prior to carrying out some sort of statistical procedure, but if you're trying to produce a graphic that gives some idea of how close multidimensional points are to one another, it might be a good choice.


Cross validation for highly imbalanced data with undersampling In my problem, I am dealing with a highly imbalanced data set, say for every positive class there are 10000 negative one. A normal starting method to train a model is to undersample the data. In this procedure, it is very important to train our model on the undersampled data and check the model evaluation on the holdout (from the original data -- without undersampling).

Now the quesion is here. KFold-cross validation actually splits the undersampled train set into K segments and take one of the folds as test set (which is now undersampled test set). I believe for model evaluation we actually need to calculate the metric of interest on non-undersampled test set (right? or I am misunderstaning sth here?). If yes, is it possible to perfrom cross validation as follows?

Split data into K segments.
Take the first Segment as test set, and undersample the rest of Folds (for example K=1 as test and K=2,3,4,5 as train sets)
Fit model on undersampled train data and calcualte the metric of interest on the test set.
Consider the other Fold as test set (this time e.g., K=2) and the rest as train set (K=1,3,4,5). Undersample train set and proceed to step 3.
Continue this procedure for the rest of the folds.

Is this a correct way of cross validation, when we undersample the data?
If yes, is possible to do it with standard libraries?

Edited on 21 of Feb:
Thanks to @Wes, I want to know if the following piece of code is the correct implementation of KFold cross validation on highly imbalanced dataset.

import numpy as np
from statistics import mean, stdev
from collections import Counter

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import KFold
from sklearn.metrics import roc_auc_score

from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import make_pipeline

# initial parameters
RANDOM_STATE = 42
RATIO = 0.033
N_SAMPLES = 1000000
K_FOLD = 5

# Generate the dataset
X, y = make_classification(n_classes=2, class_sep=1,
                           n_features=10, n_redundant=2,
                           weights=[0.9999, 0.0001], n_informative=5,
                           flip_y=0.0, n_samples=N_SAMPLES,
                           random_state=RANDOM_STATE)

print('Number of samples in each class %s' % Counter(y))

rus = RandomUnderSampler(random_state=RANDOM_STATE, ratio = RATIO)
rfc = RandomForestClassifier(random_state=RANDOM_STATE, n_estimators=100)
pipeline = make_pipeline(rus, rfc)

auc_roc = []
kf = KFold(n_splits=K_FOLD)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    y_pred = pipeline.fit(X_train, y_train).predict_proba(X_test)[:,1]

    auc_roc.append(roc_auc_score(y_test, y_pred))

print('ROC_AUC = {} +/- {}'.format(np.round(mean(auc_roc),4),
                                   np.round(stdev(auc_roc),4)))

So that I have obtained the following result ROC_AUC = 0.9374 +/- 0.037.

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and then evaluate on the non-undersampled fold of data for each iteration as you described.


Linear Discriminant - Least Squares Classification Bishop 4.1.3 Pls. refer section 4.1.3 in Pattern Recognition - Bishop: "Least squares for Classification":
In a 2 class Linear Discriminat system, we classified vector $\mathbf{x}$ as  $\mathcal{C}_1$ if y($\bf{x}$)>0, and $\mathcal{C}_2$ otherwise.
Generalizing in section 4.1.3, we define $\mathcal{K}$ linear discriminant equations - one for each class:
$y_{k}(\bf{x}) = \bf{w_k^Tx} + \mathit{w_{k0}} \tag {4.13}$
adding a leading 1 to vector $\bf{x}$ yields $\tilde{\mathbf{x}}$.
And the Linear Discriminant function for $\mathcal{K}$ class is given by:
$\bf y(x) = \widetilde{W}^{T}\tilde{x}$. The author progresses and presents sum of squares Error function as:

$E_D(\widetilde{W}) = \frac{1}{2}Tr\{(\tilde{X}\widetilde{W} - T)^T(\tilde{X}\widetilde{W} - T)\} \tag {4.15}$

My doubts are related to above equation 4.15.
Consider a 3-class system with only one observation, $\bf{x}\in \mathcal{C_2}$, my understanding:

Pls. refer $\bf{Y}$ in upper half of diagram. Will only $val(\mathcal{C_2})$ be positive: $\mathbf{x} \in \mathcal{C_2}$,
$y_{2}(\bf{x})$ > $\it{threshold}(\mathcal{C_2})$.  Is the value,
$val(\mathcal{C_k})$, negative for other classes' Discriminant
functions? If not, could you briefly explain the reason?
The error matrix, $\bf{E}$ is 1x3 matrix. $\bf{E}^{T}E$ will be a 3x3 matrix, with diagonal elements representing squared(Error) for a
class. Does $Tr$ in 4.15 stand for $trace$ - sum of diagonal elements?
If so, why do we ignore off diagonal error values/ why don't they
matter?

P.S.: If my understanding is wrong/ grossly wrong, I'll appreciate if you point out the same.

As Bishop points out throughout that section, least squares is ill-equipped for this problem, so maybe we shouldn't spend too much time understanding it.  On the other hand, clearing up misconceptions here may help elsewhere.

We would like for (i.e., least squares strives for) $\operatorname{val}(\mathcal{C}_2)$ to be close to 1 and the others close to 0.  But the values could very well be negative, positive, greater than 1, for any of the classes (the correct ones or not)!  Our final classification though is dealt with as Section 4.1.2 explains (just after equation 4.9): the class with largest $y$-value wins.
Yes, here $\operatorname{Tr}$ means the trace.  In "least squares," we're minimizing the mean squared error.  It happens that the sum of squared errors can be written nicely as the trace of these matrices, but the incidental entries of the matrix off the diagonal don't concern us here.


Machine learning methods on 1 feature dataset Assuming I have following dataset but much longer. Can I use any machine learning methods having only one feature? Giving Name and predicting Fullname. I'm newbie and think that doesn't make any sense cause it applies just basic statistics computations as mode. But maybe is something better? If some methods can be used, which ones?

Name   | Fullname
------------------
John   | Novak
Adam   | Johnson
Sophia | Loren
Dave   | Miller
Anna   | Smith
Dave   | Smith
...

Can I use any machine learning methods having only one feature?

Yes!

In fact, many NLP classifications tasks are in this format. Given 1 piece of text, classify something. For example:

Given 1 review, classify the sentiment
Given 1 news article, classify the topic
Given 1 chat message, classify the intent

And now you have:

Given 1 name, classify the Fullname

Can a better method be used?

Like you mentioned, you could just find the most common Fullname for a given name and every time you get a name you have a lookup table for the Fullname. However, what will happen when a name you have never seen before appears, how do you classify it? Are you also assuming that you already have the full list of Fullnames?

Assumption: you know all Names and Fullnames

In this case, do as you suggested. Create a dictionary mapping Name-Fullname by finding the most common Fullname for every Name.

Assumption: you know all Fullnames but not all Names

Let say you have the mappings:

Peter -> Johnson
John  -> Smith

Then, there is a name you have never seen before, Pete for example, which does not appear in your mapping table.

You could try two approaches:

The simple way - find which name in the mapping is closest to Pete using some word distance measure, like Levenshtein.
The more robust way - forget the notion of mapping table and use a machine learning model. You will need the following things:

A text vectorizer to transform your text into a numerical vector. I would suggest a character level n-gram TF-IDF.
A classifier. If you use the vectorizer I suggested, then you will need a linear classifier, like an SVM.

If you go to with approach two, when you encounter the name Pete, it will be spit into n-grams (e.g. [pe, et, te, pet, ete]) and vectorized.

Assumption: you don't know all Fullnames and you don't know all Names

This gets more interesting because you could be working with Fullname generation.

It could be used when you move to names from other countries as well.

For example, you already have the mapping:

Peter -> Johnson
John  -> Smith

Then you start dealing with Dutch names and encounter Pieter and Jan.
Then you might want to get the following results where even the Fullnames are different:

Pieter -> Janssen
Jan    -> Smeets

For this, you could use a seq-to-seq Recurrent Neural Network. The architecture can be similar to ones used for neural language translation.

However, all embeddings you create have to be character level. Instead of learning an embedding for every word, you learn for every character. You also feed your network one character at a time. This way, you will be less likely to find "out of vocabulary" tokens (except for when you find character from another alphabet).


how does model.score(X_test,y_test)'s behave? Related to model.score(X_test,y_test):

Why do we calculate score means r2score on X_test and y_test?
Should it be between $(y_{pred}, y_{test})$!

You don't specify the language or library you're using.

Assuming it's sci-kit learn in python then model.score automates the prediction of your data using X_test and compares it with Y_test and by default uses the R-squared metric to so (hence don't need to manually derive y_pred).

If you have derived the predictions anyway (e.g. using model.predict(X_test)) then you can use the metric directly e.g. r2_score(y_true, y_pred).


Deep Neural Network: Output a Magnitude, not merely a Category How does (can?) a deep neural network provide a magnitude estimate, rather than simply guessing what category an input belongs to?

For example, in guessing the weight of a cat, is there is a different way than creating discrete categories of cats (1 kg, 1.01 kg ... 10 kg), having the network guess the category from an image of the particular cat, and then deduce the weight given the category it chose?

You simply need to make your last layer have 1 neuron and no activation function, so that the output of the network is a single number.

Other than that make sure that you select a loss function for regression (e.g. mean absolute error, mean squared error, etc.)


Logistic Regression implementation does not converge I am currently trying to implement logistic regression with iteratively reweightes LS, according to "Pattern Recognition and Machine Learning" by C. Bishop. In a first approach I tried to implement it in C#, where I used Gauss' algorithm to solve eq. 4.99. For a single feature it gave very promising (nearly exact) results, but whenever I tried to run it with more than one feature my system matrix became singular, and the weights did not converge. I first thought that it was my implementation, but when I implemented it in SciLab the results sustained. The SciLab (more concise due to matrix operators) code I used is

phi = [1; 0; 1; 1];
t = [1; 0; 0; 0];
w= [1];

w' * phi(1,:)'

for in=1:100
    y = [];
    R = zeros(size(phi,1));
    R_inv = zeros(size(phi,1));

    for i=1:size(phi,1)
        y(i) = 1/(1+ exp(-(w' * phi(i,:)')));
        R(i,i) = y(i)*(1 - y(i));
        R_inv(i,i) = 1/R(i,i);
    end

    z = phi * w - R_inv*(y - t)
    w = inv(phi'*R*phi)*phi'*R*z
end

With the values for phi (input/features) and t (output/classes), it yields a weight of  -0.6931472, which is pretty much 1/3, which seems fine to me, for there is 1/3 probability of beeing assigned to class 1, if feature 1 is present (please forgive me, if my terms do not comply with ML-language completely, for I am an software developer). If I now added an intercept feature, which would accord to

phi = [1, 1; 1, 0; 1, 1; 1, 1];
w = [1; 1];

my R-matrix becomes singular and the last weights value is

w  =
  - 5.8151677
  1.290D+30

which - to my reading - would mean, that the probability of belonging to class 1 would be close to 1 if feature 1 is present about 3% for the rest. There has got to be any error I made, but I do not get which one. For both implementations yield the same results I suspect that there is some point I've been missing or gotten wrong, but I do not understand which one.

This has been asked, and answered, on CrossValidated. See https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression and https://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what for two related answers (and there are other related questions that you can explore there). That logistic regression can blow up is a known effect in computational statistics.

Also, in situations where exp(-30) gives you roughly the relative accuracy of the double type, you would want to be extremely careful with accumulation of the round-off errors, as 1+exp(-30)=1: summing the likelihood contributions may run into numerical problems, especially when you start computing numerical derivatives and gradients. For a brief introduction into the issue in application to the typical problems in statistical computing and the specific problems it tends to encounter, see http://www.stata.com/meeting/nordic-and-baltic14/abstracts/materials/dk14_gould.pdf.


Is using cross validation on your entire dataset acceptable when dealing with a small sample size Normally my practice includes using k-folds cross validation on a subset of my dataset and keep a final test set.

When dealing with an exceptionally small dataset, is using cross validation on the entire dataset acceptable?

It depends, there can be different reasons to use CV:

if you're using it to evaluate your model only, then it's fine to use the whole training set.
If you're using it to evaluate multiple models and then select the one which performs best (e.g. tuning hyper-parameters), then you shouldn't use the whole training set as you still need to evaluate the result of this tuning step on some unseen data.


Tree-based algorithms and ordinal features For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, but I am not clear for tree-based methods)
For example:
Is there a difference between converting an ordinal feature from [‘Low’,’Medium’,’High’] to [1,2,3] compared to converting it to [1,99, 876]

Edit: I misread your post: the answer is no it shouldn't matter which interval is used. So long as the order is not changed the splits your tree finds on this data will effectively be the same.
See my original answer if you want more context:
In scikit-learn specifically (I cannot speak for other implementations of tree-based models) it does not accept categorical data as input. Meaning the user must convert this to  real numbers using hot encoding, numerical encoding etc...
When fitting on any given tree - when it's iterating through possible splits it is splitting your column using an inequality (ie is this column's value bigger or smaller than this threshold value). Due to this methodology it does matter in which order your data is encoded.
If you hot-encode your data - each column will be looked at independently (a likely split would be "is this bigger or less than 0.5 as thats effectively the only split to be made when the options are either 0 or 1).
If you numerically encode your data it will just pick the best threshold that maximizes your split criterion function (usually gini inequality). The bias you add to your system by manually determining the order or the numerical encoding can affect how well the tree is able to split on this column. If you do some logical encoding whereby the target increases in value as the value of your feature increases, your tree will be able to split on this column more effectively & you will likely see better results.
For this reason I recommend numerically encoding with a TargetEncoder (https://contrib.scikit-learn.org/category_encoders/targetencoder.html). This way the order of your numerical encoder will make sense for the decision tree.


How to assign a new level to many levels of a categorical variable I have a model which has many categorical variables. For each categorical variable there are many levels, like 50~. But not all of them have significant counts. I got these counts using the function value_counts() in Python:

A                 50
B                 38
C                 26
D                 18
E                 10
...
T                 1
X                 1
Z                 1

How can I change the levels with count (say) less than 5 to a new level "others"?

for x in data.class:
    if x.value_counts() <30:
        x = "others"

Say your data is in a DataFrame df, and the column of interest is named column.

treshold = 5 #or whatever value makes sense to you

def set_levels(df):

    for i df.column.unique():
       if len(df.loc[df.column == i]) < treshold:
          df.loc[df.column == i] = 'other'

    return df

new_df = set_levels(df)


Which algorithm is perfect to determine best fit analyst based on multiple factors? Assume I have a team of 200 analyst who work on different IT tickets. I want a better ticket assignment system which considers multiple factors before ticket assignment.

Outstanding Ticket with each analyst - availability
A skill score on the type of ticket
Customer satisfaction Score
Communication score, etc

Now, as soon as a new ticket lands in the queue I'll determine how critical the ticket is, what kind of skill set is required and based on the customer/business area what kind of satisfaction and communication score the ticket is expecting.

Can someone suggest any algorithm which fits for this kind of problem - i.e. mapping the right analyst with right ticket based on multiple variables?

One approach is a set of models which take the ticket as an input and churn out scores for how critical the ticket is, what kind of skill set is required, what kind of satisfaction score will be needed, and the communication skills needed.

Based on these predictions, you could then write a simple rule-based algorithm to determine the optimal analyst to send the ticket to. These rules would be based on the analyst characteristics you mention in the question.

Depending on the form of the ticket input, the models above may require an element of natural language processing (NLP). You may also want to research recommendation systems - it's very likely that something similar has been done before.


Best way to determine the number of features for RFE (recursive feature elimination) I am applying the feature selection method, RFE (recursive feature elimination), from scikit-learn to a dataset. I do not have any pre-determined number of features for RFE and would rather get the number from data itself.

So far, I applied range of number of features, 1 to 10, for training data. For evaluation, I use the F1 from prediction outcome using the features from RFE. For serialization, I plan to use the number of features that provided the best F1.

What other methods may be used to determine the number of features for RFE? Thanks!

I encourage you to look at this method https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html#sklearn.feature_selection.RFECV, it allows you to recursively test all of your features based on the scoring method of your choice including F1


Stata: Trying to merge two data sets with common year observations I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "cpilevel". The other is our main dataset, and we are trying to merge the CPI-year dataset in order to create a new "cpilevel" variable in the main dataset associated with the same years. I am attaching photos of the CPI-specific dataset and a cross-section of the main dataset (there are far too many observations to capture the whole set). Please let me know the easiest way to merge these datasets; we have tried using m:1 codes but can't seem to get the right combination.

::EDIT::

Here are the results of the merge command. It did not accomplish what we need in terms of merging the "cpilvl" variable with the "year" variable in each of the main observations, but it seems to be on the right track?

Have a file open in your current stata session. in your case let's say the file opened is M&D Research Project one.
On the top option bar mentioning 'File', 'Edit' etc. go to 'Data' -> 'Combine Datasets' -> 'Merge two Datasets'
In Type of merges choose 'Many-to-One' since your currently open dataset is 'M&D' having multiple year observations that have to merge with one observation per year in your 'cpi' dataset.
In 'Key Variables' mention the name of the column to match/merge the data with. In your case it's 'year'.
Choose the file 'cpi' under 'Filename of dataset on disk'.

You'll have your resultant data.

NOTE: You may have to make sure the order of the year on both datasets is sorted

Also if you want to merger cpi with M&D and not vice versa, open cpi dataset first and choose many to one relation. The rest of the steps are same


Finding and ranking best semantic matches between two sets of phrases I'm looking for a proper definition for what sort of problem this is, so I can further research it on my own - though I will, for sure, appreciate any specific advice on what are industry standard ways for working on this. While I'm fairly inexperienced with NLP or recommender systems, I've done a fair amount of classical ML before, in case that matters.
The problem - given a "search" query with a list of inputs and a list of expected outputs, retrieve and rank up to N best semantic match for each input. Constraints:

All inputs and outputs are anywhere between a word and a sentence.
Number of outputs >= number of inputs.
All inputs and outputs are unique within themselves, but there can be outputs that are identical to inputs.
Every input is guaranteed to have at least one "good enough" output present.
Every output is guaranteed to be the best match for at most one input.
Labelled data is available, i.e. human-tagged queries. Inputs are sparse, i.e. inputs for queries targeting identical or very similar output sets can be very different.
Data is in English, and I work in Python - if that matters for your suggestions.

An example:
Inputs
(1) Truck
(2) Assortment of lemons, limes, and oranges.
(3) Apples, pears, and oranges.
(4) A tool with broad blade, used for digging.

Outputs
(a) Citrus fruits
(b) Portable telephone that can make and receive calls over a radio frequency.
(c) Fruits traditionally grown in Germany.
(d) Vehicles used for cargo transportation.
(e) Shovel
(f) Motor vehicle used for transportation.

Desired result (numbers are arbitrary, for illustrative purposes)
1 - d (100%), f (75%)
2 - a (95%), c (60%)
3 - c (87%), a (45%)
4 - e (100%)

Similar question suggestions by Stack Exchange are not answering my question, and searching for the answer elsewhere just pits me into endless stream of articles about sentiment analysis of IDMB or Twitter datasets.

I'm experienced primarily in recommender systems, but I've done enough work in NLP to have some ideas on how to approach this problem.
I don't know of any formal name for the problem that you are proposing, but I do know that it's going to be really hard to train a model to learn from just that data, even if you had dense labels. There's just too much unstated human context embedded into the problem to train a model on that from scratch.
Like for your example of Fruits traditionally grown in Germany. You have to find some Knowledge graph or embedding that has learned the relationship of fruits and where they grow geographically. There is a limited set of things out there that would be capable of this.
So what you need to do is apply another model or language embedding to the data, and try to engineer a solution from that.
The first thing I thought of is any large-scale pre-trained skip-gram or CBOW embeddings (these embeddings are often called "word-vectors" or "thought vectors"). A resource to the basics of that here.
The idea here is to use some pre-trained language model and calculate embeddings for your inputs and outputs. Then you just do cosine similarity between the embeddings for each input and output, and see if you can get good matches.
Because you're using sentences you're going either going to have to use a model that embeds sentences or documents (doc2vec is an example), or you're going to have to find some way to aggregate token embeddings. I would pick the former if you try this approach.
But to build on that, once you have these pretrained embeddings, you could also train your own neural net on top of those embeddings to classify for matches. I would read up on the task of question-answer for neural networks for inspiration, as I imagine you might take some ideas from that for how to associate the queries and matches for a neural net (Google "QANet" for some leads on this).
Another approach, which depending on the data you're working with, would be a knowledge graph. This solution would be more complicated, but basically you would break up the documents into different pieces (POS/NER tagging) and then search for semantic equivalents in a knowledge graph. Some example KGs can be found here.
Another thing to google would be the field of "Ontology". Very much related to knowledge graphs but may get you some more niche results that could prove to be helpful.
Some more context about the exact problem you're trying to solve or what kind of dataset you're working with may help illuminate other solutions. Hope that some of the terms I gave you pushes you into the right direction. Good luck!


Market Basket Analysis - Data Modelling Imagine that I've the following dataset:

 Customer_ID    Product_Desc
    1   Jeans
    1   T-Shirt
    1   Food
    2   Jeans
    2   Food
    2   Nightdress
    2   T-Shirt
    2   Hat
    3   Jeans
    3   Food
    4   Food
    4   Water
    5   Water
    5   Food
    5   Beer

I need to make the consumer behaviour and predicte what products are associated. For do that I think that will a good strategy make the relationships first and then count the occurrences (don't know if anyone have a better idea).

The first step is to conclude this relationships:

Jeans-T-Shirt-Food
Jeans-Food-Nightdress-T-Shirt-Hat
Jeans-Food
Food-Water
Water-Food-Beer

How can do this? With Apache PIG or with Spark?

Many thanks!!!

Let's start with your problem definition: "a good strategy make the relationships first and then count the occurrences".

That is, roughly, the basic strategy that market basket analysis algorithms use. However, algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the inherent association rules between items across all baskets. My recommendation would be to use one of these to gather the relationships between items purchased, instead of reinventing them; especially because you'll face a lot of the hard problems these algorithms already solve (namely the large search space when generating combinations of basket items).

You can use any of several libraries or languages to do this, namely R, Python, etc. Doing this in Spark would be pretty simple using MLLib, your workflow would be something like: 1) choose an algorithm, e.g. FPGrowth; 2) prepare your data to fit the format required by FPGrowth (each transaction should be an array of basket items); 3) run FPGrowth and output its frequent itemsets.

There's a good example of this at Spark's website:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

//prepare the data to use with FPGrowth
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

//create and run the model
val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

//output the frequent itemsets (items frequently bought together)
model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}


How do I turn images dataset into a numpy array? I have a directory for a dataset of images, I I want to transorm it to a numpy array in order to be able to fit an image generator to it. What I have tried to do is the following:

trainingset_temp = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

import cv2
import glob

trainingset = []
files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Haze') # your
image path
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Sunny')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Snowy')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

files = glob.glob ('/content/drive/My Drive/Colab Notebooks/Train/Rainy')
for myFile in files:
    image = cv2.imread (myFile)
    trainingset.append (image)

trainingset = np.array(trainingset,dtype='float32')

but it gives me the following error message:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-44-2a679dbff06e> in <module>()
     54     batch_size=batch_size,
     55     class_mode="categorical",
---> 56     shuffle=True
     57 )
     58

1 frames
/usr/local/lib/python3.6/dist-
packages/keras_preprocessing/image/directory_iterator.py in __init__(self,
directory, image_data_generator, target_size, color_mode, classes,
class_mode, batch_size, shuffle, seed, data_format, save_to_dir,
save_prefix, save_format, follow_links, subset, interpolation, dtype)
    104         if not classes:
    105             classes = []
--> 106             for subdir in sorted(os.listdir(directory)):
    107                 if os.path.isdir(os.path.join(directory, subdir)):
    108                     classes.append(subdir)

FileNotFoundError: [Errno 2] No such file or directory: array([],
dtype=float32)

Can someone help me? Thanks in advance.

You encounter a FileNotFoundError, which means that a file can not be found.
I suspect that you forgot the filename extension of your image. Should '/content/drive/My Drive/Colab Notebooks/Train/Haze' not be something like '/content/drive/My Drive/Colab Notebooks/Train/Haze.png' or '/content/drive/My Drive/Colab Notebooks/Train/Haze.jpg'


What is the difference between biclustering and clustering? After reading the wiki page for biclustering (https://en.m.wikipedia.org/wiki/Biclustering), I am really confused on what is the difference between biclustering and clustering?

Any explanation/comparisons would be really appreciated. Many thanks.

Clustering find groups of observations in a p dimensional space (p features).

Biclustering enables you to find clusters in a subset of features.
Then why would you need other features than this subset to perform the clustering?
Because locally you may find dimensions a & b useful to identify a cluster (green cluster in the figure) and somewhere else in the space, dimensions b & c might be useful to identify other clusters (light blue and purple clusters on the figure).

Sources :

https://towardsdatascience.com/subspace-clustering-7b884e8fff73
https://en.wikipedia.org/wiki/Biclustering


Fast AI Lesson 4 - MNIST. Confused about multiplying weights by pixels? I’m on lesson 4 of the Fast AI "Deep Learning for Coders" course, and have been back through the same lesson a few times now but I don’t think I’m quite getting a few things. I want to have an understanding of what’s going on before moving on.
This lesson is on MNIST - and Jeremy is recognising 3s vs 7s. So he has 12000 images (ignoring mini-batches) of about 800 pixels each, and his tensor has a shape of (12000, 800). He randomly generates 800 weights, 1 for each pixel, plus a bias (which I don’t quite understand the purpose of either).
So then for the first image, he multiplies the image’s value by the weight for that pixel. So if the pixel value is 0.9 and the weight is 0.1, 0.9*0.1 = 0.09. Or if the pixel value is 0.5 and the weight is 0.6, 0.5*0.6 = 0.3.
(and then a bias is added, and this is done for every image)
I don’t understand what the purpose or benefit is of multiplying these numbers together. If I wanted a score of how accurate is our weight to the pixel value, I’d do abs(value - weight).
And then adding these up would give me a score of how accurate my weights are.
This is where I’m confused. what am I missing?

The input images contain a total of 784 (28 * 28) pixels, meaning that the model has 784 inputs. If we then use one node in the next layer which is connected to all inputs we also get 784 weights as each pixel is connected to the node in the next layer. See also the following image:

The input layer contains all the different inputs to our model (i.e. all pixels) and as you can see all those inputs are connected to all nodes in the next layer. Each connection/line has a specific weight assigned to it, the number of weights for a fully connected layer is equal to n_input * n_output. In the fast ai course this means that there are 784 weights (28 * 28 * 1), while in the example below there are 12 weights (3 * 4) for the first and second layer and 8 (4 * 2) for the second and third layer.
The reason for the bias already seems explained in the notebook itself, without the bias the function is not flexible enough. If all inputs are equal to zero the output would also be zero, whereas it needs to be a number different from zero. The bias allows for this by shifting the function up and down, see also the b term for a linear regression.


Data science tools for easing the participation of a business into their scoring system I'm a working in a small company. The company sells products on a website and they have a python script that runs everyday to attribute a score to each product based on a set of parameters (google analytics events, similar products popularity, price, etc).
The problem is that the scoring outcome is not satisfying, and requiring developers to edit this script arbitrarily, based on business people assumptions, is time consuming and not a proper way to achieve what the business needs.
The ideal solution in my point of view, would be to provide the business team a way to act on the scoring logic in a GUI, integrate what they could come up with, and display the immediate result of applied modifications so they can iterate until they find the right model.
In your experience, what kind of tool/service/library could easily allow non tech users to input formulas on a set of data, which could be synced and integrated in a python script ?

From a Machine Learning perspective, the answer would be to let the non tech users annotate a sample of products with scores and then train a model which predicts scores for all the products. Then the model could be refined and updated by regularly repeating this process, possibly using active learning in order to select the most useful instances and minimize the number of required manual annotations.
Many variants are possible: users could be asked to just say whether they think a score is too high/low, or compare two products and say which one should have a higher score, etc.


Classifying time series data that overlap I am working with Time-Series Data that has to be classified into two classes (Blue and Red) or at least Classify the data as one class (Red), I'm unable to come up with features that distinctly separate the data.

Please advise as to how do I need to approach this problem.

You could calculate the distance between time series using Dynamic Time Wrapping DTW and then you could cluster them using K-means or so. Here is a python implementation of the DTW or use the dtw package in R.


How to calculate VIF in python 3.6 with scipy 1.0.0? I am trying to import

from statsmodels.stats.outliers_influence import variance_inflation_factor

This is working fine upto Scipy 0.19.

But , with Python 3.6.3 ,it's failing due to unavailability of ss module in Scipy 1.0.0.

~\Anaconda3\lib\site-packages\statsmodels\regression\linear_model.py in <module>()
     41 from scipy.linalg import toeplitz
     42 from scipy import stats
---> 43 from scipy.stats.stats import ss
     44 from scipy import optimize
     45 from scipy.stats import chi2

ImportError: cannot import name 'ss'

Can anyone please guide me how to do VIF analysis using Scipy1.0.0 ?

First, ss module is in scipy.stats not scipy.stats.stats as of scipy 0.17. Also, it is deprecated, but you seem to have figured that out.

Second, in the current linear_model.py on github, the ss package is no longer imported:

from scipy.linalg import toeplitz
from scipy import stats
from scipy import optimize

What version is your statsmodels? Sounds like you may need to update it.


How can I get rid of the white space in matplotlib? I am doing a course in edx. While plotting in python, there is always a blank space before the beginning of the plot. In StackExchange, I have found a solution, which is mentioning the axis limit. However, in maximum cases, the starting points of the plot is unknown. So, Is there any solution to this without manipulating the axis limits manually? A graph is attached herewith for making things more clear.

This stackoverflow answer should solve your issue. You can use either plt.margins(x=0) or ax.margins(x=0) to remove all margins on the x-axis.


How to calculate prediction error in a LSTM keras I have an LSTM which I have constructed and run in keras using python.  I use this model to predict $n$ points into the future for a time series forecasting problem.

When I use a method such as ARIMA to make the forecast I am able to generate prediction errors for my predictions as the model is fit by minimising the MLE using AIC for example.

Is there a way that is currently supported in keras for me to generate prediction errors for my regression predictions?  If there isn't, is there a way that I can calculate it myself?

You may use the technique explained in the article "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning"

Very briefely the technique consists of applying dropout for training and predictions.

In Keras this can be easily applied passing the training argument in the call of the Dropout layer.

import keras

input  = ...
x      = keras.layer.Dense(100)(input)
dp     = keras.layer.Dropout(0.5)(x, training=True)
output = keras.layer.Activation('relu')

model = keral.Model(input, output)

During prediction you get the mean and standard deviations:

T = 1000   # Do 1000 predictions to estimate uncertainty
predictions = np.array([model.predict(X_test)] for _ in range(T)])

pred_mean = results.mean(axis=0)
pre_std   = results.std(axis=0)

You can increase (or descrease) T if you need more (or less) precision.


Where to get models with weights instead of only weights? What's the purpose of .h5 files? I have downloaded .h5 files from qubvel/resnet and qubvel/efficientnet. I was trying to use some models as a backbone for my model but I'm getting the following error:
ValueError: No model found in the config file.

As explained here this is because the .h5 file contains only weights, not a model.
So those .h5 files are only weights. What's the purpose of having only weights without architecture?
I was trying to do following code:
resnet18_path_to_file = "models/resnet18.h5"
resnet18 = tf.keras.models.load_model(resnet18_path_to_file)
resnet18.compile()

inputs = resnet18.input
outputs = resnet18.layers[-2].output

return tf.keras.models.Model(inputs=inputs, outputs=outputs, name="custom_resnet18")

There are several options when saving and loading a keras model, as explained at https://www.tensorflow.org/guide/keras/save_and_serialize:

save the whole configuration, including the architecture, weights and even the last training state
but also the model architecture and the weights can be saved as independent files, and that is what you might have loaded: an .h5 with only the weights, and you also need the .json with the model architecture, based on the snippet below:

-model architecture and model weights to disk as separate files, which is what it might be done by the user who trained the model:
json_config = model.to_json()
with open('model_config.json', 'w') as json_file:
    json_file.write(json_config)
# weights saving to disk
model.save_weights('path_to_my_weights.h5')

So you need to do the following:
-load the model as follows (you need the model json also):
with open('model_config.json') as json_file:
    json_config = json_file.read()
new_model = keras.models.model_from_json(json_config)
new_model.load_weights('path_to_my_weights.h5')


How to use survival analysis for predictive maintenance for time series data? So, I have a dataset with daily operating conditions for different machines and a flag saying if it failed or not.
Here is a snapshot of the data.

How can I use survival analysis or any other algorithm to calculate when the machine is expected to fail in the future?
What I understand is that I can use survival package in R, but I am not able to use it for a time series data.

Whenever your task includes something like "...when XY will fail...", i'd say go for survival analysis first, it is easy and fast and it will give you overview of your data.

With your data you can either turn them into intervals to be able to plot survival curves, or proceed directly to Cox regression, which can work with continuous data and will yield the hazard ratio.

You can start with Kaplan-Meier curve (as bonus there are Confidence intervals):

km <- survfit(Surv(datetime, Failed) ~ 1,conf.int=0.90, conf.type="log-log", data=Dataset)
summary(km)
plot(km, xlab="month", ylab="estimated S(t)", main="Kaplan-Meier with log-log, C.I.=90%")

The curve will look somewhat like this:

Further you can split the curve to see if any of your parameters have different influence. You can do that by simply replace the ~1 with something like ~AttributeX

So you should get this kind of plot:

Of course R will give you various tests and p-values as well, such as Log-Rank to verify whether the influence is significant (survdiff).

Then you can proceed to Cox Regression, which will tell you what is the hazard ratio (=whether the attribute influences the hazard positively or negatively and to what extent). It looks like this in R:

cox<-coxph(Surv(datetime, Failed)~AttributeX, data=pbc)
summary(cox)

It is good practice to verify the assumptions - proportional hazard and functional form (again R will give you p-values or you can plot the residuals - Martingale or Schoenfeld).

If you are interested to know WHEN the event will occur, search for Accelerated Failure Time models, which will give you the parametric survival time distribution, where you can simply put the time and obtain the probability.

In R:

wei<-survreg(Surv(datetime, Failed)~ AttributeA + AttributeB + AttributeC,data=Dataset)

There are more possible distribution, you can check which one fits the best your data. I have never done the prediction, but there is a function predict which is described in documentation or there are already similar questions with answers on Crossvalidated, such as this.