Macro averaged in binary classification Is macro averaged the precision, recall and F1 score recommended for binary classification data? The calculation of precision is calculated for both class and averaging it makes it seems better. Or macro averaged is works better for multiclass classification only?

It's of course technically possible to calculate macro (or micro) average performance with only two classes, but there's no need for it. Normally one specifies which of the two classes is the positive one (usually the minority class), and then regular precision, recall and F-score can be used.
Commonly there is a majority and a minority class, and naturally the majority class is easier to predict for the classifier. That's why the minority class is usually chosen as the positive class: by choosing the most difficult class to predict, the performance value represents more precisely the real ability of the classifier. As a consequence, the macro-average performance is often better than the performance on the positive class, since the former includes the "easy" class.


Calculate likelihood of a date being worked based on location, hours and rate of pay I am just getting into trying some machine learning. As a C# developer I have played around with ML.NET. I'd now like try and see if it can  help with a real world problem.

We have a system that invites staff members to work shifts in multiple hospitals, if a staff member is interested they contact our admin team and are then assigned to the shift.

Many shifts are difficult to assign staff to due to the date/day, location, the hours and of course the pay available. We therefore often have to increase the rate paid per hour. When the rate is increased there is a much better chance the shift will be filled.

My shift class is displayed below.

    // SHIFT CLASS
    int shiftID
    int locationID
    int shiftGradeID

    DateTime shiftStartDate
    DateTime shiftEndDate

    decimal hourlyRate
    decimal hours

    int assignedLocumID

Using the data above I'd appreciate any advice or insight to help work out;

Based on location, date (day, month), start time, end time, grade, hourly rate and hours, how can I work out the percentage chance of a shift being filled.

Ideally I'd like to try something out using ML.NET, however I'm not sure where to start. I have 5 years of data to train the model. Appreciate any help you can provide.

If you're looking for code, I'm not familiar with C#. My answer will focus on theory.

tl;dr most machine learning-related packages have a built-in logistic regression function of some sort. That's what I'm recommending here.

More detail:

I would start with a basic model and work my way up. It sounds like this is something you can figure out using a regression model.

Based on your question: "Based on location, date (day, month), start time, end time, grade, hourly rate and hours, how can I work out the percentage chance of a shift being filled?", I understand you want:

output: probability of shift getting filled
input: location, date, start time, end time, grade, hourly rate, hours (these are $x$'s below)

If you're familiar with logistic regression:

$$log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k$$

the output of this model is the log odds of a single shift being filled (versus not filled). Note this can be rewritten as:

$$p = \frac{e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k}}{1-e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k}}$$

Where $p$ is the probability that the shift gets filled.

I would start by fitting one logistic regression model for each shift.

Be careful: multinomial regression would give you the probability of 1 shift out of k possible shifts being filled, but I don't think that's what you're looking for.

After seeing how well that worked (comparing the model predicted results to the actual percentages in the data), if necessary I would build a conditional model to take into account the probability of filling a shift, given other shifts have already been filled. Not sure how I'd do that yet, but hopefully this gives you a starting point!

EDIT based on the answer from Vikram Murthy:

I realize I forgot to mention: your response variable in this case would be 0 or 1 indicating whether a shift was filled or not. So for each shift, you would have a column indicating whether that shift was filled. That's the column being predicted. This is similar to using "dummy variables".

For example, if you have two shifts, your columns would be:

loc, day, month, starttime, endtime, grade, hourlyrate, hours, shift1_f, shift2_f

So your data might look something like this:

loc, day, month, starttime, endtime, grade, hourlyrate, hours, shift1_f, shift2_f
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 1, 0
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 0, 1
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 1, 1
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 0, 0

In this setup, this would indicate that shift 1 was filled in the first case, shift 2 was filled in the second case, both shifts in the third case, and no shifts in the last case.

Your two logistic regression models would be set up like this:

shift1_f = loc + day + month + starttime + endtime + grade + hourlyrate + hours

and

shift2_f = loc + day + month + starttime + endtime + grade + hourlyrate + hours

The proportions would be figured out automatically by the program; it's not necessary to figure them out yourself and enter them in any application of logistic regression that I've seen.


Synthetic Gradients - what's the practical benefit? I can see two motives to use Synthetic Gradients in RNN:

To speed up training, by imediately correcting each layer with predicted gradient
To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.
I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Point 1:
Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.
However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.
Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer
shouldn't improve training speed.

Point 2:
Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.
This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A
----> stop and bkprop!
fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B
----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)".
Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.
But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last
timestep of fwdprop allows to effectively train longer sequences, but
vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.
Question:
Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

-That's not necessarily correct. We typically truncate and start a new backprop because of hardware constraints, such as memory or computational speed. Vanishing-gradient can be improved by other means, such as Gradient Normalization - scaling-up the gradient vector if it gets too small beyond certain layers, or scaling-down if it's about to explode. Or even using Batch Normalization

It's important to understand how to update any DNI module. To clear things up, consider an example of network with several layers and 3 DNI modules:

 input
   |
   V
Layer_0 & DNI_0
Layer_1
Layer_2
Layer_3 & DNI_3
Layer_4
Layer_5 & DNI_5
Layer_6
Layer_7
   |
   V
output

  The DNI_0 is always trained with a synthetic gradient arriving from DNI_3 (flowing through Layer_2, and Layer_1 of course), siting several layers further.

  Likewise, the DNI_3 is always trained with a synthetic gradient arriving from DNI_5

  DNI_0 or DNI_3 will never get to see the true gradient, because true grad is only delivered to DNI_5, and not earlier.

  For anyone still struggling to understand them, read this awesome blogpost, part 3

Earlier layers will have to be content with synthetic gradients, because they or their DNI will never witness the "true gradient".

Regarding training in parallel with minibatches instead of Parallelizing via Synthetic Grads:

Longer sequences are more precise than minibatches, however minibatches add a regulization effect. But, given some technique to prevent gradient from exploding or vanishing, training longer sequences can provide a lot better insight into context of the problem. That's because network infers output after considering a longer sequence of input, so the outcome is more rational.

For the comparison of benefits granted by SG refer to the diagrams page 6 of the Paper, mainly being able to solve longer sequences, which I feel is most beneficial (we can already parallelize via Minibatches anyway, and thus SG shouldn't speed up the process when performed on the same machine - even if we indeed only propagate up to the next DNI).

However, the more DNI modules we have, the noisier the signal should
 be. So it might be worth-while to train the layers and DNI all by the legacy backprop, and only after some epochs have elapsed we start using the DNI-bootstrapping discussed above.

That way, the earliest DNI will acquire at least some sense of what to expect at the start of the training.  That's because the following DNIs are themselves unsure of what the true gradient actually looks like, when the training begins, so initially, they will be advising "garbage" gradient to anyone sitting earlier than them.

Don't forget that authors also experimented with predicting the actual  inputs for every layer.

If your layers have expensive backprop (perhaps you have Batch-Normalization or some fancy activation functions), correcting with DNI might be a lot cheaper, once it's sufficiently well-trained.  Just remember that DNI is not free - it requires matrix multiplication and probably won't provide much speed-up on a simple dense layer.

Minibatches gives us speedup (via parallelization) and also give us the  regularization.
Synthetic Gradients allow us to infer better by working with longer sequences and (potentially) less-expensive gradient. All together this is a very powerful system.


StandardScaler's mean and standard deviation for real-life data? I've heard that we should use train dataset's scale for that of test data so they are in line with each other in terms of scale.
And I know we use transform() function for the test data preprocessing.
But I am wondering how I can apply the same logic to the real-life data set after we successfully save and load a model since I don't think we can use transform() function anymore.
From what I understand, there must be a way to "extract" mean and standard deviation from the train data set and to apply it for real-life model when using the model we saved already, but how can we do that?

It has to be saved somewhere i.e. Database after the training is done.
The saved values should be used on new data and all these steps should work in a loop i.e. when you re-train again the values will be updated and saved again.
e.g. If we see the Keras pre-trained models, it provides the necessary pre-processing function. We can directly use that
from keras.applications.resnet50 import preprocess_input
train_datagen = ImageDataGenerator( preprocessing_function=preprocess_input)


Difference between IOB and IOB2 format? I have to tag a dataset for NER. I came across conll2002/esp. What I understand so far, in IOB2 format if I want to tag 'Alex Larson is going to Los Angeles for a job interview with Candace Patrick' it'll be like:

Alex B-PER
Larson I-PER
is O
going O
to O
Los B-LOC
Angeles I-LOC
for O
a O
job O
interview O
with O
Candace B-PER
Patrick I-PER

Am I right? What about IOB format?

The difference is not related to the length of the named entities. Rather, it deals with how two adjacent named entities of the same type are labeled.

In IOB1 (IOB), B- is only used to separate two adjacent entities of the same type:

Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O  # or I-PER if pronominals are being tagged
ate      O
lasagna  O

In IOB2, all entities begin with B-:

Today    O
Alice    B-PER
Bob      B-PER
and      O
I        O  # or B-PER if pronominals are being tagged
ate      O
lasagna  O

See Wikipedia


How to implement a GridSearchCV custom scorer that is dependent on a training feature? I would like to code a custom scoring function using the make_scorer function, where my custom_function(y_true, y_pred)calculates the DAILY sumproduct of y_true and y_pred and outputs, say the mean, for example. The problem is that the timestamps are made available in my X matrix as a feature and I cannot access index of current folder.
Does anyone have a clue how to make this possible?
Thanks

You should be able to do this, but without make_scorer.
The "scoring objects" for use in hyperparameter searches in sklearn, as those produced by make_scorer, have signature (estimator, X, y).  Compare with metrics/scores/losses, such as those used as input to make_scorer, which have signature (y_true, y_pred).
So the solution is just to define your own "scoring object" directly, and reference the passed X to do the computation you want.  See The User Guide for some more details.


Can you reuse observations from your train data in your final test data? For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.

Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.

I am using XGboost implementation.  Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.

I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :

df2 <- df1[sample(nrow(df1)),]

In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical.

Is it because, a large portion of my final test data already learned from the observations also found in my training data?

If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?

The Out of Bag(OOB) error in XG Boost is considered as a Validation error(test error). During Hyper-parameter selection in XG boost, OOB error is measured for different hyperparameters. In XG Boost algorithm, the decision trees are constructed by a random sample of observations and the successive decision trees are built by giving more importance to the observation which the previous trees are misclassified. As this XG boost keeps increasing the number of decisions trees, the validation error decreases and saturates at some point.

In your case, you can consider this OOB error as test error instead of creating a separate test set.

The following plot explains this more accurately


Search for hyperparameters whith different features using Random Forest I have a dataset in which I would like to perform a classification model, so I have decided to use Random Forest. The number of features that I have is approximately 200 and I would like to test which set of features gives me a better result. For the sake of experimentation, I do not want to use any feature importance method.

So what I have done is to split my features, without replacement, in groups of 20, 40,60,80 and 100 features. I am using GridSearch for tuning the hyperparameters of my RF. The question that I have is if I want to compare the accuracy of each one of the models (the one with 20 features, with 40, and so on), it would be fair to apply GridSearch with each subset of the features. Or should I only perform Gridsearch once, let's say with the model of 20 features, and then use the same hyperparameters with the set of 40 features, 60 features and so on.

Any help?

Thanks

By performing GridSearch I understand you want to say searching for the best hyperparameters.

For sake of simplicity, let say that you want to fit a linear regression with a penalty (lasso/ridge) with 1 feature and with 100 features. The hyperparameter that you are looking for is the $\lambda$ penalty.

It is easy to see that with 1 feature your model might need some parameter, it could be that the feature has a perfect distribution and it doesn´t need any penalty at all. But when we go for 100 features there is some noise and you might need penalties to make sure that your model is generalizing well. So $\lambda$ will be different

With this example, my idea is to show that hyper-parametrization is a specific problem for each task.

For Random Forest, it would be the same. Different features will require different parameters so yes, you have to do a GridSearchCV with every subset of features if you want to achieve optimality


How can I implement tangent distance for k-nearest neighbor in python/scikit-learn? My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter.

Existing packages

I have been researching existing libraries for a while.

The only thing I found was this KMeans package, for python 2 and based on implementing a C library.  I could neither load it in with ctypes nor convert it into an executable with gcc.

I also found this other C code and this Matlab script but with similar results.

Implementation

I also looked into a few papers, to see if I can implement it by myself. For instance, based on this I understand that the main thing I need to do is to calculate the tangent matrix. But, I do not understand

how do I define $s(p, \alpha)$ and especially
how do I calculate the derivatives in python.

I would be glad for any help, comment, whatsoever.

Update

As suggested, I raised the following related issues/requests:

ComeOnGetMe
Scikit-learn

Update 2

@ComeOnGetMe rewrote his code so it can be used along the scikit-learn specifications (example code). Many thanks for that! Nonetheless, when I tried to use it in scikit-learn it underperformed and was very slow, so there is further work needed with that.

Since then I also found a more detailed explanation for code implementation, although based on the C code already mentioned.

ComOnGetMe's KMeans tangent distance metric looks good, if it didn't work for you initially you should fork it and work on the code. I would contact him directly, he probably would have insight for you. Scikit-learn doesn't have a distance metric for tangent distances but the documentation states you can call a user-defined distance (all be it with overhead).

Contributing code might be a good direction to go if you would want to contact scikit-learn's developers and there are a couple of researchers with published papers implementing tangent distance as a metric, so you could have luck there too.


Transfer learning - small database I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:

1.0  0
0.4 0.6

Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:

0.8  0.2
0.6 0.4

is something like this:

1.0  0
0.8 0.2

Below you can see training + test loss/accuracy.

Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?

Seeing your graphs it does not show that the model is not learning, as the training accuracy is high. If the model wasn't learning anything even this would have been low. Also, you are right that the model is overfitting because it is performing very well on the training set and poorly on the test set.

The overfitting scenario is also confirmed by the fact that you have a very small dataset. You might want to fine-tune all the layers of the resnet as shown here. Also, try early stopping and dropout to prevent the model from overfitting.


simply use .rolling(365, min_periods=1) no need of for loop
data['52wkLow'] = data['Low']['GME'].rolling(365,min_periods=1).min()


semi-structured text parsing using machine learning I am looking for a method to parse semi-structured textual data, i.e. data poorly formatted but usually having a visual structure of a matrix which may vary a lot in content and number of items in it, which may have headers or not, which may be interpreted sometimes column-wise or row-wise, and so on.

I have read about the WHISK information extraction paper : https://homes.cs.washington.edu/~soderlan/soderland_ml99.pdf

but unfortunately, it is not very detailed and I have not been able to find a real-system implementing it, or even snippets of code.

Has anybody have an idea where I can find such help? Or suggest an alternative approach which may be suited to my problem?

Thank you in advance for your reply!

Without a sample of your data, it's unclear what's the structure of your data and what tool is suitable to process it.

Here are some blind recommendations based on my experience:

If you just need some flexibilty parsing the text record, such as variable repeat number of certain field, or conditional parsing of fields, then you should check out this python library: http://construct.readthedocs.org/en/latest/
it allows you to first define a hirachcal structure of your data, and then apply this structure to parse information from a text file. It's especial useful when parsing binary files.
If you're looking for an algorithm that can actually "understand" your text data and "infer" the structure in a smart way. Then you might want to try graph based approach:
http://kavita-ganesan.com/opinosis


When should the last action be included in the state in reinforcement learning? I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is not completely clear as different agents/environments combinations might have different performances if action was included/excluded from input states (I might be wrong).

For my specific problem:

the agent can't influence/control the states through its actions (similar to the case of a simple multi-armed bandit)
the action space is discrete
I am using a DQN based approach

I would also appreciate a general overview/rules of thumb of when to include/exclude actions as state inputs.

ps. when i say "different agents/environments combinations" in the beginning I mean using different agents to solve the same env or same agent to solve different env.

If your environment fulfills the Markov property, there is no reason to include the actions $a$ in the state $s$, as the action $a_t$ that lead to the new state $s_{t+1}$ should not provide any additional information, i.e. the knowledge of the old action should not influence the reward and transition functions from $s_{t+1}$ onwards.

Therefore, if your environment is a proper MDP, there is no reason to include actions in the next state.

It might be helpful in some rare cases, where your environment is not actually Markov to include the last action in your state, but in that case you should also think about if there is a better way to represent your state, such that it becomes Markov.

In your concrete case, where the actions do not change the state, there is no reason to include the last action as part of your state.

Side Note:
If your problem is a simple discrete bandit, you will probably be better off by looking into bandit approaches, rather than using a DQN based approach.


How to checkpoint by minibatch in Keras I know that I can use ModelCheckpoint in Keras for checkpointing a model every epoch (or every few epochs, depending on what I want).

I am getting my data for each minibatch from a fit_generator, and it takes a very long time to evaluate each minibatch. I'd like to be able to checkpoint by minibatch instead of by epoch. How can I do this in Keras?

You have to write a custom callback for this. Steps are :

Subclass ModelCheckpoint   (https://github.com/keras-team/keras/blob/master/keras/callbacks.py) or create new one if you do not need filename pattern etc.
Add method that would be called at the end of each batch

class BatchModelCheckpoint(keras.callbacks.Callback):
     def on_batch_end(self, batch, logs=None):
        self.model.save(filepath, overwrite=True)


Choosing the correct learning algorithm I am kind of new to the data mining subject but i need help to choose a learning algorithm for my application:
The problem: identifying that a certain curve or data set belongs to a certain fault in a Component.

My training data should be like this:

Motor Current Values:
[0.5,0.6,...,0.4] -> Fault in Komponent 1

[0.2,0.3,...,0.4] -> Fault in Komponent 3

[1,0.7,...,0.4] -> Fault in Komponent 2

.
.
.
[0.8,0.7,...,0.3] -> Fault in Komponent 3

And i was wondering if i should using a cluster analysis (k-means and save centroid centers then compute the distance of each new entry then give a fuzzy estimation to where which cluster my data might belong to).

Decision Tree algorithm(entropy of the values and stuff), Or should i calculate a distance between the Nominal data (Healthy(No fault) Curve) and the faulty data and play on a more simple basic decision tree with thresholds ?

Proceed with a peak analysis and count the number of peaks within my data and add it to my learning algorithm.

And when should i do a data pre-processing like normalization ?

This is an example of my entry parameters : [Speed(mm/s), Motor Current Values(Ampere), DistanceToNominalState(Ampere)]

Here is what my data looks like.

Any suggestions ?

It seems you have a data set for one component where the component suffered a fixed number of failure modes. You want to find out which data (let's assume continuous in time, so, what time) correspond to what failure mode. In other words, you are doing "pattern recognition" in your failure data.

Have you thought of using Self-Organizing Maps (SOM)? They are a sub-branch of artificial neural networks and have great capability in such problems.

You should also consider that not all failure modes appear in shape of a "peak" value. So, only looking at peaks is not a very smart way. It most probably will cover most of the failures, though, there will be moments that you miss. SOM could take care of this too.

Data pre-processing is done before the analysis. Be careful in normalization. You could miss the peaks or valley points easily if you don't pay enough attention in normalization. Don't just use any code or normalization method you find online. Test and check it with you data. For instance, some normalization could make all negative values positive which a negative value may have an important meaning in your work.

I assume you have another variable called "failure". Another approach I suggest is building a Neural Network (NN) model, which is very common. You have three input data that you mentioned, consider the "failure" variable as your target variable. Build the neural network and apply it to your data again. If the numbers of failures are little, the NN will be able to rebuild the normal behavior of your data (NN here is called a normal behavior model). When you apply it to your input data, the NN model will detect any deviation which is not expected.

MATLAB has a very good support for both of these approaches.


What type of classification algorithm for recognizing events in a set of data? I would like to know what type of machine learning to use for recognizing events in a set of data. (or at least be pointed in the right direction so I can do my own research on the topic.)

I'm trying to identify M&M color using an RGB sensor. Here is an example of some data I collected. I want to train a neural network by saying "this is what it looks like when a blue M&M passes by the sensor... And a yellow..." So it then can tell me when and what color M&M has passed by.

My type of problem is similar to this. In this case, they record accelerometer data and use it to identify the exercise someone was doing.

How can you teach an ANN to do this? And how do you teach it what an "event" looks like?

** P.S. I'm new to this forum so any modifications to my question to make it more relevant and understandable are welcome.

You will need your data to look something like this:

Blue Red Green Label
20   12  13    _
18   11  13    _
18   12  13    _
19   13  14    _
24  12   13    _
28  14   19    B
30  19   21    B
29  18   20    B
25  14   16    B
21  12   13    _
19  11   12    _
18  11   12    _

That is, every point in the data will need to be annotated with a label: _ for nothing, B for blue, Y for yellow.

A sequence labeling model can be trained to predict the label for each point in the input sequence. Note that it's common to use a "Begin Inside Outside" (BIO) scheme to help the model understand when an "event" starts and ends:

Blue Red Green Label
20   12  13    Outside
18   11  13    Outside
18   12  13    Outside
19   13  14    Outside
24  12   13    Outside
28  14   19    B_Begin
30  19   21    B_Inside
29  18   20    B_Inside
25  14   16    B_Inside
21  12   13    Outside
19  11   12    Outside
18  11   12    Outside

(there are also variants of this scheme)

As far as I know Conditional Random Fields would be the traditional approach for this kind of task, but there might more recent NN approaches that I'm not aware of.


feature scaling xgbRegressor I read for example in this answer:
Does the performance of GBM methods profit from feature scaling?
that scaling doesn´t affect the performance of any tree-based method, not for lightgbm,xgboost,catboost or even decision tree.
When i do feature scaling and compare the rmse of a xgboost model without and with minmax scaling, i got a better rmse value with feature scaling. Here is the code:
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error as MSE
import math
from math import sqrt
import pandas as pd
import numpy as np
from xgboost import XGBRegressor
import xgboost as xgb

data = pd.read_excel(r'C:...path.xlsx')
X = data.drop(['colA'], axis=1)
y = data['colA']

scaler = MinMaxScaler()
scaler.fit(X)
minmax_scaled_X = scaler.transform(X)
minmax_scaled_X
y = np.array(y).reshape(-1, 1)
scaler.fit(y)
minmax_scaled_y = scaler.transform(y)

from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(minmax_scaled_X, minmax_scaled_y, test_size =0.3, random_state=0, shuffle=True)

xg_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.7, learning_rate = 0.05,
                max_depth = 8, min_child_weight = 4, n_estimators = 600, subsample = 0.7)

xg_reg.fit(xtrain,ytrain)
preds = xg_reg.predict(xtest)
rmse = sqrt(MSE(ytest, preds))
print(rmse)

the result with min max scaling is 0.003, while the rmse without is about 3.8. I did the same with simple decision tree and got always a better result with minmax scaling.
Where is my mistake?
In other posts like the link above, answers are about that it is not good to scale when using trees.
Can I say, that min max scaling does have a positive effect on the rmse on my data?

You're also scaling $y$, then of course you are getting lower error. That question was regarding scaling $X$.
The same model will have very different error metrics when units on $y$ are changed: if I multiply all $y$ values by 100, the error will be 100 times larger, if I divide all $y$ values by 100 the error will be divided by 100.


Which approach is better? In the book Deep Learning and Neural Networks by Michael Nielsen, he used a naive approach to write the code which is a bigger considered to a code written in tensorflow.

I want to know which code is better and how it is better ?

In that book the author encourages the reader to write a neural network library from scratch, to gain an understanding of the principles of deep learning. Tensorflow is a mature deep learning library that can be used in production, and can make use of high performance computing resources like GPUs and TPUs. The code in the book is better for learning and Tensorflow is better for training models for real world applications.


Football match prediction using regression I am trying to predict goal difference of football matches in keras using a single layer Neural Network. I used mse as metrics and its a low value aroung 0.05 but some predictions has huge difference.

I tried sklearn LinearRegression too but results were almost same. Also I have tried multiple layers network too but results were almost same too.

Any ideas to lower the predicted difference?

If your MSE is low, but then your predictions on the test set are way off, i can think only of over-fitting either you're leaking information from the training set to the test set ( or vice-versa ) or you simply have a complex model ( too many features ) which you could solve by increasing your data or going for less features.

That's a first diagnosis, maybe you could plot your learning curves so you can check your variance/bias and tell which one of those problems you have.


Biasing SVM algorithm towards particular subset of data I'm training an SVM model for sentiment analysis, based on social media data eg. tweets.
The model will be trained using a small selection of a particular company's tweets in order to classify new ones. However, since the training set is too small to get an accurate model I will be combining the company's data with a much larger general tweets dataset to train the model.
Being specialised to one company, the content of the respective data is slightly different to the content of the general dataset. Since the data to be predicted is company specialised, it seems logical to me to bias the models training  towards giving greater importance to the company related tweets to improve the accuracy. My first thought was simply increasing the magnitude of the polarity of the companies tweets, ie general tweets are -1 or 1 and company tweets are -3 and 3, for example.
Is this the right idea/method?

I don't think that's a very good idea: the goal is not to make the model predict a more extreme polarity when the tweet relates to the company.
Instead you might want to consider oversampling the few instances of this specific company. For instance if you have 100 company-specific tweets and 1000 general tweets in your training set, you could duplicate the company-specific ones 10 times in order to give the specific tweets have a higher weight in the data. If possible you should tune the parameter of how many times to duplicate in order to obtain the optimal value.


Which non-training classification methods are available? I am trying to find which classification methods, that do not use a training phase, are available.

The scenario is gene expression based classification, in which you have a matrix of gene expression of m genes (features) and n samples (observations).
A signature for each class is also provided (that is a list of the features to consider to define to which class belongs a sample).

An application (non-training) is the Nearest Template Prediction method. In this case it is computed the cosine distance between each sample and each signature (on the common set of features). Then each sample is assigned to the nearest class (the sample-class comparison resulting in a smaller distance). No already classified samples are needed in this case.

A different application (training) is the kNN method, in which we have a set of already labeled samples. Then, each new sample is labeled depending on how are labeled the k nearest samples.

Are there any other non-training methods?

Thanks

What you are asking about is Instance-Based Learning. k-Nearest Neighbors (kNN) appears to be the most popular of these methods and is applicable to a wide variety of problem domains. Another general type of instance-based learning is Analogical Modeling, which uses instances as exemplars for comparison with new data.

You referred to kNN as an application that uses training but that is not correct (the Wikipedia entry you linked is somewhat misleading in that regard). Yes, there are "training examples" (labeled instances) but the classifier doesn't learn/train from these data. Rather, they are only used whenever you actually want to classify a new instance, which is why it is considered a "lazy" learner.

Note that the Nearest Template Prediction method you mention effectively is a form of kNN with k=1 and cosine distance as the distance measure.


Can HMM be used as a binary classifier? I have some time-series data, which I need to use to predict a binary label for a given time-stamp. I was initially doing it using a Logistic Regression Model but I later figured that a time-series based problem could be better solved using Hidden Markov Models. I might also save on time since this prediction is to be made for ~200,000 time-stamps.

I have been able to obtain the next n hidden states using the hmmlearn implementation, but what I really need are 0/1 labels.

How can I (if possible) use HMM as a binary classifier?

I have no experience in hmmlearn.

But if you want to receive the labels for the underlying states of the process you have to do 'decoding' by using the 'Viterbi' algorithm. It calculates the most probable state sequence for your data. So if you only have two hidden states it calculates a 0/1 label for each of your time-stamps.

If instead you're trying to discern between two different processes and recognize them, you have to train two different HMMs for the two processes and calculate the foward probabilty up to time t. Then you can choose the process with the higher probability.


How to include lifecycle configuration in Sagemaker studio's user notebook I want to use lifecycle configuration in Sagemaker studio so that on start of user's notebook it runs the given lifecycle configuration.
My lifecycle configuration will have shell script which will launch cronjob having python script to send attached notebook's running duration.
#!/bin/bash
set -e

# PARAMETERS
IDLE_TIME=120

echo "Fetching the autostop script"
aws s3 cp s3://testing-west2/duration-check.py .
aws s3 cp s3://testing-west2/on-start.sh .

echo "Starting the SageMaker autostop script in cron"
(crontab -l 2>/dev/null; echo "*/1 * * * * /bin/bash -c '/usr/bin/python3 $DIR/duration-check.py --time ${IDLE_TIME} | tee -a /home/ec2-user/SageMaker/auto-stop-idle.log'") | crontab -

echo "Changing cloudwatch configuration"
cat $DIR/on-start.sh | sudo bash -s auto-stop-idle /home/ec2-user/SageMaker/auto-stop-idle.log

Referred lifecycle configuration from here:
https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/tree/master/scripts/auto-stop-idle

Original source
You can follow this tutorial:
https://modelpredict.com/sagemaker-stop-your-instances-when-idle/
Summary
Basically, you have to go to the notebook instance where you want to apply the Lifecycle Configuration, stop it, and go to: Additional configuration > Lifecycle Configuration > [Select your script].
After this, you have to configure the proper permissions for the Lifecycle Configuration script to stop your notebook instance. To do so, create a new policy, and paste the following code:
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "sagemaker:StopNotebookInstance",
              "sagemaker:DescribeNotebookInstance"
          ],
          "Resource": "*"
      }
  ]
}

After this, attach it to your Notebook instance and it's done!


What are the possible ways to detect skin while classifying diseases? I am working on a skin disease classification problem where I have successfully created a classifier ( TensorFlow + Keras ) which can classify images of two skin diseases.

The sample image needs to be classified in this manner :

Whether the sample is an image of the skin.
Does the skin have any two of the diseases ( Melanoma or Psoriasis )
If a disease is found, to which class does it belong (CLASS1: Melanoma or CLASS2: Psoriasis

  How can create Classifiers which could carry out the following tasks?
  Do I need the image localization or CNNs or something like YoLo?
  What steps should I implement?

I have created a classifier ( with an accuracy of 96% ) which can classify an image of the two diseases efficiently. But it can't detect the presence of the disease ( Step 2 in the above task list ).

You have to have sample tissues from skins with Melanoma, Psoriasis and Healthy Skins.

As is understood in the question, you developed a solution which can distinguish with 96% accuracy between melanoma and psoriasis. You are missing from the possibility of distinguish between sick and healthy skins.

The model you are looking for is a multiclass classification. Where CLASS3: Healthy skin.

You are doing ok so far, you just need to include this category and images of healthy skins in your model.


Given a regression based model with many feature variables; what tools would you utilize to figure out which feature variables add the most variance? Given a hypothetical dataset {S} with 100 X feature variables and 10 predicted Y variables.

X1
...
X100
Y1
....
Y10

1
..
2
3
..
4

4
..
3
2
..
1

Let's say I want to improve the accuracy of Y1. I am prepared to constraint/remove the input variables in order to increase the accuracy. How would I go about finding the culprits for making Y1 more variable than needed?
E.g. I find that X49 adds the biggest swing in variance with Y1 and after constraining it Y1 is fitted better.
How would I go about finding it's X49?
EDIT: I'm asking for approaches towards sensitivity analysis. Not deciding which variables need to be removed. Let's assume all 100 X variables are important but some need to be constrained (e.g. X49)

There might be a smarter method but I would simply try to fit a model without $X_i$ for every feature $X_i$ (and also a reference model with all the features). By contrast the model where $X_{49}$ is removed should obtain the lowest variance if $X_{49}$ is responsible for a lot of variance.
Be careful that in general a feature which causes a lot of variance is an important one, since if it wasn't important then it wouldn't have much impact on the target.


Usage of Precision Recall on an unbalanced dataset I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced  as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.

Also the confusion matrix is of no help too.

I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall?

If anyone can shed any light on what approach should I take, that would really help me move forward.

When evaluating your algorithms, especially when your dataset is unbalanced, you should use more metrics than just accuracy. The accuracy is how many examples you have correctly identified in total. As you have seen if you have an unbalanced dataset where 0.5% of your instances are 1's then this will result in 99.5% accuracy if you blindly set all your outputs as zeroes. This is obviously wrong albeit the high accuracy. The accuracy is calculated as

$Accuracy = \frac{\sum{TP} + \sum{TN}}{\sum{TP} + \sum{TN} + \sum{FP} + \sum{FN}}$

where TP is true positive, TN is true negatives, FP is false positives and FN is false negatives.

If you want to capture the performance of your unbalanced dataset you should look into the percentage of FP and FN you are calculating. You can do this using the sensitivity and the specificity. Calculate the sensitivity as

$Sensitivity = \frac{\sum{TP} }{\sum{TP}  + \sum{FN} }$

and the specificity as

$Specificity = \frac{\sum{TN} }{\sum{TN}  + \sum{FP} }$.

An ideal classifier should have the accuracy, specificity and sensitivity all be 1. This would mean every sample is correctly classified. In your case where you are getting very high false negatives, you will see that your sensitivity will be very low. This is a measure with which you can state that your algorithm is performing poorly. It is good form to always include these metrics in any statistical study you are doing. Accuracy alone is not sufficient to prove that you are obtaining good results.

Moreover, there is the receiver-operator curve (ROC). This will tell you your false positive rate for any true positive rate. You can then calculate the area under this curve (AUC) to get a comparable metric of performance.

All of these should be used together when exclaiming the performance of your algorithm. The ROC and AUC can be omitted however leaving out the sensitivity and specificity of your algorithm is unwise.


Trying to understand free-energy equations in a Karl Friston neuroscience article I am trying to understand a neuroscience article:

Friston, Karl J., et al. "Action and behavior: a free-energy formulation." Biological cybernetics 102.3 (2010): 227-260. (DOI 10.1007/s00422-010-0364-z)

In this article, Friston gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and Shannon entropy. They appear on page 231 of the article as equation (5):

  The resulting expression for free-energy can be expressed in three ways (with the use of the Bayes rules and simple rearrangements):

  • Energy minus entropy

  • Divergence plus surprise

  • Complexity minus accuracy

  Mathematically, these correspond to:

The things I am struggling with at this point are:

the meaning of the || in the 2nd and 3rd versions of the equations;
and the negative logs.

Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc?

I'm not qualified to understand almost all of that paper, but, I might be able to give some intuitions from information theory that help you parse the paper.

|| denotes the Kullback-Leibler divergence. It measures an information gain between two distributions. I suppose you could say it indicates the information in the real distribution of data that a model fails to capture.

When you see "negative log" think "entropy".

In the first equation, think of it as "-ln(...) - -ln(...)". This may help think of it as the difference of entropies. Likewise in the second, read it as "D(...) + -ln(...)". This may help think of it as "plus entropy".

If you look at the divergence definition, you'll see it is defined as the log of the ratio of the PDFs. This may help connect it to logs and negative logs. Look at the definition that writes it as cross-entropy minus entropy. Then this is all a question of differences of entropies of things which may be clearer.


Face detection for different poses more robust than MTCNN? I am using the MTCNN model described on machinelearningmastery here:  MTCNN ipazc

But it won't detect certain orientations, ie. somebody lying on the ground so the top of the head points to the right of the frame and their chin to the left.  Thus, I am going to cv2.flip on the y-axis and also rotate using the method described on pyimagesearch here:  rotation of images to try the detect() method on at least 8 different orientations of the same frame and then output the one with the most detections.

This may or may not work very well on video when faces and objects need to be detected on every frame.  Thus, I am looking for links to pretrained model zoos for MTCNN and other face detection, object detection and face recognition algorithms.

If anyone is looking for something that is more robust than MTCNN, try this but you need GPU with Cuda I believe.
It says Tesla P40 on requirements but that is for training only.
I could infer on new images with the provided parameters on a GTX 1080 with Cuda.
It's called DSFD from Tencent in China.  Their results on some well known benchmark datasets surpass MTCNN.
I can concur that it detected the above-mentioned side faces (not profile but lying sideways).
It is on Github:  DSFD Tencent
There is a demo.py where you can designate your own image as an arg on command line.  If you figure out how to use this with CPU only, please let me know.


Clustering of unlabeled ship images I want to create a ship detection classifier from a dataset that is formed by 4000 photos(3072*2048).

But the dataset  that i currently have is not labeled  so i can feed it to a cnn.So i want to cluster this dataset to 2 labels(or 2 directories) ship and no_ship.I tried running k-means but the results were dissapointing.Is some other more functional  way to do this?

In addition to the answers provided, you can:
1.) Train jointly a CNN (or Autoencoder) with clustering on your data. (DCN, kmeansNet,..)
2.) Pretrain a CNN using self-supervision on your data. (Have a look into the vast self-supervision literature, e.g. this work).
3.) Use an alternating scheme to train a CNN classifier on soft-labels provided by a clustering algorithm, e.g k-means (e.g. this work).


How to properly split train and test to measure performances? Have specific question about how to measure a model's performances:

Is it correct to have common samples between the train and the test sets?
Is it correct to have duplicates samples in the test set?

welcome to the forum. I dont know what you exactly mean with 1) „common samples“.

But train and test should have no common rows/observations. They should be two distinct sets.

Having duplicates in test is not a good idea since you want to test how well your model works on data not seen so far by the model. Duplicates in test shrink the number of unique test cases. So there is little use of duplicates in the test set.


How do scientists come up with the correct Hidden Markov Model parameters and topology to use? I understand how a Hidden Markov Model is used in genomic sequences, such as finding a gene. But I don't understand how to come up with a particular Markov model. I mean, how many states should the model have? How many possible transitions? Should the model have a loop?

How would they know that their model is optimal?

Do they imagine, say 10 different models, benchmark those 10 models and publish the best one?

I'm familiar with three main approaches:

A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes, and so have 44 states for the hidden phoneme layer in a voice recognition model.
Estimation. The number of states can often be estimated beforehand, perhaps by simple clustering on the observed features of the HMM. If the HMM transition matrix is triangular (which is often the case in failure prediction), the number of states determines the shape of the distribution of total time from the start state to the end state.
Optimization. Like you suggest, either many models are created and fit and the best model selected. One could also adapt the methodology that learns the HMM to allow the model to add or discard states as needed.


How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset? I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best solution for any given algorithm (i.e. ellbow-curve, silhouette-coefficient etc.).
But I get very different results - as expected with different algorithms. K-Means is good for spherical clusters, density-based approaches for totally different cluster shapes.
Now the actual question: How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset? Is there a scientific way to go? Any comparative metric (like rand index) that can be used? Some papers on that topic? Maybe even a flowchart?

This is a non-trivial question. There is no general law to find the "best" algorithm or the "correct" amount of clusters (assuming you don't know the correct number of clusters).
As you already mentioned, certain algorithms make assumptions on the shape of your clusters or your data in general. One thing I suggest is to look at your data, check the assumptions and rule out algorithms where assumptions are violated. dimensionality reduction methods like PCA, t-SNE, UMAP, etc. are very helpful here if you work with high dimensional data.
Further have a look at the complexity of your clustering algorithm. If your are familiar with the Bias-variance tradeoff and Occam's razor you already know that simpler algorithms are less prone to overfitting and more likely to give you the correct results. BIC and AIC are quite popular measures in this regard.
Generally for unsupervised clustering it is a good idea to look at a number of different algorithms, compare them with different metrics and look where you can find the highes "agreement" among them.


Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies I'm still in the process of learning, so I'm sorry if this doesn't make much sense.

I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with imputers for numerical data and OneHotEncoding for categorical data. On the other hand, we also learned about using just pd.get_dummies which from my understanding just works as OneHotEncoding (please, correct me if I'm wrong about this).
However, using just pd.get_dummies seems to give me better results than the other method but in the other method I'm even going as far as to impute the numerical values where as with pd.get_dummies I'm not even sure what happens to missing numerical data.

So, what's going on here?

Thanks!

PS: After we use pd.get_dummies we then use "align". I read the docs but I'm still not entirely sure the purpose of doing this

X_train = pd.get_dummies(X_train)
X_valid = pd.get_dummies(X_valid)
X_test = pd.get_dummies(X_test)
X_train, X_valid = X_train.align(X_valid, join='left', axis=1)
X_train, X_test = X_train.align(X_test, join='left', axis=1)

"(Making) dummy variables" and "one-hot encoding" are the same thing, yes.  (The phrasings originate from statistics and computer science, respectively.)

pandas.get_dummies and sklearn's OneHotEncoder both implement this process, but have some differences in their (current) implementations.

Skip to the end if you're not interested in some details that might not be relevant for your underlying question.

I think there are three basic questions to ask:

What do we do if the train set has levels the test set doesn't?
What do we do if the test set has levels the train set doesn't?
Do we want one dummy variable for every level?  [Otherwise, you get perfect multicollinearity, bad for some models.]

Does "missing" count as a level?

(And to be more general, production sets count here as test sets, and validation sets can count as either training or test sets depending on your usage.)

In both (1) and (2), we'll have problems applying a model fitted on the train set to the test set without rectifying the differences.  In (1), we just need to provide the extra columns to the test set (which will be all-zeros, but that's what we expect).  In (2) we have more serious troubles: the model has no idea what to do with those new levels.  The only thing to do is to find a reasonable way to convert those new levels to the "closest" training levels (which requires domain knowledge and/or a separate analysis), or failing that, just encode those levels as all zeros (essentially converting those levels to the "reference level" of the training set, i.e. none of the training levels).

pandas is more about data manipulation and aggregation, not about modelling, so doesn't really need to care about what to do with these issues.  However, align gives us a way to work it out: it will conform the two sets' column names, filling in gaps with np.nan.  Using the left-join as in your code takes care of (1).  (Using outer-join would take care of the programmatic problem in (2), but the train set will still have a column of all nans, so the model won't actually know what to do with those levels.)  Finally, get_dummies helpfully has the options drop_first and dummy_na to take care of (3).
(pandas version 0.25)

sklearn of course is all about modeling, and cares a great deal about train/test splits.  OneHotEncoder then has the fit method that bakes in the fitting set's categories (later available with the attribute categories_), and the transform method uses only those categories for later use (with e.g. test sets).  It has a useful option handle_unknown, that when set to 'ignore' takes care of (1).  There has been discussion/work toward giving options to handle (2) [namely, providing an "other" level], as well as dealing with missing values [currently it produces an error].  It can take care of (3) with the option drop.  Things are a little more difficult here too because sklearn uses numpy arrays internally instead of dataframes, so the columns don't actually get names.  (It might be worth mentioning that earlier versions of sklearn was much more restrictive on the inputs to OneHotEncoder.)
(sklearn version 0.21)

Alright, finally to the base question.  It may be that the slightly different options above produce different enough encoded datasets as to cause you problems in modeling.  (Something else to check: the methods are dummifying the same columns, right?)  It is more likely, I think, that the imputation method you've employed has hurt performance.  I'm not familiar enough with the Housing Prices dataset to know, but if the missing values aren't missing at random, then imputation might misrepresent the truth.  And, since XGBoost can handle missing values without imputing, you might have more similar results by skipping this step.


How Tensorflow text prediction predicts without softmax activation In the Colab notebook here:  RNN text generation
in def generate_text(), there is
predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

I looked into tf.random.categorical here:
stackoverflow answer

and sort of understand how it works.

I tried to debug/figure out what it does with print statements:

for i in range(num_generate):
      predictions = model(input_eval)
      # remove the batch dimension
      predictions = tf.squeeze(predictions, 0)

      # using a categorical distribution to predict the word returned by the model
      predictions = predictions / temperature
      predicted_id1 = tf.random.categorical(predictions, num_samples=1)
      predicted_id2 = tf.random.categorical(predictions, num_samples=1)[-1,0]
      predicted_id3 = tf.random.categorical(predictions, num_samples=1)[0,0]
      predicted_id4 = tf.random.categorical(predictions, num_samples=1)[-2,0]
      predicted_id5 = tf.random.categorical(predictions, num_samples=1)[1,0]
      predicted_id6 = tf.random.categorical(predictions, num_samples=1)[2,0]
      predicted_id7 = tf.random.categorical(predictions, num_samples=1)[3,0]
      #predicted_id8 = tf.random.categorical(predictions, num_samples=1)[4,0]

      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy() #index 0 for

      print("predicted_id1", predicted_id1)
      print("predicted_id2", predicted_id2)
      print("predicted_id3", predicted_id3)
      print("predicted_id4", predicted_id4)
      print("predicted_id5", predicted_id5)
      print("predicted_id6", predicted_id6)
      print("predicted_id7", predicted_id7)

      print("predicted_id", predicted_id)

This was the output:

predicted_id1 tf.Tensor(
[[19]
 [33]
 [ 3]
 [35]
 [ 4]
 [64]
 [22]], shape=(7, 1), dtype=int64)
predicted_id2 tf.Tensor(35, shape=(), dtype=int64)
predicted_id3 tf.Tensor(19, shape=(), dtype=int64)
predicted_id4 tf.Tensor(3, shape=(), dtype=int64)
predicted_id5 tf.Tensor(38, shape=(), dtype=int64)
predicted_id6 tf.Tensor(26, shape=(), dtype=int64)
predicted_id7 tf.Tensor(36, shape=(), dtype=int64)
predicted_id 29

So it looks like there is some kind of distribution and that some of the indices pick from that distribution but the actual prediction in this case of 29 does not appear in the distribution so I am confused.  Are the elements in the distribution not integer ID's of characters in the text?  The one way I learned in the Udacity DLND was to assign probabilities to the predicted next character and choose the argmax so please feel free to enlighten me.

The logits have the distribution according to the len of vocabulary and the model training. So, you can use np.argmax(logits) to get the prediction, but normally to the application of generating script is more interesting to take into account a factor of aleatory which in this case is the function "random_categorical" which is used to get the value according to the probability. The value 29 appears in the distribution, you didn't see, because you execute the function a lot of times and every execution, "random values" will appear. If you want the exact value, use np.argmax.


Neo4j graph to cypher conversion Is there a way or tools available to generate or retrieve cypher query from a Neo4j database ? Should we need to store cypher quries along with the graph data for regeneration ?

This has already been answered in stackoverflow. There is no need to save the queries you used to populate the database, you only need to dump the contents of the database to a file:

db1/neo4j-shell -path db1/data/graph.db/ -c dump > export_data.cypher

In order to load the database dump into another database, you just supply it to neo4j shell through the standard intput:

db2/bin/neo4j-shell -path db2/data/graph.db/ < export_data.cypher


Amazon S3 vs Google Drive The majority of people use S3. However, Google Drive seems a promising alternative solution for storing large amounts of data. Are there specific reasons why one is better than the other?

Personally, we use S3 on top of GCE and really love it. Depending on how much data you're dealing with, Google Drive just doesn't quite match the 5 TB max that S3 gives you. Also, if you're using python, boto does a pretty fantastic job of making most aws services pretty accessible regardless of what stack you're dealing with. Even if you're not using python, they've got a pretty straightforward API that generally is more accessible than Google Drive.

Instead of google drive, though google did recently release a cloud storage service, apart from drive, that lets you more closely integrate your storage with any gce instance you've got, https://cloud.google.com/products/cloud-storage/

They've got an API which seems to be pretty comparable to S3's, but I can't profess to having really played around with it much. Pricing-wise the two are identical, but I think that the large community and experience with aws in general still puts S3 squarely above both google's cloud storage and google drive.


Handle 50,000 classes in OneVsRestClassifier I'm new to data science and NLP. I'm trying to solve a problem that is having 1 million rows and some 50,000 distinct classes. The dataset has some text column as a predictor and the other one is the multilabel responses. I have been using tfidf to represent the text fields and MultiLabelBinarizer to transform the labels. But MultiLabelBinarizer is giving MemoryError.

And there is no way I can pass the legacy multi-label data representation using a sequence of sequences as it seems no more to be supported in the sklearn package. So, what should be my approach?

Any help is appreciated. Thanks in advance.

As you already know, the main culprit here is the large number of classes (50k). For every sample data, you have a label of size 50k. Even if a sample only belongs to a single class, the label will still be of size 50k.

For example, sample1 has a label of A and B while sample2 has a label of A. Sample3 on the other hand has C for its label and sample4 has A. We can visualize it like this:

sample1 - [1, 1, 0, 0, 0, 0, ..., 0]
sample2 - [1, 0, 0, 0, 0, 0, ..., 0]
sample3 - [0, 0, 1, 0, 0, 0, ..., 0]
sample4 - [1, 0, 0, 0, 0, 0, ..., 0]

One solution here is to use Siamese neural network. This takes in two input at a time. Instead of training directly to learn classes as the output, we are training to learn the similarity between two samples. We first select random pairs. We use the label 1 if two samples belong to the same class; else 0.

sample1 & sample2 - 1
sample1 & sample3 - 0
sample2 & sample4 - 1
sample3 & sample4 - 0

This way, even if there is a lot of classes, you are only training with 2 labels, 1 or 0, thus, avoiding memory error.

The problem now is that you are doing a multi-classification task. For instance, sample1 and sample 2 are the same class A but sample1 also belongs to class B. My suggestion, but I'm not totally sure here, is to add multiple instance of that pair.

sample1 & sample2 - 1
sample1 & sample2 - 0
sample1 & sample3 - 0
sample2 & sample4 - 1
sample3 & sample4 - 0

As for the large number of training samples (1 million), this can be handled by using small batches during training.


ARIMAX with spark-timeseries Cloudera recently added the spark-time series library to github. According to the user docs, it definitely can fit autoregressive integrated moving average (ARIMA) models, but I see no mention of ARIMAX, which takes into account explanatory variables. Does anyone with more experience with this library know whether ARIMAX is possible in the library's current implementation? Mathematically, I understand the difference between ARIMA and ARIMAX, and I know extending it myself wouldn't be terribly difficult, but I'm right now looking for an off-the-shelf solution in spark. Can anyone recommend an alternative spark implementation?

Blast. It looks as though ARIMAX is currently a work in progress for the cloudera/spark-timeseries community. Although some work has been done on the model development side, it doesn't appear to have been merged in.


How to find optimal number of trees in random forest using Grid search in R? From below code, I am getting optimal number of mtry. What is this mtry ? and How should I find the optimal number of tree that to be assigned to Random forest algorithm so that it will give High accuracy.

Any comment will be highly appreciated ! Thanks in Advance.

classifier = train(form = Survived ~ ., data = training_set_scaled, method = 'rf')
classifier
classifier$bestTune

mtry is the parameter indicating how many of the features are checked in each split decision.  http://topepo.github.io/caret/train-models-by-tag.html#random-forest , see method 'rf'

In Random Forest, usually more trees give more stable results, and overfitting due to number of trees is rare.  Moreover, since the trees are built independently, you could just fit many trees then take subsets to get smaller models.  But, see e.g. hyperparameter tuning in mlr for how to perform grid, random, Bayesian, and other hyperparameter searches, or grid in caret or random in caret.


Is Data Science just a trend or is a long term concept? I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data science is just a mix of computer science and statistics techniques.
So I always think about this. If it is just a trend and if in 10 years from now, someone will still mention Data Science as an entire field or just a subject/topic inside CS or stats.
What do you think?

The one thing that you can say for sure is: Nobody can say this for sure. And it might indeed be opinion-based to some extent. The introduction of terms like "Big Data" that some people consider as "hypes" or "buzzwords" don't make it easier to flesh out an appropriate answer here. But I'll try.

In general, interdisciplinary fields often seem to have the problem of not being taken serious by either of the fields they are spanning. However, the more research is invested into a particular field, the greater is the urge to split this field into several sub-topics. And these sub-topics sonner of later have to be re-combined in new ways, in order to prevent an overspecialization, and to increase and broaden the applicability of techniques that are developed by the (over?)specialized experts in the different fields.

And I consider "Data Science" as such an approach to combine the expertise and findings from different fields. You described it as

  ...a mix of computer science and statistics techniques

And indeed, several questions here aim at the differentiation between data science and statistics. But a pure statistician will most likely not be able to set up a Hadoop cluster and show the results of his analysis in an interactive HTML5 dashboard. And someone who can implement a nice HTML5 dashboard might not be so familiar with the mathematical background of a Chi-Squared-Test.

It is reasonable to assume that giving students enough knowledge to apply the most important techniques from the different fields that are covered by data science will lead to new applications of these techniques, and be beneficial - also for the "purists" in these fields. The combination of these techniques is not straightforward in many cases, and can justify an own branch of research.

You also asked whether in 10 years, data science will be considered as "just a topic inside computer science". Again: Nobody can say for sure. But I wonder at which point people stopped asking the question whether "Computer Science" will one day only be considered only as a mix of (or a subject of) Electrical Engineering and Mathematics...


Can setting of different thresholds help in model performance in case of handling class imbalances? In a binary classification problem where there is class imbalance, after applying undersampling/oversampling or SMOTE techinques, is it still convention to use a 0.5 threshold if we are making the two classes completely balanced? Or should we still change the threshold based on what we are trying to optimize?

If changing the threshold improves the performance of the model, it would be better to change it and use an optimal value. Balancing the dataset does not mean that you can not change the discrimination threshold. Also, in some cases when the data is originally balanced, changing the threshold could be very useful and it is a very smart move.


Python library to detect a bank/financial institution name in a string I would like to extract bank names from a given text like wells Fargo, chase....is there a python library for this? I know there is entity tagger in space and flair but they only identify the entity (org/person)

As mentioned in the comment you can use regex but you would need to define a set of rules for it. You can try with LexNLP which is trained on legal documents and use it to extract data type like address, companies and persons.


Data Science Project Ideas I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.

I have just started with Data Science and Machine learning. I am looking for long term project ideas which I can work on for like 8 months.

A mix of Data Science and Machine learning would be great.

A project big enough to help me understand the core concepts and also implement them at the same time would be very beneficial.

I would try to analyze and solve one or more of the problems published on Kaggle Competitions. Note that the competitions are grouped by their expected complexity, from 101 (bottom of the list) to Research and Featured (top of the list). A color-coded vertical band is a visual guideline for grouping. You can assess time you could spend on a project by adjusting the expected length of corresponding competition, based on your skills and experience.
A number of data science project ideas can be found by browsing Coursolve webpage.
If you have skills and desire to work on a real data science project, focused on social impacts, visit DataKind projects page. More projects with social impacts focus can be found at Data Science for Social Good webpage.
Science Project Ideas page at My NASA Data site looks like another place to visit for inspiration.
If you would like to use open data, this long list of applications on Data.gov can provide you with some interesting data science project ideas.


Why Decision Tree Classifier is not working with categorical value? I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx.

I am trying to model a Decision Tree Classifier as part of an ensemble (soft voting system). The problem is that I have the categorical features already converted to Integer values (i.e. if blood pressure is within the medium range, then blood pressure =2 in the dataset).

The categories for the feature variables are as follows

feature#1 (total cholesterol) = 1, 2, 3

feature#2 (Systolic Blood Pressure) = 1, 2, 3

feature#3 (diastolic Blood Pressure) = 1, 2, 3

feature#4 (smoking rate) = 1, 2, 3

The target/class variable (stroke) = 1, 2 (yes and no, respectively).

The problem is that when I draw the DT (please check the picture), I expect to get a splitting decision where a certain feature equals one of the above-mentioned values (i.e. 1, 2, 3). However, it is giving me other values like 1.5, 0.5, etc.,

I have included the code and a sample dataset for anyone who can help me out. btw, I need to keep the current dataset in the number format as I am using it in conjunction with other classifiers for the soft voting classifier.

Here is the dataset and this is the code for reference, I am using Spyder with Python 3.7

Its all ok.

Exactly because your values are between 1 and 2 does he chooses 1.5. In other words trees will select 1 and 2 as values (integers not real numbers) but 1.5 is there as cut-off point between these classes.


Regression - How Random Forest and Gradient Boosting really works? I read a lot about random forest and gradient boosting, but I do not know how these two algorithms really work.

For example, see the simple picture about basketball (picture 1) from this link:

How does Random Forest and how does Gradient Boosting work?
Has each tree in the random forest different trainings data AND different features?

Sorry about m question but I don‘t find an easy non-mathematical answer.

Random Forest:  Build a decision tree.

Sample N examples from your population with replacement (meaning examples can appear multiple times).
At each node do the following

Select m predictors from all predictors
Split based on predictor that performs best via some objective function
Go to the next node, select another m predictors and repeat

Combine all trees as an average or weighted via some scheme.

Gradient Boosting

One key note is that random forest trees essentially indepedent of each other.  Boosting algorithms add a certain depedency to the model.

Initialize a model by finding the minimizer of a certain objective function
For each iteration

Compute the partial derivative of -$L(y_{i}, F(x_{i}))$ with respect to $F(x_{i})$ for all i to n
Fit a tree $h_m$ to the the result from above.
solve $\lambda_m =arg min \sum_{i=1}^{n}L(y_{i}, F_{m-1}(x_{i}) + \lambda h_{m}(x_{i})$

Update model via  $F_m(x) = F_{m-1} + \lambda_m h_{m} (x)$


Confused with the derivation of the gradient descent update rule I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation:
$$∆C=∇C∙∆w$$
where $∇C$ is the gradient vector/vector derivative of the cost function (MSE) and $∆w$ is the change in weights. It said that the goal is to make the change in cost negative. Good so far. My issue is with the next part. It states that
$$∆v=-η∇C$$
My issue is with this, and why $∆v$ is set to this. Why would we want to change the weights by a small amount of the gradient function?

Upon writing this I have realised the answer to the question. I am still going to post so that anyone else who wants to learn where the update rule comes from can do so. I have come to this by studying the equation carefully. $∇C$ is the gradient vector of the cost function. The definition of the gradient vector is a collection of partial derivatives that point in the direction of steepest ascent. Since we are performing gradient 'descent', we take the negative of this, as we hope to descend towards the minimum point. The issue for me was how this relates to the weights. It does so because we want to 'take'/'travel' along this vector towards the minimum, so we add this onto the weights. Finally, we use neta which is a small constant. It is small so that the inequality $∆C>0$ is obeyed, because we want to always decrease the cost, not increase it. However, too small, and the algorithm will take a long time to converge. This means the value for eta must be experimented with.


Reinforcement Learning Vs Transfer Learning? I recently saw a video lecture from Jeremy Howard of fast.ai in which he states that transfer learning is better than reinforcement learning. But I was unable to understand the reasoning behind it.

Can someone explain to me or point to any evidence stating which is better and why?

I didn't watch this lecture, but, the way I see it, reinforcement learning and transfer learning are absolutely different things. Transfer learning is about fine-tuning a model, which was trained on one data and then striving to work with another data and another task. For example if you use weights of pretrained model on imagenet and then implement it to your dataset, while your dataset consists of small amount of different species of birds images (which might be not sufficient to train for example unet from a scrath). Reinforcement learning is about how some agent should response to environment condition to receive high reward. There is an illustrative example with a drone making a delivery, when there is some range of restrictions of the environment. There are two links, which might be useful:
https://machinelearningmastery.com/transfer-learning-for-deep-learning/
https://skymind.com/wiki/deep-reinforcement-learning

I guess, I can't answer, which approach is better, because they aim to solve different challenges.


How do I convert strings in CSV into integer in Pandas? for my supervised classification problem,

I have a train dataset which contains past purchase data of customers and 5 new products are purchased by these customers. I have a test dataset which contains past purchase data of customers. They never bought from these 5 products.

I want to train a predictive model, I need to convert integer purchased data and new products. There are so many different products in my purchased data.

I saw that the mapping was made for true-false to convert 1-0, but in this case what can I do that?

You cannot convert your string product names to integers and expect    it to work, if you convert it to integers the algorithm you use will    expect a linear relationship between these integers, a relationship    that in fact does not exist.

It is categorical data, and you should treat it like that, you have to one-hot encode it. If you have too many products to one-hot    encode all of them try grouping them in categories and then one-hot encoding the categories.


LSTM-Model - Validation Accuracy is not changing I am working on classification problem, My input data is labels and output expected data is labels
I have made X, Y pairs by shifting the X and Y is changed to the categorical value
Labels  Count
1      94481
0      65181
2      60448

    X   Y
    2   1.0
    1   2.0
    1   1.0
    2   1.0
    2   2.0

encoder = LabelEncoder()
test_labels = to_categorical(encoder.fit_transform(values[:,1]),num_classes=3)
train_X,test_X,train_y,test_y= train_test_split(values[:,0], test_labels,test_size = 0.30,random_state = 42)

print(train_X.shape)
print(train_y.shape)
print(test_X.shape)
print(test_y.shape)

(154076,)
(154076, 3)
(66033,)
(66033, 3)
Converting this to LSTM format
train_X = train_X.reshape(train_X.shape[0],1,1)
test_X = test_X.reshape(test_X.shape[0],1,1)

# configure network
n_batch = 1
n_epoch = 10
n_neurons = 100

ModelArchitecture
tf.keras.backend.clear_session()

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(n_neurons, batch_input_shape=(n_batch, train_X.shape[1],train_X.shape[2]), stateful=True),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(100, activation = 'relu',kernel_regularizer=regularizers.l2(0.0001)),
    tf.keras.layers.Dense(3, activation='softmax')
])

model.summary()
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['acc'])
history = model.fit(train_X,train_y,validation_data=(test_X, test_y),epochs=n_epoch, batch_size=n_batch, verbose=1,shuffle= False)

Validation Accuracy is not Changing
Epoch 1/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0844 - acc: 0.4269 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 2/5
154076/154076 [==============================] - 354s 2ms/step - loss: 1.0853 - acc: 0.4256 - val_loss: 1.0813 - val_acc: 0.4310
Epoch 3/5
154076/154076 [==============================] - 355s 2ms/step - loss: 1.0861 - acc: 0.4246 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 4/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0874 - acc: 0.4228 - val_loss: 1.0825 - val_acc: 0.4310
Epoch 5/5
154076/154076 [==============================] - 353s 2ms/step - loss: 1.0887 - acc: 0.4208 - val_loss: 1.0828 - val_acc: 0.4310

What can be the changes to improve the model.
Update2
After Increasing the learning rate of Rmsprop to 0.5 , Below is the training loss and validation loss
Epoch 1/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.4314 - acc: 0.3594 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 2/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 3/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 4/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 5/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 6/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 7/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 8/10
176087/176087 [==============================] - 372s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 9/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9379 - val_acc: 0.2739
Epoch 10/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854

From the documentation:
inputs: A 3D tensor with shape [batch, timesteps, feature].
Are you saying that you want 1 input and 1 feature, but you want to output 100 neurons? I would consider adding more timesteps.
train_X = train_X.reshape(train_X.shape[0],10,1)
test_X = test_X.reshape(test_X.shape[0],10,1)

Also, I wouldn't add regularization to a ReLU activation without batch normalization. Actually, I probably would use dropout instead of regularization. Consider that with regularization many ReLU neurons may die.


How to Predict/Forecast street's Traffic based on previous values? I have a dataset which has the following 5 columns:

date, hour, day_of_week, street_id, counts

My dataset has information about the number of cars that each street (same city) has in a given hour of a certain date, and I want to predict the traffic count that a certain street has in a given hour of a certain date.

I think I could use certain variables depending on the day and hour that I want to predict, for example, if a want to predict the traffic count of a working Wednesday:

Results of others working days
Results of others Wednesdays
...

I want to use Spark MLlib to perform the prediction because I have experience with Spark and I have large datasets.

How you deal with this kind of problem?

Any ideas?

This looks like a Time Series problem. So based on a variable's past values, you try to predict the future values.

Usually an "unheard of" problem with Spark, but you are in luck ; spark-ts library seems to be doing what you need, so you don't need to code your own using MLlib. I recommend you try it out and then circle back to something in MLlib if things don't work.

They have introduced a TimeSeriesRDD and once you can encode your data in this data structure (Note that this still behaves like a normal RDD), you can play around with the models available. For example, implementing the ARIMA model would be as simple as  -

val arimaModel = ARIMA.fitModel(1, 0, 1, ts)

Hope that helps!


Visualization of multiple Markov models I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan.

Most often single models are visualized with a decision tree or transition state diagram. However, with multiple different models what are potential visualizations that could communicate the transition states that differentiate each model?

I have seen other people use a table to depict different models and the transition states.

For clarity, I am not referring to a transition probabilities chart but a method of communicating the differences between multiple models.

If we limit the question on comparing two graphs, I can propose a way based on adjacency matrices comparison. There is a sample notebook: graph_diff.ipynb

To summarize:

Having two graphs,

   A  B  C  D                  A  B  D  E  F
A  0  2  2  2               A  0  1  2  3  0
B  2  0  1  1               B  1  0  0  1  1
C  2  1  2  0               D  2  0  2  1  0
D  2  1  0  0               E  3  1  1  0  1
                            F  0  1  0  1  0

We can compare them and detect changes, producing result similar to diff output:

   A  B  C  D  E  F
A  1  0 -2  1  2  2
B  0  1 -2 -2  2  2
C -2 -2 -2 -2 -2 -2
D  1 -2 -2  2  2  2
E  2  2 -2  2  2  2
F  2  2 -2  2  2  2

Compare matrix nodes from both graphs. Edges values indicate change:

1 = same edge
0 = changed edge
-2 = removed edge
+2 = added edge

This matrix can be visualized as grid:

Or as graph:

which is more informative because shows also nodes changes:

green: new (added nodes E and F)
red: removed (removed C)
yellow: unaffected (A, B, D)

For simplicity, edges are unidirectional.


Can we use decreasing step size to replace mini-batch in SGD? As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample in each iteration? Can we compare the convergence rate of them?

Generally answer is "it's not known". Similarity of effects of increasing minibatches size and decreasing learning rate is mostly empirical, there is no known asymptotic formula for it. Also effect of small LR and big minibatch is not the same. For example batch normalization layer would act completely different on those two approaches. Probabilistic distribution of gradients produced by minibatches and single sample (or mb of significantly different size) would be also quite different


Should you use random state or random seed in machine learning models? I'm starting to study machine learning. All the examples I saw, the person that created the ML model used a random state or a random seed to stop the randomness of the process. But, in real life, when you're trying to apply a machine learning model into an actual project of a company, should you use any random state or seed? Is it right (in data science terms) to set a random state to the machine learning model and make reproducible results?

Your intuition is correct. You can set the random_state or seed for a few reasons.:

For repeatability, if you want to publish your results, share with
other colleagues
if you are tuning the model, in an experiment you usually want to
keep all variables constant except the one(s) you are tuning.

I usually set the random_state variable, not the random seed while tuning or developing, as this is a more direct approach. When you go to production, you should remove the random_state and or random_seed settings, or set to None, then do some cross validation. This will give you more realistic results from your model.


Mutate with dynamic column names dplyr Hi I have this dataset (It has many more columns)

   media   brand    radio        tv            cinema
   <chr>   <dbl>    <dbl>       <dbl>            <dbl>
  radio      0      0             0                0
  tv         0      0             0                0
  cinema     0      0             0                0
  tv         0      0             0                0
  radio      0      0             0                0
  tv         0      0             0                0

I want to obtain the following(Assign a 1 to each column based on the value of media column):

        media   brand    radio        tv            cinema
       <chr>   <dbl>    <dbl>       <dbl>            <dbl>
      radio      0      1             0                0
      tv         0      0             1                0
      cinema     0      0             0                1
      tv         0      0             1                0
      radio      0      1             0                0
      tv         0      0             1                0

Do you have any  idea on how to do it?As I have many more columns, I'm trying to find a solution using dynamic variable assignment.
Thanks

What you are trying to do is called one hot encoding or dummy encoding. Even as this may be possible to accomplish in dplyr (together with tidyr), I would recommend using the function one_hot() from the library mltools. For an explanation, see the following code snippets:

# set up sample data
df <- data.frame(
  id = 1:4,
  media = factor(c("radio", NA, "tv", "tv"), levels=c("radio", "tv", "cinema"))
)

One Hot Encoding with tidyr:

The following code snippet shows how you would accomplish one hot encoding with dplyr and tidyr (copied from this comment):

library(dplyr)
library(tidyr)

tib <- as_tibble(df)
tib %>%
  mutate(i = 1) %>%
  spread(media, i, fill = 0)

However, this code will give you the following table:

# A tibble: 4 x 4
 id    radio   tv   `<NA>`
<int>  <dbl>  <dbl>  <dbl>
  1      1      0      0
  2      0      0      1
  3      0      1      0
  4      0      1      0

Note that there is no cinema column present, even though we defined a respective level in our sample data-frame. Also, the spread()-function created an own column for values with NA's, treating them as a separate category.

One Hot Encoding with mltools (recommended):

library(data.table)
library(mltools)

dt <- data.table(df)
one_hot(dt)

This code will give you the following table:

id  color_radio  color_tv  color_cinema
1          1          0            0
2         NA         NA           NA
3          0          1            0
4          0          1            0

As you can see, the column cinema is now present even though our sample data-frame did not contain any observations of it. Also, there isn't an NA-column present anymore. The NA from the second row is instead visible in all columns (which makes way more sense, since you don't know, what category this observation IS, so you cannot know what category it IS NOT).

In addition, this code is not only easier to understand it also runs about two to three times as fast.


Continous bag of words claimed to be unsupervised, how is it working? I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series

https://www.youtube.com/watch?v=syWB-YMYZvI

https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=548s

Up to about 17 mins into the second video the lecturer says that the approach is unsupervised for CBOW as there are no labels? How can you learn a NN with no labels? This completely confused me as why are we not comparing our softmax probability vector to an actual set of outputs so that we can adjust the v_c and v_w weights accordingly. His liklihood function seems to only be concerned with the parameters v_c and v_w (and completely devoid of some kind of target label) which is bonkers to me, because could i not not just make them whatever i want? In addition how are you learning relationships between pairs of actual words, if actual target variables are not guiding you towards correct labels? Could someone please explain what is going on under the hood as i really want to understand this approach. Perhaps target labels are being considered and i've not noticed it?

Most log likelihood estimates with a window of size $m$ look something like the following $-\mathrm{log} \prod_{j=0,j \neq m}^{2m} \frac{e^{u^T_{c-m+j}v_c}}{\sum_{k=1}^{|v|} e^{u_k^Tv_c}} $

To my knowledge a likelihood function should involve data from actual observations not purely parameters, which are to be estimate, can someone explain this also?

Note i follow it more from 17 mins onwards as he talks about pairs (w,c) in D or D' respectively. I appreciate any help!

The CBOW approach is unsupervised because the network learns the distribution of word co-occurrences around each word, and this doesn't require labelling or additional input, just sequences of words.

As Mikolov et al state in one of the original articles, "the training objective is to learn word vector representations that are good at predicting the nearby words; in another, Mikolov et al say their aim is "building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word". So if you can see that for each sequence, you're taking 'x' words and training the network to predict one given the others, then no supervision is involved.

It's a little unconventional because it's not the output of the network which is important, but the weights that are learned during training - these are what is taken and used as embeddings in other tasks.

Adrian Colyer does a great general write-up of word2vec here, and the explanation from Chris McCormick here is good and quite accessible.