100% Accuracy on test dataset using a previous developed model oputput My dependent variable is a probability that is sourced from someone else's classification model. I am using this probability as a dependent variable as I don't have the actual data. On building an xgboost algorithm, the accuracy is 100%. There is no data leakage - but I wanted to ask - is a 100% accuracy possible when modeling a previously developed algorithm?

Welcome to the community. If I understood correctly:

you have is a dataset whose target labels (aka ground truth) you don't know, so you figured these out by assigning the output of another model from someone else --> I guess this pre-trained model was built with the same dataset right?
you built a new model on this dataset, using as target values the other model's output as your ground truth, giving you a 100% accuracy --> you convert the predicted probability to the corresponding label is it?

This might mean:

your model is providing the same output labels as the other model trained on the same dataset; this is not strange, mainly if the models are of the same type (e.g. tree-based models as your xgboost), mainly assuming the pretrained model outputs fit the real target values
the data is quite easy to "learn", so both models provide similar results over the same target labeling
to take into account: accuracy might not be an optimal metric, taking a look at how much unbalanced is the dataset (it is, what is the ratio between the different classes).

Nevertheless, I would make sure to find the real and precise corresponding targets to go on with confidence on your problem.


Creating radial basis for linear regression Python I'm trying to do time series forecasting with linear regression like it's done in this video: Radial basis forecasting starting from 5:50.
I understand the basic idea of basis, but I don't think I understood the usage of it in time series data correctly. I have a Pandas dataframe with daily timestamps and target variables. I tried writing radial basis function
def radial_basis(x, month):
    alpha = 0.5
    coef = -1/(2*alpha)
    return np.exp(coef*(x-month)**2)

and calculating basis for every month (x is row number). This didn't work.
Any tips on how I should try to do this?

I was able to solve this by myself.
First you need to create a column that contains day of year values from the timestamps.
Then apply radial_basis function for that column with month parameter being the middle day of every month. For example in January it's 15, February 45 etc.
With this method you can generate a spike for every month.
data_all['Day'] = data_all.Timestamp.dt.dayofyear

def radial_basis(x, month):
    alpha = 8
    coef = -1/(2*alpha)
    return np.exp(coef*(x-month)**2)

for i in range(12):
    col = 'RB' + str(i+1)
    data_all[col] = data_all.Day.apply(radial_basis, month=(15+30*i))


model.score and r2_score giving different values for a regression model I am build a linear regression model and a decision tree model using sklearn. I want to compare the performance of these two models, I have calculated the r2_score for both the models. I have calculated the model.score for both the values. I am confused which is a better metric to compare the performance of these models. Also what does model.score gives?

from sklearn.metrics import r2_score
score_DT = r2_score(y_pred_DT,y_test)

dt_score = regressorDT.score(X_test,y_test)

Both functions are the same r2 metric and should produce the same results.

Your usage of the r2_score function is wrong. The first argument should be the ground truth values and not the predicted values, so in your case it should be:

score_DT = r2_score(y_test, y_pred_DT)


Setting activation function to a leaky relu in a Sequential model I'm doing a beginner's TensorFlow course, we are given a mini-project about predicting the MNIST data set (hand written digits) and we have to finish the code such that we get a 99% accuracy (measured as 0.01 loss) in less than 10 epochs.
In principle I am getting the accuracy, but the loss only reaches <0.01 at the 10th epoch (hence assignment is counted as failed).
As per instructions, I'm not allowed to change the model.compile arguments, so I decided I can try to change the activation function to a leaky relu, using the code I was given. It is not as straightforward as it seems and everything I found online does not seem to work. Most suggestions are in the model.add() format, which I can't figure out how to incorporate/substitute in the code they provided us, without changing it too much (everything failed).
The current code is given below:

  model = tf.keras.models.Sequential([
          keras.layers.Flatten(input_shape=(28,28)),
          keras.layers.Dense(128,activation=tf.nn.relu),
          keras.layers.Dense(10,activation=tf.nn.softmax)
      ])

Any help would be appreciated!

Try this, and tune alpha.

model = tf.keras.models.Sequential([
    keras.layers.Flatten(input_shape=(28,28)),
    keras.layers.Dense(128,activation=tf.keras.layers.LeakyReLU(alpha=0.3)),
    keras.layers.Dense(10,activation=tf.nn.softmax)
 ])


Problems with shape of Conv1D on Keras I have some problems with layers construction on Keras.
I explain the whole problem:

I have a feature matrix, with dimensions: 2023 (rows) x 65 (features);
I tried to build a CNN, with Conv1D as first layer;

My code is:
def cnn_model():
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae'])
    model.fit(X, Y, epochs=100, batch_size=64, verbose=0)
    model.evaluate(X, Y)
    return model

scoring = make_scorer(score_func=pearson)
# evaluate model with standardized dataset
estimation = []
estimation.append(('standardize', StandardScaler()))
estimation.append(('mlp', KerasRegressor(build_fn=cnn_model, epochs=50, batch_size=32, verbose=0)))
pipeline = Pipeline(estimation)
kfold = KFold(n_splits=10, shuffle=True, random_state=1)
results = cross_val_score(pipeline, X, Y, cv=kfold, scoring=scoring)

The problem is that, when it runs, I get the following error:
ValueError: Input 0 of layer sequential_9 is incompatible with the layer: : expected min_ndim=3, found ndim=2. Full shape received: (None, 64)

I really don't know why this error occurs. Probably it's just a problem with parameters passing, I'm quite new on this field. Can you tell me something more? I tried a bunch of things in order to solve this error but every time I get some new errors instead of solving. Thank you all.

Just tried your cnn_model function and it works correctly for me.
Not sure about what your problem is, but it seems to be your X and Y. X must not have shape (2023, 65), but shape (datasize, 2023, 65). and Y should have shape (datasize, 1) as well. otherwise your network treats X as 2023 inputs of shape (None, 65) instead of treating it as n inputs of shape (None, 2023, 65).
here is my code in case you wanna run it (important part is the definition of X(ims) and Y(lab) :
import keras
from keras.layers import Conv1D, Dropout, MaxPooling1D, Flatten, Dense
import numpy as np

ims = np.random.randn(100, 2023, 65)
lab = np.random.randn(100, 1)

def cnn_model(X, Y):
    model = keras.Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae'])
    print('training')
    model.fit(X, Y, epochs=10, batch_size=32, verbose=0)
    print('finished training')
    model.evaluate(X, Y)
    return model

m = cnn_model(ims, lab)
m.summary()


GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`) I've been trying to make these packages work for quite some time now but with no success. Basically the error is:

GraphViz's Executables not found

EDIT: I had not posted a terminal log with the error originally. I'm using Ubuntu now so I won't be able to reproduce the exact same error I got in the past (a year ago, so far away in the past...). However, I've been experiencing a similar --- if not the same --- error in my current setup; even while using a virtual environment with pipenv. The error seems to come from lines that were described in @张乾元's answer:

Traceback (most recent call last):
  File "example.py", line 49, in <module>
    Image(graph.create_png())
  File "/home/philippe/.local/lib/python3.6/site-packages/pydotplus/graphviz.py", line 1797, in <lambda>
    lambda f=frmt, prog=self.prog: self.create(format=f, prog=prog)
  File "/home/philippe/.local/lib/python3.6/site-packages/pydotplus/graphviz.py", line 1960, in create
    'GraphViz\'s executables not found')
pydotplus.graphviz.InvocationException: GraphViz's executables not found

I've tried to install GraphViz via 2 different ways: via pip install graphviz and through the .msi package (and also tried to install pydot, pydotplus and graphviz in many different orders).

The code I'm trying to run is simply a dot-to-png converter for the Iris Dataset.

from sklearn.tree import DecisionTreeClassifier
import sklearn.datasets as datasets
from sklearn.externals.six import StringIO
from sklearn.tree import export_graphviz

import pandas as pd
import pydotplus

from IPython.display import Image

iris = datasets.load_iris()
df = pd.DataFrame(iris.data, columns = iris.feature_names)
y = iris.target

dtree = DecisionTreeClassifier()
dtree.fit(df,y)

dot_data = StringIO()
export_graphviz(
    dtree,
    out_file = dot_data,
    filled = True,
    rounded = True,
    special_characters = True
)
graph_1 = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph_1.create_png())

In Jupyter Notebooks and in Atom, the system seems to be looking for GraphViz inside pydotplus, as it points to ~\Anaconda3\lib\site-packages\pydotplus\graphviz.py. Shouldn't it be the other way around?

Lastly, I just want to point out that I've already tried adding GraphViz's path to the system's PATH using C:\Users\Philippe\Anaconda3\Library\bin\graphviz.

See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that program) with the PATH environment variable settings. Installing particular versions, or in a particular order, or manually adding a PATH fixes the problem.

It's best if the Package sets the PATH correctly and removes it when you uninstall the Package (so you don't get too long a PATH - which usually won't happen). Setting it manually prevents future breakage and forces it to work, but you need to manually remove the extra text if you uninstall the Package.

Here's the advice from those three links:

pip install graphviz
conda install graphviz

or

You need to run

conda install python-graphviz

instead of

pip install graphviz

to get these bindings, which also work with conda's Graphviz package.

or

Download and install graphviz-2.38.msi (use the newest version) from
https://graphviz.gitlab.io/_pages/Download/Download_windows.html
Set the path variable

        (a) Control Panel > System and Security > System > Advanced System Settings > Environment Variables > Path > Edit

        (b) add 'C:\Program Files (x86)\Graphviz2.38\bin'


Incremental training and Auto Machine Learning for big datasets I built a NLP sentence classifier, which uses vectors from word embedding as features.

Training dataset is big (100k sentences).
Every sentence has 930 features.

I found the best model using an auto machine learning library (auto-sklearn); the training required 40 GB of RAM and 60 hours. The best model is an ensemble of the top N models found by this library.

Occasionally, I need to add some data to the training set and update the training.
Since this autoML library isn't suitable for incremental training, every time  I need to do complete retraining, using more and more memory and time.

How to address this issue? How to do incremental training? Should I quit the usage of this library? For memory and time usage, would it be better to parallelize the training?

First of all using auto-sklearn, you can use

automl.fit(X_train, y_train, dataset_name='X_train',
               feat_type=feature_types)

    print(automl.show_models())

so you can extract the instance of the best model from the first fitting. However in order to learn incrementally you have to (in case of sklearn models) have fit_partially method. Naive Bayes varaints and other algorithms here have this functionality. So you are out of luck if these are not in the output of show_models: In this case you ought to do your own automated ml targeted on fit_partial models.
Alternative is using spark it has some cool streaming (incremental learning algos)
StreamingKMeans, StreamingLinearRegressionWithSGD, StreamingLogisticRegressionWithSGD and generally StreamingLinearAlgorithm.
To conclude, I would not use auto-sklearn if these are your problems and choose some alternatives that do work parallel.


What does the co-ordinate output in the yolo algorithm represent? My question is similar to this topic.
I was watching this lecture on bounding box prediction by Andrew Ng when I started thinking about output of yolo algorithm.
Let's consider this example, We use 19x19 grids and only one receptive field with 2 classes, so our output will be => 19x19x1x5.
The last dimension(array of size 5) represents the following:

1) The class (0 or 1)
2) X-coordinate
3) Y-coordinate
4) height of the bounding box
5) Width of the bounding box

I don't understand whether X,Y coordinates represent the bounding box with respect to the size of entire image or just and receptive field(filter). In the video the bounding box is represented as a part of receptive field but logically receptive field is much smaller than bounding box and also people might tinker with filter size, so positioning bounding boxes with respect to filter makes no sense.

So, basically what does the coordinates of bounding boxes of an image represent ?

Simply the left high most vertex of the bounding box.

With X,Y and it's height and width you can find the bounding box in the image. (With respect to the size of entire image)


How to serialize/pickle a spacy ner model? I have trained a custom SpaCy named entity recognition model. I saved the model to disk using:
nlp.to_disk()

which results in keeping the model in a folder. Is it possible to make the nlp object to a pickle file?

Yes - Here is how to pickle in Python:
import pickle

pickle.dump(nlp, open( "nlp.p", "wb" ))


Weight-initialisation and model instability Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction.

What could be the causes of such a behavior ? What would be the solutions ?

*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.

As you have noticed weight initialization is extremely important, If for instance, you initialize your network with zeros, then it's guaranteed you will not learn from data. there are many ways to initialize your data:

Most simple approach (beginner approach) :
just use random initialization

Xavier Initialization:
this might be a more intermediate approach if you are interested on this check the following link:
https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79

I'd say that you gotta keep in mind when your network is "learning" it is basically navigating in a hyper dimension. most people do not actually standardize data. Standardizing data is essential since you make learning easier for your algorithm.

So short answer: standardize your data, your algorithm will converge faster into a minimum (of course not always global).


Train a deep reinforcement learning model using two computers I would like to know if there is a way to train a deep rl model using two different computers. The first one would execute the game and send requisitions to the second computer which would store and train the model itself.

Obs: The computers aren't in the same LAN.

Thanks!!

Distributed RL is very much a thing. Google have created a distributed setup called IMPALA for this, and there are multiple instances of A3C, PPO etc available if you search. I don't know much about IMPALA, but the basic idea of the scalable policy gradient methods is to run multiple environments, collecting gradients on each server, then collating them together to create improved policy and value networks every few steps.

There are a couple of variations in strategy based on which stage of the data is shared - observations or gradients. Gradient calculation is CPU intensive, so above a certain scale it is worth having that occur on the distributed devices, depending on how intensive it is to collect experience in the first place.

  Obs: The computers aren't in the same LAN.

This may prevent you implementing anything with low-level observation or gradient sharing, unless the bandwidth between the machines is high.

The simplest way to use two computers in this case is to perform basic hyper-parameter searches by running different tests on each computer and tracking which computer has done which experiments.

  The first one would execute the game and send requisitions to the second computer which would store and train the model itself

This could work with an off-policy value-based method, such as DQN. You still need a reasonable bandwidth between the two machines, especially if the observation space is large. DQN is a reasonable choice since you don't need the environment-running machine to follow the current optimal policy - although you will still want to update the policy on the first computer some of the time.

The basic algorithms of DQN do not require much changing to support this kind of distribution. Just comment out or make conditional a few parts:

On the first machine:

Comment out or logically block sampling and learning from experience table
Maintain a "behaviour-driving" Q-value network instead of the learning network, in order to run $\epsilon$-greedy policy
Send experience to second machine instead of storing it in local experience-replay table (this is the bandwidth-intensive part)
Asynchronously receive updates to behaviour-driving Q-value network

On the second machine:

Comment out or logically block interactions with the environment
Asynchronously receive experience from first machine and add to experience-replay table
Every so many mini-batches, send updated current network to second machine

You will need to have some experience handling multi-threading or multi-process code in order to cover the asynchronous nature of the updates. If that seems too hard, then you could have the two machines attempt to update each other synchronously, or via a queueing mechanism. The disadvantage of that is one or other machine could end up idle waiting for its partner to complete its part of the work.


Dealing with zeros when plotting log-scaled data I have a non-negative variable and I'd like to plot it, log-scaled
I'm trying to understand how to deal with 0-values. One naive idea I had in mind is just to add 1 to all values (or some very low number greater than 1)
What other options are available?
Thanks

Your suggestion is a valid one, encoding variables with a known outcome once the scaling is applied. Log(1) will become zero, so just keep that in mind for your next stage. You can use clip or replace for this:
df.clip(1, df.max())

or try replacing with a NaN
df.replace(0, np.nan)

Alternatively you could do one of the following:

Drop the zero value rows e.g. df = df[df['column'] !=0] but then you lose some data.
Fill the zero values with a statistically representative value (i.e. interpolation). You can explore the Pandas interpolate method here.

Which ever method you decide upon depends on your use-case, and compatibility with the plotting function you use.


Dropping one category for regularized linear models While reviewing the sklearn's OneHotEncoder documentation (attached below) I noticed that when applying regularization (e.g., lasso, ridge, etc.) it is not recommended to drop the first category. While I understand why dropped the first category prevents collinearity, I am unsure why it is needed for regularized regression. Wouldn't this this add an additional dimension that will need to be regularized?

drop{‘first’, ‘if_binary’}

Specifies a methodology to use to drop one of the categories per
feature. This is useful in situations where perfectly collinear
features cause problems, such as when feeding the resulting data into
a neural network or an unregularized regression.
However, dropping one category breaks the symmetry of the original
representation and can therefore induce a bias in downstream models,
for instance for penalized linear classification or regression models.

When you do linear regression you have to leave out one column as it's a singular matrix and hence columns are linearly dependent and we cannot calculate the inverse.
But when you do regularization it take cares of singularity. The matrix is almost surely nonsingular. Hence we don't need to drop a column and if you drop different columns from each feature it could lead to different prediction as it would lead to bias.
Refer this.


Clustering list of list of integers I have ~100 sets of samples with integer IDs. For example, 3 of them could be:

a = [0, 1, 3, 4, 6...]
b = [1, 5, 9, 102...]
c = [1, 7, 10, 42...]

I am looking to cluster/group together these sets such that within each cluster, all the elements have at least X% common IDs with each other, where X is an input parameter.

I was thinking about using Agglomerative Clustering with 1 - %X as the distance metric, but was unsure how to modify it to account for each clusters 'information' being the common set of IDs between sets within it.
Any advice would be appreciated (including a different technique/algorithm clustering was just what came to mind)

Welcome to the community!

I would use the Mutual Information between two sets as the criterion of similarity. In your special case, set the mutual information of those sets whose intersection is less than X% to $0$. Run your clustering and see what comes out. With the same similarity matrix, you can also construct a weighted graph and cluster nodes (which is called Community Detection). This link gives you a proper overview starting from slide 37 and in slide 48 you find your own idea! If you have more questions about Community Detection, e.g. implementation, please drop a comment here.


Clustering algorithm which does not require to tell the number of clusters I have a dataframe with 2 columns of numerical values. I want to apply a clustering algorithm to put all the entries into the same group, which have a relatively small distance to the other entries. But which clustering algorithm can I use, although I do not know how many groups will be formed? It would be ideal if there is a parameter to determine the maximum distance allowed. And if there isn't such an algorithm, maybe it would be really helpful to come up with some intuitions, how such an algorithm can be implemented by myself. Thanks a lot!! :)

The data could look like this:

a,b
20,30
19,31
10,10
9,8
12,11
31,11
32,11

I would try DBSCAN algorithm first: fairly easy to tune (with, in particular, a notion of distance as you requested), and does not need to know the number of clusters.

There are a few other algorithms that can help you decide the number of clusters: Bayesian Gaussian Mixtures (see sklearn implementation) for instance, but it requires a bit more knowledge and work. There is also spectral clustering, but for this one, sklearn does not automatically find the number of clusters, so you will have to do create your own implementation and determine the number yourself, manually, by plotting the eigenvalues.


Machine Learning Steps Which of the below set of steps options is the correct one when creating a predictive model?

Option 1:

First eliminate the most obviously bad predictors, and preprocess the remaining if needed, then train various models with cross-validation, pick the few best ones, identify the top predictors each one has used, then retrain those models with those predictors only and evaluate accuracy again with cross-validation, then pick the best one and train it on the full training set using its key predictors and then use it to predict the test set.

Option 2:

First eliminate the most obviously bad predictors, then preprocess the remaining if needed, then use a feature selection technique like recursive feature selection (eg. RFE with rf ) with cross-validation for example to identify the ideal number of key predictors and what these predictors are, then train different model types with cross-validation and see which one gives the best accuracy with those top predictors identified earlier. Then train the best one of those models again with those predictors on the full training set and then use it to predict the test set.

I found both of your options slightly faulty. So, this is generally (very broadly) how a predictive modelling workflow looks like:

Data Cleaning: Takes the most time, but every second spent here is worth it. The cleaner your data gets through this step, the lesser would your total time spent would be.
Splitting the data set: The data set would be splitted into training and testing sets, which would be used for the modelling and prediction purposes respectively. In addition, an additional split as a cross-validation set would also need to be done.
Transformation and Reduction: Involves processes like transformations, mean and median scaling, etc.
Feature Selection: This can be done in a lot of ways like threshold selection, subset selection, etc.
Designing predictive model: Design the predictive model on the training data depending on the features you have at hand.
Cross Validation:
Final Prediction, Validation


How to Impute Top-Censored Data What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?

You deal with a highly skewed censored distribution which makes it really hard to get good estimates. The key question is what information is available to model the skewed income distribution (you don't say anything on that).
There is quite some literature on the issue of imputing top incomes. E.g. "Measuring inequality using censored data: a multiple‐imputation approach to estimation and inference" by Stephen P. Jenkins  Richard V. Burkhauser  Shuaizhang Feng  Jeff Larrimore. The abstract kind of nails the problem:

To measure income inequality with right‐censored (top‐coded) data, we
propose multiple‐imputation methods for estimation and inference.
Censored observations are multiply imputed using draws from a flexible
parametric model fitted to the censored distribution, yielding a
partially synthetic data set from which point and variance estimates
can be derived using complete‐data methods and appropriate combination
formulae. The methods are illustrated using US Current Population
Survey data and the generalized beta of the second kind distribution
as the imputation model. With Current Population Survey internal data,
we find few statistically significant differences in income inequality
for pairs of years between 1995 and 2004. We also show that using
Current Population Survey public use data with cell mean imputations
may lead to incorrect inferences. Multiply‐imputed public use data
provide an intermediate solution.

Also see "Recent Trends in Top Income Shares in the United States: Reconciling Estimates from March CPS and IRS Tax Return Data" to get an idea how to model a censored right-skewed income distribution in a proper way.


Why my regression model always be dominanted by one feature? I am working on a financial predict problem. which means it is a time series prediction problem.
I have three features, which have high correlation(each two's corr is about 0.6)
And I do the linear regression fit.
I assume that the coefficient should be similiar among these three features, but i get a coefficient vector
like this:
[0.01, 0.15, 0.01]
which means the second features have the biggest coff(features are normalized), and it can dominant the prediction result.
I dont know why. I think adding weak features can boost my prediction model, but i think the second feature is dominant in my model, and other features are worthless.
Why one of features can be dominant in the model, did I miss something?

The first thing that comes to my mind is that you might have not normalized your features correctly. Generally a feature ranging between a bigger range of values, compared to the other ones, is going to be more influencial in terms of the models' output.
In order to midigate this issue, one common practise is to transform your features into having zero-mean and a variance of one. In this way, it is guaranteed that all your features have identical range of values.
Other than that, It may be just that the dominant feature is indeed more indicative for your time series prediction and thus your model has learned to rely its predictions on this specific feature.


Neural networks - adjusting weights My question would be about backpropagation and understanding the terms feedforward NN vs backpropagation. I have two questions really:

If I understand correctly, even a feedforward network updates its weights (via a delta rule for example). Isn't this also backpropagation? You have some random weights, run the data through the network, then cross-validate it with the desired output, then update the rules. This is backpropagation, right? So what's the difference between FFW NN and RNN? If you can't backpropagate on the other hand, how do you update the weights in a FFW NN?
I've seen NN architectures looking like this:

Basically all the neurons are being fed the same data, right? OK, the weights are randomized and thus different in the beginning, but how do you make sure Temp. Value #1 will be different from Temp. Value #2, if you use the same update rule?

Thank you!

I'm not an expert on the backpropagation algorithm, however I can explain something. Every neural network can update it's weights. It may do this in different ways, but it can. This is called backpropagation, regardless of the network architecture.

A feed forward network is a regular network, as seen in your picture. A value is received by a neuron, then passed on to the next one.

A recurrent neural network is almost the same as a FFN, the difference being that the RNN has some connections point 'backwards'. E.g. a neuron is connecteded to a neuron that has already done his 'job' during backpropagation. Because of this, the activations of the previous output have an effect on the new output.

On question #2

Interesting question. This has to do with weight initialization. Yes, you're right, each neuron in the hidden layer accepts the same connections. However, during the initalization process, they have received a random weight. Depending on your NN libary, the neurons might also have been initialized with a random bias.

So even though the same rule is applied, each neuron has different outcomes as all it's connections have different weights than the other neurons weights.

On your comment: just because all the neurons happen to have the same backpropagation function, doesn't mean they will end up with the same weights.

As they are initialized with random weights, each neurons error is different. Thus they have a different gradient, and will get new weights.

You also have to keep in mind that for a certain output to be reached, there are multiple solutions (due to non-linearity). So due to initialized random weights, one neuron might be close to a certain solution while another neuron is closer to the other.

Additionally, as was stated in the comments, a network works as a whole. The output neuron is also non-linear, and for most test cases, the output should be non-linear and the output neuron most likely requires that the hidden neurons activate at different input values.


Tensorflow / Deepmind: how do I take actions from observations for math algorithms related to proofs? Crossposted from here
This question is to ask for directions/suggestions/help on the use of deepmind opensource libraries: DeepMind Lab or TensorFlow in Python.
Consider that I'm new to concepts like deep learning and AI.
Questions are:

Are there examples on using Deepmind or Tensorflow for math problems where I need to observe values and take actions?

Using an approach similar to to the one described in this page based on observations, actions, rewards, etc., I'd like to call a learing agent to choose among some values. I was thinking to something like this:

Input: a list of list of tuple (the list will change at each step)
Action: pick up a value from the input (based on experience)
Reward: If the value it returned was good or bad for the rest of the algorithm I'm implementing, I'll reward the deep learning agent.

Additional notes:

I cannot train the algorith in advance

The Input is something like this:
edge: (1, 2), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (0, 1), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (5, 4), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (6, 7), face_down: 3, face_up: 5, face_left: 5, face_right: 5
edge: (3, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (4, 1), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (8, 5), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (3, 8), face_down: 4, face_up: 5, face_left: 4, face_right: 5
edge: (2, 3), face_down: 4, face_up: 5, face_left: 5, face_right: 4
edge: (5, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (0, 5), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (1, 0), face_down: 4, face_up: 4, face_left: 5, face_right: 4
edge: (9, 6), face_down: 3, face_up: 5, face_left: 5, face_right: 5
edge: (0, 3), face_down: 4, face_up: 4, face_left: 4, face_right: 5
edge: (7, 9), face_down: 3, face_up: 5, face_left: 5, face_right: 5

The idea is to use the same aproach deepmind uses to play games, but instead of analyzing pixels and use the pad (up, dowm, left, right, fire, jump), is to let the learning agent to analyze some math values and, as the only action, to select one of them.
Are there other approaches or libraries/framework to address such a problem?

I would recommend you to have a look at OpenAI gym.
It will be easier for you to implement a new environment describing your problem. Regarding the agents, there are many implementations around that are compatible with OpenAI.


Why does my model learn with Ragged Tensors but not Dense Tensors? I have a string of letters that follow a "grammar." I also have boolean labels on my training set of whether the string follows "the grammar" or not. Basically, my model is trying to learn determine if a string of letters follows the rules. It's a fairly simple problem (I got it out of a textbook).
I am generating my dataset like this:
def generate_dataset(size):
    good_strings = [string_to_ids(generate_string(embedded_reber_grammar))
                    for _ in range(size // 2)]
    bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))
                   for _ in range(size - size // 2)]
    all_strings = good_strings + bad_strings
    X = tf.ragged.constant(all_strings, ragged_rank=1)

    # X = X.to_tensor(default_value=0)

    y = np.array([[1.] for _ in range(len(good_strings))] +
                 [[0.] for _ in range(len(bad_strings))])
    return X, y

Notice the line X = X.to_tensor(default_value=0). If this line is commented out, my model learns just fine. However, if it is not commented out, it fails to learn and the validation set performs the same as chance (50-50).
Here is my actual model:
np.random.seed(42)
tf.random.set_seed(42)

embedding_size = 5

model = keras.models.Sequential([
    keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),
    keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS) + 1, output_dim=embedding_size),
    keras.layers.GRU(30),
    keras.layers.Dense(1, activation="sigmoid")
])
optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)
model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(X_train, y_train, epochs=5, validation_data=(X_valid, y_valid))

I am using 0 as the default value for the dense tensors. The strings_to_ids doesn't use 0 for any of the values but instead starts at 1. Also, when I switch to using a Dense tensor I change ragged=True to False. I have no idea why using a dense tensor causes the model to fail, as I've used dense tensors before in similar exercises.
For additional details, see the solution from the book (exercise 8) or my own colab notebook. As a disclaimer: I am studying ML for fun, not for work or school.

So turns out the answer was that the shape of the dense tensor was different across the training set and validation set. This was because the longest sequence differed in length between the two sets (same with the test set).


CNN tagging such that each input could have multiple tags Thanks in advance for reading my question!
I've been using CNNs to classify text using Keras and TF. My data is strings "I read the news" or "I read machine learning news" and my labels are tags: Data Science, Reporter, Child...
My issues is that each text can have multiple labels attached to it. How should I construct my target such that I can capture each of those multiple outputs?
Description, Tag
"I read the news", Child
"I read the news", Reporter
"I read machine learning news", Data Science
"I read machine learning news", Reporter

CNN's (generally) use softmax as activation function at the last layer, which gives a probability distribution over all possible labels. You use some kind of loss function to optimize the CNN to generate "right" label for a novel sample.

Loss function used in such case is cross-entropy loss or KL-divergence which measures how close two distributions are. For multiclass classification (different from your case, where each input example can have only one label), desired distribution is (0, 0, .... , 1, ..., 0) where 1 appears at the position of actual label. Loss function penalizes the distribution which is "far" from this distribution.

For your case you can encode your output as (0, 0, 1, 0, ... 1, ..,) so on where you put ones at positions of the labels which are associated with your input and use KL-divergence as loss function and optimize your CNN to  minimize the loss.


Frequency/Count encoding How do I perform frequency/count encoding for a train and test set?

The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no specific train, and test encoding transformation). For instance:

dataset.groupby("cat_column").size()/len(dataset)

In my case now I have a train, and test set.

[First option] Is it okay (due to leakage? or there won't?) for me to use frequency encoding on the whole dataset. OR

[Second option] I should take into consideration train, and test set independence.

If the second option, how do I do this?

Encode the train set, then use the encoding values of categories in the train set for the same categories in the test set. Categories not represented in the test set would be need to be handled. OR
There's a better generic implementation?

I hope this is not a late answer but actually you can use category_encoders library, it follows sklearn's style.

Example:

import category_encoders as ce

#I'll pretend that you've already split your data into train/test

#your categorical features
cat_features = ['cat_feature1', 'cat_feature2']

#count encoder
count_encoder = ce.CountEncoder(cols=cat_features)
count_encoder.fit(train[cat_features])

train = train.join(count_encoder.transform(train[cols]).add_suffix('_count'))
test = test.join(count_encoder.transform(test[cols]).add_suffix('_count'))


Generalization of Correlation Coefficient The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the general predictive power of x over y when the relationship between x and y is not linear?

I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.

To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\dots,X_d)$. Also let $X_{ij}$ be the $i$th observation for variable $j$. These measures are called rank correlations because they can be computed using the ranks only. What I mean is that if you sort all $X_{ij}$, $i=1,\dots,n$, and replace the biggest observation by $n$, the second biggest by $n-1$, and so on (do that for all columns $j$) and call you new observations $R_{ij}$, then

the empirical Spearman's $\rho$ (matrix) is simply the Pearson linear correlation (matrix) of $(R_1,\dots,R_d)$; and
the empirical (pairwise) Kendall's $\tau$ between $X_{i_1}$ and $X_{i_2}$ is the probability of concordance minus the probability of discordance between two iid observations, say $(X_{1 i_1},X_{1 i_2})$ and $(X_{2 i_1},X_{2 i_2})$, which  can equivalently be computed from the ranks $(R_{1 i_1},R_{1 i_2})$ and $(R_{2 i_1},R_{2 i_2})$ instead.

A rank correlation between $X_{i_1}$ and $X_{i_2}$ of one indeed means perfect concordance (i.e. $X_{i_1}$ always increases with $X_{i_2}$), but that does not necessarily means they are linearly related. The ranks are linearly related.

Just to make the concept of concordance clearer, here the (bivariate) observations are all concordant

and here they are all discordant

So that when you consider a cloud of points you have some pairs that are concordant and others that are discordant.

Note that this answer provides examples, but there are many other ways to approach the question. As commented by Emre, information-theoretic measures are also an option.


Is a test set necessary after cross validation on training set? I'd like to cite a paragraph from the book Hands On Machine Learning with Scikit Learn and TensorFlow by Aurelien Geron regarding evaluating on a final test set after hyperparameter tuning on the training set using k-fold cross validation:

"The performance will usually be slightly worse than what you measured
using cross validation if you did a lot of hyperparameter tuning
(because your system ends up fine-tuned to perform well on the
validation data, and will likely not perform as well on unknown
datasets). It is not the case in this example, but when this happens
you must resist the temptation to tweak the hyperparameters to make
the numbers look good on the test set; the improvements would be
unlikely to generalize to new data."
-Chapter 2: End-to-End Machine Learning Project

I am confused because he said that when the test score is WORSE the cross validation score (on the training set), you should not tweak the hyperparameters to make the testing score better. But isn't that the purpose of having a final test set? What's the use of evaluating a final test set if you can't tweak your hyperparameters if the test score is worse?

In "The Elements of Statistical Learning" by Hastie et al the authors describe two tasks regarding model performance measurement:

Model selection: estimating the performance of different models in order to choose the best one.
Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

Validation with CV (or a seperate validation set) is used for model selection and a test set is usually used for model assessment. If you did not do model assessment seperately you would most likely overestimate the performance of your model on unseen data.


Why does PAC learning focus on learnability of the hypothesis class and not the target function? The definition of PAC learning is roughly

An algorithm is a PAC learning algorithm if it given enough data, for any target function, it asymptotically does as well as it possibly could given the functions it's capable of representing.

This definition seems sort of unambitious. In reality, I care more about approximating the target function well in an absolute sense, not just approximating it as well as my hypothesis class can muster. By some kind of no-free-lunch principle, it's probably not possible to be able to do this for all target functions, so we need to be satisfied with algorithms that learn well on a class of target functions. If I were to take a shot at defining a notion of "learning", it would look like this:

An algorithm learns a class $\mathcal V$ of random variables in $\mathcal X\times\mathbb R$ iff for any $(X, Y)\in\mathcal V$, then for any $\delta, \epsilon$, given enough i.i.d. samples of $(X, Y)$, the algorithm returns a function $h$ such that $P(|h(X)-Y|\leq\epsilon)>1-\delta$.
A class $\mathcal V$ of random variables is learnable iff there exists an algorithm that learns it.

I have two questions:

Why the emphasis on getting good bounds for approximating the hypothesis class, rather than the target function/distribution, when surely it's the latter we care about?
Is something like my definition also studied?

Fair warning, this is just an intuition and I'm not really expert in this kind of question. Nice question anyway :)
Theoretical models of learning like PAC are meant to be used to prove learnability results. Therefore what matters is not only that the definition corresponds to the intuition of what "learning" means, but also that it's actually technically possible to prove anything with this definition. I suspect that this is why (or one of the reasons why) the PAC definition is constrained to the particular class of functions that the algorithm deals with, since it makes it possible to theoretically determine what is the true best function in this class. This is needed to prove (or disprove) that the algorithm can always return it.
I also suspect that in the universe of all possible functions, the optimal function could always be some kind of infinite non-generalizing function which assigns exactly the true $Y$ for any $X$ (at least to the extent possible, i.e. when $X\neq X' \Rightarrow Y\neq Y'$). In this perspective restricting the class of target functions forces the algorithm to generalize, since it must represent the data in the finite space of the parameters of the class of functions.
This could explain why the definition proposed by OP is not usable:
if I'm not mistaken, there's simply no way to prove any learnability result with it.


Annotating the vocabulary using Word2vec model I am trying to label the vocabulary in the corpus.

I have trained the word2vec model on the corpus
I have grouped the words which are related based on the score as key as the first word as the key and remaining words as a list of 2-tuple of word and scores with respect to the key

example:
'coffee'---key
values are

[('tea', 0.8139282),
 ('latte', 0.76456803),
 ('coffe', 0.7607962),
 ('lattes', 0.756057),
 ('starbucks', 0.7158153),
 ('espresso', 0.71386236),
 ('mocha', 0.69999266),
 ('coffees', 0.6816252),
 ('frappucino', 0.67192864),
 ('cuppa', 0.66720986),
 ('cappucino', 0.6664002),
 ('chai', 0.6623157),
 ('decaf', 0.65980726),
 ('frappuccino', 0.65150374),
 ('venti', 0.6486204),
 ('expresso', 0.6369579),
 ('macchiato', 0.6280453),
 ('scone', 0.62476856),
 ('sippy', 0.6236704),
 ('cappuccino', 0.61718297),
 ('iced', 0.6130485),
 ('hazelnut', 0.6023698),
 ('mug', 0.6004759),
'
'
'
'
'

as i  know the coffee is releated to latte ,green_tea ,espresso,starbucks.. from the above data
I would like to label each word as below

latte [COHYPO] green_tea [COHYPO] espresso [HYPO] Starbucks [RELATED] tim_horton [RELATED]

COHYPO-https://en.wiktionary.org/wiki/cohyponym

[HYPO] -https://en.wiktionary.org/wiki/hyponyme

[RELATED] -the word is repeated

[MORPHO]-Morphological variant (example :Computer and computers )

[Partof]- indicates that the annotated word is a part of the word of interest

Any suggestion or ideas by which I can approach this problem

WordNet is certainly an interesting resource to explore for this task. It might not cover all your vocabulary but I can't think of any other way to capture fine-grained semantic relationship between words.


Should I relabel this data or remove the potentially leaky feature? Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.

When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.

Features:

1) current point is over threshold - 1/0
2) current point is AM - 1/0
3) current day is weekday - 1/0
4) current percentage - 0.0-1.0
5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0

Label:

1 if a point is over threshold at any point in time after now and before now+15 minutes

Features 5-9 are intended to capture the inertia of the current percentage.

From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?

Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).

Actual / Predicted   Alarm goes off      Alarm did not go off
State changed        True positive       False positive
State stayed         False negative      True negative

Note the true negatives are useless - we don't care if the state stays in the same spot (below/above threshold). Therefore, the accuracy is NOT important for this system. Do not try to minimize the wrong metric. This extends to "do not use ROC/AUC" (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).

We also would like to deal with a huge "lag", trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.


Ethically and Cost-effectively Scaling Data Scrapes Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models.

For instance, the Data Science Toolkit (or RDSTK for R programmers) allows me to pull lots of good location-based data using IP's or addresses and the tm.webmining.plugin for R's tm package makes scraping financial and news data straightfoward. When going beyond such (semi-) structured data I tend to use XPath.

However, I'm constantly getting throttled by limits on the number of queries you're allowed to make. I think Google limits me to about 50,000 requests per 24 hours, which is a problem for Big Data.

From a technical perspective getting around these limits is easy -- just switch IP addresses and purge other identifiers from your environment. However, this presents both ethical and financial concerns (I think?).

Is there a solution that I'm overlooking?

For many APIs (most I've seen) ratelimiting is a function of your API Key or OAuth credentials. (Google, Twitter, NOAA, Yahoo, Facebook, etc.) The good news is you won't need to spoof your IP, you just need to swap out credentials as they hit there rate limit.

A bit of shameless self promotion here but I wrote a python package specifically for handling this problem.

https://github.com/rawkintrevo/angemilner

https://pypi.python.org/pypi/angemilner/0.2.0

It requires a mongodb daemon and basically you make a page for each one of your keys. So you have 4 email addresses each with a separate key assigned. When you load the key in you specify the maximum calls per day and minimum time between uses.

Load keys:

from angemilner import APIKeyLibrarian
l= APIKeyLibrarian()
l.new_api_key("your_assigned_key1", 'noaa', 1000, .2)
l.new_api_key("your_assigned_key2", 'noaa', 1000, .2)

Then when you run your scraper for instance the NOAA api:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'
payload= {  'limit': 1000,
        'datasetid':  'GHCND',
        'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': 'your_assigned_key'})

becomes:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'
payload= {  'limit': 1000,
            'datasetid':  'GHCND',
            'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': l.check_out_api_key('noaa')['key']})

so if you have 5 keys, l.check_out_api_key returns the key that has the least uses and waits until enough time has elapsed for it to be used again.

Finally to see how often your keys have been used / remaining useage available:

pprint(l.summary())

I didn't write this for R because most scraping is done in python (most of MY scraping).  It could be easily ported.

Thats how you can technically get around rate limiting.  Ethically ...

UPDATE The example uses Google Places API here


Weighting influence of two neural networks on classification I'm training a model that has two neural networks. One of them is a resnet18 CNN which has as it's input images. The other one is a small one hidden layer network that has as it's input four other variables.

At this moment I concatenate the outputs of these networks in the first (and only) fully connected layer, after which datapoints are classified into three classes in the classification layer.

I was wondering how I could design my network in such a way that the two different neural networks have a specific amount of influence on the classification. For instance, I'd like the CNN to have 60% 'say'/influence in the eventual classification, and have the other NN have 40% 'say'/influence.

I can't find an intuitive way to to this in the current structure. The only way I can imagine doing it is by separating the networks, and then weighting the losses of both the networks. Does anybody know another way to achieve this without separating the networks?

As neural networks are non-parametric, it would usually be the case that you train the combined model (as you describe it, but perhaps with additional FC-layers at the end) and let the model as a single entity learn/decide how best to combine the output of your two sub-models.

If you want to the hard-coded percentages you mention, then the most straight-forward way would be to do what you mention in the last paragraph.

People often train the same (or slightly differing models) independently on the same dataset, then average the predictions of these models on the test set, which can improve overall performance by integer percentages.


Does Convolution kernel size affect number of channels? I am going through Dilated Residual Network blog post. In this, Under 2.Multi-scale Context aggregation heading, author mentioned this.

  The last one is the 1×1 convolutions for mapping the number of
  channels to be the same as the input one. Therefore, the input and the
  output has the same number of channels. And it can be inserted to
  different kinds of convolutional neural networks.

I thought, we decide number of channels in the next layer and kernels will be initialized randomly. These kernels shape is decided by us which are 1x1 or 3x3 etc., So, what did author mean when he said, 1x1 convolutions for mapping the number of channels to be the same as the input one.When, Even if its 2x2 convolutional kernel, number of channels are not changed.

Normally a convolutional neural network will get flattened into a single column vector after the convolutions and then maybe be processed by dense layer. In this model, the convolution $1\times1$ is used as an output layer. It will have $C$ channels like every other layer, but it is not dilatated. Hence, you can use this layer as the input to other convolutional neural networks.

The kernel size will not influence the channels. Imagine an RGB image with 4 by 4 pixels. If we have a $2\times 2$ convolution with $2\times 2$ stride we will get an output of dimension $3\times 2 \times 2$ (without padding). Hence the channels do not change. If we have $K$ filters we will get $K$ times a $3\times 2 \times 2$ output. If the kernel size is $4\times 4$ with a stride of $2\times 2$ we will get a $3\times 1 \times 1$ (without padding) output for each of the $K$ filters. The kernel size only influences how large the receptive field of the convolutions are. Hence, it only influences how the layer is scaling the individual dimensions of the width and height (by using RGB images as an example).

If you flatten the output of a layer you will always reduce its dimensionality to $1$. In the dilated convolutional neural network they do not have a layer that flattens the input.


Converting a string to dummy encoded variables Here's the data

PlayerID, Characters, Win or Lose

I can make it look like this

8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1, 0

Or like this

2GRG822L9,"barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,",1

The second column is an 8 character combination from 70+ n characters.

I need to encode the variables to be dummy variables, so each character gets its own column. Is there a way to do this in python/R? I'm assuming you have to leave the second column as a string rather than outputting a csv file that looks like this.

2GRG822L9,barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,1
8PYPY0LLQ,valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,0

It should probably look like this before dummy encoding (I can get rid of commas within the string)

2GRG822L9,"barbarians8, valkyrie5, chr_balloon3, fire_spirits8, minion8, firespirit_hut6, rage4, skeleton_horde3,",1
8PYPY0LLQ,"valkyrie5 ,  chr_witch4 ,  hog_rider5 ,  zapMachine1 ,  mega_minion3 ,  baby_dragon2 ,  bomber7 ,  skeleton_horde1,",0

It is simple with python:

from pandas import DataFrame
data = [('2GRG822L9',"barbarians8,valkyrie5,chr_balloon3,fire_spirits8,minion8,firespirit_hut6,rage4,skeleton_horde3",1), ('8PYPY0LLQ',"valkyrie5,chr_witch4,hog_rider5,zapMachine1,mega_minion3,baby_dragon2,bomber7,skeleton_horde1",0)]
df = DataFrame.from_records(data,columns=('PlayerID', 'Characters', 'Result'))
df = df.drop('Characters', 1).join(df.Characters.str.get_dummies(','))

The result:

    PlayerID  Result  baby_dragon2  barbarians8  bomber7  chr_balloon3  \
0  2GRG822L9       1             0            1        0             1
1  8PYPY0LLQ       0             1            0        1             0

   chr_witch4  fire_spirits8  firespirit_hut6  hog_rider5  mega_minion3  \
0           0              1                1           0             0
1           1              0                0           1             1

   minion8  rage4  skeleton_horde1  skeleton_horde3  valkyrie5  zapMachine1
0        1      1                0                1          1            0
1        0      0                1                0          1            1


how to transform a folder of images into csv file? I have a data set for segmentation project. The dataset have 2 folders containing train folder and annotated_train_data folder, both folders have images. input and label both are images. I want to make a csv file of this dataset to feed into a neural network. what is the easiest and efficient way to make csv file?

./images
./images/train
./images/train/label1.jpg
./images/train/label2.jpg
./images/train/label3.jpg
./images/annotated_train_data
./images/annotated_train_data/label1.jpg
./images/annotated_train_data/label2.jpg
./images/annotated_train_data/label3.jpg

import os
import pandas as pd

BASE_DIR = 'images/'
train_folder = BASE_DIR+'train/'
train_annotation = BASE_DIR+'annotated_train_data/'

files_in_train = sorted(os.listdir(train_folder))
files_in_annotated = sorted(os.listdir(train_annotation))

images=[i for i in files_in_train if i in files_in_annotated]

df = pd.DataFrame()
df['images']=[train_folder+str(x) for x in images]
df['labels']=[train_annotation+str(x) for x in images]

pd.to_csv('files_path.csv', header=None)

Try this.


Why the RNN has input shape error? My x_train shape is (798,3) and y_train input shape is (798, 1). I am creating a RNN like this

def create_rnn_model():
    model = Sequential()
    model.add(SimpleRNN(20,return_sequences=False,stateful=stateful,activation='relu',batch_input_shape=(1,3,1)))
    model.add(Activation('relu'))
    adam = optimizers.Adam(lr=0.001)
    model.compile(loss='mean_squared_error', optimizer=adam, metrics=[root_mean_squared_error])

    return model

But this returns the error

  ValueError: Error when checking input: expected simple_rnn_1_input to
  have 3 dimensions, but got array with shape (798, 3)

My batch size =1 and my timestep is 3 and dat_dim=1 .Then where am I doing it wrong?
Any help is appreciated.

  EDIT

I changed my x_train to shape (798,3,1) and y_train shape to (798,) and ran the model but it threw an error

ValueError: Error when checking target: expected activation_1 to have shape (20,) but got array with shape (1,)

But I can run the model with 1 unit. How do I specify the model to run with 20 units instead of one.

First for RNN part, you should consider to read documentation about input_shape and batch_input_shape.

Then your Activation layer seems to be useless because you already apply relu activation in RNN layer.

Finally, your network returns output's shape of (20,) (RNN return (20,) and activation don't change shape) and you compare this with y_train with shape of (1,). Shapes dismatch so it return this error

EDIT: You can print model.summary() to monitor input/output 's shape


When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it? I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.

This completely changed my perspective of how I work with data. Not only should I clean the data, but the data should also be formatted correctly. Seems pretty obvious when you think about it, but that is besides the point.

I decided to start applying those principles into my data cleaning workflow, however, I was wondering whether there are times where having tidy data would not be ideal?

When would we not want to "tidy our data"?

Ideally when should we use Tidy Data and when should we avoid it?

Your input to this would be highly appreciated.

In general tidy data is great... but it can quickly become unreasonably large. This is the main reason why I usually try to refactor my data in a tidy format as late as possible in the process.

Example: imagine a dataset containing $N$ instances, with columns feature1 ... featureX and result1... resultY, where the result? columns represent some value based on several methods/parameters. The tidy version would have columns feature1 ... featureX plus method (which takes as values result1,..,resultsY) and of course result for the result value. The tidy version would contain $N \times Y$ instances, i.e. for each instance it repeats the values of the $X$ features $Y$ times. If $X$ is large the dataset is going to be very large in memory (and also stored as a file).

Naturally this gets even worse when you have for example several steps involving various parameters, multiplying the instances every time is not always feasible.


Why TensorFlow can't fit simple linear model if I am minimizing absolute mean error instead of the mean squared error? In Introduction I have just changed

loss = tf.reduce_mean(tf.square(y - y_data))

to

loss = tf.reduce_mean(tf.abs(y - y_data))

and model is unable to learn the loss just became bigger with time. Why?

I tried this and got same result.

It is because the gradient of .abs is harder for a simple optimiser to follow to the minima, unlike squared difference where gradient approaches zero slowly, the gradient of the absolute difference has a fixed magnitude which abruptly reverses, which tends to make the optimiser oscillate around the minimum point. Basic gradient descent is very sensitive to magnitude of the gradient, and to the learning rate, which is essentially just a multiplier of the gradient for step sizes.

The simplest fix is to reduce the learning rate e.g. change line

optimizer = tf.train.GradientDescentOptimizer(0.5)

to

optimizer = tf.train.GradientDescentOptimizer(0.05)

Also, have a play with different optimisers. Some will be able to cope with .abs-based loss better.


getting the weights of intermediate layer in keras I have an image dataset 376 classes each class has 15 pictures corresponds to a person. I would like to get the feature vector that corresponds to each person.

What I have done is, after I compiled the model I then used this link
as a reference to get the weights of the last convolutional layer. However, when I do this, I get the error:

InvalidArgumentError: You must feed a value for placeholder tensor 'conv_layer' with dtype float and shape [?,19,19,360]

How can I resolve this issue?

Here is the code that I have done so far:

train_data = np.array(train_data, dtype=np.float32)
test_data = np.array(test_data, dtype=np.float32)
train_data = train_data / 180 # to make the array values between 0-1
test_data = test_data / 180
train_label = keras.utils.to_categorical(train_label, 376)
test_label = keras.utils.to_categorical(test_label, 376)
# CNN MODEL
model = Sequential()
model.add(Conv2D(180, (3, 3), padding='same', input_shape=(180, 180, 3),
activation="relu")) #180 is the number of filters
model.add(Conv2D(180, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
model.add(Conv2D(360, (3, 3), padding='same', activation="relu"))
model.add(Conv2D(360, (3, 3), activation="relu"))
conv_layer = model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
flatten_layer = model.add(Flatten())
model.add(Dense(496, activation="relu"))
model.add(Dropout(0.5))
dense_layer = model.add(Dense(376, activation="softmax"))
#compiling the model
model.compile(
loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy']
)
model.fit(
train_data,
train_label,
batch_size=32,
epochs=40,
verbose = 2 ,
validation_split=0.1,
shuffle=True)
# getting intermediate  layer weights
get_layer_output = K.function([model.layers[0].input],
                              [model.layers[11].output])
layer_output = get_layer_output([conv_layer])[0]

The easiest way to create a truncated output from a network is create a sub-network of it and apply weights of your trained network. The following example is a modification of what you have shown up there, but it will guide you out

Network you want to train originally

model = Sequential()
model.add(Conv2D(10, (3, 3), padding='same', input_shape=(60, 60, 3),
                 activation="relu"))
model.add(Conv2D(10, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(5, activation="softmax"))
model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy'])

model.fit(
    train_data,
    train_label)

Now create a subnetwork from which you want the outputs, like from above example

model_new = Sequential()
model_new.add(Conv2D(10, (3, 3), padding='same', input_shape=(60, 60, 3),
                     activation="relu"))
model_new.add(Conv2D(10, (3, 3), activation="relu"))
model_new.add(MaxPooling2D(pool_size=(3, 3)))
model_new.add(Dropout(0.25))
model_new.add(Flatten())

model_new.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['mse'])

# You need to apply fit on random array's created, just so as to initialise
# weights. Anyways you will replacing them with original ones from above.
model_new.fit(train_data, y=np.random.rand(40, 3610))

Now take weights from the first trained network and replace the weights of the second network like

model_new.set_weights(weights=model.get_weights())

You can check whether the weights are changed in the above step by actually adding these check statements like

print("Are arrays equal before fit - ",
      any([np.array_equal(a1, a2) for a1, a2 in zip(model_new.get_weights(), model.get_weights()[:4])]))

model_new.set_weights(weights=model.get_weights())
print("Are arrays equal after applying weights - ",
      all([np.array_equal(a1, a2) for a1, a2 in zip(model_new.get_weights(), model.get_weights()[:4])]))

This should yeild

Are arrays equal before fit -  False
Are arrays equal after applying weights -  True

Hope this helps.


Regression Model Evalation with unbalanced target; alternative to R-squared? The Data look something like this:

[0, ....,  0, 2.8, 10, 4.5,]

So there are a lot of zeros to predict. And just a few non zeroes.

Apllying R2 to a potentially predictive model for this kind of Data is not feaseable:
The difference between all zero prediction and the expected values is not that big.

Is there a R2-like metric to aid this situation ?

Looks like your data are severely unbalanced if they are almost all zeros. Try data balancing or some form of cross-validation. To your original point: there is not a simple R^2 metric fix.


Fetching rules from rpart using caret package I'm looking to find a way where I can extract rules from my decision tree. I have 8 predictors and they are all categorical variables and the response variable has three outputs A B and C.

I've developed the model and can also create decision tree but I'm having trouble in understanding the tree and be able to extract rules from my model.

The 8 predictors I have can have values a, b, c, d, e.

To give you an example, I want to extract rules from my decision tree like this:

Rule 1:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - c
Predictor 4 - b
Predictor 5 - a
Predictor 6 - d
Predictor 7 - e
Predictor 8 - a

Result: A

Rule 2:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - d
Predictor 4 - b
Predictor 5 - a
Predictor 6 - d
Predictor 7 - b
Predictor 8 - a

Result: A

Rule 3:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - d
Predictor 4 - a
Predictor 5 - a
Predictor 6 - a
Predictor 7 - b
Predictor 8 - a

Result: B

and so on and so forth.

Is there a way I have rules from my decision tree like I explained above?

Here is the result of my model:

124 samples
8 predictors
3 classes: 'A', 'B', 'C'

No pre-processing.
Resampling: Cross-Validated (10 fold, repeated 10 times).
Summary of sample sizes: 112, 111, 111, 112, 110, 112, ...
Resampling results across tuning parameters:

  cp           Accuracy   Kappa
  0.00000000  1.0000000  1.0000000
  0.06329114  1.0000000  1.0000000
  0.12658228  1.0000000  1.0000000
  0.18987342  1.0000000  1.0000000
  0.25316456  1.0000000  1.0000000
  0.31645570  1.0000000  1.0000000
  0.37974684  1.0000000  1.0000000
  0.44303797  0.7261846  0.5696935

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was cp = 0.3797468.
n= 124

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 124 79 A (0.3629032 0.2741935 0.3629032)
  2) Activity4e< 0.5 45  0 A (1.0000000 0.0000000 0.0000000) *
    3) Activity4e>=0.5 79 34 C (0.0000000 0.4303797 0.5696203)
      6) Activity7c>=0.5 34  0 B (0.0000000 1.0000000 0.0000000) *
      7) Activity7c< 0.5 45  0 C (0.0000000 0.0000000 1.0000000) *

You can use the rpart.rules function if you have used rpart to build the original tree.

library(rpart)
fit<-rpart(Reliability~.,data=car.test.frame)
rpart.rules(fit)


Why `max_features=n_features` does not make the Random Forest independent of number of trees? Consider the following simple classification problem (Python, scikit-learn)

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

def get_product_data(size):
    '''
    Given a size(int), sets `log10(size)` features to be uniform
    random variables `Xi` in [-1,1] and an target `y` given by 1 if
    their product `P` is larger than 0.0 and zero otherwise.
    Returns a pandas DataFrame.
    '''
    n_features = int(max(2, np.log10(size)))
    features = dict(('x%d' % i, 2*np.random.rand(size) - 1) for i in range(n_features))
    y = np.prod(list(features.values()), axis=0)
    y = y > 0.0
    features.update({'y': y.astype(int)})
    return pd.DataFrame(features)

# create random data
df = get_product_data(1000)

X = np.array(df.drop(df.columns[-1], axis=1))
y = df['y']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33,
                                                        random_state=1)

def predict(clf):
    '''
    Splits train/test with a fixed seed, fits, and returns the accuracy
    '''
    clf.fit(X_train, y_train)
    return accuracy_score(y_test, clf.predict(X_test))

and the following classifiers:

foo10 = RandomForestClassifier(10, max_features=None, bootstrap=False)
foo100 = RandomForestClassifier(100, max_features=None, bootstrap=False)
foo200 = RandomForestClassifier(200, max_features=None, bootstrap=False)

Why does

predict(foo10)  # 0.906060606061
predict(foo100)  # 0.933333333333
predict(foo200)  # 0.915151515152

give different scores?

Specifically, with

max_features=None, all features are selected for each tree
bootstrap=False, there is no bootstrap of samples
max_depth=None (default), all trees reach the maximum depth

I would expect each tree to be exactly the same. Thus, regardless of how many trees the forest has, the predictions should be equal. Where is the tree's variability coming from in this example?

What further parameters would I have to introduce in the RandomForestClassifier.__init__ in such a way that foo* have all the same score?

Interesting puzzle indeed.

First things first. The DecisionTreeClassifier has some stochastic behavior. For instance, the splitter code iterates through the features at random:

        f_j = rand_int(n_drawn_constants, f_i - n_found_constants,
                       random_state)

Your data is small and comes from the same distribution. What this means is that you'll have a lot of identical purity scores depending on how iteration is done. If you (a) increase your data, or (b) make it more separable, you'll see the problem should ameliorate.

To clarify: if the algorithm computes the score for feature A and then computes the score for feature B and it gets score N. Or if it computes first the score for feature B and then for feature A and it gets the same score N, you can see how each decision tree will be different, and have different scores during test, even if the train test is the same (100% if max_depth=None of course). (You can confirm this.)

During my exploration of your question, I have produced the following code with my own implementation of a random forest. Since it took me some time, I figured I might as well paste it here. :) Seriously, it can be useful. You can try to disable random_state from my implementation to see what I mean.

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
import numpy as np

class MyRandomForestClassifier:
    def __init__(self, n_estimators):
        self.n_estimators = n_estimators

    def fit(self, X, y):
        self.trees = [DecisionTreeClassifier(random_state=1).fit(X, y)
                      for _ in range(self.n_estimators)]
        return self

    def predict(self, X):
        yp = [tree.predict(X) for tree in self.trees]
        return ((np.sum(yp, 0) / len(self.trees)) > 0.5).astype(int)

    def score(self, X, y):
        return accuracy_score(y, self.predict(X))

for alpha in (1, 0.1, 0.01):
    np.random.seed(1)
    print('# alpha: %s' % str(alpha))
    N = 1000
    X = np.random.random((N, 10))
    y = np.r_[np.zeros(N//2, int), np.ones(N//2, int)]
    X[y == 1] = X[y == 1]*alpha
    Xtr, Xts, ytr, yts = train_test_split(X, y)

    print('## sklearn forest')
    for n_estimators in (1, 10, 100, 200, 500):
        m = RandomForestClassifier(
            n_estimators, max_features=None, bootstrap=False)
        m.fit(Xtr, ytr)
        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))

    print('## my forest')
    for n_estimators in (1, 10, 100, 200, 500):
        m = MyRandomForestClassifier(n_estimators)
        m.fit(Xtr, ytr)
        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))
    print()

Summary: Each DecisionTreeClassifier is stochastic, data such as yours, which is small and comes from the same distribution, are bound to produce slightly different trees, even if the random forest itself is deterministic. You can fix this by passing the same seed to each DecisionTreeClassifier which you can do using random_state=something. RandomForestClassifier also has a random_state parameter which it passes along each DecisionTreeClassifier. (This is slightly incorrect, see the edit.)

EDIT2: While this removes the stochasticity component of the training, the decision trees would still be different. The thing is that sklearn ensembles generate a new random seed for each child based on the random state they are given. They do not pass along the same random_state.

You can see this is the case by checking the _set_random_states method from the ensemble base module, in particular this line, which propagates the random_state across the ensembles' children.


tree.DecisionTree.feature_importances_ Numbers correspond to how features? clf = tree.DecisionTreeClassifier(random_state = 0)
clf = clf.fit(X_train, y_train)
importances = clf.feature_importances_

importances variable is an array consisting of numbers that represent the importance of the variables. I wonder what order is this? Is the order of variable importances is the same as X_train?

I am trying to make a plot from this. So order matters.

You can take the column names from X and tie it up with the feature_importances_ to understand them better. Here is an example -

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
import pandas as pd

clf = DecisionTreeClassifier(random_state=0)
iris = load_iris()
iris_pd = pd.DataFrame(iris.data, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
clf = clf.fit(iris_pd, iris.target)

I am taking the iris example, converting to a pandas.DataFrame() and fitting a simple DecisionTreeClassifier. Once the training is done, you can take the columns attribute of a pandas df and make a dict with the feature_importances_ output.

print(dict(zip(iris_pd.columns, clf.feature_importances_)))

This will give you what you want -

{'sepal_length': 0.0, 'sepal_width': 0.013333333333333329, 'petal_length': 0.064055958132045052, 'petal_width': 0.92261070853462157}

Hope this helps!


Support Vector Classifier and cost C Applying SVC (Support Vector Classifier) to the 1-d data shown here:

What will be the support vectors for the parameter cost C=0 and C=Infinity?
As far as I read about SVM and hyper parameter C I would say:

For C = 0 there will be no support vectors

For C = Infinity all of the data points will be support vectors

Is it correct?
Note that I'm talking about SVC not SVM

SVC aims to find a separating hyperplane. That data you have is separable. Thus the hyperplane will go between x3 and x4. This are your support vectors. Now C is a parameter which allows you to make a trade off between errors allowed and the width of separating hyperplane. But your data does not need this trade off since it is already separable. Because of that the needed error to be accepted is zero. This penalty error in svc equations is multiplied with C. Because C is multiplied with 0 it does not matter the value of C, their product will be zero anyway.


What knowledge should I gain for developing a supervised image processing software that learns how to edit photos based on past behavior? I have done several machine learning projects but all of them have been connected to the traditional machine learning (predictions, classifications, etc.). I have currently been offered a project to finish in less than 6 months.

The idea is to develop/improve a pre-existing software. The software takes the image of a molecule from an advanced molecule and then tries to highlight the cell line with red, some times the software takes the background or the lines of other cells also as the highlighted part, and thus the user has to manually edit and trim such mistakes. The idea is to make the software learn from user's edits and behavior over time.

One thing I want to know is whether such a 6-month project is realistic for someone with no background in image processing and pattern recognition? Or is it going to be terribly difficult because I only have had experience with "data-oriented"/statistical machine learning?

My other question is; what type of concepts/topics should I dig deep into to learn the fundamentals of carrying out this project?

From the description of your problem, you need both Computer Vision and Deep Learning for a task like that. It is going to be extremely difficult, but you are more in advantage than anyone else, given you have a strong statistical and machine learning background. You don't have to worry a lot about the image processing part as there are libraries that will do that for you. You can look into PIL for that.The hard part is the learning from the edits part.A simpler way to solve this problem would be to focus on image processing and pick out the cell line clearly. The other way would be to train a Convnet on a large collection of 'labelled-images' of the cell-line, so that it is able to identify it in any picture. I do think it is quite a hard problem to solve but do give it a try. Cheers. All the best.


How to use feature group? Let's say I have a data set like the following:

file   group_a_co_1    group_a_co_2    group_b_co_1    group_b_co_2
file_1  0.8 0.2 0.3 0.7
file_2  0.1 0.9 0.2 0.8
file_3  0.5 0.5 0.7 0.3
...

I wonder, whether there are ways/tricks to tell the model about the group information here: since group_a_co_1 + group_a_co_2 = 1 and the same goes for group_b. Somehow I figure if I expose the group information, the performance of my model will improve.

The information in groups 'group_a_co_2' and 'group_b_co_2' are already redundant; they do not add more information to the model. Therefore they can be removed.
Adding even more redundant information will not improve your model further.


Will the performance of my NER model improve? I am training a spacy model from scratch by creating a dataset of my own with format spacy needs it to be in, the model is an NER model and the entity i am trying to recognize is Food items. I have created a dataset with 263 rows and after training the spacy model from scratch on this dataset, my model is performing great (I am getting around 80% accuracy) and this accuracy may not look a lot but it is better and I am being able to do my task a lot better now.

Now I want to improve my model even further by increasing train data. For increasing the train data I am thinking of using the rows I am sending as test and checking manually if spacy recognised each entity correctly or not and if all entities in a sentence are recognised correctly use this sentence in my training set.

My question is will this method improve my model in any way ?
Or in general if we add data which is correctly classified by our model in training set, will the performance of model improve to some extent?

More gold data of any sort (especially at this small scale) usually leads to improvement because the model improves after seeing more examples in different contexts, but you especially want to add cases that the model got wrong (after correcting the annotation, of course) to see improvement.


Help with backpropagation equations for a simple neural network with Sigmoid activation I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem.
 As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input x will compute c=ax+b, a being a parameter and b being its bias. Then it will feed
c to Sigmoid function to compute z=1/(1+exp(-c)) which will be its output. Please correct me if I'm wrong in any of this.

Now, I'm trying to follow this article to create back-propagation differential equations. It says that dc = z * (1-z) * dz where dz is the gradient of the green edge and then da = x * dc where da is the gradient computed in this iteration of backpropagation.

My question is that there are three zs (one corresponding to each outward edge) and their value(weight) will be identical (z=1/(1+exp(-c))). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute dc which is simply z * (1-z) * dz but there are three zs and three dzs?

The gradient is actually calculated w.r.t to the weights so that each neuron would have a different 'change in the weight' according to its previous weight. In other words, not all the 3 neurons in the hidden layer will be penalized with the same value because the initial weights at t=0 are different for these neurone.

It is well explained here: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

I hope this short answer helped!


How to plot multiple bar charts with different infill and outline? I have the CSV's of data. I want to plot the distribution of every attribute or a single one to compare the distribution from other CSV's, Since all attributes are the same but the distribution is different from each other because I have made the synthetic data using the original one.
Just like shown below that one attribute distribution is different from the other, I don't know the name of this chart. How could I plot this :

Code I tried so far:
    data=pd.read_csv(r'/content/drive/MyDrive/census-income.data.csv',names=['age','class of worker','detailed industry recode','detailed occupation recode','education','wage per hour','enroll in edu inst last wk','martial status','major industry code','major occupation code','race','hispanic origin','sex','member of labour union','reason for unemployment','full or part time employment stat','capital gains','capital losses','divdends from stock','tax filer status','region of previous residence','state of previous residence','detailed household and family stat','detailed household summary in household','instance weight','migration code-change in msa','migration code-change in reg','migration code-move within reg','live in this house 1 year ago',' migration prev res in sunbelt','num persons worked for employer','family members under 18','country of birth father','country of birth mother','country of birth self','citizenship','own business or self employed','fill inc questionnaire for veterans admin','veterans benefits','weeks worked in year','year','income'])
    data=data.drop(['state of previous residence','migration code-change in msa','migration code-change in reg','migration code-move within reg',' migration prev res in sunbelt','country of birth father','country of birth mother','country of birth self','instance weight'],axis=1)
    s_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/SDV/SDV_TRAIN_DATA.csv')
    d_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/Data Synthesizer/DataSynthesizerCensusTrain.csv')
    sp_data=pd.read_csv('/content/drive/MyDrive/SyntheticData/Synthpop/CensusTrainData.csv')
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd

    # example data
    x = np.arange(len(data))

    fig, ax = plt.subplots(figsize=(50,48))

    ax.bar(data['age'], x,color="red",  edgecolor="None", linewidth=4, label='Original Data')
    ax.bar(s_data['age'],x,color="None", edgecolor="blue", linewidth=4, label='Synthetic Data Vault')
    # ax.bar(d_data['age'],x,color="None", edgecolor="green", linewidth=4, label='Data Synthesizer')
    # ax.bar(x, sp_data, color="None", edgecolor="green", linewidth=2, label='Synthpop')
    ax.legend()

    plt.show()

Use (more information in the docs):
import matplotlib.pyplot as plt
import numpy as np

# example data
x = list(range(14))
y1 = np.arange(14)
y2 = np.arange(14)[::-1]
y3 = [8 for x in range(14)]

# Interesting part
fig, ax = plt.subplots()

ax.bar(x, y1, color="red",  edgecolor="None", linewidth=2, label='1')
ax.bar(x, y2, color="None", edgecolor="blue", linewidth=2, label='2')
ax.bar(x, y3, color="None", edgecolor="green", linewidth=2, label='3')
ax.legend()

plt.show()


Why in spectral clustering number of eigen vectors is same as number of clusters that we want? In spectral clustering we take eigenvector corresponding to K smallest eigenvalues. Then we do K means clustering on these eigenvector to get final clusters. What will happen if we take different number of eigenvectors than number of clusters we want ?

Theoretically, log2(k) components could be enough.

But usually clusters are not that well balanced, that you could get all 2^l combinations of eigenvectors stable, usually one will mask the other.

If you go back to the theory of spectral clustering, you'd have one eigenvector for each connected component. Now the clusters are unfortunately connected, so we don't completely get this ideal situation, but you'd expect one eigenvector for each "almost component", too.


How to control the amount of positives in classification? I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "False". I want to train a classifier (typically XGBClassifier or LGBMClassifier are those I worked a bit with).

What I'm currently doing is that I find the best parameters using GridSearchCV. But my objective is to minimize the amount of samples I would flag as "True" once I try on the test set. Should I train the algorithm with a typical F1 metric, and ONLY THEN find the best threshold that suits my needs? Or should I create a custom metric that would implicitly force the algorithm not to flag too many samples as positive?

Hope this is clear!

as desertnaut was saying, I think you are talking about class imbalance.

If you meet class imbalance "in the wild" there are a couple of ways to tackle this. Your approach of creating a custom metric would fall under cost function based approaches. You can use cost functions that handle false negatives as worse than false positives (which is usually the case when we're talking about class imbalance).

Another way would be to use sampling based approaches. These are rather simple and I personally tend to use these more often since it's easier to play around with aka experiment with them. Two extremes that can be combined as you wish:

Boosting one class / Oversampling: Look at the distribution of Class A and Class B. Boost the class you want to have more effect to have more datasamples changing your algorithm.
Undersampling: Look at the distribution of Class A and Class B. Remove datasets of the class you want to have less effect of your algorithm.

In reality I usually go for a combination of both to have more variance in my data. For example boosting Class A by 110% and reducing Class B to 80% to achieve a balanced dataset (although it sounds to me that you want to have an imbalance just the other way round. Take care about bias with that though)


Pandas: How can I merge two dataframes? I found (How do I merge two data frames in Python Pandas?), but do not get the expected result.

I have these two CSV files:

# f1.csv
num   ano
76971  1975
76969  1975
76968  1975
76966  1975
76964  1975
76963  1975
76960  1975

and

# f2.csv
num   ano   dou  url
76971  1975 p1   http://exemplo.com/page1
76968  1975 p2   http://exemplo.com/page10
76966  1975 p2   http://exemplo.com/page100

How do I merge these for to get the result given below?

# Expected result
num   ano   dou  url
76971  1975 p1   http://exemplo.com/page1
76969  1975
76968  1975 p2   http://exemplo.com/page10
76966  1975 p2   http://exemplo.com/page100
76964  1975
76963  1975
76960  1975

f1.merge(f2, left_on='num', right_on='num', how='outer')

see https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html


250 Categorical values I have a dataset which has only categorical values. As I came across a few articles people suggested that KNN / Random forest would work for dataset like this. Though in R it couldn't handle as if contains more than 53 categories.

Whereas in Sklearn, I used  linear regression against each column to the column which I was predicting. After which I created a weighted average of all the outputs. But I'm pretty sure it's not the right way.

Can anyone suggest a better way?

PS : The dataset contains 4 columns  with only categorical values. With 200 - 250 different categories in each column.

Welcome to the site!

I think the 53 category issue is specific to R's randomForest package. If you're set on using a random forest, try other packages like ranger and Rborist.


Retrieve dropped column names from `sklearn.impute.SimpleImputer` The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops some features at will, but has no way to communicate which features have been dropped to the caller

I've been trying to come up with a workaround, but they all are extremely hackish and unreliable. Is there something I'm missing?

I've got the same issue today, and it's a shame your post got no answers. I think this question is not well addressed in the sklearn documentation. I can show you my workaround to this issue:

headers = X.columns.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

empty_train_columns =  []
for col in X_train.columns.values:
    # all the values for this feature are null
    if sum(X_train[col].isnull()) == X_train.shape[0]:
        empty_train_columns.append(col)
print(empty_train_columns)

The idea is to keep all your column names, and after you split your data check which of them completely empty in your training set. If I'm not wrong the Imputer respects the column order so, for example, you can correlate every feature with its importance if you are using Decision-Tree-based models.

I'm not satisfied with this ugly piece of code but I couldn't find a more elegant (and simple) solution.