Hyperparameter Optimization for a Machine Learning Algorithm I have a question regarding Hyperparameter Optimization for a Machine Learning Algorithm.

I try to fit a Support Vector Classifier and use Hyperparameter-Tuning (but it could be also another classifier).

My classes are highly imbalanced (20% of one class, lets call it “red” and 80% of the other, lets call it “black”).

Now, the objective of my Hyperparameter-Optimization is the cross-validation loss.

If, say, 20% of the observations are “reds” and 80% are “black”, then a really bad classifier would just label every case as “black” and achieve on average a cross-validation loss of 20%.

Now my question: If I see that the objective of my Hyperparamer-Optimization does not fall significantly below 20%, I could infer that it is useless without further analyzing. Correct? Or is there something I do not understand correctly?

This happens in all optimizations I am running now. I could conclude that my features are not informative.

I guess I have to change the objective for the hyperparameter optimization routine? (If my package allows for that)

You are right, what has to be changed is the objective. You are currently using accuracy as a measure of how good your classifier is. Accuracy is not a good measure when you have class imbalance. For this reason, other objectives have to be used. AUC is a better measure, and so is the log-loss, although not that interpretable. However, both AUC and log-loss need you method to output probabilities, not just predictions of classes. For this reason, it is required that your SVM implementation has a predict probability method.


What is the meaning of semi-handcrafted features? I know handcrafted features are features which are being created by a human made algorithm. But what are semi-handcrafted features?

It means part of it handled by the human at first, and then, base on them running a machinery learning algorithms to extract new features. For example, gather some features from the human about some pictures, and using some machinery algorithms to find some new features by some non-linear combinations of those features.


XGBoost is it possible to prevent a feature from being used twice in the same tree? I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the columns in xtrain, the feature chart changes along witht he shuffle after re-running the model.
This suggests that XGBoost is "getting stuck" on the first feature. Is there a way to prevent this? I was thinking by limiting how many times the same feature can appear in a tree?

If you mean "how many times the same feature can appear in an [individual] tree", then
you can use max_depth to indirectly limit the number of features included in a single tree, even down to one feature. Since XGBoost is designed to use weak learners, having a lower depth value is ok.
model = XGBClassifier(max_depth=n)

However, I think the problem is not that XGBoost is getting stuck on a single feature. For example, maybe the other columns contain little or no correlation to the label. How does your model perform when you remove that column entirely. That should tell you if you only have one good feature in your dataset.


StandardScaler before and after splitting data When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before splitting the data into train/test, but when i was checking some of the codes posted online (using sklearn) there were two major uses.

1- Using StandardScaler on all the data. E.g.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_fit = sc.fit(X)
X_std = X_fit.transform(X)

Or

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit(X)
X = sc.transform(X)

Or simply

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_std = sc.fit_transform(X)

2- Using StandardScaler on split data.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

I would like to standardize my data, but I am confused which method is best!

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to transform the training set, which in turn is passed downstream.

As an example, knowing the distribution of the whole dataset might influence how you detect and process outliers, as well as how you parameterise your model. Although the data itself is not exposed, information about the distribution of the data is. As a result, your test set performance is not a true estimate of performance on unseen data. Some further discussion you might find useful is on Cross Validated.


Autoregressive (AR) models constants - Time Series Analysis I'm currently struggling with different Model like AR or MA.

If AR(1) is expressed as: $y_t = \beta + \beta_t \times y_{t-1} + \epsilon_t $

How do I know what the $\beta$ 's would be? What are the dependencies?

I think a simple example would help a lot.

You have a slight typo in the notation of a AR(1) model. The correct signature is,

$y_t = \beta_0 + \beta_1 \times y_{t-1}+\epsilon_t$ or $y(t) = \beta_0 + \beta_1 \times y(t-1) + \epsilon(t)$,

where $y(t)$ and $\epsilon(t)$ are  random variables. If $y(t)$ is standard Gaussian you can estimate $\beta_{0,1}$ with a maximum likelihood estimator (MLE). If not you will need a more complex method. You can read more about it in this article on estimating at ARMA Process.


How does backpropogation behave when there is a dropout in FC layer? I read about dropout and how it helps in catering overfitting. In simple layman terms, it randomly drops some of the neurons in forward propagation. My question is that since these neurons will be 0 in forward propagation, are we supposed to update them while backpropagating?

Correction made (see edit log):

No, there is no need for that.

For all dropped neuron, the corresponding activations are simply zeroed out so the corresponding gradient will also be zero.

Now because you do not want to update the parameters, you simply multiply the gradients with the same dropout mask as was used in your current iteration.

If you're using a library, you just have to do everything as it is usually done because this is already being taken care of.


Underlying model for prediction using different prediction variables I have time-series energy consumption data for a duration of one-month. The frequency of data is half-hourly. The features of dataset are

temperature - temperature value at particular time instant
humidity -    humidity value at particular time instant
timeofday -  this corresponds to 48 half-hourly durations of a day.
weekday -  day of the week (value between 1 - 7)
prevday1 -  energy consumption at the same time-instant on previous day
prevday2 - energy consumption at the same time-instant on a day before the  previous day
prev_instant1 - energy consumption at the previous time instant. For example, for 7 pm, previous_instant1 is 6:30 pm
prev_2_hour - average energy consumption during the last two hours

Using these variable I need to forecast power (energy) consumption. The pairs plot of these variables is

I started with a linear model, but it seems that the model is performing very badly. I want to know is there any automatic technique, which can check all possible combinations of predictor variables and output the best fitted model. I can check all models (linear combinations, non-linear, splines etc) manually, but it will take huge time.

UPDATE

I am forecasting in real-time at a half-hourly rate (very short term forecasting). Also, I use separate models corresponding to different half-hours of the day. For example, In a day we have 48 half-hours so corresponding to each half-hour I have a separate model. For the whole process, I use following approach:

Create a training data set with above mentioned features. This training data set is of one month duration (30 days)
In the next step, I forecast for the next day (testing day) using specific models for each of 48 half-hour durations
At the end of the day, I retrain all my 48 models with recently updated data (recent historical 30 days)

Steps 2 and 3 continue till stopping condition. Stopping condition used is the no. of testing days. A screenshot of my training data is as:

head(training_data)

tail(training_data)

Have had the same problem before.  Used Weka tool (http://www.cs.waikato.ac.nz/ml/weka/).

The approach:
• convert time of day into a nominal (non-numeric), eg bins numbered 0-47
• convert day of week into a nominal (non-numeric)
• use M5Tree to construct a model

The rules in the tree will pop out things like "if 0 < bin < 5:30am_bin" then ... which discriminates start of day

It is better with more than 1 month of data.  There will only be 4 samples for certain days of the week, for example.  So more months would be better. And in such case again converting month to a nominal value (non-numeric) lets the rules pick up seasonal and monthly changes.


Making sense of a accuracy plot for a 5 fold training using random forest I'm using sklearn.model_selection.learning_curve for 5 fold training of data. The code is as given below.

train_sizes = [1, 100, 500, 1000, 2000, 3000, 3879]
train_sizes,train_scores, validation_scores = learning_curve(estimator = ensemble.RandomForestClassifier(),
                                                          X = X_res,
                                                          y = y_res,
                                                          train_sizes = train_sizes,
                                                          cv = 5,
                                                          scoring = 'accuracy')
train_scores_mean = train_scores.mean(axis = 1)
validation_scores_mean = validation_scores.mean(axis =1)
plt.style.use('seaborn')
plt.plot(train_sizes, train_scores_mean, label = 'Training acc')
plt.plot(train_sizes, validation_scores_mean, label = 'Validation acc')
plt.ylabel('Accuracy', fontsize = 14)
plt.xlabel('Training set size', fontsize = 14)
plt.title('Learning curves for a linear RF model', fontsize = 18, y = 1.03)
plt.legend()

Also, I'm using the random forest classifier model for a multiclass classification problem. When I plot the accuracy graph, it appears like the below.

How's it possible that the mean training accuracy for all training sizes is 100%?

Any help is appreciated.

Using all default values for the RandomForestClassifier class leads to overfitting. As stated in scikit-learn documentation,

  The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values.

Fully grown trees have a perfect score on train data, but as you can guess, that is not how random forests are supposed to be used. Try setting the various parameters of the classifier (start with max_depth for instance), and you will get more consistent results.


Converting data format I'm trying to use the recent COVID-19 data from the site of Italian Civil Protection, but they use a rather complicated time format that I'm finding troublesome as a novice to plot as data in a graph.
This is how the data is presented:

[1] 2020-02-24T18:00:00 2020-02-25T18:00:00 2020-02-26T18:00:00 2020-02-27T18:00:00 2020-02-28T18:00:00 2020-02-29T18:00:00

and I would like to use the format as DD-MM, without the time and the year.
How can I do it?


What clustering algorithm is appropriate for clustering paths? I have a dataset with vectors in 2-dimensional space that form separate sequences (paths). Full data is presented below: ,
while a random sample of 5 paths looks like below (please note that incontinuity in paths are natural for the data and doesn't mean missing values):

I would like to find similar paths, where similar would mean (in order from the most to the less prominent):

they end up in a similar region
they are similar in direct length (i.e. length from start to end on x axis)
they are similar in complexity (i.e. number of vectors)
they starts in a similar region

What clustering algorithms are natural choice for such a setup? What are things worth to be aware while clustering paths? How can I deal with the fact, that different paths has different number of vectors? How can I represent a data to take that into account?

Like Ricardo mentioned in his comment on your question, the main step here is finding a distance metric between paths. Then you can experiment with different clustering algorithms and see what works.

What comes to mind is dynamic time warping (DTW). DTW gives you a way to find a measure of "distance" (it is actually not strictly a distance metric, but it is close) between two time series. One very useful thing is that it can be used to compare two time series that are of different lengths.

There are many good blog posts on DTW, so I won't try to give yet another explanation of it. There are also many python implementations of it. And a lot of work has gone into making the algorithm very fast. DTW is a strange algorithm--in some ways very simplistic, but typically works well. Once you modify the algorithm to deal with paths, you can construct the distance matrix and use that for clustering. One common clustering algorithm that is used in conjunction with DTW is spectral clustering, since the distance matrix can be used directly (instead of the matrix of data points, which we don't have here).


Help with breaking up a column in PowerBI to make a bar chart My data set is a list of documents. I have a column in my data set - 'Independent Variable'. This column can contain nothing, i.e. 'NA', or one variable, i.e. 'var1', or multiple, i.e. 'var1;var2;var3' and is essentially metadata for the documents. I'm attempting to create a visual, like a bar chart, that would show how many documents in my data set are associated with each independent variable. I have roughly 30 independent variables, and they are all housed in the column 'Independent Variable', hence when trying to make a chart i get a ton of entries for that so each bar is 'var1', 'var2', 'var1;var2' etc. How can I break it down so that only var1, var2, var3 are my buckets and if those variables exist in the column that document is counted

You need to clean up the data. Load it into the Power Query editor.

Split the column with the variables by the delimiter ;
now you have a lot of individual columns, some have values, some are empty
select all columns except the newly split ones and use Unpivot > Unpivot Other Columns on the transform ribbon.
Now you have one row for each combination of document name and variable
Rename the columns as you see fit, delete the Attribute column and load the data.

Create a chart with the variable column in the legend.


Brownian motion in models for virus spread? Was reading a Washington Post article "Why outbreaks like coronavirus
spread exponentially, and how to flatten the curve” and it looked like they were using Brownian Motion.

(Can't directly link the graphic, but it's the third graphic in the article, showing particles bouncing around, but I've included a screenshot:)

If this is the case, how do particles in a fluid medium resemble interactions in a human population?

I read this article when it came out. I don't think the point is about whether or not people behave like particles in a fluid medium, but about how quickly people can spread the coronavirus when randomly interacting with one another in an open space (in my opinion, this model would be a reasonably good representation for a crowded area like a mall lobby or a beach where people are moving around more or less randomly).

However, even in other settings when people aren't moving around randomly, if anything, people are more likely to interact with each other in restricted spaces like sidewalks where there is less choice of where to position yourself spatially.


How to improve an existing (trained) classifier? I have a Classifier which I have trained and tested on a small dataset - receiving solid results, though I wish to improve them. If I understand correctly one way of doing so is to add more data to obtain a more precise classification rule.

When doing this should I add data to both the training set and the test set? or should I add only to the training set? or maybe I should I create new training and test sets from the 'new dataset'? (new = the old data + the new data).

In order to improve your classifier, you have few options.

Ensembling - make a group of classifier and let them predict together. Stacking, blending, bagging, boosting. Choice is yours.
Hyper parameter tunning - you did not mention your tool, but i suppose that in every solid one is option to search in parameter space to find the best combination
Sampling - you can try undersample or create new samples (SMOTE) to give your classifier more data of class which you'd like to predict
Feature engineering - get rig of noisy features and work only with those which have an effect on your prediction, consider PCA too
Scaling - normalize your data can improve the performance, some classifiers require this
Data quality - missing values, coding categorical variables or suspicious values can influence performance


Strange behavior in current Keras setting (tensorflow-1.7.0 Keras-2.1.5) when using CPU After reinstalling Keras and Tensorflow on a Virtual Machine, I have noticed that models fail to train correctly (at the very least those I have checked).
For example, taking this code from the Keras blog as a test case, instead of decreasing loss and final validation accuracy of ~0.8, loss quickly increased to ~7.5 and accuracy stayed at ~0.5.

It should be noted that when I tried it with older versions of Keras and Tensorflow (on the same computer), I did not observe this aberrant behavior.

I have repeated this experiment at three other setups in which the same behavior was observed - a two other computers using a VM, and a third computer, this time a laptop with no suitable GPU.
On the other hand, trying this on two computers with GPUs, the training progressed as expected, with loss decreasing and accuracy increasing to the level advertised in the Keras blog and observed with the previous versions of Keras and Tensorflow while using a CPU.

From the evidence I can speculate that either Keras's or Tensorflow's interaction with the CPU was changed in a manner disrupting the training processes.

Has anyone here faced a similar problem?
What can be done to solve this?

I recently had a similar trouble with porting working models to tensorflow 1.7. I found that the initialization was the culprit. Adding
model.add(Dense(2048, activation='relu', kernel_initializer=initializers.glorot_normal())) instead of model.add(Dense(2048, activation='relu')) fixed the problem.


Dimensionality reduction without select components I would like to use dimensionality reduction algorithm in my pipeline.
I have 2k features and I'm using xgboost.
My model is rebuilding each day (there are new records that should be involve to training set).

I'm looking for method for dimensionality reduction with out setting n_components.
I know that in PCA it shouldn't be set. But I'm looking for method that find something like clusters on my data and then I will use it to train my model. Of course the same flow I'll be using for prediction.

Do you have idea how should I do my data processing for this case?

It would be helpful to know a bit better what you're trying to achieve and why the selection of a specific number of eigenvalues bothers you.
From the generic information you gave it seems you're aiming at training a model on a compressed/dense representation of several features, in which case I would suggest to train an autoencoder (or something similar), on top of which you could then train whatever classifier you need.
Otherwise if the problem relies only on the amount of features you have you could try with some feature selection strategies


Learning word embeddings using RNN The common way of learning word embeddings is based on BOW, and Skip-gram models.

Is it possible to train a RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings? Basically, we train a network with positive and negative samples, and back-propagate into word vectors. What are the drawbacks of this technique?

Any reference to similar works is highly appreciated.

Yes, it is possible to train an RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings.

The word embeddings of the corpus words can be learned while training a neural network on some task e.g. sentiment classification. Before it can be presented to the RNN, each word is first encoded so that it is represented by a unique integer e.g. using a tokenizer. We add a padding token to make all the sentences of the same length.

Doing this is referred to as building an embedding layer "in front of" your LSTM/RNN/GRU network model. For the embedding layer you need to specify:

maximum length of a sequence
embedding size for each token.

The drawback is that using this technique, training will take more time because you have to not only train your prediction model but also your word embeddings. This is because the total amount of trainable parameters will be larger as compared to a model where we use a pre-trained word embedding with frozen parameters.

Finally, depending on your dataset size using pre-trained embeddings trained on some larger corpus will lead to better embeddings than training your own word embeddings on your smaller dataset.

Here is an experiment using both approaches:
https://towardsdatascience.com/machine-learning-word-embedding-sentiment-classification-using-keras-b83c28087456


Merging repeating data cells in csv I have a CSV file with around 1 Million rows.
Let say its have details like

Name      |   Age   | Salary
name 1      52       10000
name 2      55       10043
name 3      50       100054
name 2      55       10023
name 1      52       100322...

and soon .

but i need to merge the redundant details .
and need a output like

Name      |   Age   | Salary
name 1      52       110322*
name 2      55       20066 *
name 3      50       100054

you might notice that the repeating Name 1 and Name 2 details are merged and the Salary values are added .So i'm looking for a way to apply this change to my original data set. so i need a python script to fix my problem .

Pandas is a python library that you will find very useful for these types of tasks.

Here is a stack overflow post that tells you how to do what you want to accomplish.

It boils down to three very pythonic lines with a groupby and transformation followed by a drop_duplicates:

import pandas
df = pandas.read_csv('csvfile.csv', header = 0)
df['Total'] = df.groupby(['Name', 'Age'])['Salary'].transform('sum')
df.drop_duplicates(take_last=True)


can a perceptron be used for regression? In machine learning, the perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.

on wiki page, the problems section includes regression without more explanation.

can a perceptron used for regression?

The deep feedforward neural networks used for regression are nothing but multilayer perceptron architectures.

Originally, perceptrons were used as binary classifiers i.e to classify binary labels ( 0 or 1 ). But, if no non-linear activation function is applied to the dot product of the features and weights, then it is simply a linear regressor.

If the linear function is $f(x) = x$ and $N$ is the number of features then,

$\Large y = \sum_{i=0}^N x_i w_i$

or using vector notation,

$\Large y = \vec{x}.\vec{w} ...(1)$

For multiple regression, we use the below equation with mean-squared error loss function to optimize the $\beta$ parameters,

$\Large y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 ... \beta_n X_n$

Let ,

$\Large
\vec{W} = [ \beta_1 \ \beta_2 \ \beta_3 \ ... \ \beta_n ] \\
\vec{X} = [ X_1 \ X_2 \ X_3 \ ... \ X_n ]
$

Where $N$ is the number of features, $\vec{X}$ is the feature vector and $\vec{W}$ is the weights vector. Then,

$\Large y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 ... \beta_n X_n = \beta_0 + \vec{W}.\vec{X} ...(2) $

Where, $\beta_0$ is also called as the bias coefficient in Artificial Neural Networks. You can relate equations 1 and 2 and understand the concept.

  Hence, $\Large y = \vec{x}.\vec{w} + bias $ generally represents a hyperplane
  which is used in linear regression.


Keras one_hot encoding - what's the point when unicity not guaranteed? In the following example the text:

  'The quick brown fox jumped over the lazy dog.'

gets the encodings [5, 9, 8, 7, 9, 1, 5, 3, 8].

My code:

In this way the words 'brown' and 'dog' get both the encodings of 8 and 'quick' and 'jumped' 9.

What is the point of encoding if unicity is not guaranteed?

https://machinelearningmastery.com/prepare-text-data-deep-learning-keras/
https://keras.io/preprocessing/text/

It seems to be bad choice of function names since the one_hot function actually is using hashing and is equivalent to hashing_trick (with standard settings).

See this issue: https://github.com/keras-team/keras/pull/6887

When I do:

from keras.preprocessing.text import one_hot
from keras.preprocessing.text import hashing_trick

text = 'The quick brown fox jumped over the lazy dog.'
print(one_hot(text, n=9))
print(hashing_trick(text, n=9))

I get identical results:

[5, 6, 2, 6, 5, 8, 5, 4, 1]
[5, 6, 2, 6, 5, 8, 5, 4, 1]

So in case you want to have "real" (unique) one-hot encoded results, you need to resort to some other solution (there is no Keras built-in solution for this in the moment as I believe).


Classification or regression problem? I have a table with this features:
   'tourney_id', 'tourney_name', 'surface', 'draw_size', 'tourney_level',
   'tourney_date', 'match_num', 'winner_id', 'winner_seed', 'winner_entry',
   'winner_name', 'winner_hand', 'winner_ht', 'winner_ioc', 'winner_age',
   'winner_rank', 'winner_rank_points', 'loser_id', 'loser_seed',
   'loser_entry', 'loser_name', 'loser_hand', 'loser_ht', 'loser_ioc',
   'loser_age', 'loser_rank', 'loser_rank_points', 'score', 'best_of',
   'round', 'minutes', 'w_ace', 'w_df', 'w_svpt', 'w_1stIn', 'w_1stWon',
   'w_2ndWon', 'w_SvGms', 'w_bpSaved', 'w_bpFaced', 'l_ace', 'l_df',
   'l_svpt', 'l_1stIn', 'l_1stWon', 'l_2ndWon', 'l_SvGms', 'l_bpSaved',
   'l_bpFaced'

You can see that they are duplicate features with the difference that one half is for winners e.i winner_age and the other half for losers e.i loser_age. I not sure which approach to take as is not a classification problem in the sense that I don't have clear binary labels (of course I could categorize some of them) or regression because I don't have a clear target (each feature is both winner and loser). I'm trying to do some EDA first but in order to do it I have to have a clear understanding on which approach to take to start with in the first place.

If you are trying to predict the target of winner or loser, then it is a binary classification problem.
If you frame it as a binary classification, you'll have to reorganize the data as tidy data such that each row is an individual observation and each column is a feature.


Why is it bad to use the same test dataset over and over again? I am following this Google's series: Machine Learning Crash Course.

On the chapter about generalisation, they make the following statement:

Good performance on the test set is a useful indicator of good performance on the new data in general, assuming that:

- The test set is large enough.

- You don't cheat by using the same test set over and over.

Why exactly is the second point a bad one? As long as one does not use the test set for the training stage, why is it bad to keep using the same test set to test a model's performance? It's not like the model will get a bias by doing so (the test set is not updating any of the model's parameters internally).

In fact this will introduce some bias to your model because you will distinctly choose a model that performs well on the testing set. This is indeed what bias consists of. Ideally you would want to have a training set, a validation set and a testing set. You will only use the testing set right before writing the spec of your model. If the performance is much poorer than expected at this very last step, then you need to restart from the top. Get new data to reduce "the polluting" of the testing set and retrain and validate until you are satisifed then use the testing set.

Consider if your testing set does not generalize well on the actual distribution, this may be improbable but it is possible. Then a good model will perform poorly on this test set, however it would otherwise be a good model if you did not bias your decision using your test set.


Why might my validation loss flatten out while my training loss continues to decrease? In my effort to learn a bit more about data science I scraped some labeled data from the web and am trying to classify examples into one of three classes. I am running into a problem that, regardless of what model I try, my validation loss flattens out while my training loss continues to decrease (see plot below).

A few potentially relevant notes:

I have about 15,000(3,000) training(validation) examples.
Labels are roughly evenly distributed and stratified for training and validation sets (class 1: 35%, class 2: 34% class 3: 31%).
I have 73 features that consist of: 10 numerical features, 8 categorical features that translate to 43 one-hot encoded features, and a 20-dimensional text embedding
I get similar results if I apply PCA to these 73 features (keeping 99% of the variance brings the number of features down to 22).
The plot shown here is using XGBoost.XGBClassifier using the metric 'mlogloss', with the following parameters after a RandomizedSearchCV: 'alpha': 7.13, 'lambda': 5.46, 'learning_rate': 0.11, 'max_depth': 7, 'n_estimators': 221
I get similar results using a basic Neural Network of Dense and Dropout layers.
Best model I've achieved only gets ~66% accuracy on my validation set when classifying examples (and 99% on my training examples).
Admittedly my text embedding might not be fantastic (using gensim's fasttext), but they are also the most important feature when I use Xxgboost's plot_importance function. Though, I was facing a similar problem even before I added the text embedding.

This is totally normal and reflects a fundamental phenomenon in data science: overfitting. When the validation loss stops decreasing, while the training loss continues to decrease, your model starts overfitting. This means that the model starts sticking too much to the training set and looses its generalization power. As an example, the model might learn the noise present in the training set as if it was a relevant feature.
When training your model, you should monitor the validation loss and stop the training when the validation loss ceases decreasing significantly. It is also the validation loss that you should monitor while tuning hyperparameters or comparing different preprocessing strategies.


Collection of several learners I have few questions for which I could not extract answers from text books and online tutorials. Therefore, will be extremely grateful if the following points are clarified.

1) If I want to apply SVM, MLP and decision trees and combine the prediction results from these learners. So, I will have a mixture of ensemble of SVM, MLP and Decision trees OR I can use one learner only and have an ensemble of decision trees, an ensemble of SVMS and an ensemble of MLPs. Is my understanding correct?
How do I combine the prediction results from different models?

2) In ensemble learning be it homogeneous and heterogeneous, each of the learner is trained on the same data set? Is this technique known as Bagging  or simply ensemble learning? Would I be using the same subset of data for training each learner or different?

2) I have read in many tutorials that bagging and boosting is performed on similar learners. But ensemble learning can be homogeneous and heterogeneous. As a result, can bagging and boosting be performed on different learner types?

3) In Boosting, the learners are trained sequentially and therefore often only a learner of one kind is used as it is probably difficult to know if the sequence influences the learning procedure. Am I correct?

4) How is stacking different from the rest?

Here's my two cents:

Yes, if you want to perform enseble each model should be trained on the same set of data. The technique you described is stacking, because you "stack" each prediction and simply use the majority.
Ensemble Learning, to me, si more "complex", infact you can combine the different predictions from k models via linear combination, where you estimate the coefficients of the combination itself.

More precisely, you have k prediction for each of your test observations, from k models (ie: 3 or more), and you need to find the best combination (for each obs.) of these, to have the best "accuracy" (any kind of metric).

Keep in mind that it's best to combine models that have low or even opposite correlation, so that you "diversify" your predictions, that would mean better results.

Often peole train lots and lots of different model, than use a simple Neural Network using the predictions as inputs, and the true labels from the test set as the truth, just as a normal class model would do.

See the papers in the References here as a starting point.

Bagging is basically a special case of stacking via majority vote, see:

Bagging.

You propose to the same type of model (ie: decision trees) the train dataset multiple times, making new different extraction, with replacement. This is like submitting new entries, but basically you are using the same data all over. Then you average all the predictions again, this helps with overfitting but it's not always the case, because you are using the same model, and so each si highly correlated with the other.

Yes, in boosting, just like bagging you use the same model (usually a weak learner). At each stage, the learner learns from the errors made at the previus stage.

Here an excellent visual example of what's going on.

To be clear, stacking and ensemble learning can be used with lots of different models on the same train set (it's even better if the model in the "ensemble" are quite different). You can combine each prediciton with different techniques, from simple average, to the more complex use of Neural Networks.

Bagging and Boosting are based on the use of the same model, one using the same obs. multiple times, the latter uses the same weak learner on the same data, but each makes predictions using the errors made by the previous one, in a loop.

Hope this helps you a little.


If the dim of your data is more than 2, you may want to transform your data with PCA, t-SNE or Custom linear projections.

t-SNE is recommended for clustering result.

For example:

from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0)
data = tsne.fit_transform(data)

This may lose some information, but useful for visualization.


Creating a single number from a numpy array - Python I am working on a gender classification project. I am extracting the pixels of an image using a Numpy array in Python, similar to the one below:

[[[129 155 191] [123 150 185] [120 149 183]]

How can I use these values to extract a single meaningful number to be used in a csv file for K-Nearest Neighbor Algorithm. For example single number like

0.34232

In summary, how can I turn [[[129 155 191] [123 150 185] [120 149 183]] into a single number like 0.34232

Welcome to Data Science! You're question needs a little more detail... there are many many ways to make an array into a single number. You should say a little more about what you mean by meaningful.

Here are a few examples, using your example array, which may seem outrageously simple, but do indeed form the basis to many of the techniques used in modern research:

In [1]: import numpy as np

In [2]: x = np.array([[129, 155, 191], [123, 150, 185], [120, 149, 183]], dtype=np.float32)

In [3]: x
Out[3]:
array([[129., 155., 191.],
       [123., 150., 185.],
       [120., 149., 183.]], dtype=float32)

Now here are a few (randomly selected) ways to create a single number from your array:

In [4]: np.mean(x)    # the mean / average
Out[4]: 153.88889

In [5]: np.sum(x)    # the sum
Out[5]: 1385.0

In [6]: np.std(x)    # the standard deviation
Out[6]: 25.722641

In [7]: np.linalg.norm(x)    # the Frobius norm - a distance measure
Out[7]: 468.07156

In [8]: np.max(x)
Out[8]: 191.0

These might seem stupidly simple, but if we were to treat your array as a single block from a larger image-array, then these might represent pooling-layers that are used to down-sample arrays as they are passed through a neural network. Just have a look at the available pooling layers within the Keras library.

Which method you might want to use will heavily depend on your use case, your dataset and your model.


Log file analysis: extracting information part from value part I'm trying to build a data set on several log files of one of our products.

The different log files have their own layout and own content; I successfully grouped them together, only one step remaining...

Indeed, the log "messages" are the best information. I don't have the comprehensive list of all those messages, and it's a bad idea to hard code based on those because that list can change every day.

What I would like to do is to separate the indentification text from the value text (for example: "Loaded file XXX" becomes (identification: "Loaded file", value: "XXX")). Unfortunately, this example is simple, and in real world there are different layouts and sometimes multiple values.

I was thinking about using string kernels, but it is intended for clustering ... and cluseting is not applicable here (I don't know the number of different types of messages and eventhough, it would be too much).

Do you have any idea?

Thanks for your help.

P.S: For those who programs, this can be easier to understand. Let's say that the code contains as logs printf("blabla %s", "xxx") -> I would like to have "blabla" and "xxx" seperatated

How about considering each string as a process trace and applying alpha-algorithm?
That would give you a graph and nodes with a big number out-edges will most likely point to values.

You can mark these nodes and for every new string parse/traverse the graph until you reach those areas.


Help me choose a Data Science book in Python I've been a Data Scientist for a few years now, but I've only recently started to do most of my work in Python (boy, do I miss ggplot2! But altair is coming to the rescue).

I want to improve my Python skills and, since most of my work is related to developing Data Science & Analytics applications, I'd rather learn from a book about these topics than from a book on, say, Application Server frameworks. Also, since I mostly develop Deep Learning models, I was looking for a PyTorch book. However I could only find two, of which one is really crappy, and the other one is from authors I really respect, but on a topic I don't work with (NLP):

https://www.amazon.com/Natural-Language-Processing-PyTorch-Applications/dp/1491978236

So I've done a bit of research and found about the following books:

Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

Data Science from Scratch: First Principles with Python

Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython(second edition recently came out)

Python Data Science Handbook: Essential Tools for Working with Data

Which book would you choose? You're free to suggest other titles. Requirements, in order of importance:

It should be a good book about Python, so the code should be Pythonic
It should be about about PyTorch, but not only about NLP (this requirement will probably be unfulfilled, for reasons above). Not about Keras, or mainly about Keras: I already have a reference for that.
It should be about Data Science

Price is not an issue.

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow, its a great book (1/2 way thru).  However it uses Scikit as per the title and some of the mechanics inside Sckit are a black box (not explained in depth).  The general concepts however are still quite well written.

The 2nd half of the book is about TensorFlow so again, perhaps not what you want if you're focused on PyTorch ?


Statistical learning for data-limited systems I'm currently conducting a review for quantitative methods being used for tropical inland fisheries. One of the major problems for modeling methods in tropical inland fisheries is the lack of data available. Fisheries assessments are difficult with widely distributed, small-scale fisheries. As many of the people living in tropical regions are subsistence fishers, they directly consume fish without any recordings of the catch.

I'm trying to find statistical/mathematical modeling methods that are able to deal with data-limited systems. I do not have a significant background in statistics, but I am curious about the concept of "statistical learning" and data science methods in general. My understanding of current "hot" topics in data science (machine learning, AI) is that they are useful for big data. I have not come across their usefulness in data-limited methods. Although it seems as though they could potentially deal with data limitation by using a range of other data available to make hypotheses about the state of the data that is limited. Do you know of any usefulness of statistical learning or data science methods in general for data-limited systems?
Thanks!

It really depends on the data set in hand. For example for short time series you can look at this thread.

In general I would say that non-parametric statistics works pretty well for smaller datasets.


My Neural Network is not learning anything I try to train convolutional neural net for a classification problem. However, my neural net is not learning anything. It guesses only two labels and ignores the rest. Even though I try to train to overfit my neural net, the loss function is not decreasing at all. My I try to make my network go as deep as 12 layers of the convolutional neural net in order to overfit the subsampling data. However, it is not working. Do you have any suggestion what to look?

Make sure that while training, you are passing the data in a random order. It looks like you are passing all images of one class at a time then the next class etc. Shuffle the images and pass it in a random manner.


Splitting a pdf containing batch of scanned documents My question is primarily: is there any ML research paper about splitting a pdf containing a batch of scanned documents (eg bank statements) into individual documents?

I have searched for this but I have not found any relevant research paper or any application in general mentioned on the Internet.

I would be primarily interested in the feature engineering of these papers/applications but also in general in the whole approach.

"Machine Learning for Digital Document Processing: from Layout Analysis to Metadata Extraction" by Esposito, Ferilli, Basile, and Mauro goes into detail about how to create a custom system for parsing digital documents, including pdfs. It proposes a generalized process to learn any structure within documents.


How to add a calculated column in a Pandas dataframe? I am new to Python/Pandas so I'm struggling a bit here.

I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare them with the value the year before at the same day and month.

These are the first lines of the dataframe.

         Date Country       City Specie count   min   max median variance
0  2020-02-23      CR   San José   pm25    20  13.0  53.0   25.0  1232.00
1  2020-04-04      CR   San José   pm25    23  17.0  57.0   38.0  1302.57
2  2020-04-24      CR   San José   pm25    23  30.0  80.0   59.0  1966.13
3  2020-01-14      CR   San José   pm25    24  13.0  34.0   21.0   379.55
4  2020-02-07      CR   San José   pm25    23  57.0  95.0   72.0   838.97

Does anybody have an idea as to how I can proceed?


Text mining for text matching I m new in text analysis and need your advice to help medical students to write properly and correctly. The students describe sicknesses as they  observe them; however, they must use an "official sickness description". I have the data collected from the students and the correct data. Let me explain:

I have a csv table1 containing 300K rows / one column : each row is the description of health condition written by students ( they can be redundant).

I have an other table2 containing almost 200K rows and 3 columns
Column 1: the official name of the condition ( the correct one the student should use)
Column 2: a code ( number)
Column 3: the translation of "the official name of the condition" in another language.

The idea is to start from table1 and match each row with N official names (with their codes and translation). Let's say 5 official names ranked by distance computation.

I am confused:
Should I go for a recommendation engine or a search/rank algorithms?

What are the steps you can identify to accomplish this task?

PS: the final output might be an API where student start to write the description of the sickness and obtain a list of choices/recommendations where they can pick one.
I hope I was clear ! Thanks

FuzzyWuzzy would be a good python library for this purpose. A code example from the GitHub readme:

>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)

I imagine you would like to use the extract function with limit 5 with choices being set equal to all the 'official names' from your table2 csv file.

Details on installation and use are at the link above.


How to implement Classification and Anomaly detection (C++) I am creating a system using C++(DX11) and i'm reading raw data into my program, i want to classify what the 3D data-set i'm reading in is and detect any anomalies it may have when compared to a database of the same types of item. I've not really done much in regards to machine learning aside from simple KMeans and linear regression solutions, what would be the best approach to implementing something like classification and anomaly detection in my program and what background reading/research might be required?

Currently, the program can display the raw data as a 3D image.

You can try using SVM if your dataset is medium-sized or small (500-10000). You train the model with two classes: normal or anomaly (your output would be the probability of being of one class or the other).

A great library in C++ is libSVM (https://www.csie.ntu.edu.tw/~cjlin/libsvm/) which allows to perform classification or regression tasks.

There is a grest section of Dr NG's coursera ml class that focuses on anomaly detection. I would expect your classes to be imbalanced (you have much more normal cases than anomalies so you need to think about how to deal with this issue, also how do you choose the probability threshold to classify data as anomaly or not)

There are other choices of course: K-means can be used + some rule or another classifier. I would suggest you watch the Andrew NG class try his method (pdf kernel estimation) and once you got the hang of it try another model (such as SVM).


Is there a way to aggregate frequency distribution for many runs? In the following plot

I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is 30 but I'm only showing 5 runs.
My question is is there a way to aggregate all the frequency distributions for all the fitness values among the 30 runs? so that I can only show a single plot for all the runs.

My question is is there a way to aggregate all the frequency distributions for all the fitness values among the 30 runs? so that I can only show a single plot for all the runs.

Yes absolutely, you can do that with boxplots or violin plots, one boxplot (or violin plot) for each fitness value. The height and shape shows the distribution of the values across runs for one fitness value, and allows visual comparison between the different fitness values.


Do I need an Artificial Intelligence API? I'm wondering if programmers tend to use AI APIs. And if so, what are they like? And where can I find a nice one for Java?

One needs to use an artificial intelligence (AI) API, if there is a need to add AI functionality to a software application - this is pretty obvious. Traditionally, my advice on machine learning (ML) software includes the following two excellent curated lists of resources: this one and this one.

However, keep in mind that ML is just a subset of AI domain, so if your tasks involve AI areas beyond ML, you need more AI-focused tools or platforms. For example, you can take a look at ai-one's AI platforms and APIs as well as interesting general AI open source project OpenCog.

In addition to the above-mentioned AI-focused platforms, IBM's Watson AI system deserves a separate mention, as quite cool and promising. It offers its own ecosystem for developers, called IBM Watson Developer Cloud, based on IBM's BlueMix cloud computing platform-as-a-service (PaaS). However, at the present time, I find this offering to be quite expensive as well as limiting, especially for individual developers, small startups and other small businesses, due to its tight integration with and reliance only on a single PaaS (Blue Mix). It will be interesting to watch this space as competition in AI domain and marketplace IMHO will surely intensify in the future.


Residuals in a gradient boosted classification I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at kaggle.

Can someone explain what are the residuals that are modeled in case of a classification scenario?

It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to a probability. The loss function used for deciding the weights of the terminal nodes is adapted from the normal sigmoid loss to not have to map directly to probabilities. I cannot find it at the moment but it should be easy to derive.

EDIT: I found the post that I read this from the other day, it's over at stats stackexchange: https://stats.stackexchange.com/questions/204154/classification-with-gradient-boosting-how-to-keep-the-prediction-in-0-1


R recommendations while applying ggplot2 and plotly together to plot a graph I have data frame having 3 columns namely index, Actual & Predicted (all column are numeric).
I am using 2 packages plotly & ggplot2 for plotting index on x-axis & Actual and Predicted on y-axis.
 The command for plotting the graphs goes as follows-

plot<- ggplotly(ggplot(dataframe, aes(x = index))+ geom_line(aes(y=Predicted), colour= "blue")+ geom_line(aes(y=Actual), colour="red"))

plot

The R was giving the plot smoothly without any recommendations, but suddenly it started giving recommendations like,

We recommend that you use the dev version of ggplot2 with ggplotly()
Install it with: devtools::install_github('hadley/ggplot2')

So I tried installing ggplot2 with devtools as given in the recommendations.

I am able to plot the graph but still I am getting the above recommendation.

What does that recommendation mean and what to do if i am getting the recommendation even after installing it with devtools ?

Plotly is R package for creating interactive web-based graphs via plotly's JavaScript graphing library, plotly.js. The plotly R libary contains the ggplotly function , which will convert ggplot2 figures into a Plotly object.

If you are still getting the recommendation even after installing it, it seems it was not properly installed. Please refer the below git hub link and code to do the same installation properly.

Try to via Hadley at https://github.com/hadley/ggplot2

# install devtools
install.packages('devtools')

# install dependency of scales package
install.packages(c("RColorBrewer", "stringr", "dichromat", "munsell", "plyr", "colorspace"))

# Open devtools package
library(devtools)

# move to development mode
# scales and ggplot2 are installed in "~/R-dev" directory, so official version of ggplot2 are not removed.
dev_mode(TRUE)

install_github("hadley/scales", force=TRUE)
# main branch of development
install_github("hadley/ggplot2")
# (Optional) install_github("ggplot2", "kohske", "feature/new-guides-with-gtable")

# Exit dev mode
dev_mode(FALSE)

# load development version of ggplot2
library(ggplot2)


How to use text classification where the training source are txt files in categorized folders? I have 200 *.txt unique files for each folder:

Each file is a lawsuit initial text separated by legal areas (folders) of public advocacy.

I would like to create a training data to predict new lawsuit by its legal area.

Last year, I have tried use PHP-ML, but it consume too much memory, so I would like to migrate to Python.

I started the code, loading each text file in a "json-alike" structure, but I don't know the next steps:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.base import TransformerMixin
from sklearn.pipeline import Pipeline
import os

path = 'C:\wamp64\www\machine_learning\webroot\iniciais\\'

files = {}

for directory in os.listdir(path):
    if os.path.isdir(path+directory):
        files[directory] = [];
        full_path = path+directory
        for filename in os.listdir(full_path):
                full_filename = path+directory+"\\"+filename
                if full_filename.endswith(".txt"):
                    with open(full_filename, 'r', encoding='cp437') as f:
                        files[directory].append(f.readlines())

Thanks in advance

Assuming that your folders are your classes, you can match any document with the correspondent tag.
Then, for every document:
1.- Normalize the text, i.e. remove stop words (unless they make sense), stem and / or lemmatize it (unless it doesn't makes sense).
2.- Vectorize the documents, you can choose TFIDF, BOW, word embeddings etc
3.- Depending on your documents train with an MLP (in case of BOW) or an LSTM in case of words embeddings.

When you have a new document, you need to repeat the procedure using the vocabulary you created for the training set.

I had a similar use case and was enough to use BOW with a multi layer perceptron, the accuracy was above 95%, but, documents were different for each category and I removed most frequent words because there where to common.

Another solution is to perform a topic modeling on documents binding those topics to the category and then training a simple classifier (an MLP or SVM will work)


What is an aspect in opinion mining? It's quite a challenging task of aspect extraction in the field of opinion mining if you look at the number of related papers.

But what is an aspect in the field of opinion mining?

It can be explained with a simple example:

EXAMPLE REVIEW: The dashboard finishing is not good, indicators, headlights,
parking lamps and ac vents are having impressive finishing
The overall car is good.

The statement "overall car is good" indicates that there is a mix of positive and negative opinion about the car. In the aspect of the dashboard, the opinion is negative. However, the review apprising the aspects "indicators, headlights,
parking lamps and ac vents".
It might be clear by this example that the aspects are the properties or factors of the target product or event.


Does convergence of loss function is always guarnteed? Which of the following is true, given the optimal learning rate?
(i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global optimum while stochastic gradient descent is not.
(ii) For convex loss functions (i.e. with a bowl shape), stochastic gradient descent is guaranteed to eventually converge to the global optimum while batch gradient
descent is not.
(iii) For convex loss functions (i.e. with a bowl shape), both stochastic gradient descent and batch gradient descent will eventually converge to the global optimum.
(iv) For convex loss functions (i.e. with a bowl shape), neither stochastic gradient descent nor batch gradient descent are guaranteed to converge to the global optimum
Which option is correct and why?

(iii), if you add this clause

provided an optimum or smaller than optimum learning rate and the training dataset is shuffled

Why
When we get the Gradient of the full batch, it's towards the global minima. So with a controlled LR, you will reach there.
With stochastic GD, the individual gradients will not be towards the global minima but it will be with each set of few records. Obviously, it will look a bit zig-zag. For the same reason, it might miss the exact minima point and bounce around it.
In a theoretical worse case, if the dataset is sorted on Class, then it will move in the direction one Class and then the other and most likely miss the global minima.
Reference excerpt from Hands-On Machine Learning

On the other hand, due to its stochastic (i.e., random) nature, this algorithm is much less regular than Batch Gradient Descent: instead of gently decreasing until it reaches the minimum, the cost function will bounce up and down, decreasing only on average. Over time it will end up very close to the minimum, but once it gets there it will continue to bounce around, never settling down (see Figure 4-9). So once the algorithm stops, the final parameter values are good, but not optimal."

When using Stochastic Gradient Descent, the training instances must be independent and identically distributed (IID) to ensure that the parameters get pulled toward the global optimum, on average. A simple way to ensure this is to shuffle the instances during training (e.g., pick each instance randomly, or shuffle the training set at the beginning of each epoch). If you do not shuffle the instances—for example, if the instances are sorted by label—then SGD will start by optimizing for one label, then the next, and so on, and it will not settle close to the global minimum.


Is it appropriate to use random forest not for prediction but to only gain insights on variable importance? For a binary classification problem, I have a small data set with 200 observations. There are around 20 potential variables but based on variance importance I think only 2 or 3 are important for classification. This data set is too small to train a random forest model for prediction purposes, but is it okay to run a random forest model and use the variable importance feature to understand which variables are important? If only a handful of features are by far the most important, then, I think despite the small data size, it's an appropriate approach in that it'll tell me what I want. The results can then be used to perhaps build a simple decision tree or as a preliminary analysis for a future larger test.
Any flaws with this thinking?

A Random Forest model can definitely be used to help you determine feature importances. Actually, it is used as a very common strategy for feature selection. If your data is too small, my recommendation would be to treat it as if you were to make predictions with this model, meaning that you should watch out for overfitting and do a proper hyperparameter optimization (you want a simpler and smaller model here), so that the features you find are really the most relevant, and don't just appear important for being biased by the small training set.


tensorflow: is there a way to specify XLA_GPU with tensorflow? The following code is used to specify device on which tf node is running

with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

I have already known:
this post, tensorflow doc and xla demo

What i want to know is:

Is there any way to specify XLA_GPU as the device on which tf node is running?

with tf.device('/XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

executing the code above gives ValueError: Unknown attribute: 'device' in '/job:localhost/replica:0/task:0/device:/XLA_GPU:0'

this is 100% reproducible on google colab.

Check that the device is there first by examining the device names in the output from the following:

from tensorflow.python.client import device_lib
device_lib.list_local_devices()

Once you've verified that 'XLA_GPU:0' is a device on your system use something like

with tf.device('/device:XLA_GPU:0'):

Obviously you can change the names.


Question About Coming Up With Own Function for Distance Matrix (For Clustering) Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game.

A lot of the features I plan on using are unique to this game (data that can only be analyzed if one knows the game well), and thus I believe that it is best for my data that I come up a new function to generate the distance matrix that I plan on using in the various clustering algorithms later on.

My data is just a mixture of continuous and categorical data, and instead of using Gower Distance for the reasons specified above, I wanted to come up with my own formula.

For example, when comparing the download date (a time period measured by the datetime object from pandas), I would like to cross reference the download date with the time periods of the various promotions that went on and adjust the similarity score for the download date accordingly.

Now, if I have 35 million different user data, and around 20 features (for each of the 35 million different users), is this a feasible method of calculating the distance matrix?

I'm asking because I've read that using the gower distance to calculate the dissimilarity matrix for a large dataset is not feasible due to the time complexity.

I was wondering if the method I described earlier is feasible in terms of time complexity, or am I just wasting my time?

Thank you. As someone who doesn't have much experience with data science and clustering, any input would be grateful!

Thank you.

It's trivial to see that you don't want to compute the distance matrix. Just compute the size of the entire matrix, even assuming zero memory overhead... 35 million x 35 million x 8 bytes per double /2 for optimizing symmetry. That is something like 5 Petabyte?

Forget anything that is not linear. Try to shrink your data first by weeding out the easy cases first: duplicates, single-time visitors where you have too little data to say much, etc.

Then begin experimenting with a manageable sample first. If you find something on a sample then it might be useable on the entire data, too. And if a method does not work in the samples it likely will not work on the entire data (but you'll have to wait much less to know it doesn't work).


What is stored in heap structure in the following example? I am planning to use heap structure to find the minimum distance between a set of 2D points and form a cluster.. and after to spend a couple of hours surfing on the internet, I have not still gotten a clearly example.

Imagine you have a set of 2d points (x1,y1; x2,y2;....;xn,yn) and then you compute all possible distances between these pairs of points to finally get a matrix with all distances values.

After that, I want to use a min heap structure to sort data and match the pairs with the mininum distance... and here is my question...

Which value/data is really stored at every root or leaves in heap structure? I know that the root of min Heap is the shortest distance but, the data stored in every globe at Heap graph represent the index of every distances of the pairs of points or the pairs of points?

In the first case, if you store the index value located at the top of heap, Do you need to perform after a new search in the distance matrix to obtain which pairs of points form these shortest distance? Is it optimal to do that?

I hope my question was clear and thanks for your support.

You would put triplets into your heap, storing

(distance, i, j)

where distance is the distance, i and j are the indices.

When you "pop" the topmost (smallest distance) element from the heap using the appropriate algorithm, the second smallest will now become the top.

Python has a module heapq for building min-heaps. Simply add tuples with heapq.heappush, and remove the smallest with heapq.heappop. To convert an array into a heap, use heapq.heapify (faster than calling heappush for every element).

Note that putting n^2 values into a heap yields runtime O(n^2 log n^2) which is fairly expensive.


Text Mining - Similarity among words to determine thresholds While trying to find how one place (Military High School) like a school is similar to another one (Military H School or Military High S), I used the LV algorithm to find the string distance and then converted them into a percentage on a calibrated scale.

stringdist(data[i,1],data[i,2],method = "lv")

Though this is admissible, I still need a more precise way to determine the threshold. The goal is to set a threshold of acceptance for a variation of a name from the standardized name, so that say if the threshold of this word is less than say 85%, it would be rejected. This exercise is part of data pre-processing.

Any suggestions.

Have you tried word2vec models ?. One of the main application is computing similar words, as they are very closer in the feature vector space. Thus apart from High_school and H_school, also secondary_school, public_school may also be obtained as closer words. The important thing is that a huge amount of sentences are needed for this unsupervised learning model.

Ref: gensim, word2vec in java


Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels are not exclusive, a headline could be about crime and politics at the same time. I would like to compare three algorithms for this problem. I use the following pipeline to predict the labels:

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])

parameters = [
    {"clf": [RandomForestClassifier()],
     "clf__n_estimators": [10, 100, 250],
     "clf__max_depth":[8],
     "clf__random_state":[42]},
    {"clf": [LinearSVC()],
     "clf__C": [1.0, 10.0, 100.0, 1000.0],
     "clf__random_state":[42]}
    {"clf": [GaussianNB()]}
]

rkf = RepeatedKFold(
    n_splits=10,
    n_repeats=2,
    random_state=42
)

cv = GridSearchCV(
    pipeline,
    parameters,
    cv=rkf,
    scoring='accuracy',
    n_jobs=-1)

The pipeline works fine for the random forest, but breaks at the LinearSVC() with the following error:

ValueError: bad input shape (20972, 20)

If I remove the LinearSVC(), it stops at:

A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

There is a list "Support multilabel:" in the sci-kit learn documentation on multilabel classification (https://scikit-learn.org/stable/modules/multiclass.html) and only the random forest is included. However, the documentation states that "Multioutput classification support can be added to any classifier with MultiOutputClassifier."

I am bit confused, do LinearSVC() and GaussianNB() support multilabel classification when wrapped in MultiOutputClassifier()? If not, is there a workaround?

When you run your grid search, the clf step of the pipeline is replaced by each of RandomForestClassifier, LinearSVC, GaussianNB; you never actually use the MultiOutputClassifier.

You should be able to just wrap the two offending classifiers with a MultiOutputClassifier. You'll need to prefix your hyperparameters with estimator__ to get through the MOC into the underlying classifier: clf__estimator__C.


How to build recommendation system that also takes time as a feature Let's say a User was Interested in football, so we were recommending him posts about football after few days or weeks he starts watching Baseball.

How'd we go about building a recommender system that also takes recency into consideration
Or
Let's say that it knows when a user was interested in what

Use a diminishing sample weight where the newest sample = 1 and older samples get lower weights. This way if a user changes his interests over time, your recommendations will change accordingly.


How to read Feature Based Grammar from a string To read Context Free Grammars we can use

nltk.CFG.fromstring("""
S-> NP VP`...
    """)

But it can't read a feature based grammar which is like?

S -> NP[NUM=?n] VP[NUM=?n]

So how to read feature based grammar from a string without actually having to save it in a fcfg file to load it later?

You need to take care of NP and VP expansion production.

# ###################
# Grammar Productions
# ###################
# S expansion productions
S -> NP[NUM=?n] VP[NUM=?n]
# NP expansion productions
NP[NUM=?n] -> N[NUM=?n]
NP[NUM=?n] -> PropN[NUM=?n]
NP[NUM=?n] -> Det[NUM=?n] N[NUM=?n]
NP[NUM=pl] -> N[NUM=pl]
# VP expansion productions
VP[TENSE=?t, NUM=?n] -> IV[TENSE=?t, NUM=?n]
VP[TENSE=?t, NUM=?n] -> TV[TENSE=?t, NUM=?n] NP


Drawing validation set from test set I am building a 3 neural network models on dataset that is already separated  to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train set. And this gives a certain limitation or maximum capacity to my neural network model(s). By this I mean, I can not seem to improve the accuracy even if I change the hyper parameters or the parameters of my models.

I have created 3 neural networks models and varied almost everything:

Number nodes/hidden layers,
Input features (performed feature selection and space reduction),
Activation functions and loss functions,
regularization, optimizer and more,

When I try to average the predictions of the 3 models, I don't see any improvements. Although I've read a lot that if I change such parameters I might have some uncorrelated models. But this wasn't the case for me because I always find correlation between my model predictions when I compute Pearson Correlation

After building all these models, I am pretty sure that the training set and test set are not drawn from the same distribution (i.e. they are not a random split of some full original dataset), which means that other features probably also have a different distribution.

Some proposed I could merge the training+test, but I don't want to do that as this dataset was developed in this way. But I would like to draw my validation set from the test set, is this possible? Can I use a validation set randomly drawn from the test set to tune models?

Forget that you are working with a neural network for a moment. Hopefully you are also taking into account time. If you were performing a ordinary regression and time was a component among others, you would have to apply an extrapolation penalty to your confidence intervals to penalize your model for deviated from the observed range.

Another possibility is that you have an intervention in your dataset. Meaning, it is possible that something occurred and what you need to truly do is a variance test between the training and the sample data.

Traditional regression handles minimization of variance around the mean, but there is something called median regression as well, which is design for this particular type of issue, when there are problems with dispersion in the model.


Classifying based on irregular number of features Ok, so I am trying to classify a rather large data set where the training set has some peculiar issues...

There are a different number of features available for each row. For example, I might have 10 data points to work with for one prediction and only 3 for another. I would want to somehow null the empty data points because they aren't necessarily 0. We just don't have the data.

Each feature is a triplet of numbers (A,B,C), (A1,B1,C1), (A2,B2,C2). Should I separate these into individual features or is there a way to keep them grouped?

There are a few features/variables which all rows will have.

Is there preferred methodology?

| Y | v0        | v1        | v2      | v3     | v4 | v5 | v6 | v7 | v8 | v9 | A | B | C  |
|---|-----------|-----------|---------|--------|----|----|----|----|----|----|---|---|----|
| A | 15,0,7    | 18,0,2    | 23,0,0  | 21,0,3 |    |    |    |    |    |    | 1 | 0 | 5  |
| B | 15,.05,12 | 14,.05,12 | 9,.3,45 |        |    |    |    |    |    |    | 2 | 2 | 12 |
| A | 18,0,2    | 23,.03,0  |         |        |    |    |    |    |    |    | 0 | 1 | 3  |
| A | 23,0,0    | 18,0.07,2 | 15,0,7  |        |    |    |    |    |    |    | 3 | 1 | 1  |
| C | 4,.08,212 |           |         |        |    |    |    |    |    |    | 6 | 4 | 4  |
| B | 14,.05,12 | 9,.3,45   |         |        |    |    |    |    |    |    | 1 | 3 | 9  |

The simple approach is:

To use a learning algorithm which can handle missing values. You should be careful about whether these missing values occur randomly or not though, as they could cause the model to be biased.
Triplets can be separated into individual features, there's no simple way to process them differently anyway.


How regularization helps to get rid of outliers? I have heard regularization helps to get rid of outliers, how so?
'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'.
Could you shed some more light on it?

You don't get rid of the actual outliers (no data reduction). But statistical methods can be robust against outliers.

  Robust statistics are statistics with good performance for data drawn
  from a wide range of probability distributions, especially for
  distributions that are not normal. Robust statistical methods have
  been developed for many common problems, such as estimating location,
  scale, and regression parameters. One motivation is to produce
  statistical methods that are not unduly affected by outliers.

Source: wikipedia.

So, L-1 regularization is robust against outliers as it uses the absolute value between the estimated outlier and the penalization term.

Whereas, L2-regularization is not robust against outliers as the squared terms blow up the differences between estimation and penalization.


Diff. in P-value & F-Stat. Multiple linear regression Even if we have individual p-values for each predictor.

Why do we need overall F-statistic?

I read this solution but I am not sure if I get it right. Can someone please explain?

Source: "An Introduction to Statistical Learning: with Applications in R" by James, Witten, Hastie and Tibshirani

The overall F-stat says if your model is significantly better than the naive model that only has an intercept at the average of all response data pooled together. In other words, it measures if $R^2$ is significantly better than 0.

In your case, you can eyeball the $R^2$ and see that 0.897 is going to be better than 0. The F-stat and p-value will be more useful when the $R^2$ has a subtle difference from 0.

Another place where an F-stat is useful to regression is testing against a less naive model than intercept-only. In fact, the p-values on each parameter in the regression are equal to the p-value you'd get by doing the F-test comparing the full model to a model with that parameter excluded (assuming the typical nice properties that go along with doing parameter inference in linear regression).

The reason not to examine the individual p-values for each parameter is because of false discoveries, the usual business about multiple testing. Doing one F-test of the entire equation removes the possibility of false discovery of a significant parameter, in exchange for not being able to pin down which parameters are significant.


What loss function avoids overconfidence? In the case of a neural net with a relatively small training data set, doing simple classification with categorical cross entropy (log loss), it is very easy for the results of the network to be "overconfident" because that improves log loss on the training data.

What I mean by overconfident (this is partly intuition and partly something that can be derived from Bayesian stats): given 1000 training examples in 2 equal classes, it is unreasonable for any classifier to give a probability of a new input as being in one class or the other which is substantially higher than (1-1/1000), even if the classifier is perfect on the training data.  That is because if the training set was increased by just one more example, the 1001st example could break whatever rule seems to have been followed in the data set up to that point, and that would force all the predicted values/probability estimates to be updated accordingly.

However, in practice trained neural nets with log loss function will output values which are arbitrarily close to 0 or 1 given a long enough training run, because that continues improving their performance on the training data way past the point that it makes sense to do so based on the size of the training data.

The question: how can the usual log loss function be modified so that making predictions extremely close to 0 or 1 does not hugely improve log loss compared to predictions which are only somewhat close?

You could add a penalty by adding a small "distance" to every loss calculation. They you would have a "floor" below which no comparison could be below. The tricky decision is how large to make the penalty. There is a large literature on how to choose the "right" penalty that keep your overconfidence low but maximizes the predictive ability.

If you have a copy of "The Elements of Statistical Learning", then look at page 61 and the 10 + pages following that. There are also other items in the index covering penalized methods. The Wikipedia article on the Lasso methods is also a possible starting point.

When I type this into an R console on a new Linux box I'm outfitting:

 ??penalized

I get:

Help pages:

brglm::profileObjectives        Objectives to be profiled
mda::gen.ridge      Penalized Regression
mgcv::bam       Generalized additive models for very large datasets
mgcv::bam.update        Update a strictly additive bam model for new data.
mgcv::gam       Generalized additive models with integrated smoothness estimation
mgcv::jagam     Just Another Gibbs Additive Modeller: JAGS support for mgcv.
mgcv::null.space.dimension      The basis of the space of un-penalized functions for a TPRS
mgcv::pcls      Penalized Constrained Least Squares Fitting
mgcv::smooth.construct.bs.smooth.spec       Penalized B-splines in GAMs
mgcv::smooth.construct.cr.smooth.spec       Penalized Cubic regression splines in GAMs
mgcv::smooth.construct.tp.smooth.spec       Penalized thin plate regression splines in GAMs
quantreg::rq.fit.lasso      Lasso Penalized Quantile Regression
quantreg::rq.fit.scad       SCADPenalized Quantile Regression
rms::pentrace       Trace AIC and BIC vs. Penalty
VGAM::sm.ps     Defining Penalized Spline Smooths in VGAM Formulas
VGAM::summarypvgam      Summarizing Penalized Vector Generalized Additive Model Fits

I would have gotten more if I were using my older machine ( a macpro that died recently) but I havent' installed an lasso-type packages yet. There are quite a few:

> rownames(available.packages()[  grep("lasso", available.packages()[,1]) , ])
 [1] "biglasso"      "cglasso"       "CVglasso"      "DIFlasso"      "dpglasso"      "elasso"
 [7] "extlasso"      "gglasso"       "glamlasso"     "glasso"        "glassoFast"    "GPCMlasso"
[13] "grplasso"      "hglasso"       "iilasso"       "ipflasso"      "lasso2"        "lassopv"
[19] "lassoscore"    "lassoshooting" "lmmlasso"      "nnlasso"       "palasso"       "ppmlasso"
[25] "prioritylasso" "PUlasso"       "sealasso"      "sglasso"       "Tlasso"


Support Vector Machines with soft margin: solving the dual form I am currently struggling with finding an analytical solution for the $\alpha_k$.
I have derived the following constrained optimization problem:
$$
L = \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i \alpha_j y_i y_j (\textbf{x}_j^T \textbf{x})
$$
$$
s.t. \quad 0 \leq \alpha_i \leq C \quad \forall i, \quad \sum_{i=1}^{N} \alpha_i y_i = 0
$$
I had, at first, not taken the constraints into account which, after taking the derivative w.r.t. $\alpha_k$, gave me:
$$
y_k \sum_{i=1}^{N} \alpha_i y_i (\textbf{x}_j^T \textbf{x}) = 1
$$
This system of linear equations I could easily solve in Python using numpy.
But as the alpha values were way too high (as could have been expected), I went back and found that I had forgotten about the constraints.
Now, I don't know how to find an analytical solution to that.
I have tried writing down the problem using Lagrange Multipliers but that doesn't seem to get me anywhere.
I have also looked around the Internet a lot and couldn't find a single lecture/slides/etc. that actually went on from that point.
Now is my question, is there a way to find an analytical solution to that constrained optimization problem or do I have to just put all of that into a solver?
And if there is a solution, I would appreciate some hints on how to get there.
Thank you for your Time!

if you want to find an analytical solution for the problem I think that it doesn't exist.
At this point you should apply decomposition (don't try to work on all your variables,
but iteratively considers only a subset of them called working set), if (and only if) you use a working set W, |W|=2, then your subproblem has an analytical solution.
Due to the large number of alpha variables, decomposition is the basis of svm implementations, in particular the svm light algorithm uses a working set W of cardinality 2.


AttributeError: 'DataFrame' object has no attribute 'ix' I was learning Classification-based collaboration system and while running the code I faced the error AttributeError: 'DataFrame' object has no attribute 'ix'. Here is the code I write until now.
X=bank_full.ix[:,(18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36)].values

From the docs:

Warning: Starting in 0.20.0, the .ix indexer is deprecated, in favor
of the more strict .iloc and .loc indexers.

Just use .iloc instead (for positional indexing) or .loc (if using the values of the index).
To read more about loc/ilic/iax/iat, please visit this question on Stack Overflow. To quote the top answer there:

loc: only work on index iloc: work on position ix: You can get data
from dataframe without it being in the index at: get scalar values.
It's a very fast loc iat: Get scalar values. It's a very fast iloc
http://pyciencia.blogspot.com/2015/05/obtener-y-filtrar-datos-de-un-dataframe.html
Note: As of pandas 0.20.0, the .ix indexer is deprecated in favour of
the more strict .iloc and .loc indexers.


Augmenting the Cost Function with a Gaussian Prior I'm doing an online course and one of the assignment says:

  To avoid parameters from exploding or becoming highly correlated, it is helpful to augment
  our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without
  constraining their direction, and often leads to classifiers with better generalization ability.

Exactly what does "augment our cost function with a Gaussian prior" mean, and how could I go about doing that?

I tried searching on google but didn't find anything helpful.

Let's look at the example of linear regression. Instead of deriving it from solving the normal equations, we can motivate it by thinking about it as finding the conditional distribution $P(y|x)$. Let's assume that this distribution follows a Gaussian distribution with fixed variance $\sigma^2$ and mean $\hat{y}(x,w)$ and weights $w$. Assuming that samples are i.i.d. we can easily show that applying maximum-likelihood gives the same values of $w$ than minimizing the mean squared error. See example 5.5.1 for a mathematical explanation.

If we additionally assume a prior Gaussian distribution for $P(w)$ (w.l.o.g. we can assume unit variance $I$), we can now show that the posterior $P(w | x,y)$  of $w$ is also Gaussian distributed (that is because the prior is "conjugate") with variance $\frac{1}{\alpha} I$ and we recover linear regression with weight decay $\alpha w^T w=\alpha \lVert w \rVert^2_2$  (i.e. $L^2$ or Tikhonov regularization).

Intuitively, a Gaussian distribution's probability mass is centered around the mean (assumed here to be zero). If we assume such a distribution for the coefficients in the linear regression (as our prior knowledge), linear regression will favor coefficients close to zero.

All of this is nicely explained here (chapter 5) or here (chapter 18).


Applying ConvNets to classify motion/video data How would someone go about using deep learning to classify sign language gestures? For example, suppose I had video files of many different gestures. For any given gesture, I might have many videos of it, and each video would be comprised of many frames. When trying to classify MNIST digits in images the dimensions of the input are comparatively simple: height, width, and RGB channels. How would gestures (multiple frames over time) be accounted for? Would time be a fourth dimension? What should the neural net's architecture look like so as not to overfit? Should I use something instead of a convoluted neural network?

Edit:

I know that there are probably clever ways to hand-code predictors for sign language, but I'm more interested in how to architect the neural net and take advantage of the time component of the data (where there is value in the transition of video frames over time). Classifying gestures is a simplification of the actual problem I'm trying to solve, so I'm looking for an approach that would be generalizable to other types of problems where it might be necessary to look at many frames of the video to predict the target variable.

Signs are the visual equivalent of "words", and just as words can be decomposed into smaller parts, so can signs be broken down into smaller meaningful/useful units. For example, in American Sign Language (ASL), signs are decomposed into parameters such as handshape, orientation, movement, location on the body. Each parameter can take on a finite set of values, for example handshapes can be closed fist, open hand, cocked index, etc. I would expect a neural network would need to learn these parameters and their values, while learning to ignore distractor and nonsense configurations. Perhaps initial training includes learning to differentiate between signs and non-signs. And just as there are different spoken language, there are different sign languages as well. Moreover, there exist distinct sign language dialects like Black ASL that will need to be taken into account by your system.


Real time topic identification of news article Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)).

Let's say that I'd like to group together all the articles that are speaking about the same thing. However :

I want my algorithm to define the topics by itself (in opposition to "the user tells the existing topics and the algorithm assigns each article to the topics"),
I can't know the precise number topics there are (because, obviously, a new one has to be created each time something new happens),
and, as we are talking about news article, the list of topics should be expanding in real time if something new happens and new articles talk about it.

For simplicity's sake, let's assume all the articles are in the same language.

I've been using TF-IDF for this as of now, because that's what I came up with on the top of my head when I knew nothing about topic modeling, but it's not really good at its job. I'm getting into topic modeling and discovering things such as Latent Dirichlet Allocation, Correlated Topic Model, graph of words etc. I'm currently reading through all I can on the subject, but I'm probably unaware of interesting algorithms that could fit my need.

I'm interested : what approach would you take to tackle my problem ?

Thanks

You are looking for either "online" or "streaming" topic modeling.  A hierarchical Dirichlet processes can automatically choose the number of topics. Here is a paper on streaming/online Latent Dirichlet Allocation model (LDA). A Python implementation of streaming/online can be found here. If your scale warrants it, Apache Spark has an implementation of online LDA also.


recommender system: how to compare different scores when calculated individually? I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).

So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.

For example, suppose that for a given customer:

score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?

I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.

Any ideas?

PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).

You can continue to use machine learning for this model-combination step too. You're describing a form of ensembling. How about:

set aside some purchases as your validation set V, and keep the remaining as training set TRAIN
for TRAIN, use k-fold cross-validation to create your separate models
use TRAIN again with k-fold cross-validation to ensemble the models together
report performance of the ensembled models on V

Ensemble techniques will take into account the predictive quality of each individual model.


How to write custom de-identification algorithm in Python? I have tried a simple algorithm to anonymize my data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values. The data sample is available here

import pandas as pd
import uuid as u
import datetime as dt

# generate a pseudo-identifier sequesnce using python random number generator library uudi.

 def uudi_generator(length):
    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
    i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('bankcredit-data.csv')

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([dt.datetime.now]*sLength, index=dataset.index))

 #transcation record is attached to originaL data file
dataset.to_csv('bankcredit-data.csv')

#delete identifiabLe record from dataset
del dataset['firstnamme']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('deidentified-data.csv')

Unless you want to build your own, try the Faker Library for anonymity of PPI info.

pip install Faker


Extracting and tagging/labeling text from pdfs I'm trying to extract paragraphs from 230 pdfs that have similar formats, and then label the paragraphs based on the pdf's section headings. Normally, I'd convert the pdf to txt files or extract the text using pdftools on R, but this would prevent me from distinguishing the text in headings -- which is bold, has a certain color -- from the text in the body that I want to "tag". I have access to Adobe Acrobat Pro, so I can convert the pdfs to .txt, .xml, .html, .rtf, or .docx files.

Does anyone have any suggestions on the best way to proceed?
If there isn't a straightforward answer to the first question, is there one file format among the ones listed above (aside from .txt, which would strip the formatting) that would be easier to parse than others?

Thanks!

Since the main goal is to extract headers(which has different characteristics from other text). It would be best to convert pdf to xml as you will get features like style,hpos,vpos for each text in the file.
To work with xml you can refer ElemenTree.


How to identify possible false negatives in labels? I am working on a drilling difficulty prediction project using machine learning method. The dataset has lots of geological data as the main features. It also has total drilled depth of each hole.
Not surprisingly, the holes that are seeing drilling difficulty had significantly longer drilled depth than the ones that did not have difficulty. However, the geological difficulty should not be dependent on the drilled length:  Based on particular type of rock, difficulty is always there no matter if it is drilled or not. If we drilled the holes that didn't have difficulty longer, I am sure lots of them will see difficulty eventually.
Having said that, the labels probably have a lot of false negatives (because the difficulty hasn't been encountered yet). How do I identify these false negatives in the labels?

I think your problem is not with identifying the false negatives, it's with defining the classification problem properly.
If I try to phrase what you're trying to predict, it would be something like this: "Given the data about a drilling point, what's the probability of encountering a difficulty up to depth $x$".
So, if you have a hole which didn't reach that depth $x$, it's not a false negative - it's just a missing label (no test... yet).
If you want to make your prediction based on the information known before drilling (or, at least, before drilling up to a given depth $z$) - limit your model to that information.


Can I install Tensorflow in Anaconda without using Keras? Can I install Tensorflow in Anaconda without using Keras?
If I can what is the difference between using Keras with Tensorflow and only Tensorflow?

Thanks..

Can I install Tensorflow in Anaconda without using Keras?

Absolutely.

  If I can what is the difference between using Keras with Tensorflow
  and only Tensorflow?

While Tensorflow  is a super powerful numerical computation and optimization library with lots of features for building neural networks, YET it is a bit tedious and nontrivial especially for beginners to use.

Here Keras comes very handy. In short, Keras is Tensorflow abstraction. It allows to quickly and easily define a neural net even complex ones with a few lines of codes. Also note that Keras not only runs on top of TensorFlow but also on top of Microsoft Cognitive Toolkit, Theano, or MXNet. Look at at this post where the blogger compares defining a typical neural network via directly in Tensorflow and Keras. You can see that with Tensorflow it is minimum 17 lines of code, whereas Keras reduce it to 10 lines of code.


Pandas copy() different columns from different dataframes to a new dataframe I have 2 dataframes that are coming from 2 different Excel files. I want to extract some columns from one file and other columns from the second file to print a new dataframe with the copied columns.
I copied 2 columns from different dataframes (df1 and df2) but I get print only one of them (the last one) in df3. How can get all of them in the df3?
    import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = df1[['ticker_symbol']].copy()
    df3 = df2[['ASX code']].copy()

    print(df3)

You are overwriting df3.
Method-1:
   import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = df1[['ticker_symbol']].copy()
    df3['ASX code'] = df2[['ASX code']].copy()

    print(df3)

Method-2:
    import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = pd.DataFrame()
    df3['ticker_symbol'] = df1['ticker_symbol'].copy()
    df3['ASX code'] = df2['ASX code'].copy()

    print(df3)
```