Python - Calculate Cost profitability and benefit of the model I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier:

coef_gini = DecisionTreeClassifier(criterion = "gini", random_state = 100, max_depth = 3, min_samples_leaf = 5)
coef_gini.fit(training_features, training_target)

y_pred = coef_gini.predict(test_features)
y_pred

for name, importance in zip(training_features.columns, coef_gini.feature_importances_):
    print(name, importance)

print ( "Train Accuracy using Decision Trees Classifier is : ", accuracy_score(training_target, coef_gini.predict(training_features)))
print ( "Test Accuracy using Decision Trees Classifier is : ", accuracy_score(test_target, y_pred))
print ( "Confusion matrix using Decision Trees Classifier is ", confusion_matrix(test_target, y_pred))

What is the cost matrix? Is this the money that company will lost for each wrong predictive target value? Anyone have an example?

Thanks!

Confusion Matrix

A Confusion Matrix is an important tool to measure accuracy of a classification algorithm. It compares predicted class of an outcome and actual outcome.

Scenario 1: Credit Risk

Based on a credit risk scorecard, application for credit card are classified as “Good” and “Bad”. “Good” indicates applicants paying back dues on credit card and “Bad” indicates customers defaulting on the dues. Now, the customers are compared against actual performance of the customer payment behaviour after say 18 months. So, comparison of Predicted Class (“Good” or “Bad”) to actual customer behaviour state (“Defaulted” or “Regular”).

There is always a trade-off between Type I error/False Positives (accepting Bad Customers) and Type II Error/False Negatives (Rejecting Good Customers).
We generally require to optimize between False Positive Rate (Type I Error) and False Negative Rate (Type II Error).
So, role of cost matrix comes in picture to find the optimal cut off value for a classification rule.  Now, going back to Credit Risk Model. The cut off value optimize between cost of an opportunity loss (miss to accept a good customer/Type II Error) and cost of accepting a potential defaulter (involved in loss due to default).
Referencing Example is from a blog and image from Google..

Cost Matrix

Cost Matrix is similar of confusion matrix. It’s just, we are here more concerned about false positives and false negatives .There is no cost penalty associated with True Positive and True Negatives as they are correctly identified.
The goal of this method is to choose a classifier with lowest total cost.

Total Cost = C(FN)xFN + C(FP)xFP

where,

FN is the number of positive observations wrongly predicted

FP is the number of negative examples wrongly predicted

C(FN) and C(FP) corresponds to the costs associated with False Negative and False Positive respectively.
Remember, C(FN) > C(FP).

Hope this Helps..


cv2.error: OpenCV(3.4.3) (-215:Assertion failed) !empty() in function Would anyone have a tip on how to fix this empty function error in OpenCV? I am attempting to follow the guides on OpenCV.org

The script will detect a face in an image and draw boxes around detections.

import numpy as np
import cv2 as cv

face_cascade = cv.CascadeClassifier('haarcascade_frontalface_alt.xml')

img = cv.imread('images/1.png')
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
    cv.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w]

cv.imshow('img',img)
cv.waitKey(0)
cv.destroyAllWindows()

I get an error:

>>>
=== RESTART: C:/Users/Desktop/distance-to-camera/selectHaar2.py ===
Traceback (most recent call last):
  File "C:/Users/Desktop/distance-to-camera/selectHaar2.py", line 11, in <module>
    faces = face_cascade.detectMultiScale(gray, 1.3, 5)
cv2.error: OpenCV(3.4.3) C:\projects\opencv-python\opencv\modules\objdetect\src\cascadedetect.cpp:1698: error: (-215:Assertion failed) !empty() in function 'cv::CascadeClassifier::detectMultiScale'

I think this may be due to the fact that XML file is missing or the path to it is incorrect. Could you please check that? Maybe you coul try with the full path for the haarcascade_frontalface_alt.xml file.


Advantages of Binary Rating System for Collaborative Filtering Recommender Systems I notice that Netflix, which I think used to use a five-star scale for rating content and give predicted ratings for unrated content on the same scale, now just has basic like/dislike buttons.  Music streaming services like Pandora and Spotify seem to employ the same approach.  My question is what makes a binary response superior to a likert-type scale when it comes to generating the best recommendations?

Is it related to the nature of the algorithms (e.g., classifications are more accurate when there are fewer possible responses)? Or is the advantage more to do with human psychology (easier for users to accurately choose between like/dislike as opposed to love/like/neutral/dislike/hate)?

This article from Business Insider answers your question from the business perspective.

They mention that the original problem is that the way Netflix used the 5-star rating was not standard within the industry:

  Netflix’s Cameron Johnson, who oversaw the shift, told Business
  Insider that it all came from the realization that Netflix had always
  used star ratings differently than the rest of the internet, but that
  this distinction wasn't clear to users.

And the way ratings were presented to the user discouraged them to contribute. Their solution was the thumbs up/thumbs down approach because it was more clear to the user that they were training an algorithm:

  So when looking for a replacement, Netflix wanted to make sure that
  was clear. That’s why Netflix settled on “thumbs up/down," which is
  widely understood to imply that you are training an algorithm to know
  what you like, Johnson said.

I suspect there is more than that thought, and maybe binary data is simply "good enough".

This chapter from Mining of Massive Datasets has some interesting considerations on ratings versus boolean utility matrices.

From page 339:

  If the utility matrix is not Boolean, e.g., ratings 1–5, then we can
  weight the vectors representing the profiles of items by the utility
  value. It makes sense to normalize the utilities by subtracting the
  average value for a user.

Specially when treating rating matrices one need to be aware that different users may have different understanding of ratings, hence you will most likely normalise the matrix before calculating the recommendations, while for binary data this is not required.


Are there libraries similar to cleverhans for different types of classifiers? I am seeking adversarial examples for classifiers (e.g. random forest, logistic regression, multilayer perceptron) that I've already trained (through scikit-learn). But I can't seem to find any existing libraries for this purpose. This could be a fun project for me to start, but before beginning to explore my naïveté, is there any literature/code on this topic that I should be considering?

Try foolbox

  Foolbox is a Python toolbox to create adversarial examples that fool neural networks. It requires Python, NumPy and SciPy.

Also, join the google group of cleverhans, it's really good. Plus Ian Goodfellow is there along with Nicolas Papernot.


How to convert Scikit Learn logistic regression model to TensorFlow I would like to use existing Scikit Learn LogisticRegression model in the BigQuery ML. However, BQ ML currently has a hard limit of 50 unique labels and my model needs to handle more than that.

BQ accepts TensorFlow models, which do not seem to have this limit.

How can I convert existing Scikit logistic regression model to TensorFlow model?

Sure, here is a skeleton in TF 2.0.

import tensorflow as tf

weights = tf.Variable(tf.random.normal(shape=(784, 10), dtype=tf.float64))
biases  = tf.Variable(tf.random.normal(shape=(10,), dtype=tf.float64))

def logistic_regression(x):
    lr = tf.add(tf.matmul(x, weights), biases)
    return lr

def cross_entropy(y_true, y_pred):
    y_true = tf.one_hot(y_true, 10)
    loss = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)
    return tf.reduce_mean(loss)

def grad(x, y):
    with tf.GradientTape() as tape:
        y_pred = logistic_regression(x)
        loss_val = cross_entropy(y, y_pred)
    return tape.gradient(loss_val, [weights, biases])


How can I use all possible spelling correction of documents before clustering those documents? I have the data set with many documents of 50 to 100 words each.

I need to clean those data by correcting misspelled words in those documents.

I have an algorithm which predicts possible correct words for misspelled word.

The problem is I need to choose or verify the predictions made by that algorithm in order to clean the spelling errors in the documents.

Can I use all the possible correct words predicted for correct spelling in word vector to perform clustering on those data?

The problem is that you cannot be a 100% sure that the predictions of the corrections of your algorithm will be true. Such an algorithm does not exist yet.

So you have 2 options :

Either you correct the algorithm errors by yourself, but it will be very long, and if your aim is to study the performance of a method it will bias the results. I would not recommand that.
Or you can just apply your algorithm and either set a probability threshold, i.e. you let the process correct the words if the probability is higher than an arbitrary value, say 75%, or correct all the words with the highest probability correction. You can then apply the classification model, and even if some corrections made were false, it should globally enhance the result, so it should be better than not correcting.


Dropout implies stochastic descent? The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD)

I've built a system which converges nicely with momentum at 0 and learning rate 0.01, even with 100 layers stacked. With dropout, the error decreases, but jumps up and down due to dropout knocking-out certain neurons?

Is it usual to see fluctuations of error during backprop with dropout, similar to SGD?

Would that  imply I can train on huge batches instead of minibatches, without the fear of overfitting by default?
For example, using Resilient Propagation (rProp) - in my test-example the LSTM trains in just 40 iterations using rProp, instead of 1000 using SGD

It depends a bit on your definition, usually the stochastic part of Stochastic Gradient Descent refers to the fact that you sample mini batches and estimate the true gradient with this sample. Dropout adds stochastic behaviour by sampling masks that comes down to sampling new network architectures every time. You could certainly see this as a form of stochastic gradient descent but whether this is the case or not depends solely on the definition.


Is GEMM used in Tensorflow, Theano, Pytorch I know that Caffe uses GEneral Matrix to Matrix Multiplication (GEMM) which is part of Basic Linear Algebra Subprograms (BLAS) library for performing convolution operations. Where a convolution is converted to matrix multiplication operation. I have referred below article.
https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

I want to understand how other deep learning frameworks like Theano, Tensorflow, Pytorch perform convolution operations. Do they use similar libraries in the backend. There might be some articles present on this topic. If someone can point me to those or can explain with an answer.

Those Frameworks are based on Cuda, which is the parallel computing tool for GPUs. Specificaly it is cuDNN that is used by the deep learning framework.

You can find the user guide : https://docs.nvidia.com/deeplearning/sdk/pdf/cuDNN-Developer-Guide.pdf

All off the operations are described and indeed GEMM algorithms are used for convolution operations.


Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning) I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me.

In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial a_j}\dfrac{\partial a_j}{\partial w_{ji}}$ (equation (5.50)), a notation is defined as $\delta_j\equiv\dfrac{\partial E_n}{\partial a_j}$ (equation (5.51)). In my understanding $\delta_j$ is the first part in the chain rule. However in equation (5.54), the book mentioned

  As we have seen already, for the output units, we have
  $$\delta_k=y_k-t_k$$

Question 1:
$y_k-t_k$ is the error on output unit $k$, which is simply the difference between the $k$th output unit value and the corresponding target value. But, from the definition of the notation $\delta_k$, we should have $$\delta_k=\dfrac{\partial \frac{1}{2}(y_k-t_k)^2}{\partial a_k}=(y_k-t_k)\dfrac{\partial y_k}{\partial a_k}=(y_k-t_k)\dfrac{\partial h(a_k)}{\partial a_k}$$ where $h(a_k)$ is the activation function.
So why in the book $\delta_k=y_k-t_k$??

In page 242, Section 5.3. Error Backpropagation,

  Consider a simple linear model where the outputs $y_k$ are linear
  combinations of the input variables $x_i$ so that
  $y_k=\sum_iw_{ki}x_i$. For a particular input pattern $n$, the error
  function is $E_n=\dfrac{1}{2}\sum_k(y_{nk}-t_{nk})^2$, where
  $y_{nk}=y_k(\boldsymbol{x_n},\boldsymbol{w})$. So the gradient of this
  error function with respect to a weight $w_{ij}$ is given by
  $$\frac{\partial E_n}{\partial w_{ji}}=(y_{nj}-t_{nj})x_{ni}$$ which
  can be interpreted as a ‘local’ computation involving the product of
  an ‘error signal’ $y_{nj} − t_{nj}$ associated with the output end of
  the link $w_{ji}$ and the variable $x_{ni}$ associated with the input
  end of the link.

Question 2:
I am not clear with the structure of this neural network. The one in the book is a two-layer neural network with linear activation, is it?

As you point out, with $\delta_k = y_k - t_k$, the author is stating the relationship between the final units' output and the target. So Equation 5.54 is simply stating:

  the error on the $k^{th}$ output unit is the difference between its output and the target.

I believe that $\delta_k$ refers to the error simply at the output, whereas $\delta_j$ is the derivative of the output at unit $n$ with respect to any neuron back in the network, $j$. If this is the case, doing your derivative as you did, for $\delta_k$ (instead of $\delta_j$) means you are computing the error gradient between the output and the final layer. Over this layer, the activation must be linear (we do not apply a non-linearity at the final layers output). This would mean your
$\dfrac{\partial h(a_k)}{\partial a_k}$ term would actually fall away in this case (to a constant), using the $k$ subscript - and you have the answer from the author: $y_t - t_k$.

The network that is considered in your second question seems to only really discuss a single fully-connected layer, so yes, two actual layers of neurons - input and output only with no activation (so a linear activation, as you mention).


Should columns with close to zero variance be removed before or after one hot encoding? I've been using R and the caret package since a while. The caret package provides a function to reduce the number of variables - nearZeroVar() wherein variables that have close to zero variance are returned and one can remove them from the dataframe

Caret also provides another function dummyVars that converts factors to dummy variables.

Now my question is - what should the order of using the two be? Should the features with near zero variance be thrown out and then converted to dummies or should it be the other way around? I have a feeling that the order matters because in some cases dummy variables might have a lot of zeros but still be important.

I think 1st you need to remove the columns with near to zero variance(assuming that they are not nominal variables), only if you have more variables as the computation time can be reduced.

Let us consider a scenario where there are 2 columns col1, col2 and you checked for variance then col1 had 0.003 variance and col2 had 0.5 var. once you apply the function nearZeroVar(), it removes col1, you need to hot-encode only 1 variable.

By that you can save time and your computer performance won't be effected.


How to Transpose dataframe with custom output? I have a data like this:

I want to transpose it like this in python:


neural network number of input neuron I want to predict using a multi layer perceptron. My data have 3 attributes x,y,z. My window size (input) is 3. That is, using 3 sets of values for x,y,z to predict next set of x,y,z. What should be my number of input neurons?

Should it be 6 input neurons for 3 sets of values of x,y,z?
or should it be 3 input neurons and each neuron has 3 inputs for x,y,z?

My understanding of the question is that you are using a basic MLP, feed-forward neural network, to work with data that is a 3-dimensional vector which you are calling $(x,y,z)$ - although note that most notation you read would call the whole observation $x$ which would be a vector $(x_1, x_2, x_3)$. You want to accept an input of 1st, 2nd, 3rd observations of this data (a window on some possibly longer series), and predict the 4th observation.

For the given architecture, the prediction is made immediately on the supplied input - a MLP has no "memory" for previous inputs - so you will need 3 x 3 = 9 inputs to represent each (x,y,z) in the series. For a MLP it doesn't really matter how you arrange this input - e.g. you could have $(x^{(1)}, y^{(1)}, z^{(1)},x^{(2)}, y^{(2)}, z^{(2)},x^{(3)}, y^{(3)}, z^{(3)})$ or $(x^{(1)}, x^{(2)}, x^{(3)},y^{(1)}, y^{(2)}, y^{(3)},z^{(1)}, z^{(2)}, z^{(3)})$ or any other arrangement as long as you are consistent for every training example and prediction.

You don't usually want or need to somehow combine the x, y and z values to create a pre-processed layer with 3 inputs, or create a special layer that combines $x^{(1)}, x^{(2)}, x^{(3)}$. You might do that if your understanding of how the sequence worked allowed you to come up with features that modelled something important. However, the more usual approach is to allow the neural network to figure out how the multiple inputs combine from the training data.

You may want to look at alternative architecture of Recurrent Neural Network (RNN), which has more options for dealing with predictions of series. For a RNN, you would have 3 input neurons, and you would run the network 3 times in sequence and read the prediction after running it the third time.

You have noticed correctly that a MLP has no "understanding" internally that your x in (x,y,z) at step 1 is the same type of variable as the x in steps 2 and 3. The MLP would have 9 separate inputs and would ignore any direct relations, or even that there is a sequence being predicted. Whilst in a RNN, both those details are built into the model as core assumptions. In some cases that could be an important advantage to the RNN. But it is still worth trying the simpler MLP first and testing whether its predictions are good enough for purpose.


XGBRegressor vs. xgboost.train huge speed difference? If I train my model using the following code:

import xgboost as xg
params = {'max_depth':3,
'min_child_weight':10,
'learning_rate':0.3,
'subsample':0.5,
'colsample_bytree':0.6,
'obj':'reg:linear',
'n_estimators':1000,
'eta':0.3}

features = df[feature_columns]
target = df[target_columns]
dmatrix = xg.DMatrix(features.values,
                     target.values,
                     feature_names=features.columns.values)
clf = xg.train(params, dmatrix)

it finishes in about 1 minute.

If I train my model using the Sci-Kit learn method:

import xgboost as xg
max_depth = 3
min_child_weight = 10
subsample = 0.5
colsample_bytree = 0.6
objective = 'reg:linear'
num_estimators = 1000
learning_rate = 0.3

features = df[feature_columns]
target = df[target_columns]
clf = xg.XGBRegressor(max_depth=max_depth,
                min_child_weight=min_child_weight,
                subsample=subsample,
                colsample_bytree=colsample_bytree,
                objective=objective,
                n_estimators=num_estimators,
                learning_rate=learning_rate)
clf.fit(features, target)

it takes over 30 minutes.

I would think the underlying code is nearly exactly the same (i.e. XGBRegressor calls xg.train) - what's going on here?

xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In xgboost.train, boosting iterations (i.e. n_estimators) is controlled by num_boost_round(default: 10)

In your case, the first code will do 10 iterations (by default), but the second one will do 1000 iterations. There won't be any big difference if you try to change clf = xg.train(params, dmatrix) into clf = xg.train(params, dmatrix, 1000),

References

http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train

http://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor


Reinforcement learning for continuous state and action space Problem

My goal is to apply Reinforcement Learning to predict the next state of an object under a known force in a 3D environment (the approach would be reduced to supervised learning, off-line learning).

Details of my approach

The current state is the vector representing the position of the object in the environment (3 dimensions), and the velocity of the object (3 dimensions). The starting position is randomly initialized in the environment, as well as the starting velocity.

The action is the vector representing the movement from state t to state t+1.

The reward is just the Euclidean distance between the predicted next state, and the real next state (I already have the target position).

What have I done so far?

I have been looking for many methods to do this. Deep Deterministic Policy Gradients works for a continuous action space, but in my case I also have a continuous state space. If you are interested in this approach, here's the original paper written at DeepMind:
http://proceedings.mlr.press/v32/silver14.pdf

The Actor-Critic approach should work, but it is usually (or always) applied to discrete and low-dimensional state space.

Q-Learning and Deep-Q Learning cannot handle high dimensional state space, so my configuration would not work even if discretizing the state space.

Inverse Reinforcement Learning (an instance of Imitation learning, with Behavioral Cloning and Direct Policy Learning) approximates a reward function when finding the reward function is more complicated than finding the policy function. Interesting approach, but I haven't seen any implementation, and in my case the reward function is pretty straightforward.
Is there a methodology to deal with my configuration that I haven't explored?

As you say, policy gradient methods work better than value-based methods (like DQN) with continuous action spaces.

Your problem seems very similar to the MountainCarContinuous environment. If you check the leaderboard, you can see how the top score uses A2C. You can find a clear implementation of the same algorithm (using the same environment) in the following notebook: Continuous MountainCar Actor Critic Solution.ipynb.

Here you can also find a collection of Policy Gradient Algorithms. For example, PPO has proven to be very successful in environments like the one that you describe.

TL;DR Look for Policy gradient methods.


Visualizing 28 different variables with 28 different colors? ColorBrewer seems to be very useful in selecting a color pallet to represent factors that have up to 12 possible values.

I have 28.

Is it a horrible idea to represent 28 variables with color? If so, could you suggest an alternative visual indicator?

Currently I'm using the colors for column side colors in a heatmap shown below. As you can see, the Strain column is not very informative:

This is the best solution I have come up with, I simply found a larger color pallet. The largest I could find was:

http://bl.ocks.org/aaizemberg/78bd3dade9593896a59d

So I merged d3 catagory 20 with the first few from d3 category20b which was inspired from this heatmap tools heatmap tool's source code:


Best way to narrow down a list and rank based on attributes? I have a mortgage/credit data set that contains a list of customers (600k rows) and has a 100 columns inclusive of the customer's general info (address, city, zipcode, etc), income, fico scores, number of current mortgages, mortgages in the past, aggregate mortgage amounts, number of bank card trades, etc. The data pertains to customers that are already good candidates to contact for issuing a credit product, however if one is to narrow down the list to 350K:

What would be the best method to rank the list to cut it down?

PS Your insights are much appreciated.

So it a little hard to know how to answer the question. Why do you need to cut the number of records down? If you want to rank them, rank them to what purpose? Ranking implies there are positives and negatives, or at least less positives.

One way you could approach this is to perform PCA. That said, I think you should consider disqualifying some fields, like the address fields.


Determining number of clusters in high dimensions I am doing KMeans clustering for sentence embeddings and my problem is the number of clusters. In general, feature size is an order of a few hundreds (in this case 768) and my concern is the sparsity of space. I tried to use gap statistic, but it just increases monotonically and has no maximum (I ended up with max 2048 clusters). Also, embeddings lie on a n-dimensional sphere rather than fill the space uniformly. My questions is: does it really make sense to use various "clustering metrics" to determine optimal number of cluster when the feature space is large?

There isn't a correct way to approach this problem. One common way is what you are doing, i.e.  check for various values of $k$ and have a heuristic tell me the best value. Some such methods are are the elbow, silhouette and the gap statistic that you're using. Determining the number of clusters via such a method is perfectly valid; in fact that's what they're for.

Another approach would be to try hierarchical clustering on the data and see which level leads to the maximum difference in variance.


Chose the right regression analysis In R I have data where head(data) gives

day   promotion   profit   new_users
1           105    45662          33
2            12    40662          13
3            44    46800          20
4           203    54102          46

Now day is simply the day (and is in order). promotion is simply the promotion-value for the day, the profit is the profit that day and new_users is the number of new users that day.

I want investigate the relationships between promotion to profit and new_users. We see a clear positive correlation between promotion and profit, and there is also a positive correlation between promotion and new_users.
In R I simply test correlation

cor.test(data$promotion, data$profit, method="kendall", alternative="greater" )
cor.test(data$promotion, data$new_users, method="kendall", alternative="greater")

which both gives a low p-value, ie we have a positive correlation.

I want to find a point where where the increase of promotion don't increase profit or new_users that must, ie a sweet spot.

Here is 2 plots and the R code for these

plot(data$promotion, data$profit, col="brown")
plot(data$promotion, data$new_users)

How should this be done?

My thoughts where to make a regression model.
For the first one "promotion vs. new_users" one could use a poisons model because it's a count-process, so a model like this would be a good chose?

glm(formula= data$new_users ~ data$promotion, family="poisson", data=data)

Next what regression model should one chose for the next one. Is it fair to say that this regression model is a good chose ? (I use sqrt command)

glm(formula=data$profit ~ sqrt(data$promotion) , data=data)

Or maybe it's not even necessary to use a regression model at all to find a sweet spot?

Thanks.

I have now looked at 'good' new users. For each day we have a promotion value and we have a count value which is the number of new good users. This plot shows us the number of good new users we get for a promotion for each day. For example for promotion value 90 we have a day where we got 8 new good users and a day where we got 14 new good users.

What would be the right approach to find a sweet spot for the use of promotion ?

Since I cannot comment because I don't have enough reputation, I will post this as an answer.

If your goal is to "find a point where the increase of promotion don't increase profit or new_users", I won't do a simple regression, since the regression will tell you that if you do more promotions, you will always inscrease the profits. I would say that, in reality, the relationship between promotions and profits or new users is not linear. Because the number of new users is limited and the promotions are not.

A better model is to say that there is a optimal promotion that will give you the best increase of profits and new users.

(if you have a real business to optimise, I would introduce the Customer Lifetime Value of new users. Because generally, the new users you get when doing huge promotions will not come back...)


Why should I know C++ ,if I am a machine learning engineer? I see there's a lot of machine learning job openings with skills requirements, python ,R, keras,tensorflow,pytorch,spark, etc.which are completely fine & reasonable, but why many of the recruiters include C++ ,like what is use of C++ in ML research, or even creating ML pipelines ?

How much C++ should I know if I'm good at rest of ML skills ?

Many machine learning products emphasize on speed over accuracy, e.g. in autonomous vehicles, a model that predicts in milliseconds with 98% accuracy is more valuable than a model that predicts in seconds with 99.99% accuracy.

While Python provides many great packages for exploring machine learning research and testing proofs of concept, it is significantly slower than C++ and not acceptable for products with real-time requirements. If you want to know why Python is slower, check out this thread.


Covert R dataframe column to Date I have a R dataframe having a column with dates in multiple formats.
I want to convert this column to date format.
While some formats are converted others are not.
Why is the second date not converted? How can this be corrected?
Code and result:
> slno <- c(1,2)
> Date <- c('22-04-2015', '4/8/2015')
>
> df_data <- as.data.frame(cbind(slno,Date))
> df_data
  slno       Date
1    1 22-04-2015
2    2   4/8/2015
>
> date_formats <- c('%m/%d/%Y', '%d-%m-%Y')
>
> #df_data$Date <- mdy_hm(df_data$Date)
> df_data$Date <- as.Date(as.character(df_data$Date), tryFormats = date_formats, optional = F)
>
> df_data
  slno       Date
1    1 2015-04-22
2    2       <NA>
> str(df_data)
'data.frame':   2 obs. of  2 variables:
 $ slno: chr  "1" "2"
 $ Date: Date, format: "2015-04-22" ...
>

The parse_date_time function from the lubridate package will do this. This function takes an argument orders which is like the tryFormats argument in 'as.Date', except the formats allowed are more generous and lubridate-like (without punctuation, delimiters, etc.). For example, you can set orders = c("dmy", "mdy").
Date <- c('22-04-2015', '4/8/2015')
parse_date_time(Date, orders = c("dmy", "mdy"))
"2015-04-22 UTC" "2015-08-04 UTC"


Compile See5 / C50 GPL Edition See5 / C5.0 is Data Mining Tools available from rulequest

I want to compile C50 for Linux, preferably for CentOS 6.x, but I am unable to compile. I have also tried on Ubuntu, but not success there as well.

I have downloaded C50.tgz from C5.0 Release 2.07 GPL Edition
After extracting when I run ./Makefile it gives below error on Ubuntu18

./Makefile: line 9: CC: command not found
./Makefile: line 10: CFLAGS: command not found
./Makefile: line 11: S: command not found
./Makefile: line 11: LFLAGS: command not found
./Makefile: line 12: SHELL: command not found
./Makefile: line 19: src: command not found
./Makefile: line 48: obj: command not found
./Makefile: line 59: all:: command not found
cat defns.i global.c c50.c construct.c formtree.c info.c discr.c contin.c subset.c prune.c p-thresh.c trees.c siftrules.c ruletree.c rules.c getdata.c implicitatt.c mcost.c confmat.c sort.c update.c attwinnow.c classify.c formrules.c getnames.c modelfiles.c utility.c xval.c\
        | egrep -v 'defns.i|extern.i' >c50gt.c
make: /bin/csh: Command not found
Makefile:75: recipe for target 'c5.0' failed
make: *** [c5.0] Error 127
./Makefile: line 61: CC: command not found
./Makefile: line 61: LFLAGS: command not found
./Makefile: line 61: -o: command not found
./Makefile: line 67: obj: command not found
./Makefile: line 67: c5.0dbg:: command not found
./Makefile: line 68: CC: command not found
./Makefile: line 68: obj: command not found
./Makefile: line 68: -g: command not found
./Makefile: line 74: src: command not found
./Makefile: line 74: c5.0:: command not found
./Makefile: line 75: src: command not found
./Makefile: line 77: CC: command not found
./Makefile: line 77: LFLAGS: command not found
./Makefile: line 77: -O3: command not found
strip: 'c5.0': No such file
./Makefile: line 82: obj: command not found
./Makefile: line 85: .c.o:: command not found
./Makefile: line 86: syntax error near unexpected token `newline'
./Makefile: line 86: `  $(CC) $(CFLAGS) -c $<'

If I run make command, it gives below error.

make c5.0
make: /bin/csh: Command not found Makefile:60: recipe for
target 'all' failed make: *** [all] Error 127

gcc version is:

gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0

After extracting C50.tgz file, give execution rights to Makefile.

chmod +x Makefile

Then install csh

sudo apt install csh

Run below command to check if csh is install and check csh version (if installed)

dpkg -l csh

It should show output like below.

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version              Architecture         Description
+++-==============================-====================-====================-==================================================================
ii  csh                            20110502-3           amd64                Shell with C-like syntax

Now run below command.

./Makefile

It should give output like below. (Ignore the warning messages).

./Makefile: line 9: CC: command not found
./Makefile: line 10: CFLAGS: command not found
./Makefile: line 11: S: command not found
./Makefile: line 11: LFLAGS: command not found
./Makefile: line 12: SHELL: command not found
./Makefile: line 19: src: command not found
./Makefile: line 48: obj: command not found
./Makefile: line 59: all:: command not found
cat defns.i global.c c50.c construct.c formtree.c info.c discr.c contin.c subset.c prune.c p-thresh.c trees.c siftrules.c ruletree.c rules.c g     etdata.c implicitatt.c mcost.c confmat.c sort.c update.c attwinnow.c classify.c formrules.c getnames.c modelfiles.c utility.c xval.c\
        | egrep -v 'defns.i|extern.i' >c50gt.c
gcc -ffloat-store  -O3 -o c5.0 c50gt.c -lm
c50gt.c: In function ‘ListAttsUsed’:
c50gt.c:14025:12: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
      Att = (Attribute) DefSVal(D[e]);
            ^
c50gt.c: In function ‘Error’:
c50gt.c:15561:17: warning: format not a string literal and no format arguments [-Wformat-security]
     fprintf(Of, Buffer);
                 ^~~~~~
strip c5.0
rm c50gt.c
./Makefile: line 61: CC: command not found
./Makefile: line 61: LFLAGS: command not found
./Makefile: line 61: -o: command not found
./Makefile: line 67: obj: command not found
./Makefile: line 67: c5.0dbg:: command not found
./Makefile: line 68: CC: command not found
./Makefile: line 68: obj: command not found
./Makefile: line 68: -g: command not found
./Makefile: line 74: src: command not found
./Makefile: line 74: c5.0:: command not found
./Makefile: line 75: src: command not found
./Makefile: line 77: CC: command not found
./Makefile: line 77: LFLAGS: command not found
./Makefile: line 77: -O3: command not found
./Makefile: line 82: obj: command not found
./Makefile: line 85: .c.o:: command not found
./Makefile: line 86: syntax error near unexpected token `newline'
./Makefile: line 86: `  $(CC) $(CFLAGS) -c $<'

You will see a new file c5.0*
Check command line options with below command.

./c5.0 -h

it should show below output.

C5.0 [Release 2.07 GPL Edition]         Fri Mar 15 17:43:39 2019
-------------------------------

    Options:
        -f <filestem>   application filestem
        -r              use rule-based classifiers
        -u <bands>      order rules by utility in bands
        -w              invoke attribute winnowing
        -b              invoke boosting
        -t <trials>     number of boosting trials
        -p              use soft thresholds
        -e              focus on errors (ignore costs file)
        -s              find subset tests for discrete atts
        -g              do not use global tree pruning
        -m <cases>      restrict allowable splits
        -c <percent>    confidence level (CF) for pruning
        -S <percent>    training sample percentage
        -X <folds>      cross-validate
        -I <integer>    random seed for sampling and cross-validation
        -h              print this message

Now C50 is ready to be used. Here is a sample command(assuming sampledata.data and sampledata.names files exists in the same folder).

./c5.0 -r -f sampledata

After successful execution of the command, sampledata.rules will be generated in the same folder.

See full tutorial from here


Beginner math books for Machine Learning I'm a Computer Science engineer with no background in statistics or advanced math.

I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of the Machine Learning, I wasn't able to understand the great book that a friend suggest me The Elements of Statistical Learning.

Do you know any easier statistics and math books for Machine Learning? If you don't, how should I move?

Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free.

Learn Statistics - Intro to Statistics Course
Intro to Descriptive Statistics
Inferential Statistics: Learn Statistical Analysis

If I want to recommend a book, I would recommend the following book which is free under CC license. It has nice examples and is so much practical; moreover, there are lots of codes in it which help you feel statistics in real world examples.

Think Python by Allen B. Downey
Python Data Science Handbook

Also the following link may help:

From Google Itself Good And Concise


Creating a custom dataset for object detection I'm currently trying to build a model to recognize approximately 10 labels (food items) in a fairly controlled environment (refrigerator). I was unable to find datasets that worked well enough for my task, so I'm attempting to curate one myself.

So far this has been my approach:

Capture images with a smartphone
Annotate in LabelImg
Train in Detectron2 / pyTorch

I have 2 questions:

Given that the environment is somewhat consistent throughout all samples, is there a ballpark number / rule of thumb for a decent number of samples to use per class? (100 train / 100 test)? This is a proof of concept project, so I'm just looking for something with reasonable accuracy (80%+)
After I've captured my images (let's say via smartphone) are there any preprocessing steps (other than annotating) that are necessary before using as training data? (i.e resize the images, reduce file size, format)

Absolutely any help is appreciated, and of course other tips/suggestions you feel may be useful.

I have built many data sets. The latest was a data set of species of birds. I has 100 species of birds so I had 100 classes. For each species (class) I had 100 training images, 5 test images and 5 validation images. Thats as total of 11,000 images.. The classifier I built had a final accuracy of 98% on 500 test images (5 test images per specie)
Here are some things to consider in building your data set.
1- crop the data or not- depending on the images you generated via your cell phone what is on average the ratio of the pixels of the region of interest (example a milk bottle in the refrigerator) to the total pixels in the image. Your classifier will always work better when this ratio is high. You want your classifier to learn to recognize a milk bottle as a class.Any part of the image that is not a milk bottle is essentially noise.  The rule of thumb I use to build a high quality data sets is 50%. That is that on average for your data set the region of interest should occupy 50% of the pixels in the image.
2- how many training- validation-test images should you have? depends on the problem but if you follow the 50% rule above I think you can build a solid data set with about 100 images per class, 5 validation images per class and 5 test images per class so you would need about 110 total images per class.
3- image augmentation - even with the small number of images you can create a good classifier using image augmentation. This is a way  of artificially expanding the size of your data set. If you use Keras, the ImageDataGenerator provides several transforms that can be used to augment your data set. Documentation is at https://keras.io/preprocessing/image/.
4-image size - CNN's operate with a fixed size for all input images. Again in general the larger the image the better the result(assuming the 50% rule is used). However you pay a price for large images in terms of computation time and memory usage. I have consistently used 224 X 224 X 3 (color images). Depending on the importance of color(ie how important is color to discriminating between classes) you may be able to just use gray scale images  224 X 224 X 1.

5- avoiding BIAS in your data set - make it diverse as possible.
   It is easy to accidentally build an unintentioned BIAS in your data set. Here is an example. A guy created a data set with 2 classes. One set had images was of various kinds of dogs. The other set had only images of wolves. He wanted to build a classifier that if given an image would classify it as either a dog or a wolf. He built his data set and trained the classifier with great results 99% training accuracy 98% validation accuracy. That is a surprising result since a lot of dogs very closely resemble a wolf. But when random images of a dog or a wolf were presented the accuracy was essentially 50%. WHy? Well he built a big bias in his data set. 90% of the training and validation images for wolves had a white background of snow. Since he did not crop the images most of the image was white. For the dogs his images were mostly of dogs NOT in snow. So what the CNN learned was "if background is white must be a wolf, if background is not white must be a dog. So think about avoiding building an unintentional bias in your data set. In your case say you have a class ketchup bottle. Now if all you take are images of HEINZ ketchup bottles (probably with the text HEINZ on the label) your model will not know anything about ketchup bottles that aren't Heinz ketchup bottles. Good luck I hope your task works out well.
One more thing, why are you considering doing all the work of taking cell phone images? Why not just do a Google search for the items of interest and download the images. For example do a search for "ketchup bottle images" You will find hundreds of images (with diversity which is important) that you can just download.


Minimum number of trees for Random Forest classifier I am searching for a theoretical or experimental estimation of the lower bound for the number of trees in a Random Forest classifier.

I usually test different combinations and select the one that (using cross-validation) provides the median best result.

However, I think that there may be a lower bound on the number of trees to use, given $m$ observations and $n$ features, to reduce the variance contribution to the error. Is there some test or reference I may check to correctly tune my classifier?

This is not necessarily an answer to your question. Just general thoughts about cross-validating the number of decision trees within a random forest.

I see a lot of people in kaggle and stackexchange cross-validating the number of trees in a random forest. I have also asked a couple of colleagues and they tell me it is important to cross-validate them to avoid overfitting.

This never made sense to me. Since each decision tree is trained independently, adding more decision trees should just make your ensemble more and more robust.

(This is different from gradient boosting trees, which are a particular case of ada boosting, and therefore there is potential for overfitting since each decision tree is trained to weight residuals more heavily.)

I did a simple experiment:

from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.grid_search import GridSearchCV
import numpy as np
import matplotlib.pyplot as plt
plt.ioff()

df = load_digits()
X = df['data']
y = df['target']

cv = GridSearchCV(
    RandomForestClassifier(max_depth=4),
    {'n_estimators': np.linspace(10, 1000, 20, dtype=int)},
    'accuracy',
    n_jobs=-1,
    refit=False,
    cv=50,
    verbose=1)
cv.fit(X, y)
scores = np.asarray([s[1] for s in cv.grid_scores_])
trees = np.asarray([s[0]['n_estimators'] for s in cv.grid_scores_])
o = np.argsort(trees)
scores = scores[o]
trees = trees[o]
plt.clf()
plt.plot(trees, scores)
plt.xlabel('n_estimators')
plt.ylabel('accuracy')
plt.savefig('trees.png')
plt.show()

I am not saying your are committing this fallacy about thinking more trees can cause overfitting. You clearly are not since you have asked for a lower bound. This is just something that has been bugging me for awhile, and I think it is important to keep in mind.

(Addendum: Elements of Statistical Learning discusses this at page 596, and is in agreement on this with me. «It is certainly true that increasing B [B=number of trees] does not cause the random forest sequence to overfit». The author does make the observation that «this limit can overfit the data». In other words, since other hyperparameters may lead to overfitting, creating a robust model does not rescue you from overfit. You have to pay attention when cross-validating your other hyperparameters.)

To answer your question, adding decision trees will always be beneficial to your ensemble. It will always make it more and more robust. But, of course, it is dubious whether the marginal 0.00000001 reduction in variance is worth the computational time.

Your question therefore, as I understand, is whether you can somehow calculate or estimate the amount of decision trees to reduce the error variance to below a certain threshold.

I very much doubt it. We do not have clear answers for many broad questions in data mining, much less specific questions like that. As Leo Breiman (the author of random forests) wrote, there are two cultures in statistical modeling, and random forests is the type of model that he says  has few assumptions, but is also very data-specific. That is why, he says, we cannot resort to hypothesis testing, we have to go with brute-force cross-validation.


How does Snowball Stemmer work? I have been reading about Snowball Stemmer.
I wonder how does it work?
Does it use rules to stem word
or does it use machine learning to do that?
I checked snowballstem.org but could not find the answer!

First of all, Snowball is not a type of stemmer, but a programming language used to implement stemmers.
The reason why the Snowball language was created was to make it more convenient to implement linguistic rules. So to answer your question, they are rule-based, not ML-based.


Model Selection using Bias Variance Trade Off I have a Regression Model with Train MAPE as 6% and Test MAPE as 15%.
This appears to me as a clear case of over fitting. But can I still use this model assuming 15% Error is not a bad number after-all. Is this there a flaw in this thinking?

Yes, assuming you haven't overfitted on the test set (which may happen after extensive hyperparameter optimization), you can assume that your model has a MAPE of 15%.
However, if you limit the overfitting, the test performance would probably go down!


What do "compile", "fit", and "predict" do in Keras sequential models? I am a little confused between these two parts of Keras sequential models functions. May someone explains what is exactly the job of each one? I mean compile doing forward pass and calculating cost function then pass it through fit to do backward pass and calculating derivatives and updating weights? Or what?

I have seen in some codes, they only used compile function for some of their LSTMs and fit for some other ones! So I need to know each of these functions do what part of the work(training a neural network).

It's also interesting for me to know what exactly do predict function as well.

Very thank you in advanced!

Let's first see what we need to do when we want to train a model.

First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile)
Secondly, we will want to train our model to get all the paramters to the correct value to map our inputs to our outputs. (fit)
Lastly, we will want to use this model to do some feed-forward passes to predict novel inputs. (predict)

Let's go through an example using the mnist database.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

Let's load our data. Then I normalize the values of the pixels to be between 0 and 1.

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

Now we need to reshape our data to compatible with Keras. We need to add an additional dimension to our data which will act as our channel when passing the data through the deep learning model. I then vectorize the output classes.

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Now let's define our model. We will use a vanilla CNN for this example.

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

Now we are ready to compile our model. This will create a Python object which will build the CNN. This is done by building the computation graph in the correct format based on the Keras backend you are using. I usually use tensorflow over theano. The compilation steps also asks you to define the loss function and kind of optimizer you want to use. These options depend on the problem you are trying to solve, you can find the best techniques usually reading the literature in the field. For a classification task categorical cross-entropy works very well.

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Now we have a Python object that has a model and all its parameters with its initial values. If you try to use predict now with this model your accuracy will be 10%, pure random output.

You can save this model to disk to use later.

# Save the model
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

So, now we need to train our model so that the parameters get tuned to provide the correct outputs for a given input. We do this by feeding inputs at the input layer and then getting an output, we then calculate the loss function using the output and use backpropagation to tune the model parameters. This will fit the model parameters to the data.

First let's define some callback functions so that we can checkpoint our model and save it model parameters to file each time we get better results.

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

Now we have a model architecture and we have a file containing all the model parameters with the best values found to map the inputs to an output. We are now done with the computationally expensive part of deep learning. We can now take our model and use feed-forward passes and predict inputs. I prefer to use predict_class, rather than predict because it immediately gives me the class, rather than the output vector.

print('Predict the classes: ')
prediction = model.predict_classes(x_test_reshaped[10:20])
show_imgs(x_test[10:20])
print('Predicted classes: ', prediction)

  Predicted classes:  [0 6 9 0 1 5 9 7 3 4]

The code to print the MNIST database nicely

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()


How to apply data binning on reviews data? I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or classification the only method to bin long-text data?

You can bin on Len of the data. you can also bin on Keywords you are interested in. You can use Countvectorizer with YellowBricks library to arrive at Frequency Plot. You can tokenize into N grams and find the frequencies. Depends on your problem statement that you are trying to solve. You can do Sentiment Analysis on each of the reviews and bin them into positive negative and neutral. There are many ways you can think off.

I hope this helps.


Machine Learning models in production environment Lets say a Model was trained on date $dt1$ using the available labeled data, split into training and test i.e $train_{dt1}$, $test_{dt1}$. This model is then deployed in production and makes predictions on new incoming data. Some $X$ days pass, and there is bunch of labelled  data that is collected in between $dt1$ and $dt1 + X$ days, lets call it $Data_x$.  In my current approach, I take  random samples out of  $DATA_x$ (take for e.g 80/20 split) ,

So,
$80\%$ of $DATA_x$ = $train_x$ (new data used to fine-tune the existing model trained on $dt1$)
$20\%$ of $DATA_x$ = $test_x$ (new data added to the $test_{dt1}$)

This process of fine-tuning repeated as time passes.

By doing this I get an ever expanding test set, as well as I prevent retraining the whole model (essentially I can throw away the old data as the model has learnt from it). The new model generated is just a fine-tuned version of the old one.

I have some questions, regarding this approach:

Are there any obvious drawbacks in doing this?
Would the model ever need to be completely retrained (forgetting everything that was learnt before, and training the model with new train/test splits ) after some time or can the approach I described above continue indefinitely ?
What should be the condition for swapping the existing deployed model with the newly fine-tuned model ?

I think this is a good approach in general. However:

Fine-tuning your model (online learning) depends a lot on the algorithm and model how well this works. Depending on your algorithm it might be wise to retrain the whole thing
Your sample space might change overtime. If you have enough data maybe retraining every few days/weeks/months over only the last year worth of data might be better. If your old samples don't represent the current situation as well having them included might hurt your performance more than the extra samples help
The biggest condition is if it's tested and how much downtime it involves, but in general swapping more times is better, and this can be automated


Semi-supervised learning for regression It is mentioned on this page that Label Propagation of scikit-learn can be used for regression also. However, nothing is mentioned on its API page.

How can I do semi-supervised learning for regression with Label Propagation or any other method?

It seems to be an error in the documentation as the target is validated to be a classification target.

 check_classification_targets(y)

The default label propagation would however just propagate the existing 'labels' i.e. regression values, so not much would be gained in this case.


CNN Architecture for Multiple Instance Learning I have a binary classification problem where I have a bag of documents (image files) that I need to classify - the bag that is, not the individual document. However, a bag can have a different number of documents and the combination of documents will determine the classification. E.g.

Bag1: A, B, C     label = Pass
Bag2: D, E                Fail
Bag3: F                   Pass

In Bag1, the presence of A, B, or C alone may not be enough to Pass, but together they do. In Bag3, F itself might be enough to pass.

I have already done OCR to represent all the text at the Bag level. For the images, all I can think is to average the pixel values of each image in the Bag to represent it as a single image, and then train with a CNN.

Is there any better architecture or method for handling something like this?

As I understand there is some information that needs to be present in order for the example to "pass".
I have a few questions as well as a suggestion.
First off do the information come in different forms? Meaning, different fonts, different kind of documents for the same information etc?
Is the cnn mandatory? Any more info, regarding the domain of the application or the nature of the problem that you could share?

Suggestions:

Since you have done the OCR I would suggest a pipeline of different models. Process the outputs of the OCR so they can create a secondary dataset in order to employ NLP. Meaning decode the OCRs to produce plaintext outputs, which would serve as the secondary data-set.
Another thing you could try is to identify the category of each document just like any other multi-class classification. After that you could, employ another classifier based on which categories pass your test (if there 's a discrete number) and continue like so.
Example: Identify Type A through Type F documents. Set the combinations that would serve as true labels, e.g occurrence of Type F, occurrence of Type A,B,C (any other combination would be False) and then train.

Finally there's a paper on a different domain, though an approach similar to theirs could be followed. They present a Bag-of-Visual Words (BOV) method for object-based classification in land-use/cover mapping.

  ...we can characterize an image by a histogram of visual-word count [6]. The visual vocabulary provides a “midlevel” representation which helps to bridge the huge semantic gap between the low-level features extracted from an image and the high-level concepts to be categorized

Object Classification of Aerial Images With Bag-of-Visual Words


Scrapping data from a CSV file So, I'm a rookie at web scrapping and Python in general, and I'm having quite a hard time learning it. I'm trying to get some data from a csv file, but so far I could only split lines and columns. I don't have the slightest idea of how I'm supposed to proceed from now on.

From a certain CSV file, I'm supposed to say which Scandinavian country (Sweden, Norway and Denmark) won the most gold medals in the following categories: curling, skating, skiing and ice hokey, starting from year 2001.

Here's what I've managed to do so far:

import requests
from bs4 import BeautifulSoup

url = "https://sites.google.com/site/dr2fundamentospython/arquivos/Winter_Olympics_Medals.csv"

# Usando requests
csv = requests.get(url).text

lines = csv.splitlines()

for l in range(1, len(lines)):
  columns = lines[l].split(',')
  print(columns)

Any comments will be greatly appreciated

CSV is a format meant to machine and human-readable. Most programming languages have support to use CSV's, and Python is certainly one of them.

There is a standard library that would help, you will find it here: https://docs.python.org/3/library/csv.html#module-csv

I would recommend however to install the pandas package, to read the CSV into a structure made for analysis. There is a tutorial here: https://www.datacamp.com/community/tutorials/pandas-read-csv


Kfold or cross_val_score I am all new with ML. I try to understand what is Kfold and cross_val_score.
I made this model:
RandomForestRegressor(max_depth=17,n_estimators=93,criterion='mse')
modelfinal.fit(xtrain1, ytrain1)
mrse_test = np.sqrt(mean_squared_error(y_pred=modelfinal.predict(xtest1), y_true=ytest1))

It gave me:

Train/test
R2
mae
mrse

Train
0.94
13.52
22.07

Test
0.605
34.57
56.592

Because on the train, it gave me 0.9 as R2 and only 0.6 on test.
I have been told  I should use Kfold cv to see if I have over overfitting or not.
But I just can't understand how it will help me, and what I did is right or not.
This is what I did:
cv=ShuffleSplit(5,test_size=0.2)
score=cross_val_score(modelfinal, X, Y, cv=cv)

This is what I get:
array([0.59618812, 0.57799759, 0.55214982, 0.63904499, 0.65027832])

What can I conclude from this?

Overfitting is visible when the performance on the training set is much higher than the test set, so you clearly had overfitting with your first experiment already.
Cross-validation is not a way to see overfitting by itself, it's a way to obtain a more reliable performance value by using several splits (removes chance factor) and using an overall larger test set (the full data).
One thing I notice in your experiment is that the max_depth parameter is set to 17 which is very high, especially in case your training set is not too large. This means that you're allowing the model to create very big and complex trees, so there is more chance that it would represent small patterns which happen by chance in the data. Try reducing this parameter value, hopefully this will avoid or at least decrease the overfitting. If it's not enough there might be some work to do at the level of features: too many features and/or not enough instances can cause overfitting as well.


Unskewing the Data with the PCA's Help I'm making some RFM Analyses (Customer Segmentation) and, in order to feed the RFM data to K-Means, I need to unskew the data, as K-Means works best when dealing with symmetrical distributions.

One of the best practices I've found so far is to use log transforms in order to unskew the data. However, after a bit of experimenting (and, hopefully, correct intuition), I've also found that PCA is working really well at unskewing the data, without the need of performing a log transform. Is this supposed to work or am I in danger of making incorrect assumptions?

On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?

On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?

This is your solution, this is the normal, most used and best practice on what to do with skewed data


How can I create .nii (nifti) file from 3D Numpy array I have a prediction numpy array. How can I make a .nii or .nii.gz mask file from the array?

You can this using nibabel:
import nibabel as nb

ni_img = nib.Nifti1Image(numpy_array, affine=np.eye(4))
nib.save(ni_img, "dicom_volume_image.nii")


Sklearn tfidf vectorize returns different shape after fit_transform() I'm new to ML and trying out basic samples using sklearn. I have achieved converting text (single dimension) to numbers using TF-IDF and got the predictions correct.

Now I have a different use-case to predict Work Hours based on Marital-Status and Relationship Status so my training data is now two dimension.

I have DataFrame total of 24420 rows * 2 columns of shape --> (24420, 2)

After this DataFrame is been passed to TfidfVectorizer.fit_transform() function it turns to shape --> (2, 3) Not sure why!

And while training mnb.fit(x_train_tf, y_train) it returns an error

  ValueError: Found input variables with inconsistent numbers of samples: [2, 24420]

You could see the initial DataFrame shape of 24420*2 it transposed to 2*24420

All code is up here.

Questions:

Why after fit_transform() the shape changes to (2,3) what happened to (24420,2) data? What does this denotes?
Why does the ValueError: Found input variables with inconsistent numbers of samples [2, 24420] occurs?
Why initial DataFrame shape changed to 2*24420?

What is wrong with the implementation?

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this -

>>> x_train_tf.toarray()
array([[ 0.70710678,  0.        ,  0.70710678],
       [ 0.        ,  1.        ,  0.        ]])

If you check what features you are getting, you will see -

>>> tfidf.get_feature_names()
[u'marital', u'relationship', u'status']

What's happening is, while passing dataframe, the TfidfVectorizer is only taking the column names and converting them into numeric form.

I don't think you need to use tfidf here. As far as I understand, your data is categorical text, so use pandas.get_dummies() instead of tfidf. This will convert your categorical data to numeric form which you can use for modelling.

>>> pd.get_dummies(x_train.iloc[:10,])
       marital-status_Divorced  marital-status_Married-civ-spouse  marital-status_Never-married  marital-status_Widowed  relationship_Husband  relationship_Not-in-family  relationship_Own-child  relationship_Unmarried
1521                   1                          0                              0                        0                       0                            1                         0                        0
2274                   0                          1                              0                        0                       1                            0                         0                        0
20209                  0                          1                              0                        0                       1                            0                         0                        0
5529                   0                          1                              0                        0                       1                            0                         0                        0
27639                  0                          1                              0                        0                       1                            0                         0                        0
26670                  0                          0                              1                        0                       0                            1                         0                        0
16635                  0                          0                              0                        1                       0                            0                         0                        1
30824                  0                          0                              1                        0                       0                            1                         0                        0
15181                  0                          0                              1                        0                       0                            0                         1                        0
9119                   1                          0                              0                        0                       0                            0                         1                        0

Here, for demo purpose, I have taken only 10 rows.


Loss being outputed as nan in keras RNN Since the first Epoch of the RNN, the loss value is being outputted as nan.

  Epoch 1/100
  9787/9787 [==============================] - 22s 2ms/step - loss: nan

I have normalized the data.

    ...,
    [9.78344703e-01],
    [1.00000000e+00],
    [9.94293976e-01]]])

  Example of my X_train (float64 of size (9787,60,1))

-

array([6.59848480e-04, 6.98212803e-04, 6.90540626e-04, ...,
   1.00000000e+00, 9.94293976e-01, 9.95909540e-01])

  Example of my y_train (float64 of size (9787,))

My RNN:

# Initialising the RNN
regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True, input_shape =
(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))

# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

# Adding the output layer
regressor.add(Dense(units = 1))

# Compiling the RNN
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32)

It could possibly be caused by exploding gradients, try using gradient clipping to see if the loss is still displayed as nan. For example:

from keras import optimizers

optimizer = optimizers.Adam(clipvalue=0.5)
regressor.compile(optimizer=optimizer, loss='mean_squared_error')


Interpretation of scikit-learn one class svm scores How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way to tell when one sample is "more anomalous" than other? The predict() and decision_function() functions have sign information, yet the score_samples function does not have this.
from sklearn.svm import OneClassSVM

X = [[0], [0.44], [0.45], [0.46], [1]]
clf = OneClassSVM(gamma='auto').fit(X)

clf.predict(X)
# array([-1,  1,  1,  1, -1])

clf.score_samples(X)
# array([1.7798..., 2.0547..., 2.0556..., 2.0561..., 1.7332...])
```

The sample_scores values, along with a cutoff threshold value, are used to determine whether a value is an outlier or not. You should be careful if you try to compare these sample_score values to see which values may be more anomalous than others. If you are looking to eliminate outliers solely to eliminate them from your data-set, using this value alone on a outlier would not be reason enough to eliminate it.


Data prediction book While I was studying, few years ago , one of the most interesting topic was evolution, genetic algorithms and neural networks. Many of the problems I faced could be solved by using that knowledge.

I assume that since that time world has found more interesting algorithms, do you recommend some books worthy of reading in that domain ?

Mainly I am looking a way to find patterns in huge amount of data.

Ex. Having energy consumption for few years for one building.

Lets have an algorithm that is trying to find all possible repeatings in many variations.

Obviously at the beginning it should find that at the weekend energy consumption is less then average between Mon and Fri, but is that possible that an algorithm would tell me sth like this ? :

Every Friday at third week in even months user sleeps for 3 hours and in uneven months 5 hours ?

Or the algorithm finds it self that user like to save energy so if previous month he sees that he spent more then usual , next month he is trying to spent less,

or lets assume that user eats breakfast at work but if he eats it at home then he will stay whole day at home, then lets assume that he has off day, then check energy usage after breakfast if is high, which means user is preparing to leave or is small which means that he wants to sleep and basically stays home.

So I was wondering if this possible to auto detect this kind of patterns ?

I am enthusiast of c# and interested in R.

please refer this question

Books recommended by Yann Lecun can be found here and by G. Hinton here


Handling features with multiple values for clustering Suppose I have a movies dataframe in pandas. One of the features is Genre.

It has a list of genre names. for example:

Movie_ID    Genre

1        [Action, Thriller, Drama]
2        [Romance, Comedy]
3        [Action, Romance]

How do I use this column for a clustering problem, say K-means clustering?

You can easily use get_dummy function in Pandas to convert them to numerical vectors.

The idea is that categorical variables do not have a numerical intuition e.g. when it comes to the definition of Distance. But just imagine you have one feature Genre with 3 values Comedy, Romance and Crime. Then you can model them in a 3-dimensional space by saying Comedy = (1,0,0), Crime = (0,1,0) and Romance = (0,0,1). It replaces 1 feature with three but intuitively works well.

Update

I just understood your question after editing it! It was a bit fuzzy previously. But I keep my initial answer and add an update.

In this case use the values of the feature Genre (unique values of union of all genre sets in that column) as new features and determine their presence with 1 and 0 otherwise. Should work.

Movie_ID    Action  Thriller  Drama  Romance  Comedy

1             1       1         1       0       0
2             0       0         0       1       1
3             1       0         0       1       0


Choosing class labels from annotated data For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500  of them are same posts.  How should I decide the category label now?

For example,

AnnotatorId  Post  Labels
-------------------------
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E

Will the label for post1 be A,B,D through majority voting? Or, is there some better, commonly used approaches?

That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric:
\begin{eqnarray}
\kappa &=& \frac{p_{o}-p_{e}}{1-p_{e}}\\
       &=& 1 - \frac{1-p_{o}}{1-p_{e}},
\end{eqnarray}
where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.


Why the Silhouette Score and optimal number of Cluster changes when using 2D and 3D data? I am experimenting with Kmeans clustering. My data (vectors) was in 300 dimensions which I am converting into 2D and 3D using PCA. Now, to find the optimal number of clusters, I used the Silhouette score. However, for 2D the best Silhouette score is showing for 3 clusters (silhouette score = 0.45), and for 3D it is showing 9 clusters (silhouette score = 0.3861).
I want to know whether it is normal? If yes, what is the reason for this? What should I choose 2D or 3D?
Also, the reason for experimenting with 2D and 3D is because I wanted to plot the 3D graph using seaborn.

Yes it can happen. In fact it is quite normal since there are different clusters in 2D and different in 3D, since more or less information is added to data (by having more dimensions). This is a by-product of the curse of dimensionality.
Adding as more relevant information as possible would make clusters more close to underlying groups. So 3D would be better than 2D. This is a general observation. There are of course cases where projecting data in a low-dimensionalal manifold is indeed better since it can eliminate noise and/or capture specific attributes better than clustering on all (possibly irrelevant) dimensions (another by-product of the curse of dimensionality).
If the relevant information in your data has low dimensionality but this information is correlated along many dimensions in the original data then a feature extraction method is needed in order to capture the low-dimensional relevant information from original data (eg PCA, ICA ,..).
For some references along this direction see for example:

How to cluster in High Dimensions
An investigation of K-means clustering to high and multi-dimensional biological data
How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?


How to preprocess data in a data table? (e.g. change a value in each row for some feature) I Have a general question - how to preprocess data in Orange. E.g. I want to do something with each row value of some feature. How can I do this visually? And how can I do this via Python Script?

So far, I have only some ideas how to do this via python script:

Here is my code:

minimum = min(list(map(lambda row: row[0], in_data))).value

from copy import deepcopy
out_data = deepcopy(in_data)

for row in out_data:
    row[0].value = row[0].value - minimum

But it fails, because seems like the Value object is immutable...
Is there any way to create my own Value object with new data, and then do something like:

out_data = list(map(lambda row: Value(row[0].variable, [row[0].value - minimum]) + row[1:], in_data))

I am also interested in a "Visual widget-based" alternative, if it exists...

To scale the data, you can use Preprocess widget. To construct new features, e.g.

new_feature = old_feature - some_values

you can use Feature Constructor widget.


Find the column(s) name where the value of the variable matches a regex I am trying to filter a data frame where I need to search for a regular expression within the data frame.

dim(df)
[1] 10299 561

I wanted to know if there is a way, I could find the name(s) of the column(s) where a particular regex may be available.

As an example, how do I find the name of the column where 'rsi' is present. I am referring to iris.

Would this work for you,

 names(iris)[apply(iris, 2, function(x) any(grepl("rsi", x)))]


Overfitting for minority class after SMOTE w/ random forests I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained.

However, I am now getting most results as class 1 when I test my model. I just tried to test it on the training set and this is what I got:

Without SMOTE

With SMOTE

I've also tried hyperparameter optimisation, but that hasn't worked

Thanks

PS: Used SMOTE implementation in pandas with UnbalancedDataset library

smote algorithm depends on the data set you have.
If you have severe data imbalance, like the one you have in your case smote algorithm may not be able to help if the variations within the minority class is very high and the similarities  between the two classes is very high.
But How to know if this is the case.
Try to duplicate samples from the minority class train a non linear svm and check the results if the classification accuracy is very low then this the case.

Smote use knn to create new samples but if the variation within the minority class is very high then using smote will use samples that are not real neighbors even.
To be honest with you , there is no clear solution for this problem but i can suggest the followings:
1. Try borderline smote : it is a modified version of smote algorithm
2. Try smote boosting : it is a modified version of adaboost where adaboost algorithm is augmented with smote
3. If you can modify the smote boost to consider  borderline smote instead of smote


How is it possible for RNN to do sentiment analysis? I'm wondering how RNN can be used when doing sentiment analysis.
It seems that the characteristic of RNN is to remember what appeared in the past and determine the value of the present (future), but I think this RNN's characteristic is useless when analyzing sentiment.
I am curious to see how it is possible to do emotional analysis by remembering the words that have appeared so far.
I think the way to train vectors that show how many times specific words appear is the only way to analyze sentiment.
If some RNN model train a sentence "good movie" and label "positive", can it predict a new sentence "movie good" as a "positive"?
Also, if a normal model(just fully connected deep neural network) train a sentence "good movie" and label "positive", is it hard for this model to predict a new sentence "movie good" as a "positive"?

RNNs do not learn to predict sentiment. They learn correlations between the input data and the target labels. If they see that every time the input contains the word "bad" they have to generate the label "negative", then they will learn it. If they see in the training data that the previous phenomenon happens always except when there is a "not" before "bad", then they will learn it. Depending on how the data distribution is, they may generalize the negation to any combination of verbs and adjectives, or maybe not, and they only handle negation appropriately when the input data is very very similar to the training data.
Your question seems to aim at understanding the "internal dynamics" of the RNN when it is predicting sentiment. While there is some research in that direction, I think they can still be considered as "black boxes", in that we do not actually understand the functions modeled by neural networks and therefore their outputs are not "explainable"


Origin of the Boolean Model of Information Retrieval Simple question, but I can't really find the answer to that: Who "invented" Boolean Retrieval? Of course, I assume that the concept grew over time, but is there a paper or publication that mentions/defines the Boolean Model as a whole for the first time?
On Wikipedia, the book by Lancaster and Fayen (1973) is cited, but I couldn't find any definition in there, either.

Looking for "Boolean Retrieval" in google scholar with upper date limit finds two papers that seems related from the sixties.
https://heinonline.org/HOL/LandingPage?handle=hein.journals/juraba8&div=16&id=&page=
https://apps.dtic.mil/sti/pdfs/AD0673362.pdf
There is a reference to Non-Boolean retrieval from 1963, hinting that the term was known before.
Try checking the references in these papers.


Keras y-labels range between 0 and 1 instead of binary? I have X values and corresponding y-labels, until now I used to round my labels <0.5 to 0 and >0.5 to 1. Is it possible to use values between 0 and 1 for "y train"? Using Keras and Tensorflow. Appreciate any suggestions.

Assuming y_train is already in range [0, 1], it should be enough to specify the correct loss function while compiling a model (model.compile).

Loss functions can be found here.

Chose one which is not for categorical variables (since you have a range [0, 1]).


Guys I am trying to create a application to create training data import os
import cv2
import matplotlib.pyplot as plt
from colorama import Fore
import random
import numpy as np
import pickle

training_data = []
def create_training_data():
path = os.getcwd()
path_dataset = os.path.join(path,'dataset')
categories = ['a','b']
for category in categories:
    path_categories = os.path.join(path_dataset,category)
    index = categories.index(category)
    for img in os.listdir(path_categories):
        try:
            path = os.path.join(path_categories,img)
            img_array = cv2.imread(path,cv2.IMREAD_GRAYSCALE)
            training_data.append([img_array,index])
        except Exception as e:
            print(Fore.RED+path)
    print(Fore.GREEN+f"imported all data from cateogry {category}")

#For every object in the training dataset
#training_data[element][data type*]
#0 - Stores the values of all images
#1 -  stores the values of all category

create_training_data()

print(training_data[1201][1])
plt.imshow(training_data[1201][0],cmap="gray")
rows,columns = training_data[0][0].shape
print(f"rows-{rows} cols-{columns}")
plt.show()
random.shuffle(training_data)
X = []
Y = []

for feature,label in training_data:
X.append(feature)
Y.append(label)

X = np.array(X).reshape(-1,rows,columns,1)

pickle_out = open(X.pickle,'wb')
pickle.dump(X,pickle_out)
pickle_out.close()

pickle_out = open(Y.pickle,'wb')
pickle.dump(Y,pickle_out)
pickle_out.close()

After I run this script I am getting an error:

Traceback (most recent call last):
  File "trainingdata.py", line 55, in
    pickle_out = open(X.pickle,'wb')
AttributeError: 'numpy.ndarray' object has no attribute 'pickle'

The error in the line pickle_out = open(X.pickle,'wb') is that the name of file must be in quotes. So the correct code would be:

pickle_out = open('X.pickle','wb')
pickle.dump(X,pickle_out)
pickle_out.close()

Do the same for Y also. Also you can do it more concisely in the following way:

with open('X.pickle', 'wb') as f:
    pickle.dump(X, f)


Big Data - Data Warehouse Solutions? I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo.

Many times I find myself need to perform "joins" across databases, for example a student ID that appears in multiple databases with data that I want to gather. The joins are usually done after data is streamed out of the database, since the data is not located in the same database, and this sometimes requires hours just for thousands of records.

Can services such as AWS redshift or Google BigQuery allow you to somehow "import" data from many data sources and then you can perform SQL queries to join them?

How about Hadoop and Hive? Where we steam data out from the database and place it as files in Hadoop, and let Hive Query the data?

Can services such as AWS redshift or Google BigQuery allow you to
  somehow "import" data from many data sources and then you can perform
  SQL queries to join them?

It depends on your data and the type of joins you are performing. But, yeah, databases like Redshift can definitely perform better in your use case as they are column-based databases. Read this post and the associated answers for understanding how columnar data stores handle data.

  How about Hadoop and Hive?

Hadoop + Hive is mostly a DIY hosted/cloud version of what Redshift gives you on cloud.


Modeling when the response variable has too many 0's and few continuous values? For problems where the data represents online fraud or insurance (where each row represents a transaction), it is typical for the response variable to denote the value of fraud committed in dollars. Such a response value might have less than 5% non-zero values denoting fraudulent transactions.

I have two questions regarding such a dataset:

What algorithms can we use to ensure that the model not only predicts the fraudulent transactions accurately, but also predicts the value of fraud associated with these.
Assuming that we can quantify the cost involved in each false positive (tagging a non-fraudulent transaction as fraudulent) and cost incurred due to a false negative (tagging a fraudulent transaction as non-fraudulent), how can we optimize the model to maximize savings (or minimize losses)?

How about

Ordinary Least Square (OLS) regression? Since you have a class imbalance, you might want to combine that with boosting algorithms.
If you have a function to quantify the cost involved with FP's and FN's, use any optimization technique you can find. My favorite is genetic algorithms. You may also try linear programming.


How can I save a final model after training it on chunks of data? After training a model on chunks, how can I save the final model?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

Saving a model like this will just give me the model trained on the last chunk. How can I avoid that and get the overall model trained on every chunk?

UPDATE:
 Most of the real-world data sets are huge and can’t be trained in one go. How can I save a model after training it on each chunk of data?

df = pd.read_csv(“an.csv”, chunksize=6953)
for chunk in df:
  text = chunk[‘body’]
  label = chunk[‘user_id’]

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3 )

  text_clf = Pipeline([(‘vect’, TfidfVectorizer()),
  (‘tfidf’, TfidfTransformer()),
  (‘clf’, LinearSVC()),
  ])

  text_clf.fit(X_train, y_train)

  # save the model to disk
  filename = ‘finalized_model.sav’
  joblib.dump(model, filename)

Will saving it this way give me the model trained on the entire dataset? I want the model trained on every chunk. Any help?

One option is Dask framework's incremental learning. That would automate the loading of chunks and applying partial_fit in scikit-learn.
The code would be something like:
from dask_ml.datasets import make_classification
from dask_ml.wrappers import Incremental

X, y = make_classification(chunks=25)
estimator = LinearSVC(random_state=10, max_iter=100)
clf = Incremental(estimator)
clf.fit(X, y, classes=[0, 1])

joblib.dump(clf, filename)


Training loss stuck in the starting epochs but then starts decreasing. What could be the reason for it? I am training a model where I found a unique problem that for starting 4 epochs, my loss did not change with the epochs but after that, it started changing. Could it be because of the high learning rate, local minima or something else like some regularisation parameter is high??

It could be any of those factors. The root cause can be found through experimentation. Hold everything constant and change a single factor. Then systematically change each factor while holding all other factors constant.


Need some help on data mining platform recommendation Currently I need one data mining platform (or solution) meets such requirements:

Data Source

I can import data from: CSV, MySQL, PostgreSQL, HDFS.

Optional: NoSQL, MongoDB.

Interface

I can assemble models and datas with drag & drops (like Weka).

Algorithms

I can model with common machine learning algorithms: Logistic Regression, Random Forest, SVM, word2vec, etc.

It would be awesome if it had deep learning support.

Optional: PMML

Customization

I can implement my own algorithms (under certain standard).

Optional: I can implement my own APIs (under certain standard).

Programming language is not limited.

Visualization

Simple visualization is okay.

License

Commercial solution is acceptable, open-source solution would be better (I can do some tweaks in order to meet my requirement).

There are two software I've investigated: Weka and SPSS Modeler. Weka doesn't support HDFS, while SPSS Modeler doesn't support customized algorithms.

As @oW_ mentioned, Orange seems to be a good choice.

Data Source

Orange3-spark provides Hive/Spark support, cool!

Interface

A little laggy, but acceptable.

Algorithms

Full algorithm supports (based on scikit-learn). I can implement deep learning stuff myself.

Customization

Highly supported. Widgets & addons can be developed.

Visualization

Not bad.


Python - Check if text is sentences? So I have a scraper that gets articles. However, it doesn't always work properly. I want to get better at checking when it doesn't work. For example, the following is something like I want it to scrape:

  Hello. This is a sequence of sentences that are put together. They don't have to follow this exact format, but something very close to this would be nice! Just basically stuff like this put together with the occasional weird formatting, which depends on what is scraped.

But I might also get something that is obviously not text:

  REGISTER | LOGIN | LOGOUT | Sign in to your account Forgot your password?  {* #signInForm *}....

Is there any python library that checks the general format of strings? Basically, I am scraping articles and want to see if the text scraped is article-y. If there isn't a python library, would the best way to go be some sort of regex matching? Is this possible to do reasonably well?

Any help would be greatly appreciated, thanks!!

I would try a semi-supervised learning technique where it passes you scraps and asks you to label them. What you're looking for will likely be kind of domain specific depending on the type of site. In the end you'll probably have a bunch of heuristics like:

If length < 50 and contains "LOGOUT", "REGISTER","SIGN IN","LOGIN"
If count of "|" > 1
If count of all upper case words > 1


predict with Multinomial Logistic Regression If I execute the following code I have no problem:

require(foreign)
require(nnet)
require(ggplot2)
require(reshape2)

ml <- read.dta("https://stats.idre.ucla.edu/stat/data/hsbdemo.dta")
ml$prog2 <- relevel(ml$prog, ref = "academic")
test <- multinom(prog2 ~ ses + write, data = ml)
predict(test, newdata = dses, "probs")

but if I try:

require(caret)
ml <- read.dta("http://www.ats.ucla.edu/stat/data/hsbdemo.dta")
ml$prog2 <- relevel(ml$prog, ref = "academic")
test <- train(prog2 ~ ses + write,method="multinom" ,data = ml)
predict(test$finalModel, newdata = dses, "probs")

it returns Error in eval(expr, envir, enclos) : object 'sesmiddle' not found, why?

That's not the error I get and I'm thinking that you left out some code. I get:

predict(test$finalModel, newdata = dses, "probs")
Error in as.data.frame(newdata) : object 'dses' not found

I don't know why I see this so much, but you should avoid using the finalModel object for prediction. train is doing some things that the model from multinom sin't aware of (such as expanding dummy variables). Try using

predict(test, newdata = dses, "probs")

and never use predict(test$finalModel) again...

Max


YOLO annotation guildelines: overlapping and partially visible objects Can you please advise on how I should annotate objects that are partially visible or overlapping?

There is only 1 restriction, your truth bboxes with the same class_id shouldn't be overlapped more than IoU > 45%, because Yolo uses nms_threshold = 0.45.

In general, you should mark your objects in such a way as you want to detect these objects.