Is it good practice to use SMOTE when you have a data set that has imbalanced classes when using BERT model for text classification? I had a question related to SMOTE. If you have a data set that is imbalanced, is it correct to use SMOTE when you are using BERT? I believe I read somewhere that you do not need to do this since BERT take this into account, but I'm unable to find the article where I read that. Either from your own research or experience, would you say that oversampling using SMOTE (or some other algorithm) is useful when classifying using a BERT model? Or would it be redundant/unnecessary?

I don't know about any specific recommendation related to BERT, but my general advice is this:

Do not to systematically use oversampling when the data is imbalanced, at least not before specifically identifying performance issues caused by the imbalance. I see many questions here on DataScienceSE about solving problems which are caused by using oversampling blindly (and often wrongly).
In general resampling doesn't work well with text data, because language diversity cannot be simulated this way. There is a high risk to obtain an overfit model.


Confusion matrix plot with python I'm for a function that can plot the following plot using python:

Disclaimer,

You can use PyCM which is a python module in order to save your confusion matrix and its evaluation parameters in a html report.

You can use the following commands:

from pycm import *
y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [2, 0, 2, 2, 0, 2, 1, 2, 2, 0, 2, 2]
cm = ConfusionMatrix(y_actu, y_pred)
cm.save_html(filename,color=(100,50,250))

and the result will be as the following:


Modeling events at irregular intervals I've got data representing a sequence of events at irregular intervals and an outcome. There are a fixed number of event types, about 10 of them. The outcome being modeled is binary.

Arguably the following are important to incorporate into the model:

1) Which events occur

2) The order in which the events occur

3) The time lag between events

I.e:
User has event A at start, 5 months later event C, and outcome is positive.
Another user has event B at start, 3 days later event A, 2 days later event D, and outcome is no.

I've also got some static (non sequence data) for each user which I'd like to incorporate.

What is the best approach to model this?

Thanks in advance.

There are multiple ways to tackle this. I'll suggest two here.

[1]
The one that probably requires the least amount of preprocessing work is to throw the event data into a recurrent neural net that can handle variable sequence lengths. Map the event categories to a small embedding space and run it through the RNN. You can then concatenate the remaining static features and put a feed-forward classifier on top.
This would take care of 1) and 2). In order to account for 3) there are several tricks.

You can append the time to event to the embedding vectors
You can multiply the embedding vectors with a "time mask" (i.e. feed the sequence of event lagging times into a RNN with sigmoid activation und multiply element-wise. The idea is that the sigmoid activation will put a weight on the event based on the time passed, e.g. an event many months ago will be unimportant and therefore receive an activation close to 0). See section 3.1. in this paper
That paper also suggests another solution where you can learn a joint embedding of time and event (section 3.2).
Note that both methods report only minor (but consistent) gains over standard RNNs.

[2] Given that your number of event categories is low you could also just one-hot encode the event or use a tree-based method like gradient boosting and feed the events, the time between events and your remaining features as separate inputs into the model. Unless your dataset is very huge this will perform equally well if not better because you don't have to learn embeddings and have a lot less model parameters.

The only difficulty is that you will now have to fix the sequence length. But you can pad shorter sequences with N/A or 0 values and I would not expect any performance decrease because of it!


Best method to deal with too many zeroes in regression problem? I'm trying to solve a simple regression problem using TensorFlow and Pandas to see what's the expected conversion rate for a given product given how much I'm willing to pay for each visit (cost per click or cpc).

Each data point represents a single day and its given acquisition cost and conversion rate. I expect conversion rate to decrease exponentially as visits increase because my audience gets wider and not as focused anymore.

The data seems to agree with this:

The problem is that for the majority of data points, the conversion rate is zero (i.e. no products were sold) so any model is likely to perform well by merely guessing zero sales. For this particular subset (one of the best selling products) the ratio of zero-to-non-zero is roughly 4:1 but in the general data it's about 300:1.

The two best ways that I can think of dealing with this are

a) summarizing data by time periods large enough to contain at least one conversion in them (problem with this is that I lose data by having to settle for the average cost instead of the more detailed daily data)

b) splitting the problem into a classification problem (will I sell at least one unit) and a regression problem (given that I'm going to sell at least one unit, how many units can I expect to sell)

Do either of those sound like a smart idea? Are there any tried-and-true methods to solve this kind of problem?

Option (b) is a good approach. If you use a classifier that outputs probabilities, you can even multiply the two model outputs together to calculate a risk expectation that's interpretable as the joint likelihood.

If cpc is discrete valued and non negative, you could interpret that as a count and use a zero-inflated Poisson model. If it's not, you could convert it by rounding to the nearest cent or something like that.


CUDA_ERROR_OUT_OF_MEMORY: out of memory. How to increase batch size? I have one GPU: GTX 1050 with ~4GB memory.
I try Mask RCNN with 192x192pix and batch=7. I got an error:  CUDA_ERROR_OUT_OF_MEMORY: out of memory
I found this
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

  File "<ipython-input-2-0806c9f06bd0>", line 3
    session = tf.Session(config=config, ...)
                                       ^
SyntaxError: positional argument follows keyword argument

Environment:
1.CUDA 10.0
2.cuNDD 10.0
3.tensorflow 1.14.0
4.pip install opencv-contrib-python
5.git clone https://github.com/thtrieu/darkflow
6.Allowing GPU memory growth

Reference


API to find out how many executors are running my Spark jobs? I'm writing code that captures statistics from Spark runs and stores the results in XML files. I'd like to capture, ideally with the REST API, information about the number of executors and the number and size of the Data Frames that were created. Is there an easy way to do this? What are the API calls?

Keep in mind that the number of executors is independent of the number of partitions of your dataframe. You set the number of executors when creating SparkConf() object.

https://spark.apache.org/docs/latest/submitting-applications.html

To retrieve the number of executors from code, it depends of the API you use. For python, this is not currently implemented. For Scala, you can. Here is a good stack post :

https://stackoverflow.com/questions/38660907/how-to-get-the-number-of-workersexecutors-in-pyspark

Number of partitions refers to the number of parts your dataframe is composed of. With pyspark API, you can get it by transform your dataframe to a rdd and then use method .getNumPartitions() :

num_partitions = df.rdd.getNumPartitions()


Principal component analysis I have a data set that looks like the following:

 Time        V1     V2    V3     ...    V40
13:00        0.44   0     0.33          0.55
13:01        0.55   0     0.34          0.52
13:02        0.58   1     0.20          0.58
.
.
.
15:01        0.57   0     0.24          0.70

Where V2 is the binary equivalent of on/off switches. Currently, I am still pre-processing my data and normalized the data-set from (0,1) using sklearn.preprocessing. I am wondering if applying dimensionality reduction/PCA to my dataset will affect the outcome of my model and whether if it is advisable to use it to process my data.

If you have multi dimensional data, it's very hard to visualize. PCA
  helps us in reducing the dimension of the data set by keeping maximum
  co-variance in top K Features. So PCA will help us in presenting the
  data in most of the cases.

So if you perform PCA, you will definitely lose some data and creating a model with less data, will definitely affect the model outcome.


Python OneHotEncoder Using Many Dummy Variables or better practice? I am building a neural network and am at the point of using OneHotEncoder on many independent(categorical) variables.  I would like to know if I am approaching this properly with dummy variables or if since all of my variables require dummy variables there may be a better way.

df
    UserName    Token                       ThreadID    ChildEXE
0   TAG     TokenElevationTypeDefault (1)   20788       splunk-MonitorNoHandle.exe
1   TAG     TokenElevationTypeDefault (1)   19088       splunk-optimize.exe
2   TAG     TokenElevationTypeDefault (1)   2840        net.exe
807 User    TokenElevationTypeFull (2)      18740       E2CheckFileSync.exe
808 User    TokenElevationTypeFull (2)      18740       E2check.exe
809 User    TokenElevationTypeFull (2)      18740       E2check.exe
811 Local   TokenElevationTypeFull (2)      18740       sc.exe

ParentEXE           ChildFilePath               ParentFilePath
splunkd.exe         C:\Program Files\Splunk\bin C:\Program Files\Splunk\bin 0
splunkd.exe         C:\Program Files\Splunk\bin C:\Program Files\Splunk\bin 0
dagent.exe          C:\Windows\System32         C:\Program Files\Dagent 0
wscript.exe         \Device\Mup\sysvol          C:\Windows  1
E2CheckFileSync.exe C:\Util                     \Device\Mup\sysvol\ 1
cmd.exe             C:\Windows\SysWOW64         C:\Util\E2Check 1
cmd.exe             C:\Windows                  C:\Windows\SysWOW64 1

DependentVariable
0
0
0
1
1
1
1

I import the data and using the LabelEncoder on the independent variables

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

#IMPORT DATA
#Matrix x of features
X = df.iloc[:, 0:7].values
#Dependent variable
y = df.iloc[:, 7].values

#Encoding Independent Variable
#Need a label encoder for every categorical variable
#Converts categorical into number - set correct index of column
#Encode "UserName"
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Encode "Token"
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Encode "ChildEXE"
labelencoder_X_3 = LabelEncoder()
X[:, 3] = labelencoder_X_3.fit_transform(X[:, 3])
#Encode "ParentEXE"
labelencoder_X_4 = LabelEncoder()
X[:, 4] = labelencoder_X_4.fit_transform(X[:, 4])
#Encode "ChildFilePath"
labelencoder_X_5 = LabelEncoder()
X[:, 5] = labelencoder_X_5.fit_transform(X[:, 5])
#Encode "ParentFilePath"
labelencoder_X_6 = LabelEncoder()
X[:, 6] = labelencoder_X_6.fit_transform(X[:, 6])

This gives me the following array:

X
array([[2, 0, 20788, ..., 46, 31, 24],
       [2, 0, 19088, ..., 46, 31, 24],
       [2, 0, 2840, ..., 27, 42, 15],
       ...,
       [2, 0, 20148, ..., 17, 40, 32],
       [2, 0, 20148, ..., 47, 23, 0],
       [2, 0, 3176, ..., 48, 42, 32]], dtype=object)

Now for all of the independent variables I have to create dummy variables:

Should I use:

onehotencoder = OneHotEncoder(categorical_features = [0, 1, 2, 3, 4, 5, 6])
X = onehotencoder.fit_transform(X).toarray()

Which gives me:

X
array([[0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       ...,
       [0., 0., 1., ..., 1., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 1., 0., 0.]])

Or is there a better way to approach this this?

If your categorical variables include variables that suggest some numerical values like ranks, you should consider just label encoding them. (for e.g. First, Second, Third, and so on can be encoded as 1, 2, 3 and so on).

Also, find out if all these categories are important. Plot some graphs(such as histograms, distribution plots) to visualize the dataset. Remove categories that don't seem necessary.(e.g. in the above example, if First occurs more than 80%, you should consider if that certain features really contributes to your model.)


How to predict the dealer whether pick up the goods next month? Here, I want to predict dealers(about 600) whether pick up the goods(about 30) next month. As you see, there are about 18,000 possibilities and it's difficult to predict.

By the way, now I have nearly three years'data(2017,2018 and 2019), and now I just use the common object in the last several months as the prediction.

For example, if dealer A purchases the goods item 1 in September, October and November, then I predict he will purchases item 1 in December. However, this is not exactly accurate, and the precision is about 0.6 while the recall is about 0.3.

So any other way to predict it?

Sure.

Question you should answer is it multiclass or mutlilabel.

In any case I would really advise you to take a look at this list of similiar problems/competitions. Since your question is open-ended I could write 10 pages of approaches. Yours best bet is to invest some time, find similiar problem and look what other people did and dig it up.


Date and time extraction from Excel file in Python or Pandas I have an Excel file with a column that contain both date and time in a single cells as follows:
2020-07-10T13:32:01+00:00
I'm wondering how to extract this cell, split date and time (that are separated by a "T") and write them into 2 new columns "Date" and "Time", and able to use them afterwards to, for example, do Time math operations.
I'd have a start from pandas:
    df = pd.read_excel('file.xlsx')
    def convert_excel_time(excel_time):
    return pd.to_datetime()

but I actually don't know if it is achievable with pandas.
If I have this:

I would like to calculate how many minutes passed (for the same date) for the same ID column (2nd column).
I guess I can use python tdelta, or is there a pandas alternative?
Thank you.

Pandas can parse most dates formats using
import pandas as pd
pd.to_datetime(df["name of your date column"])

You can also cast the desired column to datetime64 which is Numpy dtype
df = df.astype({"column name": "datetime64"})


Classifiers and accuracy I would like to ask you how to use classifier and determine accuracy of models.
I have my dataset and I already cleaned the text (remove stopwords, punctuation, removed empty rows,...).
Then I split it into train and test.
Since I want to determine if an email is spam or not, I have used the common classifiers, I.e. Naive Bayes, SVM and logistic regression.
Here I just included my train and test datasets: nothing else!
I am using Python to run this analysis.
My question is: should it be enough or should I implement new algorithms?
If you could provide me with an example of how an already existing algorithm was improved it would be also good.
I read a lot of literature regarding accuracy of text classification and in all the papers the authors use SVM, Naïve Bayes, logistic regression to classify spam.
But I do not know if they built their own classifier or just used the existing one in Python.
Any experience on this?

The question mixes two different notions: models (or algorithms) and accuracy. Let me clarify them.

Model (or Algorithm) is a classification technique and 'Accuracy' is one of the ways to evaluate the performance of the models.

You can choose any models(Naive bayes, SVM or other deep learning techniques) to implement your classifier. They are independent of 'Accuracy' or 'F1' or any other measures by which you want to test the performance.

At first, you shall pre-process the text (remove stopwords, punctuation, etc.) , and pre-processing is a choice that says how the data should look like before getting into the model. They do influence the model's performance, but not to a great extent when done right. Usually, pre-processing is applied on both the train and test set.

Model performance: Once you implement you model, you may want to see how well it generalises (performamce on unseen data). So, you shall split the dataset into two halves: training and test set. (Usually most of the authors split into 3 portions: training set, validation set(to avoid over-fitting) and test set). You shall train the model with training set, and the test set is used to evaluate the performance of the model.

Model evaluation: Once the model is trained on training data, predict the labels on the test set. So, you have two set of labels on test set: 1: ground truth (the actual labels indicated by the test set) and 2: predicted labels (the labels predicted by the model).
Now, use the evaluation metric of your choice (Let's assume that you want to choose 'accuracy' as evaluation metric).
Accuracy can simply be calculates as: (#Number of correctly predicted samples / #Total number of samples) * 100 . Where #Number of correctly predicted samples is the count of samples for which the ground truth label and predicted labels are the same.


How to create a confusion matrix for one node of a decision tree? I am doing past papers for my data science exam and was curious about one of the questions. They ask us to create a confusion matrix by hand for one node of a decision tree.
I understand how to create a decision tree for an entire model, but I am unsure on how to create one for just one variable. Should the entire first row be 0s other than the class A as it should always predict class A or am I missing something.
I have attached the decision tree and the exact wording of the question below. Thanks!
"(iv) Present a confusion matrix for the [leftmost terminal] node."

CM is about True Vs Predicted.
Since only one node is in the discussion which means we will have only one column of "Predicted" but all the possible rows of "True"
Let's assume 100 samples in the Node. The Node must be classified as "A"
Below should be the CM - Rows are "True" and the Column is the "Predicted"
\begin{array} {|r|r|}
\hline
 &A &B &C &D &E &F \\
\hline
A &62 &- &- &- &- &- \\
\hline
B &21 &- &- &- &- &- \\
\hline
C &13 &- &- &- &- &- \\
\hline
D &00 &- &- &- &- &- \\
\hline
E &02 &- &- &- &- &- \\
\hline
F &02 &- &- &- &- &- \\
\hline
\end{array}


Installing Prototype Widgets in Orange running on Mac I am having trouble downloading some add-ons for the Orange data mining software that are not available through the normal add-ons menu. To do this, I am attempting to install a particular set of add-ons from the GitHub page here.
I am very novice when it comes to using Terminal and terminal commands. Could anyone help provide some insight on how to install this package?
Some people have suggested that they should appear in the prototypes list, but they do not. Does anyone have Orange running on Mac so that they can try this to see if it works?
Specifics for the exact code that I need to implement in terminal to get this to work would also be helpful.
Thanks!

The easiest way to install Orange's addons is through the application itself. Open Orange, in the menu click Options -> Addons. In the popup window mark Orange3-Prototypes and click OK. Note, that doing so you will get the latest version that is published on PyPi.

If you would want to install the bleeding edge version directly from github — assuming that you installed Orange by downloading a bundle from it's website and not for example by using Anaconda — do the following. Open a terminal and paste this command:

/Applications/Orange3.app/Contents/MacOS/pip install git+https://github.com/biolab/orange3-prototypes.git

Doing so you will trigger the pip that came with the bundle and instruct it to install the addon directly from github.

Further, to install opencv, which is required by some widgets (e.g. webcamecapture) run

/Applications/Orange3.app/Contents/MacOS/pip install opencv-python

Note, however, that this is unofficial version of opencv but it seems easier to install.


Can a decision tree learn to solve a xOR problem? I have read online that decision trees can solve xOR type problems, as shown in images (xOR problem: 1) and (Possible solution as decision tree: 2).

My question is how can a decision tree learn to solve this problem in this scenario. I just don't see a way for any metric (Information gain, Gini Score, ...) to choose one of the splits in image 2 over any other random split.

Is it possible to solve the presented problem with a decision tree?
Would using a random forest solve the problem in any way?

Thank you in advance.

Yes, a decision tree can learn an XOR.

  I have read online that decision trees can solve xOR type problems...

Often things are phrased not carefully enough.  A neural network can perfectly sort a list of integers, but training one to do that would be quite hard.  Your image shows that a tree can easily represent the XOR function, but your question is how to learn such a tree structure.

  My question is how can a decision tree learn to solve this problem in this scenario. I just don't see a way for any metric (Information gain, Gini Score, ...) to choose one of the splits in image 2 over any other random split.

Indeed, the first split is probably quite random, or due to noise (if you go for $\operatorname{sign}(x\cdot y)$ with continuous $x,y$ instead of the discrete $x,y$ and XOR).  But, as long as your algorithm makes the plunge with one of those first splits, the next splits are obvious and your tree will make them.

  Is it possible to solve the presented problem with a decision tree?

Here's a notebook (github/colab, suggestions welcome) demonstrating that yes, a (sklearn) decision tree can learn $\operatorname{sign}(x\cdot y)$ (perhaps with some errors when points are extremely close to 0); but it also goes on to show some of the difficulties, e.g. when variables other than $x,y$ are available to the tree to split on.  Up-shot: noise variables can wreck that first split I mentioned above, and even useful variables can make the tree lose track of the XOR.

  Would using a random forest solve the problem in any way?

Probably not the basic problem, but it looks like it helps with, e.g., the noise variables above.


LSTM doesnt find finer dependencies than the Random Forest model I am currently implementing a forecasting model. The whole builds on a regression. The first model is an LSTM network based on Keras and the second a Random-Forest model based on sklearn.

I use the same input values and target variables for both models. The forecasts of the LSTM network are closer to the actual observed values (better rmse and r2) (probably due to the memory / lookback), but the Random Forest model provides better altitudes and depths. It seems that the Random-Forest model has found better dependencies than the LSTM model.

LSTM:

Random-Forest:

What is the best way for the LSTM model to understand finer dependencies? Should I use an ensemble or stack method from LSTM + RF? Does it make sense?

Some (possibly stupid) ideas for combining the two models:

Make an average of the two forecasts
Instead of a naive average you use LSTM for estimating the coarse daily trend and the RF for the fine details, maybe like so:

a. Smooth (maybe with a loess or a kernel) the LSTM to get the daily trend

b. For each day, subtract the mean volume call from the prediction of the RF, to get the fine details

c. Add the forecast of two models

General suggestions for improving your model:

Instead of training a model on the whole series of call volume, you might be better off feeding your model one day at a time, e.g. you cut the time series between 8 am and 5 pm every day. After all, you do not want your model to spend parameters on learning that no calls are made at night or during the weekend.
Add the day of the week as a parameter in the model, as well as some information on the call volume in the past (like previous day) if you think that there might be some time correlation between consecutive days.
If you think that the day of the week has a strong effect on the call volume (e.g. if each day has a special, recurring pattern) then you might consider creating a separate model for each day of the week.


What is difference between feed forward neural network and LSTM? What is the difference between feed-forward neural network and LSTM? How do they differ in their architecture?

A feed-forward neural network looks like this:

input -> hidden layer 1 -> hidden layer 2 -> ... -> hidden layer k -> output.

Each layer may have a different number of neurons, but that's the architecture.

An LSTM (long-short term memory cell) is a special kind of node within a neural network. It can be put into a feedforward neural network, and it usually is. When that happens, the feedforward neural network is referred to as an LSTM (confusingly!).

So how does an LSTM work? Have a look at my answer here: Forget Layer in a Recurrent Neural Network (RNN) -

tl;dr: an LSTM cell has three gates, each of which is used to modulate its input in some way: an input gate, a forget gate, and an output gate. It has a "memory" and an output that get modified by the gates. That is, within a single LSTM cell:

(input & previous cell state) ->  (input & forget gates) -> (update cell state)

(input & previous cell state & updated cell state) -> (output gate)

You can stack these cells in many different configurations. So there is no single "LSTM network" - rather a set of many possible architectures that can be built from these basic nodes.

Hope that gets you started!


Machine learning and time-based data I want to predict conversion rates for an eCommerce store. I have data from Google Analytics with features like averageSessionDuration, bounceRate, numberOfVisitorsBySource etc. and the corresponding conversion rate.
That's the specific background for a generic question: I am not sure how I should select the time units for my data. Should I export the data by day, by week, by month or by year?

When I have one row per day, I have a lot of data (a lot of rows),
but also a lot of noise, because the conversion data for one day is
sparse.
When I have one row per year, I have less noise, but also less data
(less rows).

In general, what's the right approach to select the right time frame per row in such a case?

I think you should first export it with a timestamp and then add a day, month ... other features to the data. This might help in better accuracy.
For more details, you can refer: This article


what is the difference in terms namely Correlation, correlated and collinearity? A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting in concert to explain the variation in a dependent variable.Could someone clarify the terms ?

Pearson correlation is the usual correlation when nothing further is specified and specifically refers to linear association.
$$\rho_{XY}=\dfrac{cov(X,Y)}{\sigma_X\sigma_Y}$$
In the world, people use “correlation” to mean any kind of association, but this is wrong from the standpoint of statistics. Arrange points symmetrically on a parabola and run them through that equation; you’ll get zero correlation, despite the obvious relationship.
There also is Spearman correlation, which does Pearson correlation on the ranks of the values.
If the points are $(0,1)$, $(2,4)$, $(3,3)$, the Spearman correlation is calculated by converting the $x$-values to their ranks and the $y$-values to their ranks: $(1,1)$, $(2,3)$, $(3,2)$. Then run the transformed points through the usual equation for (Pearson) correction.
To separate “correlation” and “correlated”, the former is a noun while the latter is an adjective. If there is “correlation” between two variables, they are “correlated”.
Collinear seems to come up in the context of regression and refers to predictor variables that are correlated. The related “multicollinear” means multiple regression predictors that have a linear relationship with another predictor, as if you could regress one predictor on some of the others and get decent accuracy. “Multicollinearity” seems to be the more common term to use when we talk about related predictors, as “collinear” variables strikes me as perfectly related with a correlation of $1$ (think of measurements in both meters in kilometers), while multicollinearity, to me, does not mean a perfect predictive ability unless “perfect” multicollinearity is specified.
“Collinear” and “multicollinear” are adjectives; “collinearity” and “multicollinearity” are the nouns.


If ReLU is so close to being linear, why does it perform much better than a linear function? ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers.

I'm a beginner to deep learning research and methodologies but I've already seen several examples that claim that using ReLU as activation functions on a network will be superior than other common functions, including simply linear ones, of course.

ReLU has 1 point of "drastic" change, and is "otherwise linear".

My questions are:

Is there research or opinions on why this change significantly affects the performance of a network?
Is there something unique about $0$, the point of change? What if we move the point of change to $1$ instead for instance?

My intuition is that, as $0$ separates the line, it separates the sign function and that may have high affect on certain types of data that rely on positive/negative classification, perhaps.

How relevant is the dataset being learned? Do we see that ReLU is "better" on most examples of datasets?

Why ReLU is better than the other activation functions

The above question is relevant. Mine is more focused on the difference between ReLU and a linear function.

A neural network is a succession of layers (dense/linear, convolutional). After each layer, you need a non-linear activation function. Why do you need a non-linear activation function? Because composing two linear transformations is equivalent to having a single linear transformation:

$ (x \cdot W_1 + b_1) \cdot W_2 + b_2 = x (W_1 \cdot W_2) + (b_1 \cdot W_2 + b_2)$

Therefore, stacking linear transformations would be pointless, because the whole network would be equivalent to a linear transformation.

A ReLU may seem similar to a linear function, but its non-linear nature gives the network the ability to model non-linear functions.
No, there is nothing unique about $0$ being the point of change. Most neural network layers have a bias term that makes the point of change irrelevant, as during the training the bias will be adjusted to the proper value.
While in deep learning everything is data-dependent, the fact that the ReLU's changing point is irrelevant is independent from the data (see previous answer).

The advantage of ReLU is against other non-linearities, like sigmoid and tanh, which suffer the vanishing gradient problem. ReLU itself has its own problem, namely the "dying ReLU problem".


Basics: What is the correct sequence for preparing simple data for ML? I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one).

I was just wondering what would be the best way to organise the data to avoid redundancy with the following steps:

Feature Selection
Account for gaps in data(Imputing)
One hot encoding to allow non numerical features
Split data into training and validation sets
Then proceed to fit and predict with the model.

My main query is whether it is better to split the data before encoding it, or to only do the split after completing the encoding?

The same encoding needs to happen on both the train and the test set. So, encode before splitting.


Why am a getting wrong prediction when combining two list of samples, which individually gives correct prediction? So I am coding in Python. I have to set of samples. Set1 contains samples of class A and the other set, Set2 contains samples of class B. These samples taken are a part of the training dataset. When I am predicting set1 and set2 individually, the classification is perfect. Now when I am merging the two sets for prediction into one set, the prediction gives the wrong result for the samples in Set2, i.e., predicting the samples of set 2 to be in class A. However, samples belonging to Set1 are predicted to be in class A in the merged set. Why is this happening? (I have saved the model and loaded that model for further prediction, the same error persists)

model.add(Dense(newshape[1]+1, activation='relu', input_shape=(newshape[1],)))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(50, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          metrics=['binary_accuracy'])
model.fit(X_train, y_train,validation_data=(X_test, y_test),validation_split=0.2, epochs=500, batch_size=25, verbose=0)

You have a mistake somewhere in either the data you provide as training data or the model you use for predicting.

If it was really the same model trained on the full data in all the cases, then any given instance in set2 would always be predicted with the same class (right or wrong), independently from the other instances in the test set. This implies that it's not the same model being applied when you predict individual sets or both merged.

As a test, save your model somewhere first then apply it to your different cases (the model that you saved, don't train it again).

Ok so you are trying to do 3 experiments:

Training on full training set produces modelA, then apply modelA on set1
Training on full training set produces modelB, then apply modelB on set2
Training on full training set produces modelC, then apply modelC on both set1 and set2

But if this was really what you are doing the three models would be the same: modelA = modelB = modelC. Basically you could do this instead:

Training on full training set produces modelD
apply modelD on any set

Based on your description what probably happens is something like this:

Training on instances of class A (error) produces modelA, then apply modelA on set1 -> perfect results
Training on instances of class B (error) produces modelB, then apply modelB on set2 -> perfect results
Training on instances of class A (error) produces modelC = modelA, then apply modelC on both set1 and set2: perfect for class A, completely wrong for class B

You need to check which instances you give as training set in each case.


How can I extract numerical information conditional on other information? Given text data, I'd like to extract numerical information based on the presence of other information.

For instance, "There are 119 people in the gym right now. 100 people in the weight room, 19 on treadmills". If I was interested in "weight room" then I would extract "100" or if I was interested in "treadmills" then I would extract "19".

I can create dependency trees with Python's spaCy package. Is there a way to extract these dependencies?

If your data is always as clean as this, you might be able to solve the problem using simple regular expressions. You could simply look for the first number that occurs before the string of interest.

As to the dependency parsing using spacy, this is the output that your sentence would get

There expl are VERB []
are ROOT are VERB [There, people, now, .]
119 nummod people NOUN []
people attr are VERB [119, in]
in prep people NOUN [gym]
the det gym NOUN []
gym pobj in ADP [the]
right advmod now ADV []
now advmod are VERB [right]
. punct are VERB []
100 nummod people NOUN []
people ROOT people NOUN [100, in, ,, 19]
in prep people NOUN [room]
the det room NOUN []
weight compound room NOUN []
room pobj in ADP [the, weight]
, punct people NOUN []
19 appos people NOUN [on]
on prep 19 NUM [treadmills]
treadmills pobj on ADP []

Meaning that you would need to first locate weight room, and then trace back to 100, which in this case might simply be more complicated than using regular expressions.


Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2) I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure).

I wish to compute: for each of the 500 txt files, find the cosine similarity with all the videos. After this rank the videos for each txt files and save that ranking list in another file.

So far, I am able to read all the txt files. This is the code:

import gensim
import glob
# go to directory
all_files = glob.glob("ExtratingConceptFromVideoScripts/iir/*.txt")
# go to files
indata = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            indata = indata +  (d[0], )
print(indata[0])  # print the contents of first file

This is the output:

Online edition (c)2009 Cambridge UP An Introduction to Information Retrieval Draft of April 1

Now, I read the contents of the json file:

import gensim
import glob
# go to directory
all_files = glob.glob('ExtratingConceptFromVideoScripts/data/corpus.json')
# go to files
vid = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            vid = vid +  (d[:8], )
print(vid)

Output:

(['{"0": {"metadata": {"id": "fQ3JoXLXxc4"', ' "title": "| Board Questions | 12 Maths | Equivalence Class | Equivalence Class Board Questions |"', ' "tags": ["Board Questions"', ' "12 maths"', ' "12 maths Board Questions"', ' "Previous Year Board Questions"', ' "Maths Board Questions"', ' "Board questions based on Equivalence Classes"'],)

Now, my problem is: the cosine similarity matrix finds the cosine similarity between the set of documents in one tuple (ie. it finds the cos sim between all files in doc iir). How to compute the cos sim of each iir doc with each doc in the vid.

I tried to convert the vid json file to tuple, and then compute the cos sim between iir and vid, but it gives an error.

Can anyone help me with explaining how to find cos sim between files with different formats, how to loop through files and compute cos sim, how to rank docs on the basis of highest cos sim.

Similarity is computed in a vector space model with a primary assumption that the pair of objects for which you want to compute similarity must be represented in the same vector space using some pre-processing operation.

Second dataset is a JSON. You'll need to extract text from it. One way to do it: you can use title and tags strings for each video, concatenate it to a string to find string representation for each video. Use any text vectorization method post that.

So get the video metadata into a text and then you can compute similarity.


Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator? So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here;

from pyspark import SparkConf
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import mean
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

from pyspark.ml import Pipeline
from pyspark.ml.classification import (RandomForestClassifier, GBTClassifier, DecisionTreeClassifier)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

conf = SparkConf()
conf.set('spark.logConf', 'true').set("spark.ui.port", "4060")

spark = SparkSession.builder.config(conf=conf).appName("Gradient Boosted Tree").getOrCreate()

data = spark.read.parquet('/mydata/location)

def yt_func(x):
    if x <= 10:
        yt = 0
    else:
        yt = 1
    return yt

yt_udf = udf(yt_func, IntegerType())

data = data.withColumn('yt_1',yt_udf(data['count']))

datasub = data.select('feature1', 'feature2',
                      'feature3', 'feature4',
                      'feature5', 'feature6',
                      'feature7', 'feature8',
                      'feature9', 'feature10',
                      'feature11','feature12',
                      'feature13')

datasub = datasub.na.fill(0)

from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols = ['feature1', 'feature2',
                      'feature3', 'feature4',
                      'feature5', 'feature6',
                      'feature7', 'feature8',
                      'feature9', 'feature10',
                      'feature11','feature12',
                      'feature13'], outputCol = 'features')

output = assembler.transform(datasub)

finaldata = output.select('features','yt_1')

train_data,test_data = finaldata.randomSplit([0.7,0.3])

finaldata.show(20)

dtc = DecisionTreeClassifier(featuresCol='features',labelCol='yt_1')
rfc = RandomForestClassifier(featuresCol='features',labelCol='yt_1', numTrees=70)
gbt = GBTClassifier(featuresCol='features',labelCol='yt_1')

dtc_model = dtc.fit(train_data)
rfc_model = rfc.fit(train_data)
gbt_model = gbt.fit(train_data)

dtc_preds = dtc_model.transform(test_data)
rfc_preds = rfc_model.transform(test_data)
gbt_preds = gbt_model.transform(test_data)

dtc_preds.show()
rfc_preds.show()
gbt_preds.show()

accuracy_eval = MulticlassClassificationEvaluator(metricName = 'accuracy', labelCol='yt_1')
recall_eval = MulticlassClassificationEvaluator(metricName = 'weightedRecall', labelCol='yt_1')

print 'dtc accuracy:', accuracy_eval.evaluate(dtc_preds)
print 'dtc recall', recall_eval.evaluate(dtc_preds)

print 'rfc accuracy:', accuracy_eval.evaluate(rfc_preds)
print 'rfc recall', recall_eval.evaluate(rfc_preds)

print 'gbt accuracy:', accuracy_eval.evaluate(gbt_preds)
print 'gbt recall', recall_eval.evaluate(gbt_preds)

When I run this I get the following;

dtc accuracy: 0.98596755767033761
dtc recall: 0.98596755767033761

rfc accuracy: 0.98551077243825225
rfc recall: 0.98551077243825225

gbt accuracy: 0.98624595624862965
gbt recall: 0.98624595624862965

What is confusing me here is why I am getting the same values for the accuracy and the recall.... they are EXACTLY the same. Surely this isn't correct....??

Any ideas?

It's correct. You can see comments to this effect in the docs/source. In a multiclass setting, overall precision and recall are the same.

Precision and recall in the "binary" classification case is really reporting precision with respect to one of the two classes: the positive one. There are in fact two classes, but we tend to think of it as class vs not-class.

However in multiclass, there is no positive and negative. All N classes are equal, and precision and recall are computed over all N classes, not N-1. This ends up making their definition the same.

To see it, consider the confusion matrix of a multiclass classifier, with predicted class down the rows and actual classes across the columns. "True positives" are equivalent to all the diagonal entries: these are the cases where predicated and actual classes match. "False positives" occur when the predicted class is wrong: these are all the entries in each row that aren't on the diagonal. "False negatives" occur when the actual class isn't the predicted one: these are all the entries in each column that aren't on the diagonal.

But the sum of rows (less the diagonal) is the same as the sum of columns (less the diagonal). So the false positive and false negative count are the same. And from the definition of precision and recall, this makes their values the same.

In fact, both equal the accuracy of the multiclass classifier.


Xgboost take k best predictions I have a mission of classification with a lot of classes. I am comparing some ML algorithms for this case and I would like to try xgboost.
I am writing in python and I am trying to get the best 3 predictions using this algorithm but I couldn't find any method in xgboost API the fits what I want

Any recommendations ?

After fitting the model you can use predict_proba( ) from the docs here

This returns a numpy array with the probability of each data example being of a given class. The three highest probabilities will be your best 3 predictions.


Can distribution values of a target variable be used as features in cross-validation? I came across an SVM predictive model where the author used the probabilistic distribution value of the target variable as a feature in the feature set. For example:

The author built a model for each gesture of each player to guess which gesture would be played next. Calculating over 1000 games played the distribution may look like (20%, 10%, 70%). These numbers were then used as feature variables to predict the target variable for cross-fold validation.

Is that legitimate? That seems like cheating. I would think you would have to exclude the target variables from your test set when calculating features in order to not "cheat".

There is nothing necessarily wrong with this.  If you have no better information, then using past performance (i.e., prior probabilities) can work pretty well, particularly when your classes are very unevenly distributed.

Example methods using class priors are Gaussian Maximum Likelihood classification and Naïve Bayes.

[UPDATE]

Since you've added additional details to the question...

Suppose you are doing 10-fold cross-validation (holding out 10% of the data for validating each of the 10 subsets). If you use the entire data set to establish the priors (including the 10% of validation data), then yes, it is "cheating" since each of the 10 subset models uses information from the corresponding validation set (i.e., it is not truly a blind test). However, if the priors are recomputed for each fold using only the 90% of data used for that fold, then it is a "fair" validation.

An example of the effect of this "cheating" is if you have a single, extreme outlier in your data. Normally, with k-fold cross-validation, there would be one fold where the outlier is in the validation data and not the training data. When applying the corresponding classifier to the outlier during validation, it would likely perform poorly. However, if the training data for that fold included global statistics (from the entire data set), then the outlier would influence the statistics (priors) for that fold, potentially resulting in artificially favorable performance.


Multi Object Detection Might sound like a very basic question, but in a detection problem with multiple objects (solved by regression) how does the weight update work such that regression equations of all classes are satisfied all at once. How is it that boxes of other objects do not interfere with the training of box weights of our object?
Help appreciated a lot. Thanking in advance! :)

Because a NN doesn't see a box. The only job it does is to minimize the loss using Feedforward and Backpropagation.

Let's say the image has 1 object -
=> 4 o/p values => 4 o/p neurons => 4 y_train values => 4 losses into the NN
=>To have the loss minimum it will adjust itself to have all the 4 o/p neurons as close to the truth as posible.

Let's say the image has 2 object -
=> 8 o/p values => 8 o/p neurons => 8 y_train values => 8 losses into the NN
=>To have the loss minimum it will adjust itself to have all the 8 o/p neurons as close to the truth as posible.

The difference - Obviously, later will be a bigger network and need more epochs to get the golden values of weights

If you have just started Deep Learning -
Michael Nielsen


Machine learning algorithm to classify blog posts So I have a large collection of blog posts containing title, content, category, tags and geo-location fields and I'm looking to achieve three things:

Assign a category (or multiple categories) to all the posts and any new ones. I have a strict vocabulary of categories.
Add new tags to the posts that might be relevant to the post.
Mark the post if it contains information about a place. For example: Lorem ipsum dolor sit amet San Francisco, consectetur adipiscing elit.

I've been looking into different machine learning algorithms, most recently decision trees, but I don't feel that is the best algorithm to work out the problems above (or that I haven't understood them enough).

Many of these posts already contain categories, tags and geo-location data. Some do not contain any information and some have only a few details.

What would be the best machine learning algorithm to look into to solve each of the three areas?

Question 1: Category prediction

To predict the category of a new blog post, you could do the following:

Build a MLP (multilayer Perceptron, a very simple neural network). Each category gets an output node, each tag is a binary input node. However, this will only work if the number of tags is very little. As soon as you add new tags, you will have to retrain the network.
Build a MLP with "important words" as features.
If you have internal links, you might want to have a look at "On Node Classification in Dynamic Content-based Networks". In case you're German, you might also like Über die Klassifizierung von Knoten in dynamischen Netzwerken mit Inhalt
You could take all words you currently have, see those as a vector space. Fix that vocabulary (and probably remove some meaningless words like "with", "a", "an" - this is commonly called a "stopword"). For each text, you count the different words you have in your vocabulary. A new blog post is point in this space. Use $k$ nearest neighbor for classification.
Use combinations of different predictiors by letting each predictor give a vote for a classification.

See also

Yiming Yang, Jan O. Pedersen: A Comparative Study on Feature Selection in Text Categorization, 1997.
Scikit-learn: Working With Text Data

Question 2: Tagging texts

This can be treated the same way like question 1.

Question 3: Finding locations

Download a database of countries / cities (e.g. maxmind) and just search for a match.


Dealing with low-information centroids using Nearest Centroid Classifier and bag of words method I am currently working on a problem where we have projects and e-mails that belong to a single project each.

My goal is to create a recommendation system for incoming e-mails which presents the projects the e-mail might belong to.

The number of projects is constantly growing, just like the number of e-mails. This is why I decided to use the Nearest Centroid Classifier because the "training" of new classes is easy (after all, just calculating a mean over the e-mails belonging to a centroid) and it seemed to be promising to me.

I use NCC in combination with the bag of words method and for that I am calculating the scores of the words via TF-IDF.

The data pool is not the greatest actually, which is another reason I tried using a less complex model like NCC. I only have 5000 useful e-mails and around 300 projects.

The problem is though that when I calculate the distances to every project centroids some centroids win in every case. For nearly every e-mail the first 10 centroids of the recommendation are the same, all the time. When I had a look at them I noticed that the "best" centroids are simply centroids which do not hold much information, they seem to have very few text data. And if the e-mail does not have much text obviously the error is low and thus the distance is low.

Is there any way to deal with that problem? Or is TF-IDF and NCC not a good combination?

This is a standard problem with distance/similarity measures between texts of different length. I'm not aware of any standard way to solve it, but in your case I would simply try to remove any email shorter than a certain length from the training set (you can experiment with different thresholds). This would hopefully force the centroids to be more specific, the goal being that none of them can easily attract all the instances.


How to compare topics generated from topic modeling from different datasets? I have two datasets of a similar theme. Let's assume Dataset A and Dataset B. Using the top2vec model (https://github.com/ddangelov/Top2Vec) (https://arxiv.org/abs/2008.09470) on each dataset, I came up with a certain number of topics. Now, I want to compare both datasets' topics.
How can I do this? Clustering or any other method will work as far as I can compare the generate topics computationally.

I'm not aware of any standard options to do that, but I could suggest a couple directions. Note that in both ideas it's possible to compare topics individually or globally.
Applying evaluation measures on the instances
In this idea you would apply the model obtained from dataset A on all the instances of dataset B (and of course you can do the converse as well). This would give you a prediction of topic for every instance in B (typically using the max probability topic for the instance) from both model A and model B.
Then you would need a method to match the topics between A and B, i.e. convert topics from A to B. This can be done using the number of instances they have in common, I know that there is some literature and several methods which have been used in supervised evaluation measures for word sense induction, I would guess that there are similar things for topic modeling (not sure). Once the matching is done, various evaluation measures can be used to compare the topics. The idea is to consider the topics obtained from A as the predictions to be compared against the gold-standard topics from B. Note that most measures are symmetrical, so it doesn't matter which one is considered the prediction or the gold.
Comparing words probabilities
I'm not familiar with top2vec but usually it's possible to obtain the top words for every topic together with their associated probability. Assuming this is possible here, the idea would be to compare either:

The top N words for every pair of topics (every topic from A against every topic from B). This can be done very simply by counting the number of words in common between the two top N words: the more the topics are similar, the more they have words in common in their top N.
the whole probability distribution of the words for every pair of topics, using a distance measure such as KL divergence (there are other options). This value will represent how closely related two topics are.


max_iter hyper parameter in sklearn.cluster.MiniBatchKMeans What is the significance of max_iter in sklearn.cluster.MiniBatchKMeans? Is this the maximum number of times partial_fit() can be executed on batches of data?

max_iterint, default=100
Maximum number of iterations over the complete dataset before stopping independently of any early stopping criterion heuristics.

It's the number of iteration over the full dataset.
Number of partial_fit will depend on batch_size

batch_sizeint, default=100
Size of the mini batches.

partial_fit(self, X[, y, sample_weight])
Update k means estimate on a single mini-batch X.

Ref :
User guide
Class Def.


How are clusters from DBSCAN sometimes non-convex? I've been using clustering in my bag of ML techniques for quite some time now, and  I've never found a satisfying answer to this question.

In DBSCAN, we define a maximum radius with which to form clusters.  The algorithm will scan the space and group together points that are ALL reachable from one another.  However, we can sometimes end up with a non-convex cluster.

My confusion is around how the notion of a "radius", which describes a convex object, can be an input to an algorithm which results in a non-convex object?

A cluster in DBSCAN consists of multiple core points.

The radius is the area covered by a single core point, but together with neighbor core points the shape will be much more complex. In particular, they can be much larger than epsilon, so you should choose a small value, and rely on this "cover" functionality.

Wikipedia has an example of a non-convex cluster


Advice on what Machine Learning Algorithms to study for a Job to candidate matching algorithm I have asked in a few places and this seems to get downvoted for some reason. If this is not the place to ask this then some advice on how and where to ask it would be appreciated.

I'm creating a web platform 2 sided market that matches available consultants to short term contracts.

We currently have a rules-based algorithm that matches based on a small number of features like industry, job title, rate, and availability. For that list of consultants, It then calculates a percentage match based on other features like skill set, qualifications, certifications, years of service, etc.

We have data on candidates that were accepted and candidates that were rejected.

I've spent a year revising maths, statistics, and studying basic machine learning with python. But I am no closer to understanding how machine learning could be used to match consultants to assignments.

I assume what I need to look into is feature engineering and some sort of classification algorithm. It would seem that a contract assignment would have an ideal candidate and so we are matching all other candidates to the features of the ideal. Is it some sort of unsupervised learning to classify the candidates into groups and putting forward those candidates that fall in the same group?

Does anyone have an idea on how best this would best be done?

I am looking at what to study to get closer to a solution so any advice to so that I can get to a point of understanding how this could best be done would be appreciated.

Very likely, there is too much noise in the data. Meaning there are hidden pieces of information that are dominating the decision for acceptance or not, which you don't find in the data. For example, if the client finds the consultant sympathic. Or if the client had a good day when deciding about the consultant.

I doubt that the rule system is overly complicated, so my advise is to stay with it.

If you want to get started with machine learning, look for MNIST examples (a handwritten digits dataset) and I recommend to use Keras with Tensorflow as a Backend.

(by the way: I'm a machine learning ;-) )


Is pre-processing always neccessary? I'm working on classification of two classes of Raman spectra. And while I was working on finding the optimal steps for pre-processing, I started to wonder if it is really necessary.

I have a lot of Raman spectra data and and most of them differ between both classes even to the naked eye (characteristic peaks or the lack of).
While I do have many datasets I know were not recorded under exact same conditions, I also have a few sets for which exact same conditions apply for both classes. They still look obviously different from each other. Since my primary goal is to develop a robust algorithm with an unsupervised classificator, I now start to wonder:

Assuming the setup and thus noise and any sort of distortion is the same in both classes, even if dominant, do I need to bother with pre-processing when various classifiation models already return satisfying results with the raw data?

Most (if not all) papers I read about Raman spectra pre-processing (there's a lot on that topic) gave me the impression, the authors are always interested in separating the Raman spectrum of interest from all the unwanted other signals (fluorescent background, white noise, cosmic spikes etc). While this certainly would be interesting, I actually only want to sort them.

I'm asking this question from a rather hypothetical standpoint. I know that in reality you can hardly guarantee exact same conditions (especially with random occurrences like cosmic spikes).

[Not specific for Raman spectroscopy]

There is a trend in some areas of machine learning to aim at end-to-end learning, that is, devise machine learning algorithms that take raw input data and gives the desired output (e.g. classification label, regression values). This way, end-to-end learning avoids introducing expert knowledge or preprocessing, usually relying only on huge amounts of data. This has been specially true in the last years in the field of deep learning, where images are taken with no preprocessing, audio is not filter-banked, and games are played by receiving only raw pixels.

So it is certainly possible to skip data preprocessing completely.

We should nevertheless take into account that most of the successful end-to-end machine learning approaches impose constraints that enable the learning to exploit the very aspects of the input data that were previously defined by experts. An example for this are Convolutional Networks, that impose a feature locality constraint that enables the learning to identify features, which could have been pre-defined as gabor filters instead.

BUT, in more traditional data science, feature engineering is a key part of the process of devising ML systems and data exploration and expert knowledge are normally used to identify the kind of preprocessing to be applied to data.


Recommendations for learning DCGANs? I mean stuff that doesn't use Python, Keras or TensorFlow. I have been looking for an in-depth explanation on how to implement a DCGAN from the ground up so I can have a complete understanding of the concept. Even some literature on GANs from the ground up would be useful.

All of the stuff I have read thus far leaves out a lot of the details, falling back on third party libraries, hiding the implementation.

Is my best bet to read papers? Trudge through references until it makes sense? Is there not a comprehensive text because of the novelty?

Checkout the playlist of Ahlad Kumar on YouTube. He has a wonderful playlist for theory and implementation of Gans. Although the implementation is using tensorflow, it will satisfy your curiosity of understanding the concept without getting your hands dirty with coding from scratch without using deep learning framework.

Conditional GANs:
https://youtu.be/7Tlk3Gql-Wg


Why do machine learning engineers insist on training with more data than validation set? Among my colleagues I have noticed a curious insistence on training with, say, 70% or 80% of data and validating on the remainder.  The reason it is curious to me is the lack of any theoretical reasoning, and it smacks of influence from a five-fold cross-validation habit.
Is there any reason for choosing a larger training set when attempting to detect overfitting during training?  In other words, why not use $n^{0.75}$ for training and $n - n^{0.75}$ for validation if the influence really is from cross-validation practices carried over from linear modeling theory as I suggest in this answer?
I posted a similar question on stats.stackexchange.com but based on the response thought I might have a more interesting discussion here.  The concept of training for multiple epochs is, in my opinion, inherently Bayesian and thus the concept of cross-validation may be ill-suited at worst, unnecessary at best, for reasons I suggest in that post.

The reasoning will be: "The more data for training the better". Then you have to keep in mind that the validation/hold-out set has to resemble how it should work on production/testing.  The theory is that the larger the training data, the better the model should generalize.
The validation set can be much smaller, on extremely big dataset you can make it even 0.01% of the data, and there should be no problem.
In basic cases you don't even need to do the K-fold, this makes the training more expensive and only for hyperparameter search and inside the training set it needs to be done.
For your case, you can consider the split you want. Just keep the balance of the training data to be as larger as possible and validation data to resemble the best the production environment as possible.


Algorithm selection rationale (Random Forest vs Logistic Regression vs SVM) I want to understand the criteria of selection of ML algorithms, i.e., what are the guidelines on which algorithm to be selected in which case?
The reasons I know are:

Logistic regression to be picked in case we want to advise the impact on y variable on what changes on any x variable.
Random forest works good on mixed data and very effective for categorical data. Also, it does feature selection first (so dimension reduction is not needed).
Random forest not to be picked with high featured and multiple category data due to its high processing time.
SVM works well with the closely placed data points like in image processing identification of dog vs cat.

But these are not sufficient enough to pick anyone, as I don't have any reason for why which algorithm not to be picked, like when to choose SVM over Logistic regression or RF over Logistic regression.
The only rationale I have is the performance, so I run all algorithms and who ever performs best that I select (but this is not right way).

I suppose I will suggest as a starting point and expand on what you suggested by just adding the following

Knowing the type of data you are working with and it's characteristics, (categorical, supervised/unsupervised, data size etc.).
Knowing what accuracy requirements you need, timeframe and computational power you have at your disposal vs accuracy and really answering "why, am I trying to solve this problem?"

After answering these questions you can at least narrow down slightly what you may use (and eliminate those you clearly don't believe fit).  After that, I suppose it's trial and error, experience and comparing to others who dealt with similar datasets and problems.

I have this crude flow chart I found in my favourites from the scikitlearn website. Not sure where I found it to be honest.  Take it for what you will, hopefully it helps somewhat:

https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html


Which algorithm would you use for text categorisation using clustering? If you are asked to do text categorization using clustering. Which algorithm would you use and why?

tf-idf vectors are an easy start, but it can be hard to cluster very high dimensional data.

You might try topic modeling (LDA, LSI for example) to reduce the dimensionality of your features.

A newer approach is paragraph vectors, which learn a distributed representation of arbitrary-length text. Here is an implementation in python.

Learning a reasonable, lower dimensional representation of the text can help with the issues that arise trying to cluster high dimensional data.


Plotting an empty bin in a Seaborn histogram I'm going through this YouTube series on simulation by The Coding Train. I'm trying to graph some filtered random numbers, but seaborn is leaving an odd gap in the very middle of the histogram.

My data is filtered by collecting random numbers bigger than the output of some function, like $y = x^2$. I also tested graphing the output from random integers from 0 to 100 and did not get the same gap in the histogram, so I thought the issue might be my data. However, I also made a graph using pygame and there is no gap, I also inspected my data; numbers from 1 to 100 are all represented.

Code:

# Choose a function.
# Make randints a, b.
# If b > function(a), return b.
# Repeat and collect b.
# Graph b as histogram.

import collections
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

sample_size = 10**4
dice_max = 100
def function(x): return (x)**2

def main():
    # Debug: Make random data.
    # values = np.random.randint(0, dice_max, sample_size) # Test different data. Problem goes away.

    # Debug: Inspect values then press enter.
    # for i,(k,v) in enumerate(sorted(collections.Counter(values).most_common())): print(i,k,v); input()

    values = [montecarlo(function) for _ in range(sample_size)]
    pg_histograph(collections.Counter(values))
    sns.set()
    plt.show(sns.distplot(values, bins=dice_max, kde=False, norm_hist=False))

def montecarlo(function):
    """Make randints a, b. If b > function(a), return b."""
    y, b = 1, 0
    while not b > y:
        a, b = (np.random.randint(dice_max) for _ in range(2))
        y = function(a)
    return b

def pg_histograph(bins):
    """Make histograph using pygame."""
    import pygame as pg
    window_width = 800
    window_height = 800
    bar_color = pg.Color('black')
    bg_color = pg.Color('white')
    pg.init()
    window = pg.display.set_mode((window_width, window_height))
    window.fill(bg_color)

    bar_width = int(window_width / len(bins))
    max_count = bins.most_common(1)[0][1]
    for bin_, count in bins.items():
        bar_height = round(count / max_count * window_height) # Fit y values to window.
        x = bin_ * bar_width # Fit x values to bar size.
        y = window_height - bar_height # Bars extend downward from x,y, so they need to be shifted up by bar_height.
        pg.draw.rect(window, bar_color, (x, y, bar_width, bar_height))
    pg.display.update()

main()

Plotting empty bins can happen in Seaborn when plotting histograms with discrete data.
Plotting empty bins can be prevented by adding discrete=True.
sns.histplot(values, bins=dice_max, discrete=True)


Simple Explanation of Apache Flume Can anybody explain Apache Flume for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

Please

As you would explain for a non-technical person

What is it used for?
Data ingestion into a distributed datastore (e.g. HDFS). See image (I did not make the image and am only including the image for visual aid). There are other tools that will help you with the ingestion of data as well (Storm and Sqoop are mentioned).
At which stage of a BigData analysis is it used?
It is used for data ingestion into your distributed datastore (e.g. HDFS). So for example a webserver is running logging information into /var/logs/webserver.log. Apache Flume can look at that file, grab what it needs out of it and send it to HDFS. Once the data gets put into your datastore you can then utilize other tools to analyze the imported data (e.g. Hive, Pig, MR, etc.)
And what are prerequisites for learning it?
Understanding of how to write scripts, edit configuration settings, and your way around Linux would be the absolute minimum to get started. This set of instructions are old but a starting point would be to look at the hortwonworks tutorial on flume. http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/
If you would like me to elaborate more I would be glad to but I wanted to try and meet your requirement of a simple, short explanation.


how to retrain model with periodic new features? I've trained a gradient boosting classification model. But, suppose i've a set of fixed features F1,F2....Fn and new features which are added weekly (no. of actions done in that week). So, after 2 weeks dataset to be trained on is :

   Fixed           Dynamic
F1 ,F2 .....Fn    W1 ,W2

After 3 weeks

  Fixed           Dynamic
F1 ,F2 .....Fn    W1 ,W2, W3

How do we approach this problem on production server, is there any approach available which allow model to be retrained on new features and not only on new observations ?

The way gradient boosting is constructed and trained, there is not an obvious solution for this without just training from scratch. Other models might be more suitable for this (Adding this to a neural network and retraining this will take less time than from scratch I think). Another approach would be to use these time-based lag features as a time series, that way you just input your features and it will learn relationships between the past few weeks instead.


Vectorizing/Parallelizing DataFrame indexing I want to make the following Python data-processing code more efficient by replacing for loops. Is there any way to vectorize code like this?

I have a DataFrame object df that looks somewhat like:

names number
bob 5
sara 10
bob 8
foo 12
moo 16

I want to subset the DataFrame to find out all the rows associated with each name, and then perform an operation on number. This is what I am doing now:

for myName in set(df['names']):
    nameSubset = df.loc[df['names']==myName]
    operation(nameSubset['number'], **args)

'''Basically,perform an operation on the `number` column of nameSubset.'''

Is there any way to make this code run faster? Theoretically, this could be made faster if, instead of running through each myName at a time, the computer could process several myNames at any given moment. I'm not sure how to vectorize/parallelize my code to make this happen though.

Is that what you want?

In [261]: df
Out[261]:
  names  number
0   bob       5
1  sara      10
2   bob       8
3   foo      12
4   moo      16

In [262]: def my_op(ser):
     ...:     return ser.sum()
     ...:

In [263]: df.groupby('names').agg({'number':my_op})
Out[263]:
       number
names
bob        13
foo        12
moo        16
sara       10


KeyError: Selecting text from a dataframe based on values of another dataframe I have the following two dataframes badges and comments. I have created a list of 'gold users' from badges dataframe whose Class=1.
Here Name means the 'Name of Badge' and Class means the level of Badge (1=Gold, 2=Silver, 3=Bronze).
I have already done the text preprocessing on comments['Text']and now want to find the count of top 10 words for gold users from comments['Text'].
I tried the given code but am getting error:
"KeyError: "None of [Index(['1532', '290', '1946', '1459', '6094', '766', '10446', '3106', '1',\n       '1587',\n       ...\n       '35760', '45979', '113061', '35306', '104330', '40739', '4181', '58888',\n       '2833', '58158'],\n      dtype='object', length=1708)] are in the [index]". Please provide me a way to fix this.

Dataframe 1 (badges)
   Id | UserId |  Name          |        Date              |Class | TagBased
   2  | 23     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   3  | 22     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   4  | 21     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   5  | 20     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False
   6  | 19     | Autobiographer | 2016-01-12T18:44:49.267  |   3  | False

Dataframe 2 (comments)
   Id|                    Text                             |    UserId
    6|  [2006, course, allen, knutsons, 2001, course, ...  |    3
    8|  [also, theo, johnsonfreyd, note, mark, haimans...  |    1

Code
for index,rows in comments.iterrows():
  gold_comments = rows[comments.Text.loc[gold_users]]
  Counter(gold_comments)

You can consider this simple example, and take this forward to solve your problem. I have data set of quotes about animals and fruits. and I need to find out the top occurring word in each category. Count Vectorizer will be useful here
Consider data:

Code Snippet:
from sklearn.feature_extraction.text import CountVectorizer

def return_word_count_segment_wise(data, type):

    tfidf_vec = CountVectorizer(max_features=5)
    model = tfidf_vec.fit(data[data['Type'] == type].description)
    model_transform = tfidf_vec.transform(data[data['Type'] == type].description)

    feature_list = model.get_feature_names();
    count_list = model_transform.toarray().sum(axis=0)
    return dict(zip(feature_list,count_list))

return_word_count_segment_wise(data, 'Animal')

Outputs: {'cats': 3, 'is': 2, 'love': 4, 'my': 4, 'than': 3}
return_word_count_segment_wise(data, 'Fruits')

Outputs: {'fruit': 8, 'of': 5, 'that': 2, 'the': 3, 'we': 3}

Answering question asked in comment:
Try to merge both the dataframes, and then call the function while filtering out the customer segment using class (1/2/3)
merged_df = pd.merge(badges, comments, on = 'UserId')

return_word_count_segment_wise(merge_df, 1) # Get top 10 words for Gold class
return_word_count_segment_wise(merge_df, 2) # Get top 10 words for Silver class
return_word_count_segment_wise(merge_df, 3) # Get top 10 words for Bronze class

And just in case you can't merge, you can filter out the another dataframe using this dataframe
to_check = comments[comments['userId'].isin(Badges[Badges['class'] == 1].userId)]
return_word_count_segment_wise(to_check, 3)


Attributing causality to single quasi-independent variable Apologies if this isn't the correct place to ask - I'm not sure if this fits best with Stats or Data Science.

I'm using analytics to help marketers identify attributes of their users correspond to successful conversions (such as someone buying a product, signing up for a newsletter, or subscribing to a service). Attributes could be things like which site they came from (referrer), their location, time/day of week, device type, browser, etc.

What I'd like to say (although I'm not certain it's possible) is to isolate differences in conversion rate to an individual attribute, something like, '11% of your users from Facebook converted whereas only 3% of non-Facebook users converted', which would mean that the attribute 'referrer' and the level of the attribute 'Facebook' are responsible for driving conversions.

Given that I may have 100s of quasi-independent variables, is it even possible to isolate the effect to one variable and one level of that variable? As opposed to a combination of them that is more likely to be driving the difference? If so, what technique or conceptual paradigm do I use to identify which variable-level is responsible for the greatest lift in my dependent variable, conversion rate?

I would suggest you to consider either direct dimensionality reduction approach. Check my relevant answer on this site. Another valid option is to use latent variable modeling, for example, structural equation modeling. You can start with relevant articles on Wikipedia (this and this, correspondingly) and then, as needed, read more specialized or more practical articles, papers and books.


Clustering users based on buying behaviour I have a set of data which indicates purchase transaction of users (~1 million records). User can have more than 1 purchase across time. Data is spread over 6-7 months.
Attributes that I have are

userid
merchant
amount
day of purchase
date of purchase
time of purchase

Goal -

Cluster users based on their purchase behaviour, so that I can target set of users (Promotions) based on their cluster behaviour.
Eg - I want to send promotions to user who execute XX behaviour

What I have tried -

I tried KMeans Clustering in Python Sklearn for 10,000 sample records. By using "elbow method" I have identified that 6-7 cluster are optimal.
Still, Clusters are not really helpful in identifying similar group of users. Clusters does not make sense.

Questions -

1) Am I doing this correctly?

2) What would be the ideal algorithm for this type of use case. Should I be looking in Clustering or something like Recommendation algorithm (Group similar users)

3) Questions/Goal/Target group of users might change over time. How can we handle that?

Big Picture:

First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform.  Some people may tend to buy things on payday (15th and 30th of the month), some people might like to shop on Sundays, some people might like sports stores, people who shop on Wednesdays could like to buy woman's clothes, midnight shoppers could tend to focus on "as seen on TV" products.  There might be signal there, but with only 6 features (really 5), it might be tough to find it.

Normalization/Standardization

Make sure you standardize your data before using k-means clustering. Otherwise, your distance metric will loose rotational invariance and the clusters will become nonsense from feature to feature.

Feature Engineering:

You will want to do some feature engineering to help your clustering algorithm find signal in the data:

1) Get rid of userid unless it has some meaningful information like it is a proxy for when they first signed up i.e. ask yourself whether this would help the algorithm cluster the data or provide noise to confuse the algorithm. The answer is likely the latter.

2) Do you have the merchant name only? Have you one-hot encoded the merchant list? Can you turn the merchant name into one or more variables with lower cardinality like groupings for food, clothes, household, sporting, etc?  Can you find other information on merchants (other data), join it into your data and enhance this feature to make it much more useful?

3) Amount can possibly be teased out a bit, but be conservative with this one. You could possibly flag purchases over $100 or $1000 as is_expensive and is_very_expensive.  Also possibly squaring the amount could account for buyer behavior around big ticket items.  This might just add noise, so be careful with this one.

4) Day of week should be first turned into a numerical variable 1-7, but then I would also recommend turning cyclic variables into two spatial dimensions.  I explain how to do this here, and in addition to time, it can be applied to day-of-week also. I recommend keeping both the 1-7 format and the two dimensional spatial format.  Are there other days that you want to specifically highlight?  You could, for instance have an is_weekend variable that is 1 for Saturday and Sunday and 0 for other days. You could have another feature for is_boring_day that is 1 for Monday, Tuesday, and Wednesday and 0 for other days. Play with it, think outside the box... this can be a very useful feature but requires some coaxing at times.

5) Date of purchase is pretty loaded in that you can pull out the months and turn them into cyclic variables.  You can also pull out the day of the month in a similar way.  You can create at payday flag for the 15th and 30th and create a feature like 1/(1+days_since_payday) since people might be more likely to buy on payday. You can add a seasons feature from the date. Finally, you can use raw date in days since your earliest piece of data.

6) Similarly for time, you should turn this into a 2D spatial feature. This will make it easier to pull out the spending behavior of people who shop at certain times of the day.

Clustering Algorithms

k-means can be a bit blunt at times.  I suggest you try at least two more clustering algorithms besides k-means. DBSCAN is a very sophisticated clustering tool and works well for noisy data. It also benefits from not having to specify the number of clusters a priori. Experiment with changing the neighborhood size and how much noise is allowed to see how this effects your clustering. Try at least one more clustering type in the scikit arsenal... they are very easy to swap out once you do it once.

Other Methods:

Finally, you can play with trying to other supervised learning methods. Maybe try separating your data into training and testing sets and try predicting purchase amounts based on the other features.  Or try classifying the type of purchase based on the other information.  Maybe try training a SVM for classification and a Bayesian-ridge regressor for predicting the purchase amount.

Handling changing user data:

Changing data can be handled in a number of different ways.  The most obtuse is simply to retrain your model periodically, while either aggregating all of the data, or forgetting some of the older data due to a lack of relevance. The next step in sophistication is to use an online learning algorithm, which continually updates the model based on new data.

In general, just have fun with it.

Hope this helps!


does tensorflow use opencv to covert image to numpy array Does tensorflow use opencv to covert image to numpy array ?
How is feature extraction done in tensorflow ? How is it different from opencv model ?

TensorFlow has two ways of processing image data

Built-in Image Ops  ( https://www.tensorflow.org/api_guides/python/image )
tf.keras Pre-processing ( https://keras.io/preprocessing/image/ )

Feature extraction is at pixels / channel level. Higher level features (Such as Edge / Contrast) are learnt typically by CNN layers.

If needed, you can feed Image Data + Features extracted by OpenCV to TensorFlow.


best activation function for ensemble? i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will be used as input for the final model (logistic regression too).

the default base model used a softmax function. i think transform a confident value into a probability will lose much information. so i have a plan to change the softmax into activation function.

what i learn in my CNN class, Relu is the best default activation function for an image. but my case is multi-class classification email.

which activation function should i choose?
Relu, sigmoid or other?

thanks

I think you will have to experiment; there isn't generally a one-activation-fits-all for hierarchical models. Give the ones you mentioned a try, but perhaps starting with sigmoid and tanh and also a LeakyReLU.

The reason I'd perhaps leave out a normal ReLU at first is because that too would potential trim out information before the final model, as it simply removes negative weights to zero. A good final model should be able to correct for this, but I think its unnecessary to force it to.


Why does a machine learning model of English can not be directly applied on right to left languages, like Arabic Hindi, and Urdu I have a Logistic Regression model that works fine when used with an English Dataset. But, when I pass a same format data set in a different language, (Language is URDU which is right to left and UTF-8 encoded ) the model does not predict.

I understand that the preprocessing differs for URDU, Arabic, and Hindi but even if it has been performed separately, still the model is unable to predict at all.

I would like to know why the same model can not perform on different languages directly. Good or bad results are another topic. How can I make a model which is scalable enough to perform on both English and Arabic? What are the constraints in doing so?

We would need to know more about how your model actually works. Logistic regression means that the model transforms an input in numbers to output in numbers using a sigmoid function. But your input is not numbers, it's text.

That means that you're doing something already to transform the text to numbers before you're doing Logistic Regression. That's why your question is a little confusing.

Also, are you training a model on English text, and evaluating on a separate English test set, and then training a second model on Arabic text and evaluating that on Arabic test data? If you try to train a model on one language and evaluate it on another it won't work.

What library are you using? NLTK? Sklearn? All of these have simple text classifiers.

It's possible you're using a bag of words model, where each word is given an ID and transformed to a sparse vector, and the entire document is transformed to a long vector whose size is the length of your vocabulary. You might be using a tf*idf stage which weights words like "the" with a lower weight than content words.

A model like this doesn't need to break necessarily if you change language. However there are a few important points you need to know:

there will be a tokenisation stage. Arabic tokenisation in particular is different from English because some prefixes and suffixes are attached to the word without spaces (e.g. article al- ال, preposition "to" li- لِ, etc.)  You need to have replaced the tokenisation stage correctly.
If there is a stopword removal stage you should replace the stopwords with the list of the new language.
If there is a lemmatisation stage you need to find an Arabic/Hindi/Urdu lemmatiser or stemmer. Otherwise you will have a problem with sparsity, e.g. an inflection of a word will occur in the test text and your model won't recognise it because the word appeared in a different inflection in the training text.
Some languages don't use spaces between words like English, the obvious example is Chinese, however I don't think this is an issue for the languages you mentioned.

Whether the language is RTL or LTR makes no difference for the classifier since that only affects how the text is displayed on screen by a browser or word processor. The bytes are in the same order in an Arabic text file as an English text file.

However any model architecture would work fine with any language provided the preprocessing is adapted correctly to the language. If you used a character based model with neural network you could even make a model that works for any language without worrying about adapting the preprocessing, for example BERT has a multilingual pretrained model which can be run directly on almost any language. However these models tend to be bigger and require a powerful computer and even a GPU.

If you wanted a model that would work on text in any of the four languages you have mentioned, you could train four separate models and have a language identification component which first identifies the language and passes the input to the correct model.


What is a policy in machine learning? While I was reading the paper "Grounded Action Transformation for Robot Learning in Simulation", I came across the term "policy". Could someone explain to me what that actually is (in general and in the particular context of the paper)?

A policy is a state-action mapping.  A 'state' is a formalism used in AI that represents the state of the world, i.e. what the agent's idea of the world is.  The action is, naturally, what action it should take in that state.  A policy just maps states to actions.

One of the basic problems in AI is how to maximize reward over time at some task.
 One strategy for the agent is to try to understand the system and predict the results of their actions, and the reward that would follow.  Another strategy is for the agent to try lots of things and record the results.  Either of these (eventually) allow the agent to calculate a good policy - and once it's calculated, the difficult computational work is done and the agent just has to 'look up' what action to take in each state.


Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors? I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.”
(Source)

Since kNN is not a linear model (it is nonparametric), is it appropriate to use $R^2$ to evaluate it? If not, what error metric would be appropriate? I'm using the kNN classifier provided by scikit-learn in python.

$R^{2}$ is a cursory measure of goodness of fit of a linear model to the data and it is used in regression analysis. It appears to be a popular choice when deciding between linear and non-linear regression models.

It seems you intend to use kNN for classification, which has different evaluation metrics than regression. Scikit-learn provides 'accuracy', 'true-positive', 'false-positive', etc (TP,FP,TN,FN), 'precision', 'recall', 'F1 score', etc. for evaluating performance of a classifier.

bonus: kNN are simple classifiers that are used when data shows local structure but not global structure, i.e. data with same label are found in multiple localized clusters in the data space. The key parameter to tune is 'k', which should be based on the size of these localized clusters as you would stand to achieve considerably different results with different values of 'k'. You can do a grid-search (also in scikit-learn) to tune your parameters (including 'k' of course).

evaluation metric: accuracy or ROC or mean Average Precision or F1-score: There are several metrics to choose from and the choice is based on the task you are performing and what your priorities are. (you can look up these metrics for more details) but you can begin with accuracy for starters.


Predicting object by features probabilities I have the definition of an object provided as features probability.
Each object has it's own feature importance and probabilities.
For example for object "X", I have "color" feature (with the weight of 0.8) - the object can be blue in 80% of cases and black in 20% of cases. And "shape" feature (with the weight of 20%) - square in 30% and round in 70%.

I'm trying to create a "predictor", so if I'm observing something blue and round - (0.8 x 0.8) x (0.2 x 0.7) - probability for object X.

Does it make any sense mathematically?
If this method sounds reasonable enough, how should I handle really small numbers (I can have a really long vector of features, the final number will be really small)?

What you are attempting to do is so much like the Bayes decision theory; you can find the math behind it here. In probability theory and statistics, Bayes’ theorem (alternatively Bayes’ law or Bayes' rule, also written as Bayes’s theorem) describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For example, if cancer is related to age, then, using Bayes’ theorem, a person’s age can be used to more accurately assess the probability that they have cancer, compared to the assessment of the probability of cancer made without knowledge of the person's age. One of the many applications of Bayes' theorem is Bayesian inference, a particular approach to statistical inference. When applied, the probabilities involved in Bayes' theorem may have different probability interpretations. With the Bayesian probability interpretation the theorem expresses how a subjective degree of belief should rationally change to account for availability of related evidence. Bayesian inference is fundamental to Bayesian statistics.

Take a look at here and here too.


How to use machine learning to create combine of opposite images side by side Inspired by: Two Worlds Pictures

I just want to create a Machine Learning Model that can automatically combine the opposite images into 1 image.

I am thinking about 2 possible solutions:

Pose Estimation: Detect humans and their poses from image data and search an archive by pose, but totally out of context.
Land Lines: explore similar lines.

It's just my ideas, do you have any recommendations?
Thanks

Maybe something like GANs would work. It can use some base picture and change the context, add some opposite attributes. There is an example of Generative Adversarial Style Transfer Networks to clarify what I mean http://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w41/Palsson_Generative_Adversarial_Style_CVPR_2018_paper.pdf In this example age is change by the model and it looks very natural, therefor the state of appearance could be change too (from poor to rich or vice versa)
https://neurohive.io/en/news/gated-gan-style-transfer-with-multiple-styles/
The first image with a girl conveys the idea of gan using, but it should be thought over more.


What is the feedforward network in a transformer trained on? After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns.

What I learned about neural nets is that they learn based on a target variable, through back propagation according to a particular loss function.

Looking at the architecture of a Transformer, it is unclear to me what the target variables are in these feed forward nets. Can someone explain this to me?

Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be

("the cat is black", "die Katze ist schwarz").

In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but using embeddings incl. positional information). This is what you calculate your loss on, run backprop on, and derive the gradients as well as weight updates from.

Accordingly, you can think of the light blue feed forward layers of a transformer

as a hidden layer in regular feed forward network. Just as for a regular hidden layer its parameters are updated by running backprop based on transformer $loss(output,target)$ with target being the translated sentence.


KDE on TF-IDF - sensitive bandwidth I'm clustering texts based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can spread very randomly, I found Kernel Density Estimators relevant.

However, the scores of KDE are very sensitive to the choice of bandwitdh hyperparameter, which I couldn't pre-estimate. Most bandwidth values end up with either infinite score for points outside the cluster, of zero for the points in the cluster. I need a way to 'automatically' choose the bandwidth, so that my results will yield scores that make sense for the points in the cluster (larger values) and the points outside of it (smaller values).

I tried:

Both Silverman and Scott factors methods to evaluate the bandwidth depending on #points and #features, both were far from relevant in my case
GridSearchCV returns the minimal bandwidth in the grid
Different kernel types (all the relevant ones are similarly sensitive)
Reduce dimension, but as expected this severely hurt the KDE results without making the bandwidth that less sensitive

from sklearn.neighbors import KernelDensity

# indexes of points in cluster3
docs = np.where(y_pred == 3)[0]

kde = KernelDensity(kernel='gaussian',bandwidth=0.399).fit(X_tfidf[docs].todense())

# evaluate scores on all points
scores = np.exp(kde.score_samples(X_tfidf.todense()))

Note that there are ~2200 features in the TF-IDF, a few dozen points (40-120) in each cluster the KDE was fitted to, and about 4000 points in total.

Any ideas on anything (even beyond KDE) are welcome, thank you

GridSearchCV returns the minimal bandwidth in the grid

Then alter your grid to have a lower lower bound.