How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background? I am currently a postdoc and my PhD was in applied mathematics in the area of numerical analysis and electromagnetic/acoustic wave propagation. There was no statistical element to my PhD, it was completely deterministic. I took several probability/statistics and one machine learning module 5-6 years ago during my BSc, and a stochastic ODE module during my MSc but that's about it..its been all applied mathematics since then.

I am considering leaving academia and entering industry and it seems like there are far more jobs in the area of data science/machine learning than there are for my skillset.

If I left academia and began 'studying up', how long do you think it could take me to gain the skills required for a data science/machine learning position in industry?
It seems like there is a very wide variety of science/machine learning techniques and obviously there isn't time to learn all or even most of them. So what approaches are absolutely essential for data science/machine learning in industry these days and what is the most efficient route to gaining these skills?

I think you already know enough applied mathematics to begin with. You can pick-up rest of it as required.

One option is :

Start with an online course that provides high level overview of machine learning and types of algorithms (E.g.: https://www.coursera.org/learn/machine-learning)
Start applying the knowledge in real world problems as soon as possible.
Learn various types of neural networks (deeplearning.ai is one place to start)
Apply knowledge to real world problems (Such as Audio/Video classification , Natural Language)
Get an internship

This will take 5 - 6 months.


Are there better solutions on visualizing of two variables having very different range and mean on same graph? I'm now working on an analysis comparing time consuming of a pairwise experiment. AI and human processed same subject in experimental group and processing time recorded. The time consuming of AI are significantly shorter than human and two variables(time of AI route and time of human route) have different distribution which makes it is weird to show them on a same graph.

The head of record table is like this.

The graph visualization will be used in an essay focusing on showing the difference of their means and their ranges rather than the distributions. Because of that, the data should be presented objectively without transformation.

What below are my solutions and all of them seems ugly.

Is there any trick you guys using to visualize such data? Please teach me. Thank you

The only problem is that you did not use colors in first plot. That can change the game a lot!

the other point is to re-think of the term "ugly" as all your plots look normal. Specially Histogram and BoxPlots. If you want the first scatter plot to be more well-spread, you need to scale your features. Try a standard scaler and see the results. I strongly recommend checking this to get a better feeling about scaling and plots.

The other points is the term "comparing". What do you want from the comparison? If it is correlation then the first plot is showing it to some extent (which can be validated by calculating Pearson correlation and its p-value). If you want to see the difference between classes then second and third are doing a great job! you can enrich them with a F-value and justify your results.


How to train ML model with multiple variables? I am trying to learn Machine Learning concepts these days. I understand in a traditional ML data, we will have features and labels. I have following toy data in my mind where I have features like 'units_sold' and 'num_employees' and a label of 'cost_$'. I would like to train the model to learn these features and label for a particular 'city' and 'store'. For example if I perform Linear Regression, the model learn intercept and coefficient for the city and store it relates to. When I input units_sold and num_employees for next year, I get the prediction.

city        store   units_sold  num_employees   cost_$
New York    A       10          4               11000
New York    B       12          4               11890
New York    C       14          5               15260
New York    D       17          6               17340
London      A       23          5               22770
London      B       27          6               25650
London      C       22          3               21450
Paris       A       4           2               5200
Paris       B       7           3               9590

I'm trying to brainstorm about it and would like to know how to approach this problem?

This is very little data so doing much with it is very difficult. especially considering you expect to get 'next year' estimates without having any historical data to base yourself from.

However, if you want to be able to estimate a cost given your input features which are city, store, units_sold and num_employees then this problem setup is very standard in machine learning.

First I will put your data into a pandas DataFrame and I will encode your categorical features using numerical values. I did this randomly, but I think you can get better results by getting the latitude and longitude of the stores, as well as some data about the neighborhood (density, wealth, etc.).

import pandas as pd

data = {'city': ['New York', 'New York', 'New York', 'New York', 'London', 'London', 'London', 'Paris', 'Paris'],
        'store': ['A', 'B', 'C', 'D', 'A', 'B', 'C', 'A', 'B'],
        'units_sold': [10, 12, 14, 17, 23, 27, 22, 4, 7],
        'num_employees': [4,4,5,6,5,6,3,2,3],
        'cost': [11000, 11890, 15260, 17340, 22770, 25650, 21450, 5200, 9560]}

df = pd.DataFrame(data)

df['store'] =df['store'].astype('category').cat.codes
df['city'] =df['city'].astype('category').cat.codes

When I am working with very little data I always like to pull some visualizations to give me some intuition about the data. Let's first see how the features are correlated.

import matplotlib.pyplot as plt

plt.matshow(df.corr())
plt.xticks(np.arange(5), df.columns, rotation=90)
plt.yticks(np.arange(5), df.columns, rotation=0)
plt.colorbar()
plt.show()

We can see that the cost is associated directly with units_sold. That is not much of a surprise I guess. But it also correlates with the num_employees and is strongly inversely correlated with the city.

If we plot the num_employees and units_sold we can see that the correlations observed above more clearly.

A predictive model

Now we want to be able to give a model our inputs and get an estimation of the cost.

Let's first put our data into a training and testing set. This is very problematic with such a small dataset because you have a very high probability of ending up with a training set that omits a city, or a store type. This will significantly affect the abiltiy of the model to predict an output for data it has never seen.

The labels of the data are the cost.

import numpy as np
from sklearn.model_selection import train_test_split

X = np.asarray(df[['city', 'num_employees', 'store', 'units_sold']])
Y = np.asarray(df['cost'])

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

Let's start with a standard linear regression

from sklearn.linear_model import LinearRegression

from sklearn.linear_model import LinearRegression

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

plt.plot(lineReg.predict(X_test))
plt.plot(y_test)
plt.show()

  Score:  0.963554136721
  Weights:  [ 506.87136393  -15.48157725  376.79379444  920.01939237]

A better alternative is to use ridge regression.

from sklearn import linear_model
reg = linear_model.Ridge (alpha = .5)
reg.fit(X_train, y_train)
print('Score: ', reg.score(X_test, y_test))
print('Weights: ', reg.coef_)

plt.plot(reg.predict(X_test))
plt.plot(y_test)
plt.show()

  Score:  0.971197683706 Weights:  [ 129.78467277    2.034588
  97.11724313  877.73906409]

If you do another split and rerun both these models you will see that the performance varies greatly for both of them as we expected due to the limited data. However, the effect is much more pronounced for the simple linear regression.

To get a better idea of the actual results, let's do something a bit naughty. We will split the data over and over again to get different models and get the average of their scores.

scores = []
coefs = []
for i in range(1000):
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
    lineReg = LinearRegression()
    lineReg.fit(X_train, y_train)
    scores.append(lineReg.score(X_test, y_test))
    coefs.append(lineReg.coef_)
print('Linear Regression')
print(np.mean(scores))
print(np.mean(coefs, axis=0))

scores = []
coefs = []
for i in range(1000):
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
    lineReg = linear_model.Ridge (alpha = .5)
    lineReg.fit(X_train, y_train)
    scores.append(lineReg.score(X_test, y_test))
    coefs.append(lineReg.coef_)
print('\nRidge Regression')
print(np.mean(scores))
print(np.mean(coefs, axis=0))

  Linear Regression
  -1.43683760609
  [ 1284.47358731  1251.8762943   -706.31897708   846.5465552 ]

  Ridge Regression
  0.900877146134
  [ 228.05312491   95.33306385  123.49517018  873.49803782]

We can see that across 1000 trials the ridge regression was by far the superior model.

Now you can use this model to estimate costs by passing the model a vector with the features in the same order as the dataset as follows

reg.predict([[2,  4,  1,  12]])

The resulting score is

  array([ 12853.2132658])

This is not enough data to do any machine learning regression reliably. However you can get some insight into what factors affect your cost the most.


How to read the labeled enron dataset categories? I am trying to use the labeled Enron dataset (link) but I am really confused about the labeling system they use. I understand the Cat_[1-12]_level_weight is some form of confidence level. This dataset labeled by multiple students. Cat_[1-12]_level_weight increases with the number of the same label assigned by multiple students to a certain row (sample). But, what is for instance the Cat_1_level_2? From the overview (in the link), I guess Cat_1 mean "Course genre" and level_2 mean  "Purely Personal (49 cnt.)" ? If so, why the Cat_1_level_2 has values like 1, 2, 3, 4, etc.? I am really confused.

Looking at the first link from your post there is an in-depth overview of the different categories. Cat_#_level_1 denotes the top level category (12 in total) whereas Cat_#_level_2 denotes the second level category (up to 19 categories, depending on the level 1 category). E.g. an email with the labels Cat_3_level_1 and Cat_6_level_2 has the label california energy crisis / california politics (category 3.6). You therefore need to combine the two levels to get to the label that belongs to the second level category.


How to "reshape" into square matrix for numpy.linalg.solve()? I'm trying to find the intersection of lines $y=a_1x+b_1$ and $y=a_2x+b_2$ using numpy.linalg.solve(). What I can't get my head around is how to correctly make $A$ a square matrix for solve() to work. I'm familiar with solving linear equation systems, but there's something here I don't get.

What I'd like to do is:

def meeting_lines(a1, b1, a2, b2):
    a = np.array([[a1], [a2]])
    b = np.array([b1, b2])
    return np.linalg.solve(a, b)

def main():
    a1=1
    b1=4
    a2=3
    b2=2

    y, x = meeting_lines(a1, b1, a2, b2)

Where I expect $y=-3$ and $x=1$. However, this fails with numpy.linalg.LinAlgError: Last 2 dimensions of the array must be square.

Thank you very much for your help, trying to figure this out has messed up my day already!

You should formulate your lines as follows to have $(x, y)$ as unknowns:
$$\begin{align}
\left.\begin{matrix}
a_1x-y=-b_1\\
a_2x-y=-b_2
\end{matrix}\right\}
\rightarrow
\overbrace{
\begin{bmatrix}
 a_1& -1\\
 a_2& -1
\end{bmatrix}
}^{\boldsymbol{a}}
\overbrace{
\begin{bmatrix}
 x\\
 y
\end{bmatrix}
}^{\boldsymbol{x}}
=
\overbrace{
\begin{bmatrix}
 -b_1\\
 -b_2
\end{bmatrix}
}^{\boldsymbol{b}}
\end{align}$$
Therefore, the code should be:

import numpy as np

def meeting_lines(a1, b1, a2, b2):
    a = np.array([[a1, -1], [a2, -1]])
    b = np.array([-b1, -b2])
    return np.linalg.solve(a, b)

a1=1
b1=4
a2=3
b2=2
x, y = meeting_lines(a1, b1, a2, b2)
print(x, y)

which outputs:

1.0 5.0


Labels are not given for multiclass classification problem I have probably a weird question.
If you are dealing with a multiclass classification problem, do you always have already determined target output/labels?

I have e.g. a huge data set with a lot of features about different city areas (population, population density, number of services, banks and so on). I want to classify objects (houses, buildings) in these city areas based on these features, whether they are near the city center or not, let's say I want to have 3-5 labels at the end. But I don't know myself yet, how should I determine these labels.
Is there a specific approach to solve this? Had anyone a similar problem? Please advise

P.S. Earlier I calculated the distance between some object (e.g. house) and city center point (based on latitude&longitude). And based on distances I generated labels. But this approach is not universal when we have different cities of different sizes.

P.P.S. Do I have to follow probably unsupervised learning methods? Do clustering and find the clusters. Then analyze the clusters to give meanings to those identified clusters. And then solve the problem as a multiclass classification problem?

Your problem is associated to "unsupervised" learning in machine learning. You do not have a data set that has training data - meaning that data points with correctly specified labels are not known yet.

You can try different approaches to group/label your data set using the given features. You probably have to check by yourself if your model is "auto"-labeling your data correctly.

Clustering (k-Means)
Decision Trees
Auto-Encoding with NNs

More Approaches


Do data scientists use Excel? I would consider myself a journeyman data scientist.  Like most (I think), I made my first charts and did my first aggregations in high school and college, using Excel.  As I went through college, grad school and ~7 years of work experience, I quickly picked up what I consider to be more advanced tools, like SQL, R, Python, Hadoop, LaTeX, etc.

We are interviewing for a data scientist position and one candidate advertises himself as a "senior data scientist" (a very buzzy term these days) with 15+ years experience.  When asked what his preferred toolset was, he responded that it was Excel.

I took this as evidence that he was not as experienced as his resume would claim, but wasn't sure.  After all, just because it's not my preferred tool, doesn't mean it's not other people's. Do experienced data scientists use Excel?  Can you assume a lack of experience from someone who does primarily use Excel?

Most non-technical people often use Excel as a database replacement. I think that's wrong but tolerable. However, someone who is supposedly experienced in data analysis simply can not use Excel as his main tool (excluding the obvious task of looking at the data for the first time). That's because Excel was never intended for that kind of analysis and as a consequence of this, it is incredibly easy to make mistakes in Excel (that's not to say that it is not incredibly easy to make another type of mistakes when using other tools, but Excel aggravates the situation even more.)

To summarize what Excel doesn't have and is a must for any analysis:

Reproducibility. A data analysis needs to be reproducible.
Version control. Good for collaboration and also good for reproducibility. Instead of using xls, use csv (still very complex and has lots of edge cases, but csv parsers are fairly good nowadays.)
Testing. If you don't have tests, your code is broken. If your code is broken, your analysis is worse than useless.
Maintainability.
Accuracy. Numerical accuracy, accurate date parsing, among others are really lacking in Excel.

More resources:

European Spreadsheet Risks Interest Group - Horror Stories

You shouldn’t use a spreadsheet for important work (I mean it)

Microsoft's Excel Might Be The Most Dangerous Software On The Planet

Destroy Your Data Using Excel With This One Weird Trick!

Excel spreadsheets are hard to get right


Are there any tree-based models that use a genetic algorithm to generate the trees? I have a large dataset (195 features x 20m samples) that I have trained using XGBoost. I would like to see if a genetic algorithm can beat XGBoost since the data has so much noise it is prone to overfitting.
I would like to use a  tree-based model so I don't have to standardize the data, and the features do have some interrelationships.
Are there any python packages that have this all done? Ie.that can create trees through a genetic optimization process?

Very new release:
genetic-tree

The main objective of the package is to allow creating decision trees that are better in some aspects than trees made by greedy algorithms.
The creation of trees is made by genetic algorithm. In order to achive as fast as possible evolution of trees the most time consuming components are wrtitten in Cython. Also there are implemented mechanisms for using old trees to create new ones without need to classify all observations from beggining (currently in developmnet). There is planned to allow multithreading evolution.
The created trees should have smaller sizes with comparable accuracy to the trees made by greedy algorithms.
Also worth to be checked:
PyGAD

PyGAD is an open-source Python library for building the genetic algorithm and optimizing machine learning algorithms.
References:
https://hal.inria.fr/hal-01405549/document
https://www.hindawi.com/journals/tswj/2014/468324/
https://www.kdnuggets.com/2018/07/genetic-algorithm-implementation-python.html


customer segmentation with unbalanced data I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some clusters but I am more interested in the best approach to do it.

My data has basically 3 different products. Each product has a dozens of features. The issue is that one product is purchased by 95% of the customer while the other 2 are only purchased by 10 to 20% of the customers ( each customer can purchase any of the 3 products multiple time). It means that my dataset has a lot of zeros for product 2 and 3 (which is also informative).

I have done segmentation on each product individually and now I am trying to do a clustering based on all the data . I basically merged 3 datasets by customer to get one file.

Note that the data file include 1 to 2 millions of rows.

I have looked at the distribution of every variables and while some have a nicely normal distribution, many other have peak at 0 and  sometimes 1 ( I have several percentage variables).  Some examples below:
enter image description here

After some data cleaning I used K-mean in pyspark to get some clusters. I can see some interesting segment based on the distribution of some of the features in the cluster. However I am not sure if using euclidian distance is the best option in my case. I have been thinking of trying other algorithm such as k- medoid but it also come with challenges as I need to transform to panda data frame and then run it in python which might  fail with so much data.

What would be the best way to handle this scenario when you have features for 3 different products but 2 of them are not purchased as much as the first one?All the features are numerical in my file but  I also thought about adding binary variables to tell if customer purchased the product or not and then use them as features.

Split your data into zero and non-zero valued to get like 8 clusters.

That is by far the best clustering you can find in your data because of the data. I doubt you can find much more than that. Have you tested if any of the "interesting" patterns you found with k-means are better than these?


Which machine (or deep) learning methods could suit my text classification problem? I am long-time engineer with almost zero machine learning experience, who is trying to determine a good starting point to solve my problem (hopefully using machine learning).

The problem (I'll keep it simple):

Ultimately, I wish to be able to automatically assign a category to a financial transaction description
For example: "121217 POST XX123 TONYS COFFEE" with an amount of $5, should map to "Food & Drink"
Transaction descriptions are unstructured and often consist of unnatural language; also sometimes words can be truncated, or concatenated to others words
There may or may not be "features" such as country/region codes, dates, etc., in a description
Overall, there is no guarantee to the order or structure of the tokens in a description
A transaction will always have an amount

Assumptions:

I maintain a list of potential categories (maybe, 20 or 30 in total)
I could maintain a huge list of business names mapped to their distinctive category (may not be necessary to use, though?)
There is an existing set of training labelled data (raw descriptions/amounts and the category they belong to; in the thousands, not millions)

A best effort to extract a business name from a description, could be considered, but it would be great if that was not necessary for the accuracy of a final system.

I was originally thinking about NLP somewhat, but as this is fairly non-natural language, with no semantics, I believe there is no use of NLP. NER (named entity recognition) is maybe not really helpful either, as it is generally required to understand a text somewhat in order to determine entities.

I am toying with the idea of downloading GloVes pre-trained word vectors (https://nlp.stanford.edu/projects/glove/), to help determine words related to categories, though I am unsure how as of now or how well that might work. The idea might be if I trained something to say "Jimmy's Diner" -> "Food & Drink", then, for example, "Bobby's Rest" might map to "Food & Drink" too, as that is the nearest category to it in terms of word relationships/distance. It's depends on the possibility of being able to query the word embedding in such a way, as well as train it.

I guess in other to train a system using labelled data, I'd need to extract features from the description. The problem is, what features? Some features might be useless (unique identifiers, concatenated words, etc.). I'd need the system to be somewhat forgiving in terms of polluting it with useless features (avoidable to a degree, but probably not inevitable).

Either way, it would be great to hear how some of you experts might begin to approach this: what ML techniques would you deem most suitable?

I have researched machine learning and deep learning & the associated frameworks quite a bit over the last few days, but there is so many areas, with so much potential, that it is hard to to know where to begin.

You state that some of the words may occasionally be truncated or concatenated. Thus I would extract n-grams from your Strings and then use that into a bag-of-words vector.

How does this work?

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

Bag-of-Words builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.

Due to the high number of grams that will result, you will want to do some feature dimensionality reduction. You can use techniques such as PCA and LDA to determine which features (grams) are most pertinent to your decision boundary.


Pandas: Compare two Dataframe and Groupby categorical My Question is about pandas DataFrame, I have two DataFrame both follow the same structure.
df_1:
Index    Text     Category
0        Text 01        1
1        Text 02        1
2        Text 03        1
3        Text 04        1

df_2:
Index    Text     Category
0        Text 05        2
1        Text 02        2
2        Text 09        2
3        Text 04        2

I want to marge both for this purpose I use pd.concat(df_1, df_2) but it simply merge both files, but I want to merge in that manner.
df_merge:
Index    Text     Category
0        Text 01         1
1        Text 02       1,2
2        Text 03         1
3        Text 04       1,2
4        Text 05         2
5        Text 09         2

But I really don't know how to do that.

Try:
Minimum reproducible example:
 A = pd.DataFrame({"Text":["01", "02","03","04"], "Category":[1,1,1,1]})
 B = pd.DataFrame({"Text":["05", "02","09","04"], "Category":[2,2,2,2]})

 C = pd.concat([A,B], ignore_index= True)

 C.groupby("Text").Category.unique().to_frame()

Result:


How is the backbone of two neural networks trained? Suppose, I have a backbone network(convolutional neural network). After this network ends, the output is fed into two neural networks. Both building on the outputs of the feature extractor(CNN). Now if I want to train this complete network from scratch on two different tasks, the weights of the layers after the backbone network can be updated easily, but how should I update the weights of the backbone network. I mean I can compute gradients with respect to two losses, shall I take the mean for the gradients in the backbone or it has to be some weighted sum? if it is the weighted sum then how would the parameters of the weighted sum be updated?
Thanks

In General any sort of Gradient-based learning is done on scalar functions, so those functions f: ℝ^n ↦ ℝ. (In fact this is the meaning of a Gradient). Mainly if you want to define any minimization problem, you need a single value to minimize and not more.
This means: Ultimately your loss always has to be a scalar (a single number). Combining the gradients in the middle (so before backpropping into your backbone) would ultimately be equal to just combining the losses. And a weighted loss will be easier to implement.
For reference, you can look this talk by A. Kaparthy on how Tesla does multi-task learning on a single Backbone and how they deal with combining different losses.


What model should I use for multiple time series input I want to predict bacteria plate count in the water from time series(around 10000 values in a row) of water temperature on a one minute granularity, and other daily climate data including min and max temperature, rainfall, solar exposure, day_of_the_week etc for a sequence of 20 days each before the sample was collected for test. There are 700 different locations in the building.

My initial approach was to join all time series of the same location into one row/one series. I used RNN LSTM but the testing accuracy is not so good(65%-75%). I guess the reason could be that joining different time series(eg. water temperature and rainfall) may have led to gradient explosion due to the contrasting nature of data, thus I tried gradient clipping and there was no improvement in testing accuracy. I understand that in general RNN is good for time series data and CNN is good for image preprocessing. But my case is slightly different as I'm having multiple time series joined together to form one time series. I'm wondering if CNN or GRU would be a better model to use.

The data looks like this:

     x0  x1    x2   x3   x4   x5   ... x10000 Date       max_t1...max_t20 min_t1...min_t20 rf1... rf20 sol1...sol20 d_wk1... d_wk20
1    40 31.05  25.5 25.5 25.5 25   ...  33    2019-01-01 26.2  ...        20.2  ...         0 ...      32.4...       4 ...
2    35  35.75 36.5 36.5 36.5 36.5 ...  29    2019-01-03 24.8. ...        18.4  ...         0 ...      28.8          6 ...
⋮     ⋮   ⋮      ⋮    ⋮    ⋮     ⋮          ⋮

where

max_t1, ..., max_t20 represent max temperature from day1 to day20(Date day);

min_t1, ..., min_t20 represent min temperature from day1 to day20(Date day);

rf1, ..., rf20 represent rainfall from day1 to day20(Date day);

sol1, ..., sol20 represent solar exposure from day1 to day20(Date day).

d_wk1, ..., d_wk20 represent which day it was of the week from day1 to day20(Date day)

These are all the features beside water temperature data(so there are around 100 new columns in total).

Update Of Question:

I have checked many CNN case studies online but very few are on non-image data. I fit the data using CNN on Keras but the accuracy level is very low(<60%). What could go wrong? Is there anything I can do? A friend suggested XGBoost- is there a way to apply the algorithm to my data? Any idea is appreciated.

So far I've used something like this:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(merge.iloc[:,1:10170], merge[['Result_cat','Result_cat1']].values, test_size=0.2)

import numpy as np
import keras
import tensorflow
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.optimizers import SGD
from tensorflow.python.keras.optimizer_v2.adam import Adam

model = Sequential()

model.add(Dense(1000, input_shape=(10167,)))
model.add(Activation("relu"))
model.add(Dense(512, activation='softmax'))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))

model.fit(X_train, y_train, batch_size=10000, epochs=1000)
score = model.evaluate(X_test, y_test, batch_size=10000)

You mention having 2 types of data:

Water temperature with a minute-granularity
Climate data taken 20 days prior

I'm not sure what you mean with "joining the series into one row/series", but I don't think that it make sense to combine these two types as the time-frame is just too different. In this case it would probably be better to combine different layers to deal with the discrepancy. I would use a RNN layer for the water temperature and experiment with different layers (for example regular dense layers) for the remaining input, treating it as fixed inputs (like the location variable) instead of dynamic time-step input like the water temperature.

Since the climate data is 20 days prior, I don't think that it'll really influence the water temperature on a minute basis; the climate data is more like a foundation, similar to  the location variable.


LightGBM choice of evaluation metric I have past data of a large number of people who applied for a loan and their movement through 8 different stages, from start of application to loan being paid out.

I am trying to build a model that would predict how many people will get their loan completed (paid out) in the next 5 days (a working week).
I'm using LightGBM and it would learn based on past data, and then use the model on people currently at various different stages and predict how many of them will likely be completed in the next 5 days.
Those 8 stages can only happen one after another like a pipeline, so stages nearer to the last stage 'Completion' have a higher probability of being complete.

The model gives a good accuracy, but a poor precision score, there are too many false positives.

I primarily used the Mean Squared Error, any recommendations for another evaluation metric?
Since the 8 stages follow a pipeline in which they can only go one after another, can I somehow define them differently than I did?

Regarding 2) Your task sounds like a classical case for survival analysis. From the description of your problem, I cannot really say if this would be an option, but I just want to let you know about it: https://en.m.wikipedia.org/wiki/Survival_analysis


Factors of Variation in Deep Learning In the textbook im reading about Deep Learning, i found :

  When designing features or algorithms for learning features, our goal
  is usually to separate the factors of variation that explain the
  observed data [...]
  the factors are usually not combined by multiplication.

What's the meaning of the bold phrase ?

ps: what's your definition for "factors of variation" ?

Factors of variation are some factors which determine varieties in observed data. If that factors change, the behavior of the data will change.

The bold phrase said these factors are usually independent and one factor does not change by changing the value of the others.


How to select one record from multiple record for a subject during analysis? I have a dataset where I am working on a binary classification. I have two classes of subjects. One is Outpatients and Other is Inpatients. (66:33 is the class proportion)

My objective is to identify the risk factors that influence hospital admission (Inpatients).

But the problem here is, I have my dataset like as below

1) Let's say we have a subject called "John". He has visited hospital 19 times based on my data duration from Jan 2001- Dec 2005. All of his 19 visits are outpatients.

2) Let's say we have another subject called "Jack". He has visited the hospital 34 times based on data duration from Jan 2001-Dec 2005. Out of 34 visits, he has been admitted as inpatient 18 times and rest 16 are outpatient visits.

So now my question is

1) Usually for analysis, we only see one record per subject/individual. Right? But now on what basis should I pick that one record?

Meaning, for John out of his 19 visits, which one should I pick?

Similarly for Jack, out of his 18 inpatient visits, which one should I pick?

I choose only one out of 18 from Jack because we don't need his outpatient info as we already have a separate group of outpatients and jack is considered for Inpatient class (because he has inpatient records too unlike John).

2) Is it really necessary to have only one record per person for analysis? Is there anyway to do this? Or is it like I have to represent in aggregate form the info of multiple records in one record? Is there any theory that allows analysis of multiple records for an individual?

Hope my question is clear and kindly request you to help me

I will try and be as concise as possible. First, let's redefine the way you think about your data points. There can ever only be two types of visits in terms of time. Periodic and Non-Periodic. Let's call each visit an event. Some events could be related to chronic conditions where periodic visits are quite common. Some events could be related to flu, head injury, etc., These are non-periodic visits. You need to think about what you are trying to predict. Are you predicting an inpatient visit based on periodic events or non-periodic events?

1) I would not recommend picking a single record in multiple visits as it is insufficient. For example, if I come in for a visit for a Blood Pressure Check-Up and you pick that event, then that means you only get that information but where as I would be visiting for a follow up check up or something like that or it could be pre-surgery visits. Some of these events could be both inpatient and outpatient depending on the nature of the event. It is also possible that some of the outpatient visits will lead to an inpatient visit. Jack's earlier outpatient visits could be a sign of him coming in as an inpatient. All of John's visits are outpatient but will he ever be an in-patient? Is that what are you trying to predict?

2) You can definitely have multiple records per person and treat them as a sequence of visits over a period of time. This basically means that you are treating these visits as sequences. Most of the health care related risk prediction stuff is based on sequences. It is just the way you treat sentences. Each patient is described by a sentence of visits. This meaning, you can leave John's visits and Jack's visits as they are and use padding techniques to make them appropriate for a deep learning solution to classify which one would be inpatient.

You can refer to this paper.

This is definitely not concise but the number of permutations and combinations when it comes to healthcare stuff is high. Please let me know if you've got any questions.


Training Models with 1 Hidden Layer for a multi-layer neural network is stochastic gradient descent(SGD) guaranteed to reach a global optimum?

It depends on the complexity of the data. If the data representation is enough for one hidden layer then yes, it is possible, or at least it'll be able to find the best solution possible.

If the dataset is highly dimensional then it's unlikely that the neural network reaches a good generalization of the problem. So a local optimum is probably the only conclusion that the network will be able to find.


is there cross validation for xgb classification for multi labels? is there cross validation for xgb classification for multi labels?
I have been search but can not find any cross validation for xgb classifier
is using cross validation for xgb or xgb classifier worth?
and is there any example I can find?

Single or multi label doesn't make the difference. Cross validation is only split methodology. It just divides records in your data set to separate train and test splits. Python wrapper implements scikit API, so it'll work with any of the selection methods. Metrics will work too. Just remember to one-hot encode your labels.

  Is using cross validation for xgb or xgb classifier worth? and is there any example I can find?

If you only can afford it, do it for every pipeline you can. You split the data to see how your model generalizes new, unseen records. These splits are randomized, so you preferably would like to repeat it some number of times in a way all of the records has been used some times for training and testing. That's what cross validation does. It's best to obtain distribution of metrics from 50-100 train/test cycles.


Data leakage and predictive models: should we use past predictions as a feature? I want to develop a Random Forest Classifier model to predict whether or not a customer will convert 7 days from today. The model is re-trained once a week and makes predictions for the following week. The features I use were created using 2 years worth customer behavior data.

Since I'm running this model and generating predictions once a week, I have been storing the conversion predictions for every customer on a weekly basis. Furthermore, I will know if my previous predictions were correct or not.

So let's say I'm training the model again today for the coming week and I want to include following features:

last_wk_predictions = Probability of Purchase predicted last week for each customer (between 0 and 1)
did_convert_last_wk = did they convert last week? (0 or 1)

Is there a specific name for this approach? Is it considered data leakage to include past predictions?

It depends on the business problem you are solving. You might see data leakage if your initial models were not so good. For the weekly problem it will be good to get the real ground truth (even a fraction of the week's data will be good) and add it for reevaluation of the model and possible retraining. Now if you are thinking that past purchase will make the customer return back (yes if the products are good and quickly consumed making the customer repeatedly want it again) then yes adding every week's ground truth will definitely greatly increase reduce the misclassification for returning customers for same products. But as I said in my first sentence if you only include predictions of past week into this week the model might go wrong since first iterations errors will start influencing the later iterations and the model will move away from good prediction as time passes by. If you include predictions of past week into this week and the customer's returning behaviour has got nothing or very less to do with the current behavior (like non-chronic patient visiting the hospital for Outpatient visits for minor ailments) then you are introducing data leakage which will give wrong predictions as time passes by.


How to set different weights for different training samples? Suppose that your supervised learning training set is made out of 3 different datasets, merged into a big one. Because of the way each of those was labeled before merging, you might be suspicious that one of them (maybe the smallest one) is more "important" than the other ones, meaning that their labels are more reliable. The others might contain more errors.
How could you weight the most reliable data points for the ML model to pay more attention to them and increase the loss when it makes a mistake on those samples? And is there a simple way to implement this using scikit-learn?

In scikit-learn, most algorithms (SVM, Decision Trees, SGD, etc.) have a sample_weight argument that you can pass when fitting. In your case, you could provide a different weight based on which of the 3 datasets the data point comes from.
If the algorithm you want to use doesn't provide the sample_weight argument, you can always sample with replacement. Simply put, you give each sample weight, and then you create your dataset by sampling them with replacement. This means that instances with higher weights may appear multiple times in your dataset.


Binomial family in logistic regression I was asked in an interview why do we use the binomial distribution in logistic regression and how is it related to the class that we are predicting?

Could anyone explain, without any mathematical equations, why do we use binomial instead on any other distribution?

From wikipedia:

  ..., the binomial distribution with parameters n and $\rho$ is the discrete probability distribution of the number of successes in a sequence of n independent experiments, each asking a yes–no question, and each with its own boolean-valued outcome: a random variable containing a single bit of information: success/yes/true/one (with probability $\rho$) or failure/no/false/zero (with probability $\rho = 1 − \rho$).

So if you know that logistic regression is performed in order to model a binary output variable to some modelling question (i.e. to give 0 or 1, yes or no, etc.), it would make sense to base any probabilistic assumptions on a distribution, which shares this feature. Therefore, a binomial distribution may make sense compared to a continuous distribution, such as a Gaussian or Cauchy.


Keras bug NasNetlarge no top I am trying to use NasNetlarge in Keras without the top but I cant get rid of the top:
from keras.applications import NASNetLarge
model = NASNetLarge(input_shape=(224, 224, 3), include_top=False,
                    weights="imagenet")

If I use any shape other than the (331, 331, 3) shape, I get the error that I must use (331, 331, 3) if I include_top=True even though I set include_top=False. If I set the input shape to (331, 331, 3) and include_top to False, it downloads the weights with the top. Am I doing something wrong or is there a bug in the Keras module?

What is your keras version? Two things you could try are:

Update your keras to latest version
If the above does not work, open the nasnet.py file in keras, and change this part of the code:

elif default_size == 331:  # large version
    if include_top:
        weight_path = NASNET_LARGE_WEIGHT_PATH
        model_name = 'nasnet_large.h5'
    else:
        weight_path = NASNET_LARGE_WEIGHT_PATH_NO_TOP
        model_name = 'nasnet_large_no_top.h5'

    weights_file = get_file(model_name, weight_path,
                            cache_subdir='models')
    model.load_weights(weights_file)

to this:

elif default_size == 331:  # large version
    if include_top:
        weight_path = NASNET_LARGE_WEIGHT_PATH
        model_name = 'nasnet_large.h5'
    else:
        weight_path = NASNET_LARGE_WEIGHT_PATH_NO_TOP
        model_name = 'nasnet_large_no_top.h5'

    weights_file = get_file(model_name, weight_path,
                            cache_subdir='models')
    model.load_weights(weights_file, by_name = True)


How to handle differences between training and deploying of an RL agent Hi I am training an RL agent for a control problem. The objective of the agent is to maintain temperature in a zone. It is an episodic task with episode length of 10 hrs and actions being taken every 15 mins. Ambient weather is one of the state variable during the training. For training process a profile of ambient temperature has been generated for each hour of the day and used for training. I have trained the agent using PPO algorithm and the agent training is converging. I wish to deploy this model for a real world case and  have 2 questions regarding it.

If I train an agent for taking actions for 15 mins during the training process, is it ok I make my agent take actions at every 5 min interval during deployment?
If I train an agent on a particular ambient temperature profile, will the agent by heart / remember the profile used for training and expect the same temperature profile during deployment for it to work well ?

Can someone help me with these two things.

Thanks

If I train an agent for taking actions for 15 mins during the training process, is it ok I make my agent take actions at every 5 min interval during deployment?

It is impossible to say in general. Depending on the nature of the environment and controller, this may work just fine, or may completely destroy the ability of the agent to function at all.

I would suspect that you could get away with this change, if your controller is adjusting things that manage rates of change - e.g. the power levels of heating or cooling. That is because the amount of change forced per time step (by both controller and environment) is likely to scale with the size of time step. The agent is also likely to choose to continue the same action at $t' = 1, 2, 3$ that the original agent chose at $t = 1$, because the state will not have changed as much.

Much better though would be to alter your training simulation to be more accurate in this regard. There is a trade-off here. The more faithful your simulation is to real world use, the more likely it is that you will train an agent which works well in the real world. However, creating precise and accurate simulations can be hard work. I cannot tell how much work going from 15 minute time steps to 5 minute ones would be for you - just saying the change, it doesn't seem like much effort, but perhaps it is not under your control.

  If I train an agent on a particular ambient temperature profile, will the agent by heart / remember the profile used for training and expect the same temperature profile during deployment for it to work well ?

Over-fitting and failure to generalise are issues with neural networks wherever they are used. Yes, this is possible.

You can control for one or both of these changes between training and production systems - time step differences and temperature profile. Check to see whether your controller has generalised enough to deploy it, by testing your agent in a simulation with characteristics closer to production. For instance, you can have the simulation work with newly generated, and previously unseen (to the agent) external temperature profiles.

You can also, at least in theory, decrease the time step size in the simulation environment, although I recommend that you make that change during training instead, unless there is some strong reason why it is not possible. It is better to fix parameters that you can, rather than expect the agent to generalise over them - especially in your case where you only have one target value.

If you are not able to adjust your simulation to use it for testing, then you have to test your trained controller in a real-world environment. At that point you will want to monitor it closely to see how well it performs. That is good practice regardless of an additional test phase in simulation. Having a test phase in simulation is simply a good idea to mitigate risk and reduce costs.


Gradient for starting Backpropagation I was reading this nice tutorial about Pytorch's basics:

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

In the first example (pure Numpy), the author starts the backward phase by setting as his "first" gradient the numerical result of the loss function calculation multiplied by the constant 2:

grad_y_pred = 2.0 * (y_pred - y)

Why does he multiply by 2?

This is because of the loss assumption is the (Mean) Squared Error $\mathcal{L} = (\hat{y} - y)^2$ and the derivative is

$$
\frac{\partial}{\partial \hat{y}} \mathcal{L} = 2 (\hat{y} - y)
$$

which is then passed "backward" for use in the chain rule.


How to recommend items after finding similar users in recommendation system As the title explains my problem, I'm done with creating a recommendation system that can give me similar users for any given new user. The problem I face is, If I extract the list of products that these similar users liked the most, how do I weight these items and recommend, say only 3 of these items.

You can use some kind of distance metrics.

For example, you identify three users that have the lowest distance from your target user and propose one item from each, or three items from the closest user, whatever works best for your usecase

I guess you've used clustering to cluster the users together. Maybe that using product clustering in conjunction with user clustering can give you good results?


Prediction interval for general non-linear regression I have been working on a predictive model. With each prediction, we need to provide a score to express the confidence about our prediction. So I am looking at prediction interval (PI). In linear regression, I believe these can be obtained and well-documented. However, I am yet to find much reference for non-linear regression (such svr, gbr or other blackbox method for regression). Two methods that I have seen are given below:

1) Using bagging, we can generate many point prediction of each new data point, and then we get the interval from the distribution of these predictions around each new point.

2) using Quantile regression to get the upper and lower bound of the new point.

Personally, I do like the bagging method, although I don't feel very convinced. Hence, I am reaching out to the community to get a general opinion or some other ideas that I haven't seen so far.

Bootstrapping (which I think you are referring to as bagging, a specific algorithm that incorporates bootstrapping) is your friend here.

Simply train N models using N datasets, where these N datasets are created by sampling from your original dataset with replacement. Using these N different models, generate N predictions on an observation of interest.

You can now use any sort of method to derive the confidence interval of interest with these N predictions, whether that is using a normality assumption, empirical percentiles (simple bootstrapping), etc.

The one drawback I see with this method is that if you have many observations to predict, this will quickly become computationally expensive. That being said, bootstrapping is easily made parallel.


Does running a Decision Tree classifier several times help? To introduce, I am a novice in ML techniques. I recently had to write a scikit-learn based decision tree classifier to train on a real dataset. Someone suggested me that I must run mu model several thousand times and plot the accuracies on a graph. Here's the rub: I manually ran it around 20 -30 times and every time it gave the same accuracy (for both gini and entropy base). Is that wrong? Should it show slight variations every time?

The scikit-learn DecisionTreeClassifier takes a parameter called random_state. If this is set to an integer, your model should produce the same results every time.

The person suggesting you run the model many times would be correct, assuming you allow for no set random state. This means the results should be slightly different every time, because there is some random selection going on in the algorithm. Here is an example from the splitter classes:

# Draw a feature at random
f_j = rand_int(n_drawn_constants, f_i - n_found_constants,
               random_state)

If are are not setting that random state (or any other kind of random seed), I am not sure off the top of my head, why or how the results would always be identical.


How to design batches in a stateful RNN I am using TF Eager to train a stateful RNN (GRU).

I have several variable length time sequences about 1 minute long which I split into windows of length 1s.

In TF Eager, like in Keras, if stateful=True, "the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch." (source)

Thus, how should I design my batches? I obviously cant sample random windows from random sequences. I also cant split a sequence into windows and place adjacent windows in the same batch (e.g. batch 1 = [[seq1 0-1s], [seq 1 1-2s], [seq1 2-3s], ...]), as the state from the previous window wont get passed to the next window, as is the point of a stateful RNN.

I was thinking of mixing sequences in the same batch as in:

batch 1 = [[seq1 0-1s], [seq2 0-1s], [seq3 0-1s], ...]
batch 2 = [[seq1 1-2s], [seq2 1-2s], [seq3 1-2s], ...]
...

However, there the issue is that the sequences have different length, and thus some will finish before others.

So what is the best way to implement this?

(FYI, I couldn't find anything in the academic literature or blogoshhere which discusses this, so refs would be great)

Thanks!

Your specific case

After [seq1 0-1s] (1st sec of long sequence seq1) at index 0 of batch b, there is [seq1 1-2s] (2nd sec of the same sequence seq1) at index 0 of batch b+1, this is  exactly what is required when we set stateful=True.

Note that the samples inside each batch must be the same length, if this is done correctly, difference in sequence-length between (not inside) batches causes no problem. That is, when all samples from batch b are processed, then next batch b+1 will be processed, and so on and so forth.

A general example

As a general example, for stateful=True and batch_size=2, a data set like

seq1: s11, s12, s13, s14
seq2: s21, s22, s23
seq3: s31, s32, s33, s34, s35
seq4: s41, s42, s43, s44, s45, s46

where sij denotes j-th time step, must be structured like

    batch 1         batch 2         batch 3         batch 4

0   s21, s22        s23, <pad>      s31, s32, s33   s34, s35, <pad>   ...
1   s11, s12        s13, s14        s41, s42, s43   s44, s45, s46

or like (with overlap)

    batch 1         batch 2         batch 3

0   s21, s22        s22, s23        s23, <pad>    ...
1   s11, s12        s12, s13        s13, s14

where, for example, long sequence s21, s22, s23 (3 time steps) is broken down to two sub-sequences s21, s22 and s23, <pad>. Also, as you see, it is possible to have batches with different sequence lengths (by using a custom batch generator).

Note that <pad> (padded values) should be masked to prevent RNN from considering them as actual values (more info in this post). We can also avoid using padded values by opting for batch_size=1 which might be too restrictive (more info in this post).

Here are two examples of a sequence with 5 time steps:

            s11   s12   s13   s14   s15

example 1   23,   25,   27,   24,    28     # 5 temperature readings for t to t+4

example 2   I,    like, LSTM, very,  much   # 5 128-dim word embeddings

Some resources

You may find this article on stateful vs stateless LSTM helpful. Some quotes from the article:

  The stateless LSTM with the same configuration may perform better on
  this problem than the stateful version.

and

  When a large batch size is used, a stateful LSTM can be simulated with
  a stateless LSTM.

This blog on Stateful LSTM in Keras by Philippe Remy
Some opinions on Keras github, like this.


How to identify similar words as input from a dictionary Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below

Input Terms

Subaaaash
Paruuru
Mettromin
Paracetamol
Crocedinin
Vehiclehcle
Buildding

Dict terms #this dict has around million records and I have to find a closest match for input term from this dictionary

Metformin 250 MG
Metformin
.....
Crocin
Vehcile
Paru
Subash

Now, I expect my output to be like as shown below

As you can see that red colored Paru is not a correct match for paracetamol, but that's the closest match we can get for the input term paracetamol from the dictionary. So, I would like to do matching based on

1) Word sounds (when pronounced). Phenotics

2) Spelling mistake corrections

3) Find the best matching word from the dictionary for input terms

Can you let me know how can I do the above?

So, your question concerns how to effectively translate the input words into their proposed correct words (e.g. Paruuuu --> Paru) via phonetics and spelling mistake corrections.

My first idea on this would be to use a deep sequence to sequence model. In a sequence to sequence model, we encode the input word (e.g. Paruuuu) as a sequence of characters into an encoder (effectively an RNN / LSTM, etc.) into a "hidden representation".
Then you decode your hidden representation as a sequence of phonemes (which denote how we pronounce the word) with a decoder (again, another RNN / LSTM, etc.).
Then, we can take this sequence of phonemes as input into another encoder and then decode with a neural network, where the output layer is a softmax layer, which computes the probability distribution over all words in your vocabulary (in you Dict terms) and select the word the highest probability.
 So overall, we have:

Encode input word --> Decode output phoneme sequence --> Encode output phoneme sequence --> Decode with neural network to classify word.

 The proposed method is supervised, so of course, you will need examples of input words and their correct "translations" (e.g. ("Paruuu", "Paru"))

 Here is an article which gives a good intuition behind sequence to sequence models which can classify your input words: https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346


Python Machine Learning/Data Science Project Structure I'm looking for information on how should a Python Machine Learning project be organized. For Python usual projects there is Cookiecutter and for R ProjectTemplate.

This is my current folder structure, but I'm mixing Jupyter Notebooks with actual Python code and it does not seems very clear.

.
├── cache
├── data
├── my_module
├── logs
├── notebooks
├── scripts
├── snippets
└── tools

I work in the scripts folder and currently adding all the functions in files under my_module, but that leads to errors loading data(relative/absolute paths) and other problems.

I could not find proper best practices or good examples on this topic besides some kaggle competition solutions and some Notebooks that have all the functions condensed at the start of such Notebook.

I don't think there are best practices in this developing area yet but, in addition to cookiecutter, there were some interesting ideas shown in a tutorial at the SciPy 2016 conference.
Personally, I try to minimize the number of sub-folders in a project unless I have a really good way to distinguish them and have good reason to keep them separate. Bad organization is nearly as bad as no organization. I think best practice might depend on use case -- not every project needs the same amount of boilerplate.


how are split decisions for observations(not features) made in decision trees I have read a lot of articles about decision trees, and every one of them only focused on telling how a feature/column is considered for split, based on criterion like gini index, entropy, chi-square and information gain. But, not one talked about the observation part.
Example: Let's say I have a dataset with 3 independent features and 1 discrete target variable, namely height_in_cm(like 130, 140), performance_in_class(like below average, average, very good), class(like 7th, 8th or 10th class) and plays_cricket(1 for yes & 0 for no) is the target variable. So, with target variable as the root node and then for splits, I may try all the features iteratively and settle with the one that I had most information gain with or with most pure nodes. ex, let's start with any variable, let's take the first one, height_in_metres and after two splits with two child nodes, like height < 120 in one child node and another child node with height > 120 and i will then calculate gini impurity and they turned out to be 0.45 and 0.49 respectively.
Questions:

just like i iteratively try all the feature combinations, do i also need to try all the combinations of a feature split, for above case, height < 100 & height > 100, then height < & height > 110 & height < 90 & height > 90 and so on. How to do this and what's the efficient way.
just like there are metrics like gini impurity and entropy to measure the quality of a feature split, are there any metrics to measure the quality of split based on observations?

Do i also need to try all the combinations of a feature split, for above case, height < 100 & height > 100, then height < & height > 110 & height < 90 & height > 90 and so on

Suppose that you only have 3 posible values on height 80,90,100. Then the decision tree will try to split on if X >= 80,X >= 90 and X >= 100. If the instance does not satisfy the condition will fall on the other leaf.
So if the split is above the threshold it will fall into another leave, if its below it will fall into another. And then there recursively.

So the answer to this is NO. It will only try on of the logical conditions -- >= for example. I believe this depends on the implementation.
I encourage you to build a decision tree from scratch to fully understand the algorithm.
I reccomend you this video https://www.youtube.com/watch?v=y6DmpG_PtN0 but there is a lot of tutorials there. Choose the one that suits you most.


Version of Perceptron If we change the $ywx<0$ condition (for performing update) to $ywx<1$ like in SVM (but without adding regularization to maximize the margin), is there any difference from the basic perceptron (the one with the aforementioned $ywx<0$ condition)?

Old question, but in case anyone is still interested in an answer...
In the perceptron algorithm a point $x$ has a label $y$ equal to 1 or -1. The predicted label for a point is $w\cdot x$. The goal is to separate the points with an hyperplane orthogonal to w in such a way that the points with label 1 are on one side and the points with label -1 are on the other side. Mathematically, the "sides" of the hyperplane are characterised by the equations $w\cdot x>0$ and $w\cdot x<0$, respectively. ($w\cdot x=0$ is the equation of the hyperplane itself).
Therefore $x$ point is correctly labelled (it is on the right side of the hyperplane orthogonal to $w$) if

$y=1$ and $w\cdot x>0$, or
$y=-1$ and $w\cdot x<0$

In both cases, a correctly labelled point corresponds to $yw\cdot x > 0$, while an incorrectly labelled points corresponds to $yw\cdot x \leq 0$.
The whole algorithm and the proof of its convergence are based on this simple observation.
If we change the condition for update to $yw\cdot x < 1$, the algorithm would not be able to find a separating hyperplane. For example, consider the simple dataset $x_1 = (2,0), y_1= 1$ , $x_2 = (-2,1), y_2= -1$.

Start: $w=(0, 0)$.
Step 1: $y_1w\cdot x_1 = 0 < 1$ therefore $w \leftarrow w + y_1x_1 = (2, 0)$
Step 2: $y_2w\cdot x_2 = 4 > 1$, do not update $w$.
Step 3: $y_1w\cdot x_1 = 4 > 1$, do not update $w$. We went over all points without updating $w$, therefore exit the loop.
Result: the hyperplane orthogonal to $w=(2, 0)$ is the y-axis, which clearly does not separate $x_1$ from $x_2$.

TLDR;  Update conditions different from $yw\cdot x \leq 0$ do not offer any guarantees for finding a separating hyperplane.


Encoding features for multi-class classification I have a question regarding how to setup a dataset for modeling.
Let’s say I have a dataset representing which car a person will buy depending on some characteristics:
The dependent variables are individual cars (Car 1, Car 2, … Car 100).
The independent variables are:
Budget (of the buyer)
Favorite Color (of buyer)
…..
…..
Color (of Car 1)
Color (of Car 2)
….
Color (of Car 100)
MPG (of Car 1)
MPG (of Car 2)
…..
MPG (of Car 100)
Let’s assume this is a multi-class classification problem. So, only one of the cars can be chosen in each situation.
My question is: is it appropriate to have independent variables like that - that are specific to each of the dependent variables? (Color of Car X, MPG of Car X, …). Is it appropriate to just fit a row like that into a model? How does the model know to understand that each of the Colors are discussing the same feature? Color
Lastly, is there a name for this type of data/problem? I'm not sure how to look for it on Google.

Color is a categorical feature.
One of the most common methods to encode categorical features is one-hot encoding. Color could be encoded as an indicator vector. The color of the current car would have a 1 at the appropriate index. For example, [1, 0, 0, …, 0] for a red car and [0, 1, 0, …, 0] for a blue car.
There are other options for encoding categorical features such as binary, count, hash, or label encoding.


Getting Error in RStudio while loading a package- 'tm' I've installed the R package tm. Now I am trying to load it, but it's giving me this error and warning everytime, I try.

The package is in this location-

‘C:/Users/Google/Documents/R/win-library/3.3’

Error Message--

Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) :
  there is no package called ‘slam’
In addition: Warning message:
package ‘tm’ was built under R version 3.3.1
Error: package or namespace load failed for ‘tm’

Any thoughts?

This is a dependency error. You're just missing one of the packages that tm assumes you have.

One solution is to simply directly install that package:

install.packages("slam")

another is to use a package manager that takes care of this for you:

install.packages("pacman")
pacman::p_load(tm)


Autoencoder or layer-based dimensionality reduction? I have a few TB of wide data. I want to reduce the number of features in my dataset before feeding my dataset into a classification model... or should I not?

Obviously, I will want to try both methods if my cluster budget allows, but is there any obvious theory as to why either of the following approaches would be better than the other?

Autoencoder (tanh, tanh, tanh layers)
Or: drastically reduce the number of inputs, perhaps via dropout, to the second and third layers of the classification model itself. Also, droput

Autoencoder is a good option for doing dimensionality reduction, however, you will need another training for autoencoder.

If you want dimensionality reduction before feeding into a classification model, how about PCA and t-SNE? They don't require training process for doing dimensionality reduction.


How to access the data of the column on which some groupby operation has been carried out? Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries in the second column for each entry in the first, applying count method grouped by entries on the first is the classical solution. And it works as intended.
Here's something concrete-

There are multiple entries for each of the county e.g. California. I found the number of entries for each county by executing this code-

Now I want to have something like this-
|               county|             number_of_venues|
|---------------------|-----------------------------|
|Alameda              |                  some number|
|San Diego            |                  some number|
|Yolo                 |                  some number|

and so on.
I tried achieving this by-
df = pd.DataFrame(california_venues.groupby('county').count())

Which gives me this weird dataframe-

Where county is not a key. It just sits on top of it. Trying to access the entries of the column through county raises KeyError.
My plan was to extract the two relevant columns once the desired dataframe had been formed.And finally rename the column name(s).
What are my options to get the form described by me in the markdown table?

As county is not a key when you make the new dataframe so you can't access it.
So you just need to reset index so that county can be accessed when required.
df = pd.DataFrame(california_venues.groupby('county').count().reset_index())


How does PCA and clustering treat different type of predictors where order of their values signify the result differently? Assume we have 3 variables for 3 countries as shown below:

|  Country |  GDP  |  Child Mortality  |  Health Expenditure  |
|  X       |    5  |                8  |                   4  |
|  Y       |    7  |                3  |                   1  |
|  Z       |   12  |                4  |                   9  |

Now if GDP and health expenditure increase, it is a better country. While if child mortality increases it is a worse country.

Q1. How does PCA and clustering take care of these type of predictors appropriately without having the intelligence of knowing which set of predictors makes it a better country as value increases and which set of predictors make it a worse country as value increases?

Q2. Do we need to translate some predictors like child mortality with some inverse values (eg: divide by 1 or something else on scaled data), so that even for that column an increase in value signifies a better country, so the behavior is similar to columns like GDP / Health Spend etc, so our PCA and clustering algorithms are better able to model and provide good clusters ? Note, that there is no y in the model, we just have a list of countries which we need to cluster.

Anything of this sort is done in modelling and any techniques for these ?

OR what I am seeing here is invalid, and we just dont need to care about any translation (other than simple feature scaling) and directly run PCA/clustering algorithms ? And if we do this how does it take care to create appropriate clusters mathematically ?

Similar combinations of values will be clustered together regardless of their meaning in your model.  The meaning of the clusters is for the model builder to determine.

You could engineer another feature (call it ‘goodness’) that takes the GDP and divides by child mortality (for instance) in order to add more variables to determine clusters, by dividing GDP by mortality counties with lower GDP and higher mortality rates would have a smaller “goodness” value while higher GDP and lower mortality would have higher “goodness” values and then would give the model a better chance to cluster on this factor.


Using a decision tree with 3 dimensional input points I implemented a normal classification tree (that uses the Gini index to look for a split). I am using it to predict the age of people. My input data was a series of points on 1 axis (Only X coordinate), and output was an age (9 years old, 10 years old....).

Example:

p1 = 1.3, p2 = 3.4, p3 = 2.1 ........ => Output = 9 years old

p1 = 1.4, p2 = 2.4, p3 = 2.6 ........ => Output = 10 years old

My tree looks something like (just an example):

if p1 > 1
    if p2 < 3
        output = 10 years old
    else
        output = 9 years old
else
    ...

This is working fine.

Now I want to modify it, to work with 3D points. I mean points that have X, Y, Z and not just X.

So my input data would be like:

p1 = [1.2, 1.5, 4.3], p2 = [4.2, 1.3, 5.2] .... => output = 9 years old

How do I approach this? should my tree use each coordinate as a seperate input (p1x, p1y, p1z, p2x, p2y....) or is there an approach that can be taken for my case?

Note: I am using classification trees but if you think there's another way of predicting in my case, I would love to know about it.

Also I'm using python code, but I'm looking for the logic not the programming part.

Thanks

To use a single vector (p1x, p1y, p1z, p2x, p2y....) per person would be the most easy and obvious way.

Maybe, your prediction would also improve by including some other transformations of these features. But usually trees are eager to use all the information you feed them efficiently. So just a 9-dimensional vector would do.


Neural Networks : Can I use both sigmoid and tanh as activation functions? In a neural network architecture  can I use the sigmoid function in some layers and the tanh function in the others? Is it a good choice?

Yes you can. There are no hard rules against having different activation functions in any layer, and combining these two types should give no numerical difficulties.

In fact it can be a good choice to have tanh in hidden layers and sigmoid on the last layer, if your goal is to predict membership of a single class or non-exclusive multiple class probabilities. The sigmoid output lends itself well to predicting an independent probability (using e.g. a logloss (aka cross-entropy) objective function).

Whether or not it is better than using sigmoid on all layers will depend on other features of your network, the data and the problem you are trying solve. Usually the best way to find out which is better - at least in terms of accuracy - is to try out some variations and see which scores best on a cross-validation data set. In my experience, there often is a small difference between using tanh or sigmoid in the hidden layers.


What are the best practises to decide whether a variable is categorical? What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are the best strategies when categorising variables?

For example, the dataframe I'm working has several categorical variables such as is_holiday that has labels for several holidays. However certain variables like visibility_in_miles suggest that those too need to be treated as categorical. part of the reason is that while most variables have hundreds of unique values, some have only 9 points.

The number of categories within a variable does not matter whether a variable is categorical or not.

Categorical variables are mutually exclusive, unordered groups. For example, Christmas and Halloween are different holidays but have no order in the concept of "holiday-ness".

Ordinal variables which are mutually exclusive, ordered groups with no consistent measure of the distance between the ordering. For example, ranking items (e.g., first, second, third, …). There could be big (or small) differences between each of the places.

Numeric variables have consistent differences between values. The difference between 10 and 11 miles is the same as the difference 20 and 21 miles because "mile-ness" is a consistent measure.


Clustering customer data stored in ElasticSearch I have a bunch of customer profiles stored in a elasticsearch cluster. These profiles are now used for creation of target groups for our email subscriptions.

Target groups are now formed manually using elasticsearch faceted search capabilities (like  get all male customers of age 23 with one car and 3 children).

How could I search for interesting groups automatically - using data science, machine learning, clustering or something else?

r programming language seems to be a good tool for this task, but I can't form a methodology of such group search. One solution is to somehow find the largest clusters of customers and use them as target groups, so the question is:

How can I automatically choose largest clusters of similar customers (similar by parameters that I don't know at this moment)?

For example: my program will connect to elasticsearch, offload customer data to CSV and using R language script will find that large portion of customers are male with no children and another large portion of customers have a car and their eye color is brown.

One algorithm that can be used for this is the k-means clustering algorithm.

Basically:

Randomly choose k datapoints from your set, m_1, ..., m_k.
"Until convergence":

Assign your data points to k clusters, where cluster i is the set of points for which m_i is the closest of your current means
Replace each m_i by the mean of all points assigned to cluster i.

It is good practice to repeat this algorithm several times, then choose the outcome that minimizes distances between the points of each cluster i and the center m_i.

Of course, you have to know k to start here; you can use cross-validation to choose this parameter, though.


evaluation metrics for multiple values per session I have an application that executes my foo() function several times for each user session. There are 2 alternate algorithms that i can implement as "foo" function and my goal is to evaluate them based on execution delay .

The number of times foo() is called per user session is variable but will not exceed 10000.
Say delays values are:

Algo1: [ [12, 30, 20, 40, 24, 280] , [13, 14, 15, 100], [20, 40] ]
Algo2: [ [1, 10, 5, 4, 150, 20] , [14, 10, 20], [21, 33, 41, 79] ]

  My question is whats the best metric to pick the winner ?

possible options

average from each session, and then evaluate cdf
median from each session and then evaluate cdf
anything else ?

Here is a suggestion:

Standardise everything (if you ommit this than some big number like 9999 can ruin everything), than take average value per user session. Than, optionally, mutliply this number by x/10 for example, where x is the sample size in the use session (think of it like evidence where more samples add more confidence) and finally average by number of sessions for the algorithm.


How to extract a column that has the highest value within row in Hive? I have a table, more or less in the following format

col1 col2 col3 ... col100
val1 val2 val3 ... val100

Where val* are doubles. Is there a way to extract for each row in which column is the highest value within row in Hive?

For example, for table like

col1 col2 col3
2     4   5
8     1   2

I would get

col3
col1

I can't test in hive, but a possible SQL query is as follows (greatest returns the maximum value from the list):

select
  case
   when col1 = greatest(col1,col2,col3) then 'col1'
   when col2 = greatest(col1,col2,col3) then 'col2'
   when col3 = greatest(col1,col2,col3) then 'col3'
  end as c1
from test;

Additional note:
you should check, how ties are to be handled, in my solution I simple take the first column.


Find correlation between grades from two raters The question is whether we can find a correlation between two sets of grades (categorical data).
Let’s say we have a dog competition and there are 1000 dogs participating.
There are two rounds of assessment
first round
dog owners give their assessment on the scale from A to C. Where A is excellent and C is bad. There are four criteria for assessment during both tours (behaviour etc).
second round
one judge gives his assessment of one dog based on the same assessment criteria as in round 1. however, grades vary from M - meeting expectation, E - exceeding expectation, B - Bellow expectation.
We understand that M is B, E is A and B is C.
After two rounds our table would look like:

| dog             | round one | round two |
| --------------- | --------- | --------- |
| Dog1_criteria1  | A         | B         |
| Dog1_criteria2  | A         | E         |
| Dog1_criteria3  | A         | E         |
| Dog1_criteria4  | B         | M         |
| Dog2_criteria1  | A         | E         |
| Dog2_criteria2  | B         | M         |
| Dog2_criteria3  | A         | E         |
| Dog2_criteria4  | C         | B         |
....

How do we find a correlation between the two sets of answers? Thank you!

You can treat this as a type of inter-rater agreement problem and use Cohen's Weighted Kappa or a similar measurement. Weighted kappa takes into account the distribution of ratings for each round and the difference between grades.
Three matrices are involved: the matrix of observed scores, the matrix of expected scores based on chance agreement, and the weight matrix. Weight matrix cells located on the diagonal (upper-left to bottom-right) represent agreement and thus contain zeros. Off-diagonal cells contain weights indicating the level of separation between ratings. Often, cells one off the diagonal are weighted 1, those two off 2, etc.
source:Wikipedia
The equation for weighted κ is:
$$\kappa = 1 - \frac{{\sum_{i=1}^k}{\sum_{j=1}^k}w_{ij}x_{ij}}{{\sum_{i=1}^k}{\sum_{j=1}^k}w_{ij}m_{ij}}$$
where k=number of codes and $w_{ij}, x_{ij}, m_{ij} $ are elements in the weight, observed, and expected matrices, respectively. When diagonal cells contain weights of 0 and all off-diagonal cells weights of 1, this formula produces the same value of kappa as the calculation given above.
In practice, you may want to use an implementation in Python, R, or a statistical software package rather than manual calculations.
Here is some intuition from a similar example on why to use weighted kappa (round 2 grades are converted to round 1 scale):
| dog             | round one | round two |
| --------------- | --------- | --------- |
| Dog1_criteria1  | A         | C         |
| Dog1_criteria2  | A         | B         |
| Dog1_criteria3  | A         | A         |
| Dog1_criteria4  | B         | B         |
| Dog2_criteria1  | A         | A         |
| Dog2_criteria2  | A         | A         |
| Dog2_criteria3  | A         | A         |
| Dog2_criteria4  | C         | C         |

You could look at % agreement and give a score of 6 of 8, or 0.75. That seems good, but suppose the judges just gave everyone an A. That would also be 0.75. So we need to factor in the frequency of the grades and the probability of agreement between each combination. That's where the expected matrix comes in.
Then there is the degree of agreement between two ratings. You usually want to assign more agreement to an A/B or B/C than to an A/C. And the differences may not be linear at all. The weight matrix allows you to account for the differences.
The observed matrix is simply the count of observations for each possible pair of ratings.
A final note: there are several variations of Kappa, such as quadratic weighted kappa. Most of them should work well for comparing grades.


Does the predict function in machine learning understand categorical data I understand that before feature engineering one has to split the dataset into train and test data, so as to avoid bias in the analysis. I also understand that the machine learning model does not understand data apart from numerical data, thus encoding is required, which is a part of feature engineering. My question is, do I encode the test data separately or does the prediction function understand categorical data.

This depends somewhat on the model and language (implementation).

First please understand that categorical data is not the same as non-numerical data! Many models can handle categorical data (e.g. regression formats) just fine and some can even handle non-numerical data.

Finally and most important for you feature engineering has to be done on the whole data set before the train/test split. All models can only predict on data that has the exact same input formats as the data it has been trained on!

So yes if you one-hot encoded some column it also needs to be one-hot encoded for the prediction.


Function related to Orange GUI I am a beginner to Orange GUI data mining software and I have some questions related to that.
First, I would like to know if there is any search function so that I can search if a column contains a specific word.
Second, does Orange has any functions for dimension reduction?

Yes, Orange has a Dimensionality Reduction Widget under the Unsupervised Learning tab. Click PCA (Principal Component Analysis)


sklearn FeatureUnion vs ColumnTransformer I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ColumnTransformer vs FeatureUnion, and they are not the same. Please advise if the following are equivalent, if not what the problem is. The data is from kaggle
https://www.kaggle.com/ronitf/heart-disease-uci

from sklearn.model_selection import train_test_split
cat_attribs = ['sex','cp','fbs','restecg','exang','ca','thal']
num_attribs = ['trestbps','chol','thalach','oldpeak','slope']
X_train,X_test,y_train,y_test = train_test_split(heart_df,y,test_size=0.25,random_state=100)

Approach #1, using column transformer

from sklearn.compose import ColumnTransformer
ct = ColumnTransformer([('oneHot', OneHotEncoder(categories='auto'),cat_attribs)
                       , ('minMax',MinMaxScaler(),num_attribs)])
ct_result = ct.fit_transform(X_train)

Approach #2, using FeatureUnion

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import FeatureUnion
from sklearn.pipeline import Pipeline
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names].values

num_pipeline = Pipeline([('selector', DataFrameSelector(num_attribs)),
                        ('minMax',MinMaxScaler())])
cat_pipeline = Pipeline([('selector', DataFrameSelector(cat_attribs)),
                        ('oneHot',OneHotEncoder(categories='auto'))])
full_pipeline = FeatureUnion(transformer_list=[
    ('num_pipeline', num_pipeline),
    ('cat_pipeline', cat_pipeline)])
fp_result= full_pipeline.fit_transform(X_train)

I've tried this with an sklearn builtin dataset rather than yours, but the only difference appears to be the order of the columns.  Switching the order of the elements in the transformer lists produces the same results.  (In both cases, the numeric columns and categorical one-hot encoded columns are separated from each other, but are placed in the order that they appear in the transformer list.)


Where is the Backward function defined in PyTorch? This might sound a little basic but while running the code below, I wanted to see the source code of the backward function:
import torch.nn as nn

[...]
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
loss.backward()

So I went to the PyTorch GitHub and found the CrossEntropyLoss class, but without any backward function defined. Moving up, CrossEntropyLoss extends _WeightedLoss >> _Loss >> Module then still nothing.
So, where is the backward function defined?

Backward function is same for all type of layers.
Look at 155 number line here.
https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/tensor.py
and it will forward you to here
https://github.com/pytorch/pytorch/blob/35bd2b3c8b64d594d85fc740e94c30aa67892a34/torch/autograd/__init__.py


Multi-class classification v.s. Binary classification If a training set has 5 classes including :
"label-A", "label-B", "label-C", "label-D", "others".

But the problem is much simpler, it wants to distinguish each input whether it is belongs to "label-ABCD" or "others". In this case, there are two solutions to solve this problem in my mind.

Solution-1 : Train a 5-classes classifier, when the classifier predicts the input as "label-A" or "label-B" or "label-C" or "label-D", we relabel it as "label-ABCD".

Solution-2 : Train a 2-classes classifier, we relabel the data as "label-ABCD" which is labeled as "label-A" or "label-B" or "label-C" or "label-D". And then it becomes a binary-classification problem.

My question is...

Which way can the model get a better performance in "theorem"?
In real case, this two cases get almost the same performance by CNN classification model, and I am wondering if I adopt a weaker classifier like C4.5, Naive Bayes, SVM...which way will win?

Thanks!!

The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.

Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. This means that is A and B are different in some way, but this difference is irrespective of the classification with "others" then there is no need to learn that distinction.

For example: if you want to detect dog, cat, human with features such as weight, height and number of legs. The number of legs feature will have relatively low importance, because cats and dogs will likely all have 4 legs. However if I want to classify cat/dog vs humans, then the number of legs will be the most important feature. It might be the only feature you need.

One caveat may be severe class imbalance. By combining your classes in this way you may end up with an over representation of ABCD. You can use techniques such as anomaly detection to train a model on your ABCD data and then detect whether a novel instance falls within this distribution, or is an outlier in which case you would label it as "others."


Can somebody explain me this method? CNN Keras - starter def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]

    for File in FolderList:
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
            LabelList.append(classes.index(os.path.splitext(File)[0]))
            print(FolderList)
    return ImageCV, LabelList

I got this method, and I'm not understanding properly what it do line by line, can someone help me?

def ReadImages(Path):  #path to dir with all images
    LabelList = list() #Initialized empty labels list
    ImageCV = list()   #Initialized empty ImageCV list initialized
    classes = ["nonPdr", "pdr"]  #classes labels all image files startwith
    #return list of image folders in main dir path/
    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]

    #loop over dir of image folders
    for File in FolderList:
        #for loop returns index of each image in a image folder
        #and the image name
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            #Read image file resize it and append it to ImageCv list
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
            #return string of characters before dot 1 in file name
            #return index from classes !!!dangerous line assumes all files
            #of form nonPdr.* or pdr.*
            #append that index value either 1 or 0 to libelist
            LabelList.append(classes.index(os.path.splitext(File)[0]))
            print(FolderList) # print the folder just parsed
    return ImageCV, LabelList # return the two list created at the start
    #now containing images and labels based on names of image files


Using random forest to learn Imbalanced Data (rare disease) I met a question when I ran the random forest. I used "V1", "V2", "V3" to predict a binary outcome (1: sick; 0: no) with random forest.

I got a very high accuracy score (99%) however, when I check the confusion matrix, it shows that none of sick individuals were caught in testing data set (30% of entire data set). Here is the confusion matrix:

[[856   0]

[  9   0]]

This result means that 0 out of 9 people was detected as sick and it causes my attention. Maybe because the data set is imbalanced (very few sick individuals)?

I would like to see if there is any other ways to detect sick individuals rather than a high accuracy rate, which means it is OK it has higher false positive rate but I would like to catch all 9 (true positive) individuals.

Thanks!

Use class weights, to weight errors, so that "incorrectly labelling a sick person as healthy" is penalized more than "incorrectly labelling a healthy person as sick".  Or look up any of the other standard techniques for dealing with class imbalance.


Evaluate clustering by using decision tree unsupervised learning I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion and obviously if someone is aware of this method it would be great if he/she could explain to me the whole idea.
Clusters have been made to the data set (sample of 250000 rows and 5 features out of 500000 rows) by using k-prototypes as one of the features is categorical.
All the combinations of k= 2:10 and lambda = c(0.3,0.5,0.6,1,2,4,6.693558,10) have been made and 3 methods to figure out the best combination have been use.

Elbow method (pick the number of clusters and lambda with the min WSS)

Silhouette method pick the number of clusters and lambda with the max silhouette)

Decision tree

They build a decision tree for the data and after that they calculated for every different clustering combination the following value:
(inverse leaf size weighted within cluster purity)* cluster size/ total obs
and the picked the combination which had the max value. (k=10 and lambda=4)
So my question is:
Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?
Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is?
And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?
I would really appreciate if someone can help me with that.
Thanks in advance!

That's a good question.

Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?

Clustering and simple decision tree fitting are used together in many cases such as:

First, like you mentioned quality of clustering can be measured by using decision tree leafs. I heard this calculation first time (I know some other measures very similar to it) but it makes sence since it still measures how are clusters are distinct from each other and dense.
Second (and most used one), fitting a decision tree by assigning cluster labels as class labels. The fit in here should be overfit (training error should be close nearly 0). This let you when you have a new customer (let's say segmentation in e-commerce) you don't have to calculate all distances and find clusters, you just predict the new customer with the tree and assign cluster = segment label to her/him.

Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is?

No, you cannot. The tree fit well (nearly overfit). Since you want to obtain cluster separation rules in your data rather than obtain a model.

And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?

In that stage you should think why are we doing this segmentation? With all possible cluster models try to simulate your business approach and compare results.

Hope it helps!


If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally).

They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different.
Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST.
Here are those resources,
SBERT paper: https://arxiv.org/abs/1908.10084
SBERT implementation: https://github.com/UKPLab/sentence-transformers