How to quantify the numerical influence of categorical variables? I have trained a simple neural net, to make predictions based on three inputs. For examples sake lets say I m trying to work out "how many days does it take to complete an accademic study", where the data could be:

f1 = institution
f2 = student age
f3 = study topic

and the label is the total number of days to complete.

If you imagine the following data structure:

f1, f2, f3, label

A,  D,  G,   v
B,  E,  H,   w
C,  F,  I,   x
A,  E,  H,   y
A,  D,  G,   z

Is there anyway that I can work out how long each feature adds to the total number of days to complete, so for example I want to be able to say that if you choose f1 = A that 4 days will be added to the total days to complete the study.

I can work out how much difference there is between each choice, eg choosing f1 = A might be 7 days quicker than if you chose f1 = B, but I do not know how to roughly ( maybe average ) identify how long each choice represents.

Yes you're talking about training a model, then doing sensitivity analysis to determine the numerical influence of each explanatory variable in isolation:

Some considerations/tips:

you said the three features are categorical. Presumably student_age is ordered categorical, and the other two are unordered. But you didn't say how many levels they each have (cardinality), or how many bins. You don't necessarily want to (say) bin student_age in bins of 1 year each. You might bin in say 2/3-year increments, or quantiles (see the R cut command), e.g. student_age: [0-17, 18-20, 21-23, 24-28, 29-34, 35+]. Similarly, if you have a long tail of smaller institutions with few exemplars, you might bin them together, or bin by country/ region/ size/ focus/ ranking.
what is the product of cardinality of each variable, vs the size of the dataset? Do you have 1 exemplar for each possible combination? >10? <0.1? If you don't have that much data, you might prefer to use Linear Regression rather than NN, or at least a shallower NN rather than deep. With Linear Regression you can allow quadratic interaction terms, nonlinear terms etc - again
(with Linear Regression you can directly see the influence just by directly reading the model coefficients e.g. Princeton: -1.4, Princeton.Age22_24: -3.6)
if you specifically want to use NN, see the lime package. Also see the many articles/tutorials on sensitivity analysis with NNs, and CrossValidated answers on sensitivity analysis.
deep NNs suffer from a lack of explainability; you can view the many individual layer weights but it's near-impossible to picture how that translates into effect on output.


Topic Segmentation - should it be done in Raw, TfIdf or Semantic Space? Let's assume we have a collection of documents and wish to perform some unsupervised topic segmentation.

As always, we will perform some preprocessing (including tokenization, accent-removal, lowercasing, lemmatizing and such) and transform the lists of tokens into either raw-counts or tfidf-vectors. We'll call this matrix M.

Now we have several possible approaches to perform some simple bag-of-words topic segmentation:

Apply a matrix decomposition method (LSI, LDA, NMF) directly to M and use the resulting components as the topics.
Embed each vector of M into a semantic space (LSI, word2vec) and then apply a matrix decomposition method on the semantic space.
Apply a clustering method (kM, DBSCAN, MSC, GMM) directly to M.
Embed each vector of M into a semantic space and then apply a clustering method on the semantic space.

I have two questions:

Are there any other alternatives to bag-of-words topic segmentation that I have not considered yet?
What are the conceptual differences between the methods described above and which one(s) are recommended?

Thanks in advance!

Clustering method can applied directly to TfIdf matrix (which will be generally sparse) or to documents in the derived semantic space like LSI.  Since LSI embedding is a dimensionality/noise reduction step, it is a good idea to cluster on documents embedded in semantic space. Clustering methods will generally use Euclidian distance and will probably perform better on dense matrix. If the clustering algorithm implementation that you use doesn't support sparse data out of the box (most implementations don't) then definitely use a transformation like LSI first and then apply clustering.
LSI method itself is a matrix decomposition operation on the original TfIdf matrix. So applying matrix decomposition on top of LSI is not needed.


Multi-dimensional regression with ensemble models? I'm curious to know whether boosting, random forests or other types of ensemble models can perform multi-dimensional regression.
To be precise: That means multiple outputs (multi-dimensional labels) that are not intended for classification. For example predicting a persons height and weight from some data where there are two outputs each corresponding to one of height/weight (the labels and output values can be normalized if necessary). You can easily do this with neural networks of course. I am not interested in the trivial case where you simply have multiple models, one per regression output.
If yes, are there any libraries available that support this (the ones I've looked at do not)?

I think there are some algorithms in scikit-learn that support this kind of multi-output with correlated inputs/outputs (e.g. not using one learner per output, this can be achieved using the MultiOutputRegressor meta-estimator): RandomForest, DecisionTree and LinearRegression.
See : https://stats.stackexchange.com/questions/153853/regression-with-scikit-learn-with-multiple-outputs-svr-or-gbm-possible

and

http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py
for a comparison between the two approaches.


What is the difference between "Python interactive" and "Python 3" kernels When creating a new notebook using the web interface, Jupyter gives me two options: 'Python Interactive' and 'Python 3' (see screenshot).

However, I've been unable to find any indication on what the differences between these two kernels (which I assume they represent) are. Both are python 3.6.7 on my system. The search term jupyter combined with "python interactive" or "python 3", even with quotes, yields only generic descriptions of Jupyter, which is not really surprising. Even the Jupyter kernels list doesn't really help.

So, what is the difference?

Interactive mode gives immediate feedback, i.e. every single line will be executed immediately, without possibility to script it i.e. following warnings:

  >>> if interactive:
  ...   print("do it")
  ... print("cant you?")
    File "<stdin>", line 3
      print("Done")
          ^
  SyntaxError: invalid syntax

so the fact that they are together confuses him, but if you seperate it

  >>> if interactive:
  ...   print("do it")
  do it

  print("cant you?")
  cant you?

So the main point is to seperate, and be carefull with new command lines.

Python 3 is standard scripting python version.

Why differentiate? Interactive to test quick and dirty, Python3 to write scripts


Machine learning for missing data in time series We have two time series columns - column A is the reference column ( source of truth) and column B is a ''cousin'' of column A, in the sense that it exhibits ( or should exhibit) the same patterns, evolution, rates of changes etc, as column A.

However, at certain periods column B values for wtv reason start exhibiting abnormal values which we are comfortable to classify as outliers. We are happy to remove these value as outliers.

Now given the unequal timeseries ( post outlier removal of column B) would like to explore machine learning techniques that would impute data in column B, using column A as the reference or source of truth, effectivly replicating the behavior of column A to impute missing data in column B.

I've researched many models to use, regression, LTSM etc. hoping for some expertise that would help shed light on what technique would best work for the problem statement at hand.

If column B truly exhibits the same patterns, evolution, and rate-of-change as column A, then that means the values of column B are just a translation of A.  That is, $B_i = A_i + c$, where $c$ is some constant.

If this is the case, then you don't need machine learning to impute values.  Knowing $A$ is just as good as knowing $B$.  You just need to estimate $c$.  One good way to estimate $c$ would be to compute the mean difference between $B$ and $A$.

On the other hand, if B is related to A, but does not exhibit the exact same patterns and rate-of-change, then machine learning might be a little more appropriate.  You'll want to train a regression model that predicts B as a function of A, then use the model to fill in missing values of B.

Given the degree of coupling you describe between A and B, linear regression is probably a good place to start.  You can experiment with higher-order polynomials (quadratic, cubic, etc) if linear regression doesn't get the job done.  An easy way to get started would be to experiment with polynomial regression in Excel or Python.


Theoretical work on validity of restricting movement of Centroid of K-Mean I recently received a manuscript for review in which author used ~1000 "fake" data points, so that the final centroid of K-mean stays within the required range. Neither me nor the author seems to have background in data science and the paper is more of application into our research area.

I have tried to find published work related to such method of restricting k-mean centers, but failed to do so. However, on simple logic, it seems like valid way, so maybe author used wrong terminology.

Hence, I would like to ask, is this a valid way to restrict k-mean center and are there any published work on it?

I highly recommend finding a source explains how k-means work and understand it well. The K-means is well known, so it is hard to find a reference talk about it as an algorithm or explain how it work.

I noticed you stating "author used ~1000 "fake" data points, so that the final centroid of K-mean stays within the required range" which is always going to be true. K-means is about calculating the mean (average) of data points used, which assure (always) to end with a centroid/s within the range of data used.

The power of this algorithm (K-means) is calculating the mean iteratively to reach stability of means (centroids). In another waord, in each iterate, means shift to be centered of denses. That give, if you in case of finding 1 K (one centroid) you will find it by one iterate.

Me personally suggest start with some videos, and go forward. Here is the first result on YouTube about k-means https://youtu.be/_aWzGGNrcic.


Is this a tried alternative to word embedding for NLP? I'm searching for research related to my idea, but apparently cannot articulate it well enough to the search engines to show me what's been published on this.
My idea: in a deep learning context (text classification), instead of inputting the text as integer vectors representing words in a vocabulary, which get substituted with dense vectors (word embeddings), what if instead, you represent each phrase almost like how a vector of "pixels" represent an image (then use 2d convolutions)? For example, each character, digit, and symbol get an integer value, just like how pixels are represented. So abc becomes 0, 1, 2. The phrase could be formatted as a rectangle at word boundaries or something, but the gist of it, a phrase is formulated as a 2d array where each element is a character from some character set.
Is this a known/tried approach, and what name does it go by? From its apparent lack of popularity, I assume this formulation is far worse than word embeddings, but I'm curious about it.

It has never been tried as far as I know. And the reason is that, IMHO, it would give you inconsistent results. I am talking in particular about unwanted arithmetic properties of such vector.
Do you want to represent an a, b, c vector of letters {1,2,3} knowing that
a < b < c  ?
This makes sens with ints, not with char/tokens.
The good news is you can still use 2d conv, once you assembled embedding vectors into larger phrase vectors.


Confusion matrix logic Can someone explain me the logic behind the confusion matrix?

True Positive (TP): prediction is POSITIVE, actual outcome is POSITIVE, result is 'True Positive' - No questions.
False Negative (FN): prediction is NEGATIVE, actual outcome is POSITIVE, result is 'False Negative' - Why is that? Shouldn't it be 'False Positive'?
False Positive (FP): prediction is POSITIVE, actual outcome is NEGATIVE, result is 'False Positive' - Why is that? Shouldn't it be 'True Negative'?
True Negative (TN): prediction is NEGATIVE, actual outcome is NEGATIVE, result is 'True Negative' - Why is that? Shouldn't it be 'False Negative'?

A confusion matrix is a table that is often used to describe the performance of a classification model. The figure you have provided presents a binary case, but it is also used with more than 2 classes (there are just more rows/columns).

The rows refer to the actual Ground-Truth label/class of the input and the columns refer to the prediction provided by the model.

The name of the different cases are taken from the predictor's point of view.

True/False means that the prediction is the same as the ground truth and Negative/Positive refers to what was the prediction.

The 4 different cases in the confusion matrix:

True Positive (TP): The model's prediction is "Positive" and it is the same as the actual ground-truth class, which is "Positive", so this is a True Positive case.

False Negative (FN): The model's prediction is "Negative" and it is wrong because the actual ground-truth class is "Positive", so this is a False Negative case.

False Positive (FP): The model's prediction is "Positive" and it is wrong because the actual ground-truth class is "Negative", so this is a False Positive case.

True Negative (TN): The model's prediction is "Negative" and it is the same as the actual ground-truth class, which is "Negative", so this is a True Negative case.


How to select 500 most pertinents tags among 10000? Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags.

How to select 500 tags in order to cover the largest set of documents ?

Firstly, I chose the 500 most frequent tags. If we keep only these 500 tags, 70% of documents keep at least 1 tag.

I'm looking for a better method to pick the 500 most pertinents tags.

example :

my best set of 5 tags to cover all documents is [italian, german, french, chineese, japan].

it is not [italian, spanish, indian, finnish, english] although these are the most frequent tags.

I tried a Single Value Decomposition on the matrix : [documentID, set of tags]. but after that ? is it a good idea ? how to get 500 tags from the SVD results ?

That is a set cover problem. The documents (universe) are being covered by a set of tags (elements).

In the case of maximizing the number of documents with a limited number of tags, a greedy approach can be taken. Find the single tag that covers the most documents. Then find the next tag that covers the most documents not already covered. Continue the process until the number of tags has been reached.


When to build separate models I'm pretty new to predictive modeling, but am interested in generating predictions for credit card account spend. These are existing accounts.

The data I have available to me is Card Type (i.e. Platinum, Black, Gold) and spend/transaction data over the last year.

I have a two questions:

How do I decide whether to use linear regression or a decision tree/random forest? Or do I try many techniques?
I expect spend will differ considerably across Card Types; does this suggest I should build separate models by Card Types? If I had a another set of attributes that I thought might segment customers, but wasn't sure, how would I evaluate that?

If there are guides online that answer the above, that'd be much appreciated as well. Haven't found any good ones (or at least I think not)

short answer: assess simple models first, and then build more complex ones if necessary.

For a roadmap, I'd go like this:

get familiar with the different categories of machine learning problems: classification vs regression problems. Your problem falls in the regression category (at least at first sight, you could also reframe it into a classification problem if necessary)
get familiar with the main corresponding algorithms
get familiar with the strategies to train / test a model performance
since you are explicitly asking for online resources, you can have a look at Andrew Ng's machine learning course (especially the first weeks of the course). It's not free, but you can also have a general overview buy looking for his Machine Learning Yearning Book. Another resource coming to my mind is Machine Learning Mastery website (can't remember the guy's name) where you can find some introductory ebooks to answer those questions. It's well written and can provide you a good lift

Does this help?


How to define quadratic weighted kappa as eval_metric in catboost classifier I am using catboost for a multiclass classification problem. I want to use quadratic weighted kappa as the evaluation metric.

Catboost already has WKappa as an eval_metric but it is linearly weighted variant, not quadratic.

clf = CatBoostClassifier(objective='MultiClass',
eval_metric=q_weighted_kappa(),
learning_rate=0.01,
n_estimators=750,
od_type="Iter",
depth=35,
early_stopping_rounds=500,
random_seed=42,

)

I need help to define the function q_weighted_kappa() that I can use as specified above.

Sklearn kappa has 2 variants: simple and weigthed. It seems WKappa refers to weighted (quadratic) kappa in terms of sklearn, and Kappa - to simple.


Mathematics: Writing down a three-class classifier confusion matrix Confusion matrix 2A three-class classifier is evaluated on a test set
of 900 samples which containsall three classes in equal
proportions.
• Classes 2 and 3 are always classified correctly
• Class 1 is confused with class 2 in 50% of the cases,
  and with class 3 in 20% of the cases.

Write down the confusion matrix!

Like the name "confusion matrix" already says, I'm confused.
I'm using a binary classification confusion matrix which looks like that

I show you this matrix, because there are a lot of different looking classification matrices with a different order of the true positives, true negatives and so on.
According to this task, I must use a three-class classifier matrix. So first I divided the 900 Samples to 3, so I get 300 for each class. Then I substracted 50% and 20% of 300 to get 90 samples for the first class.
I read, the row index is for the true class and the column index is, what the classifier says.
My confusion Matrix looks like this:

I think, the confusions of class 1 are false positives.
Did I write the matrix correctly?

No, you did it incorrectly, you need to switch rows with columns (transpose). Next time, please check if a similar question has not been asked before. For the explanation, see https://stats.stackexchange.com/questions/318780/how-to-understand-confusion-matrix-for-3x3 .


Why/When should I use VGG16 to do fine-tuning? Why or When should I use VGG16 in my cnn? what is the pros and cons to use this model?

I search but not found this answer. If you have references, I appreciate

I think you mixed up the terms that you do here. The task is called transfer learning and you FINETUNE the model.

The reasoning is that after experimentation we found that earlylayers of CNN captures the general features of images such as edges,lines, etc, and later layers capture more specific features e.g. faces, shape of object. We only wish to alter the few layers or even only the output classifying layer.
Pretrained model is trained with datasets that are most likely bigger than your dataset. Also because of reason number 1 and we have trained it with bigger dataset, we have first few layers that is able to generalize well and hence helps you combat overfitting.
You save time and resource by doing this. First because you are only backpropagating within last few layers.
Architecture-wise, people spend time researching how every features interact with image and hence it is recommended to use this architecture and published/publicly known result implies that it gives good result.

So in general it is recommended practice to do transfer learning for image classification unless you have a very strong reason not to do so. There are many resources that tells you how to do it.

Imo, VGG16 is outdated, it is also heavy and inefficient. You might want to check efficientnet.

Medium source regarding CNN filter. Medium source


How can I explain this chart showing 5-days moving average? I have plotted the frequency of items sold through time, trying to determine the trends by moving average.
I considered a 5-days window. I would like to know if this approach makes sense and how I could interpret the results.
It's my first time with time-series and moving average (I have no a scientific background at all).
I hope you can help me.

Yes it makes sense, a moving average makes the curve "smoother" in the sense that it's less sensitive to short variations. This usually makes it easier to observe the general tendency.
You could also try different time periods for the average, e.g. 10 days or 15 days.
It looks to me like there's a moderate increase trend in your data, but the variations are important and the time window is short, so it's too early to be sure. You could apply linear regression to confirm the increase.


Problem with basic understanding of polynomial regression I have an understanding of simple linear regression. Clear that  results in a fitted line like this:

However, studying polynomial regression is a bit of a challenge having some questions about the process. I understand the idea that fitting a curviliear "line" could follow the data more precisely. But seeing the following Python code in scikit-learn:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = 2 - 3 * np.random.normal(0, 1, 20)
y = x - 2 * (x ** 2) + 0.5 * (x ** 3) + np.random.normal(-3, 3, 20)

x = x[:, np.newaxis]
y = y[:, np.newaxis]

polynomial_features= PolynomialFeatures(degree=2)
x_poly = polynomial_features.fit_transform(x)

model = LinearRegression()
model.fit(x_poly, y)
y_poly_pred = model.predict(x_poly)

suggests for me that here we "just" projecting the original features into 2nd degree new polynomial features, while still fitting the simple (straight) line on it. Still, we get the following solution with magically curved line:

Here arise my questions:

How could the straight line in figure #1 become a curved one in
figure #2 when we just introduce some new features but still fit the
original LinearRegression model? A cannot see why couldn't the same
estimator found curve in case 1 when it is able to find it in case
2? There is no hint for the estimator in the syntax in case 2 that "ok, let's apply a curve now instead of a line", right?
I read that polynomial regression is still linear, what is the exact
interpretation of this statement? "Linear" means no straight line but
any curve, is this what this want to say? If so, what is "nonlinear"?
Additionally, having read about penalty terms for polynomial regressions, I read that the
introduction of higher order features has the effect that
coefficients tend to grow along the magnitude. This is illustrated
here:

I cannot figure out here why does coefficients for a specific same feature (like x_1) increase just because there are more additional polynomials (e. g. x_2, x_3 etc) while stay low just being alone? Quite confusing.

"Linear regression" (aka. "ordinary least squares", OLS) refers to the type of estimator. Linear here means that you minimise the sum of squared residuals for a given (linear additive) regression equation. You can write a simple model:

$$y = \beta_0 + \beta_1 x_1 + u. $$

This would fit a linear function with intercept $\beta_0$ and slope $\beta_1$. So the "fitted" function would be:

$$ f(x) = \beta_0 + \beta_1 x. $$

You can "propose" basically any functional form to be fitted by the linear model, e.g. with a squared term (just add $x^2$ as feature):

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x^2_1 + u. $$

This will give you the (estimated) quadratic function:

$$ f(x) = \beta_0 + \beta_1 x + \beta_2 x^2. $$

In essence, you can add any kind of linear transformation of $x$, e.g. $log(x)$, $exp(x)$ etc.

"Non-linear" models (there is no really clear definition), would be e.g. logistic regresion, non-parametric regression, tree-based models, neural nets etc. While each coefficient $\beta$ directly gives you a "marginal effect" in linear (OLS) regression, this is not the case for the other (non-linear) models.

When you use regularisation in linear models, you simply add a penalty term to the loss function, so you minimise the sum of squared residuals given a penalty term. However, regarding your last question, I'm not quite sure what your actual question is.


Classification: How to manage data sets where one data row depends on another data row I am trying to classify  heading, image and image caption  of a webpage. I am preparing data by scraping selected URLs(around 1000) using XPath of DOM elements I need. Each data row in the CSV file contains tag name, x-coordinate, y-coordinate, text-size etc with three target labels heading, image, image caption but a text simply will not become image caption until unless it is under an image. Is there any way to create this dependency among rows.

I assume that you are solving a supervised classification problem, that is, you train your model on a labeled sample. I can think of two approaches to this problem.

I. Classify tags, use neighbor tags for the features

For each tag you can calculate features like:

(x, y) distances to the closest image
(x, y) distances to the closest image above this tag
number of images near this tag
...

Such features could be fed into any classifier like SVM or decision tree.

II Classify pairs of tags

For each tag $i$, you can consider all other tags $j$ and predict probability that $i$ is a caption of image $j$. This prediction may be based on concatenation of:

features of $i$
features of $j$
joint features of $i$ and $j$, like vertical and horizontal distance from $i$ to $j$.

After these probabilities are predicted, you can classify $i$ as a caption indeed, if any of these probabilities exceeds some threshold.


List value in Pandas DataFrame column makes analysis harder Should I move to database?
I have a list of courses data in JSON format that looks like this:
courses = [
  {
    course_id: "c_01",
    teachers: ["t_01", "t_02"]
  },
  {
    course_id: "c_02",
    teachers: ["t_02", "t_03"]
  }
]

And a list of teachers that look like this:
teachers = [
  {
    teacher_id: "t_01",
    teacher_fullname: "teacher_01"
  },
  {
    teacher_id: "t_02",
    teacher_fullname: "teacher_02"
  },
  {
    teacher_id: "t_03",
    teacher_fullname: "teacher_03"
  }
]

I also have other data like courseworks and submissions that I want to cross with these to create summary analytics like:

¿Which are the courses of a teacher?
¿Who is the teacher with most courses?
¿What is the average coursework quantity per course per teacher?
etc...

I loaded each list (courses, teachers, courseworks, submissions) into DataFrames, but I'm having a hard time selecting the courses of a teacher using vectorized methods.
Programming attempts

Tried using DataFrame.query but failed to use array methods inside the query
Tried using Series.isin but as it can't hash the list inside the Series teachers is useless.

Data manipulations attempts
Then I tried to flatten the teachers Series as follows:
courses = [
  {
    course_id: "c_01",
    teacher_id: "t_01"
  },

  {
    course_id: "c_01",
    teacher_id: "t_02"
  },
  ,
  {
    course_id: "c_02",
    teacher_id: "t_02"
  }

  {
    course_id: "c_02",
    teacher_id: "t_03"
  }
]

Which works because it allows merges and all kinds of queries, but because of combinatorial explosions I ended up with thousands extra rows I no certainty that the aggregated numbers are correct.
My last approach was to add one column per teacher in each course like this:
courses = [
  {
    course_id: "c_01",
    teacher_01: 1,
    teacher_02: 1,
    teacher_03: 0,
  },
  {
    course_id: "c_02",
    teacher_01: 0,
    teacher_02: 1,
    teacher_03: 1,
  }
]

It actually works perfectly and the aggregations become very easy to do. One concern is that I may end up with thousands of columns (I'm not sure if that's a problem) and the other is that each time a run an analysis on a batch of data I'll end up with different columns so it will be harder to make cross analysis between different datasets.
Anyway, my final thoughts are that I should store everything in a database and query the information that I need already "joined" to perform an easier and cleanier analysis process.
so...Should I move to database? Am I missing something?

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.
Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. If you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.


How to measure the correlation between categorical variables and a continuous variable I have the following list of the names of the categorical variables in my dataset:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

in this function definition, I am one-hot encoding each column with the help of a for loop:

def feature_encoding(df, categorical_list):

    # One Hot Encoding the columns gathered in categorical_columns
    for col in categorical_list:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[categorical_list])

        # drop the old categorical column from original df
        df.drop(col, axis = 1, inplace = True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

I don't want to use all of these columns in my training so now I am at the dimensionality reduction phase of my work.  I want to measure the correlation between each of these columns and my SalePrice variable (numerical) and weed out the columns with low correlation.

I've read that Chi-square test is generally used for measuring the correlation of categorical variables but I have not seen an implementation where it was a list of categorical variables vs a continuous variable.

The Chi-squared test measures the relationship between two categorical variables.

To measure the relationship between a categorical feature and a continuous feature, you can use an ANOVA test.

As an aside, you don't need to use a for loop for your encoding - get_dummies has the parameter columns which allows the user to specify which columns are encoded.


What are regret bounds? I searched for the term and it appears in a few articles but it is used without explanation. The only explanation I could find is in a PhD thesis: "Regret bounds are the common thread in the analysis of online learning algorithms. A regret bound measures the performance of an online algorithm relative to the performance of a competing prediction mechanism, called a competing hypothesis."

I am still confused after reading this (I did not read the rest of the thesis as it is way above my understanding in that field). Could someone please explain? Many thanks in advance!

Reinforcement Learning models are often measured relative to each other and sometimes relative to optimal behavior.

Regret is a common measure of performance for Reinforcement Learning systems.

Reinforcement Learning model performance is stochastic. Thus, is better to run the same model many times to estimate the distribution for performance. From that distribution, bounds can be estimated. Those bounds are similar to confidence intervals for scalar parameter estimates.


Do I need to learn Hadoop to be a Data Scientist? An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn Hadoop to be a Data Scientist?

Different people use different tools for different things.  Terms like Data Science are generic for a reason.  A data scientist could spend an entire career without having to learn a particular tool like hadoop.  Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data.

I would say that a data scientist should be familiar with concepts like MapReduce, distributed systems, distributed file systems, and the like, but I wouldn't judge someone for not knowing about such things.

It's a big field.  There is a sea of knowledge and most people are capable of learning and being an expert in a single drop.  The key to being a scientist is having the desire to learn and the motivation to know that which you don't already know.

As an example:  I could hand the right person a hundred structured CSV files containing information about classroom performance in one particular class over a decade.  A data scientist would be able to spend a year gleaning insights from the data without ever needing to spread computation across multiple machines.  You could apply machine learning algorithms, analyze it using visualizations, combine it with external data about the region, ethnic makeup, changes to environment over time, political information, weather patterns, etc.  All of that would be "data science" in my opinion.  It might take something like hadoop to test and apply anything you learned to data comprising an entire country of students rather than just a classroom, but that final step doesn't necessarily make someone a data scientist.  And not taking that final step doesn't necessarily disqualify someone from being a data scientist.


Are stationarity and low autocorrelation the prerequisite of regression model? As said in the title, are stationarity and low autocorrelation the prerequisite of general / linear regression model ? That is, if a time series is non-stationary or has large autocorrelation, would it be easier or harder to be predicted using regression model such as linear model and deep learning ?

For (linear) regression models, the following assumptions have to hold:

Linear Relationship
Multivariate normality including your error term with zero mean + some infinite variance following a normal distribution
No-autocorrelation
No (strong) multicollinearity
No homoscedasticity

Non-stationarity (caused by changing mean or/and variance over time) is prevailing in most level data (e.g. stock prices) - as it has a unit root or is trend stationary.

To cure your data from non-stationarity in most of the times, it is sufficient to use the relative change (percentage change - or log changes). Taking logarithmic changes, can also cure of homoscedasticity a bit.

To test for stationarity, you can use the Augmented-Dickey-Fuller-Test where the Null hypothesis states that a unit-root is present in your data set, thus you have a non-stationary variable.

Coming to your second question about time-series and deep learning. Unfortunately, I haven't found a good paper/article yet that discusses the assumptions on the main data properties that have to be satisfied to get correct statistical results. See my own post - not answered yet

However, as deep learning models make use of the same underlying statistical assumptions - e.g. probability distributions etc. or even the underlying models (see e.g. linear activation function) - I guess, in a time-series framework, the same assumptions have to hold, otherwise, there is the chance of spurious regression.


How do I determine if a neural network can be computed by a network that contains only an input layer and an output layer? I have a homework problem where the neural network below is given with its description.  We have been asked "Can the functions that the above network computes also be computed by a network that contains only an input layer and an output layer that has a single node?" If we can, we're to provide the network, weights, and activation function. And if not, we have to explain why.

I am very unsure how to approach this problem as I am not that strong with neural networks. Any starting point would be very helpful.

Thanks in advance.

You are asked whether we can mimic the given neural network with a new network that contains only a input layer and an output layer, which in turn means whether $a_5$ can be represented as a linear combination of $x_1, x_2, x_3, x_4$. Since every action the neural net does is linear it can be represented as a smaller one without a hidden layer.

Let's calculate the weights

$$a_5 = (a_1*\theta_9 + a_2*\theta_10 + a_3*\theta_11 + a_4*\theta_12)*C $$
$$a_5 = ((x_1*\theta_1 + x_2*\theta_2)*\theta_9*C + (x_1*\theta_3 + x_2*\theta_4)*\theta_{10}*C + (x_3*\theta_5 + x_4*\theta_6)*\theta_{11}*C+ (x_3*\theta_7 + x_4*\theta_8)*C)*\theta_{12}*C $$
$$a_5 = C^2 * (x_1*(\theta_1*\theta_9+\theta_3*\theta_{10}) + x_2(\theta_2*\theta_{9}+\theta_4*\theta_{10}) + x_3(\theta_5*\theta_{11}+ \theta_7*\theta_{12}) + x_4(\theta_6*\theta_{11}+\theta_8*\theta_{12})) $$

So our weights are:
$$<\theta_1*\theta_9+\theta_3*\theta_{10},\theta_5*\theta_{11}+ \theta_7*\theta_{12} , \theta_2*\theta_{9}+\theta_4*\theta_{10}, \theta_6*\theta_{11}+\theta_8*\theta_{12}> $$
and our activation function is multiplication by $C^2$


On design of the training set: conceptual question I am curious to know how training data should be constructed so that it scales to examples that are not a part of the training data. For example, the problem that I am facing right now is in the application of identifying or distinguishing the frequency response of time series that are generated from different distributions. So I constructed $p$ number of examples each from Gaussian, Uniform, Poisson and a kind or colored noise say pink. The White noise examples (Gaussian, Uniform and Poisson) are labelled as 1 and colored noise as 0. Using Neural Network the classification works fine. Now I wanted to do sensitivity analysis by checking if the trained network can classify white noise from another distribution and also colored noise say red. Both the tests failed. NN failed to classify them. But, as soon as I included the red and the new kind of white noise in the training data and tested on a different trail (time series), the NN could classify it.

QUESTION: This behavior makes me wonder if machine learning algorithm sare incapable of distinguishing examples from different systems eventhough the examples in testing have similar properties as those used in training. In this case, eventhough white noise appears similar but since they are generated from different distribution or say systems the training data must include examples from all the generating mechanism or systems, otherwise in testing the ML model fails to recognize it. Is this the usual behavior?

One of the basic assumptions governing Machine Learning is that samples from the training set must follow the same underlying distribution as samples from the test set (and so must any other sample you want fed to your model)!

This is why, usually, we randomly partition the same dataset into training and test sets.

This is actually one of the main reasons ML models underperform in some real world applications. You might have trained your model in a specific dataset, but overtime the data slightly changed its characteristics and new data differ from the old ones that were used to train the deployed model. In this case you need to retrain your model on the new data.


how can i compare two classification algorithms? all,
i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?
i have used mcnemars test:
from mlxtend.evaluate import mcnemar_table
from mlxtend.evaluate import mcnemar
lgbm_pred = second_best.predict(x_test)
xg_pred = chosen_model.predict(x_test)

tb = mcnemar_table(y_target=y_test,
                   y_model1=lgbm_pred,
                   y_model2=xg_pred)

chi2, p = mcnemar(ary=tb, corrected=True)
print('chi-squared:', chi2)
print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)
i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.
https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers-and-assess-statist
basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach - does anyone also know how i can use permutation tests?

Performance of a model can be judged along several dimensions:

Accuracy - Is the train-test validation performing well?

Overfitting - Is the difference between training score and the validation score minimal?

Efficiency - Is the model light-weight, does it compute and calculate fast?

Complexity - Is the model easy to explain, does it use minimal transformations?

Deployability - Will I be able to deploy and use the model easily?

Rather than from a pure statistics perspective (which is just dimension 1 & 2) judge the models from all perspecties. If they still perform equally well than they are in fact equal (not surprising for two similiar models performed on the same data set).


Understanding one of the assumptions of linear regression: Multicollinearity I've read that multicollinearity is one of the main assumptions of multivariate linear regression - Multicollinearity occurs when the independent variables are too highly correlated with each other.
However, when learning linear regression, one of the key topics is the idea of introducing interaction terms into the model to model the interaction effect which is when the effect of an independent variable on a dependent variable changes, depending on the value(s) of one or more other independent variables.
Aren't these two statements contradictory? If there were really interactions between $X_1$ and $X_2$ in the model $y = \beta_0 + \beta_1*X_1 + \beta_2*X_2$ surely we should remove either $X_1$ or $X_2$ so that the independent variables in the regression model are no longer correlated thus the multicollinearity assumption holds. Putting in the interaction terms seems to ignore this assumption and rather introduce a further term to further complicate it.
From a modeling standpoint this makes sense, but doesn't the mathematics breakdown if we do this?

Interaction effect and Colinearity have two different meanings.
Multi-colinearity simply tells if two or more predictors are correlated i.e. change in one changes the other. I believe there is no confusion about this as you have also mentioned in your question.
But Multi-colinearity doesn't need the response variable for it to be figured out.
Interaction by definition, are always in the context of how predictors relate to the outcome.

More formally, two or more predictors are said to interact if their combined effect is different (less or greater) than what we would expect if we were to add the impact of each of their effects when considered alone.

A simple and intuitive example can be - consider the effects of water and fertilizer on the yield of a field corn crop. With no water but some fertilizer, the crop of field corn will produce no yield since water is a necessary requirement for plant growth. Conversely, with a sufficient amount of water but no fertilizer, a crop of field corn will produce some yield. However, the yield is best optimized with a sufficient amount of water and a sufficient amount of fertilizer. Hence water and fertilizer, when combined in the right amounts, produce a yield that is greater than what either would produce alone

It is represented as -
$y =β_0+β_1x_1+β_2x_2+β_3x_1x_2+error$
$β_3$ explains the interaction between $x_1$ and $x_2$
Interaction can be Additive (no interaction), Synergistic(+ve), Antagonistic(-ve), Atypical depending on different parameters values.
Reference
Quotes, example and the equation is from -
Feature Engineering and Selection: A Practical Approach for Predictive Models, Max Kuhn and Kjell Johnson
Good read for an in-depth explanation of different Feature engineering concepts.
Internet link


Representing 2 dimensional data for facial recognition My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model to recognize if a face belongs to a person in a smaller set of people and if so, to which one.
Hence, each face is represented by a list of (x, y) coordinates denoting each landmark point recognized by the Google library. Variations in the absolute locations of the points and their relative position to each other are cues telling people apart.

Therefore, from a list of points such as this one:

I get a list of coordinates with the person one-hot encoded at the end:

1975.72,120.265564,2108.6755,141.79669,2344.71,214.35474,[...], 000010000

and I want to create a .csv file containing multiple rows for each person.

My problem is, that I do not know how to represent that the points are actually grouped by two as (x, y) coordinate of the very same point in a way that my model can understand it.

Since I'm new to this, I'm not even sure if my model actually needs to understand this in order to recognize someone out of a list of points so sorry, if this is a stupid question. Also, I do not need the problem to be solved, I only request help with the notation system in my training/test/validation data.

Clarification

I already know how to extract the list of points from an image using Google ML-Kit. What I do not know is how to represent points in a way that the net will pick up that x and y belong together constructing a 2d object.

For example, I could have a raw csv file like in the example and all I would need to do is splice the content alongside the commas, but then the net would never know about the relationship. Instead of a list of 1d values, however, I could also expose the net to a list of 2d points such as [[x, y], [x, y], [.....]], but I don't know if data with n-dimensional elements needs any special attention or the net can understand it as if it were a regular list with consecutive elements.

You are describing is a Cartesian coordinate system where each point is an ordered pair of signed numbers.

In order to compare across people, you need to define the same origin (0,0) and define all points relative to that. Typically, the origin of a face is the tip of the nose.

The input to a model would be a collection of these Cartesian coordinate pairs for each training instance.

Google's Firebase ML-Kit has a very specific API for Face Detection  with specific facial landmarks and each point is a VisionPoint. Given the predefined, rigid, and hierarchical structure, it would be more useful to store the data as JSON to preserve the relationship between label and coordinate pair.


How to detect if a person is interacting (as in touching) some object in an image? I am currently working on a human interaction problem, which tries to identify if a person touched a predefined object.

Current Approach: I am using open-pose to estimate the pose of the person then manipulating the keypoints returned into some logic (simple if else statements) which tells whether person interacted or not. However it seems to be a dirty workaround to me and is not a full proof solution. So is there any better approach I can go for?

Is there any predefined model or white paper I can refer for this problem?

I am struggling on how to detect the pose of a person in an image? Is there a way to classify the pose of a person in three categories like front, back and side?

There are papers that show how to estimate different poses. For example this one gives an overview of different methods: PoseTrack: A Benchmark for Human Pose Estimation and Tracking. You can take this benchmark and group all the poses into needed categories. For example, by manually assigning your groups. And then you use the categorization of poses like in paper and convert to the poses you need.

One of the solutions for pose tracking is Mask R-CNN. It uses a neural network with convolutional layers and predicts in parallel class, box and binary mask. Here is the original Mask R-CNN paper. And here is the ProTracker algorithm that achieved one of the best results on PoseTrack.


Gradient Descend Method Hello!

I started learning the Gradient Descend Method in order to solve some regression problems, I can say I know what the algorithm does overall but I can't understand why in every cost function, the x^i is not taken as an argument!

Lets suppose we have n sized input data (x1,x2,...,xn) , each input data has m attributes. In the cost fucntion we basically have an xi from (x1,x2,...,xn), but which one? We just take one at random? And that remains starndard throughout the algorithm? I did attach the image to provide an example of what I'm saying

The equations are not showing quite what you think. For the equations you have copied, when you see $\mathbf{x}^{(i)}$, you should not think of it as

  The ith feature of an example record $\mathbf{x}$ - [INCORRECT]

Instead it is

  The ith example record from the training dataset - [CORRECT]

So, $\mathbf{x}^{(i)}$ in this case is actually a vector including all features of a single example, and the equations show vector arithmetic. The parameter $\theta$ is also a vector.

The individual feature indexing is shown with a subscript $j$ i.e. $x^{(i)}_j$ is the jth feature of the ith record. The equations are also using bold $\mathbf{x}$ for a vector quantity and non-bold $x$ for a scalar quantity to try and make it clearer.

Take care, when you have multiple dimensions of data to work through, different document sources will use different conventions and indexing schemes. After a while, when you know what to expect is going on, you will spot the differences quickly and figure out which convention is in use. This is only a minor problem, and can crop up if you switch learning sources - e.g. watch a video lecture by one researcher then read a book by another.

  Lets suppose we have n sized input data (x1,x2,...,xn) , each input data has m attributes.

There are $m$ examples in the dataset. The data is $(\mathbf{x}^{(1)}, \mathbf{x}^{(2)},...,\mathbf{x}^{(m)})$

There are not $m$ components of each vector. It is not stated how many components there are in the equations.

  In the cost fucntion we basically have an xi from (x1,x2,...,xn), but which one? We just take one at random? And that remains standard throughout the algorithm?

Both equations sum over the whole dataset to calculate cost function or its gradient.


Softmax classifier never allows for 100% probability in LSTM? When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this:

$$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$

notice, LSTM's result is a $h=tanh(c) \circ \sigma(p)$

Where c is the cell state passed through a tanh as well, and $\circ$ is the component-wise product of two vectors.

Recall that tanh never goes beyond -1 and never goes above 1; The $\sigma$ never goes below 0 and never above 1

Does this mean that if we have a 4-neurons on the output - the best guess a network can make will be 1, -1, -1, -1?

When softmaxed, this will produce $\frac{2.72}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}$ which are these probabilities:

0.7,

0.1,

0.1,

0.1

and we can never get ~100% certainty, (Edit: or in this example even above 70%) no matter the learning rate?
Is there a way to combat it without destroying its derivative working nicely with Cross-entropy?

Edit

As a solution, I've tried to determine how close to $e^{−1}$ and $e^1$ the $e^x$ actually lies. Basically an 'inverse lerp', and it indeed puts values to 0 and 1 range. However, I am not sure if it affects derivatives & if it no longer will work out of the box when using Cross Entropy.

Here is the usual way to compute softmaxed-vector:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
   softmaxedVec[i] =  exp( tanhVec[i]);
   totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
   softmaxedVec[i] = softmaxedVec[i] / totalSum;
}

and here is what I am now trying to do:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
   softmaxedVec[i] =  inverse_lerp(0.3678,  2.71828,  exp( tanhVec[i]));
   totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
   softmaxedVec[i] = softmaxedVec[i] / totalSum;
}

//returns percentage: 0 the value sits on min, 1.0 the value sits on max
float inverse_lerp(float min, float max, float currValue){
    return (currValue - min) / (max - min);
}

However, with this I am afraid that I might have lost that gorgeous $\frac{\partial E}{\partial W} = expected - target $  (curtesy of softmax & cross entropy working together)

..or if I basically ruined the benefit of softmax

If want you mean by 100% certainty is:

1,

0,

0,

0,

as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because:

$$ e^{h_i} \neq 0 $$ $$ \forall h_i \in \mathbb{R} $$

See this link for better explanation. Note that $ e^x $ can be close to 0 but it will never be 0.
The kind of output that you want is the one that the max function would output, sadly the max function is not differentiable so a soft version is needed hence softmax.

I think that the problem is that you are using softmax after tanh, and this so you must eliminate one of the "squashing" functions, see this discussion

The reason to use softmax is if the classes are mutually exclusives, also one of the advantages of using softmax is that it reduces sensitivity to outliers by giving at least some probability to the other values. See this links for a more detailed explanation 1 and 2.


Does neuron have weight? My understanding is that a connection between two neurons has a weight, but a neuron itself does not have a weight.

If connection c connects neurons A to B, then c has a weight w, but A and B don't have a weight. w determines if A has a strong influence on B or a weak influence on B.

But this Wikipedia article says:
"The connections between artificial neurons are called ‘edges’.
Artificial neurons and edges typically have a weight that adjusts as learning proceeds."

https://en.wikipedia.org/wiki/Artificial_neural_network

The 2nd sentence seems to imply that a neuron has a weight, too.

Does a neuron really have a weight?

If you look at the 2nd equation under "Propagation function"

$p_{j}(t)=\sum _{i=1}^No_{i}(t)w_{ij}+w_{0j}$

The $w_{0j}$ is the constant (bias), so it's reasonable to write that the $w_{ij}$ weights are associated with the connections, and the $w_{0j}$ is a weight associated with the nueron itself.

Additionally if you assume $o_{0}(t)=1$ then you can rewrite the equation above as

$p_{j}(t)=\sum _{i=0}^No_{i}(t)w_{ij}$

(i.e. note the sum is from 0 to N, not 1 to N)

When you do this it no longer really makes sense to make a distinction between a "weight" and a "bias".


When I try to predict with my model I get an Attribute error After I've created my model using keras sequential, I tried to start predicting on a small sample to see if it would work however I get this error and I have no idea why.

  error : AttributeError Traceback (most recent call last)
   in () ----> 1 example_results
  = model.predict(example_batch) AttributeError: 'NoneType' object has no attribute 'predict'

The code for my model can be found here :

https://colab.research.google.com/drive/1mxRIx5ufVsA0ljdTpL0Ud0qm2y39eyGX

I've done it using this method before but it refuses to work this time around.

In cell 10 of your notebook you write:

model = build_model() #Perfect

In cell 11 you write:

model = model.summary() #Uh oh!!

later you call

example_result = model.predict(example_batch) #model.summary() returns None

and python tells you

  AttributeError: 'NoneType' object has no attribute 'predict'

This is because you reassigned model in cell 11 to, well, nothing. You should remove the model = in cell 11 and your code will run perfectly!

redo cell 11 to read:

 model.summary()


Keras: Misunderstanding what Keras does when invoking a Model, providing an Input? In a class named Generator, I have defined a model with the below method define_model:

def define_model(self):
    conv2d = Generator.__last_block()
    output = Activation('tanh')(conv2d)
    model = Model(self.input_layer, output)
    return model

In another file, I am writing the entry-point of my Python program, which calls the generator's define_model method:

generator = Generator()
generator_model = generator.define_model()

input_low_resolution = Input(shape=low_resolution_shape)
generated_high_resolution_images = generator_model(input_low_resolution)

As I'm reading a course, I however have a queston about this code: what does Keras do when it executes the line generated_high_resolution_images = generator_model(input_low_resolution)?

As far I can understand, it doesn't define a model (my model is already defined thanks to generator.define_model()!). And since the Keras methods train, fit or other aren't called, I deduce that this line doesn't train the model.

By the way, it's a really weird line for me, because it passes a parameter to a reference (the reference to the object is generator_model and the parameter is input_low_resolution). Normally we pass parameters to the reference's methods (constructor and other methods).

Keras does a lot of stuff when you call a model. I guess that the most important is that it defines the trainable variables and the graph based on the inputs and outputs. For instance actually making any dense layers, and not just the representations.

For instance, if you did mod1=generator_model(input_low_resolution) and mod2=generator_model(input_low_resolution), then if you train mod1 then mod2 would not be affected as they have different parameters. So the .define_model just makes a "handle" to your model, which when called actually "builds" it.

This functionality is very useful when you want to mix-and-match multiple layer configurations, or working with more complex structures like for instance GANs or ADDA.

I would argue that your implementation is not exactly canonical and actually I'd recommend looking at the tensorflow.keras.Model example. Here they introduce a very nice, canonical way of making your own model.

Furthermore maybe Keras Model class API and Getting started with the Keras functional API can be of help.


Feature importance parameter in machine learning models like Naive Bayes Sorry for vague heading for the question. My question is that, is there any way to compare features (or attributes) used in machine learning algorithm? I have used Naive Bayesian classifier for binary classification which consists of total 6 features. I want to compare this features with one another and also list out features in priority order of their significance in models(how important that feature is). For example we use information gain or gini index to decide how effective feature is in CART. How can I compare this in naive bayes? I have read about confusion matrix and f measure but they are good at defining goodness of models and not features.

Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. The feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn for instance), the variability of those probabilities with respect to the classes should express the importance of those features.


LSTM sequence prediction: 3d input to 2d output I have this LSTM model
model = Sequential()
model.add(Masking(mask_value=0, input_shape=(timesteps, features)))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences=False))
model.add(Dense(features, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

and shapes X_train (21, 11, 5), y_train (21, 5).
Each timestep is represented by 5 features and
return_sequences is set to False because I want to predict one 5D array (the next timestep) for each input sequence of 11 timesteps.
I get the error

ValueError: y_true and y_pred have different number of output (5!=1)

If I reshape the data as X_train (21, 11, 5), y_train (21, 1, 5) instead I get the error

ValueError: Invalid shape for y: (14, 1, 5)

Note: the value 14 is due to the fact that I'm using cross validation.
What should I do?
Edit
I changed the model to
model = Sequential()
model.add(Masking(mask_value=0, input_shape=(timesteps, features)))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(features, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

and used the same shapes as before.
Here model.summary() gives
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
masking_1 (Masking)          (None, 11, 5)             0
_________________________________________________________________
lstm_1 (LSTM)                (None, 100)               42400
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 505
=================================================================

The idea is to produce a multilabel classification. After training the model, I evaluate it on the test data and this is what I get:

X[0] =  [[0 0 0 0 0],[1 0 0 1 0], ...,[0 0 1 0 0],[0 0 1 0 0]]
y_true[0] = [0 0 1 0 0]
y_pred[0] = 2

which is not what I want. How can I get an output of the same shape as y_true, so as to transform it into a multilabel classification?

What are your features like?  Given that you have a Dense layer outputting a softmax of size 5, this implies that all you want to predict is 1 feature, a categorical feature with 5 options.

If this is not true, more information about the features is needed to help here.

Your Y-variable for each feature should be of size (num_samples, time_step_len, num_categories_of_feature).  You need to one-hot-encode each categorical feature separately, which gives the last dimension size, num_categories_of_feature.  As you have it currently, the Y_train size is (num_samples, features).  So, as you have the problem framed, the network has no way to learn the sequence patterns, as you only give it the end result.  You should create your Y_train data to be the true value for the next time-step, for every time-step.  Hence, (num_samples, time_step_len, num_categories_of_feature).  Side note: I've only worked with LSTMs/RNN's on one problem, and this is how I did it.  I cared about learning the sequences in it's entirety, because my inputs at prediction time are variable.  If you always have 11 time-steps and always just want the next time-step prediction, this might not apply.  I really don't know to be honest.

This is where I'm not totally sure if this is the only way to do this, but the way I think of this problem for wanting to predict 5 categorical variables, you need a way to output softmaxs for each variable.  A softmax activation of size "features", like you have it here, is estimating a probability distribution of size 5, which implies your Y variable is only 1 categorical feature that has 5 potential values. So, you will need to set up your network to have 5 outputs with independent softmax outputs the size equal to the number of categories for each variable.  A single softmax should only be used to estimate a distribution over a single class variable.  10 options for feat1?  Softmax of size 10.  etc.

losses = {"feat1_output": "categorical_crossentropy", "feat2_output": "categorical_crossentropy", "feat3_output": "categorical_crossentropy", "feat4_output": "categorical_crossentropy", "feat5_output": "categorical_crossentropy"}
lossWeights = {"feat1_output": 1.0, "feat2_output": 1.0, ... , ...}# if everything is equal, dont worry about specifying loss weights.
metrics = {"feat1_output": "categorical_accuracy", "feat2_output": "categorical_accuracy", "feat3_output": "categorical_accuracy", "feat4_output": "categorical_accuracy", "feat5_output": "categorical_accuracy"}
opt = Adam(lr=init_lr,decay=init_lr / num_epochs)
model.compile(loss = losses, loss_weights = lossWeights, optimizer=opt, metrics=metrics)

Now, you will be optimizing 5 loss functions at the same time, one for each categorical prediction.  You must now have 5 Y-variable datasets, each of size (num_samples, time_step_len, num_categories_of_feature).  You will then give 5 y datasets for the outputs in the fit function, as a list.  However, to properly name the output layers, you will need to specify the names for the output layers in the model definition.


how to compute bernoulli entropy? I am reading gail implementation code in openai baselines. they compute bernoulli entropy as one of the loss in adversary network loss function.

In their code, they implement bernoulli entropy as this:

def logsigmoid(a):
'''Equivalent to tf.log(tf.sigmoid(a))'''
return -tf.nn.softplus(-a)

def logit_bernoulli_entropy(logits):
ent = (1.-tf.nn.sigmoid(logits))*logits - logsigmoid(logits)
return ent

also there is a reference of another openai implementation, it's the same code but I can't see any explanation to it.

I checked that the equation to compute bernoulli entropy is:

$ -p\log p - (1-p)\log(1-p)$

I think the later equation is the right way to compute bernoulli entropy, but the first one should be right too as it's written in openai's implementationn. I can't see any similarity, is there any relationship between these two expression?

Good observation, and yes, they are in fact equivalent ways of computing the entropy of a bernoulli random variable.

To begin, you have to notice that in the openai code, we do not have the value of $ p $ passed to the function, instead we have $ logit(p) $ which is defined to be:

$ logit(p) = log\frac{p}{1-p} = log(p) - log(1-p)$

Also, two formula to keep in mind. The first is the sigmoid function which is:

$ sigmoid(x) = \frac{1}{1+e^{-x}} $

And the second is the sigmoid of the logit which is:

$ sigmoid(logit(p)) = \frac{1}{1+\frac{1-p}{p}} = p$

(i.e. the inverse of the logit function is the sigmoid function)

Now we are ready to go from the first equation (used in the code) to the second equation (the generic equation to compute the bernoulli entropy)

$ entropy = (1 - sigmoid(logit(p)))*logit(p) - log(sigmoid(logit(p))) $

$ = (1 - p)*logit(p) - log(p) = (1 - p)*(log(p)-log(1-p)) - log(p) $

$ = (1 - p)*log(p) - (1-p)*log(1-p) - log(p) $

$ = -p*log(p) - (1-p)*log(1-p) = entropy $

Thus the formula used in the code is equivalent to the second formula you provided.


What is the best way to feature engineer features which have more than one repeated values? What is the best way to feature engineer features which have more than one repeated values ?
I want to parse this data and finally keep in a pandas df for further analysis.
Example,
I have data of people's profile which consists of
Name, Age, Gender, Company, Degree

Now it is easy to keep Name , age and gender which has specific single value, but company can have more than one value or multiple value like someone worked with Google or Microsoft or both Google, Microsoft.
Same case with Degree , people can have single as well as multiple values together.
Right now I have kept them as comma separated values like if someone has more than one company then value is  Google, Microsoft.
While I encode them using say sklearn Label Encoder I get different codes like
Google = 1
Microsoft = 2
Google, Microsoft = 3
Which I guess is not very accurate, as when the data increases it will explode with number of combinations also, if I have to find similar features of those who worked at Google I might not get the correct answer as code 2 and code 3 will never match.
Is there a better way to handle such data ?

Depending on the kind of ML problem you're facing, there might be more or less suitable methods.
Have you tried one-hot-encoding ? It should actually answer your question, by defining people as dense vectors filled with 0s excepted at a given company's coordinates, where they'll show a 1.
Well it's kinda bruteforce and the drawback is that it will for sure increase the dimension of your feature vectors, but it's a good starting point.


Predicting a random variable Y will increase within n minutes if random variable X increases I am starting to learn machine learning and I would like advice of how to model that random measure Y will increase within n minutes if random measure X increases.

X and Y have different units (GBP and USD) so I am using returns (percentage change) to make them the same unit.

My data looks like this

Time, X, Y
00:00, 1%, 0%
00:01, 0%, 1%
00:02, 2%, 0%
00:03, 0%, 2%

Just from observing this data you can see that Y increases after one second after X increases but not sure the best way how to model this!

You need to look for lag effects - the easiest thing is to try a number of lag time windows and look at how they correlate with target. If you think there is a distinctive lag impact, then at some point the correlation strength should increase.


How to split pieces of dataframe and create new dataframes based on it? I have the following dataframe that I need to split bu Product and create a new dataframe for each product contain the series ds and y. How can I do that?

One way of doing this is explained here, for example:

import pandas as pd
import numpy as np

df = pd.DataFrame([('bird', 'Falconiformes', 389.0),
                   ('bird', 'Psittaciformes', 24.0),
                   ('mammal', 'Carnivora', 80.2),
                   ('mammal', 'Primates', np.nan),
                   ('mammal', 'Carnivora', 58)],
                   columns=('class', 'order', 'max_speed'))

Simply do the following, would return groupby separated dataframe, stored in a list:

dfs = [x for _, x in df.groupby('class')]

#dfs
[  class           order  max_speed
 0  bird   Falconiformes      389.0
 1  bird  Psittaciformes       24.0,
     class      order  max_speed
 2  mammal  Carnivora       80.2
 3  mammal   Primates        NaN
 4  mammal  Carnivora       58.0]

Please note here you should lookup via list index not the class names, for example df[0]:

   class    order           max_speed
0   bird    Falconiformes   389.0
1   bird    Psittaciformes  24.0

And df[1] the other class!


Text classification with Word2Vec on a larger corpus I am working on a small project and I would like to use the word2vec technique as a text representation  method. I need to classify patents but I have only a few of them labelled and to increase the performance of my ML model, I would like to increase the corpus/vocabulary of my model by using a large amount of patents. The question is, once I have train my word embedding feature, how to use this larger corpus with my training data - my labelled data?
My data set is composed by 2000 patents which are labelled.
The patents used to train my word embedding corpus are 3 millions (some of my 2000 labelled patents are already included in this larger corpus) which I trained using Gensim.
Do you have any suggestions on how to do it?
Thank you very much in advance.
Rob

Use large amount of un-label data to finetune the BERT based model. You can train BERT in unsupervised manner. Then, use that bert to get embeddings of your input text of label data and train a classifier.


Explanation behind the calculation of training loss in deep learning model I am trying to model an image classification problem using convolution neural network. I came across a code on Github in which I am not able to understand the meaning of following line for loss calculation in the training loop.
I am omitting most of the detail and only placing the relevent code-

for batch_idx, (data, target) in enumerate(final_train_loader):
     loss = criterion(output,target)
     #Idea behind the below line
     train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

Cross-entropy loss function is being used here.

The line you're asking about

train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

is basically calculating the average train_loss for the finished batches

To illustrate, suppose 4 batches have been done (with average loss named avg_loss) and current is calculated from 5th batch (with loss named new_loss)

The new average loss is from

$\frac {4 \times \text{avg_loss} + \text{new_loss}} {5}$

This is exactly the same as

$\text{avg_loss} + \frac {\text{new_loss} - \text{avg_loss}} {5}$

which is the calculation done by the code


K-Means clustering for mixed numeric and categorical data implementation in C# I am a research scholar in data mining. I'm interested in C# implementation of K-Means clustering algorithm for mixed numeric and categorical data.

The solution is described here: K-Means clustering for mixed numeric and categorical data

C# implementation can be found in ALGLIB library, which I strongly recommend: http://www.alglib.net/translator/man/manual.csharp.html#gs_packages


How can I do simple machine learning without hard-coding behavior? I've always been interested in machine learning, but I can't figure out one thing about starting out with a simple "Hello World" example - how can I avoid hard-coding behavior?

For example, if I wanted to "teach" a bot how to avoid randomly placed obstacles, I couldn't just use relative motion, because the obstacles move around, but I don't want to hard code, say, distance, because that ruins the whole point of machine learning.

Obviously, randomly generating code would be impractical, so how could I do this?

Not sure if this fits the scope of this SE, but here's a stab at an answer anyway.

With all AI approaches you have to decide what it is you're modelling and what kind of uncertainty there is. Once you pick a framework that allows modelling of your situation, you then see which elements are "fixed" and which are flexible. For example, the model may allow you to define your own network structure (or even learn it) with certain constraints. You have to decide whether this flexibility is sufficient for your purposes. Then within a particular network structure, you can learn parameters given a specific training dataset.

You rarely hard-code behavior in AI/ML solutions. It's all about modelling the underlying situation and accommodating different situations by tweaking elements of the model.

In your example, perhaps you might have the robot learn how to detect obstacles (by analyzing elements in the environment), or you might have it keep track of where the obstacles were and which way they were moving.


Using word2vec with mixed language data I scraped my Facebook chat history and wanted to try out some basic machine learning stuff with word2vec. However, the data has all sorts of stuff - links, emoji, Cyrillic alphabet, etc. Even if I manage to clear some of those, would it be possible to process the Cyrillic alphabet with word2vec?

Word2Vec uses the bag-of-word model as input which means that you can use whatever alphabet. Building a bag of word just consists of attributing one feature per word in the entire dictionary of your corpus. You can do this with a cyrillic alphabet as with other alphabet.


Need help on Time Series ARIMA Model I'm working on forecasting daily volumes and have used time series model
to check for data stationarity. However, I'm strugging at forecasting data with 90% accuracy. Right now variation is extremely high and I'm just unable to bring it down.

I've used log method to transform my data. Please find the link to folder below which contains ipynb and csv files:

https://drive.google.com/drive/folders/1QUJkTucLPIf2vjo2mRmoBU6be083dYpQ?usp=sharing

Any help will be highly appreciable

Thanks,
Rahul

Before anything, draw a graph.

You'll notice that you have at least three process: one around 20-25, one around 60-90, and one around 250-450. A closer look tells you that the first one is on Saturdays, the second on Sundays and the last one for other working days.

Further notice that the Sundays series is different since 2016-12-26. And have a close look to Christmas and Banking holidays to decide if they behave like Saturdays, Sundays or ordinary days.

Also notice a set of outliers between 2017-12-21 and 2018-01-02. Remove them before to fit. You also have overproduction by the end of November 2016. Remove them also.

Then make a 7-day moving average, which as you will see is rather linear, except for an underproduction in January 2017, for which you may have an external explanation (following the change of end 2017).

With this linear fit, you may have a look to a weekly seasonality. And reach a 70-80% accuracy.

But forget a 90% accuracy. There is much more fluctuation in your data.


Can someone explain the Tensorflow code below? I have been trying to understand the below code (written in Tensorflow), but unable to do so as I am not very proficient with Tensorflow. I am looking for some help. Below is the code:
    if model == 'mse':
        rec_loss = tf.losses.mean_squared_error(img, img_rec)
        kld_loss = -tf.reduce_mean(0.5 * (1 + z_log_sigma_sq - z_mu ** 2 - tf.exp(z_log_sigma_sq)))
    else:
        if model == 'gaussian':
            log_sigma = tf.Variable(0.0, trainable=False)
        elif model == 'sigma':
            log_sigma = tf.Variable(0.0, trainable=True)
        elif model == 'optimal':
            log_sigma = tf.log(tf.sqrt(tf.reduce_mean((img - img_rec) ** 2, [0, 1, 2, 3], keepdims=True)))

        rec_loss = tf.reduce_sum(gaussian_nll(img_rec, log_sigma, img))

Where the gaussian_nll function is defined below:
def gaussian_nll(mu, log_sigma, x):
    return 0.5 * ((x - mu) / tf.exp(log_sigma)) ** 2 + log_sigma + 0.5 * np.log(2 * np.pi)

I have these questions:

What is the difference between log_sigma under model == 'gaussian' and under model == 'sigma'? (only difference in the definition of the variable is the flag trainable)
What is the difference between the tf.losses.mean_squared_error under the model=='mse' and log_sigma under model == 'optimal'? Aren't they both similar?

They are correct, but I am unable to correctly read it. Any help regarding the code would be greatly appreciated. For those of you who wants to take a look at the full code on github, here is the link github page (the above piece of code appears between lines 34 and 44)
Thanks once again for your help.

gaussian is different from sigma, because sigma is trainable while gaussian is not. It implies that the value of sigma can be optimized but not of gaussian.
optimal is different from mse as

in the case of mse, the final loss is computed as MSE.
but in the case of optimal, we use gaussian as the loss function, with MSE acting as the variance of the gaussian kernel.


Creating training data My task is to classify free text originated from customer complaints about our product.

I have created a Taxonomy and have around 10 different categories. I've realized that these categories include keywords.

Example:

"Customer doesn't understand how to use the product".

Keywords: understand, knowledge, know, aware.

Record:

Training, Customer doesn't understand how to use the product

I'm using Google Prediction API. When training the model, I would categorize previous text as: "Customer doesn't understand how to use the product" - Training.

How can I add keywords to free text/training data to help the model perform better and provide a better confidence level?

Data in training set:

Training, understand knowledge know aware
Training, Customer doesn't understand how to use the product

Right now, I'm adding Keywords into same training data, but looking for a better suggestion.

Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is balanced, stratified sampling. Assuming sufficient labelled data, ensure that your final training set has a balanced number of text examples for each class/label. Depending on your situation, you may need to over/under sample or somehow deal with the problem of highly imbalanced classes (see 8 tactics to combat imbalanced classes).

The simplest NLP approach to use a bag of words technique, simply indicating the presence/absence of a word in the sentence. Thus each sentence becomes represented as vector of length n, where n = the number of unique words in your data set.

data_set <- c("the big dog", "the small cat", "the big and fat cow")
words <- strsplit(data_set, split = " ") #tokenize sentences
words
##[[1]]
##[1] "the" "big" "dog"
##
##[[2]]
##[1] "the"   "small" "cat"
##
##[[3]]
##[1] "the" "big" "and" "fat" "cow"

vec <- unique(unlist(words)) #vector representation of sentences
##[1] "the"   "big"   "dog"   "small" "cat"   "and"
##[7] "fat"   "cow"

m <- matrix(nrow = length(data_set), ncol = length(vec))

for (i in 1:length(words)) { #iterate the index of tokenized sentences
  vec_rep <- vec %in% words[[i]] #create binary word-feature vector
  m[i,] <- vec_rep #update matrix
}

df <- data.frame(m, row.names = NULL)
names(df) <- vec
df
##   the   big   dog small   cat   and   fat   cow
##1 TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE
##2 TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE
##3 TRUE  TRUE FALSE FALSE FALSE  TRUE  TRUE  TRUE

Sometimes you can increase performance by adding bi-gram and tri-gram features.

##given: "the big dog"
##unigrams <- {the, big, dog}
##bigrams <- {the big, big dog}
##trigrams <- {the big dog}

Sometimes weighting words by their frequency improves performance or computing the tf-idf.

Another way to increase performance, in my experience, has been custom language feature engineering, especially if the data is from social media sources replete with spelling errors, acronyms, slang, and other word variants. Standard NLP approaches will typically remove stop words (e.g., the, a/an, this/that, etc.) from vector representations (because closed class, high frequency words often don't help discriminate among class/label boundaries). Because vector representations are typically highly dimensional (approximately num of unique words in the corpus/data set), dimensionality reductions techniques can increase performance. For example, one can compute chi-sq, info gain, etc. on a word feature's distribution across classes -- only keep those features/words above some threshold or below some pre-established p value).


Detecting specific points in (noisy) dataset In my recent work I've came across a problem where I need to find certain points in quickly oscillating data. Let's work with syntethic data instead of real measurements and ignore the noise for the time being. These points are called stationary phase points (SPP for short). For human eye they are quite easy to spot: the oscillation slows down locally. An example:

Some important things to note:

It's always a local maxima or minima (so a reliable peak detection algorithm can help a lot, which I've been  working on recently)
The graph is ideally normalized to [-1, 1], in real measurements it's a little distorted
Usually the Nyquist–Shannon sampling theorem is not satisfied at the borders of data and the oscillation is avaraged out. (In the synthetic data I created it's satisfied, see below for a real-world dataset.)
The general number of SPPs are 1 or 2, 3 or 4 are really rare, but exist.
A SPP can continiously move around in the x direction as conditions are changing, can vanish at the borders of x-axis, and also one SPP can separate into two different SPPs which are moving away from each other.

The main goal would be reliably monitoring these points movement (ideally in real time, but first the SPP detection from x-y data is the target).
There are some other generated graphs with indication for SPPs, and the last is from real data.

To solve this, my first thought is to detect all the extremal points in data, then generate their consecutive distances, and where that distance exceeds a certain threshold value, it's a SPP. But setting a good threshold, correctly identifying extremal points seems quite hard and unique for each graph.

My question is:
 Do you think it would worth using a machine learning tool there, and if yes, which algorithm would be the most suitable? As a physicist I'm inexperienced, I only have basic knowledge about ML (but willing to learn!)
In general, can you think of any useful algorithm or idea for such situation?

Note: Mostly I'm working in Python with numpy/scipy.

As far as local max and min detection concerned I use scipy.signal.find_peaks and scipy.signal.find_peaks_cwt along with scipy.signal.savgol_filter.

This is a common problem in signal processing. It would be useful to look into the rich history of anomaly detection in signal processing.

One option would be to apply a Fourier Transform which decomposes a function into its constituent frequencies. Define a baseline set of frequencies. Then anomaliness frequencies could be found relative to the baseline.


Understanding Transformer's Self attention calculations I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-framework#comment-160771

What is the value of Key, Value in the self attention calculation of Transformer model?
Query vector is embedding vector for the word that is queried, is that right?
Is attention calculated in RNN is different from self attention in Transformer?

What is the value of Key, Value in the self attention calculation of Transformer model?

You have the queries matrix Q. Imagined it to a coordinate system. the x-axis is the key and y-axis is the resulting value.

Query vector is embedding vector for the word that is queried, is that right?

Yes, for the word the word or token

Is attention calculated in RNN is different from self attention in Transformer?

RNNs is a family of neural networks. They don't include attention per se.
A resource, that helped me greatly is: https://jalammar.github.io/illustrated-transformer/ maybe it will you also to better understand transformers.


How to approach graphed data for a binary classification system? I'm looking for articles and other reading material on handling and modeling graph-shaped data for classification systems. As I've never working with graphed base machine learning anything from novice to advanced would be good. I'm after solving graph problems, but representing data that natively fits into a graph diagram in a way that would benefit machine learning on it.

I would like to experiment with predicting based on a graph of directed interactions between nodes in a system as well as incidental events occurring on one or more of the nodes (In hopes of deriving information from correlations between # of nodes similar incidents happened at, relations between nodes with similar incidents, etc).

Anything from published articles to experience tips and best practices would be highly appreciated.

Super comprehensive question. You are basically asking for tones of directions! I'd suggest to start with link prediction problem. So assume you have a directed/signed/weighted graph and you want to find the most probable next interaction (link/edge). I'd suggest to have a look at Jerome Kunegis papers and you may see it as a starting point and go on from there.


Using Binary Image Classification on Video Using tensorflow/Keras, I have built a good model which is currently binary classification.  For example, training labels would be images of a person's knees bent or knees not bent.

The idea behind such a model could be using a continuous video feed, and when it detects either knees bent or not, a certain probability would output.  It works, however, since it's binary, I have probabilities close to 0 or 1 even though a person is not even in the video feed.

Any ideas on how I can change my model (perhaps multi-class) in over a continuous feed - which could be hundreds of images over a minute would only give me probabilities for those events of knees bent or not?  If going multi-class, would I then have to also train all other potential images?  That seems close to impossible, even with using transfer learning with ImageNet.  Any suggestions?

Using a multi-class network would probably be the most efficient approach.

Training two networks (as suggested in the comments: one network to detect the presence of a person and one to detect the state of the knee) is unnecessary and overly complex.  You can train the network on three classes: bent knee, straight knee, absent knee.  For the absent knee, you do not need to train on all possible images, just images you expect to see.  For instance, these negative data could be the background of your intended images; take a picture with the knee and one without.

If speed is a major requirement, the model architecture becomes critical.  I suggest exploring MobileNets as they are quite fast and do not sacrifice too much accuracy.


how to build a predictive model without training data neither historical data I m trying to score "how much a product is expected in the market". I created some features:
How much this product is used each year.
Where was it used .
how many product for each country.
the main component of this product
compared the product with other similar products.
etc.
I built a big matrix/table with all these information.
I want to create a prediction model, where the input is a main component of a product and the output is ( expected with a pourcentage).

I dont have a training data ( just features without related results which indicates if the product is expected or no)
I dont have historical data to confirm the results of my work.

Some advices ?

Thank you all

If you do not have data then I think the problem is more of research than  Machine Learning.
Ask your research team to gather primary data about the product by conducting surveys, polls,etc.

Depending upon the data you get from research, you can use some statistical methods like hypothesis testing to solve the problem.


Which dataset I should use when I am retraining my model? I trained my deep learning model using x dataset and now I got new dataset and I give it name as y. I want to retrain my model on this new dataset which is y. Do I need to use x+y dataset or just y? Can you tell me what is right approach and what are the effects of that?
If nothing like that and other approaches are available please post it that helps a lot.
Thanks in advance.

So, I understand the question is asking what data should you use, when given new data.
You might want to provide more context to your question as this might seem like a loose answer. The way you go forward with this depends on a couple of factors:

Is the initial dataset ($D_1$) have the same features/columns as the new dataset ($D_2$), such that you can integrate them (as you said). This obviously can help to answer whether to use either $D_1$/$D_2$ or both.
It also depends on the problem specification: if this new dataset provides new features (maybe more accurately measured), which are more appropriate to the problem spec, then obviously you should then work with $D_2$ and train the model with this dataset. If the problem spec can be solved using both datasets, then use both as this can help to improve generalisation performance more so than using $D_2$ alone.


Error while using lightGBM's cv() function for a regression problem I am trying to use lightGBM's cv() function for tuning my model for a regression problem. My main model is lightgbm.LGBMRegressor(). However, I am encountering the errors which is a bit confusing given that I am in a regression mode and NOT classification mode. Below are the code snippet and part of the trace.

param_grid = {
    'class_weight': [None, 'balanced'],
    'num_leaves': list(range(30, 150)),
    'learning_rate': list(np.logspace(np.log(0.005), np.log(0.2), base = np.exp(1), num = 1000)),
    'subsample_for_bin': list(range(20000, 300000, 20000)),
    'min_child_samples': list(range(20, 500, 5)),
    'reg_alpha': list(np.linspace(0, 1)),
    'reg_lambda': list(np.linspace(0, 1)),
    'colsample_bytree': list(np.linspace(0.6, 1, 10)),
    'objective': 'regression'
 }
lgb_train = lgb.Dataset(X_train,y_train)
cv_results = lgb.cv(params, lgb_train, num_boost_round = 10000, nfold = 10, metrics = 'rmse', shuffle=False,
           early_stopping_rounds = 100, verbose_eval = False, seed = 50 )

Part of the trace of the error I am getting is

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
<ipython-input-84-3f76f5345fda> in <module>
      1 # Perform cross validation with 10 folds
      2 cv_results = lgb.cv(params, lgb_train, num_boost_round = 10000, nfold = 10, metrics = 'rmse', shuffle=False,
----> 3            early_stopping_rounds = 100, verbose_eval = False, seed = 50, )

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/engine.py in cv(params, train_set, num_boost_round, folds, nfold, stratified, shuffle, metrics, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, fpreproc, verbose_eval, show_stdv, seed, callbacks)
    456     cvfolds = _make_n_folds(train_set, folds=folds, nfold=nfold,
    457                             params=params, seed=seed, fpreproc=fpreproc,
--> 458                             stratified=stratified, shuffle=shuffle)
    459
    460     # setup callbacks

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/engine.py in _make_n_folds(full_data, folds, nfold, params, seed, fpreproc, stratified, shuffle)
    315         else:
    316             tparam = params
--> 317         cvbooster = Booster(tparam, train_set)
    318         cvbooster.add_valid(valid_set, 'valid')
    319         ret.append(cvbooster)

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/basic.py in __init__(self, params, train_set, model_file, silent)
   1552                 train_set.construct().handle,
   1553                 c_str(params_str),
-> 1554                 ctypes.byref(self.handle)))
   1555             # save reference to data
   1556             self.train_set = train_set

/anaconda3/envs/py36/lib/python3.6/site-packages/lightgbm/basic.py in _safe_call(ret)
     44     """
     45     if ret != 0:
---> 46         raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
     47
     48

LightGBMError: Unknown objective type name: r

I don't understand the error. Moreover, if I remove the 'objective':'regressor', then we are getting the error: y variable is only 1 class which seems to me to be referring to a classifier.

Any help would be great.

Thanks

Answer
In the param_grid dictionary, pass every hyper-parameter as array e.g., in your code above, you missed-out to place 'objective': 'regression' part of param_grid as an array. Even though, you have are using single value place it as array i.e. 'objective': ['regression']
Therefore, you can update below piece of code in your script and re-run. It will remove the error.
`
param_grid = {
    'class_weight': [None, 'balanced'],
    'num_leaves': list(range(30, 150)),
    'learning_rate': list(np.logspace(np.log(0.005), np.log(0.2), base = np.exp(1), num = 1000)),
    'subsample_for_bin': list(range(20000, 300000, 20000)),
    'min_child_samples': list(range(20, 500, 5)),
    'reg_alpha': list(np.linspace(0, 1)),
    'reg_lambda': list(np.linspace(0, 1)),
    'colsample_bytree': list(np.linspace(0.6, 1, 10)),
    'objective': ['regression'] # <<< CHANGE REQUIRED TO remove ERROR
 }