Rank links from rss feed I am trying to create a script to filter the most "intersting" articles from an rss feed and rank them.

feeds = ['http://feeds.theguardian.com/theguardian/technology/rss',
         'http://rss.cnn.com/rss/money_news_international.rss',
         'https://news.ycombinator.com/rss?format=xml',
         'http://feeds.reuters.com/reuters/companyNews',
         'http://feeds.reuters.com/reuters/businessNews']

I tried using applying the K-means algorithm to RSS feeds to filter the most popular articles from thousands of links in an attempts to reduce my personal RSS reading time.

However, I feel that this might not be state of the art.

Any suggestions on papers, actual implementations or approaches to get a proper list of "must-read" articles.

I appreciate your replies!

Big Note: Unless you do not define what "interesting" is, Machine Learning can not do anything for you. Clustering only tells you which documents are similar to eachother but YOU need to define which cluster is interesting (BTW, clustering in terms of context is called Topic Modeling as each cluster of written text is about some topics).

Now my suggestions:

Topic Modeling

Use a simple LDA to detect topics of each document and cluster docuemnts based on topics. For each topic, print first 5 dominant keyword and if they interest you, then rad the articles.

Keyword Search

Set up a list of keywords you would like to read about. Using a simple TF-IDF model you can find "high score" docuemnts according to your favorite keywords.

Keyword Extraction

You may extract keywords of each document usng RAKE with a score higher than a manual threshold. See if keywords interest you or not. (conceptually close to LDA but here you can extract keywords from each individual document, while LDA makes sense if you have a corpus of text.)

Update

According to the additional info in the comments, the question turned even more towards classic recommender systems.

We have the number of clicks per article and we use it as the interestingness score. Now one may say (a very basic idea to be honest):

Crawl all the articles and Use RAKE to extract keywords from n most interesting articles. Use those keywords as search queries.
Use BM25 algorithm to search those queries and get bm25 score for each article according to queries.
Aggregate those bm25 score with interestingness score (e.g. simply multiply them) and rank articles from most interesting to least interesting.


Model not learning when using transfer learning I am working on a personal project on image classification (two classes) and am trying to see how the MobileNet v2 structure would perform. While training the training accuracy is already quite high after the first epoch after which is continues to increase. The validation accuracy, however, seems forever stuck around the 50% mark.

I have tried both SGD, Adam, and RMSprop as optimizers, however they both produce the same results. I have also used both a MobileNet v2 with and without ImageNet weights without seeing any difference. Other CNN architectures work perfectly fine, achieving up to ~95% validation accuracy. I am using the following code for my model:

from keras.applications import MobileNetV2
from keras.models import Sequential
from keras.layers import GlobalAveragePooling2D, Dropout, Dense

basemodel = MobileNetV2(input_shape=(128, 173, 3), include_top=False, weights=None)
model = Sequential(
    [
        basemodel,
        GlobalAveragePooling2D(),
        Dropout(0.5),
        Dense(64, activation="relu"),
        Dropout(0.3),
        Dense(1, activation="sigmoid"),
    ]
)
model.compile(loss="binary_crossentropy", optimizer="sgd", metrics=["accuracy"])

Is there something obvious that I am missing or am I just using incorrect hyperparameters for my dataset?

Well the critical step for finetuning/transfer learning any model in tensorflow[1,2+]:

use right preprocessing for images (find the preprocessing which requires mobilenetv2)
start with "imagenet" weights
change the BatchNormalization behavior, in tf the default momentum value is 0.99 which is not best, in pytorch it is set to 0.9 which is better when finetunig your model and you don't have lot of data (see the link and this for more)


extraction information from resume I have a project in machine learning in which I need to analyze a
  curriculum vitae. for that I have to write a python program. It uses
  basic techniques of Natural Language Processing like word
  parsing,chunking,reg ex parser. If you run the algorithm you can
  easily capture information like name,email id,address,educational
  qualification,experience in seconds from a large number of documents.
  but I have a confusion between two methods LSTM (RNN) and NER? What do
  you think is the best method?

LSTM is a neural network architecture used for sequence prediction, whereas NER is name of NLP task.

The two are incomparable, as one is a method and the other type of problem.

The relationship between them is that LSTMs can can be used for sequence tagging, which can be used for NER.


Scientific name for the problem given in the description. (In machine learning what kind of problem it is termed as) Features marked as boolean values (around 50,000) , need to score a set of fixed output (around 25,000).

I have wikipedia topics relevant to a particular url as features. And the output is the advertisement keywords which will perform well on those url. If I have the training data, i.e. advertisement keywords performance on urls. Given a url (I can extract the features i.e. relevant wikipedia topics), I need to predict which keyword might perform well on that url.

What is the scientific name for such kind of problems in which there is a big boolean input set and I need to score a big fixed output set. Can you name few algorithms for such kind of problems.

Ok. I think I'm getting a clearer picture. Firstly any name isn't going to be different just because the data is large. Of course in the big data community anyone is free to coin a term and time will show if it gets adopted (same goes with what we might consider a scientific name for that matter too).

Another thing is input output. These terms can be confusing as what is input by a comp scientist into a training model is both the input and output of what a statistician would define.

I'm not aware of a name for the problems but names for models addressing such a problem include logistic regression and probit regression. Both deal with a(n) (set of) independent variable(s) - the feature name in your case - and a resultant dependent variable - the binary relevancy. The regression looks at fitting a linear function to a hidden variable; whose threshold gives the binary result. Fitting an "input" training set then allows the scoring based on this linear function, whether you want a binary score or a real valued one (from the hidden variable) that will be more indicative of the probability of your prediction.


MLflow real world experience Can someone provide a summary of the real world deployment experience of MLflow? We have a few ML models (e.g., LightGBM, tensorflow v2, etc.) and want to avoid framework like SageMaker (due to customer requirement). So we are looking into various ways of hosting ML models for inferencing. Latency is one key performance metric that is very important to us. MLflow looks like a good choice for us.
It would be greatly appreciated if the users of MLflow can share their experience.

I am currently using MLflow together with Databricks at my company, and up to now, I would say it is quite ok for managing models versions, tracking models metrics and building the models workflow from a dev/experimenting/training status to production.
I am not sure if it can be considered a standalone tool for that, meaning that you should combine it with something like Azure Databricks (what I currently use).
It's nice the way you can customize the tags, metrics or artifacts that you might want to keep logged after each run or model training, as follows:

where you can also visualize several run experiments as:

(these models are for the shake of the example, so metric values are not optimized)
and storing the generated models files (e.g. a pickle file) in the corresponding location:

Once it is generated, you can register your best model via the databricks UI (in case you finally use it) or via mlflow code as follows (here I register the last one, you should take the one with the best prefered metric which mlflow also gives you directly as an option):

so as to be loaded later on and make predictions with your model.
Once you your are happy with some model, you can transition it:

Another nice feature I like is that it offers you custom modules for the most used frameworks like scikitlearn, xgboost, keras, etc
and build a container image, deploy it... but this is another longer topic.
Of course, you can also use MLflow with another third party or even locally.
About the performance point, do you mean the inference itself or some other source of latency? You can make inferences with your model via SPARK UDFs (PySPARK in my case) info, in case you need several predictions as fast as possible or simply for batch inference.


Terminology of time series The terms Time Series Analysis, Time Series Forecasting, and Time Series Modeling are widely used. But I don't see the clear difference in between.
Does time series analysis mean the statistical analysis of time series.
Do the time series modeling and forecasting deal with the machine learning methods of dealing with time series.
Can anyone explain the differences between these terms?

I think these are often used colloquially as synonyms, but let's try to find the differences.
Each of them begins with "Time Series" (TS). So the difference lies in the three following terms. here with my interpretation:

Analysis - wanting to describe and understand characteristics the observed data coming from the generating function$^1$.
Forecasting - wanting to somehow predict the future output of the generating function.
Modeling - wanting to describe and understand the generating function itself, via an approximation.

So as you see, I think time-series analysis and time-series modelling are almost the same thing. I just feel the latter most likely involves really using a real model, with parameters. That doesn't mean specifically parametric or a non-parametric approaches.
I personally wouldn't draw a line between them in terms of "uses statistics" vs. "uses machine learning", as your question suggests.

A natural progression in approaching a time-series problem (in my opinion) would be:

understand the data (TS-analysis)
understand the underlying function (TS-modelling)
test how well I have understood the function and its output (TS-forecasting)

$^1$
With the term "generating function", I don't mean the strict mathematical definition, rather just the underlying process or system that is generating the observed data


Conv-2 CNN architecture - CIFAR-10 I have a CNN architecture for CIFAR-10 dataset which is as follows:

  Convolutions:           64, 64, pool

  Fully Connected Layers: 256, 256, 10

  Batch size:             60

  Optimizer:              Adam(2e-4)

  Loss:                   Categorical Cross-Entropy

When I train this model, training and testing accuracy along with loss has a very jittery behavior and does not converge properly.

Is the defined architecture correct? Should I have a maxpooling layer after every convolution layer?

The Conv-2 architecture is an attempted implementation of the following paper.

The following table is taken from The Lottery Ticket Hypothesis By: Jonathan Frankle, Michael Carbin

Thanks!

Usually, the setup of having two conv. layer followed by one pooling layer works well for CIFAR. Since your batch size is relatively high and the learning rate relatively low these are probably not the source for the lack of convergence. For further analysis a plot of your train and validation loss (not accuracy) would be helpful.

Therefore, I suggest the following steps:

1. Plot train and valid loss curves to understand where you're standing in terms of convergence and overfitting/underfitting, e.g. see this article on how to read loss curves.

2. If your current model capacity is insufficient train for more epochs or add another stack of layers with the same structure (features should be increasing throughout the network by for example first having two conv. layers with depth 32 and the next two conv. layers with depth 64)

3. If overfitting becomes a problem add dropout

Moreover, I do not think you need multiple fully connected layers at the end but they might not do harm either. Adam might work well the standard learning rate here but if you have manually optimized yours that is fine too (the learning rate is, in fact, the most influential hyperparameter so it almost always makes sense to optimize it).

In deep learning it usually makes a lot of sense to check what networks have worked on the task you're working on (or similar tasks). For CIFAR you find a simple but well working network here.


How to calculate Precision and Recall using confusion matrix in Matlab? I am working on 3 class problem.How to calculate precision,recall ,f-score,MCC of each class while using  MATLAB. Here is my confusion matrix:

2775  0    0
1    591   0
4     0   845

I am calculating Accuracy from Confusion matrix in this way:

Accuracyy = 100*sum(diag(confusionMat))./sum(confusionMat(:));

I want to measure below performance measures for each class.I know the formulas but how to execute this in MATLAB. Please help!!! I would really appreciate! Thank you!

Precision=TP / (TP + FP)
Recall= TP / (TP + FN)
F-score = 2*TP /(2*TP+ FP + FN)
Matthews Correlation Coefficient (MCC)

if yHat are your predictions and yval are your y true then

tp = sum((yHaT == 1) & (yval == 1));
fp = sum((yHaT == 1) & (yval == 0));
fn = sum((yHaT == 0) & (yval == 1));

precision = tp / (tp + fp);
recall = tp / (tp + fn);
F1 = (2 * precision * recall) / (precision + recall);


Mini batch size and reset states 3 is a big file, but I would like to reset the state after mini_batch_size of 50.

n_epoch=10000
n_batch=50

# create and fit the LSTM network
model = Sequential()
model.add(LSTM(3,batch_input_shape =(n_batch,trainX.shape[1], trainX.shape[2]),stateful=True))
model.add(Dense(1))
model.add(Activation("linear"))
model.compile(loss="mse", optimizer="adam")
model.summary()

#fitting model
for i in range(n_epoch):
    history=model.fit(trainX, trainY, epochs=1, batch_size=n_batch,verbose=2, shuffle=False)
    model.reset_states()

I am getting the error:

value error:  In a stateful network, you should only pass inputs with a number of samples that can be divided by the batch size. Found: 63648 samples<

How can I train a LSTM in mini_batch size of 50 (a number which is not divisible by trainX)?

You can simply make your training data a shape that is divisible by 50.
    In [1]: sample_size = 63648

In [2]: to_remove = sample_size % 50

In [3]: end_index = sample_size - to_remove

In [4]: model.fit(trainX[:end_index], trainY[:end_index], ...

What are the dimensions of trainX and trainY? You are not specifying a validation dataset, is that intentional?

You say you want to reset the state after a mini-batch, but you are resetting the state after each epoch (one loop through your entire dataset).

One more comment: the n_epochs = 10000 seems excessive.


Converting a regression problem into a classification problem I have a regression problem where a neural network has to predict a value from $0$ to $19.999...$. I would like to convert this regression problem into a classification problem with the classes $0, 1, 2, ..., 18, 19$. Therefore I am using a $\mathrm{softmax}$-output layer.

Now, if I use a categorical crossentropy-loss, the loss is the same if the network classifies an input as $1$ instead of $14$ and if the input is classified as $13$ instead of $14$. I like to punish the loss more if the predicted value / class is farther away from the true value / class.

Is there a better loss function than categorical crossentropy for this use-case?

I think what you should be looking for is ordinal regression/classification in order to utilize the ordering of the classes.

Example python/matlab implementations, with some extra resources.

For fast integration with your system, you could create a new ordinal regression model using the outputs of the already trained network as input to the new ordinal model and the true ordinal value as the output.

For another quick/dirty solution you could try simple Mean Absolute Error or Mean Squared Error, according to this study, between the predicted and target classes of your system, as it is right now, instead of the categorical cross-entropy error function.


What kind of RAM to choose for data analysis? I'm planning to upgrade my computer and also want to buy some new DDR3 RAM modules. Making some research I realized that there are 2 main characteristics of RAMs (beside capacity): frequency and latency.

I want to use my computer for data analysis as well (like Kaggle competitions). Usually I work in R and sometimes in Python.

I understand that there is a trade-off between latency and frequency (that is also reflected in the price sometimes). My question is which one is more important for data analysis purposes? How can it effect real world performance? How much difference should I expect between choosing a model with overall better and lower parameter values?

UPDATE: I don't ask which RAM is better in general. I need answer from users who know how memory management happens in R (or Python). How are the different ML algorithms managed in memory and which RAM characteristics are preferred to support it. I would also appreciate personal experiences on this field.

It's not easy to compare two RAMs with different frequency and latency, since both of them affect your performance with not the same way.

The short answer is:

If you have two RAMs with same capacity and frequency, choose the one with lower latency.
If you have two RAMs with same capacity and latency, choose the one with the higher frequency.

From lifehacker.com

  Essentially, you have two things to worry about when it comes to RAM "speed": frequency, which deals with how much data can be transferred to the stick at one time, and latency, which is how quickly it responds to requests. In the current market, as you get to higher frequencies, latency tends to increase, so in many cases, they tend to balance each other out. Buying RAM with a higher speed doesn't matter a ton.

The long answer:

Don't worry about RAM. Yes, if you have a crappy RAM, you will also have a problem with your performance. But, the most important thing in Data Analysis is the performance of your code. An example from my personal experience:

I had a text dataset, about 30GB. I needed to create a corpus for a simple TF-IDF. The brute-force code, spent about 15 hours, whereas a map-reduce algorithm spent about 1.5 minute for the same dataset.

I work on a MacBook Pro with 16GB RAM. I didn't have any problem with performance. However, if you have huge datasets and even code improvements and RAM upgrades cannot help, you may reconsider to move to Hadoop or similar.


How to plot similarity of two datasets? I'm performing some simulations, and at the end I get a CSV file with three columns. One column holds the values for the x-axis, which was also input to the simulation and theoretical calculations, second one holds theoretically expected values, and the other column holds the values obtained by the simulation. I was planning to plot something like this:

But that does not look good in my case, as the values in y-axis normally keep doubling, and the values for the x-axis exponentially increase, so most of the points end up getting collected at the lower left part, near the intersection of x-axis and y-axis of the plot. Therefore, I need a different way to plot my data, which will be more visually appealing and inform how close the simulation results are to the theoretical expected ones. For example, some of my values can be seen below (and they keep increasing in such a way):

x         = [2, 4, 8, 16, 32, 64] # partially removed for brevity
expected  = [47.9995, 95.9783, 191.9127, 383.9708, 767.8831] # partially removed for brevity
simulated = [48, 96, 191.8, 383.8, 767.4] # partially removed for brevity

What is a good way to plot such a data that doubles in the y-axis and exponentially increases on the x-axis all the time, and to view how similar the two datasets actually are?

Here is an example of r lattice xyplot using log scale on the x axis and the difference of your two measures I(expected - simulated)

df <- data.frame(
x         = c(2, 4, 8, 16, 32, 64),
expected  = c(47.9995, 95.9783,
191.9127, 383.9708, 767.8831,
1457.2771),
simulated = c(48, 96, 191.8, 383.8,
767.4, 1458.1228))
xy <- xyplot(I(expected - simulated) ~ x ,
auto.key=TRUE,
data =  df ,    type=c("p","g"),
scales=list(x=list(log = 10) ),
ylab="difference expected - simulated", xlab="x", main="Simulation Results" )
print (xy)

Note, that I added a 6th result to your sample data,that was missing.


How to learn a classifier from a dataset with high imbalance What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the "target" class being much rarer than the "background" class)? For example,

Should one first down-sample the majority/background class to reduce its frequency and then readjust the probabilities reported by the learning algorithm? How should one do the readjustment?
Should one use different approaches for different learning algorithms, i.e., are there different techniques for dealing with imbalance in SVM, random forests, logistic regression, etc.?

A common strategy for dealing with imbalance is to penalize harder the missclassifications that select the class with higher frequency.

In a binary classification problem you could penalize by dividing 1/n where n is the number of examples of the opposite class.

See the following from Prof. Jordi Vitriá

This is the loss function for structured output SVM.

The problem you mention is common in object recognition and object classification in images where much more background images are used than images containing the object. A stronger case happens with exemplar SVM's where just a single image of the object is used.


Predicting binary target value based on unlabelled features I have a dataset of around 15 stocks having the following data format :

TimeStamp | F1 | F2 | F3 | ...Fn | Y

TimeStamp: Date and Time (masked)

Fi : A random feature(numeric) which contains some information about the target Variable

Y: A binary 0/1 target variable to predict.

The time column is meta data, not a feature. All other columns are features.

The dataset is sufficiently large. I need to make a predictive model for predicting binary values using the random unlabelled numeric features( 71 total features) . Can you help me how to proceed with this problem ?

You have lots of choices here. You did not say whether the Fi are categorical or numeric. If numeric, logistic regression is a simple and obvious place to start. Logistic regression also works with categorical variables, but best if the number of such variables is not large and the number of categories is small. The logistic regression will let you know if anything is going on at all, but it may not give you the best predictive power.

For a small number of categorical variables, I would try a log-linear model.

Assuming more variables:

If categorical, then I would try a decision tree. Decision trees also work with numerical variables or both -- so it's a good second choice either way, but more suited to categoricals.

If you are getting nice results from the tree and the number of variables is large, I would then move to a random forest. These don't add much, in my experience, when the number of variables is small (say 30 or less), but they can greatly improve the stability and predictive power of your model when you have a large number of variables.

If the variables are numeric, a support vector machine, or a Gaussian classification model, would improve on your logistic regression.

Note. I have a statistics background, rather than a machine learning background, so I tend not to take my explanatory variables at face value. It's nice to get the children off the street, so if the number of variables is not super huge, I might try a lasso regression to see if I can drop any of the variables. If a number of your explanatory variables do not actually give much information about the target, they will basically add noise and instability to your model. But I'm old fashioned. A lot of people just throw everything in the hopper and leave it at that.


How do I change the location of Google BigQuery So, I'm trying to use Google BigQuery for the first time for a project of mine, and I'm a bit confused. The documentation isn't helping much, and it looks like all the Google employees are gone thanks to the current epidemic, judging by the blogpost by Google.

I've got several .csv files containing tables of unlabeled numerical data uploaded to Google Cloud storage, in their Australian servers. What I want to do is to use BigQuery to perform k-means clustering and analysis of correlations between variables on them. However, I'm having issues trying to get the data transferred into BigQuery.

I tried using the "Data Labeling" tab to create a data set like their documentation suggests, but it only reads one CSV at a time (I want to combine all the CSVs into one dataset), and none of the data types seem to fit at first glance, with the only one that sort of matches being "text". However, once it finishes its process, it looks like the resulting dataset is empty, when the original CSVs have hundreds of rows and dozens of columns. I've also tried using the "Tables" tab to create a data set, but it is giving me an error message about needing to put my models in the same region as my data, and I can't see how to fix this; the only options it seems to be giving me seem to be "Global" and "European Union", so I'm not certain how to set it to "Australia".

Does anyone have any ideas about how to fix this?

An anonymized version of the data sort of looks like this (but the actual CSV is much larger, with many more rows and columns):

  Item_ID,From_Position,To_Position,Data_One,Data_Two,Data_Three
  AA22DDA002,103,104.0,182.35643356,8.12345678,-2.32453256,0.03234213
  AA22DDA002,104.0,105.0,483.23424678,3.34627319,-4.37600000

Based on my experience with how finicky CSV can be, I strongly suspect an encoding problem. A lot of CSV readers simply throw-away the data if they encounter illegal characters or when they can't figure it out. I don't know what BigQuery does in that regard, but considering their help on loading CSV files it's worth a shot to make sure your data is clean and according to how BigQuery wants it before attempting anything else.

  Limitations

  When you load CSV data from Cloud Storage into BigQuery, note the following:

  CSV files do not support nested or repeated data.
  If you use gzip compression BigQuery cannot read the data in parallel. Loading compressed CSV data into BigQuery is slower than loading uncompressed data.
  You cannot include both compressed and uncompressed files in the same load job.
  When you load CSV or JSON data, values in DATE columns must use the dash (-) separator and the date must be in the following format: YYYY-MM-DD (year-month-day).
  When you load JSON or CSV data, values in TIMESTAMP columns must use a dash (-) separator for the date portion of the timestamp, and the date must be in the following format: YYYY-MM-DD (year-month-day). The hh:mm:ss (hour-minute-second) portion of the timestamp must use a colon (:) separator.

  CSV encoding

  BigQuery expects CSV data to be UTF-8 encoded. If you have CSV files with data encoded in ISO-8859-1 (also known as Latin-1) format, you should explicitly specify the encoding when you load your data so it can be converted to UTF-8.

  Delimiters in CSV files can be any ISO-8859-1 single-byte character. To use a character in the range 128-255, you must encode the character as UTF-8. BigQuery converts the string to ISO-8859-1 encoding and uses the first byte of the encoded string to split the data in its raw, binary state.

BigQuery expects UTF-8, you provide CP-1252. CP-1252 should, in my opinion, have died out by now. So either it only looks like such or you have a program that doesn't care much about following recent standards (those exist, I use one of them every day).

Also, make sure your localisation is set correctly. Some CSVs are separated by ; instead of , and some locales consider , a thousands separator while others see it as decimal separator. So please verify whether the locale of your CSV is the same as the locale of your schema.

Your default location can be set in the .bigqueryrc file. The schema's location can be set using the bq command, like this:

bq --location=location load \
--source_format=format \
project_id:dataset.table_name \
path_to_source \
schema

  Where:

  location is the name of your location. The --location flag is optional. For example, if you are using BigQuery in the Tokyo region, you can set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  format is NEWLINE_DELIMITED_JSON or CSV.
  project_id:dataset.table_name is your project ID.
  project_id:dataset.table_name is the dataset that contains the table into which you're loading data.
  project_id:dataset.table_name is the name of the table into which you're loading data.
  path_to_source is the location of the CSV or JSON data file on your local machine or in Cloud Storage.
  schema is the inline schema definition.

  Example:

  Enter the following command to load data from a local CSV file named myfile.csv into mydataset.mytable in your default project. The schema is manually specified inline.

bq load \
--source_format=CSV \
mydataset.mytable \
./myfile.csv \
qtr:STRING,sales:FLOAT,year:STRING

More on locations here.

Both your encoding and locale are suspect based on the details you have provided.


Confusion in backpropagation algorithm I have been trying to understand the backpropagation for a while now.
I have came across two variants of it.

In the Andrew Ng class the derivatives of the weights of hidden layers are calculated using the error signal that is distributed back to the hidden node.
In Geoffrey Hinton class the derivatives of the weights of hidden layers are calculated using the derivatives of the next layer that are already computed and from my knowledge of calculus that makes more sense.

Can someone explain how the first variant works?

The first variant is the second variant, or more accurately there is only one type of backpropagation, and that works with the gradients of a loss function with respect to parameters of the network.

This is not an uncommon point to have questions about though, the main issue that I see causing confusion is when the loss function has been cleverly constructed so that it works with the output layer activation function, and the derivative term is numerically $\hat{y} - y$, which looks the same as taking the linear error directly. People studying the code implementing a network like this can easily come to the conclusion that the initial gradient is in fact an initial error (and whilst these are numerically equal, they are different concepts, and in a generic neural network they don't have to be equal)

This situation applies for the following network architectures:

Mean squared error $\frac{1}{2N}\sum_{i=1}^N(\hat{y}_i - y_i)^2$ and linear output layer - note the multiplier $\frac{1}{2}$ is there to deliberately simplify the derivative.
Binary cross-entropy $\frac{-1}{N}\sum_{i=1}^Ny_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1-\hat{y}_i)$ and sigmoid output layer. The derivative of the loss neatly cancels out the derivative of the sigmoid, leaving you with gradient at the pre-transform stage of $\hat{y} - y$.
Multi-class logloss with one-hot encoding of true classes $\frac{-1}{N}\sum_{i=1}^N\mathbf{y}_i\cdot\text{log}(\hat{\mathbf{y}}_i)$ and softmax output layer. Again the derivative of the loss neatly cancels out, leaving you with gradient at the pre-transform stage of $\hat{y} - y$ for the true class.

So when you are told that backpropagation processes an "error signal" or "the error" backwards through the network, just mentally add "the gradient of" to the start of the phrase. Some people will say it knowingly as shorthand, others might be honestly confused.

The same applies to deeper layers, although then there is no other source for the confused "this is the error being distributed" other than as shorthand for "this is the [gradient of the] error being distributed".


In DQN, why not use target network to predict current state Q values? In DQN, why not use target network to predict current state Q values, and not only next state q values?
In doing a basic dq learning algorithm with nn from scratch, with replay memory, and minibatch gd, and I'm implementing target neural network to predict at every minibatch samples current and mext state q values, and in the end of minibatch, sync target network, bu I notice the weights diverge very easily, maybe because I used nn to predict current state q values? If yes, why not?
the agent code:
def start_episode_and_evaluate(self, discount_factor, learning_rate,
                                epsilon, epsilon_decay = 0.99, min_epsilon = 0.01):
        state = self.env.reset()
        done = False
        while not done:
            if np.random.uniform(0, 1) < epsilon: action = self.env.action_space.sample()
            else: action = np.argmax(self._nn.predict(state))
            next_state, reward, done, _ = self.env.step(action)
            self._replay_memory.put(state, action, reward, done, next_state)
            if len(self._replay_memory) >= self._batch_size:
                for state_exp, action_exp, reward_exp, done_exp, next_state_exp in self._replay_memory.get(batch_size=self._batch_size):
                    z, a = self._target_nn.forward_propagate(state_exp) #HERE I HAVE TO USE TARGET NETOWRK (_target_nn) OR THE MAIN NETWORK (_nn)?
                    q_values_target = np.copy(a[-1])
                    if done: q_values_target[action_exp] = reward_exp
                    else: q_values_target[action_exp] = reward_exp + discount_factor * np.max(self._target_nn.predict(next_state_exp))
                    self._nn.backpropagate(z, a, q_values_target, learning_rate)
                self._sync_target_nn_weights()
                epsilon *= epsilon_decay
                if epsilon < min_epsilon:
                    epsilon = min_epsilon
            state = next_state

source: https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project/blob/main/src/core/dqn_agent.py

Bellman equation for deterministic environments is given as follows
$$
V(s) = max_aR(s, a) + \gamma V(s')
$$
Where $V$ is a value function, $R$ is a reward function, $s$ is current state, $s'$ is next state, $a$ is an action. In DQN when optimizing $V$ it is assumed that $V(s')$ has on average lower error than $V(s)$. This is intuitively because $s'$ is closer to the end state for which $V$ is known and so there are fewer steps in which error can accumulate. This way we can progressively lower errors of $V$ for all states starting from the end states up to the starting state. I don't know how exactly you use current state to update weights but assuming it looks like this
$$
\delta V(s) := \alpha(max_aR(s, a) + \gamma V(s) - V(s))
$$
instead of
$$
\delta V(s) := \alpha(max_aR(s, a) + \gamma V(s') - V(s))
$$
i.e. you don't use $V(s')$ in the weights update what happens is the same error propagates over and over again because information about rewards in the later states is not propagated to earlier states.
As a concrete example imagine an environment with 3 states $s_1$, $s_2$ and $s_3$ where $s_1$ is starting state $s_3$ is the end state, the action space is only one move: go from state $s_i$ to state $s_{i+1}$ and reward function is 1 for moving to $s_3$ and 0 otherwise. What happens then is
$$
\delta V(s_3) = 0 \\
\delta V(s_2) = \alpha (max_a(R(s_2, a) + \gamma V(s_2) - V(s_2)) = \alpha (1 + \gamma V(s_2) - V(s_2)) \\
\delta V(s_1) = \alpha (max_a(R(s_1, a) + \gamma V(s_1) - V(s_1)) = \alpha (0 + \gamma V(s_1) - V(s_1))
$$
Because starting value for $V(s_1)$ is 0 it's gradient will always be 0 and the network will never learn the value of this state. That's why it must take a loook at values of its next state.


Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem between 0 and 500. Using transaction amount as a feature does improve target class separability. However, I can't help but wonder if there is a better way to use this variable. To explain, I care more about getting the high transaction amount values correct than I do the low ones. However, the model does not currently comprehend that. I have taught the XGBoost algorithm that the positive class is in effect more important by adjusting scale_pos_weight, but I haven't thought of a way to teach the algorithm that high transaction amount values are more important.
EDIT: I wanted to provide a bit more detail. After some additional reading, I think what I may be looking for is some kind of custom objective function. Possibly something like what is being discussed here.

Since you are using boosting models you can add something named monotonic constraints:
It is often the case in a modeling problem or project that the functional form of an acceptable model is constrained in some way. This may happen due to business considerations, or because of the type of scientific question being investigated. In some cases, where there is a very strong prior belief that the true relationship has some quality, constraints can be used to improve the predictive performance of the model.
I also recommend to read this post


(Labeled, if possible) time-series datasets for anomaly detection I would like to create a big list of available time-series datasets for anomaly detection. I'm especially interested in the following:

The time-series data should be segmented into cycles
Ideally, these cycles should be of the same length
These cycles should be labeled as normal/anomalous

But anything goes. I will be sharing the ones I found below.

Kaggle's NAB: a variety of sources such as AWS server metrics, Twitter volume, advertisement clicking metrics, traffic data, and more. Data is labeled.
Kaggle's Wafer: manufacturing data, 2K datapoints, 143 labeled anomalies. Measures are taken every 10 milliseconds.


How is the hidden state of a GRU initialized This is a GRU. Now, what will be the value of $h_t$, at $t$=$0$. That is, what will be the value of the hidden state at just the starting?

If you think intuitively what h_t (that is h_t-1) shoud represent is, very freely speaking, amount of rembering from the previous steps. Since you are just starting, and want to remember everything, very intuitive thing is to set the value at 1. Than you can see in the equations that the mutiplication with h_t-1 makes no difference in the beginning.


Constant Learning Rate for Gradient Decent Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step
of the gradient descent process.  What would be the impact of using a constant value for $\alpha_n$ in gradient descent?

Intuitively, if $\alpha$ is too large you may "shoot over" your target and end up bouncing around the search space without converging. If $\alpha$ is too small your convergence will be slow and you could end up stuck on a plateau or a local minimum.

That's why most learning rate schemes start with somewhat larger learning rates for quick gains and then reduce the learning rate gradually.


Prerequisites for Elements of Statistical Learning I am working through Elements of Statistical Learning, and unfortunately have found great difficulty in following the math. I have taken the standard series of Calculus courses (i.e., up to multi-variable calculus) and linear algebra, and have taken introductory (non-calculus-based) statistics and econometric courses.

Please  let me know if there are any courses or textbooks you would recommend in order to get my mathematics/statistics knowledge up to speed. For example, I am considering working through the relevant portions of Statistical Inference (Casella, Berger) as a prereq.

Thank you!

The authors of Elements of Statistical Learning have come out with a new book (Aug 2013) aimed at users without heavy math backgrounds.

An Introduction to Statistical Learning: with Applications in R

Download from here: http://faculty.marshall.usc.edu/gareth-james/


What is wrong with a neural network model which is so dependent on the seed of initialization? I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it is used for the test. There are two classes in the dataset which one of them has 75 samples and the other one has 249 samples. When I train the model with different seeds (actually when I re-run the training function), its results on the test set have a high variance, and the accuracy of the classification for test set changes about 20%. What is wrong with the model? How can I make it stable? How can I report the results of it?

One thing you can do is train your model N times and report the average and standard deviation of the accuracy.

Is the train/test split fixed? That means, do you use the same train set for every evaluation? And do you have a stratified split? That means the class imbalance of your data is also present in the train and test data. Since your data is imbalanced imagine you have a split where most of your train data consists of class 0, your model will learn to achieve a low train-loss with predicting 0 all the time, but you will have a high test-loss since you have way more samples of class 1 in your test set.

It probably helps to check your predictions. Fix seeds and stratify your splits.


How to build a mean prototype from data I have a dataset with physiological measures of subjects along time. I would like to create (or select) a mean prototype example in order to be able to identify in new examples how far are they from the mean prototype. A second issue will be to select a threshold to determine what is considered near or far.
Each example has 20 numeric features and I have around 300 examples per subject.

First ideas (disregarding outliers):

To iterate through all the examples of a subject and find the one who has the minimum mean distance to all the other examples. This will select a specific example from the dataset.
To use an evolutionary algorithm to find a prototypical example which has the lowest mean distance to all the other examples. This will create a new example that can be used as prototype.

Now I would like to determine when a new example is close, far or very far from the prototype (mean). A possible approach is to set two thresholds distance to determine to which class or case corresponds the new example (close, far or very far). How could I determine these thresholds? Possibly using number of standard deviations? What other approaches can be followed to perform all this?

Let's assume the distance metric is already selected.

The object with the lowest mean distance (= the object with the lowest distance sum) is known as the medoid and the basis of k-medoids algorithms such as PAM. Because you must not use k-means with arbitrary distances.


How to break down large SVM classification model? I have a classification problem with large number of classes: feature set is 512 Dimension, number of classes are around 3000. This is a face identification problem. (identify among 3000 celebrities, whose face it is. The feature is extracted using FaceNet.)

The issue for training such a SVM model is too slow:

I used sklearn SVC, it came with below result:

RAM usage: > 100GB // I eventually ended up using virtual memory of 100GB
Training time: > 30 hours
Classification time: > 1 hour per face
Other issue: Single CPU usage, no parallelization

To summarize, it is not practical to use above naive way of training for such a SVM model.

My Question: What is a practical way of optimizing SVM training / usage of this scale input data? (~ 3000 classes, feature_size = 512)

I think that using SVMs for your model is the main problem.

SVMs (linear or otherwise) are originally intended for binary classification. There are various procedures for extending them to a multi-class problem. The most common methods involve transforming the problem into a set of binary classification problems. The most common strategy is using "One-Versus-All" classifiers (OVA classification), and choosing the class which classifies the test data with the greatest margin. Another strategy is to build a set of One-Versus-One classifiers, and to choose the class that is selected by the most classifiers. Both these strategies are computationally expensive both in training and in testing.
I know there are other, more complicated methods, but I don't think that any of them is intended for such a high number of classes.

The most trivial way would be to use a neural-network model for such a problem. If you want to use one of the more classic algorithms, perhaps using decision trees will work (or even k-nearest neighbors).


Scaling features in artificial neural networks So it is a well known thing that it is a good idea to scale features/training samples in the training set, so that the values do not differ too much in the absolute sense. For example we want to train a neural network to learn a simple quadratic function y = x*x.

The training data, say, looks like x = [1, 2, 3, 4, 5, 6] for input data and y = [1, 4, 9, 16, 25, 36] for target data.

Obviously, one would scale both the input and the output data by something like x := x/6 and y := y/36, so all the training values live in the [0:1] range.

My question is, what if I then want to predict the output value for the input 7?
If I feed 7 / 6 into the network and then rescale the output by doing output := output * 36, I will not get the correct result. How to resolve such an issue?

As per your assumption, if you're scaling the features so that all of them are transformed between 0 and 1, the case where you're inputting 7 will result in a rescaled feature which will be greater than 1 (7/6 > 1). So, this tells us that your scaling technique of dividing inputs by a specific number isn't a really good scaling approach.

For your specific problem, scaling won't really be necessary, as explained here.
The important chunk says:

If the input variables are combined linearly, as in an MLP, then it is rarely
strictly necessary to standardize the inputs, at least in theory. The reason is
that any rescaling of an input vector can be effectively undone by changing the
corresponding weights and biases, leaving you with the exact same outputs as you had
before. However, there are a variety of practical reasons why standardizing the inputs
can make training faster and reduce the chances of getting stuck in local optima.
Also, weight decay and Bayesian estimation can be done more conveniently with
standardized inputs.

But if you're still interested in scaling the data, here's a nice discussion which can be explored to get an idea about standardized scaling/normalization techniques.


Multi-class classification metrics in R and Python I would like to ask if there is a metric in R and Python that serves not only for binary classification. I've found Matthews Correlation Coefficient works well in Python, but in R is only a binary version. Of course, I'm talking about "real" not just binary metrics - not something like "It's binary, but you can use one vs one or one vs all."

How about the MLmetrics package in R? It has multi class log loss function that you can use to assess the performance of your model.


SMOTE for regression I am looking into upsampling an imbalanced dataset for a regression problem (Numerical target variables) in python.

I attached paper and R package that implement SMOTE for regression, can anyone recommend a similar package in Python? Otherwise, what other methods can be use to upsample the numerical target variable?

SMOTE for Regression

smoteRegress: SMOTE algorithm for imbalanced regression problems

Update:

I found the following python library which implements Synthetic Minority Over-Sampling Technique for Regression with Gaussian Noise

smogn

I think SMOGN will work for your problem. The method is described in a paper titled: "SMOGN: a Pre-processing Approach for Imbalanced Regression". You can find it on arXiv. There is also a python implementation called "SMOGN" which can be installed through PyPI. You can find the package description at https://pypi.org/project/smogn/


Where can I learn the complete mathematics involved in LDA? I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems.

Where can I find good reading material which explains the concept in depth, especially by taking an example?

Welcome to DS.SE!

I believe the Latent Dirichlet Allocation paper by Blei, Ng, and Jordan (all big names in AI/ML/NLP) is a point to start. They provided the math foundation as well as multiple examples.

See: Blei, David M., Andrew Y. Ng, and Michael I. Jordan. "Latent dirichlet allocation." Journal of machine Learning research 3.Jan (2003): 993-1022.


Extracting specific information from multiple unstructured website I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (and content not always as sentences). Is there any better approach apart from writing custom web scraper/parser?

I tired using Named Entity Recognition and Part of speech tagging approach, but not sure if it helps in this case (or is there something about NER I am missing?)

Is there any better approach apart from writing custom web scraper/parser?

Probably not. Although this problem might be tractable using machine learning and natural language processing techniques in future, it would be far more work, and a much larger challenge, than the typical 80/20 rule you get from a web scraper plus regular expressions.

E.g. you write a web scraper with your best guess at the rules for extracting names. It finds 800 names correctly out of 1000. You analyse the problem sites, and add custom CSS selectors and tweak some of the regular expressions. That finds 150 of the remaining 200. Repeat the analysis and fix, you are down to maybe 10 problem sites, each requiring an individual fix. It is dull, repetitive work.

The alternative of attempting this problem using machine learning and NLP: First get the true labels for 10,000+ sites . . .


What is "noise" in observed data? I am reading pattern Recognition and machine learning by Bishop and in the chapter about probability, "noise in the observed data" is mentioned many times. I have read on the internet that noise refers to the inaccuracy while reading data but I am not sure whether it is correct. SO, what actually is noise in observed data? And what is additive noise and Gaussian noise ?

When you have sensors, the values you receive change even if the signal that was recorded didn't change. This is one example of noise.
When you have a model of the world, it abstracts from the real relationships by simplifying things which are not too important. To take into account for the simplification, you model the error as noise (e.g. in a Kalman filter).
But noise sources can be anything. For example, in an image classification problem, data compression can distort an image. Images can have different resolution; low resolution signals are harder to classify than high resolution figures. Aliasing effects can also distort images.

And what is additive noise?

Suppose your system equation is
$$z = H \cdot x$$
where $z \in \mathbb{R}^{n_m}$ is the observation, $x \in \mathbb{R}^{n_x}$ is the state you're interested in and $H \in \mathbb{R}^{n_m \cdot n_x}$ is a transformation matrix. Then the noise could interact with your system in any way. But most of the time it is logical and practical that the noise is additive, meaning your model is
$$z = H \cdot x + r$$
where $r$ is sampled from a random variable of any distribution.

What is (additive) Gaussian noise?

$$r \sim \mathcal{N}(\mu, \sigma^2)$$
See normal distribution


Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns? I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data.

Is there a way to include the new data without restarting the process like this?

model1 = initial variables
model2 = new variables
model3 = x*initial variables+y*new variables

As far I know, if you want to extend your model training, then number of feature should be same as previous one.

  In case you have more features, then you have two options:

  Drop those new features
  Retrain your model from scratch

  In case you have less features, then again you have two options:

  Place some values in missing features. Yo can use Impute classes, Refer Link for more information
  Retrain your model from scratch


Can I feed only images to simple CNN model without using label data? I have only image dataset. I want to take only feature map from simple CNN model so can i give only images to model without using label data? How to fit to madel only images?

To train the model, you need an objective to minimize. Very often, labels are used for this, for example, for an object recognition problem, to tell the model the kind of object in an image. You then train the CNN to encode features in the intermediate layers that are useful to solve this problem. The objective to minimize here has something to do with how often you identify the right object.

You can also train a model on an objective without labels, for example in an auto-encoder, where you use a CNN to find a representation of the input that has a lower dimension than the input, and then you typically use the same architecture but backwards to decode it back to the original input. The objective here is the similarity between the input and the output image.

A CNN trained this way, can be useful as a compression algorithm, because it encodes the image to some smaller representation in the middle layer of the auto-encoder, in such a way that you can rebuild the image to some level of accuracy.

So it's possible, but there is no reason to think that a CNN trained for an auto-encoder would be useful in an object recognition problem.


Many separation line using RBF kernel in SVM Below is my code, it take a range of a number, creates a new column label  that contains either -1  or 1.

In case the number is higher than 14000 , we label it with -1 (outlier)
In case the number is lower than 14000 , we label it with 1 (normal)

## Here I just import all the libraries and import the column with my dataset
## Yes, I am trying to find anomalies using only the data from one column

df['label'] = [-1 if x >= 14000 else 1 for x in df['data_numbers']]  #What I explained above

data = df.drop('label',axis=1)
target = df['label']
outliers = df[df['label']==-1]

outliers = outliers.drop('label',axis=1)

from sklearn.model_selection import train_test_split
train_data, test_data, train_target, test_target = train_test_split(data, target, train_size = 0.8)
train_data.shape

nu = outliers.shape[0] / target.shape[0]
print("nu", nu)

model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00005)
model.fit(train_data)

from sklearn import metrics
preds = model.predict(train_data)
targs = train_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
train_preds = preds

preds = model.predict(test_data)
targs = test_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
test_preds = preds

from mlxtend.plotting import plot_decision_regions                                 # as rbf svm is used hence lot's of  decision boundaries are drawn unlike one in linear SVM
# the top one central points with blue quares are outlietrs while at the bottom they are orangy triangles(normal values)
plot_decision_regions(np.array(train_data), np.array(train_target), model)
plt.show()

Output from training data

accuracy:  0.9050484526414505
precision:  0.9974137931034482
recall:  0.907095256762054
f1:  0.9501129131595154
area under curve (auc):  0.5876939698444417

Output from test data

accuracy:  0.9043451078462019
precision:  1.0
recall:  0.9040752351097179
f1:  0.9496213368455713
area under curve (auc):  0.9520376175548589

My graph seems to be having so many sepearation lines, I was thinking I would only be getting one that differentiates between the outliers and the normal data.

The OneClassSVM is an unsupervised algorithm that is supposed to learn the normal data distribution. This means that the algorithm will model the boundaries of areas of high likelihood for your data points to be drawn from.

Even though you know which points are outliers, the SVM will only know that there is roughly a proportion of outliers of nu, therefore the 14000 boundary is not intuitive at all in terms of distribution.

What the SVM will do that explains your multiple lines is:

If you have a hole in what you consider the "normal zone" - let's say as an example you have no data points between 2560 and 2985 - the SVM can decide that this is an area of low likelihood for your distribution and therefore build two vectors to exclude it from the learned normal distribution.
At the opposite, if you have several points clustered together above 14000, since the SVM has no clue this is supposed to be the anomalous zone, it can detect a zone of high likelihood for the data points to be drawn from, and build two vectors to include it in the learned normal distribution.

Now regarding your metrics:

You probably have an imbalanced dataset since we are talking about anomaly detection (otherwise it makes no sense!), so you should not use global accuracy here because it is misleading.
There is one thing that should alert you: The train AUC is very low, while the test AUC very high and this is not normal. So your algorithm poorly learns the target normal distribution.

The high test AUC can be explained, but first let's use some notations:

The real normal data distribution will be called *the target normal distribution", in the sense that you want to learn it. It is the interval [0;14000[
The real anomalous area (cannot talk about a distribution in this setting) will be called "the target anomaly area". It is the interval [14000;+infinity[
The learned normal distribution will be called the "learned normal distribution", obviously.
The learned anomalous area will be called the "learned anomalous area"

So the learned distribution consists in small clusters around areas of higher training data density. Since you (supposedly) have a few outliers and a lot of normal points, the learned distribution still intersects well with the target normal distribution, but not with the target anomalous area, where only a few areas will belong to the learned normal distribution. You can verify this on your graph: there are only a few vectors in the target anomalous area.

Now when you apply the algorithm on the test set, it is not likely that a point from the test set in the target anomalous area falls into the intersection of the learned normal distribution and the target anomalous area, because this intersection is very small. But this is only due to the small number of data points in this anomalous area, not to a successful learning process. Hence these points will still be labeled as anomalies and you get a misleading high AUC score.

At the contrary, the target normal distribution and the learned normal distribution intersects correctly, because there was more data points to learn from. So most of the normal points in the test set are classified correctly.

To sum up, your algorithm learns a completely different distribution from what you excpected. You can try to use a higher gamma value to smooth the kernels. but you will probably not avoid the classification of the areas of higher density above 14000 as normal data points. So watch out for the train AUC!

It would be very interesting to investigate these statements with graphs, you could as an example plot the [14000; +infinity[ zone, highlight the decision function boundaries and plot train and test set data points. You should get confirmation of what I am saying above (I will do it later and add the graphs if I find some time!)


Determining size of FC layer after Conv layer in PyTorch I am learning PyTorch and CNNs but am confused how the number
of inputs to the first FC layer after a Conv2D layer is calculated.
My network architecture is shown below, here is my reasoning using
the calculation as explained here.
The input images will have shape (1 x 28 x 28).
The first Conv layer has stride 1, padding 0, depth 6
and we use a (4 x 4) kernel. The output will thus be (6 x 24 x 24),
because the new volume is (28 - 4 + 2*0)/1.
Then we pool this with a (2 x 2) kernel and stride 2 so we get an output of (6 x 11 x 11),
because the new volume is (24 - 2)/2.
Same thing for the second Conv and pool layers, but this time with a (3 x 3) kernel in the Conv layer, resulting in (16 x 3 x 3) feature maps in the end.
My assumption would then be that the first linear layer should have 144 inputs (16 * 3 * 3),
but when I calculate the inputs programatically, I get 400. What did I miss?
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 4)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.fc1 = nn.Linear(400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, len(classes))

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features # 400, not 144

Related but less so: is there a reasoning used by people to get a good kernel size, number of layers
and number of pool layers or does everyone just look at what the SOTA papers do?

Hello and welcome to Stack Exchange!

The answer to your question is quite simple: you did not use the correct formula.

The formula you used is (assuming we are working with square inputs)

$$
W'=\frac{W-F+2P}{S}
$$

but the correct formula is

$$
W'=\frac{W-F+2P}{S}+1
$$

Now if we redo your calculations starting with $(1 \times 28 \times 28)$ inputs:

$$
W^{(1)}=28-4+1=25\\
W^{(2)}=\lfloor\frac{25-2}{2}+1\rfloor=12\\
W^{(3)}=12-3+1=10\\
W^{(4)}=\lfloor\frac{10-2}{2}+1\rfloor=5
$$

Considering that the second convolution layer has 16 output channels (or feature maps), you can indeed then calculate the number of inputs as $16\cdot5^2=400$.


LeNet-5 - combining feature maps in C3 layer Famous LeNet-5 architecture looks like this:

The output of layer S2 has dimension: 10x10x6 - so basically an image with 6 convultions applied to it to derive features.

If each dimension was again submitted to 6 filters the resulting output would be of 10x10x36 however it is 10x10x16. Initially I stumble on it but finnaly I udnerstood that this is done be combining inputs from layer S2 and applying one kernel on it as it's explained in the article:

  Layer C3 is a convolutional layer with 16 feature maps Each unit in
  each feature map is connected to several 5x5 neighborhoods at
  identical locations in a subset of S2s feature maps

.

  The rationale behind the connection scheme in table I is the
  following The 1rst six C3 feature maps take inputs from every
  contiguous subsets of three feature maps in S2. The next six take input
  from every contiguous subset of four. The next three take input from
  some discontinuous subsets of four Finally the last one takes input
  from all S2 feature maps Layer C3 has 1,516 trainable parameters and
  151,600 connections

and the roadmap of it is provided in the table:

What I am still not uderstand is how exactly should I combine them?

In previous layer I've just applied 6 kernels on 1 dimension, resulting in 6 dimensions what was understandable. Here I am a bit lost to be honest :(

Please help.
Mateusz


What is the geometrical representation of the y value in a 2-variable perceptron plot? This figure represents a perceptron model with a 2 dimensional feature vector input.

The hypothesis space of the perceptron is defined by this set:

$\{y | y = w\cdot x + b\}$

What is the geometrical representation of the y value in this perceptron plot? Is it the perpendicular distance from a point to the separating hyperplane?

in the context of binary classification, Y indicate the point at ($x_1,x_2$) belongs to positive(1) or negative(-1)

this piece of code is helpful to understand this

import numpy as np
import random
def generate_data(no_points):
    X = np.zeros(shape=(no_points, 2))
    Y = np.zeros(shape=no_points)
    for ii in range(no_points):
        X[ii][0] = random.randint(1,9)+0.5
        X[ii][1] = random.randint(1,9)+0.5
        Y[ii] = 1 if X[ii][0]+X[ii][1] >= 13 else -1
    return X, Y
X, Y = generate_data(100)
plt.scatter(X[:,0],X[:,1],c=Y)

in this figure, positive(1) class is shown as yellow, and negative(-1) class is shown as purple.


Given a 12x12 binary image (only black and white pixels) what is its dimensionality? And how can I define dimensionality of a data space? Suppose I have a grid 12x12 of pixels that can be only black or white. I can't understand if the dimensionality is 2 or 3. I mean... Is dimension given by 12x12 or 12x12x2 ?

There is some ambiguity in dimensionality you ask for. The vector space that your input lies in is $12\times12 = 144$ dimensional. So, if you're going to apply some kind of dimensionality reduction algorithm, e.g. PCA, this is the dimension. But, we sometimes tend to refer to dimension as the shape of the tensor, which is $2$ in this case. Some libraries might read the image into $12\times12\times3$ tensors, in which the third entry indicates RGB decomposition, even if the image is black and white. In this case, the tensor dimension is $3$.


What is the best way to compare various classification models? If I have two classifiers for example Neural network and support vector machine , now I want to know what would be the best way to identify which is good classifier , should it be based on classification accuracy or by analyzing confusion matrix or average precision score, f1 score etc.

I have seen in most of the papers related to image classification that they only take classification accuracy as the parameter to compare different models.

Since the problem you are trying to tacle is image classification, then classification accuracy is the appropriate measure of comparison. Also you could consider the Precision, Recall and F1 metrics (for multi-class problems). See below there is description of an extension about these multi-class metrics.

A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Another appropriate metric is AUC/ROC, which has to be extended cosnidering the multi-class case. See the link below

Receiver Operating Characteristic (ROC)


How is the weight matrix set and its dimension in CNN I am trying to calculate the output size of each layer and the number of parameters for 3 class classification using CNN. I have calculated till the final maxpooling layer and would really appreciate help in understanding how to reach the fully connected layer.

In Matlab I checked that the size for fully connected (FC) is 1176 and the weights are 3*1176 I am struggling to understand what these 2 numbers mean and how they have been calculated. I could guess that 3 comes from the number of classes but how did 1176 come? It does not match my calculation.

Question: How to determine the dimension of the last layer- FC? Since I have 3 classes, do I have 3 layers? Please correct me where wrong. Attached is the screenshot which shows the dimensions for the weight as 3*1176 .

The 1176 is the number of neurons in the layer before the last layer, which is the number of pixels in the convolutional layer but flattened (height*width*n_filters, i.e. 7*7*24=1176). The 3 comes from the number of neurons in the final layer, since you have 3 classes your final layer has 3 neurons, 1 for each class (which will output zero or one). This then gives a weight matrix of 3x1176 for the weights between the second to last layer and the last layer. Using torch we can show the dimensions of the data passed between the layers in the network. In the example below I have omitted the batch normalization and relu layers since they do not affect the dimensions. The dimensions are defined as follows: batch_size * n_channels * height * width.

import torch

x = torch.randn(1, 1, 28, 28)
x.shape
# torch.Size([1, 1, 28, 28])

conv1 = torch.nn.Conv2d(in_channels=1, out_channels=12, kernel_size=3, padding=1)(x)
conv1.shape
# torch.Size([1, 12, 28, 28])

pool1 = torch.nn.MaxPool2d(kernel_size=2, stride=2)(conv1)
pool1.shape
# torch.Size([1, 12, 14, 14])

conv2 = torch.nn.Conv2d(in_channels=12, out_channels=24, kernel_size=3, padding=1)(pool1)
conv2.shape
# torch.Size([1, 24, 14, 14])

pool2 = torch.nn.MaxPool2d(kernel_size=2, stride=2)(conv2)
pool2.shape
# torch.Size([1, 24, 7, 7])

pool2.view(-1).shape
# torch.Size([1176])


Low leves of probability observed after modelling.Is it right to scale the probability I have done modelling on imbalanced class , without any sampling methods.
Event rate is around 0.1 ,After modelling I am getting probalities more at the lower side close to zero.I have tried differnt models and an ensemble as well using caretlist in R.I am getting AUC around 79.I have rank ordered the probability and the maximum probability predicted is less than .5.

My question is that

1)Is there any thing i can try for getting the predicted probabilities ranging from 1(or in the higher level close to 1 eg : .7,.8 etc)

2)Is it right to transform the predicted probabilities between 0 and 1 by by applying min - max normalization and considering all probabilities above .5 have chance of winning and less than .5 have a chance of losing

Note:I have tried sampling methods like smote, rose and up sampling but resulted in low AUC because of the small number of observations.

Please let me know if any clarification is needed.
Any assistance would be really helpful.

It sometimes happens that a classification model will output probability estimates that are all in the low range. That means the model does not make any predictions that it is very sure are the positive class. Since only 10% of your data falls into the positive class, it appears to be a difficult problem to predict using your model.

Do not transform the probability estimates. When this happened to me multiple times, the predictions were still useful. For example, you may be interested in the highest 10% of predictions. You can always set your prediction threshold anywhere between 0 and 1.


Two different cost functions for neural networks, how they can give the same result? One is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K}\Big[y_{k}^{i}\log\big((h_{\theta}(x^{i}))_k\big)+(1-y_{k}^{i})\log\big(1-(h_{\theta}(x^{i}))_k\big)\Big]$$

The other one is: $$J=-\frac{1}{m}\sum_{i=1}^{m}\Big[y^{i}\log(a^{i})+(1-y^{i})\log(1-a^{i})\Big]$$

As I can see those two equations are not equal. How both can be used to calculate cost function?

Also, one of them using $h$ function which is $a$ of output layer, whereas others are using $a$ ($a$ is $f(w*x)$ where $f$ is activation function). When I looked from the book "Pattern Recognition and Machine Learning" from Bishop, he used $a$ for both of the equations. But from another source which I took equations from $h$ is used. But using different $a$ values and using just one of them (namely $h$ which is $a$ of output) are totally different things.

Both sources are reliable, what am I missing?

I don't remember exactly what the book has mentioned, but I guess the difference between the two is due to having one or multiple features. I guess it is already mentioned in the book. They are the same. One is for multi-dimensional input, and the other is for one-dimensional. One sigma is iterating over the features and the other is iterating over the examples. You can put $k$ to one to achieve the simpler formula.


How to find which patch in orignal image does an activation correspond to in vgg net after the final pooling layer So I am working on the NeurIPS 2019 reproducibility challenge, The link to the paper is https://arxiv.org/abs/1806.10574. So basically we have a vgg-16 net with the final fully-connected layers removed, so we get the 7x7x512  dimensional activation maps, Now the paper introduces a novel prototype layer in which we compare these 7x7 activation maps with 1x1x512 dimensional prototypes using the L2 norm, and wherever the prototype has the maximum similarity after considering all the input images we equate the prototype to that 1x1x512 patch in the activation maps

Now all is good and fine, but the paper further wants to find the patch in the original image corresponding to the 1x1 patch, so we can say that the prototype is effectively somewhat that patch in the original image.

Now what I found out is that the 224x224 to 7x7 transition can be effectively done by 212x212 kernel with 32 strides (using the effective local receptive field) image explaining this  (Not mine, copied the image from somewhere)

Okay so now if I get a 1x1 activation which I need to find the original image patch of ..its quite easy...but the problem is that in VGG net we apply zero-padding of 1 many time, so effectively we are getting a padding of 90(calculated using input size of 224, kernel size of 212 and stride of 32) in the original image , so suppose I get the top-left 1x1 activation as the one I want to find the patch of, then it will have a lot of portion = 0 due to the heavy effective padding in the original image, So is this okay ?

Sorry for the extremely long question, but it's a conceptual one and needed a deep explanation

Tracing activation through a neural net is complex. OpenAI Microscope helps visualize it.


Understanding of naive bayes: computing the conditional probabilities For a task on sentiment analysis, suppose we have some classes represented by $c$ and features $i$.

We can represent the conditional probability of each class as: $$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)}$$
where $w_i$ represents each feature and $c$ is the class we have. Then empirically, we can represent $$P(w_i|c) = \frac{n_{ci}}{n_c}$$
$$P(w_i) = \frac{n_{i}}{n}$$
Our priors for each classes are then given by: $$P(c) = \frac{n_c}{n}$$ where:

$n$ is the total number of features in all classes.

$n_{ci}$ represents the number of counts of that feature $i$ in class $c$.

$n_c$ is the total number of features for the class, and

$n_i$ is the total number of features for all classes.

Is my understanding of the above correct? So given these $P(c|w_i)$ probabilities for each word, I'm the naive bayes assumption is that the words are independent, so I simply multiply each word in a document for a certain class, i.e. to compute $\prod P(c|w_i), i \in N$ where $N$ is the number of words in the document. Is this correct?

To actually compute the conditional probability numerically, would it suffice to do the following:

$$P(c | w_i) = \frac{P(w_i|c) \cdot P(c)}{P(w_i)} = \frac{n_{ci}}{n_c} \cdot \frac{n_c}{n}\cdot \frac{n}{n_i} = \frac{n_{ci}}{n_i}$$

The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability.

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$.

Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$

where the second equation holds because of the naïve Bayes assumption.

For classification purposes you can ignore $P(w_1,\ldots,w_N)$ because it is constant (given the data). The formula is still simple ("naïve") but doesn't simplify quite as much.

  The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability.

Keep in mind that while Naïve Bayes is a decent classifier for many applications, the generated probabilities are usually not very representative.


Prove that the expected value of the training error is equal to generalization error for a class of binary clasiffiers Take $ H $ to be a class of binary classifiers ever a domain $X$. $D$ is a unknown distrbution over $X$, and $f$ is the target hypothesis in $H$. If we fix some $h$ $\in H$ we have the expected value of the training error equal to generalization error, that is:

$\mathbb{E}[L_S(h)]=L_{(D,f)}(h)$

Could you please come up with the proof of this statement?
Would you recommend another page inside stackexchange for these theoretical questions about machine learning?

Thank you.

Let us call $L=L_{(D,f)}(h) = \mathbb{P}_{x \sim D} [h(x) \neq f(x)]$.

We know that $L_S(h)= \frac{1}{m}|\{x\in S : h(x) \neq f(x)\}|$, where $m$ is the size of $S$.
Then $L_S(h)$ has values of the form $\frac{i}{m}$ with $0 \leq i \leq m$.

Also $\mathbb{P}[L_S(h)=\frac{i}{m}] = {m \choose i} L^i(1-L)^{m-i} $.

(Not sure why, but make sense since the probablility of taken $i$ $x$'s such that $h(x) \neq f(x)$ is precisely $L^i$, same thing with $(1-L)^{m-i}$ when $h(x) = f(x)$. Additionaly,
\begin{equation}
\label{1}
1=\sum_{i=0}^m {m \choose i} L^i(1-L)^{m-i}
\end{equation}

In this way:
\begin{align}
\mathbb{E}[L_S(h)] &= \sum_{i=0}^m {m \choose i} L^i(1-L)^{m-i}*\frac{i}{m} \\
                   &= \sum_{i=1}^m {m-1 \choose i-1} L^i(1-L)^{m-i}\\
                   &= L \sum_{i=0}^{m-1} {m-1 \choose i} L^i(1-L)^{m-1-i}\\
&=L
\end{align}


Is it a good idea to train with a feature which value will be fixed in future predictions? I am facing a regression problem and I have one feature that has some relevant correlation with the output. The value of this feature will be fixed in ALL the predictions I will use this model for.

Should I keep it in my model or not?

Thanks

Given that in your training data this feature has different values and some predictive power, I think not keeping this feature would be a mistake (without looking into overfitting due to having too many features). You cannot just discard the feature from your training set if it does influence the target because then these would be from a different population than your predictions and it will be able to learn from the other features.

Extreme example where x_2 will always be 5 in the future:

x_1  x_2  y
2    8    6
3    7    5
2.5  5    1.5
3    5    0.5

Just removing x_2 loses a lot of information and would create a significant bias towards higher targets.


Is it possible to change test and train data size when using crossvalind function with Kfold param? I was looking at MATLAB Help and want to work with "crossvalind" function.
It would two parameters that you can use it.

If you use "HoldOut" you can define partition size of test and train data set and when you use "Kfold" you can define "fold counts".

Now I want to know is there a way to use "Kfold" parameter and define partition size of test and train data sets?

I checked it and it seems when you use "Kfold" as bellow code, it always default partition size like 75% to 25% .

indices = crossvalind('Kfold',species,10);

K-fold means that the validation step will be performed k times, each of them using a fraction $\frac{k-1}{k}$ for training and $\frac{1}{k}$ for validation. If you want a fixed validation fraction, choose the number of folds that fits:

90%/10% : 10-fold
75%/25% : 4-fold
etc.

I don't know about Matlab libraries or functions, but if you want to do custom folds, you'll probably have to sample train and validations sets "by hand".


Which machine learning algorithm should I apply for differentiate question difficulty level with users' result Here's the scenario, There's a database with thousands of single-option questions for testing a specific skills, and a large number of users (either professional or amateur in this skill), each of which answer 10 random questions from the database.

The only thing that I can think of is to differentiate question difficulty level according to the correct rate of each question. But how could I take fully use of other informations like:

the correct rate from the user's perspective and feedback to its own influence to the difficulty level of questions (if user A answers 9 out of 10 questions correct, then the incorrect one (question_10) is more likely to be harder, than an user B answers 1 out of 10 questions correct and question_10 is in the incorrect set)
the answer time for each question by each user

Could anyone give me some ideas on this model, like where should I delve more into to make the difficulty level of question more accurate? Great thanks!

I suggest you look at item response theory and the Rasch model.  They construct a model that attempts to identify both (a) the proficiency/knowledge/ability of the user and (b) the level of difficulty of the question.


Dealing with a dataset with a mix of continuous and categorical variables How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the algorithm with categorical data converted using one-hot encoding?

For example, 2 columns are: Age and Race, Age being continuous and Race is categorical.

Update: Ideal buckets for a continuous variable is not known.

To clarify, you mean mixed variables in one column? e.g. ABC123

If yes, you create two additional columns: one with categorical and one with numerical values. Afterward, you can encode them (one hot encoding not always necessary).

Detailed explanation in Chapter 11 "Feature Engineering Mixed Variables":

https://www.udemy.com/feature-engineering-for-machine-learning/learn/v4/content


Clustering Customer Data I dont know if this kind of question is allowed but i kinda hit a wall. I know about some clustering algorithms. I already implemented Fuzzy C-Means and Gaussian Mixture Model, but I dont really know what's the efficient way to cluster customer data and there is no label at all.

Since it's company data I can't say the detail, but if this helps here is the columns of the data:

First of all, I group by using panda's for each customer data so there's no duplicate. And then i just soft cluster to 15 clusters. Why 15 ? I tried to cluster it with number of product categories as number of clusters (since this what my supervisor asked me unless i can propose better method).

Is this the right way to do it ? or there is some papers that explain better methods?

what I make will be used for real marketing on e-commerce so I'm scared if my method screws the company up or something.

The answer could be anything according to your data! As you can not post your data here, I propose to spend some time on EDA to visualize your data from various POVs and see how it looks like. My suggestions:

Use only price and quantity for a 2-d scatter plot of your customers. In this task you may need feature scaling if the scale of prices and quantities are much different.
In the plot above, you may use different markers and/or colors to mark category or customer (as one customer can have several entries)
Convert "date" feature to 3 features, namely, year, month and day. (Using Python modules you may also get the weekday which might be meaningful). Then apply dimensionality reduction methods and visualize your data to get some insight about it.
Convert date to an ordinal feature (earliest date becomes 0 or 1 and it increases by 1 for each day) and plot total sale for each customer as a time-series and see it. You may do the same for categories. These can also be plotted as cumulative time-series. This can also be done according to year and month.

All above are just supposed to give you insight about the data (sometimes this insight can give you a proper hint for the number of clusters). This insight sometimes determines the analysis approach as well.

If your time-series become very sparse then time-series analysis might not be the best option (you can make it more dense by increasing time-stamp e.g. weekly, monthly, yearly, etc.)

The idea in your comment is pretty nice. You can use this cumulative features and apply dimensionality reduction methods to (again) see the nature of your data. Do not limit to linear ones. Try nonlinear ones as well.

You may create a graph out of your data and try graph analysis as well. Each customer is a node, so is each product when each edge shows a purchase (directed from customer to product) and the weight of that edge is the price and/or quantity. Then you end up with a bipartite graph. Try some analysis on this graph and see if it helps.

Hope it helps and good luck!


How do attention mechanism in CNN for images? I read some attention mechanism papers, but I could not understand how it can be applied to an image (classification, detection, etc) using a CNN model. How does it affect the alignment scores and the weights?
I would like to understand the application of that mechanism on images.

I would suggest you refer to the paper by Hu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.

The brief idea is that the network learns the 'areas' to focus on that are on the feature maps (the last layer of the feature detectors) which can be in return mapped back to a certain location on the image.
It is not a complex topic but requires some knowledge in RNNs and LSTMs as well that help with sequential data. Best of luck.


Recommender system based on purchase history, not ratings I'm exploring options for recommender systems optimized for the insurance industry, which would take into account

i) product holdings

ii) user characteristics (segment, age, affluence, etc.).

I want to stress that

a) there are no product ratings available, thus collaborative filtering is not an option

b) recommended products don't have to be similar to products that have already been purchased, thus item-to-item recommendations are most probably not relevant.

Keep in mind that in insurance you rarely want to recommend similar products to those already purchased ones, as someone with the Car insurance is unlikely to want to buy another Motor product, rather Home or maybe Travel, etc.

That's why I want to develop recommendations on similarities between the users based on their purchase history and/or demographics

Ideally, I'd like to be able to implement it in R, if not possible, then in Python. Thanks for help and suggestions!

You could use Content based filtering but then you have to intelligently pre process the data to extract all the contents of the products. Also, that might lead to leaving a some features, This article is a great head start after you preprocess all the data.

Also, you could make pseudo ratings for product vs a customer. That would depend on your problem statement. Some few suggestions could be the number of times the customer bought the particular product in last one month or you could also take an index which would define how frequently the customer buys that product which, mathematically would be last_two_purchases/interval_of_purchase or could also take an average of last few purchases and intervals.

After making this pseudo rating you could convert all the content based features into numerical ones and use Latent factor model for collaborative filtering. Refer this video. Python could be used for this.


build biased image dataset for emotion analysis This is a pre-project question.
I would like to find or build a biased dataset to demonstrate what happens if training data are biased (biased distributed ethnicity for exemple).
I try this for the dataset (also this one) and Azure custom vision for the training.
My question is: if I remove one part of the images, would it be enough to bias the dataset and make the prediction correct only for subsets of the original dataset?

There is both subjective and objective approaches to removing bias ( de-baising techniques ) from the training datasets. It is observed that the sources of bias generally arise from the following data quality issues.

Over and under-sampling
Skewed samples
Feature choice/limited features
Proxies/redundant encodings
Biases and injustice in the primary data source

The essential steps for de-biasing require identifying the subspace of bias within the dataset. Then the key task is to neutralise and soften the bias if not completely equalising them.

There are interesting approaches for de-biasing image datasets. Deb-face is trying to address the problem in automated face recognition algorithms by constructing a de-biasing adversarial neural network that learns to extract disentangled feature representations for both unbiased face recognition and demographics estimation. Adversarial learning is adopted to minimise the correlation among feature factors so as to reduce the bias influence. Please refer to the approach paper in the following site.

DebFace - Debiasing Face Recognition


Is there any option to represent 160 discrete variables each one with their own color in scatter plot? Good morning,
I want to know if it is possible to represent with different colors in scatter plot more than 10 parameters instead of putting others and assign a color if the amount of parameters overpass 10. I was following the instructions on the web but the discrete variables are not working. I used the color icon and I assigned colors to all my discrete variables, but when I sent this information to scatter plot it didn't work. The program puts random colors ignoring the ones that I selected. I don´t know if there is any trick to make it work.
Is there any method to apply to represent 160 parameters each one with their own color?

I will be really grateful if someone can help me.
Thank you

The number of colours you can use in a scatter plot will vary depending on what software or programming language you are using.

The main issue is that scatter plots might not be the best option to use if you want to distinguish 160 different variables, particularly if you are only distinguishing them by colour.

If you can group the variables you might be able to make it work with less colours, or consider using a different type of graph.


What's the name of data points that trained model gets on input in production? I need some machine learning vocabulary advice.

What do you call input data that a trained model gets in production?

I know that labelled or unlabelled data it gets during training is called training set or training data.

When it's ready, after initial training, you deploy it in production to solve real problems. How do you call the input data that the model receives at that point?

I think you're talking about production data.  This is typically real data that you generate or acquire somehow, and it's the data you expect your trained model to classify or use to make predictions.

You mention that this is after initial training.  I'm assuming you've already used test data to measure your model's accuracy, and this is in fact now in production.


Differences between class_weight and scale_pos weight in LightGBM I have a very imbalanced dataset with the ratio of the positive samples to the negative samples being 1:496. The scoring metric is the f1 score and my desired model is LightGBM. I am using the sklearn implementation of LightGBM.

I have read the docs on the class_weight parameter in LightGBM:

  class_weight : dict, 'balanced' or None, optional (default=None) Weights associated with classes in the form {class_label: weight}. Use this parameter only for multi-class classification task; for binary classification task you may use is_unbalance or scale_pos_weight parameters. The 'balanced' mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)). If None, all classes are supposed to have weight one. Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified.

On using the class_weight parameter on my dataset, which is a binary classification problem, I got a much better score (0.7899) than when I used the recommended scale_pos_weight parameter (0.2388). Should I use the class_weight parameter or the scale_pos_weight parameter to balance the classes?

You can achieve the same results by using either class_weight, scale_pos_weight and is_unbalanced for binary classification on unbalanced dataset.
Setting
class_weight = {0: (number of negative samples / number of positive samples),
                1: (number of positive samples / number of negative samples)}

is the same as setting is_unbalance = True or scale_pos_weight = (no. of negative samples / number of positive samples).


Detecting features in XY Plots using CNNs I have a simple classification problem - if two features plotted on a simple XY plot show a "kink" or characteristic turn, then the label is TRUE, otherwise FALSE.  I've been attempting to detect the "kinks" in these plots using fully-connected NN's without much success.  I thought about converting my XY plots into images, and then feeding them into a CNN, but that seems to be fairly inefficient.  Does anyone have any advice for this situation?  A kink is circled below.  Thanks

I am a bit surprised that fully connected neural networks are unable to learn this. It might be due to one or more of the many possible problems like few data or overparameterisation.

Keeping this aside, your problem seems to be solvable by tree classifiers like CART, Random Forest, Gradient Boosting Machine or XGBoost.

The reason I think tree methods can work is that the problem can be solved using a decision tree. To detect the kinks one needs to find whether the datapoint is below the red line within some specific range of X and Y. Once the kink is detected it is easier to predict TRUE or FALSE as you mentioned.


Bert and SVM classification I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction methods such a bag of words. You can then use something like liblinear to obtain a SVM LibLinear model that is able to classify your data. On the other hand, you can build a model by concatenating Bert with a Dense layer. You can then fine tune this model and again, you obtain a classifier. Where would you use either one of them and why?

BERT is a transformer-based model. The pre-trained BERT can be used for two purposes:

Fine-tuning
Extracting embedding

You don't need to use an SVM once you're keyed into a BERT architecture. Your BERT model will generate embeddings and can be fine-tuned (ala ULMfit last layer) to perform a specific task. You could potentially just use the embeddings and then perform the task with another model but the performance would likely not be better.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.


Random Forrest Sklearn gives different accuracy for different target label encoding with same input features I'm using sklearn Random Forrest to train my model. With the same input features for the model I tried passing the target labels first with label_binarize to create one hot encodings of my target labels and second I tried using label_encoder to encode my target labels. In both cases I'm getting different accuracy score. Is there a specific reason why this is happening, as I'm just using a different method to encode the labels without changing any input features.

Yes. With y being a 1d array of integers (as after LabelEncoder), sklearn treats it as a multiclass classification problem. With y being a 2d binary array (as after LabelBinarizer), sklearn treats it as a multilabel problem.

Presumably, the multilabel model is predicting no labels for some of the rows.  (With your actual data not being multilabel, the sum of probabilities across all classes from the model will probably still be 1, so the model will never predict more than one class.  And if always exactly one class gets predicted, the accuracy score for the multiclass and multilabel models should be the same.)


Can anyone help with me how to train new data with already saved pickle file? Can anyone help me with the code me to train new data with already saved pickle file?
I've trained the model with RandomForestClassifier from sklearn and saved the model into .pickle
Now I'm trying to retrain the model on new data with same features. I want to use pickle file to train one new data. Can anyone help me with this in terms of code?

Load the pickle file and your new data. Fit the loaded model on the data
model = pickle.load(open(pickle_file,'rb'))
model.fit(x_new, y_new)