Possible for batch size of neural network to be too small? We know that when a batch size is too large, the model might not be able to converge. But what is the drawback of having a batch size be too small (say batch_size = 1) other than taking a long time to train? A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?

A batch size of 32 is commonly used and referred to as "small," but
  why don't we use a smaller batch size of 1 to guarantee convergence?

The issue is that a small batch size both helps and hurts convergence. Updating the weights based on a small batch will be more noisy. The noise can be good, helping by jerking out of local optima. However, the same noise and jerkiness will prevent the descent from fully converging to an optima at all.

It is a trade off between these factors, the best batch size will depend on the shape of the error manifold. Larger batch sizes are better on convex errors and smaller batch size are good on errors with lots of deeper local optima.


Data Mining Gear/Goods Websites for Specific Prices This is a bit of a stretch topic, but I find myself often browsing gear websites (like, Gear Patrol and Uncrate) for birthday and gift ideas.  I find it cumbersome to click through each and every article looking for items that are either within my price range or within my price range && category of item that I'm looking for.

I know that I can go to Amazon and set specific filters for items that I am looking for, but I enjoy receiving suggestions from websites that are known to give quality product recommendations.

From a data mining / web scraping standpoint, does anyone know of any resources of tutorials online that provide guidance for similar products?

Is R or Python better suited for web scraping applications?

Any insight into my idea would be greatly appreciated :)

I would look into the Python package BeautifulSoup. It parses HTML documents into a tree structure and allows for all kind of filtering and manipulation of the tree. I've taken a look at the Gear Patrol website and I would advice these steps:

Scrape the front page looking for articles. By taking a look using a web inspector like the one in Chrome (Ctrl+Shift+I) you can look at the HTML structure. Looking there we can see that the links to the articles are in:

body > div:content > div:home-this-week > div:posts hfeed > article > div:body-wrap

Now you look for all article tags and create a list of the corresponding URLs. You can use the link of the pagination to retrieve more article links or use their URL structure to make an educated guess where more articles are.

Now that we have a list of URLs, iterate over them and parse these pages in BeautifulSoup to retrieve the necessary information in a similar fashion. I advice taking breaks of a few seconds in between requests for new pages to not get banned from their server and not overly disrupt it. There are a lot of tutorials available how to do webscraping using BeautifulSoup and this site looks like an easy victim.


How to reshape ndarray to calculate the mean squared error? y_train.shape = (7654,1)
y_test.shape = (1914,1)

mae = mean_absolute_error(y_train, y_test) gives:
"ValueError: Found input variables with inconsistent numbers of samples: [7654, 1914]"

So I am trying to reshape y_train to (1,7654) and y_test to (1,1914). Would that work? I tried y_train = y_train.reshape(X.shape[1:]) but that doesn't work.

The issue is not the shape of your data. If you would like to calculate any kind of score for either your train or test predictions you need to compare the true labels to the respective predictions (either for the training data or the test data but not mixing these two):

y_train_pred = model.predict(X_train)
mae_train = mean_absolute_error(y_train, y_train_pred)

And correspondingly for your test data:

y_test_pred = model.predict(X_test)
mae_test = mean_absolute_error(y_test, y_test_pred)

(for the example I am assuming that your model object has been assigned to the variable model and used the predict() syntax of SKLearn here)


Add new attribute to the input of nerual network I have few students'(38) sequential data. For example, one student sequential data is like:

A
B
B
A
C
D
E
.
.

Each student has different length of sequences and their used letter was within this 5 letter. So, to predict the next letter/letters I used feed-forward neural network and recurrent-neural network(LSTM). I got reasonable accuracy with these models.

Now, I want to add a new attribute of the students to the input. Suppose, the students are from four different departments.
I added the one hot vector of a particular department to the corresponding student.

I used Leave one out as a cross-validation. But for this new input, the testing accuracy is always 100% which is not possible.

My question is, Are there any methods for adding a new attribute to the sequential data?

When adding an attribute, increase the number of neurons in the input layer, then train the network again.

Having 100% accuracy may indicate that the model suffers over-fitting. To solve the problem:

1- make sure that the model doesn't see the test set at training time

2- If you have sufficient number of records, I would recommend going for K-folds validations rather than leave-one out cross-validation (for example 70% training 30% testing)

3- if the number of records is insufficient, (such as in your case: 38 records at best won't allow you go deep), either

try to get more more data
or simplify your model (maybe just 1-2 layers with 1-3 neurons!)
or try machine learning methods other than neural networks. plz see (https://www.cs.sfu.ca/~jpei/publications/Sequence%20Classification.pdf).


When to use GAN over conventional sampling methods? Let's say I have a dataset from a diabetes hospital which has 30000 Type 2 diabetes and 300 Type 1 diabetes patients.
So this dataset has millions and millions of other data points like lab measurements, drugs prescribed and diagnosis data.
Now my objective is to build a model which can classify Type 2 and Type 1 diabetes patients.
As you can see that the dataset is highly imbalanced and I don't have enough T1DM patients to understand their patterns/behavior/feature which can help me differentiate them from T2DM.
So, my question now is when should I use sampling approaches like oversampling and when should I use GANs?
Should I select features of my interest and then apply oversampling or should I apply GAN?
update (addition to the above scenario)
Let's consider another scenario. I have a dataset which has only 300 T1DM patients (there are no T2DM patients). Now, I would like to just increase my dataset size. Let's also think that I don't have any ML task in my mind (meaing classification/ Regression etc) but I know that 300 samples are very less and can never be used for any meaningful analysis. So, now I would like to increase the dataset size and the use it for analysis. Here, GAN is the only solution for the synthetic data? Since no model is involved, I can't apply oversampling etc. can help me with this?

Here are the Options:

Oversampling- sure, there are some possibly good ones like SMOTE etc. Just apply it after Train test split to avoid leakage.

Undersampling - reducing the 30000 to a certain number where what is left is representative of the Information you Need to classify this class. You could, for example, apply some unsupervised learning to see which clusters inside this 30k are available and then only sample from These clusters until you have 300 examples. Or apply other undersampling techniques.

GANS- finally even tough really powerfull in certain Scenarios they are also VERY expensive. I would advise you to try GANS as your final Resort since it will take time for the Network to generate good examples.

CONCLUSION: Maybe you expected a decisive yes or no for GANS, but the truth About it is, its an Experiment. It might work, and it might not. Just like there are situations where NN are terrible.
After update:
Theoretically you can apply most of them without any labels, just mark These 300 Points as one label and see what you get. Ofcourse without a clean Goal in mind you could justify any Output as reasonable.
SMOTE has many variants. SMOTE should be treated as a conservative density estimation of the data, which makes the conservative assumption that the line segments between close neighbors of some class belong to the same class. Sampling from this rough, conservative density estimation absolutely makes sense, but does not work necessarily, depending on the distribution of the data.
There are more advanced variants of SMOTE carrying out more proper density estimation. Here is a repo with a lot of smote variations .
Here is also git repo for tabular GANS data Augmentation, this should be easy to consume


How to verify hand written signature? I trying to create a model for determining whether a questioned hand written signature matches known signature samples, and predict if the signature is genuine or forgeries.

I'm guessing I'll have to use a CNN and probably some variation of PCA.

How would you tackle the "signature verification" problem?
Is there an already trained models for that? or datasets?

All I could find are research articles, but:

no GitHub projects.
no AWS/Azure/Gcloud service.
no Kaggle completions or datasets.

Maybe I not searching for the right thing!

Any dataset of signatures I could use, and ideas on how to approach that kind of problem would be very helpful. tnx :)

One of the key features in hand-write is the frequency of the signature, i.e. high frequencies - how "shaky" is the signature, and low frequencies - how rounded is the signature etc.
For that, you would like to use Fourier transformation on the image, and train the results on labeled data, if available.


what are the steps in adaboosting? I went through adaboost tutorial and below are my simplified understanding:

Sample weight of equal value is given to all sample in dataset.
Stumps are created which uses only one feature from data set.
Using total error and sample weight stump importance is calculated.
Samples weights are changed i.e. samples which were predicted wrongly by stumps with high importance get sample weight increased and rest of the sample weight are decreased in the same order.
The process is repeated till the number of iterations mentioned where sample weight provides path for training.

Does adaboost contain multiple stumps with different splitting value for a single feature?
As mentioned, are the stumps created as first step or is it a continuous process?

Firstly, i want to notice that adaboost is a ensemble of stumps, and each stump is added to the ensemble sequentially, trying to compensate the errors of the existing ensemble.

Saying that:

In every step you have a new weighted dataset, and the Adaboost algorithm tries to fit a stump that splits the new dataset best. So yes, it is possible, and it does happen, that a single feature is used more than once (but the splitting value calculated by the stump should be different in order to reduce the ensemble's error.)
Stumps' creation is a continuous process, they are calculated in
every step for the new weighted dataset.


Get insights from Random forest::Variable Importance analysis I run variable importance on my Panel data (TV viewing over specific period) which consists of the old-Panel (Panel 0) and the new panel (Panel 1)

I am interested in understanding the differences in viewing based on the panel, as well as demographics like region. I used Avg daily total viewing in minutes as target variable and select age_group, region and big cities as predictor variables in R ( group by/repeat by panel ( 0 and 1) )

I visualized the result in R , in each chart’s Y-Axis is the prediction probability, X-Axis shows each variable’s values. How can we interpret the outcome to turn it to actionable insight?

In other words, what is the difference between plotting mean(prediction) vs. plotting mean(actual)? And how to make conclusion from it?

There is a number of different ways to view variable importance for random forests.
A great explanation of them

Variable importance measures

The most basic way is to look at how many times the variable was included in the trees in the random forest. This has the problem that it does not include how much of an effect the variable has on each tree
An improvement on this is to calculate the decrease in impurity that the variable cause in a tree when adding that variable to a tree. This is the default measure for the importance() function in the RandomForest R library. This has the problem of being biased towards continuous variables and ones with high carnality.
Another approach is the permutation importance. How this works is that it shuffles the variables in one column of the data and computes the decrease in accuracy. in R this is importance(type = 1). This is quite a good measure however it needs the features to be normalized to get the best accuracy (Shown in this paper). Its a little slower but still quite fast.
The last approach is the drop column approach. This works by dropping a feature then calculating how much the accuracy decreases from training the random forest on all the features. This is the slowest but most accurate.

Interpreting the results

These methods will all give a list of the feature with a measure of how good it is. This will give you an idea of how the performance of the random forest will change because of that feature.

Ie you might find Age is the most important feature for prediction viewing time. You can then look at a graph of age vs view time to see how those two variable interact.

With a drop column approach there is the possibility of finding that removing a feature will increase accuracy by the decrease in accuracy being an negative value.

For applying to the domain of the problem that is up to you and the real world problem that you are trying to solve with this model.

Hope this helps :)


Gunicorn workers timeout I'm using Flask where i load some pre-trained machine learning models once. I'm also using Gunicorn usually with 2 or 4 workers to handle parallel requests.

Every request contains some texts that i want to analyze.

I'll explain my problem with a example:
My Flask server with Gunicorn and 2 workers is up and loads my models once for every worker.
Then i send two parallel requests. The first will run analysis on the 1st worker with 500 texts and the second on the 2nd worker with 2000 texts.
The problem is that the second request will stop the analysis after some period of time and reload the models for this worker.

Does Gunicorn contains a default worker timeout and how it can be solved?

You can use the --timeout flag to increase the worker timeout.

Run gunicorn --help for more information about the available flags


Machine Learning: Take into account a variable if a condition is met (depending on another variable) I am working on a set of student data to train some models. I have the gender variable and I can also retrieve data about how many other girls go to this student's class. I would like to take into account in the modelling that if it is a girl, whether there are other girls in her class or not. This variable shouldn't have any interference in the cases where the student is a boy.

Any idea how I should treat this?

If you are interested to see if there are any other girls, you'll likely need to create a new indicator variable in your modelling data - perhaps something like:

other_girls_in_class = 1 if gender = female and there are other girls in the class
other_girls_in_class = 0 if gender = female and there are no other girls in the class

You need to treat these kind of approaches with care as it will implicitly contain some information on student gender.


What is the relationship between hard-sigmoid function and vanishing gradient descent problem? I think that the vanishing gradient problem occurs when the derivative of activation function is less than 1.
The deeper the neural network is, the f' * f' * f'... operation will result in the gradient closer to zero.

However, if we use hard-sigmoid as an activation function (its derivative is 0 or 1), the vanishing gradient problem can be solved.

I think the dying problem (derivative = 0 @ ReLU) is different from the vanishing gradient problem.

Is it right?

The vanishing gradient problem appears with activation functions that squash their input in a very small output range, like sigmoids and tanh. The input range is large and therefore the probability for it to fall in the squashing part of the sigmoid is very high. Chain several sigmoids and there you have vanishing gradients,

Typical examples of such a type of activations are sigmoid and hyperbolic tangent. Hard sigmoids are no exception. Their main difference with normal sigmoids is that they are computationally cheaper.

ReLU's are different. Half of their range is linear. That's why they don't suffer from vanishing gradients. However, they have their own problem: dying ReLU. This happens when the input falls far in the left side and cannot escape.


What's a classifier capable of predicting a variable number of classes I want to solve what I understand as a classification problem regarding tagging. Let's say an Entity can have 0 or more tags and I want to be able to predict which tags (if any) an entity might get

I would like the classifier to output 0 or more tags.

I have never run into this kind of structure with variable length in input data (many tags) and output (0, one or more tags for each entity) before and I'm not even sure what terminology / words to search for.

This is an interesting problem. What is your xtrain? I guess it boils down to a multi label problem. In a simple setup, you would train on the presence of a certain label (present or not). So for n tags/labels, you would train n+1 (1 is no label) models.

I‘m not really in to multilabel problems, but I guess this is the way to go.


Emotional tension score in sentences I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence transmits wish, hate, or fear. I looked at some studies on sentiment analysis, but haven't seen any relevant result. Most of the them are on negativity or positivity of a sentence.

Consider multi-label classification, instead of a binary sentiment your dataset would have a level/degree/probability target for each emotional label of whatever dimension size you want.

In the context of neural networks, it's just a matter of replacing softmax with sigmoid loss layer.

If getting such large dataset with desired supervised labels is hard or not possible, one idea may be to use models trained on any sentiment datasets out there (there are even some multi-dimensional too) and apply additional un(semi)-supervised learning techniques to extract what you want.


Predicting credit applications with timeseries I was wondering what the best way is to make a model for predicting credit applications.

I have two tables, which look like this:

   client_nr  yearmonth  total_nr_trx  nr_debit_trx  volume_debit_trx  ... etc.
1          1     201201            94            49           6527529
2          1     201202            85            58           3475518
3          1     201203            94            61          31317405
4          1     201204            85            52          18869967
5          1     201205            93            53           2893105

  client_nr  yearmonth  credit_application  nr_credit_applications
1          1     201201                   0                       0
2          1     201202                   0                       0
3          1     201203                   0                       0
4          1     201204                   1                       1
5          1     201205                   0                       1

The goal is to determine which clients are likely to apply for a credit. So far, I have made sequences of size (months, features) for each client,
using the first table.

My questions:

Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?
Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?
Would it be better to use regression using nr_credit_applications, or classification on credit_application?

Intuitively I think that the model would need some additional features based on the economic context in order to be more accurate, and that would also be a part where the evolution across time really matters.

  Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?

If possible you need to have a full time series for a client as an instance, so I'd say splitting on the client number is much better. But select the client numbers randomly from the full set of client ids, because it's possible that the clients number are assigned following a particular order in time.

  Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?

That depends on the exact goal and on the kind of algorithm being used, but afaik usually with a time series the label is predicted at any given time given the past and current features and the past labels (or past predicted labels).

  Would it be better to use regression using nr_credit_applications, or classification on credit_application?

This is more a matter of convenience for your application or for the algorithm being used, as the accuracy should be very similar (assuming you use the same kind of method of course).


How to define a multi-dimensional neural network with keras I have implemented a simple neural network with keras that takes an input of 50 values and returns a classification of '0' or '1'.  I believe the model is expecting an input shape of (50, 1).  I'd like to add another 50 data values for each input, but I'd like them to be associated with the original 50 respective inputs.  So instead of making the input of shape (100, 1), I guess I'd like to make it of shape (50, 2).  I would like the neural network to know from the start that each input feature has two values associated with it, instead of it thinking there are 100 separate input features.  Here's what I have so far:

model = Sequential()
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(100, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Can anyone show me the way the alter this structure to accept my new input shape?

If you want to use a simple neural network which takes in a 50x2 matrix you can do this y flattening your matrix to a 100x1 vector. This will make no difference in the way that the neurons are processed. Each neuron will take all the inputs and make a decision on its weight accordingly.

If you want some neighborhood feature selection then you can use a convolutional neural network like

model = Sequential()
model.add(Conv2D(32, kernel_size=(2, 2),
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))
model.add(Conv2D(64, (2, 2), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(1, activation='softmax'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

This will do a feature mixing within a 2x2 window across your matrix which will thus mix your two values before feeding them to the deeper part of the network.

Doing this is also a good idea as it serves as a means to reduce the complexity of your model which may lead to better results.


What to report in the build model, asses model and evaluate results steps of CRISP-DM? I would greatly appreciate if you could let me know what to report in the following steps of CRISP-DM?

Build Model: what should be reported for parameter settings, models and model description? I used grid search to tune hyperparameters.
Assess Model: what should be reported for model assessment and revised parameter settings?
Evaluate Results: what should be reported for assessment of data mining results?

In fact, I used just Logistic regression to do a classification task using a procedure like what is depicted below:

Could the final model evaluation on test data set be considered as a part of Evaluate Results or it should be done on new data?

The way you are trying to present the outcome is pretty good.

I cannot say that the following procedure is the standard procedure in my scenario I did something like this:

This is how I presented to my managers to make them understand the procedure which I followed.

I made a slide for each and every segment and highlighted the points which are necessary for them to be known.

While showing the results, I named the data(accuracies) differently

Train Set Accuracy: Which is 70% of the total dataset, derive the accuracy % also called model accuracy
Test Set Accuracy: Test set which is 30% of the total dataset, derive the accuracy called Test Accuracy
Blind Test Accuracy: Which is completely new data, here I trained the model with whole dataset and then tested using the new data(you can also call it as Validation set)

This is how I presented:

Since yours is also a classification problem to explain it better I gave them the breakdown:

Do let me know if you have any issues, would love to help you.


What is the difference between data analysis and machine learning? It seems to me that machine learning (especially deep learning) can work with thousands (even millions) of different inputs. After training an ML model on the inputs, the computer has hopefully "learned" something about the connections between these millions of inputs, and can hopefully make predictions regarding new unseen inputs.

What is the difference between machine learning and data analysis?

For me, data analysis represents something like....

What is an insightful KPI to get out of our data? If we are an
eComm retailer, for example, that might be what % of page visits did
we have to pages which we still had in stock?    or
What percentage of orders are dispatched within 2 days?

Machine learning, is a whole different game. You might re-state Q1 and Q2, for example:

Can we build a predictive algorithm that predicts what pages
will be out of stock?
Can we build an optimizer which helps us to dispatch orders faster?

In some deeper level, data analysis you are typically using the machine to automate some fairly trivial tasks. With machine learning, you are letting the machine automate a lot of the learning process - where you would be typically passing in that knowledge as a data analyst.

Does that help clarify at all?


Data.frame vs Data.table in R? Data.table package is claimed to be faster than Data.frame. what are the implementation changes that made this possible? How can one leverage the power of this package for data analysis?

You have a nice free course about it on DataCamp

  Data Analysis in R, the data.table Way

The author of the course is Matt Dowle, the guy that actually wrote the data.table package.

Hope that helps.


RBF kernel can classify two classes as in figure? As you can see, I have some points (belonging to red and blue class), and I would to use an RBF kernel but I think that an RBF kernel can make points linearly separable only if they are located in perfect circular way. In this case I don't know how modify the kernel (or which parameters use) for respect the "oval" aspect that these data have.

Your understanding that an RBF kernel can make points linearly separable only if they are located in perfect circular way is not correct. For your dataset, is very easy to use the RBF kernel to separate the two classes.
For simplicity let's assume you are using a SVM classifier.
It is true that the RBF kernel looks spherical, not elliptical. However, the decision boundary is also determined by the support vectors. Unless you only have one support vector for one of the classes, the decision boundary is usually NOT spherical.
You can easily visualize the decision boundary given by SVM. For example, I simply modified the code here (using sklearn.svm) to use your dataset, and set C=10, the result looks like:

So a more general question is:
How does the "shape" of kernel affect the shape of the decision boundary?
We can visualize the effect with some self-defined kernels. In the figure below, in the upper row we plot the shape of the kernels used (same color denotes the same kernel value), while the lower row shows the SVM decision boundary with that kernel.

So we can see that the "shape" of the kernel does affect the shape of the decision boundary, but only its details. The overall shape is still determined by the actual supported vectors. BTW in this case all 17 data points are supported vectors.


How to handle "unknown" category in machine learning classification problems? Tutorial problems come in the form of binary or mult-class classification where data are all properly labelled. In real-life applications, there are incoming data that do not belong to any category and cannot be classified. How can we handle these data which fall into "unknown" category? The universe of "unknown" can be far more than "known". So, the data for "unknown" can be too much and lead to class imbalance. How do we train the model to deal with "unknown" data? Or do we ignore it?

I think this is one of those topics with the most frustrating answer - it depends.

To your questions:

How can we handle these data which fall into "unknown" category?

There are many ways of doing this. Some are very simple and some are more complex, but they all depend on you understanding your data and what exactly causes the missingness - e.g. is the data missing at random or is there a specific cause driving it?

Some techniques to treat missing values (in increasing order of complexity):

Exclude all missing values. This may be fine if you have a large amount of data and few missing values (not always the case - you allude to this by mentioning the remaining data set may be imbalanced).
Replace / group the missing values with an appropriate value - e.g. replace missing values with the mean of the variable / group missings with the most populous level.
Impute missing values using models / equations - e.g. Multivariate Imputation by Chained Equations (MICE).

How do we train the model to deal with "unknown" data?

This depends on the model or technique you are using. Some techniques deal with missing values well (e.g. xgboost), while others do not (e.g. R's ranger implementation of random forests). You should take into account the model you are using when deciding on how to treat your missing values.

Or do we ignore it?

Ignore missing values at your peril!

Hope that helps!


Unable to make accurate predictions? I have a dataset of diabetes patients and I am trying to predict the next blood glucose level. I have attached an image below and I have about 1600 records in that csv file containing data of 10 patients. Each patient is uniquely identified by the Id column and the Glucose_t-1 means the glucose value in which the patient had before the current reading(Glucose_t) likewise this apply to Glucose_t-2 and Glucose_t-3. And same applies to the Insulin_t-1,Insulin_t-2. The event column is the Glycemic event in which the current reading of the blood glucose value falls to. For example,

blood glucose value <= 70 then 0,
70< blood glucose value <=180 then 1,
blood glucose value > 180 then 2.

I have applied different regression algorithms like Logisitic Regression, Random Forest Regression and so on but I was unable to predict the Glucose_t value accurately. The accuracy comes to 0.008.. which is very depressing :(. Please any help to improve the accuracy of will be greatly appreciated. Thanks.

It's a quite complex problem and there might be better options for the design (I'm thinking maybe something more specific to times series)...

However before that there's a more obvious problem to solve: it seems that you are calculating accuracy on the "Glucose_t" numeric value, right? If yes this is incorrect and that would explain your terrible results:

Accuracy is an evaluation measure for classification tasks, not for regression tasks: accuracy simply checks for each instance whether the predicted value (which is supposed to be a categorical value) is the same as the gold true value, and then divides the number of correct cases by the total. Naturally it's very hard to predict the exact true value in the case of numbers: if the true value is 183 but the algorithm predicts 184, then accuracy would count that as incorrect even though it's very close. That would explain why your accuracy is very low.
Typical regression evaluation measures are mean absolute error and mean squared errors. These measures (and their variants) are designed to calculate how far the predicted numerical value is from the true value. If you use these keep in mind that it's an error score, so the lower the value the better.
If you want to use accuracy, it would make sense to use it on your "event" value (which is derived from your "glucose_t" prediction I assume?): in this case you have 3 classes (categorical values), and I bet the accuracy will be much better. Note that accuracy can be biased by class imbalance, so don't forget to check the confusion matrix.


How do you ascertain the quality of your data? How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such reason is we need to meet deadlines for reports & publications. I am wondering if anyone has adopted or come across a method/guidelines that help to find issues within the data (time-saving tips), so we can analyze the data efficiently. Please share your experience, tips, etc.

This entirely depends on what you wish or aim to do with your data, and what you mean by your tag data-cleaning. This can involve the technical process of sanitising data i.e. fixing broken XML, but also the manual deletion of outliers etc.

Are you going to perform statistics to measure the correlation between certain variables?

Or:
Are you going to include the data into a machine-learning solution?

In any case, you can plot descriptive statistics to get an impression of the quality of your data, such as:

Box-plots to detect outliers
Standard deviation, mean, mode, median to assess if it's normally distributed or not
More, don't know from the top of my head

Remember that those measures give you an impression of the quality of your data, but even good measurements can come from bad data, and possibly vice versa.

In short, the best way of assessing the quality of your data is to assess the quality of your results, which are acquired generally after several experiments. Only then will you see how well your data is suited for the problem you are trying to solve. That - or you need to get your hands dirty and examine the data qualitatively before you use it.

Data (in assumingly large quantities) are generally used to identify certain patterns within the phenomena described by the data. It is very important to describe your task in terms of a (research) question and a proposed solution or hypothesis. Then, you can assess whether the data is suitable for your particular (research) project. Keep in mind that, theoretically, you can try to answer numerous questions on the same data, or answer the same question on different data sets. The goal is the right balance.


Categorical variable for Arima Does Arima support the usage of categorical variable? Some ways to get it working can be using one-hot encoding to represent categorical variables, but I am not sure how good it is.

One-hot encoding is the way. For ARIMA models, no other configuration is possible. Whether it is good or not, it depends on the quality of your data, your variable choice, and a correct model specification. Unfortuntately, we can't say a priori whether the result is going to be good or bad.


Multiclassification with large number of labels I am attempting to build a classifier with a large input space of one hot encoded vectors. The output should be a vector of labels, with 10000 possible labels each. For example, the labels could correspond to something like: (Class 10, Class 992, Class 800, Class 23, Class 30) etc. My first attempt was to one hot encode this output as well, resulting in an output space of size (10000, 5). However, I quickly discovered that to have an output space this large, I would easily exceed the memory on my GPU (16gb).
Question: What is the best way to encode and implement such an output space in PyTorch?
My first thought was to drop the one hot encoding and encode the outputs exactly as I did above, dropping the word class obviously so that the output would be a 5 dimensional vector (10, 992, 800, 23, 30). However, I wasn't sure which loss function to use here: obviously MSE or other such regression losses would not be appropriate. After doing some research, I found someone who recommended using TensorFlow's softmax_cross_entropy_with_logits (sorry I can't seem to find where I originally saw this anymore).
Using https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/59197990#59197990, I was able to implement a criterion in PyTorch after removing the masked parts, that would accept tensors of size (batch_size, 5) for both labels and outputs and return a number.
However, I'm not sure I'm using this correctly. Attempting to pass the same arguments into the TensorFlow version that this is supposed to be equivalent to throws dimension errors. Also, testing this on some fabricated instances don't seem to match up with what my intuition says it should:
label = torch.tensor([[8.0], [8.0]]) # Corresponds to a two instances of class 8
out = torch.tensor([[7.0], [8.0]]) # Corresponds one incorrect prediction
criterion(out, label) # Expecting a nonzero loss because the first prediction was incorrect
# However, returns tensor(0.)

It is not behaving as expected even with a single label, let alone five. I am thinking with five, if I can get the loss function working for one, to do a balanced sum of each individual.

I think your understanding of the output vector is not correct (Not just for Neural Network but any Model)
We don't encode the output to reduce the dimension. Output has not much contribution on the RAM and computation, it's the input size and dimension. Simple OHE Or just the label encoding will work(Loss function will change as per the case)
What you should decide first-
 Is this a Multi-class problem or Multi-label
 This will decide your Encoding i.e. [1, 1, 0, 0, 1] Or [1, 0, 0, 0, 0]
 This will decide the last layer's Activation Function and the Model's Loss function
It is obvious that for such a high output dimension, you will need a very large input dataset with a good variance over all the 10K class.
To reduce output dimension -
You may use any clustering technique Or your domain knowledge to club different classes.
And then, you should plan about Torch and TensorFlow


Is there any difference between 'classification' and 'categorization' based on machine learning terminology? When I was learning about classification models, it came to my mind that if there is any difference between "categories" and "classes" on the basis of machine learning terminology? If there is no difference then why do we say classification models, why not categorization models?

As noted by @Emre in the comments:

"Same thing. Just call it classification; it's the more common name for the task."

Although there is no difference with respect to machine learning, this paper may provide another perspective in detail.


How to split a dataset into train and test sets for time series (multiple step-multiple output forecasting)? I am trying to use an LSTM neural net to do multiple step / multiple output forecasting (I predict multiple values in one time knowing some values in the past). But, I have realized that I must be careful when splitting my univariate series into training and test sets. In this post, I am going to share what must be done in my opinion, but I really would like to have yours.

$(X_k)_{1\leq k\leq N}$ denotes my univariate time serie.

So basically, multiple output models are complex as they are trained to learn the dependencies between inputs and outputs as well as between outputs. This can be a problem as I transform my time serie like that :

$ D = (x=(X_k, ..., X_{k+p-1}) , y=(X_{k+p},..., X_{k+p+n-1}))_k $
with $p$ my number of steps and $n$ my number of outputs ($1 \leq k\leq N + 1 - (n+p)$). I want to split it into training and test sets. So, I must get :

$ D_{\text{train}} = (x=(X_j, ..., X_{j+p-1}) , y=(X_{j+p},..., X_{j+p+n-1}))_j $ where $j \leq N  - 2(n+1) - p$

$ D_{\text{test}} = (x=(X_i, ..., X_{i+p-1}) , y=(X_{i+p},..., X_{i+p+n-1}))_i $ where $i+p+n-1=N$

In this case, I can avoid overfitting due to a naive splitting of my dataset $D = D_{\text{train}} + D_{\text{test}}$ where $+$ denotes a concatenation operator (preserving the timeline). Indeed, this would lead to over-fitting in my opinion as I would have in my train some part of my test.

For a better understanding, let's take an example :
My univariate series is denoted as $(X_1,...,X_6)$. I want to predict $n=2$ values in the future knowing $p=2$ values of the nearest past. So, I transform my dataset as:

$ D = (x=[(X_1,X_2),(X_2,X_3), (X_3,X_4)] , y=[(X_3,X_4),(X_4,X_5), (X_5,X_6)])$

I split it naively, I get:

$ D_{\text{train}} = (x=[(X_1,X_2),(X_2,X_3)] , y=[(X_3,X_4)])$

$ D_{\text{test}} = (x=[(X_3,X_4)], y=[(X_4, X_5)]) $

I am afraid that it will lead to overfitting as the data $(X_3, X_4)$ is in both datasets. Maybe I am wrong, that's I want to know :)

Would you like to tell me if I am wrong and what do you think about?
Thanks.

** Edits are written in bold **

Assuming you have sales data for 36 month and you want to forecast for next 6 or 9
months. Lets create the train and test data at 70 and 30.

  train data= 36*70/100= 26(rounding value of 25.2)
  test data= 10(leftout sample)

Now we'll build a model on train data and forecast values on test data(those 10 points), to measure model performance we use MAE or MSE, always MAE or MSE values should be less. Once you model is doing good then forecast for future 6 to 9 months. i.e multiple output in one time prediction.


Testing unsupervised clustering Assume we train a KMeans model using data X. This will give a set of centroids that can be used to cluster data X* using a Nearest Centroid Classifier.

If we use a density-based model such as DBSCAN to train a model using data X, how can we use it to classify future data X*? Is KNN suitable?

Thanks!

General speaking, you can assign the cluster to the new value using the method of that clustering method. For example, if you are using the DBSACAN algorithm, you can assign the cluster to the new value, if it is reachable from one core points of a cluster (in a tie situation, we can assign the more density cluster or many other methods).


One Neural network with multiple outputs or multiple neural networks with a single output? I an building a feed forward deep learning model using tabular data. The inputs are numeric features or categorical features (represented with embeddings). The outputs are the same number of numeric input features.
Is there any known research or models out there which verifies that using a single model with multiple outputs would be better/worse than multiple models, each with a single output?
In essence, with N observations and M outputs, a single model minimizes:
$$
\frac{1}{N}\sum_n^N\sum_m^M \left(y_m^{(n)} - \hat{y}_m^{(n)} \right)^2
$$
while multiple models with single output, each minimize:
$$
\frac{1}{N}\sum_n^N \left(y_m^{(n)} - \hat{y}_m^{(n)} \right)^2
$$
For a single value of $m$.
Any reason one would be preferred over the other, or do I just have to try and see for myself?

Given the information you provided, the most honest answer is: You have to test it by yourself, there is no general answer for it.
Still, it has been shown empirically in research that a neural network may benefit from having multiple outputs.
So let's say we have a neural network that has multiple outputs. Further, let us group them into specific tasks:
For example:

The output neurons of group 1 tell if the image containts a dog or a cat.
The output neurons of group 2 tell the size of the animal (width and height)
The output neurons of group 3 tell the color of the animal's hair (in some encoding)

and so on...
A common example would be Faster-RCNN vs Mask RCNN.
Assume that $g$ denotes the number of different groups of output neurons.
Now if you take a feed-forward neural network, you will have common layers that eventually branch to the different output groups. Let us call $\pi$ the function that maps an input image to this particular last common layer $L$ and let $\phi_{j}$ be the function that takes the information from layer $L$ to output the result of group $j$.
Thus, given an input image  $\mathbf{I}$, the neural network maps it to $\begin{pmatrix} \phi_{1}(\pi(\mathbf{I})) \\ \vdots \\ \phi_{g}(\pi(\mathbf{I})) \end{pmatrix}$.
The output of the last common layer $\pi(\mathbf{I})=:\mathbf{f}$ can be understood as an image descriptor $\mathbf{f}$ of the input image $\mathbf{I}$.
In particular, all predicted outputs rely on the information contained in $\mathbf{f}$.
$\textbf{Therefore}$: Merging multiple outputs into a single neural network can be understood as a regularization technique. The image descriptor $\mathbf{f}$ must contain not only the information if the images shows a dog or a cat, but also all the other information. It must therefore be a more comprehensive (or "more realistic") description of the input, which makes it more difficult for the network to overfit. The network cannot solve a specific task using a non-plausible explanation, as the corresponding image descriptor would lead to bad results  on the other tasks.
As a consequence adding additional (auxiliary) tasks to the neural network can improve the accuracy on the initial task, even if you are not interested in predicting these additional tasks.
So essentially, if there is a common description of your data, that can be used to solve your required tasks, the system may benefit by using one model with multiple outputs.
You may have a look into the literature, e.g. collaborative learning, multi-task learning, and auxiliary tasks.
I hope this answers your question.


Large variation in cross-validation scores I'm training a CNN with 5-fold stratified cross-validation. On the first fold, my accuracy is ~80%, on each subsequent fold the accuracy is ~50%. Finally, upon fitting the entire training set my accuracy jumps back to ~80%. I can see that on the folds with bad performance the accuracy and loss never really change. Is it simply that one fold received all/most of the signal from the dataset or is there something else going on here?

Based on the description my guess is that any or all of the following problems occur:

the data is not randomized, which could explain why the first fold get a very different performance than the others
perhaps the dataset is very small, causing massive variations in performance and overfitting.
in any case the model is very unstable, possibly due to overfitting caused by the model being too complex and/or dataset too small.


How to present longitudinal data to LSTM for multiclass prediction I need to implement a deep learning algorithm to predict an ordinal value, called 'Entity', using longitudinal health records data. I read a few articles and guides but I couldn't find a clear explanation or example on how to organize input data; the only thing that I've understood is that I need to use an LSTM node which is designed exactly for this kind of problem.
To clarify let me bring an example, let's say that I have this table:

ID
N1
C1
B1
Out

2
25
3
0
1

2
32
4
1
2

2
52
1
0
1

3
12
2
1
2

3
56
4
0
1

3
73
3
1
3

3
12
2
1
2

The table contains variables of different kinds like numerical, cardinal, and boolean; I need to teach my algorithm to predict the future output with longitudinal data (for example if I input the first two rows with ID=2 the output should be 1 which is in the third row with ID=2).
The rows with the same ID are different data from the same person, checked at different times.
I've understood that LSTM are layers useful when there is the need to catch time relations and the input is in shape (samples, timesteps, features); given that i have few doubts about how to treat and model the training data:

My dataset is not homogeneous in the timesteps dimension (persons may have different number of rows), but I think it's ok to have tensors with different timesteps length, is that correct?
if the answer at the previous question is yes, can i train my network with tensors of growing dimension of the same patient? For istance(notice that i want to predict the future Out):

train1 x:[25, 3, 0] -> label:2;
train2 x:[[25, 3, 0],[32, 4, 1]] -> label:1;

BONUS) Since is a complicated problem I was even wondering if I am looking at it in the wrong way and I should treat it like a kinda-regression problem like the one in that example https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
To give a different perspective of the problem let's say doctors classify Alice by looking at the current situation and at previous visits, I would like to implement this behaviour in an algorithm so that when I have Bob current situation and clinical history I can make a prediction for him too based on the "rules" that i learned from Alice examples.
The alternative behaviour, that I expressed in the BONUS question, is if I want to reproduce the learning pattern of a Doctor which look at a graph like the one in the figure below and learn how to predict the trend of the y value in another patient based on the "rules" learned in Alice's graph.

I'm implementing that on Python using Keras, but I am stuck in this preprocessing phase and I can't proceed.

Use a LSTM, where the input is a sequence, specifically the sequence of records for that patient.  You can train the LSTM to predict anything you want -- e.g., the next step.  In LSTMs, the sequence normally is fixed-length (the same length for all people; you pad shorter sequences); each element of the sequence is a feature vector of same length (the same length for each element in the sequence.)  Each person in your dataset gives you at least one training instance for the LSTM.
It's beyond the scope of this question how to encode inputs, but you can read about standard methods; it's not specific to a LSTM.  For instance, you can encode categorical variables with a one-hot vector, and continuous variables directly.  Similarly for outputs, but you'll need to choose an appropriate loss function (e.g., cross-entropy loss for categorical variables, MSE for continuous values, etc.).  None of that is specific to LSTMs, so read up on how that is done with neural networks and the same methods will apply.
You'll probably need a lot of data to get the LSTM to work well.


Using CNN on one class only I want to use InceptionV3 on one class of image (ie. to detect one type of object/one label only). For example, to detect 'penguin' or 'no penguin'. All my images have been cropped down to mostly display the object, so there is barely any background present and the object is in clear view. I have used bottlenecking; I have frozen the top layer of Inception so that I can retrain Inception to classify my object. However, this means that I have no alternative class to train with since I only want to classify one object.
The issue is that val_accuracy is always equal to 1, implying a perfect model, which my better nature says is too good to be true. Therefore, I believe the problem is that I have only provided the model with 1 class. How can I overcome this?
Finally, I have tested with InceptionV3 and an out-of-the-box model just to check that my Inception set up was not the issue. Both yielded varying training accuracy, but again, the basic model produced val_accuracy equal to 1 every time. I have also experimented with train-test splitting such as 10% to train with and 90% for validation. To my knowledge, a benefit of transfer learning is that you can use small datasets to train with. Even so, my dataset is 3300.

Since you are only using one class for training, you can't expect the network to understand what the object is not.
For example, if you want to classify cats you have to pass to the network images that are not in your main class [dogs, cars, boats, random_noise, penguins]. If you try to pass any other image to your current model it will answer with great confidence that it is a cat.
Your validation accuracy will always be 100% if you have no other classes in your dataset.
For a classification problem you need at least two classes, and your train/val/test subsets should have enough images of each class so your metrics become meaningful.


Deploy pretrained model to simple web app As a small personal project, I wanted to try and deploy this image classification model with a simple web app (input image, output classification and heatmap). There are a couple pretrained models on that link. What is the absolute easiest way to do this (don't need production level capabilities)?

Flask: https://flask.palletsprojects.com/en/1.1.x/

It's a Python framework for dynamic web resources. Should be reasonably easy to load up the estimator and create an endpoint that you can post an image to and get the output you want.


How to develop a good model for predicting electricity usage? My goal is to come up with a set of research questions for which to drive my self study into my data.

I have Smart Meter data that is taken every minute, on my home's power usage.

How could I possibly use the time series data on my household energy consumption that would help me extract practical insights from it?

Adding as a answer instead of comment due to insufficient reps.
Consider developing a model that can identify the pattern of electricity and water usage per house throughout the year. And predict the upcoming requirements. It will be a time series problem. This will help in maximising resource utilisation and better availability of resources among everyone.
Later, this can help in smart power and water distribution which is not yet applied.
If you want a bigger community working into this data set, you can share it online or provide it to Kaggle. Many people are out there ready to help in it.


How do I split contents in a text that would include two or more different themes (context) in NLP? For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning"
the output should be:

The airline companies have affected by Corona since march 2020
a crime has been detected in Noia village this morning

the text has no Breaks. I know it is not a one-click solution, but if anyone knows a methodology or techniques to solve such a problem, please provide me with resources.

First reading the question I thought this is very easy and I started searching and trying out some libraries (i.e. nltk, and spacy). Here are my attempts and clearly showing that none of them works. I was thinking that these libraries will generate a parsing tree or a context-free grammar to detect sentences, but they are splitting based on a certain characters such as periods or commas.
So the other solution is to learn a classifier that learns the boundaries of sentences. A simple solution can be to use random forest classifier or SVM to serve as a binary classification task for each token. But the missing part is the data to train the model on and what to consider as labels (a BIO tagging scheme is a good idea here). I include two papers here that can be a good starting point for you to first read their methodology and use the datasets they are mentioning to do the task.

Du, Jinhua, Yan Huang, and Karo Moilanen. "AIG Investments. AI at the FinSBD task: Sentence boundary detection through sequence labelling and BERT fine-tuning." Proceedings of the First Workshop on Financial Technology and Natural Language Processing. 2019.
Rudrapal, Dwijen, et al. "Sentence boundary detection for social media text." (2015).


Unsupervised Clustering high dimentional data not having estimation for K I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster as representative of that cluster (meaning that each node in that cluster is mostly similar to that representative compared to any other node in other clusters). So we won't have any noise (meaning that all nodes should be represented). I have done K-means, Kmedoids, hierarchical, dbscan, hdbscan, etc. Each having their own issues.

Kmeans doesn't report one of the nodes as center for the cluster but the center point which might not be one of our nodes. So we switched to Kmedoids. which the results are too dependent of the initial seed. Also not knowing K is another problem.
Then we used elbow method to find K. But since not sure what is the upper limit for K, the plot is not very Elbow-y but more like a gradual decrease. Then we tried Sillouette0score method and I got WAY too big number for clusters (something over 200 which doesn't seem right to the field expert). Same problem with the number for K generated by Affinity Propagation (too big of K).
Since we have tens of thousands of data, Meanshift doesn't relly work properly. Also HDBSCAN reports 264 labels which again seems unreasonable.

I would like to try some dimensionality reduction methods for this data. But not sure what would work well. tSNE is too focused on visualization which doesn't seem best fit for our use. Any advice would be VERY appreciated.

I point out several things first:

You have an unsupervised problem so the first to know is don't search for The Right Answer. There is no right answer and you take the best you get acording to some pre-defined criteria!
not knowing K is another problem: This is one of many questions in unsupervised learning which do not have any answer! All those methods like elbow are just heuristics.
Kmeans doesn't report one of the nodes as center for the cluster: K-Means can be modified to do so. The simplest trick is to find the closest neighbor of final representatives from the data point and set them as representatives of clusters.
Some methods that you mentioned like K-Means and Sillouette assume gaussian clusters. If your clusters are not properly compact and well-separated, then they mislead you.

After these, let's have a look at your problem. Try to get an insight about your data with visualization techniques. tSNE is pretty sensitive to its parameters and needs more investigation. Start with PCA, KPCA or LLE to get some idea about data. In case they didn't give you a proper embedding, then go for UMAP. UMAP is also sensitive on parametrs but less than tSNE and it's pretty faster. It will give you more compact clusters if it finds any which is a bless for your work. At the end if none of them worked maybe try tuning tSNE parameters and see if you can find a good embedding.

On top of UMAP or tSNE you can try your clustering alorithm and it will be a logical pipeline for your problem.

Good Luck!


testing of data in random forest? I want to clarify point in random forest literature. In random forest we select random features from a subset of data, i.e it is subset of features from subset of data for a single tree. But when we test the data, let's take a test sample 'v' from the test data. The data has say '5' features.

Tree one was constructed using features 3 and 2.
Tree two was constructed using features 2,3, and 5.
Tree three was constructed using features 1,2 and 3.
Tree four was constructed using features 1,4,and 5.

at leaf node we will have probability showing to which class does the 'v' point belongs.

My question is : Will each forest choose same features from the test data sample 'v' as it used when the tree was constructed. ?
for example will tree one will use features 3 and 2. and tree two use feature 2, 3 and 5. respectively

Yes, it has to use the same featues. If not, the training algorithm does not make any sense.


What is the correct procedure when "joining" data takes ~6 hours? I am dealing with bike-share data. I have 2 DataFrames:

trips_df (subset shown), total entries = 1,048,568

weather_df (subset shown), total entries = 2,654

I am trying to calculate and attach the total_precipitation for each trip, as a column. I do this by looking up the start_timestamp and end_timestamp datetime for each trip from trips_df, in the weather_df, and summing the precipitation_amount within those times, then attaching that value back in the trips_df under the new column. I can attach the code if it's helpful.

I ran the code on a subset of 65 entries and it took ~1.3s. (CPU times: user 1.27 s, sys: 8.77 ms, total: 1.28 s, Wall time: 1.28 s). Extrapolating that performance to my entire data, it would take (1.3 * 1048568)/65 = 20971.36seconds or 5.8hours.

What am I supposed to do in this situation? For context, this is a Kaggle style data science project so I'll have to do further data wrangling, and data extraction then apply a predictive model.

There is no ideal way of doing that type of join efficiently in Pandas.

One possible option is using numpy.piecewise to define matches based on the timestamps from each dataframe. An example of apply numpy.piecewise can be found here.


Classification execution time - Orange Is there any way to find out execution time for classification algorithms in orange?

you can just use time library and give time counter like time save to variable and after classification take time now save to variable and subtract both and print the result you get the execution time


What is Inductive bias Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where a is bias term
Do we have any difference between bias and inductive bias.
How Inductive bias is helpful in generalizing
Any Kind of reference is helpful
Thanks in Advance

This is a tentative general answer.
Bias in statistics (and machine learning) is the difference between an estimated value and the true value. ie $\text{bias} = \hat{y}_{\text{estimated}} - y_{\text{true}}$.
Another (losely related) concept of bias especialy in mathematics is any, primarily, constant term that is added to something (and thus makes a difference between the two values and thus introduces bias). Bias in neural networks is meant in this mathematical sense, rather than the statistical one.
Apart from the above, bias as a term has been introduced in other fields, for example in psychology (eg cognitive bias). The term inductive bias comes from machine learning.
This sense of bias refers to the initial assumptions some entity or algorithm takes for granted and tries to learn based on them. So the induction made is influenced by these initial assumptions, and if these are proved wrong, then there will be bias in the usual statistical or mathematical sense, that is the estimation will differ from the true value.
In machine learning and statistics there is what is called Bias-Variance tradeoff meaning that the total Mean Squared Error (MSE) can be decomposed into bias (in the statistical sense), variance and irreducible error. How is this related to generalisation? Well to generalise well variance should be minimised (in most cases), so since irreducible error is out of control, one way is to increase bias, in the statistical sense (one way or another), if MSE is to be held the same.


What is the distribution of categories in imagenet training set (ILSVRC2012) http://arxiv.org/pdf/1409.0575v3.pdf Table 2
says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set.

Ideally I'd like to avoid downloading the 138 GB just for this purpose as I otherwise don't need it.

I was wondering if anyone knew the exact numbers per class in the training set, i.e., how likely is each class in the training set?

I couldn't find an URL text file for the ILSVRC2012 training set, but for complete imagenet you can download the URLs only as a text file: http://image-net.org/download

I wrote the following script to get a feeling for the data:

#!/usr/bin/env python

"""Analyze the distribution of classes in ImageNet."""

classes = {}
images = 0

with open("fall11_urls.txt") as f:
    for i, line in enumerate(f):
        label, _ = line.split("\t", 1)
        wnid, _ = label.split("_")
        if wnid in classes:
            classes[wnid] += 1
        else:
            classes[wnid] = 1
        images += 1

# Output
print("Classes: %i" % len(classes))
print("Images: %i" % images)

class_counts = [count for _, count in classes.items()]
import matplotlib.pyplot as plt
plt.hist(class_counts, bins=range(max(class_counts)))
plt.show()

which gave:

Classes: 21841
Images: 14197122

Classes which have less than 100 examples are pretty much useless. Lets remove them from the plot. Also increase the bin size to 25:

#!/usr/bin/env python

"""Analyze the distribution of classes in ImageNet."""

classes = {}
images = 0

with open("fall11_urls.txt") as f:
    for i, line in enumerate(f):
        label, _ = line.split("\t", 1)
        wnid, _ = label.split("_")
        if wnid in classes:
            classes[wnid] += 1
        else:
            classes[wnid] = 1
        images += 1

# Output
print("Classes: %i" % len(classes))
print("Images: %i" % images)

class_counts = [count for _, count in classes.items()]
import matplotlib.pyplot as plt
plt.title('ImageNet class distribution')
plt.xlabel('Amount of available images')
plt.ylabel('Number of classes')
min_examples = 100
bin_size = 25
plt.hist(class_counts, bins=range(min_examples, max(class_counts), bin_size))
plt.show()

Or with seaborn:

import seaborn as sns
sns.distplot(class_counts, kde=True, rug=False);
sns.plt.show()

Top 10

The top 10 classes with most data are:

top10 = sorted(classes.items(), key=lambda n: n[1], reverse=True)[:10]
for class_label, count in top10:
    print("%s:\t%i" % (class_label, count))

n02094433:    3047 (Yorkshire terrier)
n02086240:    2563 (Shih-Tzu)
n01882714:    2469 (koala bear, kangaroo bear, native bear, )
n02087394:    2449 (Rhodesian ridgeback)
n02100735:    2426 (English setter)
n00483313:    2410 (singles)
n02279972:    2386 (monarch butterfly, Danaus plexippus)
n09428293:    2382 (seashore)
n02138441:    2341 (meerkat)
n02100583:    2334 (vizsla, Hungarian pointer)

Using http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n02094433 you can look the names up.


SHAP Explanations in case of repeated train/test split I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that follow (respectively a summary plot Figure A and a dependece plot Figure B):

This is easy if I have a single model, with a single train/test split. However, this is risky as I might have a selection bias, so I usually evaluate the accuracy of my models with 500 random train/test splits (then calculate average values of Accuracy and Cohen's Kappa).

If I do so, how can I get the above plots, considering average information from all the 500 XGBoost models?
Three strategies come to my mind, but none seems to solve the issue:

1- Train a model on all sample (without split) and calculate SHAP values on that. I would keep calculating accuracy and Kappa on the 500 models with train/test split.

2- Select the model of 500 with the best fit on test set and calculate SHAP values on that.

3- I can calculate SHAP values for each feature, for each of 500 repetitions and then calculate their mean and standard deviation. Then I can produce an histogram out of that.

N.1 does not seem to be the correct approach. N.2 also seems to be biased, and things might change from one model to the other.
N.3 is what I have been doing so far, as it seems to be the most reliable approach. However, it has the drawback that I cannot produce the graphs that I would like to have (Figure A and B).

Any idea to produce meaningful graphs (A and B) considering information from a repeated train/test split approach?

I'd propose doing a similar thing that you do with your accuracy and Kappa metrics — calculate SHAP values for all 500 splits, and take the average of these $n\_samples \times n\_features \times 500$ matrixes in the third dimension to get $n\_samples \times n\_features$ matrixes, which you can use to create your desired plots.


What is difference between Multi-class One vs All and Multilabel Classification? Although Multi class is different from Multi label classification, whats difference does adding One vs All make in Multi-class.

Edit 1:
http://scikit-learn.org/stable/modules/multiclass.html#multilabel-learning

In this Link the part where it mentions supported classifiers under bullets of Multiclass as One-Vs-All and Support Multilabel is confusing.

In multiclass classification each class is mutually exclusive, but in multilabel classification each class basically represents a different binary classification task.

An example.

Multiclass: Images that could contain a dog, a cat or a frog. Each image contains only one of the animals.

vs

Multilabel: Movie Genre Classification based on poster images. You have a poster image from a movie and want to determine whether the movie is a drama, action, thriller etc. A movie could belong to multiple of the these genres.

So to answer your question, the one-vs-all strategy in multilabel classification basically separates the k binary classification tasks. So using the above example, you would have k binary classifiers, where each one would basically represent each genre.
So you would have a binary classifier for drama, one for action, one for thriller etc.


Pruning tree using REP i create a decision tree model using c4.5 algorithm. After create the model, i evaluate model using 10 fold cross validation and classify model using test data to get accuracy. And then I run prune tree with REP.

My question is what data should i use to compare accuracy (after prune and before prune) to decide to remove the leaf? and from where i get the accuracy? using cross validation or classify test data?

The reduce error pruning strategy works in the following way:

Train a tree on a training data set
Fit a pruning data set (which is different than the training data set) to the tree. What you will have is the tree as was learned at step one, but for each node you will have some instances from the pruning data set.
Prune each node from the tree if the misclassification error computed for the instances from the pruned data set is not larger than the original misclassification error rate computed on the training data. For example, suppose that you do a binary classification and you build a tree from a training data set. Now you fit data from the pruning data set and some instances arrives in a given non-leaf node. Suppose that at that node arrives $2$ positive instances and $8$ negative instances. This gives an error rate of $0.2$. Now from that node the instances are split given to the criteria already learned into two children node, one will have $2$ positive instances and $2$ negative instances, and in the other child node goes the other $6$ negative instances. The cumulative error on children is also $0.2$. This means that going further with the split would make no sense, since the data varies a lot. That means you can actually cut the child nodes and make the original non-leaf node a leaf node.

The point of using fresh data (pruning data set) is to check if the split is useful or the noise from data covers the signal given by the split.

To evaluate the performance of both models, before and after pruning, you need another different data set, named validation data. To see way you have to understand that pruning is actually also part of extended training. The only difference than usual learning is that in training with REM you learn how to extend using some data and you learn how to unlearn using some other data.

If you use the same data from training to do pruning you simply do not progress (you can stop early). To use for pruning the data set for testing will invalidate you model selection criteria, since it will almost always be the case that the pruning tree performs better (original tree learns from training data set, pruned tree learn from both). I dare to say that if you would like to asset the prediction error, you will have to use a different data set, or envelope everything in a cross validation.


Combining Two CSV's in Jupyter Notebook I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another csv.
In Jupyter Notebook
Thank You!

You can try the below code to merge two file:
import pandas as pd
df1 = pd.read_csv(‘first.csv’)
df2 = pd.read_csv(‘second.csv’)
df = df1.merge(df2, on=‘Column1’)


Check Overfitting in CNN I am kind of new to NLP and text classification with Convolutional Neural Nets, and I have trained my first models quite recently. I am a little bit concerned with overfitting. I am doing multilabel classification, so my output is a list of 9 numbers (one per label) containing probabilities. I have to set a threshold to the output to get a list of 0s and 1s and assign labels to sentences. When I train the models, I draw a couple of plots to check if there is overfitting or not.

I draw the variation of the F1 score, precision and recall depending on the threshold I set on the output from the model. I see that if I try the trained model with the training data the plots show a perfect F1 score, precision and recall but it shows something much poorer with the test data. In the plots, the orange line corresponds to the training data and the blue one to the test data. Shouldn’t both curves be closer? It seems to me that the training data is perfectly fit whereas the test data is further behind. Is this a good way of checking overfitting? Or is there any other way of checking it?

One way to detect overfitting is to monitor train- and validation error. If both are diverging during training an overfitting situation is given. With Keras you could set verbose=1 while training to monitor the metrics or you could plot your model history:

History = model.fit(...)
History.history #holds val and train error for all trained epochs


Loss function for an RNN used for binary classification I'm using an RNN consisting of GRU cells to compare two bounding box trajectories and determine whether they belong to the same agent or not. In other words, I am only interested in a single final probability score at the final time step.

What I'm unsure about is how to formulate the loss function in this case. I see two options:

1) force the network to output the correct label at every time step i.e. if I am providing a positive training sample whose output should be 1, then my loss function would be a vector of ones subtracted by the network's output at each time step

2) only check the output at the final time step and use only that in my loss function.

Intuitively, the second option makes more sense, but I'm sure there are other factors that come into play too.

The second option is the good one.

You can select the last output that correspond to a not padded input and using it for your loss.
Or you can directly express that explicitly : in keras set the flag return_sequence to False and your RNN layer will only give you the last output that correspond to a not padded input.

If I were you I would try to put a dense layer between the RNN layer and the last output. Don't forget to use a softmax activation function for the last layer in order to get probabilities.


What is the best way to deal with image sizes in object detection using YOLO v. 4? I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios.
Examples of using Darknet / YOLO from the command line such as in this Colab notebook simply throw images at the darknet executable without any apparent considaration for image size or aspect ratio. This makes me think YOLO does some scaling/cropping/insertion of blank parts automatically.
Then when I look at the darknet_images Python module that comes with the Darknet/YOLO framework, the approach seems to be to use cv2 to scale images to the size of the network prior to detection in YOLO. In particular, this seems to generally skew the aspect ratio. Now I am confused: should I scale or not prior to detection via darknet.detect_image()?
I can create image objects with darknet.make_image() using the image's native size and feed them to darknet.detect_image() without any complaints, so what does Darknet do about the size in that case?
I am also wondering if it would be better to scale a, say 1000x500 image to my Darknet network's size of 608x608 as a 608x304 image in a 608x608 "frame", leaving the remainder of the frame blank (white or black)? Or should I flat out scale the 1000x500 image to 608x608, distorting the aspect ratio heavily? I suppose YOLO's training takes some amount of distortion into account via data augmentation, but I guess there must be limits to how much distortion it can tolerate?

I seem to have figured it out now. When using darknet.detect_image, it calls predict_image which in turn is network_predict_image. The latter resizes the image if it is not already the same size as the network's input layer.
If you instead call darknet detector test from the command line, as I have seen examples do as mentioned in my original question, the "chain of command" goes through test_detector which resizes the image if it is not already the size of the network's input layer.
So, in any case it seems that images get resized to fit the network's input layer along the way and I do not have to do anything in particular to handle it.


ML technique to predict next online I want a direction for ML technique to predict the next time you will be online in chat app

My table contains; user id, timestamp & status

status is a categorical variable (online, composing, offline), I'm saving a row for each status change which means user online, maybe writes something to me (composing) and afterwards went offline, so I can extract the time online (currently I don't see a reason for this feature)

I want to export the model to PMML

What is the best approach if I want to a service to get the next timestamp that user will be online ?

That sounds like time series forecasting. Here's a related post:

https://stats.stackexchange.com/questions/212912/forecast-time-series-data-with-external-variables


Image Classification task with unevenly sized images Generally speaking, what's the best way to approach an image classification task with unevenly sized images (e.g. example 1 has size 300x240, example 2 has size 240x224 etc.)? Ideally I would like the dimensions to be consistent across all images before commencing CNN training.
I'm quite new to deep learning and would really appreciate any help.
Thanks

The size of the input image is usually decided by computational resources you have. If input image has larger dimension you need to increase filter size, then model will have more parameters, hence you need more computational resources and longer training time.
Usually, I resize the images to min(heights of all images), min(widths of all images).
If there are larger samples of particular dimension in the dataset and you have enough computational resource, it might also makes sense to resize to that dimension.
Try with different settings, pick the one that best works for you.


Mercer's Theorem importance I understood that Mercer's Theorem extends the definition of kernels also for infinite input space.

In Machine Learning realm our training set is always finite and hence the input space is always finite. So why are we interested in infinite input spaces?

Where is the flaw in my reasoning?

Mercer's Theorem and infinite-dimensional spaces aren't used directly. It justifies use of things like the Gaussian kernel in SVMs. Mercer's theorem says this kernel is just an inner product in some other space, but we need not figure out what that space is, or a mapping to it. The fact that it exists is essential to proving that the SVM still works.


SVR - RMSE is much worse after normalizing the data I'm building a model using a custom kernel SVR that looks into a few of my dataframe's features and checks the proximity/distance between each pair of datapoints. The features are weigthed and the weights were calculated using cross validation.
Initially my dataframe was not normalized, and the model's results were not very good (RMSE higher than 25% of the target range). Because I had read that SVR is sensible to scale, I decided to try to normalize the data, which resulted in much worse "predictions".
The original results in terms of Root Mean Squared Error were as follows:
RMSE (test_set):  6.59
RMSE (training_set):  6.56
RMSE (validation_set):  5.90

The new results, with the normalised data are the following:
RMSE (test_set):  2404.68
RMSE (training_set):  148.06
RMSE (validation_set):  2546.44

These values are very much outside the normal range of my outcome variable. This makes me suspect I'm doing something wrong.
Notes:

I recalculated the kernel's weights after normalizing.
I normalized the data after splitting it into train/test/validation.
I didn't normalize the y (outcome variable) vector.

To normalize the data, I used the following code:
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()

def normalize(df):
    x = df.values
    df_scaled = sc.fit_transform(x)
    return pd.DataFrame(df_scaled, columns=df.columns)

X_train = normalize(X_train)
X_test = normalize(X_test)
X_validation = normalize(X_validation)

Then I use these 3 variables to train and test the model.
regressor = SVR(kernel=my_kernel)
regressor.fit(X_train, y_train)

Any tips on what I'm doing wrong? Thanks.

You normalized it after splitting into train / test / validation, but you're doing this wrong. You need to normalize the training set X_train_normalized = scaler.fit_transform(X_train), and then use the same statistics to normalize the test and validation sets: X_valid_normalized = scaler.transform(X_valid); X_test_normalized = scaler.transform(X_test).
If you don't do it this way, you're normalizing to different distributions (mean and variance are different). So if you're normalizing to a z-score, replacing each feature value with how many standard deviations from the mean it is, a value of 1.0 that the model sees in the training data has a different original value than a 1.0 in the validation set and test set.
So the right way is to calculate the statistics for the training set (the fit part of fit_transform) and assume the validation and test sets are from the same distribution (you're only supposed to know about the training set at training time). Then use those same statistics on the validation and test sets to make sure you're comparing apples to apples.


Any tool to load large csv files quickly on windows 7 I am working with relatively large CSV files (1.5 MB+) and while testing the code, I have to frequently open up the CSV files to read a particular row, validate the data are there and then close the CSV file and move to another one.

Files which are larger than 1.5 MB are taking about 40 seconds per file open on my computer. Is there any tool that will allow me to open large CSV files quickly (I am only looking to read the CSV file and therefore a CSV reader that opens up files quickly will be useful)

Alternatively, if there is any tool that will perhaps when the first 1000 lines
or the last 1000 lines will be good.
I know I can do this in UNIX by using commandline utilities but then they dump the data in different files and again I will have to delete those files. Moreover, I'm looking for a solution on Windows.

I hope somebody can help resolve this. Thank you for any inputs!

notepad++ will open a file of that size very quickly.


Credit scoring using scorecardpy with XGBoost I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried to use it with my XGBoost like in an exemple but my ROC AUC fell to 0.5 and I don't see what I am doing wrong. Thanks for your help.

data = pd.read_csv('data.csv')

train_index = data['date'] < '2018-04-01'
test_index = data['date'] >= '2018-04-01'

data_final = data.drop('date', axis=1)

df_train = data_final[train_index]
df_test = data_final[test_index]

data_final_vars = data_final.columns.values.tolist()
y=['label']
X=[i for i in data_final_vars if i not in y]

# woe binning ------
bins = sc.woebin(data_final, y="label")
sc.woebin_plot(bins)

# binning adjustment
# # adjust breaks interactively
# breaks_adj = sc.woebin_adj(dt_s, "creditability", bins)
# # or specify breaks manually
breaks_adj = {
    'age': [26, 35, 40, 50, 60]
}
bins_adj = sc.woebin(data_final, y="label", breaks_list=breaks_adj)

# converting train and test into woe values
train_woe = sc.woebin_ply(df_train, bins_adj)
test_woe = sc.woebin_ply(df_test, bins_adj)

ytrain = train_woe.loc[:,'label']
xtrain = train_woe.loc[:, train_woe.columns != 'label']
ytest = test_woe.loc[:,'label']
xtest = test_woe.loc[:, test_woe.columns != 'label']

print("shape of xtrain: {}".format(xtrain.shape))
print("shape of xtrain: {}".format(xtest.shape))

from xgboost import XGBClassifier

XGB = XGBClassifier(n_estimators=100, n_jobs=6, verbose=1)
# List the default parameters.
print(XGB.get_xgb_params())

# Train and evaluate
XGB.fit(xtrain, ytrain, eval_metric=['rmse'], eval_set=[((xtrain, ytrain)),(xtest, ytest)])

# # Classifier

from sklearn.metrics import roc_auc_score

probs = XGB.predict_proba(xtest)
roc = roc_auc_score(y_true=ytest, y_score=probs[:, 1])
print("RF roc score: {}".format(roc))

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(ytest, probs[:,1])
plt.figure()
plt.plot(fpr, tpr, label='XGBoost Classifier (area = %0.2f)' % roc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('XGB_ROC')

from sklearn import model_selection
from sklearn.model_selection import cross_val_score
kfold = model_selection.KFold(n_splits=10, random_state=7)
modelCV = XGB
scoring = 'accuracy'
results = model_selection.cross_val_score(modelCV, xtrain, ytrain, cv=kfold, scoring=scoring)
print("10-fold cross validation average accuracy: {}".format(results.mean()))

# score ------
card = sc.scorecard(bins_adj, XGB, xtrain.columns)
# credit score
train_score = sc.scorecard_ply(df_train, card, print_step=0)
test_score = sc.scorecard_ply(df_test, card, print_step=0)

# psi
sc.perf_psi(
  score = {'train':train_score, 'test':test_score},
  label = {'train':y_train, 'test':y_test}
)

It happened to me as well, although I used a logistic regression model not XGBoost.

The problem is not about which model to choose, but rather there is something wrong with woebin_ply function. I didn't read the source code but the woe value I'm getting doesn't match the value for the corresponding bin/input value (You can double check your results as well).

After manually matching input value with bin with corresponding woe value, my scorecard model performs at similar level with my benchmarking models.

Hope this help!


Keras: Very high loss for Autoencoder I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet:
input = Input(shape=(768,))
hidden1 = Dense(512, activation='relu')(input)
compressed = Dense(256, activation='relu', activity_regularizer=l1(10e-6))(hidden1)
hidden2 = Dense(512, activation='relu')(compressed)
output = Dense(768, activation='sigmoid')(hidden2) # sigmoid is used because output of autoencoder is a set of probabilities

model = Model(input, output)
model.compile(optimizer='adam', loss='categorical_crossentropy') # categorical_crossentropy is used because it's prediction of multiple labels
history = model.fit(x_train, x_train, epochs=100, batch_size=50, validation_split=0.2)

I ran this in Jupyter Notebook (CPU) and I am getting loss and validation loss as:
loss: 193.8085 - val_loss: 439.7132
but when I ran it in Google Colab (GPU), I am getting very high loss and validation loss:
loss: 28383285849773932.0000 - val_loss: 26927464965996544.0000.
What could be the reason for this behavior?

You should not use the categorical cross-entropy loss, but the binary cross-entropy.
The categorical cross-entropy is meant for categorical probability distribution. In your scenario, this means that only one label would be active at the same time. If multiple labels can be active at the same time, then they are independent, and you need to compute the binary cross-entropy.


What are the exact differences between Deep Learning, Deep Neural Networks, Artificial Neural Networks and further terms? After having read some theory I am getting a bit confused about the following terms:

Deep Learning
Deep Neural Network
Artificial Neural Network
Feedforward Neural Network

So, what seems clear to me is that Deep Neural Networks are Artificial Neural Networks with multiple layers (usually more than 1 hidden layer). However I have read several times that:

  "Deep Neural Networks are feedforward Neural Networks with many layers."

I know what a feedforward Neural Network is, but to my understanding Deep Neural Networks is a term for ALL Artificial Neural Networks with multiple layers between input and output layer? Shouldn't there also be for instance Deep Recurrent Neural Networks? Is it correct that Deep Neural Networks must feedforward Neural Networks? This would in turn mean that Deep Recurrent Neural Networks cannot be referred to as Deep Neural Networks.

Moreover, I see there is a large variety of deep learning architectures, such as:

Convolutional Neural Networks
Residual Neural Networks
Deep Belief Networks
Deep Boltzmann machines
...

However, now also Wikipedia gives me a hard time to distinguish all the terms, saying: "Deep learning architectures such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks have been applied to fields including..."

So, my concrete questions rising from all the thoughts above are:

Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks?
If the answer to 1) is yes: is the Wikipedia definition correct as you can read it up there? This would mean that i.e. a Convolutional Neural Network with multiple layers must be referred to as a Deep Convolutional Neural Network, which is not a subclass of Deep Neural Networks?
Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers?
Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?

I will try to explain it in the simplest way I can-

Deep Learning - Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. It is a subclass of Machine Learning where (in layman terms) we use only Neural Networks. https://www.mathworks.com/discovery/deep-learning.html

Deep Neural Network (DNN) - This is just a term for a neural network with many layers and many nodes in layers, which is not usually found in a shallow network. There is no clear boundary between a shallow and deep neural network. I suggest you watch Andrew Ng's course on deep learning to get a better understanding.

Artificial Neural Network (ANN) - This is just another term for a neural network, all neural networks are ANN's.

Feedforward Neural Network - It is a type of neural network where there is no feedback connections. In technical terms, the information flows only in one direction (input to output) in the forward propagation stage.
https://towardsdatascience.com/deep-learning-feedforward-neural-network-26a6705dbdc7

  Shouldn't there also be for instance Deep Recurrent Neural Networks?

There is no need for Deep Recurrent Neural Networks, it would simply be Recurrent Neural Networks.

  Is it correct that Deep Neural Networks must feedforward Neural Networks?

No, RNN is a type of Deep Neural Network which is not a feedforward NN.

So, to answer your questions,

  Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks?

No, the term Deep Neural Networks belong to all Neural Networks with multiple layers. (Architecture does not matter).

  Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers?

Yes, it is a collective term for ALL Artificial Neural Networks with multiple layers.

  Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?

Deep learning architectures just refer to the different neural network architectures used for different tasks - CNN for image processing, RNN for sequence (text, audio etc) processing etc. So, when you are doing a particular task, you can use the associated architecture name. For general purpose, you can just use the term DNN.

Hope this clears your confusion.


Is a CNN architecture more susceptible to adversarial attack if it is built on ResNet? The title kind of sums up my question. But more generally, is a CNN architecture more susceptible to an adversarial attack if it is built on top of another open source architecture, like ResNet. Can something that spoofed ResNet also spoof another model built on top of ResNet?

(Assuming you do not have access to the gradient of the secondary model, but of course do have access to ResNet)

No.  Both standards CNNs and ResNets are highly vulnerable to adversarial examples; neither one is particularly better than the other.  "Open source" is not really relevant here.

If you're asking about ability to craft adversarial examples when the attacker doesn't have access to the model itself, you might be interested in black-box algorithms for generating adversarial examples.  There are ways known that an adversary can find adversarial examples without having access to the model.  Broadly speaking, there are two common attack approaches:

Exploit transferability: Generate adversarial examples for some other model that is known; then they will often fool the secret model as well.
Exploit API access: if you can make queries to the model (send it an input and get back the label that the classifier outputs), then there are new attack algorithms that are able to find adversarial examples.  ZOO is one example, but there are more recent advances as well.

Broadly speaking, there is no known defense for adversarial examples that seems to be very effective in general.


Knowing Joint probability distribution between feature-label space I am doing a course CORNELL CS4780 "Machine Learning for Intelligent Systems". you can find the link here for the one I am going to refer 1st lecture

The professor explains, we have a sample

  $D ={ (X_1,y_1),(X_2,y_2), \ldots,(X_n,y_n)} \sim P$
      Where, (Xi,yi) is a feature-label pair. There is a joint distribution
      over the feature-label space and is denoted by $P$.

We never have access to the $P$, Only God knows $P$.  What we want to do in this supervised learning  task is to take data from this distribution and learn a mapping/function form $X$ to $y$.

I agree/understand till this point.

Then, Professor goes on to make a statement in the lecture, precisely at 34 minutes 26 seconds, that

"IF we had access to this distribution, everything would be easy". But he doesnt explain this statement.

Now my question is What would have been easy if we knew about the distribution ? Does he mean, if we had access to the distribution then we would know the probabilities of each of $(X_i,Y_i)$ pair. Then we can learn a mapping/parameters such that we reduce out of sample error?

The interest is to predict $y$.

If we know the real distribution $P$ of $(x,y)$, there is no need to build any more machine learning model. Given $x$, we can directly consult $P$ to know the probability of $P(y|x)$. For the case of discrete $Y$, we have $P(y|x)=\frac{P(x,y)}{\sum_z P(x,z)}$.