Improving Accuracy of the Deep Learning Model In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code:

from keras import models
from keras import layers
from keras import regularizers
model = models.Sequential()
model.add(layers.Dense(5,activation="relu",input_shape=(train_x.shape[1],)))
model.add(layers.Dense(1,activation="sigmoid"))
from keras import optimizers
#network = model.compile(optimizer=optimizers.RMSprop(lr=0.001),loss="binary_crossentropy",metrics=["accuracy"])
network = model.compile(optimizer=optimizers.Adam(lr=0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False),loss="binary_crossentropy",metrics=["accuracy"])
result = model.fit(train_x,train_y,epochs=60, batch_size=32)

Following is the result of a few Epochs:

   Epoch 1/60
   500/500 [==============================] - 1s 3ms/step - loss: 0.7253 - acc: 0.5520
   Epoch 2/60
   500/500 [==============================] - 0s 137us/step - loss: 0.6379 - acc: 0.6640
   Epoch 3/60
   500/500 [==============================] - 0s 134us/step - loss: 0.6035 - acc: 0.6880
   Epoch 4/60
   500/500 [==============================] - 0s 158us/step - loss: 0.5852 - acc: 0.6980
   Epoch 5/60
   500/500 [==============================] - 0s 136us/step - loss: 0.5864 - acc: 0.7140
   Epoch 6/60
   500/500 [==============================] - 0s 134us/step - loss: 0.5552 - acc: 0.7240
   Epoch 7/60
   500/500 [==============================] - 0s 141us/step - loss: 0.5475 - acc: 0.7280
   Epoch 8/60
   500/500 [==============================] - 0s 164us/step - loss: 0.5340 - acc: 0.7460
   Epoch 9/60
   500/500 [==============================] - 0s 138us/step - loss: 0.5389 - acc: 0.7280
   Epoch 10/60
   500/500 [==============================] - 0s 139us/step - loss: 0.5374 - acc: 0.7540
    ===================For the simplicity I am sharing first and last few epochs result=
   Epoch 55/60
   500/500 [==============================] - 0s 161us/step - loss: 0.4947 - acc: 0.7800
   Epoch 56/60
   500/500 [==============================] - 0s 168us/step - loss: 0.5058 - acc: 0.7660
   Epoch 57/60
   500/500 [==============================] - 0s 158us/step - loss: 0.5011 - acc: 0.7700
   Epoch 58/60
   500/500 [==============================] - 0s 154us/step - loss: 0.5062 - acc: 0.7660
   Epoch 59/60
   500/500 [==============================] - 0s 156us/step - loss: 0.5040 - acc: 0.7600
   Epoch 60/60
   500/500 [==============================] - 0s 147us/step - loss: 0.4994 - acc: 0.7800

Using the above configuration (I have also tried different Neural Network architecture, the above one looks fine), I am able to achieve the best accuracy so far and which is train accuracy ~ 78 % and test accuracy ~ 72 %. I also tried with Logistic regression but in this case train accuracy ~ 65 %.

Here it looks like overfitting occurs so I tried with L2 Regularization and Dropout but none of them help to achieve the better accuracy. Unfortunately, I can't generate more data.

What should I do to achieve the better accuracy of the model (given a limited amount of data)? How can I increase the efficiency of both train and test data?

You can try bunch of things like:

Data augmentation.
Transfer learning on model trained on similar dataset.
Train multiple neural network with different intialisation of weights and then take majority voting during inference.

The last point matters if you're training from scratch because weights intialisation direct the model towards different minima. So with average of different networks you're essentially taking average of different minimas which might increase overall accuracy.


Classification of data generated by radar using FFT I have time domain data which is having binary label in form of 0 and 1. I applied FFT to all the features and label as well. The problem is my output label is now not binary anymore. How to solve this classification problem?

If you are dealing with a classification problem, you should treat the $0$ and $1$ as labels.

The goal of FFT is to convert time domain to frequency domain.

You do not want to include the label as part of your FFT procedure.


Does it make sense to "reorder" a categorical feature to make it monotonic? Sorry for the vagueness of the title; I'll explain what I mean. I'm doing the Kaggle Titanic beginner tutorial. The label you're interested in is the "survived" rate (0 or 1), which you want to predict in the test data.

One of the first things I did was plot the average survival rate for each of the features, for each one, taking the average survival rate for that value of the feature. For example, for the "Sex" feature, you can look at the average survival rate for males and females, which gives you good info you can use.

For features like "Fare" and "Age", which are more continuous, it wouldn't really make sense to take the average in the same way, because there aren't necessarily many people with the same exact age. So, I did "banding", where I do a histogram-like thing of saying that everyone from 0-10 is one band, 10-20 another band, etc, and those bands are now categorical features that can be averaged over well (this is a good move, right? at least initially?).

This is what I get:

That's the background. Now, here's my question. I want to start with a simple model like linear regression. For some features, like Sex, or Pclass, it's clear that a linear model could fit it very well. For some however, like AgeBand, there's not really a clear, monotonic trend for increasing AgeBand...but there definitely is a difference in the survival rate between different bands!

So what you could do is, sort those "non monotonic" features by the survival rate, so it is monotonic, and then use them for linear regression. Is that a good idea or no?

For example, here's the AgeBand feature sorted by the survival rate:

I would not recommend doing this in general as it will likely lead to overfitting.

While a particular reordering might improve fit on your dataset, the goal is to create a model with good predictive power. If you received never before seen data from an independent second, third, one hundredth Titanic, would the ordering still be the same? I doubt it. You'd probably be incorrectly using the somewhat arbitrary ordering fixed on the first, real Titanic on new data where the ordering is no longer the same.

I think the story goes that the young and the elderly got preferential treatment and this is why their survival rate was higher. Therefore your AgeBand curve should be approximately U-shaped. (Which it is, approximately.) If you insist on linear regression, I don't see how you could ever (directly) include this nonlinear (U-shaped) effect.

The fewer assumptions (particularities) go into the choice of features, the better. It would then make more sense to me to reorder such that you pick bands alternately from the two ends (starting from the middle): 3, 4, 2, 5, 1, 6, 0, 7 (or in reverse order, starting from the ends). Or, even better, pool the age bands into coarser groups: {1,2,3}, {4,5,6,7}, {0}. I think these are justified by your exploratory data analysis.

When I look at the range of values in the AgeBand plot ($0.33$ to $0.55$), this variable does not actually explain as much as Pclass, Sex, FareBand. In the end you might end up dropping it from your variables anyway, so once again, the more attention you pay to it, the more you're at risk of overfitting.


Technical term for using regular expressions to classify text? Background

I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext.
He wants to classify the claims as either having been "approved", "denied", or "remanded".
Each file has a section starting with the string "ORDER" followed by some sentences that explain the decision.
I suggested that we use regular expressions to:

extract the "ORDER" section,
separate the section into separate orders (as each case can involve multiple claims),
extract the claim from each order, and
classify each claim as approved or denied depending on whether the string "approved" or "denied" appears in the order regarding that claim.

Problem / Question

The researcher needs to describe his method in his paper and wants to know what the correct / technical term for this method of classification would be:

  I do want to be able to explain each step, including how you changed it to a standard format. Is there a genre of text analysis we can use to classify your ultimate regular expressions approach? It would certainly be data mining, right?

If the answers are stored correctly with no errors, confusion, or ambiguity then the process is a straight up search and match. If there are typos, misspellings, or what-have-you, the process is a Fuzzy string search.

  Here's an excellent Medium article delving into the details of a fuzzy string search.

An example where these cases might exceeded is when approved and denied are included in the same transcript. Say 5% of all your cases are such. You'd likely want to apply some machine learning classifier to correctly categorize your input, in which case, your Q might be about the depths of possible solutions.


Get most likely topic per document in pandas dataframe using gensim I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call model.get_document_topics(model_corpus) to get a list of list of tuples showing the topic distribution for each document. For example, when I am working with 20 topics, I might get the following for the first three documents in my data frame:
[(5, 0.11253482), (7, 0.75876033)]
[(19, 0.96343607)]
[(0, 0.010002977),
 (1, 0.010002977),
 (2, 0.010002977),
 (3, 0.010002979),
 (4, 0.8099435),
 (5, 0.010002977),
 (6, 0.010002977),
 (7, 0.010002977),
 (8, 0.010002977),
 (9, 0.010002977),
 (10, 0.010002977),
 (11, 0.010002977),
 (12, 0.010002977),
 (13, 0.010002977),
 (14, 0.010002977),
 (15, 0.010002977),
 (16, 0.010002977),
 (17, 0.010002977),
 (18, 0.010002977),
 (19, 0.010002977)]

This means that the most likely topic for document_1 is 7, for document_2 is 19, and for document_3 is 4. The primary output that I would like to see is simply this most likely topic for each document. The way I'm doing this now is using a loop:
import numpy as np
import pandas as pd

def get_max(doc):
        idx,l = zip(*doc)
        return idx[np.argmax(l)]

data['doc_topic'] = [get_max(doc) for doc in model.get_document_topics(model_corpus)]

I have around 80k documents in my data frame, so this code takes about 45 seconds to execute. But since gensim has already done all the computations, I keep thinking that that 45 seconds of computational time is simply spent on reorganizing data, so there must be a more efficient way of doing this.
If possible, a secondary output that would be nice to have is the document-topic matrix, such that each row corresponds to a document in my data frame, and each column represents the probability (or similarity) of the document to the topic. So this would yield a DxT matrix, where D is the number of documents, and T is the number of topics.

Timing your get_max function on the largest list you provided gives a time of a little over 9 us which seems quite efficient. I have, however, written another function that does the same but takes a little more than 3 us, so almost 3x as fast. For smaller lists this speedup will become even greater (~10x speedup for lists of size 1 or 2).
doc = [(0, 0.010002977),
 (1, 0.010002977),
 (2, 0.010002977),
 (3, 0.010002979),
 (4, 0.8099435),
 (5, 0.010002977),
 (6, 0.010002977),
 (7, 0.010002977),
 (8, 0.010002977),
 (9, 0.010002977),
 (10, 0.010002977),
 (11, 0.010002977),
 (12, 0.010002977),
 (13, 0.010002977),
 (14, 0.010002977),
 (15, 0.010002977),
 (16, 0.010002977),
 (17, 0.010002977),
 (18, 0.010002977),
 (19, 0.010002977)]

def get_max(doc):
    idx,l = zip(*doc)
    return idx[np.argmax(l)]

def get_max_new(doc):
    return sorted(doc, key=lambda x: x[1], reverse=True)[0][0]

%timeit get_max(doc) # 9.14 us

%timeit get_max_new(doc) # 3.53 us

There may be even more speedups that can be achieved, however that is more difficult for me to test at the moment.


Finding the worst affected industry due to COVID in terms of unemployment My goal is to find the worst affected industries from COVID—19 in terms unemployment.
In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each month and business distribution data.
Business distribution data contains number of establishments in each county by their respective industries. (Manufacturing -121, Accommodation and Food Services -564, Construction-32 etc.)
Unemployment rate data gives monthly unemployment rate in each county.
From this data, what would your recommendation be to find the worst and least affected industry?
Any help is highly appreciated. Thanks in advance.

Thank you clarifying the question and the current data available. This is a really interesting project idea and I look forward to hearing more developments on this.
So, with what I have seen so far, without the monthly breakdown of number of businesses by month, then you you can't look at the affect of COVID-19 on an industry-level, unless you compare 2019 with 2020 data.
However, with the data so far you can look at how COVID-19 is affecting unemployment rates and see if they affect certain groups of counties which have a certain industry composition/distribution.
If, for example, particular counties with similar distributions of industries (e.g. 90% Manufacturing, 5% Retail and 5% Information) show greater unemployment from those with a different composition.
Let me know if I need to clarify further. But again, very interesting project, please do keep me in the know with how this goes.


How can causal inference be used with machine learning? I am wondering how Causal inference is being used with machine learning and specially where in the data science project ? I have been searching for an answer and I've came to the conclusion that Causal inference can be used after the modeling phase in order to confirm some correlations between variables and the target/outcome.

For example if the model has a good accuracy and gives you a high correlation/association between input A and the target B you may want to perform a causal inference to validate that A has an effect on B.

I would like to know if my understanding is correct and also know if there is other application of Causal inference with Machine learning.

Your understanding is correct. Finding correlations between variables is simple but turning them into causal assertions needs an extra effort. Causal inference is used mostly to reach a "prescription" in the form of "do X so that Y happens".

When not to use Causal inference:

If it is possible to do experiments, causal inference can be avoided. For example, A\B tests let you study the effect of a change in two groups and reach a causal conclusion. For example, result of an A\B test would be "users in group A which see a button with color intensity 50 clicked 10% more than group B with color intensity 40", so do X='increase color intensity of the button' so that Y='more click' happens. With larger, more uniform groups, your assertion would be more reliable.

Causal inference in Machine learning:

In most machine learning projects these type of experiments are possible and mostly cheap, therefore why bother? Moreover, specially in predictive projects, value comes from correlated relations. Knowledge of causal relations, which are a subset of correlated relations, does not add value.

Causal inference:

When you work with historical data or you can only "observe" the data without affecting it, causal inference comes into play. Generally, causal inference is a controversial topic as it tries to extract causal relations from observational data (as opposed to experimental data in A\B tests).

To best of my knowledge, the main contributor to causal inference is Prof. Judea Pearl. His underlying tools are probabilistic graphical models (PGMs) and do-calculus. These tools let us explicitly encode our assumptions about the mechanics of data generation, and reach causal conclusions. So when an assertion like "do X, so that Y happens" goes wrong, we can track the problem in our assumptions in a principled way. For example, we may have ignored an important hidden variable that by including it our conclusion would change. He fundamentally says, anyone who reaches a "prescriptive" conclusion is doing causal inference, so it is better to lay out your assumptions explicitly, to prevent correlation-causation problems going unnoticed.

Some interesting resources:

Simpson's paradox is an entry point to become interested in causal inference,
This paper of Judea Pearl connects the paradox with causal inference.


Acceptable variation in accuracy of each k fold when using K-Fold Cross Validation? I have a relatively small dataset consisting of 1432 samples.

I have trained a Random Forest Classifier and performed KFold CV. The results of running 10 Fold CV are as follows:

=== 10 Fold Cross Validation Scores ===

CVFold 1 = 90.2%
CVFold 2 = 87.6%
CVFold 3 = 86.7%
CVFold 4 = 86.7%
CVFold 5 = 83.9%
CVFold 6 = 75.8%
CVFold 7 = 87.2%
CVFold 8 = 82.8%
CVFold 9 = 86.1%
CVFold 10 = 89.3%

Mean Cross Validation Score:  85.6%

I am just not sure how to explain why there are such high variances between some of the folds, i.e from 75.8% in fold 6 to 90.2% in fold 1.

My understanding is that it is simply that the classifier found the samples in the 1st fold (90%) easier to classify than it did in fold 6 (75%) but I'm actually not entirely sure if that is the case.

I understand that each case is different but is it common to have such variances? And is it acceptable?

Edit: some details regarding my data

I have 5 classes, which are imbalanced:

class 1 - 5%

class 2 - 10%

class 3 - 15%

class 4 - 60%

class 5 - 10%

I am using SMOTE to balance the classes.

It's indeed a quite large variation, but nothing alarming since 9 out of 10 folds are within the range 0.8 to 0.9.

There are many possible factors: yes a fold can be easier than another one by chance, but it also indicates that the training process is a bit unstable. Increasing the number of instances and/or reducing the number of features often helps reducing the variation.


Improving accuracy of Text Classification I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, economics, etc., but are very closely related and in some cases even partially overlapping. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.

Data Preparation -

Broke the documents in list of words.
Removed stop words, punctuations.
Performed stemming.
Replaced numerical values with '#num#' to reduce vocabulary size.
Transformed the documents into TF-IDF vectors.
Sorted all words based on their TF-IDF value and selected the top 20K words, these will be used a feature list for the classification algorithm.
Used SVM.

I have 4,500 categorized documents with 17 categories, and I used 80:20 ration for training and test dataset. I used Sklearn python library.

The best classification accuracy I have managed to get is 61% and I need it to be at least 85%.

Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.

First of all good job done in processing the data and coming up with your base model. I would suggest few things that you can try:

Improve your model my adding bigrams and tri-grams as features.
Try doing some topic modelling like latent Dirichlet allocation or Probabilistic latent Semantic Analysis for the corpus using a specified number of topics - say 20.  You would get a vector of 20 probabilities corresponding to the 20 topics for each document. You could use that vector as input for your classification or use it as additional features on top of what you already have from your base model enhanced with bigrams and trigrams.
Another thing I would say is try using a tree based classifier ensemble to capture non-linearity and interactions between the features. Either of random forest or gradient boosting would be fine. In gradient boosting you can use  xgboost as its a pretty good package that gives good classification.
If you are familiar with Deep Learning you can give a try with a Recurrent Neural network architecture(mostly the LSTM versions).

Hope this helps and let me know if this improves your classification accuracy.


How to draw line plot with multiple lines on Tableau? I have 4 different measures which I want all on the SAME line plot. They do have the same scale. I tried dual axis but it doesn't work for more than 2 plots.

Use "Measure Values" in your Measures Tab. Pull it in Row, and then under the Measures Marks, pull in all the measures you want to display.


python: why add new features of image to be worse result with xgboost there are about 93 rows of data with features, two classes label. And, there are about 49 one-hot value features, and there are about 10 features continuous value. I split the data randomly by train and valid data to predict it with following code:

model = XGBClassifier()
model.fit(X_train2, y_train2)
y_pred = model.predict(X_valid)
accuracy = accuracy_score(y_valid, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

I found that when I just used the one-hot value features, the accuracy is 78.95%, but when added another 10 features with continuous values, and the accuracy is reduced sharply, and is 57.89%.More, based on the feature importance, the ranking top 5 features is the continuous values features, also other continuous features have importance value. But most of the one-hot value feature importance value is 0, just 7 features have value. Of course, just use the one-hot value, there are just about 17 features have value, others are 0.

In addition, just used the 10 continuous features, the accuracy is 68.42%.

I don't know clearly why caused such kind of result.
now I guessed that one of reason may be the number of data is not so big, and the features is almost about 59 features.

Guessing by your reported accuracies and the 93 sample points, it looks like you have just 19 samples in your validation set (the three accuracies you report then are 11/19, 13/19, and 15/19).  So the smallest change in accuracy you can get is more than 5 percentage points, and your reported swings don't seem so drastic.

However, continuous variables can lead to overfitting, if your model learns each particular value separately (which would also lead to your continuous variables getting high feature importance scores).  The default XGBClassifier parameter max_depth=3 makes that a little less likely, but the default n_estimators=100 for only 74 samples might well be at fault.  I'd agree with bradS that you should tune hyperparameters a bit.  Also, it's probably worth at least throwing something else like logistic regression at the data.  But again I'm not sure how much fine-tuning you can realistically do with just 19 validation points.


Is there any reason why providing symbolic features into an MLP wouldn't outperform feeding raw pixels to a CNN in a RL task? I am tackling a RL problem (relaxed version of Space Fortress) with DQN. The usual approach would be to feed pixels into a CNN but that is usually very slow. I am considering feeding symbolic features such as coordinates, velocity and angles of the relevant elements into a Multi Layer Perceptron because then the training would be faster and I could try more things quicker. My idea is that the perceptual part of the task is not that interesting to learn here and I would like to focus on the RL part instead (I am experimenting with Hierarchical RL).

Given that with all the symbolic features that I would provide it would possible to reconstruct the full image and hence the state of the environment, would there be any reason that a CNN pixel based approach would be better?

PT: The environment has a fixed number of elements

Thanks

If the features are able to fully define the image, there would be no reason to use a pixel-based approach.  You are essentially performing lossless compression and (nearly) instantly acquiring a feature set, so you would see a dramatic performance difference between the CNN and MLP.

If you needed a quick solution, CNNs are great out-of-the-box algorithms, but if you are willing to put in the time to develop an MLP, it would provide a performance boost without losing accuracy.


feature names in LogisticRegression() I want to know feature names that a LogisticRegression() Model has used along with their corresponding weights in scikit-learn. I can access to weights using coef_, but i did not know how can pair them with their corresponding weights.

We if you're using sklearn's LogisticRegression, then it's the same order as the column names appear in the training data. see below code.

#Train with Logistic regression
from sklearn.linear_model import LogisticRegression
from sklearn import metrics

model = LogisticRegression()
model.fit(X_train,Y_train)
#Print model parameters - the names and coefficients are in same order
print(model.coef_)
print(X_train.columns)

You may also verify using another library as below

import statsmodels.api as sm
logit_model=sm.Logit(Y_train,X_train)
result=logit_model.fit()
print(result.summary2())


Determining correlated product categories using store purchase history I have a large dataset that contains product purchase history, like so:

userID  productID   category        subcategory
123     ABC         Kitchen         Knives
123     BEA         Kitchen         Organization
233     ZZS         Electronics     Phones

For a first project, I'm looking to answer the question: "What discrete groups of categories/subcategories do shoppers tend to shop in?". For example, we may find that shoppers who buy Monitors are highly likely to buy Keyboards and Mice as well.

Any direction on getting started on a problem like this is appreciated!

This is classic market basket analysis.

Clustering is the weong tool, you want frequent item set mining and association rules instead.

https://en.wikipedia.org/wiki/Association_rule_learning


How to determine threshold in Sigmoid function Context:

I picked up data-set from here and tried to run Logistic Regression on it. Since I am not very much aware of MATLAB, I converted "Strings" to "Numbers" with my own using "NUMBERS" software.

What I want to achieve:

After running the LR algorithm when I tried to predict the value of existing data points, I am getting values ranging between 0-1 (as it should be), but since my job it to predict whether it is either 0 (yes) or 1 (no), that means I need to find a cut-off line (threshold) in my prediction (This could probably be done by comparing actual value by predicted value).

Question:

How can I figure out the threshold for predicted result such that result is assumed to be 1 if predicted value > threshold, otherwise 0?

Predicted values can be found here.
I am assuming predicted values are correct as cost curve is showing asymptotic nature. I have pushed my work here, you may want to cross-validate and provide me few more key points.

As per Andrew Ng's Course, if you use the sigmoid activation, the outputs represent the probability of either outcome 0 or outcome 1.

So the decision boundary is 0.5

if prediction > 0.5 , the prediction is 1

if prediction <= 0.5 , the prediction in 0

Here's a screenshot from Andrew Ng's slides:

I have gone through your code and results. There seems to be something wrong with the implementation as none of your predictions give a value greater than 0.5 .

I couldn't pin down the problem. Some debugging will be needed on your side.

What's paradoxical is that your loss is reducing. I suspect this might be because your data set is unbalanced i.e., you have 221 0s and about 30 1s. This could be the reason for other problems as well.

Consider the wikipedia example, where the values are correctly matching.

If you manage to find the error or the solution conclusively, please post it here so that we can all learn.

Hope this helps!


How can collaborative filtering be extended to include more features? Looking at the following:
https://realpython.com/build-recommendation-engine-collaborative-filtering/#using-python-to-build-recommenders
I can see that userID, itemID, rating are the standard features used in a collaborative filtering model. However, my question is how can I incorporate more features into the model (such as the reviewText)?

There's a lot of different ways to embed contextual user & item features into recommendations systems. These algorithms are often called "Hybrid Recommendation Systems".
The highest performers are usually big complicated neural nets, but where I would start with is Factorization Machines.
FMs are a staple in the recsys community today, and is probably the best thing to try first if you're trying to build a hybrid recommendation model.
Some characteristics of FMs to be aware of:

They perform well with sparse data
FMs have linear complexity, so they train fast and once deployed they predict fast.
Very flexible in terms of input data, it can take any real valued feature vector.
They're much more easily interpretable than neural networks, if you know how to interpret a regression you're probably capable of figuring out what the FM is doing.
They are capable of both regression (ratings) and binary classification. In my experience you can mix implicit and explicit feedback by weighting interactions differently.
They have a very unique data model. You will notice that in the FM data matrix, there are at least 4 sections, the user-index, user-features, item-index, and item-features. You will sometimes see a interaction-feature section as well. The fundamental idea here is that each row is a record of all the data involved in the interaction between a user and an item (or interactions between whatever). The user & item features section will look more familiar, they are the features of the respective users and items. What may be new to you is the user and item indices. It's basically two separate one-hot-encodings for both the users and items, but you should only ever have 1 user and item on for any interaction. This will make more sense once you see the matrix visuals, I suggest starting at it for awhile to wrap your head around it.

Some additional resources/further reading:

Jefkine Blog
Berwyn's Blog
FastFM Python library (This is an older library, it works well and is good for out of the box performance but if this is for a long term commercial project I would suggest looking for a more modern implementation)
LightFM Python Library (Another FM Library that I personally prefer over FastFM)
RankFM Python Library (Another FM Library specifically adapted for when you only have implicit feedback and want to rank/recommend items.)


real word dataset on company full name with commonly used short name I am involved in a work where i have to recognize company when user does not provides its full legal name. Database only has full legal name which is rarely used by human user. Like no body calls google inc but they just call google. My job is to match that user provided name with database name. The approach i am thinking is to create short name of company from legal name and index that so it what user used could be directly matched.

I was thinking of writing a classifier that could be trained on real world data of full name and short names map and then used to create short names from legal names in database. Are there any such dataset currently available that i can use as input source for my trainer?

I think a better idea would be to use approximate string matching techniques: the general idea would be to compute a similarity score for each candidate term, then consider it a match if the score is higher than some threshold. This approach can be refined by  computing a set of similarity scores according to different similarity measures (see e.g. this list) and train a classifier using these scores as features. It's even possible to use more sophisticated approaches involving training a model taking into account the probability of specific edits, but it shouldn't be necessary in a simple case like this.


how bidirectional neural networks can be applied on time series while we do not know the future data? I have read about bidirectional neural networks. It seems that they need input from both past and future. so lets say we are going to predict the energy use of one hour ahead having the energy use of last 168 hours as inputs to the network. So using bidirectional RNNs it also needs the future data, which is what we are going to predict! Can you please guide me about my confusion?

Thanks

You are correct in your general understanding of bidirectional recurrent neural networks, they do utilize information from both the past and the future.

However, they are not usually used for predicting the future. Instead they are mostly used for tasks like: Speech Recognition, Translation and Handwritten-Recognition.

For these uses, the "prediction", or more generally the output of the model, is based on a global-scale (a big chunk of text like a full sentence or a paragraph), while the bidirectional behavior works on a local-scale (single words).

In simple terms, when we want our model to predict the meaning of a full sentence. We need it to understand the meaning of the specific words composing it. But in order to do that, we use the words that come before (past) and after (future) each specific word.


Dealing with informative missingness How can I deal with a time series that contains missing data which means something?

So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a loss of information.

So, you can think of it as something like "measure x only if condition y was met".
In my case a certain price is measured only if there are people interested in that on a given day.
This leads to a time series with several thousands data points, of which around 40% contains no value.

I want to use a 1D-CNN as part of an anomaly detector in order to find errors in the data. The CNN delivers predictions which are compared to the actual data. If the difference between the prediction and the real data point is above a certain threshold I want to mark that as an anomaly. (the missing_values however are no anomaly)

I thought of using embeddings to deal with missing values but am not sure if this would be the correct way of handling this.

I don't see how embeddings can help. Embeddings serve to lower the dimensionality of your data. I think you just need to introduce a single feature that is 1 if there were people interested and 0 if there were none. Then if you have sufficient data, the convnet should be able to understand the relation between the time-series and the feature you added.

It's a bit hard without having any idea about what your data looks like, but you might also be able to ignore the missing data, and train a model that predicts a continuous time series. Then the model may flag an anomaly when there is data missing, but you only actually declare an anomaly if the data is not missing and the anomaly is real.


Confusion Matrixs for Binary classifier I am new to modeling, and I am practicing building a logistic regression model. I would like to create a confusion matrix, but my code doesn't seem to work.

Here is the code for the model (which works):

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

#train, test = train_test_split(hp[age], test_size=0.3)

#from sklearn import preprocessing
X = hp['age'].values.reshape((32561,1))
#X = hp[['age','hours-per-week']].values
y = hp['evalinvest'].values

LogReg = LogisticRegression()
LogReg.fit(X,y)
print(LogReg.score(X,y))

0.916710174749

Here is where I am having diffculty:

# Confusion Matrix
import numpy as np
from sklearn.metrics import *
CM = confusion_matrix(X,y)
print ("\n\nConfusion matrix:\n", CM)

It runs and outputs results, but I don't feel like it is correct.

Confusion matrix:

 [[  0   0   0 ...,   0   0   0]
 [  0   0   0 ...,   0   0   0]
 [385  10   0 ...,   0   0   0]
 ...,
 [  1   0   0 ...,   0   0   0]
 [  3   0   0 ...,   0   0   0]
 [ 33  10   0 ...,   0   0   0]]

Then, when I run the following code, it doesn't work:

tn, fp, fn, tp = CM.ravel()
print ("\nTP, TN, FP, FN:", tp, ",", tn, ",", fp, ",", fn)

error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-68-dca3ebbdc69a> in <module>()
----> 1 tn, fp, fn, tp = CM.ravel()
      2 print ("\nTP, TN, FP, FN:", tp, ",", tn, ",", fp, ",", fn)

ValueError: too many values to unpack (expected 4)

I think you are confused on many levels here.

Logistic regression is a classifier. It gives you the probability that given an $x_{i}$, the probability it belongs to a class or classes.
So you will be having your evalinvest as classes, not continuous values.
When you call predict method on top of trained LogisticRegression model, it predicts class for each $x_{i}$ in your test set. So you expect a single array of class labels they belong to.
sklearn.metrics.confusion_matrix function takes in the original class values to the ones that are predicted by the model you had trained and returns what $x_{i}$ has been classified into what class. So it returns an n(c) x n(c) array where n(c) is the number of classes. So the diagonal of the matrix indicates the number of elements of class i being classified as class i. And an ideal model should be expected to get this numbers good. The mistake which you had made you sent in your X (train) matrix into it and y (labels) into it, which is wrong. You are supposed to send in y_true(true labels) and y_pred (predicted labels from the model). Check the documentation for more.
Ideally the function should have thrown an error. But unfortunately you had only one row in your X and one in y and they both are of same size and it passed assert it had to make.
And np.ravel converts an multi-dimensional array into 1D array. So if you doing a binary classification, you would be having a 2x2 matrix, whose flattened array would have been four elements and the assignment would have worked. But the ravelling you had made releases $n(c)^2$ elements. So there you go ValueError. See the docs

I hope my other answer on confusion matrices may clear things a little more.

Hope it clears some mistakes you are making.


Big data case study or use case example I have read lot of blogs\article on how different type of industries are using Big Data Analytic. But most of these article fails to mention

What kinda data these companies used. What was the size of the data
What kinda of tools technologies they used to process the data
What was the problem they were facing and how the insight they got the data helped them to resolve the issue.
How they selected the tool\technology to suit their need.
What kinda pattern they identified from the data & what kind of patterns they were looking from the data.

I wonder if someone can provide me answer to all these questions or a link which at-least answer some of the the questions. I am looking for real world example.

It would be great if someone share how finance industry is making use of Big Data Analytic.

News outlets tend to use "Big Data" pretty loosely.  Vendors usually provide case studies surrounding their specific products.  There aren't a lot out there for open source implementations, but they do get mentioned.  For instance, Apache isn't going to spend a lot of time building a case study on hadoop, but vendors like Cloudera and Hortonworks probably will.

Here's an example case study from Cloudera in the finance sector.

Quoting the study:

  One major global financial services conglomerate uses Cloudera and Datameer to help
  identify rogue trading activity. Teams within the firm’s asset management group are
  performing ad hoc analysis on daily feeds of price, position, and order information. Having
  ad hoc analysis to all of the detailed data allows the group to detect anomalies across
  certain asset classes and identify suspicious behavior. Users previously relied solely on
  desktop spreadsheet tools. Now, with Datameer and Cloudera, users have a powerful
  platform that allows them to sift through more data more quickly and avert potential
  losses before they begin.

.

  A leading retail bank is using Cloudera and Datameer to validate data accuracy and quality
  as required by the Dodd-Frank Act and other regulations. Integrating loan and branch data
  as well as wealth management data, the bank’s data quality initiative is responsible for
  ensuring that every record is accurate. The process includes subjectingthe data to over
  50 data sanity and quality checks. The results of those checks are trended over time to
  ensure that the tolerances for data corruption and data domains aren’t changing adversely
  and that the risk profiles being reported to investors and regulatory agencies are prudent
  and in compliance with regulatory requirements. The results are reported through a data
  quality dashboard to the Chief Risk Officer and Chief Financial Officer, who are ultimately
  responsible for ensuring the accuracy of regulatory compliance reporting as well as
  earnings forecasts to investors

I didn't see any other finance related studies at Cloudera, but I didn't search very hard.  You can have a look at their library here.

Also, Hortonworks has a case study on Trading Strategies where they saw a 20% decrease in the time it took to develop a strategy by leveraging K-means, Hadoop, and R.

These don't answer all of your questions.  I'm pretty sure both of these studies covered most of them. I don't see anything about tool selection specifically.  I imagine sales reps had a lot to do with getting the overall product in the door, but the data scientists themselves leveraged the tools they were most comfortable with.  I don't have a lot of insight into that area in the big data space.


intuition behind the difference between likelihood function of discriminative and generative algorithms I am currently studying the online material of Stanford CS229 and I came across the likelihood function for discriminative(for example, regression) and generative algorithms(for example, naive bayes):
+Discriminative:

+Generative:

In both cases, m is the number of training examples and all training examples are independent of each other. What I am wondering is why in discriminative likelihood function, the formula is the product of conditional probability of y given x and in generative likelihood function, the formula is the product of joint probability of x and y? Is there some reasoning behind this choice?

Discriminative model learn to classify an x into class y: the conditional probability distribution p(y|x).

Generative models learn the joint probability distribution p(x,y).It can be transformed into p(y|x) by a Bayes rule.

You get different likelihood functions because the way those models classify data is different.


machine learning algorithms for 2d data? I'm looking for a supervised learning algorithm that can take 2d data for input and output. As an example of something similar to my data, consider a black image with some sparse white dots. Blur that image using a full range of grayscale. Then create a machine that can take the blurred image as input and produce the original sharp image as output.

I could make some sample 1D data by taking a region/radius around the original sharp point, but I don't know the exact radius. It would be significant data duplication and a lot of guessing.

Any good algorithm suggestions for this problem? Thanks for your time.

You can simply convert a MxN matrix (your image) into a MN dimensional vector. For instance if you have 10 images each of which 10x10 pixels, you can convert them to 10 vectors each of them containing 100 element (you can do it row-wise or column-wise, does not matter). Then putting all these vectors on each other constructs a nxd matrix where n is the number of images and d is the number of pixels in each image.

So far we just prepared data for analysis. Now let's go to the question:

I'm looking for a supervised learning algorithm that can take 2d data
for input and output.

What does that mean?

First- The dimension of input-output is not a property of a machine learning algorithm.

Second- By 2d, you mean a matrix? I'm afraid your data is not really 2d! If you mean a matrix, a MxN matrix is a data of M samples in N dimensional space. also those vectors you may produce from your images are in MN dimensional space.

Third- A supervised algorithm takes some teaching samples (called training) to learn a task. What are those teachers here? If you have a training data which contains some blurred and corresponding original images then you can try to learn the place of white pixels by exploring blurred area. My suggestion:

If a smoothing filter has been applied to your image, the blurred pixels most probably look like a gaussian. So first extract the gaussians.
As you already have the place of sharp whites correspond to those gaussians you can use the mean, location of the mean and also variance as features (inputs) and the place of sharp white pixels as the target (output).

The classifier could be a simple neural network (a MLP should work here) or a simple SVM.

The answer above can not be evaluated without more information about your data and a more specific description of the question (e.g. What are your images exactly? if you can post one of them here I can probably help more)


Prove data points not enough? Estimate required number of data points? I am working on a problem with only ~ 30 data points, I need to use these 30 points to build a machine learning model (ridge regression, random forest ... etc.), then use it to predict on another thousands of data points.

After some experiments, I believe the data set is too small. However, I am wondering is there a standard approach to prove that whether we have enough data points? And how to determine how many data points are needed? Thanks!

It depends on what you mean by "needed". More training data = lower variance with the same model complexity. So when you have a small training dataset, your models will have to be very simple (like linear regression), so to have best generalization. And very simple models tend to badly represent reality.

The answer is: you need so many points, such that your validation error is below desired threshold. 30 is probably much too low for any more complex machine learning model than linear regression with one, maybe two features.


What does these points mean in Naive Bayes? I have two concept related questions related to Naïve Bayes.

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

Regards, Akshit Bhatia

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

Imagine a classifier for sentiment analysis. For a strongly positive word like $w=great$, the conditional probability $p(w|pos)$ is going to be quite high whereas $p(w|neg)$ is going to be quite low, so the posterior $p(pos|d)$ for a document $d$ containing this word is likely to be much higher than $p(neg|d)$.

Now what happens with a neutral word $w=today$? Neither $p(w|pos)$ or $p(w|neg)$ is going to be much higher than the other. So all other things being equal, the difference between the two posterior probabilities is not going to depend much on this word compared to other more relevant words, for instance "great".

  It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

In practice the independence assumption almost never holds with real data. For example words in a text actually depend on each other, that's how they make sense together in a sentence. This is true for entities like "Hong Kong" but also for virtually any sentence. For instance "I love chocolate but you hate it" doesn't mean that same as "You love chocolate but I hate it", or "it chocolate you hate love but" which means nothing. NB will treat all these variants the same way: basically the model assumes independence for the purpose of making things simpler and easier to compute, and it turns out that it works quite well in general, despite the massive simplification.


Can we use boosting algorithms like Adaboost and gradient boosting with only one classifier I have been working on ensemble learning and I came across this doubt that unlike other ensemble learning algorithms like voting classifier a can we only use one classifier with boosting.

Boosting typically only use one algorithm as it's base learner (almost exclusively decision trees). However, you could use a mixed set of algorithms as your base learners.

Something like this:

Boosting round 0: Add decision tree
Boosting round 1: Add neural network
Boosting round 2: Add KNN
Boosting round 3: Add decision tree
...

The reason you only see boosting using the same algorithm is probably just because it works better. I speculate that the diversity that comes from using several algorithms shine more when they are trained in parallel and combined. In boosting the base learners are trained in a sequence.


How to check quality of latent space like in β-VAE article? There is a nice plot in the β-VAE article that shows quality of latent space code:

Is there a general way to visualize or analyze latent space code dimensions so that is would be clear if they are too entangled or a mess?

UPD:

  The dataset consists of Gaussian blobs presented in various locations on a black canvas. Top row: original images. Second row: the corresponding reconstructions. Remaining rows: latent traversals ordered by their average KL divergence with the prior (high to low). To generate the traversals, we initialise the latent representation by inferring it from a seed image (left data sample), then traverse a single latent dimension (in [−3, 3]), whilst holding the remaining latent dimensions fixed, and plot the resulting reconstruction. Heatmaps show the 2D position tuning of each latent unit, corresponding to the inferred mean values for each latent for given each possible 2D location of the blob (with peak blue, -3; white, 0; peak red, 3).

This means that to generate such heatmaps one should be able to smoothly move the blob while knowing it position. So one should actually know what is the best latent encoding. And the plot is actually a comparison with this best latent encoding.

I'm curious if there is a way to measure or plot "something" that would help to understand the quality of the latent code when I don't know what is the best latent code.

Not that I know of. Notice that those heat maps rely on the human knowing what they want, and then devising this heat map approach to visualizing it.

For many papers interested in latent generative disentanglement, the most popular way to "check" the quality is via interpolating in the latent space and visualizing the output, seeing it across each disentangled dimension. One can then "see" whether they are indeed disentangled or not.

Ideally there is some other way to quantify this disentanglement, but it usually requires additional information. For instance, for images, one might have labels that one can use (e.g., brightness, person identity). Or, as in the $\beta$-VAE paper, one can use a toy problem where the underlying latent factors are known. Then one can devise a way to check whether one has actually separated those latent factors. Ultimately evaluation depends on the data and task.

If one just wants to visualize the latent space, usually these interpolations plots are popular. But it is qualitative. Often a quantitative evaluation of the latent space requires an additional downstream task.


Cluster very many sparse binary vectors I have a very big set of high-dimensional, but sparse binary vectors. Each vector represents a "one-hot-style" n-gram sequence of words where each index of the words that occur in the n-gram is set to 1, all others to 0. E.g.

[0,0,0,0,0,1,0,0,0,0,1,0,0,0,...]

An efficient representation for a 5-gram:

[5,10,235,1253,5521]

What I would like to do is generate such n-gram representations from a large text corpus. Hence, this may easily result in a vocabulary size of hundreds of thousands (depending on potential pre-processing steps including stopword removal, stemming, etc.) and even more distinct n-grams (m).
This could be represented as a matrix of size m x vocab_size.

Subsequently, I would like to reduce the number of rows by clustering similar n-grams together while retaining the contained words. For instance:

[5,7,10,12,15],[5,8,10,15,18] -> [5,7,8,10,12,15,18]

My naive approach would be to apply some traditional clustering algorithm in order to assign each entry in the initial n-gram set to a specific cluster and then perform a union over all the n-grams assigned to a cluster.

1st question: what would be an sensible algorithm for clustering sparse binary representations?
2nd question: how could this be done with a very big data set?

I don't think clustering is the right approach here. You want to merge them automatically, but clustering is too unpredictable for this; you need to manually examine clustering results.

I suggest you look at near duplicate detection instead. You want to merge documents that have a high overlap. There are some interesting - and scalable - algorithma for that. In particular, minhash (and later work) is worth looking at.


Fastest way to relearn machine/deep learning I hope I came to the right place to ask this question.

Back when I was at collage I studied machine and deep learning in-depth. My whole programme was based on those areas. I knew all underlying maths, even today I know how to derive backpropagation for any feed-forward network. Well, maybe I would need to take a peek. But I still understand the math and I can follow without problems. Back then (2017) I was even doing something with and researching GANs which were novelty back then.

Basically, I was at the hotspot at that time. I was working with all sorts of algorithms, from logistic regression, SVMs, to MLPs, CNNs, RNNs (mostly LSTM) and was trying already mentioned GANs. Oh, heuristics too: GAs, tabu search, simulated annealing, etc. There was also some NLP involved too.

And then after college I went to gaming industry, heh. I was/am still, during that time, working with ML/DL (and some OpenCV) but mostly easy, toy projects (although one project was real life project, but it was easy, I had to extract written digits from paper and classify them).

So, my question is, what is the best (and fastest) way to get back on the track considering my, I would say, pretty strong background? I saw that Kaggle has some courses on their site, for example course on DL is estimated 4 hours, course on feature engineering is also 4 hours. That is not a lot of time, but I am afraid it would be too easy for me and consequently a waste of time.

What are some good resources to refresh/relearn ML/DL considering my background?

If you enjoy reading, that's probably the best ML/DL book ever.


How to calculate ideal Decision Tree depth without overfitting? What would be a good way to go around finding the best depth for a DecisionTree (in SKLearn)? How can I tell if I've gone too deep and am overfitting? I know I can find the best parameters with f.e. GridSearchCV, but the best score might not mean I get the best classifier as I may overfit the tree to the data.

No! the best score on validation set means you are not in overfitting zone. As explained in my previous answer to your question, overfitting is about high score on training data but low score on validation. So to avoid overfitting you need to check your score on Validation Set and then you are fine. There is no theoretical calculation of the best depth of a decision tree to the best of my knowledge.

So here is what you do:

Choose a number of tree depths to start a for loop (try to cover whole area so try small ones and very big ones as well)
Inside a for loop divide your dataset to train/validation (e.g. 70%/30%)
Each time train your decision tree with that depth on training data and test it on the validation set, then keep the validation error (you can also keep the training error)
Plot the validation error (you can combine it with evolution of training error to have a prettier plot for understanding!)
Find the global minimum of validation error.
Then you can narrow your search in a new for loop according to the value you found to reach a more precise value


How do I choose number of clusters when Eigengap heuristic suggest 1 for spectral clustering? Eigengap heuristic Method suggest number of clusters k is usually given by the value of k that maximizes the eigengap (difference between consecutive eigenvalues).
I plotted the Eigenvalue distribution for my Data:
[

Eigengap heuristic would suggest cluster number: 1.
How should I choose number of clusters in this case ?

The first eigenvalue in your plot is non-zero, whereas the last is one.
This seems to suggest that the Laplacian whose eigenvalues you are computing is different from the standard Laplacian $ L = D - W$, where $D$ is the matrix whose diagonal entries are degrees of nodes and $W$ is the weighted adjacency matrix.
So in your plot, $\lambda_{19} = 1$ and $\lambda_{18} \neq 1$ but is $\approx 1$  which is indicative of the graph being one connected component.
So the correct way to interpret your plot would be to invert it ($\lambda \mapsto 1 - \lambda$) and then arrange them in increasing order, or to arrange them in decreasing order as-is.
Then you can use the eigengap heuristic to choose a suitable value for the number of clusters (7 seems to be a good choice as $\lambda_{13}-\lambda_{12}$ is large compared to $\lambda_{14}-\lambda_{13}$, or maybe 3 if the number of clusters is a constraint).
This is described in greater detail in Luxborg's excellent tutorial on Spectral Clustering, and you can find a footnote in Ng et al's seminal paper on the same.


Having and issue plotting horizontal chart So for some weird reason I can't manage to fix the plotting issue

Any suggestions?

from sklearn.metrics import confusion_matrix

List = []
for i in range(len(model)):
  cm = confusion_matrix(Y_test, model[i].predict(X_test))

  TN = cm[0][0]
  TP = cm[1][1]
  FN = cm[1][0]
  FP = cm[0][1]

  List.append(((TP + TN) / (TP + TN + FN + FP))*100)

  print()

import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
bars = ('Logistic', 'K Nearest Neighbor', 'K Nearest Neighbor', 'Support Vector Machine ', 'Support Vector Machine ','Support Vector Machine ','Support Vector Machine ' )
percentage = np.array([List[0], List[1], List[2], List[3],List[4],List[5],List[6]])

new_labels = [i+'  {:.2f}%'.format(j) for i, j in zip(bars, percentage)]

plt.barh(bars, percentage, color='lightskyblue', edgecolor='blue')
plt.yticks(range(len(bars)), new_labels)
# Show graphic
plt.show()

You have to change one line in your code.

This

plt.barh(bars, percentage, color='lightskyblue', edgecolor='blue')

should become this

plt.barh(range(len(bars)), percentage, color='lightskyblue', edgecolor='blue')

What you are changing is the y position of your bars. If you look at your original bars there are only 3 values there, so bars were overlapping.

Once you change that to  range(len(bars)) you are using the same tick values that you set in plt.yticks(range(len(bars)), new_labels).


Model building with neural networks Assume the existence of a collection of physical parameters and a collection of output variables which may depend on the physical parameters. An example in the training dataset consists of a vector with the actual measurement of the physical parameters, and another vector with the output values which were measured.

To my understanding, I can train a neural network such as a MLP to learn the mapping from the measured input parameters to the measured output values. If a model could be learned and assuming the learned relations are due to causality and not some unfortunate correlation, then for an arbitrary input vector the model could predict the vector of output values. Note that
the relationship between input and output is not a 1:1 map, rather n:1. Different parameter choices may thus lead to the same output.

I want to go one step further and define an interval for each output value. I then want to know how I must set the ideal input parameter values to ensure that the output values are in the corresponding intervals, i.e. how to alter inputs for the output values to be within the ranges. I tried feeding the output values as input and trying to predict the physical parameters, but I have problems due to the ratio input/output. I have 130 physical parameters and only 10 output variables. How can I achieve this with a neural network?

I would assume that the physical input parameters occur in some sort of sequence & the Neural Network can easily identify patterns from it.

I would suggest you do some scaling on the output parameters. Or use non-linear activation functions so that your output ranges from upper bound & lower bound of that function.

Also i would like to suggest that you use LSTM. Recent time-series application using LSTM have promising outputs. Their application ranges from text generation to pixel filling in images & videos. They can be used to get bounded outputs & using torch neural networks allow you to implement them as per your requirements.

You can find implementation guide for Neural Network using torch library here


Can we use sentence transformers to embed sentences without labels? I was trying to use this project :
https://github.com/UKPLab/sentence-transformers
for embedding non english sentences, the language is not a human speaking language, its machine language (x86)
but the problem is i cannot find a simple example where it shows how can i embed sentences using a custom dataset without any labels or similarity values of the sentences.
basically i have an array of sentences lists without any labels for sentences or similarity values for them, and i want to embed them into vectors in a way that it preserves the semantic of the sentence the best way possible, so far i have used word2vec and doc2vec using gensim library so i wanted to try this method to see if its any better?

The link you provided of Siamese Bert is an instance of a Bert or Roberta finetuned on STS or NLI data. Which can have the format sentence 1 is similar 3 out of 5 to sentence 2 (STS). Hence, is supervised, it does not fit your purpose.
Nonetheless, do not despair, there are some that do not require training, although may not perform as good as the supervised one. The below use word embeddings which you can train on your data corpora to generate sentence embeddings:

Word Mower's Distance
Sentence Embedding S3E

Or by feeding just sentences line by line:

DeCLUTR
Sent2Vec

P.S. I have not tried all of the solutions, to my knowledge I suggest these, cause either they are quite known or are quite recent.


What is wrong with my logistic regression implementation? Recently, I implemented the LR algorithm in Python. The main part of the code is as following(I didn't use mini batch in my code. Instead, I use the whole batch to compute gradients every time):

class Logistic():
def __init__(self):
    self.w = None
    self.lr = 10.
    pass
def train(self, xs, ys):
    m, n = xs.shape
    ones = np.ones([m, 1])
    xs = np.hstack([xs, ones])
    ys = np.expand_dims(ys, -1)
    self.w = np.ones([n+1, 1], dtype=np.float64) * 1.0
    epochs = 100
    for epoch in xrange(epochs):
        y_ = self.sigmoid(-np.dot(xs, self.w))
        # loss = -1.0/m * np.sum(ys * np.log(y_) + (1 - ys) * np.log(1 - y_))
        tmp1 = np.sum(np.log(y_[np.where(ys==1)]))
        tmp2 = np.sum(np.log(1 - y_[np.where(ys==0)]))
        loss = - (tmp1 + tmp2) / m
        print("epoch: %d, loss: %f" % (epoch, loss))
        print("y_: %f, %f" % (np.min(y_), np.max(y_)))
        grad =  np.sum((y_ - ys) * xs, axis=0) / m
        self.w -= self.lr * np.expand_dims(grad, -1)
        print("grad: %f, %f" % (np.min(grad), np.max(grad)))
        print("w: %f, %f" % (np.min(self.w), np.max(self.w)))
        print ""

The dataset I used is MNIST. I marked all digits 0 as class 0, and all other digits as class 1. Then I get this binary classification problem. I test my algorithm with many different learning rate, from 1e-6 to 10, and it turns out all of them produces good results(about 98% accuracy on test set). As far as I know, if the learning rate is to big, LR will not converge. But here although I used very big learning rate, the algorithm still converge to about 98% accuracy. Is there an explanation for this?

What is the ratio between 0's and the other numbers? If the ratio is too low, let's say 2%, if you classify everything as not 0 you will have high accuracy.

EDIT AFTER COMMENTS

Imagine there are 98 non zeroes and 2 zeroes, you don't need any classifier to have a 98% of accuracy...

Accuracy is not the best metric for unbalanced data-sets, let's try precision and recall and take a look to the confusion matrix


How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas? How do you see in pandas the element of a csv table with many columns (>25) which the names of its columns is more than 10 character?
I have 5000 rows and 32 columns and the label of some columns are more than 10 characters.
How I ca see them and work with different columns?
Excel does not work! All of the items are sloppy
Access is OK but could not detect the long labels of items!
What is your solution for it?

I dont know which compiler you are using but mine is jupyter.

You can make wider notebook by:

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

By doing that you can display more columns with df.head(). If you cant display still ("..." between columns), you can use iloc. For you data I used df.iloc[:5,:15] and df.iloc[:5,15:].

Also which is a better solution you can change pandas display options by:

pd.set_option('display.max_columns', 500)

Here is the screenshot of my notebook:


to generate consistent encoding for words in Keras using tf.keras.preprocessing.text.one_hot I am using keras(tensorflow) to convert text into encodings using tensorflow.keras.preprocessing.text.one_hot
I have used it for training dataset as below
from tensorflow.keras.preprocessing.text import one_hot

corpus = ['nice app']
onehot_repr = [one_hot(words, 10000) for words in corpus]

print(onehot_repr)
# [5779, 2969]

It's ok upto this point.
But when I use the one_hot for my testing set it generates different encoding.
I have created a Flask API to test, So how can use same encoding for both train and test set
Result from API is :
[[5129, 4965]] for same text ['nice app']

Keras' one_hot function has many limitations. The biggest issue is that the function does not actually do one hot encoding, it does the hashing trick.
One possible fix is to use keras' hashing_trick function. It allows the hashing function to specified. If you pick a stable hashing function like md5, then the values will be consistent across runs.
Here is an example:
from tensorflow.keras.preprocessing.text import hashing_trick

corpus = ['nice app']
text_hashed = [hashing_trick(text=words, n=10_000, hash_function='md5') for words in corpus]
assert text_hashed == [[9146, 6067]]


optimizing a linear optimization function with linear constarints and binary variables I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization.
I have an ojective function that needs to be maximized
def objective(bat1,bat2,bat3,bat4,bat5,bat6,bat7,wk1,wk2,ar1,ar2,ar3,ar4,ar5,bowl1,bowl2,bowl3,bowl4,bowl5,bowl6):
    total_score_batsman =  bat1*60 + bat2*40 + bat3*36 + bat4*35 + bat5*25 + bat6*22 +bat7*9
    total_score_wks = wk1*24 + wk2* 14
    total_score_ar = ar1*45 + ar2*24 + ar3*15 + ar4*1
    total_score_bowler = bowl1*64 + bowl2*47 + bowl3*16 + bowl4*7 + bowl5*5 + bowl6*4
    return total_score_batsman + total_score_wks + total_score_ar + total_score_bowler #needs to be maximized

constraints
#budget constraint

def budget(bat1,bat2,bat3,bat4,bat5,bat6,bat7,wk1,wk2,ar1,ar2,ar3,ar4,ar5,bowl1,bowl2,bowl3,bowl4,bowl5,bowl6):
    batsman_budget = bat1*10.5 + bat2*8.5 + bat3*10.5 + bat4*8.5 + bat5*9.5 + bat6*9 +bat7*9
    wk_budget = wk1*8.5 + wk2*8
    ar_budget = ar1*8.5 + ar2*9 + ar3*8.5 + ar4*8
    bowler_budget = bowl1*9 + bowl2*8.5 + bowl3*8.5 + bowl4*8.5 + bowl5*9 + bowl6*9
    total_budget = batsman_budget + wk_budget + ar_budget + bowler_budget
    return total_budget

total_budget <= 100 #constraint

#player_role constraints

bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 >= 3
bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 <= 5

wk1 + wk2 = 1

ar1 + ar2 + ar3 + ar4 + ar5 >= 1
ar1 + ar2 + ar3 + ar4 + ar5 <= 3

bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 >= 3
bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 <= 5

# no of players in a team constraint
bat1 + bat2 + bat3 + bat4 + bat5 + bat6 + bat7 + wk1 + wk2 + ar1 + ar2 + ar3 + ar4 + ar5 + bowl1 + bowl2 + bowl3 + bowl4 + bowl5 + bowl6 = 11

where bat1,bat2,bat3.....bowl5,bowl6 are 0 or 1

Its completely a linear problem, and there is no requirement of non linear optimization techniques.Can some one help me with how to solve such problems or are there any libraries in python which will help me solving this?
Thanks

Use scipy.optimize.minimize of the negative objective function.
Use the link to play and modify as you wish but here is application in the nutshell.
Lets say you have your data
> array([[ 0.00749589,  0.01255155,  0.02396251,  0.04750988, 0.09495377]
>        [ 0.01255155,  0.02510441,  0.04794055,  0.09502834,  0.18996269],
>        [ 0.02396251,  0.04794055,  0.09631614,  0.19092151,  0.38165151],
>        [ 0.04750988,  0.09502834,  0.19092151,  0.38341252,  0.7664427 ],
>        [ 0.09495377,  0.18996269,  0.38165151,  0.7664427,   1.53713523]])

You should define your constraints dictionary in the following way (for example):
> cons = ({'type': 'eq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},
> {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},
> {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})

(accesing array/columns with indexing (x[0]) thats exactly your bats, bowls etc columns).
Next just pass your negative objective function and you maximized it.


Using Generative Adversarial Networks for a generation of image layer Has anybody seen any application that would use GAN that would take input image and would output image of the same size, that could be used as a layer for the first image. The layer would contain in eg. point of interest in the input image. Would that even be a good practice of the GAN usage?

I am looking for articles, examples of application that used something similar.

In terms of generating an image "layer", that is just the same as generating an output image that can be overlaid on the input using standard graphics software. If you want pixel-level accuracy in the output then the output will need to be the same size as the input, otherwise it could be smaller, provided it is the same aspect ratio, and in which case it would need to be scaled up in order to be used as an overlay. In any case, as the output of a GAN can be an image (and often is), then this part is easy.

The "G" in GAN stands for Generative. The purpose of a generative network is to create samples from a population, where there are typically many possibilities. Those samples can be conditioned on some additional data, and that additional data could be an image, although many examples will simpler conditioning, such as a category that the training output is representative of.

One possibility for using a GAN, is where your population contains a range of traits, and you can calculate vectors that control that trait. So you can take an input image, reconstruct it in the GAN then modify it by adding/subtracting the trait-related vector. An interesting example of this is Face Aging With Conditional Generative Adversarial Networks, and similar examples are around of adding/removing glasses etc. For this to work for you, you would literally need images that had your points-of interest in them and ones without them, and then you would be able to control addition/removal of points-of-interest. The network would not detect these points in the input, instead it would add them into the output. From reading your question this does not seem to be what you want.

A similar paper uses a GAN to remove rain from photos, based on training many images with and without rain then learning the "rain vector", encoding new images with rain in them into the GAN's internal representation and subtracting this "rain vector".

GANs conditioned on input images (as opposed to categories or internal embeddings) are also possible - this example of image completion might be closer to your goal. If your points of interest are variable with many options feasible, then it could work for you.

However, if your points of interest are supposed to always be the same pixels in each image, then your goal might be better defined by strict ground truth and become more like semantic segmentation, which can be attempted with variations on CNNs, such as described in this paper by Microsoft. These are much easier to set up and train than GANs, so if you can reasonably frame your problem as pixel classification from the original image, this is probably the way to go.


Why is my loss blowing up after adding regularization I tried to add L2 regularization to a network class I wrote however when I train it the loss blows up even though accuracy also increases. Can someone explain where I am going wrong? (I am using the formulas from here)
The update to minibatch (The (1-eta*(lmbda/n)) coefficient to w is what I added)
def update_mini_batch(self, mini_batch, eta, lmbda, n):
    # n is the number of training samples being trained from
    # Turn the mini_batch with one dimensional samples into two matricies and then transpose them to get samples in columns
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*mini_batch)]

    gradient_b, gradient_w = self.backprop(matrix_x, matrix_y)

    self.bias = [b-(eta/len(mini_batch))*db for b, db in zip(self.bias, gradient_b)]
    self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*dw for w, dw in zip(self.weights, gradient_w)]

The function that evaluates cost (I am using Quadratic Cost)
def evaluate(self, data, lmbda):
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*data)]
    output_matrix = self.feedforward(matrix_x)

    cost = self.cost_func.apply(output_matrix, matrix_y)
    #L2 Regularization
    cost += lmbda/(2*(matrix_y.shape[1])) * sum(np.linalg.norm(w)**2 for w in self.weights)
    acc = np.sum(output_matrix.argmax(axis=0)==matrix_y.argmax(axis=0))

    return cost, acc

An example of my cost and accuracy during training
Epoch 0 done! Cost: 11.938649143175008. Accuracy 7397 / 10000
Epoch 1 done! Cost: 16.017232330762045. Accuracy 7381 / 10000
Epoch 2 done! Cost: 21.62351585060393. Accuracy 7431 / 10000
Epoch 3 done! Cost: 30.96422767377938. Accuracy 7498 / 10000
Epoch 4 done! Cost: 45.75409202821266. Accuracy 7669 / 10000
Epoch 5 done! Cost: 67.47752609972852. Accuracy 7691 / 10000
Epoch 6 done! Cost: 97.56030814767621. Accuracy 7574 / 10000
Epoch 7 done! Cost: 133.3273570333546. Accuracy 7503 / 10000
Epoch 8 done! Cost: 174.7085211732363. Accuracy 7341 / 10000

After running this for longer the cost still increases continually and no change in eta or lambda changes this fact. Once thing I noticed was that the individual MSE error was behaving normally and it was just the magnitude of the weights that was increasing.

When you use regularization, your loss will be larger because you add the regularization term. So, it is normal if your best loss without regularization is lower than your best loss with regularization.
However, adding regularization should not affect convergence in the long-term. Meaning that even though your overall loss might be larger, it should decrease after some epochs.
I see that it increases for the first few epochs. I would suggest running your model for longer and see how the loss behaves. If an epoch takes long, use a small chunk of your dataset to investigate the behaviour of the loss. It may be possible that for the first few epochs, the loss doesn't start converging quite yet.


How to solve the gradient descent on a linear classification problem? I have a problem which i have attached as an image.

Problem is in image attached

what I understand

error function is given by: $e(y, \hat y)=0$ if $y \cdot a(x-b) \ge 1$ or $e(y, \hat y) = 1-y\cdot a\cdot (x-b)$ if $y a(x-b) < 1$.

Gradient descent at current $t$ is (1,3).

Gradient Descent($E_{in}(a,b)$) as per definition should be equal to partial derivative of the equation of $E_{in}(a,b)$ wrt $a$ and $b$. ($w$ is equivalent to $[a, b]$, according to me)

My doubt

Please note that when i say sum over N points, i mean to use that greek symbol used for summing of N points

I am not sure how would I calculate the partial derivative in this case. When we say to fix '$a$' and vary '$b$', does it mean to find differentiation only wrt '$b$'? Which would mean that gradient($E_{in}$)= $-1 / N (\sum_{i=1}^N y_i a)$. But this removes dependence on $x$ and that's why I doubt my approach.
The final equation for whom derivative needs to be done is:
$1/N \sum_{i=1}^N 1-ya(x-b)$ for N points misclassified. But as per the dataset, no point is misclassified as the error function for each point with the (a,b)=(1,3) is equal to 0.

For point 1, where x=1.2 and y = -1, as $y \cdot a \cdot (x-b) =
(-1)\cdot (1)\cdot (1.2-3)=+1.8$. This means that $e(y_1, h(x_1)) = 0$.

So what should be the answer to this question?

I hope my doubt in the question is cleare to audience. But in any case please let me know if there is something that I am unable to explain.

It appears to me that your gradient calculation is off:

$$  \frac{\partial E_{in}(a,b)}{\partial{b}} = \frac{1}{|D|} \sum_i \frac{\partial e(y_i,h_{a,b}(x_i))}{\partial b} $$

Where (with some lenient notation) :

$$  \frac{\partial e(y_i,h_{a,b}(x_i))}{\partial b} = \frac{\partial e(y,s)}{\partial{s}} * \frac{\partial h_{a,b}(x_i)}{\partial{b}} $$

We can show that :

$$ \frac{\partial e(y,s)}{\partial{s}} = -y $$ if $$1-y.s >0$$ and, otherwise: $$ \frac{\partial e(y,s)}{\partial{s}} = 0 $$

This give you an expected dependency on x, as the value change the criterion.

And :

$$ \frac{\partial h_{a,b}(x_i)}{\partial{b}} = -a $$

So you have :

$$  \frac{\partial E_{in}(a,b)}{\partial{b}} = \frac{1}{|D|} \sum_i   \mathbb{1}_{y.x<1}*y*a $$


How to consume single piece of text for classification in a model? Here's a high-level blueprint of my model -

Input data::

CATEGORY    SERVICE     TITLE                               DEPARTMENT

apple       fruits      i love eating fruits.               fruitshop
mango       fruits      mangoes are yellow in color.        fruitshop
cycle       vehicle     that cycle is really expensive.     motorshop

I convert this to a sparse matrix with Tf-Idf scores, where the columns are the title columns tokenized.

I am using LinearSVC and StratifiedKFold (n_splits = 10, random_state=777, shuffle=True) able to achieve a prediction accuracy of 73%.

Got 2 doubts -

1) What happens when a string like "fresh fish" is used for classification?
Because the words "fresh" and "fish" are never used before, which department will this be classified into?

2) How can I consume this string "fresh fish" in my model?

Currently, I have something like this -

...
# X.shape = (1181, 1930)
# y.shape = (1181,)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)
...
pr = mod.predict(X_test)
print(pd.DataFrame(confusion_matrix(y_test, pr)))
...

I tried giving input as print(mod.predict([["fresh fish"]])) but got an error -

ValueError: X has 1 features per sample; expecting 1930

Please advise. Thanks.

You have to repeat the same preprocessing steps for prediction as you did during training. Let's say you trained like this:

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)

vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X_train)

mod = LinearSVC()
mod.fit(X_train, y_train)

Then you need to use that fit TfidfVectorizer to transform your test prediction as well (which you seem to have managed). Like this:

X_test = vectorizer.transform(X_test)
pr = mod.predict(X_test)

And the same goes if you just want to feed in a manual text:

X_manual = vectorizer.transform(['fresh fish'])
pr = mod.predict(X_manual)


Collinearity and Outlier Removal I am playing with a credit fraud detection dataset at Kaggle. An imbalanced dataset with about 0.1% of fraud transaction. The features are 28 PCs out from a PCA exercise done by the data publisher + time & txn amount and a class variable of 0/1 for legit/fraud txn.

From my brief understanding, collinearity should have been dealt with during PCA. However, I found that PCs are still correlated among the fraud cases (if you break the dataset up into legit/fraud cases). What should be a good approach to minimise that effect for fraud detection using a Naive Bayes classifier?

Another thing is that I have been taught in DS101 to deal with outliers. However, I don't seem to think removing outliers is a wise choice given that fraud can be an outlier by itself. What are some common approach to deal with outliers while not removing them?

P.S. I am fairly new to Data Science so any good directions on the above topics would be welcome. It is just not as clear cut as I have seen in the introductory text.

Thanks.

In general machine learning algorithms, if feeded with large training datasets are able to deal both with outliers and multicollinearity. PCA is a dimension reduction techniques and surely helps with multicollinearity. Naïve Bayes assumes independence of its input features (the word naïve comes from this property). So after PCA Naïve Bayes has more chance to get better results.
If you find that PCs are still correlated among the fraud cases I don't think it is an issue. However, you could try to preprocessing your data removing highly correlated variables according to some criteria. The caret library has many functionalities for preprocessing and this tutorial cover interesting stuff besides software applications.

https://topepo.github.io/caret/pre-processing.html

About outliers I am always skeptic about any outlier removal except in the case could be due to miscoding.


Reading meters with tensorflow I'm new to ML world and been reading about ML and TensorFlow.
My goal is to read the following example in real time with Android phone:

So I tried firebase ML OCR and it works really good, it reads the complete value but it does not read the decimal point and also reads a lot of the surrounding text.
So my idea is that I should first detect black and red bounding boxes and then detect individual numbers inside

is this the right way to go? How would I accomplish this?
Also how do you use two kinds of a model, one to extract a part of the image (black and red bounding areas) and then pass them to OCR model?
What about last digit which can always be in between two numbers (example: 1 and 2)?

Two options :

Use pre-built libraries for OCR + Bounding Box detection   (E.g.: https://www.pyimagesearch.com/2018/08/20/opencv-text-detection-east-text-detector/ for Bouding Box detection and then OpenCV / Tessract for OCR)

Train a Deep learning model for Text Detection in scene . Examples : https://github.com/qjadud1994/CRNN-Keras  and https://github.com/mvoelk/ssd_detectors


Predicting future data with date I have a data as below:

Date                                 Quantity   Rate      Value
01-Jan-18   0.5HP Motor Xcite        9 nos      2277.82   20500.39
02-Jan-18   0.5HP Pump Booster       1 nos      11648.30  11648.30
03-Jan-18   0.5 HP Trashmasher       1 nos      3390.70   3390.70

I want to predict the future value of the quantity/rate/value based on particular date for the given particular.  Which model can be used to predict the future value for the given particular for a given date in the future? (I am a R developer and just started and I don't know Python)

You could use R to apply time series analysis to your data, i.e. as described here.


Bayes Classifier as a general model Is it correct to argue that the Bayer Classifier is an ideal classifier, which is taken as a model by every other implemented classifier?

Yes, this is correct, the Bayes Classifier is in fact theoretically optimal too. You can quite easily verify this, this is a link to the proof for the answer (https://cs.stackexchange.com/questions/72295/showing-that-bayes-classifier-is-optimal)


Single image feature reduction at inference time : SVM I am trying to train a SVM classifier using scikit-learn.. At training time I want to reduce the feature vector dimension.
I have used PCA to reduce the dimension.
pp = PCA(n_components=400).fit(features)
features = pp.transform(features)

PCA requires m x n dataset to determine the variance. but at the time of inference I have only single image and corresponding 1d feature vector.. I am wondering how to reduce feature vector at inference time in order to match the training dimension.
Or if anyone can suggest some other dimension reduction technique that can be used with single image wiil be highly appreciable.

Just like all other Class, you have  different methods i.e.fit, transform, fit_transform

fit(self, X[, y]) Fit the model with X.
fit_transform(self, X[, y]) - Fit the model with X and apply the dimensionality reduction on X.
transform(self, X) - Apply dimensionality reduction to X. X is projected on the first principal components previously extracted from a training set.

So, you fit on the data received until the current data and then use transform on the current data.
It's similar to the way we will do Scaling in online learning e.g. You fit your StandardScaler on train data and transform the new data on that.


Validation showing huge fluctuations. What could be the cause? I'm training a CNN for a 3-class image classification problem. My training loss decreased smoothly, which is the expected behaviour. However, my validation loss shows a lot of fluctuation.

Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Additional info: I'm fine-tuning the last layer of a Resnet-18 that was pre-trained on ImageNet data in PyTorch. I must note that I'm using a weighted loss function for the training phase since my data is highly unbalanced. To plot the losses, however, I use the unweighted loss as to be able to compare the validation and training loss. I would use the unweighted loss, were it not that the distributions of the training dataset and validation dataset are somewhat different (they are both highly unbalanced however).

From simple inspection of your plot, I could make a few conclusions and list things to try. (This is without knowing any more about your setup: training parameters and model hyperparameters).

It looks like the loss is decreasing (put a line of best fit through the validation loss). It also looks like you might be able to train for longer to improve results, as the curve is still headed downwards.

First I will try answer your title question:

  what is the cause of the fluctation in the validation loss?

I can think of three possibilities:

Regularisation - to help smoothing the learning process and make the model weights more robust. Adding/increasing your regularisation will prevent large updates to weights being introduced.
Batch size - is it relatively small (e.g. < 20?). This would mean that the measured mean error at the end of the network is computed using only a few samples. With a batch size of, say 8, then getting 4/8 correct and compared to getting 6/8 correct has a large relative difference when looking at the loss. Taking the mean of the errors with such small batches will lead to a not-so-smooth loss curve. If you have enough GPU memory/RAM, try increasing batch size.
Learning Rate - might be too large. This is similar to the first point regarding regularisation. To make smoother improvements, you might need to slow down the pace of learning as you approach a loss-minimum. You can make this perhaps run on a schedule, whereby is is reduce by some factor (e.g. multiply it by 0.5) every time the validation loss has not improved after, say 6 epochs. This will prevent you from taking big steps and then maybe overshooting a minumum and just bouncing around it.

Specific to your task, I would also suggest trying to perhaps unfreeze another layer, to increase the scope of your fine-tuning. This will give the Resnet-18 a little more freedom to learn, based on your data.

Regarding your last question:

  Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Should you be worried? In short, no. A validation loss curve like yours can be perfectly fine and deliver reasonable results; however, I would try some of the steps I mentioned above before settling for it.

Should you just pick the best performing model? If you mean taking the model at its point with best validation loss (validation accuracy), then I would say to be more careful. On your plot above, this might equate to around epoch 30, but I would personally take a point that has trained a little more, where the curve gets a little less volatile. Again, after having tried some of the steps outlined above.


Adding feature leads to worse results I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables.

I compare the results of a GBM model (using python xgboost and light GBM) and I found that it doesn't matter what are the hyper-parameters of the model the 'thinner' version leads to better results (AUC) even that all 20 original variables are included in the wider version.

when I compare the same using Lasso model - the wider version is better (~1% higher) as expected.

I guess it can be related to to the randomness in the GBM but I was surprised to see that GBM doesn't fix it along the way.

Any explanation of the phenomena will be appreciated.

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect.

Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the  increased complexity overweight the added value from extra features.

I understand you‘ve tested quite wide range of hyper-parameters and enough combinations. If you apply regularisation via colsample_bytree and/or colsample_bylevel it might happen that at some stage the randomly chosen columns (features) are less informative than your original features and the algorithm is forced to use these features for further splits. Does it make sense to you?

The number of added features might be crucial, if it is too high relative to original 20 the new features become just too dominant. For example, this might happen if one adds some nominal features with high cardinality, which are then dummy-encoded.

In order to improve the results with wider data you might want to play with the parameters controlling the early stopping and stop the fitting even earlier.

Edit

The rationale for my last suggestion:
I assumed that worsened performance on wider dataset is due to overfitting, i.e. significantly worse performance on test dataset than on training data. Early stopping should prevent / control the overfitting, but it seems it didn’t work so good in your case and thus should be improved.

You can and should test different combination of new features, but trustworthy quality metric is crucial for the model choice. If you significantly overfit, the performance on training data (or even cross-validation) will not help you to choose the model (or feature combination) with good performance on test data set.


Weka: Implementation of Random Forest I am wondering how random forests are exactly implemented in Weka. This paper is very specific about RFs in Weka, but the description of its learning process in chapter 2 seems strange to me.
They say:

Bootstrap samples $B_i$ for every tree $t_i$
A random subset of features is selected for each $t_i$
Information gain is used to grow unpruned trees $t_i$

My questions:

Shouldn't step 2 be repeated on all levels of the decision tree? Otherwise each tree will never see some of the features
Whats the default when setting numFeatures=0. I think this is the number of features that is available for each split. Is it the square root of the number of all features?
Is really information gain used for determining the best split attribute?

I am using Weka 3.8.3 - not sure if this matters.

Thanks for all hints :)

Your linked paper appears to be wrong about feature subsetting.  I couldn't find it in the documentation for randomForest, but the source for randomForest uses randomTree for the base models, and in that documentation it says

  a tree that considers K randomly chosen attributes at each node.

So the selection seems to happen at each split.
(Note that xgboost has feature subsetting at each of the tree, the level (depth), and the node.  I don't see any obvious reason that one or more of these options should always be preferable...)

For default number of features, sqrt(m) is the most common, but it looks like Weka uses lg(m).  See option -K at
https://weka.sourceforge.io/doc.dev/weka/classifiers/trees/RandomForest.html

Yes, Weka uses the Quinlan family of decision trees, which split using information gain (as opposed to CART, which uses gini).


Input for individual perceptron in input layer in MLP For better understand of neural networks I started implementation of Multi Layer Perceptron. For now I'm implemented single Perceptron that resolve XOR problem. From this point I want start build MLP but I'm not sure if I correctly
 understand MLP structure. Assume I have data instance with 3 attributes and 2 classes and 5 input perceptrons. Should I give every perceptron same input and every output of perceptron should go to every hidden layer perceptron's input? I attach neural net pic.

With MLPs you can follow a set of simple rules:

The number of neurons in your input layer equals the number of features / data instances you have.
A single neuron takes only one input (one feature)
The output from one neuron can go to multiple neurons in the next layer which may or may not be fully connected.

May the force be with you ;)


How to increase the weight when it comes to outlier detection Let's say we have feature A, B, C, D, E to represent one observation in an outlier detection model. We are using scikit-learn outlier detection in our case.

AFAIK, if we normalize all the features, they are as equally important as others.

What if I want to make B very important, e.g. if A, C, D, E increase 20%, the observation might still be considered as normal case while B increases 10%, the observation needs to be marked as an outlier.

Is there any way to increase or decrease features' weight in outlier detection model?

Please look at my answer at Identfying spikes in data
.

A general method could be to calculate a (moving) average and standard deviation.
Outliers are points that differ more than 3 times the standard deviation.

In your case, B will probably have a standard deviation of 3%, because 10% is considered an outlier. For A, C, D, E, the standard deviation may be 7%, because 20% is not considered an outlier.

Relative importance does not make something an outlier. A large deviation does.