Get Logistic regression scores in CNN using Keras from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K
from keras.regularizers import L1L2

batch_size = 128
num_classes = 10
epochs = 2

# input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
             activation='relu',
             input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(10, activation='softmax',
kernel_regularizer=L1L2(l1=0.0, l2=0.1)))
model.compile(optimizer='sgd',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

The Above code uses logistic regression With the CNN in Keras. How do I get the score assigned to each image by the logistic regression function.

The evaluate method return the loss value & metrics values for the model in test mode.
Instead You should use

y_pred = model.predict(x_test, batch_size=batch_size)

As it generates output predictions for the input samples.
For more information, read Keras official documentation.


How would I apply anomaly detection to time series data in LSTM? I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, with anomalies marked as black dots below:

I've read loads of articles about how to classify with text/sequence data but there's not much on univariate time series data- only timestamps and randomly generated values with a few anomalies. One answer linked to the following method:

  Anomaly detection is done by using the prediction errors as anomaly indicators.Prediction error is the difference between prediction made at time t−1 and the in-put value received at time t. The prediction errors from training data are modeled using a Gaussian distribution. The parameters of the Gaussian, mean and variance, are computed using maximum likelihood estimation (MLE). On new data, the log probability densities (PDs) of errors are calculated and used as anomaly scores: with lower values indicating a greater likelihood of the observation being an anomaly. A validation set containing both normal data and anomalies is used to set a threshold on log PD values that can separate anomalies from normal observations and incur as few false positives as possible. A separate test set is used to evaluate the model.
  Source

I understand overall why this method would be used but I have little idea of how to implement it with a Sequential Keras model. Does anyone have or know of some example code? Bonus if there is a way to visualize this because I don't know how I would go about that side of things either.

Please help and thank you!

I've been in that situation before,

there's this article on medium where the guy uses keras,tf for predicting credit card fraud detection using autoencoders which have Dense layers, but you can try the same with LSTM, can't say for sure whether it will work, but if in case it doesn't work, please try Conv1d because nowadays convolutional networks are more promising than LSTMs and GRUs -> source

Even if autoencoder doesn't help, that guy has visualizations for seeing anomalies, you must at least try them.

Hope this helps.


How to represent linear regression in a decision tree form I have read that decision trees can represent any hypothesis and are thus completely expressive. So how do we represent the hypothesis of linear regression in the form of a decision tree ? I am referring to the equation

w0 + w1.x1 + w2.x2 + .... + wn.xn = yn

The basic idea would be to divide up your feature space in small multi-dimensional intervals, and then assign to each point in a given interval the average value that your linear regression model has in that interval. This is something you can do with a tree.

This is similar in spirit to approximating (in 1D space) the function $y = x$ with a "piecewise constant", "staircase-like" function like http://mathworld.wolfram.com/NearestIntegerFunction.html: you could divide your 1D space in equal intervals (e.g. of length 1), and you'd assign to each interval the average value that the function $y = x$ has in it.

Note that such "piecewise constant" function can be defined as a tree:   suppose you wanted to know the "tree" approximation of $y = x$ for $x^* = \pi = 3.14159..$, then you could do:

Is $x^* > 0$? Yes
- Is $x^* < 0.5$? No (if it was, I would have approximated $x^*$ with 0)
- - Is $x^* < 1.5$? No (if it was, I would have approximated with 1)
- - - Is $x^* < 2.5$? No (if it was...)
- - - - Is $x^* < 3.5$? Yes -> I approximate $x^*$ with $3$ (3 is the average of $y = x$ between 2.5 and 3.5).

Note that as the size of the intervals shrinks (in "tree language", as you grow your tree more and more), the better the approximation would become. Also, rather than decision one would speak of regression tree in this context.

To generalize this idea, just imagine that you could carry out a similar procedure for any linear function in 1D $y = ax + b$ using a "staircase-like" function, and in N dimensions for any linear function in N-D $y = a_0 + a_1 x_1 + \cdots + a_n x_x$.
Actually, you don't have to restrain to linear functions, as as you said trees are more flexible still (you just need to appropriately assign the values to each interval of your feature space)!


Using feature importance to decet latent variables and grouping Is it possible to use feature importance from Random Forests (e.g. based on gini impurity) or other models to determine which features I can use to group the rows of my dataset homogeneously?

For example, let's say I have a dataset with N rows and p columns, (one of the columns used as the label in my training task). I train the model and I get a rank of the importance of my features. Only 5 features are more important than a random feature "artificially" added to my dataset (this is used to understand if some features do not add any "predictable power" to my model). Can I assume that using these 5 features as dimensions to group my rows, provide "homogeneous group w.r.t. my output"?

Apologies for the confusing description of my problem. Rather than a strict answer I'd like to be pointed to the right direction to make some research into this. For now this is solely based on my intuition and I could be totally off-road.

Is there any area of ML/Stats/etc. where this is done?

For example, Latent Class Analysis seems something similar to this, am I wrong?

Example:

I want to predict the probability of an person to drink more than 0.5 liter of beer in a month. I have information about each individual in the population such as: age, sex, geographical area (US state), height, weight, etc.

My output is

0 = avg(liter of beer per mount) < 0.5
1 = avg(liter of beer per mount) >= 0.5

I train a model (let's say a Random forest) and the feature importance ranks says that "age", "weight" and "height" are useful in predicting the consumption of beer, while the other features do not add much to the predicting performance (or at least those are ranked as less important).

My assumption is that if I group the population if using the three "important" features these groups will share some similar characteristic, hence there's a latent variable which groups these individual together.

Groups:

1) Age: [21, 25], weight(Kg): [70, 75], height(cm): [170, 175]

2) Age: [21, 25], weight(Kg): [70, 75], height(cm): [175, 180]

3) etc.

You might be able to segmentate your items into groups that share a beer consumption trait by focusing on those variables. It's a good idea if you started with a lot of columns . Of course, expect more meaningful groups if your model is simpler rather than complex - a small decision tree rather than a random forrest, for example.

Why don't you do some descriptive statistics on these columns, dividing first your data in the target variable? The visualization of the different patterns can be quite inspiring.

Also, test your intuition: cluster your data based solely on these columns and check whether each cluster is significantly biased towards one of the target values.

Something that you can also try is using the notion of proximity between data points you get from a random forrest model in your clustering algorithm.


Saving Large Spark ML Pipeline to HDFS I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum message size:

    scala> val mod = pipeline.fit(df)
    mod: org.apache.spark.ml.PipelineModel = pipeline_936bcade4716
    scala> mod.write.overwrite().save(modelPath.concat("model"))
    18/01/08 10:00:32 WARN TaskSetManager: Stage 8 contains a task of very large size
    (755610 KB). The maximum recommended task size is 100 KB.
    org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task
    2606:0 was 777523713 bytes, which exceeds max allowed: spark.rpc.message.maxSize
    (134217728 bytes). Consider increasing spark.rpc.message.maxSize
    or using broadcast variables for large values.

Making the following assumptions about the problem area:

It's not possible to decrease the size of the model  AND
It's not possible to increase the maximum message size to a point where the pipeline would fit in a single message.

Are there any methods that would allow me to save the pipeline successfully to HDFS?

You can do the following:
A Pipeline can be made of other pipelines. Isn't that great?
A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a Transformer.
This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform on the DataFrame.


How to turn linear regression into logistical regression I followed these articles to implement logistic regression.

I'm confused however because after training the model and getting the weights for my variables I don't now how to use the one-hot vector to turn this into confidence scores for the different classes.

I've got the formula: y' = x1W1 + x2W2 + x3W3 + b

I've got values for all Ws and b.

I've got my one-hot vector: [[1, 0, 0], [0, 1, 0], [0, 0, 1]]

How do I combine all this into confidence for each class?

You should use softmax to convert your output in probabilities. For only two classes, you have the formula $P(x  \in class 1) = \frac{\exp(y_{\text{class1}})}{\exp(y_{\text{class1}}) + \exp(y_{\text{class2}})}$. It mentioned in your article.


Text mining in Amazon product review using R. I wasn't able to extract the particular product's review Text mining on Amazon product review using R Program. I wasn't able to extract the particular product's review(i.e.If iphone 11 has 6k review, I need to extract all of it.) I'm getting only one column labelled x.

Please let me know where I need to make necessary changes. I need those for performing sentiment analysis.
install.packages("rvest")

library(rvest)

install.packages("xml2")

library(xml2)

install.packages("magrittr")

library(magrittr)

url <-"https://www.amazon.in/Apple-iPhone-11-128GB-Black/product-reviews/B07XVLW7YK/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews"

apple <- NULL

for(i in 1:100)

{

murl <- read_html(as.character(paste(url,i,sep = "=")))

rev <- murl%>%

html_nodes(".review-text")%>%

html_text()

apple <- c(apple,rev)

}

write.table(apple,"apple1.txt",row.names = TRUE,col.names = TRUE,fileEncoding = "utf-8-mac")

apple<-read.csv(file.choose())

View(apple)

Change your url to:
url <- "https://www.amazon.in/Apple-iPhone-11-128GB-Black/product-reviews/B07XVLW7YK/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber"

Otherwise, as is what your current code is doing, you keep looping only the first page of reviews.


Theoretical treatment of unlabeled samples In a typical supervised learning setting with a few positive and a few negative examples, it is clear that unlabeled data carries some information that can benefit learning and that is not captured in the labeled data. For example one can estimate mean values, bounds and some other geometrical characteristics of the data-set with much higher precision if you do not discard the (massive) unlabeled data.

On the other hand, the most common ML algorithms from Neural Networks to SVM do not take advantage of this information (at least in their standard, most common form). My question:

Is there any theoretical framework where unlabeled data is treated in the supervised setting?

I can think of semi-supervised ways to approach this (first cluster and then label the clusters). Are there any other?

You already dropped the key word "semi-supervised" in your question. Indeed, semi-supervised learning is the answer to your question. Search for this term with your favourite search engine or library catalog to find out about algorithms for semi-supervised learning.


How to tell a boosting model that 2 features are related and should not be interpreted stand-alone? I am using XGBoost for a machine learning model that learns from tabular data.

XGBoost uses boosting method on decision trees. When I look at the decision-making logic of decision trees, I notice the logic is based on 1 feature at one time.  In real life, certain multiple features are related to each other.

Currently, when I feed data to the model, I simply feed all the features to it without telling the model how certain features are related to each other.

Let me describe a hypothetical example to be clearer. Suppose I have 2 features - gender and length of hair. In this hypothetical problem, I know from my domain knowledge that if gender is female, length of hair matters in determining the outcome. If gender is male, length of hair is irrelevant. How do I tell the machine learning model this valuable piece of information so that the model can learn better?

I am using XGBoost on python 3.7

I am going to talk about some ways you could do it later but first I want to talk about whether you should!

If the relation that you describe exists XGB will be able to learn and detect it! There is no real benefit in "hard-coding" a rule into the algorithm, it won't speed up the training, it won't benefit accuracy, etc. Simply put the benefit of ML algorithms is that they are able to detect exactly these relationships and model them in the best possible way.

Now if you still insist that this is something that must be done, you can. The easiest way to achieve this would be feature engineering:

Introduce NAs -simply leave out hair length of male respondents and fill with NA
Create interaction factors - instead of having hair length and gender as a simple variables you could also code it in a way that represents the known interaction like this:

gender_hair = [male, female_short,female_medium,female_long] # example factor levels

But again if you compare models with those engineered features to a simpler model you will see no benefit I'd wager.


Compute Accuracy of k-means Could you please provide me an example of how I can compute the accuracy for a kmeans clustering?
I split my dataset into train and test sets and computed the predicted clusters for the train set. However, I do not know how to compiute its level of accuracy and see if it is good not also for the test.

An example with a sample dataset of your choice it would be extremely useful.

K-Means is a clustering technique NOT classification. You don't have the ground truth here to compare with. Hence accuracy doesn't make any sense.

You can train the model and with the test data predict which cluster the test data belongs to.

Try to visualize it, it shall be helpful.


training one SVM model for predicting more than one response variable Please I want to know if there is any SVM R package that can handle more than one response variable (y) at a time. that is to train one model for predicting more than one response variable. it could be regression or multi class classification problem.

Thanks for your help

e1071 is a highly popular and therefore safe solution for this purpose.

But if data starts to get bigger I would go with classyfire. It's a quite fresh package built on e1071. It is highly optimized and have some other nice feature too. In my experience it produced similar results to e1071 but with a much faster speed as it was able to utilize all 4 core of my processor.


Preference Matching Algorithm There's this side project I'm working on where I need to structure a solution to the following problem.

I have two groups of people (clients). Group "A" intends to buy and group "B" intends to sell a determined product "X".

The product has a series of attributes x_i and my objective is to facilitate the transaction between "A" e "B" by matching their preferences. The main idea is to point out to each member of "A" a corresponding in "B" who’s product better suits his needs, and vice versa.

Some complicating aspects of the problem:

The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design which is rare among the population and I can’t predict. Can’t previously list all the attributes;
Attributes might be continuous, binary or non-quantifiable (ex: price, functionality, design).

Any suggestion on how to approach this problem and solve it in an automated way?
The idea is to really think out of the box here so feel free to "go wild" on your suggestions.

I would also appreciate some references to other similar problems if possible.

This can be a cross between machine learning and simple matching exercise.

I think X_i tend to be rather defined and finite, while A_i can be vague and not finite. From a pure algorithm perspective I would search for instances where X_i = A_i and store the results into a container of sort. The more hits for certain X'es where X_i_n = A_i_k the more points X scores. X'es are then presented to A in the order of points from best match to lowest match.

Onto the machine learning mechanism, as the algorithm serves a lot of As (by that mean thousands and thousands, even millions) patterns will start to develop and certain combination of A_i's will be more prevalent, or in other words, worth more to other A_i's for a certain category of A. Using these patterns, the weighting of points will be re-balanced for higher chance of hitting the correct offers.

Kind of like how a search engine works.


Can you use clustering to pick out signals in noisy data? As my first project into data science, I would like to pick out the main clusters in noisy data. I think a good example would be trying to pick out certain links on a given StackExchange question that has a number of answers. The most common type of link is a link to a question on the SE network. The next common is either tag links, or links to user profiles. The remaining links might be random links included in posts, which is considered noise.  Ideally, I'm looking for a solution where I don't know how many clusters of links there will be ahead of time.

I've implemented my first attempt using scikit-learn and KMeans. However, it's not ideal because I appear to have to specify the number of clusters ahead of time, and I think the random, noisy links get grouped improperly. I also think it's more effective on a larger corpus compared to the relatively small one of URL tokens (though that's just a guess).

Is there a way to do this type of clustering, where the number of clusters is unknown or where one of the clusters is a sort of miscellaneous cluster containing objects that don't closely match the other clusters?

Have you looked at DBSCAN? It is a density-based spatial clustering of data with noise that can define non-linear clusters (unlike k-means).

It doesn't require knowing the number of clusters. However, it does require two parameters (minimum cluster size and neighborhood size) that measure density. But you may be able to estimate them in your particular domain.


How can I reduce the volume of data (No column to be dropped)? I have a data set of 80,000 samples (40k 3 axis accelerometer and 40k Gyro data). I am trying to implement KNN and Random Forest for activity recognition on ESP8266 Node MCU. The limited memory of the MCU is the bottleneck of the process.
Is there any method that can reduce the dataset to, say 5000 records without losing any vital information and without affecting overall accuracy? Dimensionality reduction, as I could understand with my non-mathematical background, is reducing the data by dropping less important columns. However, in my case, I cannot drop any column (only 6 columns are there which are x,y,z values of accelerometer and gyroscope).
Sample data:

You are the domain expert on this, but these are the two practical approaches, I have utilized depending on the use case and effect on performance:
1. Downsample: Let say, randomly choose 5,000 out of 80,000 records. To maintain the same proportion of classes as in the population, go for stratified random sampling.
2. Reduce the precision of measurements (i.e. less decimal points) and aggregate data to capture the frequency of similar records as a new feature. If you have repeating records you can do this without compromising on the precision too.


Is it correct to say that a high variance feature will be important to my model? Analysing 3 features I was faced with the following situation:
Two of them have almost no variance and the last one have a greater variance than the other (almost two times). Is it correct to assume using only this information that the feature with more variance will be more important to my model ? Also, if a feature has 0 variance the model simply won't learn anything from it, am I right?

Lastly, if you have some nice references I would love to read them.

High feature variance does not imply any sort of relationship to the target variable you're interested in modeling. Suppose you are looking at a population and have two variables for each person, their height in meters and annual income in dollars. The variance of height will be a small number (heights don't vary by much more than 1 meter), but the variance of income will be a large number (incomes can vary by thousands). If you want to predict the person's arm span, their height will be a much better predictor than their income, despite the fact that it has a numerically smaller variance.
A zero variance feature does not have any predictive power, but even numerically small fluctuations can be very important depending on your goal.


How to pass a sequence of 4 images into LSTM and CNN-LSTM I got an assignment and stuck with it while going down the rabbithole of learning PyTorch, LSTM and cnn.
Provided the well known MNIST library I take combinations of 4 numbers and per combination it falls down into one of 7 labels.
eg:
1111 label 1 (follow a constant trend)
1234 label 2 increasing trend
4321 label 3 decreasing trend
...
7382 label 7 decreasing trend - increasing trend - decreasing trend
The shape of my tensor after loading of the tensor become (3,4,28,28) where the 28 comes from the MNIST image's width and height. 3 is the batch size and 4 is the channels (4 images).
I'm somewhat stuck with how to pass this into a PyTorch backed LSTM and CNN as basically all Google searches lead to articles where simply one image is passed in.
I was thinking of reshaping it to 1 long array of (pixel values) where I put all of the values of the first image row by row (28) after each other, then appended by the same approach for the second, third and fourth image. So that would make 4 * 28 * 28 = 3136.
Is my way of thinking on how to tackle this a correct one or should I rethink? I'm rather new to this all and looking for some guidance on how to go forward. I've been reading loads of articles, YT videos, ... but all seem to touch the basic stuff or alternatives of the same subject.

CNN-LSTM are used in text recognition. How is text structured? Characters are stacking in a horizontal manner. Do the same with your MNIST images and for the ground truth make proper labels (should be of 4 size).


In which situation should we consider a dataset as imbalanced? I'm facing a problem about making a classification on a dataset. The target variable is binary (with 2 classes, 0 and 1). I have 8,161 samples in the training dataset. And for each class, I have:

class 0: 6,008 samples, 73.6% of total numbers.
class 1: 2,153 samples, 26.4%

My questions are:

In this case, should I consider the dataset I used as an imbalanced dataset?

If it was, should I process the data before using RandomForest to make a prediction?

If it was not an imbalanced dataset, could somebody tell me in which situation (like what ratio for each class) I could consider a dataset as imbalanced?

I think you can speak of imbalanced targets if (in case of a binary classification problem) the classes are not represented in a 50:50 manner. This is almost always the case.
With about 25/75 in your case, I would see this as „imbalanced“. There are some strategies to deal with this problem, such as (re)sampling so that you achieve a 50:50 balanced sample (essentially you will lose observations in the majority class here). Alternatively you can use synthetic oversampling (SMOTE) and related rechniques.
However, some packages come with built-in options to deal with unbalanced targets, e.g. sklearn‘s random forest (option class_weight). Check the docs.


Difference between Information Gain and Mutual Information for feature selection What is the difference between information gain and mutual information?

At this point, I understand that information gain is calculated between a random variable and target class for classification while mutual information is calculated between two random variables.

Does mutual information become the same as information when it is calculated between a random variable and target class?

Information Gain (IG) is the measure of entropy gained due to operations performed on an assigned dataset or random variable. Now, entropy here is just the variation in our dataset; so lesser the variation, lesser the entropy and in turn greater correlation among our dataset.

As you've hinted on a classification task, Mutual Information (MI) is a one-sided metric, whereas information gain is a two-sided metric. Actually both measure how relevant a feature is to a specific target class, & I've often observed various sources using the term information gain interchangeably with mutual information, as they are symmetric.

Important to note is while mutual information measures only the positive features, information gain measures both, negative as well as positive features of our data. For mathematical completeness, information gain of X given Y is given by IG(X|Y) = H(x) - H(X|Y), and mutual information I(X;Y) is given by I(X;Y) = sum_x sum_y P(X,Y) log {P(X,Y)/P(X)P(Y)}.

Following links might help grasp better understanding of these terms:

Part-1 | Probability Distribution
Part-2 | Information Theory

Hope it helps!


Spam detection in social media I want to train a binary classification algorithm for spam detection using labeled data set of mentions from social media. These mentions have the following features:

  URL, text message, posting date, is deleted, author

where the author has its own fields:

  URL, nick, type (person or community), number of subscribers, registration date, last activity date

I have $\approx 13.5k$ observations with $\approx 450$ spam messages among them.

My suggestions:

transform data features in real numbers
binarize URLs by media sources (e.g. facebook, instagram, etc.)
use SVM with Gaussian kernel for all data

Questions: Am I on the right way? What else can I do? How can I process text data?

Thanks in advance for any suggestions!

The approach seems correct, don't worry. But I'd suggest to use Naive Bayes Classifier instead of SVM - it perfectly works for this task (and actually popular email services use this algorithm for spam detection).


Feature Selection Phase I am trying to predict the overall age of an opportunity (creation date - closing date) this is my response variable

lets say an opportunity  passes through 3 stages
to close

For example: Opp x stayed in

stage 1 : 30 days
stage 2 : 10 days
stage 3:  20 days

At stage 3 I might close it same date or wait some time

so if I waited some time to close, it will be createdon: 22/11/2018 & closedon:9/2/2019

There is opp y , where i close it in same date of stage 3, so createdon:22/11/2018 and closedate: 21/1/2019

Summary

+---------+--------+--------+--------+--------+
| OppName | oppAge | stage1 | stage2 | stage3 |
+---------+--------+--------+--------+--------+
| x       |     79 |     30 |     10 |     20 |
| y       |     60 |     30 |     10 |     20 |
+---------+--------+--------+--------+--------+

my question is :

Can I include stage1,2,3 as my independent variables to create a regression model?
They seem to nearly make the model ideal, so is it better to include maybe only stage 1? without 2 &3

Work Done ,Added Edits

I transformed stages to a categorical nature, for example <30 days to 1 month
30-60 day transformed it to 1-2months and so on

 +-----------+
|  stage1   |
+-----------+
| <1month   |
| 1~2months |
| 6~7months |
+-----------+

Then I did one-hot encode to the stages like stage 1
Then I stopped ,wasn't sure whether to include everything or what?

One-hot encode those three Stage variables instead of including all three of them. And again, include any of those at all, ONLY if they hold relevance so a pre-hand PCA won't be a bad idea overall.


Algorithmically extract seasonality in time series data Suppose you are trying to measure if the seasonality of a particular event stream is consistent i.e. the events in a time series happen more or less in pattern like fashion. How can you algorithmically measure and extract seasonality?

Currently, I am using autocorrelation to view the clearly seasonal (monthly) data events. I am struggling to figure out how to write an algorithm that could test whether or not something is seasonal by month.
The picture below shows the results of the pacf function within the statsmodels.tsa.stattools library. From the graph it is clear, how can I use the array to determine seasonality? Is there a library that would be useful for this?

This question is similar, but did not get any great answers:
https://stats.stackexchange.com/questions/225003/test-for-trend-and-seasonality-in-time-series

You can try FBProphet for your timeseries analysis. You will be able to check the seasonality, stationarity on your data.


Why is the idf important in tf-idf when it seems to just re-scale your features? I am trying to understand why tf-idf is useful. As I understand the formula to work out the tf-idf is:

Can someone explain what is wrong with the reasoning below:
Imagine I have 100 documents that have a text corpus of 10,000 words. As I understand each document will have 10,000 features once transformed using tf-idf. For a single feature we get the tf part by counting the number of occurrences of that word in each document. We then multiply that feature by the idf.
TL;DR
My question is given that we multiply the entire feature by a constant (the value of log(N/dfi)), how does that help in making a model better. As I understand multiplying a feature by a constant doesn't help as the model can figure out this new scale.
Edit
After applying tf we might get a document-term matrix like the following:
word /
doc num. computer walk smell help warmth
1        1        0    2     1    0
2        0        2    3     1    1
3        0        1    0     0    3
4        1        2    0     1    0
5        0        1    2     2    1

Then after tf-idf we might get the following:
word /
doc num. computer    walk        smell      help        warmth
1        1*log(5/2)  0*log(5/4)  2*log(5/3) 1*log(5/4)  0*log(5/3)
2        0*log(5/2)  2*log(5/4)  3*log(5/3) 1*log(5/4)  1*log(5/3)
3        0*log(5/2)  1*log(5/4)  0*log(5/3) 0*log(5/4)  3*log(5/3)
4        1*log(5/2)  2*log(5/4)  0*log(5/3) 1*log(5/4)  0*log(5/3)
5        0*log(5/2)  1*log(5/4)  2*log(5/3) 2*log(5/4)  1*log(5/3)

tf-idf seems to just scale each feature here by the same value so why does that help certain models like decision trees that can handle different scaling to predict better?

To complement my comment I'm taking those paragraphs from data camp tutorial in which they explain this in a very clear way


What would you do in Knn specific case I'd like to know what would you do in this specific and unrealistic case appliying Knn when k = 1, k = 2 and k = 3.

Class 1 individuals: [1,1] [2,2] [2,4] [2,5]
Class 2 individuals: [3,1] [4,1] [4,2]
individual to classify: [2,1]

Plot:

I don't know if there would be any criteria rather than the generation order to choose the class it belongs.

When k=3 it is a simple case, the 3 nearest examples share the same distance to the new sample. Two of them are from class 1 and a single example is from class 2, so the new sample will be classified as class 1.

When k=2 it is more tricky. Assuming that the 2 features (x,y) share similar importance to the classification (otherwise distances should be calculated with some predetermined weights), you can:

Choose randomly 2 out of the 3 candidates (you are basically acknowledging that for this case, your model doesn't have enough information for classification based on 2-NN).
take all candidates into account (i.e. for cases like this you go to 3-NN).

When k=1, same options as with the k=2 case (you will still need to go up to 3-NN).

In this specific case, because your 3 nearest neighbors are all of the same distance, I would choose to use K-NN. However for a similar case with 4 samples at the same distance, 2 of each class, I would choose randomly (not going to 5-NN).


Performance of four GTX 1080 Ti's versus one Tesla V100 for Deep Neural Network training While this may be slightly off-topic, this question does pertain to data science and machine learning. I want to train a VGG16 model on Imagenet from scratch. For this purpose, I am looking into either buying four to six GTX 1080 Ti's or one Tesla V100. I have a feeling that four GTX 1080 Ti's will perform far better than a single V100, but has there been any practical research into this?

EDIT - Each GTX 1080 Ti has 11.3 teraflops of FP32 performance (or 45.2 teraflops for four 1080 Ti's, or slightly less since the scaling isn't linear), whereas the V100 is 100 teraflops, or more than double the performance of four GTX 1080 Ti's. Anyone have any benchmarks for these two on imagenet performance?

This link has more realistic benchmarks of 1080ti vs Titan V (same as the V100 but less RAM), using pytorch. Its not nearly the best case scenario of 10x speedups, and seems to be a more modest 20% to 50% boost. This is fairly expected, nVidias grandiose performance claims tend to be only applicable to fairly restricted benchmarks that don't scale well to real world usage. You'd probably get a lot more out of 4 1080ti cards than a single V100 unless you REALLY need a large amount of RAM in a single card.


Reshaping of data for deep learning using Keras I am a beginner to Keras and I have started with the MNIST example to understand how the library actually works. The code snippet of MNIST problem in Keras example folder is given as :

import numpy as np
np.random.seed(1337)  # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils

batch_size = 128
nb_classes = 10
nb_epoch = 12

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
nb_pool = 2
# convolution kernel size
nb_conv = 3

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
..........

I am unable to understand the reshape function here. What is it doing and why we have applied it?

mnist.load_data() supplies the MNIST digits with structure (nb_samples, 28, 28) i.e. with 2 dimensions per example representing a greyscale image 28x28.

The Convolution2D layers in Keras however, are designed to work with 3 dimensions per example. They have 4-dimensional inputs and outputs. This covers colour images (nb_samples, nb_channels, width, height), but more importantly, it covers deeper layers of the network, where each example has become a set of feature maps i.e. (nb_samples, nb_features, width, height).

The greyscale image for MNIST digits input would either need a different CNN layer design (or a param to the layer constructor to accept a different shape), or the design could simply use a standard CNN and you must explicitly express the examples as 1-channel images. The Keras team chose the latter approach, which needs the re-shape.


Anomaly detection without any knowledge about structure I have an interesting question, my code needs to be able to handle structured data where I don't know much about the structure at development time. I know the samples follow a schema that can be nested and the leafs contain some basic primitives like floats, integers and categoricals. I will have access to the schema at training time. I want to train a model that can detect anomalies or more directly whether or not a sample comes from the original distribution of data.

One way would be to just flatten the data and train an isolation forest. From my intuition, this breaks with highly correlated features. Another approach would be to build a neural network architecture that contains some of the structure and primitive types of the passed schema. You could build an autoencoder approach and measure reconstruction error, which I feel like would handle correlated features a lot better because these correlations are captured by the encoder-decoder network. A possible issue would be how to build this reconstruction error metric with different primitives.

Does anyone have any ideas on this or possibly papers that handle related challenges?

EDIT:

I will clarify some things, sorry for being vague. At the time of designing the algorithm I don't know the structure of the dataset involved but at the moment of training the model this structure is known. The fact that the schema could be nested is not so relevant, let's ignore that part and assume we have a flat, tabular dataset where we know the dtypes. It's likely these dtypes are mixed so I'm looking for a way to reason about anomalies on this set.

Graph Embeddings

You try to use graph embeddings. These would offer the ability to map your graph-based data into a single latent space, at based on the vertices, the edges or the graph schema as a whole. This has become somewhat of an exciting research topic within research over the last few years; combining graph analysis with modern embedding techniques.

Here is one survey paper that list approaches to these embeddings, along with the authors' own libraries for both static graphs and dynamic graphs. Looking through the results of that survey paper (they perform some becnhmarks towards the end), it would seem that the most promising model would be:

node2vec - if you want to classify the nodes as cleanly as possible (reference implementation)
Structural Deep Network Embedding - if you want to re-construct the graph (reference implementation)

Anomaly Detection

As to the end goal of anomaly detection, I am not sure which of the embedding methods would be best suited. It might be a case as to whether you want to preserve more local or global structure (as with the trade-off between methods such as t-SNE and PCA).

I think you can imagine what to try out with the embeddings in order to detect anaomalies. There might be more clues in the papers linked above as to which way researchers tend to go for certain problems.

Given the hierarchical nature of your graph data, I would be tempted to start by looking into using Hierarchical Density Based Spatial Clustering with Applications of Noise (i.e. HDBSCAN) - the main library specifically includes a few few methods of outlier detection! Here a citation from the linked paper:

  Based on these definitions, we can devise an algorithm DBSCAN* (similar to DBSCAN) that conceptually finds clusters as the connected components of a graph in
  which the objects of X are vertices and every pair of vertices is adjacent if and only
  if the corresponding objects are ε-reachable w.r.t. user-defined parameters ε and mpts.
  Noncore objects are labeled as noise.


Why a Random Reward in One-step Dynamics MDP? I am reading the 2018 book by Sutton & Barto on Reinforcement Learning and I am wondering the benefit of defining the one-step dynamics of an MDP as
$$
  p(s',r|s,a) = Pr(S_{t+1},R_{t+1}|S_t=s, A_t=a)
$$
where $S_t$ is the state and $A_t$ the action at time $t$. $R_t$ is the reward.

This formulation would be useful if we were to allow different rewards when transitioning from $s$ to $s'$ by taking an action $a$, but this does not make sense. I am used to the definition based on $p(s'|s,a)$ and $r(s,a,s')$, which of course can be derived from the one-step dynamics above.

Clearly, I am missing something. Any enlightenment would be really helpful. Thx!

In general, $R_{t+1}$ is is a random variable with conditional probability distribution $Pr(R_{t+1}=r|S_t=s,A_t=a)$. So it can potentially take on a different value each time action $a$ is taken in state $s$.

Some problems don't require any randomness in their reward function. Using the expected reward $r(s,a,s')$ is simpler in this case, since we don't have to worry about the reward's distribution. However, some problems do require randomness in their reward function. Consider the classic multi-armed bandit problem, for example. The payoff from a machine isn't generally deterministic.

As the basis for RL, we want the MDP to be as general as possible. We model reward in MDPs as a random variable because it gives us that generality. And because it is useful to do so.


Train a point rather than a bounding box for object detection In the graph below, object detection is done through a point that points to the right place rather than a bounding box (e.g., using faster RCNN). What method enables to  train such a point?

Ususally the x,y coordinates of the top-left corner of the bounding box and the width and height of the bounding box is returned by objection detection algorithms (eg: Haar Cascades)
You can always convert a bounding box to a point via the equation:

$BoundingBoxCenter_{x}= \frac{BoundingBoxCorner_{x} +
 BoundingBoxWidth}{2}$
$BoundingBoxCenter_{y}=
 \frac{BoundingBoxCorner_{y} + BoundingBoxHeight}{2}$


How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning? How to implement STS(Semantic Textual Similarity) on unlabelled dataset. Dataset column contains Unique_id, text1(contains paragraph), text2(contains paragraph).

Ex: Column representation:  Unique_id   |    Text1    |     Text2

Unique_id 0

Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua   reynolds portrait of omai will get a public airing following fears it would stay hidden because   of an export wrangle.

Text2 then requested to do so by  spain   s anti-violence commission. the fine was far less than the expected amount of about Â£22 000 or even the suspension of his coaching licence.

Unique_id 1

Text1 groening. gervais has already begun writing the   script but is keeping its subject matter a closely guarded secret. he will   also write a part for himself in the episode. i ve got the rough idea but this is the most intimidating project of   my career.

Text2 philadelphia said they found   insufficient evidence to support the woman s allegations regarding an alleged   incident in january 2004. the woman reported the allegations to canadian   authorities last month. cosby s lawyer walter m phillips jr said the   comedian was pleased with the decision.

In the above problem, I've to compare two paragraphs texts i.e. Text1 & Text2 and then I've to compare semantic similarity between two texts. If they are semantically similar then it will print '1' if not then '0'

Any refrence implementation link or any suggestions !

Thanks in advance

You could take pretrained embedder and look for distances between the embeddings.
There's LASER from Facebook. This is an unofficial pypi package, it replaces some of the internal tools used for tokenization and BPE encodings. I've used it extensively and it works just fine. It encodes your text as 1024-element numerical vector. Then you can just calculate distance metric between embeddings i.e. euclidean.


How this visualisation was made? I really like how this visualization represents the survey participants.

Is any tool for that?  (Or R/python library?)

My go to library would be matplotlib, with which it is relatively easy to generate something similar.

I don't have the correct font family to render the exact output as above, but this hopefully illustrates the point

Source Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Create the data to plot on
# create a 2d array of evenly-spaced points on a grid
xy_range = np.arange(10.)
data = pd.DataFrame([[x, y] for x in xy_range for y in xy_range], columns=['X', 'Y'])

# color the points
# all green first and then with exceptions
data['color'] = 'green'
data.loc[(data['X']>=0)&(data['Y']==1), 'color'] = 'yellow'
data.loc[(data['X']>=8)&(data['Y']==2), 'color'] = 'yellow'
data.loc[(data['X']>=6)&(data['Y']==0), 'color'] = 'blue'

# We'll use this to calculate axis fractions
max_x = data['X'].max()
max_y = data['Y'].max()

fig, currAX = plt.subplots(figsize=(5,5), facecolor='lightgray')

# plot/mark each point as an annotation
# we do this so we can get a custom emoji instead a marker
for x0, y0, color in zip(data['X'], data['Y'], data['color']):
    plt.annotate(s=u'\u263B', xy=(x0/max_x, y0/max_y), fontname='STIXGeneral', color=color, size=15, ha='center', va='center')

# clear the axis lables, ticks, and lines
currAX.get_xaxis().set_visible(False)
currAX.get_yaxis().set_visible(False)
plt.axis('off')

plt.show();


Unable to understand the usage of labels argument in sklearn.metrics.f1_score I am trying to model a dataset with RandomForest Classifier.
My dataset has 3 classes viz. A, B, C. 'A' is the negative class and 'B' and 'C' are positive classes.

In GridSearch I wanted to optimize on F1-score since the number of samples in all the classes are not evenly distributed and class 'A' has the highest number of samples.

That is where I wanted to understand the usage of labels argument.
The doc says:

  labels : list, optional The set of labels to include when average !=
  'binary', and their order if average is None. Labels present in the
  data can be excluded, for example to calculate a multiclass average
  ignoring a majority negative class, while labels not present in the
  data will result in 0 components in a macro average.

I could not understand it properly. Does it mean, In my screnario I should have labels as labels = ['B', 'C'], just the positive class?
Kindly Help

custom_scoring = make_scorer(f1_score, labels=[???],average='weighted')
clf = RandomForestClassifier(class_weight='balanced', random_state=args.random_state)
grid_search = GridSearchCV(clf, param_grid=param_grid, n_jobs=20, scoring=custom_scoring)

The F1 measure is a type of class-balanced accuracy measure - when there are only two classes, it's very straightforward, as there's only one possible way to compute it. With 3 classes, however, you could compute the F1 measure for classes A and B, or B and C, or C and A, or between all three of A, B and C.

It seems that the "labels" parameter is telling the method which classes to compute your measure over. Since F1 is already class-balanced, you probably want to include all three labels for your measure. This parameter is likely more important for imbalance-insensitive measures like raw accuracy, as it will allow you to compute accuracy in a subset of the data - in the documentation example, they use it to exclude a majority class, allowing the user to evaluate the accuracy in the minority only. If you have a huge imbalance, say 99% of your data is of one class, your accuracy measure will be completely dominated by accuracy within that class - for this reason, it might be more informative to see how well the classifier does in the 1% only.


What's wrong with my implementation of the Adaline algorithm? I'm working through the textbook called Learning From Data and one of the problems from the first chapter has the reader implement the Adaline algorithm from scratch and I chose to do so using Python. The issue I'm running into is that the weights for my $\textbf{w}$ immediately blow up to infinity before my algorithm converges. Is there something incorrect I am doing here? It looks like I am implementing it exactly as the text describes. Below I've provided the question and my Python code. Here $\textbf{y}$ takes on the values of -1 and 1. So it is a classification problem.

import numpy as np
import pandas as pd

#Generate w* vector, the true weights
dim=2
wstar=2000*np.random.rand(dim+1)-1000

#Generate the random sample of size 100
trainSize=100
train=pd.DataFrame(2000*np.random.rand(trainSize,dim)-1000)
train['intercept']=np.ones(trainSize)
cols=train.columns.tolist()
cols=cols[-1:]+cols[:-1]
train=train[cols]

#Classify the points
train['y']=np.sign(np.dot(train.iloc[:,0:3],wstar))

#Now we run the ADALINE algorithm on the training data
#Declare w vector
w=np.zeros(dim+1)

#Column of guesses
train['guess']=np.ones(trainSize)

#s column
train['s']=np.dot(train.iloc[:,0:3],w)

#Set eta
eta=5
iterations=0
while (all((train['y']*train['s'])>1)==False):
    if iterations>=1000:
        break
    #Picking a random point
    randInt=np.random.randint(len(train))
    #Temporary values for calculating new w
    temp_s=train['s'].iloc[randInt]
    temp_x=train.iloc[randInt,0:3]
    temp_y=train['y'].iloc[randInt]
    #Calculating new w
    if temp_y*temp_s<=1:
        w=w+eta*(temp_y-temp_s)*temp_x
        #Calculating new guesses and s values
        train['s']=np.dot(train.iloc[:,0:3],w)
        train['guess']=np.sign(train['s'])
    iterations+=1

First of all, let me add this schema which I think is quite nice to understand the transition and improvement from the initial Rosenblatt's perceptron and the Adaline algorithm:

In Adaline, provided that the cost function (your y(t)-s(t)) is differentiable, the weights can be updated and there is no restriction of y and s having the same sign: the objective is to minimize the cost y-s.
Below you can find the code provided in the excellent book by Sebastian Raschka:
class AdalineSGD(object):
"""ADAptive LInear NEuron classifier.
    Parameters
    ------------
    eta : float
    Learning rate (between 0.0 and 1.0)
    n_iter : int
    Passes over the training dataset.
    shuffle : bool (default: True)
    Shuffles training data every epoch if True
    to prevent cycles.
    random_state : int
    Random number generator seed for random weight
    initialization.
    Attributes
    -----------
    w_ : 1d-array
    Weights after fitting.
    cost_ : list
    Sum-of-squares cost function value averaged over all
    training samples in each epoch.
"""
def __init__(self, eta=0.01, n_iter=10,
                shuffle=True, random_state=None):
    self.eta = eta
    self.n_iter = n_iter
    self.w_initialized = False
    self.shuffle = shuffle
    self.random_state = random_state

def fit(self, X, y):
    """ Fit training data.
    Parameters
    ----------
    X : {array-like}, shape = [n_samples, n_features]
    Training vectors, where n_samples is the number
    of samples and
    n_features is the number of features.
    y : array-like, shape = [n_samples]
    Target values.
    Returns
    -------
    self : object
    """
    self._initialize_weights(X.shape[1])
    self.cost_ = []
    for i in range(self.n_iter):
        if self.shuffle:
            X, y = self._shuffle(X, y)
        cost = []
        for xi, target in zip(X, y):
            cost.append(self._update_weights(xi, target))
        avg_cost = sum(cost) / len(y)
        self.cost_.append(avg_cost)

    return self

def partial_fit(self, X, y):
    """Fit training data without reinitializing the weights"""
    if not self.w_initialized:
        self._initialize_weights(X.shape[1])
    if y.ravel().shape[0] > 1: #if we have more than one sample
        for xi, target in zip(X, y):
            self._update_weights(xi, target)
    else:
        self._update_weights(X, y)

    return self

def _shuffle(self, X, y):
    """Shuffle training data"""
    r = self.rgen.permutation(len(y))

    return X[r], y[r]

def _initialize_weights(self, m):
    """Initialize weights to small random numbers"""
    import numpy as np

    self.rgen = np.random.RandomState(self.random_state)
    self.w_ = self.rgen.normal(loc=0.0, scale=0.01,
                               size=1 + m)

    self.w_initialized = True

def _update_weights(self, xi, target):
    """Apply Adaline learning rule to update the weights"""
    output = self.activation(self.net_input(xi))
    error = (target - output)
    self.w_[1:] += self.eta * xi.dot(error)
    self.w_[0] += self.eta * error
    cost = 0.5 * error**2

    return cost

def net_input(self, X):
    """Calculate net input"""

    return np.dot(X, self.w_[1:]) + self.w_[0]

def activation(self, X):
    """Compute linear activation"""
    return X

def predict(self, X):
    """Return class label after unit step"""

    return np.where(self.activation(self.net_input(X))
                    >= 0.0, 1, -1)


Having trouble understanding the x and y axis in SVM when training and testing data I wrote some code based on this article.

In the code in the article they have created a partition of 80 percent test and 20 percent data

#What percentage of data you want to keep for training
percentage = 80
partition = int(len(hog_features)*percentage/100)

Later they have created the following variables, the data frame is two dimensions np array --- data_frame[hog_features,labels]:

x_train, x_test = data_frame[:partition,:-1],  data_frame[partition:,:-1]
y_train, y_test = data_frame[:partition,-1:].ravel() , data_frame[partition:,-1:].ravel()

clf.fit(x_train,y_train)

now what I don't understand is why for the second dimension of the array (the labels) for the x variables they're including everything except the last value and for the y variables the second dimension of the array is just including the last element of the array from what I can understand using array slicing (Maybe I'm wrong).

The answer to your questions is simply : that is how training works :

The training set is a matrix in which the last column correspond to the labels and the rest correspond to the samples.

When slicing  the matrix, you have two parts: the first one concerns the rows, the second one concerns the columns.

ROWS:

:partition means in fact the first 80% rows - this concerns the training set x_train, y_train

partition: means the last 20% rows - concerns the test set, x_test and y_test

COLUMNS:

:-1 mean all but the last column - concerns x_train and x_test

-1: mean just the last column - concerns y_train and y_test

Therefore, here you get for x_train and x_test everything but the last value. Be careful, in fact you get 80% of the data is for x_train (:partition) and 20% is for x_test (partition:). Now, for the labels you get only the last column, and 80% of the rows are for y_train while 20% of the rows are for y_test.

One more detail, your x_train and y_train is a 5D array, according to the documentation.


Which statistical test tells which classifier performs better than the other? I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do it? Will McNemar's test be enough to validate it? Which statistical test will tell me which classifier is better than the rest? And how?

Cochran's Q test

Is a generalisation of the McNemars test and can be used to see if there is a truly better classifier for the metric chosen. You can ofcourse also do pairwise Mcnemare test and draw conclusions from there.

NOTE:

These things are expensive


Weights shared by different parts of a transformer model Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?

Updated answer
The Transformer model has 2 parts: encoder and decoder.
Both encoder and decoder are comprised of a sequence of attention layers. Each layer is comprised of a combination of multi-head attention blocks, positional feedforward layers, normalization, and residual connections.
The attention layers from the encoder and decoder are slightly different: the encoder only has self-attention blocks while the decoder alternates self-attention with encoder attention blocks. Also, the self-attention blocks are masked to ensure causal predictions (i.e. the prediction of token N only depends on the previous N - 1 tokens, and not on the future ones).
In the blocks in the attention layers no parameters are shared.
Apart from that, there are other trainable elements that we have not mentioned: the source and target embeddings and the linear projection in the decoder before the final softmax.
The source and target embeddings can be shared or not. This is a design decision. They are normally shared if the token vocabulary is shared, and this normally happens when you have languages with the same script (i.e. the Latin alphabet). If your source and target languages are e.g. English and Chinese, which have different writing systems, your token vocabularies would probably not be shared, and then the embeddings wouldn't be shared either.
Then, the linear projection before the softmax can be shared with the output embedding matrix. This is also a design decision. It is frequent to share them.
Finally, the positional embeddings (which can be either trainable or pre-computed) are shared for the source and target languages.

Old Answer (mistakenly addressing multi-encoder/decoder transformers)
In general, parameters are not shared in multi-encoder/multi-decoder transformer architectures, in the sense that each encoder/decoder has its own parameters. This is because sharing parameters defeats the very purpose of having multiple encoders/decoders: if you have two encoders that share parameters, they are effectively the same encoder.
This, of course, is not a rule, and there may be cases where it makes sense to share some or all encoder/decoder parameters. An example could be a scenario where two sentences in a certain language (or two very similar languages, like dialects) are received as input. It may be beneficial to share parameters between both encoders if the amount of data in one of them is not enough.
Background: multi-encoder and/or multi-decoder Transformers are generally not used for "simple" sequence to sequence tasks like machine translation. There are certain special cases where this kind of architectures have been used with different purposes, e.g.:

Automatic Post-Editing (APE) consists of having a primary machine translation system which output is refined with a secondary translation system that corrects the errors from the primary system. The secondary system normally receives as inputs both the original source sentence and the translation from the primary system and generates the fixed translation as output. An option for this kind of scenario is having a dual-encoder single-decoder transformer, where the output of the encoders are both fed to the decoder, either concatenating them (example) or injecting them to different attention blocks (example). In this scenario, normally there is no parameter sharing at all.

Multimodal translation: in this case, we receive different data modalities (e.g. speech and text), so we need modality-specific architectures, so normally there is no parameter sharing.

Note that here I understand that the multiplicity in encoders and/or decoders happens both at training and inference time. There are other cases where such a multiplicity happens only at training time (as in multi-task learning), but at inference time only one encoder and decoder are selected for each case, e.g.:

Multilingual translation: while many multilingual MT systems have a single model for all supported languages (normally supplying special tokens that identify the source/target language), it is possible to have language-specific encoders or decoders. You can see examples of this kind of architecture in works like this or this. In this scenario, each encoder/decoder has its own parameters and there is no sharing.


Derivates with respect to a vector Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,
I want to know how can we take the derivative of $f \text{ w.r.t. } X$? I understand differentiation w.r.t one variable, but how does differentiation of another vector/matrix w.r.t a vector work?

Differentiating a function w.r.t a variable gives us the rate at with the function changes when we change the variable by a small amount. What does differentiating w.r.t a vector signify?

$f =X^T Y$ looks like this
\begin{equation}
 f= \begin{pmatrix}
  \sum_i x_iy_{i1}\\
  \sum_i x_iy_{i2}\\
  \vdots\\
  \sum_i x_iy_{in}\\
 \end{pmatrix}
\end{equation}
where $x_i$ is the $i^{th}$ element of $X$ and $y_{ij}$ is the $(i,j)^{th}$ element of $Y$.
Now, to get the gradient w.r.t $X$, is equivalent to deriving each element of $f$ w.r.t each element of $X$. This will lead to a 2D matrix:
\begin{equation}
 \nabla_X f =
 \begin{pmatrix}
  \frac{\partial}{\partial x_1} f_1 & \ldots & \frac{\partial}{\partial x_n} f_1\\
  \vdots & \vdots & \vdots \\
  \frac{\partial}{\partial x_1} f_n & \ldots & \frac{\partial}{\partial x_n} f_n
 \end{pmatrix}
\end{equation}
We can therefore easily see that
\begin{equation}
\frac{\partial}{\partial x_j} f_i
=
y_{ij}
\end{equation}
Therefore $\nabla_X f = Y$.


Interpretability of RMSE and R squared scores on cross validation I'm working on a regression problem with 30k rows in my dataset, decided to use XGBoost mainly to avoid processing data for a quick primitive model. And i noticed upon doing cross-validation that there's a noticeable difference between R² for train and R² for CV => clear signs of overfitting. Here's my code for CV :

oof_train = np.zeros((len(train_maisons)))
ind = 0
cv_scores = []
train_scores=[]
for ind,(ind_train,ind_val) in (enumerate (kfolds.split(X,y))):
    #ind=+1
    X_train,X_val = X.iloc[ind_train],X.iloc[ind_val]
    y_train,y_val = y.iloc[ind_train],y.iloc[ind_val]
    xgb = XGBRegressor(colsample_bytree=0.6,gamma=0.3,learning_rate=0.1,max_depth=8,min_child_weight=3,subsample=0.9,n_estimators=1000,objective='reg:squarederror',eval_metric='rmse')
    xgb.fit(X_train,y_train)
    val_pred = xgb.predict(X_val)
    train_pred = xgb.predict(X_train)
    oof_train[ind_val] += val_pred
    score_fold_validation=np.sqrt(mean_squared_error(y_val, val_pred))
    score_fold_train=np.sqrt(mean_squared_error(y_train, train_pred))
    train_scores.append(score_fold_train)
    cv_scores.append(score_fold_validation)
    #r2_score(y_val, grid.best_estimator_.predict(X_val))
    print('Iteration : {} - CV Score : {} - R² Score CV : {} - Train Score : {} - R² Score train : {}'.format(str(ind+1),score_fold_validation,r2_score(y_val, val_pred),score_fold_train,r2_score(y_train,train_pred)))
end_train_score=np.mean(train_scores)
train_scores.append(end_train_score)
end_cv_score=np.mean(cv_scores)

Using SquaredError as objective ( loss function ) , evaluating with RMSE and R², here are the metrics' outputs :

CV Score : 96416.84137549331 - R² Score CV : 0.6545903695464426 - Train Score : 30605.655815355676 - R² Score train : 0.9730563148067477

My question: is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters ). And one last thing, is my result on X_train indicative that my features are informative enough to learn the target? or is the R² train score somehow biased?

Note : In this code i'm using 10 folds for CV. Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.

And one last thing, is my result on X_train indicative that my features are informative enough to learn the target? or is the R² train score somehow biased?

High scoring fits on training data does not necessarily indicate that your features are informative enough to learn the target in a general fashion. Only your cross validation scores can do so.

  Note : In this code i'm using 10 folds for CV. Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.

I wouldn't expect wildly significant differences so it depends on how much better they were, but remember that you're randomly selecting a subset of data for each fold. It's plausible that when you ran it for 3 folds the model was coincidentally better able to learn from the selected training sets to predict the validation sets. Generally speaking, the more folds you use the more confident you should be in the reliability of the score.

  My question: is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )

I'm no professional, but I would consider that to be severely overfit. I would in the first instance return to hyperparameter tuning to try and bring them in line. I'll also  perform the obligatory plug of RandomizedSearchCV at this point: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

EDIT:

Since you've asked for further details:

1) Does a high score against training data indicate that the features are informative enough to learn the target?

Consider the following example:

import numpy as np
from xgboost import XGBRegressor

x = np.random.rand(5000, 12)
y = np.random.randint(0, 200, 5000)

rfr = XGBRegressor(colsample_bytree=0.6,gamma=0.3,learning_rate=0.1,max_depth=8,min_child_weight=3,subsample=0.9,n_estimators=1000,eval_metric='rmse')
rfr.fit(x, y)
rfr.score(x, y)

  Out[26]: 0.999918392694166

Entirely random targets trained against entirely random inputs, and still scoring almost perfectly. That's a completely useless model; validating against your training data does not allow you to determine that your features are informative enough to allow you to predict the target against data that your model has not seen. Only cross-validation against unseen data can do this.

2) Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.

The RNG gods smiled upon you. Again, consider an example:

from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import KFold

df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv', index_col=None)

x = df.drop('species', axis=1)
y = df.species

xgb = XGBClassifier(max_depth=3)

kf = KFold(n_splits=10, shuffle=True)

for train_idx, test_idx in kf.split(x):
    xgb.fit(x.iloc[train_idx], y.iloc[train_idx])
    print(xgb.score(x.iloc[test_idx], y.iloc[test_idx]))

  Out[17]: 0.9333333333333333
  0.9333333333333333
  0.9333333333333333
  1.0
  0.8666666666666667
  1.0
  1.0
  0.9333333333333333
  0.8666666666666667
  1.0

That code splits the data 10 times and then trains and scores the model on each selected slice. As you can see, there's a 13.4% difference between the highest and worst scoring slices.

Your 3-Fold run coincidentally gave you 3 'good' folds. The more Folds you have, the more representative of the true result your cross-validation is.

3) is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )

Well, whether or not it's overwhelming is something of a matter of opinion. In my mind however a loss against the validation set that is 300% the training score is very severely overfit indeed, but if your validation RMSE is still within the margin of error you're willing to accept then I suppose you could go ahead and use it anyway. It's really your call.

To reduce the overfitting, you need to tune your hyperparameters better. Reducing max_depth and increasing min_samples_split is my usual go-to with trees. If your revised model (exhibiting either no overfitting or at least significantly reduced overfitting) then has a cross-validation score that is too low for you, you should return at that point to feature engineering. We can highlight the effect of hyperparameters on overfitting quite easily by plotting the effect of tweaking them:

from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.preprocessing import OneHotEncoder

df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv', index_col=None)

ohe = OneHotEncoder(sparse=False)
x = pd.DataFrame(ohe.fit_transform(df[['cut', 'color', 'clarity']]))

df.drop(['cut', 'color', 'clarity'], axis=1, inplace=True)

df = df.merge(x, left_index=True, right_index=True)

x = df.drop('price', axis=1)
y = df.price

cv = {}
for i in range(20):
    xgb=XGBRegressor(max_depth=i+1)
    x_train, x_test, y_train,  y_test = train_test_split(x, y)
    xgb.fit(x_train, y_train)
    cv[i+1] = (xgb.score(x_train, y_train), xgb.score(x_test, y_test))

train = []
test = []
for i in cv.keys():
    train.append(cv[i][0])
    test.append(cv[i][1])

fig = plt.figure()
plt.plot(train)
plt.plot(test)
plt.legend(['train', 'test'])
plt.xlabel('max_depth')
plt.xlim(0, 20)
plt.ylabel('R^2 Score')
plt.show()

Apologies for the stupid x-axis labels :P. We can see the affect of the max_depth hyperparameter on the model's tendency to overfit easily; anything over 3-ish and the train/test scores begin to diverge.

Hope that those revisions are more helpful to you :)


Which metric to use for evaluating a rating system I have a system which gives a star rating of the quality of work(on scale of 1-5, 1 being extremely poor and 5 being exceptionally good).
An expert labelled a test set with their ratings of quality of work.
The data is imbalanced in the 5 classes(1,2,3,4,5), very few samples for Extremely poor and Exceptionally good ratings.
Is there a metric which can take into account consideration that if system rated extremely poor quality work as very good or exceptionally good, it should be penalised more, as compared to when it rated an extremely poor rating as poor or average? And again due to imbalanced class distribution, straightforward accuracy is not the best measure.
Any suggestions for metric?

Performance Measures for Multi-Class Problems
For classification problems, classifier performance is typically defined according to the confusion matrix associated with the classifier. Based on the entries of the matrix, it is possible to compute sensitivity (recall), specificity, and precision. For a single cutoff, these quantities lead to balanced accuracy (sensitivity and specificity) or to the F1-score (recall and precision). To evaluate a scoring classifier at multiple cutoffs, these quantities can be used to determine the area under the ROC curve (AUC) or the area under the precision-recall curve (AUCPR).
All of these performance measures are easily obtainable for binary classification problems. Which measure is appropriate depends on the type of classifier.
F-Measure/F1-Score for Imbalanced Classification
Classification accuracy is widely used because it is one single measure used to summarize model performance. F-Score provides a way to combine both precision and recall into a single measure that captures both properties.
Alone, neither precision nor recall tells the whole story. We can have excellent precision with terrible recall, or alternately, terrible precision with excellent recall. F-measure provides a way to express both concerns with a single score.
Once precision and recall have been calculated for a binary or multiclass classification problem, the two scores can be combined into the calculation of the F-Measure. The traditional F measure is calculated as follows:
$\begin{equation}
\nonumber
F-Measure=\frac{2 * Precision * Recall}{Precision + Recall}
\end{equation}$
For a multi-class classification problem, we don’t calculate an overall F-1 score. Instead, we calculate the F-1 score per class in a one-vs-rest manner.
Here are simple Python functions that do exactly that:
def precision(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    len1 = len(y_pred)
    if len1 == 0:
        return 0
    else:
        return len(i) / len1

def recall(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    return len(i) / len(y_true)

def f1(y_true, y_pred):
    p = precision(y_true, y_pred)
    r = recall(y_true, y_pred)
    if p + r == 0:
        return 0
    else:
        return 2 * (p * r) / (p + r)

For imbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC or Micro and macro averages of the F1-score
Weighted F1-Score was calculated by F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:
$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$
therefore favoring the majority class.
Micro and macro averages of the F1-score
Micro and macro averages represent two ways of interpreting confusion matrices in multi-class settings. Here, we need to compute a confusion matrix for every class $g_i \in G = \{1, \ldots, K\}$ such that the $i-th$ confusion matrix considers class $g_i$ as the positive class and all other classes $g_j$ with $j≠i$ as the negative class. Since each confusion matrix pools all observations labeled with a class other than $g_i$ as the negative class, this approach leads to an increase in the number of true negatives, especially if there are many classes.
To exemplify why the increase in true negatives is problematic, imagine there are 10 classes with 10 observations each. Then the confusion matrix for one of the classes may have the following structure:
Prediction/Reference    Class 1     Other Class
Class 1                   8             10
Other Class               2             80

Based on this matrix, the specificity would be $\frac{80}{80 + 10} = 88.9\%$ although class 1 was only correctly predicted in 8 out of 18 instances (precision 44.4%). Thus, since the negative class is predominant, the specificity becomes inflated. Thus, micro- and macro averages are only defined for the F1-score and not for the balanced accuracy, which relies on the true negative rate.
In the following, we will use $TP_i$, $FP_i$, and $FN_i$ to respectively indicate true positives, false positives, and false negatives in the confusion matrix associated with the $i-th$ class. Moreover, let precision be indicated by $P$ and recall by $R$.
The micro average has its name from the fact that it pools the performance over the smallest possible unit (i.e. over all samples):
$\begin{align*}
P_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i+FP_i} \\
R_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i + FN_i}
\end{align*}$
The micro-averaged precision, $P_{micro}$, and recall, $R_{micro}$, give rise to the micro F1-score:
$F1_{\rm{micro}} = 2 \frac{P_{\rm{micro}}  \cdot R_{\rm{micro}}}{P_{\rm{micro}} + R_{\rm{micro}}}$
If a classifier obtains a large $F1_{micro}$, this indicates that it performs well overall. The micro-average is not sensitive to the predictive performance for individual classes. As a consequence, the micro-average can be particularly misleading when the class distribution is imbalanced.
The macro average has its name from the fact that it averages over larger groups, namely over the performance for individual classes rather than observations:
$\begin{align*}
P_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i+FP_i} = \frac{\sum_{i=1}^{|G|} P_i}{|G|}\\
R_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i + FN_i} = \frac{\sum_{i=1}^{|G|} R_i}{|G|}
\end{align*}$
The macro-averaged precision and recall give rise to the macro F1-score:
$F1_{\rm{macro}} = 2 \frac{P_{\rm{macro}}  \cdot R_{\rm{macro}}}{P_{\rm{macro}} + R_{\rm{macro}}}$
If $F1_{macro}$ has a large value, this indicates that a classifier performs well for each individual class. The macro-average is, therefore, more suitable for data with an imbalanced class distribution.
References:

http://text-analytics101.rxnlp.com/2014/10/computing-precision-and-recall-for.html
https://sebastianraschka.com/faq/docs/multiclass-metric.html
https://stats.stackexchange.com/questions/463224/which-performance-metrics-for-highly-imbalanced-multiclass-dataset
What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?
https://www.datascienceblog.net/post/machine-learning/performance-measures-multi-class-problems/
https://sebastianraschka.com/faq/docs/computing-the-f1-score.html


Binary classification of similar images with small region of interest I have a dataset of microscope images and I want to train a ML/DL algorithm to perform binary classification. The positive class is when there is only one cell in the image, and the negative class is everything else (i.e. when there are either more than 1 cells, or no cells at all).

Below is one of the original images. (there is a cell in the curved site on the center of the image)

Due to the big size of the images (2048x2048) and the excess of information (the cells can only be in the tube-system), I decided to preprocess them. So, I set everything outside the tube-system to 0 (black) and I crop all the images to the boundaries I got by averaging the images of the whole dataset. Below you can see the end result. (there are 2 cells in the tube, one in the center and one at the upper left part)

Then I tried to train a CNN (using python and TensorFlow). I played a few times by changing its hyperparameters, but I had no luck. I think the problem is that the cells (region of interest) are occupying a very small portion of the image, which makes it hard for the algorithm to focus on. To make things worse, there are random dust particles around which make the image noisy.

Do you have any ideas of how I could perform a robust binary classification in such a dataset?

I would use a CLAHE preprocessing, SIFT Image features and mask the wrong detected keypoints out as geometric constrains. then i would count sift keypoints in a image without and keypoints to get a thredhold boundary and classify by amount of robust keypoints or logistic regression


Number of parameters in seq2seq model I am trying to compute the total number of parameters in Ilya Sutskever's seq2seq model.
In the paper, the authors mention a total number of 384M trainable parameters, or to be exact:

  The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM)

In short, their model has:

one encoder LSTM and one decoder LSTM
each LSTM has four layers
each LSTM layer has 1000 units
input: 1000-dimensional word embedding of 160K possible input words
output: naive softmax over 80K possible words

Following this post and using the formula
$$
num~params = 4 (nm + n^2)
$$

I was able to compute:

For the encoder LSTM

n = 1000 for all layers
m = 1000 for the first hidden layer, because of 1000 dimensional embedding
m = 1000 for all other layers, because that's the number of units in each hidden layer

which yields:
input -> first hidden layer: 8M
first -> second            : 8M
second -> third            : 8M
third -> last              : 8M
total                      = 32M

which I am guessing is what they mean when they say 32M recurrent connections for the LSTM?

For the decoder LSTM

computations are exactly the same as above, with the caveat that the input size is 1000 because that's the output size of the encoder LSTM.

total                      = 32M

Where are the other 320M parameters?

The computations made so far account for 64M parameters.
We still haven't accounted for the final softmax layer, which admittedly could be pretty large since its output is 80K, and 320M can be suspiciously decomposed as 80K * 4K, but I'm not sure about this.

My questions are:

can the naive softmax layer alone account for the 320M missing parameters?
shouldn't we also be using the fact that this is a recurrent neural networks to compute additional weights, as in this post?

Thank you

I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation?

The reason for the remaining 320M parameters

in the input of the encoder, computing the word embeddings requires taking a one-hot vector of 160K words and transforming it into a 1K dimensional word embedding. Thus this accounts for 160M parameters;
the softmax layer of the decoder takes a 1K dimensional input (the hidden state of the last layer of LSTM cells) and transforms it into a vector of 80K probabilities. This accounts for 80M parameters;
the input of the encoder is given not only by the hidden state of the decoder, but also by the previous output of the encoder. This requires taking a 80K dimensional vector of possible words and transforming it into a 1K dimensional word embedding. Thus this accounts for 80M parameters.

Does this explanation make sense?

Recurrent connections

As far as I understand, once the cell state of the LSTM is computed, it is transferred as is to the next timestep. Thus there are no additional parameters from the recurrent network (such as, e.g., was the case in this post).


Is there a comprehensive learning plan for a self-teaching beginner in data science with no programming background? Situation

I want to be a data scientist, but I have no programming or maths background.
I have more than the basic understanding of python which I taught myself over a couple of months. Right now I am stranded, because I do not know how to frame a study plan?

I am working in a marketing firm with lots of data, so I have an income for the time that I study and I have data to apply my knew knowledge on. I am not trying to rush up and want to build up a strong base and grow upon it over a longer period of time. I know it is not going to be easy, but I feel strongly motivated.

Questions

Could you point out comprehensive learning plans with references to books, courses, and learning materials for self-teaching beginners? Could you also point out any articles or presentations in which former beginners and transitioners share their experiences about successfully becoming data scientists?

Short Answer:

Practice.  You already see it's both math and programming, so start practicing your math and your programming and eventually you'll be there.

Long Answer:

Data science happens at the intersection of math and programming.

A lot of people start with some intense math (biotech, physics, engineering, or if you're lucky statistics) and thus have to learn the programming from scratch.

A lot of others start with being some professional developer, and then have to learn the math.

But as you said, you know neither.  So check your motivations.  Do you really want to be a data scientist, or are you just caught up in the buzz that still surrounds it?  You really have to love it to endure the massive learning curve.  And if you love it, figure out why you're only now getting started.

Alright.  Still on board?  If so, you really have three paths as I see it.

you can follow along with some mooc or series of tutorials to spoon feed you some basic project.  It is a start.
You can first try to break into being an analyst first.  Could be a much shorter path.  Best case scenario, analysts can turn directly into data scientist.  Worst case scenario, you get stuck not quite living your dream.  If you take this route with the intention of eventually becoming a data scientist, be verrry wary of the tools you use.  Highly automated analyst tools keep you locked into what the tools can do.  And if the tools don't have any statistics or machine learning baked in, then you'll get stuck.
The hard way.  If you're serious about it, then know you need to learn both overlapping languages - math and programming.  Ask around and do your research.  Find what programming languages will work best in the niche you expect you'll most like.  A part of you "do I really want to be a data scientist?" should includes the types of things you want to do as a data scientist.  Out of your answer, pull a few examples of actual projects.  Of those projects, pick the easiest and learn the math behind it.  Like financial stuff?  Perhaps some simple time series prediction on some open financial data.  Like marketing?  Perhaps start with looking up A/B tests.  Yes, this one may include also doing #1, but with the very important extra layer of sincerity around why you're doing it.  And a LOT of the job of a data scientist is being able to take a question nobody's asked yet, and figure out how to apply your tools to solve it.  Many moocs rob you of that most-necessary layer of first framing the problem before trying to solve it.


Can GPS coordinates (latitude and longitude) be used as features in a linear model? I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and end points; and (2) estimating the amount of crime for a specific point.

I'd like to use a linear regression model. However, can I use these GPS coordinates directly in a linear model?

Latitude and longitude do not have an ordinal property, such as with a person's age. For example, the two points (40.805996, -96.681473) and (41.226682, -95.986587) do not seem to have any meaningful ordering. They are just points in space. I was thinking of replacing them with categorical US zip codes and then doing one-hot encoding, but that would result in a lot of variables.

You cannot use them directly, as it is unlikely there is a true linear relationship unless you're looking to predict "how far east or north" someone is. As mentioned in the comments, you need to convert them into zones. If you wanted to keep it really simple, you could use a kNN clustering algorithm with a low number of potential clusters and then assign each instance a new feature with the cluster ID, and then one-hot encode that.

You may also want to read about how people interpolate coordinates to predict values across a whole map. The first example is with temperature stations, but you can also imagine it being "hot zones" for crime.

(DOCS)


Computer vision: Identifying different items in screenshot Lets say I have a screenshot like this:

I want to be able to detect/localize each item on the floor, however, there 1) can be any number of items in the image and 2) each item is different

I have a candidate list of all possible items. In reality each one is labelled and separated into individual image files:

I've thought about training something like a convnet, but that feels like it might be slow because I'd need to segment each screenshot with multiple sliding windows and feed each one forward through the net. Creating those sliding window segments for each screenshot will likely take a long time. I'd like the entire detection process to be completed quickly (<1sec)

Whats the most efficient way of doing this detection task? I will be implementing this using Javascript

The main problems are:

The item count is unknown. There could be 0 items in the screenshot, or there could be many
There are a lot of possible targets, each one of which are different in shape, colour, and structure
Javascript isn't the fastest of languages for this type of work. This is a semi strict requirement unless its absolutely impossible to do this using JS. The fallback language would be Python

This is a good case for computer vision.  You know exactly what the icon looks like (basically pixel-perfect), and you want to find whether it exists anywhere in the screenshot.  This is an image matching problem.

Normalized cross-correlation (NCC) is a standard way to check whether small image $I$ (your icon)  appears anywhere in the large image $I'$: you compute the NCC and then look for a location where the value of the NCC is large; that is your candidate match.  You can now apply this to the set of candidate icons: for each candidate icon, search whether it appears anywhere in the screenshot, using NCC.  OpenCV and other computer vision libraries should include normalized implementations of the NCC computation.

For speed, you can also try using image pyramids, but try using NCC directly on the full screenshot first.


About Hadley Wickham paper on tidy data I read this paper and understood it very well, but I miss negative impact that will happen if we don’t fix the 5 types of messy data introduced in the paper (page 5), or even how fixing them will make the analysis easier. Can any one give examples for this?

The general idea is to standardize the format of the data so that it can be used consistently across a wide range of methods for analysis and prediction.

The negative impact is the same for all 5 types of messy data: one would spend much more time implementing functions which do nothing more than converting from one very specific format to another. Additionally through these cumbersome conversions one is more likely to introduce errors or omissions in the data.

Basically following the "tidy data philosophy" can save a lot of time and makes your data readily usable with a wider range of methods.


Are there machine learning methods that can give captions to the New Yorker magazine cartoons? Are there  machine learning methods that can give captions to the New Yorker magazine cartoons?

Yes. Image captioning is a very active area of Deep Learning, at the intersection between Computer Vision and Natural Language Processing.

These models are based on an Encoder-Decoder structure. In its most classical form, the Encoder consists of Convolutional layers that process pixel data. Their representation is then fed to a Decoder based on Recurrent layers (usually LSTM or GRU) that generate the output sentence.

I suggest you to take a look at this tutorial on Image Captioning in TensorFlow 2. You can download the Notebook and run. That specific model is based on attention mechanism, which is a fancier version of seq2seq models. You can find tons of other models and tutorial simply googling them.


Can LSTM be used for non time series data? I have a dataset -

This is a TOR network traffic dataset with labels added as TOR/ Non TOR.
I want to run an LSTM on it and classify it as Tor/Non Tor. Is that possible since this is not a time series data?

If the packets are sorted in time and you think there might be some correlation between packets that help identify TOR/nonTOR, then you may try an LSTM.
I suggest you add a new column with the delta time since the previous packet.


Custom loss function with both min(y, p) and max(y,p) I'm creating a neural network in tensorflow and need to minimize the following loss function:
$\frac{max(y,p)}{min(y,p)}$ Where $y$ represents the true value and $p$ the predicted value. Since the loss function is not differentiable it becomes a problem when using gradient descent.
Update, now I'm trying to implement:
$p > y \rightarrow loss = log(p)-log(y)$
$p < y \rightarrow loss = log(y)-log(p)$
$p = y \rightarrow loss = log(y)-log(y) = log(p)-log(p)$
This is my code:
def custom_loss(y_true, pred):
    if pred > y_true:
        custom_loss = K.log(pred) - K.log(y_true)
    elif pred < y_true:
        custom_loss = K.log(y_true) - K.log(pred)
    else:
        custom_loss = K.log(pred)-K.log(pred)

    return custom_loss

if __name__ == '__main__':
    run = 1

    X = np.load("vectors_normal_2way.npy")

    with open("target2.pickle", "rb") as file:
        target_dict = pickle.load(file)

    target_strings = [*target_dict]

    Y = np.array([])

    for target_value in target_strings:
        Y = np.append(Y, target_dict.get(target_value))

    for i in range(run):
        X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
        model = get_model()
        start = time.time()
        model.fit(X_train, Y_train, verbose = 1, epochs = 1, batch_size = 32)
        end_time = time.time()-start
        pred = model.predict(X_test)[:, 0]

But I get the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError:  The second input must be a scalar, but it has shape [32]
     [[{{node gradient_tape/custom_loss/cond/StatelessIf/gradient_tape/custom_loss/weighted_loss/Mul/_17}}]] [Op:__inference_train_function_1040]

The code can run when I have batch_size = 1

A loss function calculates the error over all the data presented to it. For neural networks, that is an average over the mini-batch. Your code might only handle scalars (hence working when batch=1), it should handle vectors.
The K looks like you are using tf.Keras. The documentation gives examples of custom loss functions. It helps to subclasses tf.keras.losses.Loss which expects a call() method that contains the logic for loss calculation using y_true, y_pred.
class CustomLoss(Loss):

  def call(self, y_true, y_pred):
    pass


how do I predict the next's alarms ? (time series) I'm trying to solve a time series forecasting problem, where the main goal is to read data with various alarm logs and make a prediction about what may happen in the future.
Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the location of the problem etc. are stored in the alert. These fields are all categorical variables.
If someone already had an identical problem, it could help me find the best technologies and which ML models are most used in these cases and some practical examples of models.
I need to predict multiple occurrencies of future alarms. In this case the response variable is problem name.

**I know that in a pre-processing phase I have to encode the variables.
EDIT:
my goal is to have a multi-step output of the "problem name" label. This label is not numeric or percentual, its a string (categorical variable). i guess i will have to feed a model with data resembling something like this (after noramalization):

I guess that for this problem I will have to use LSTM models but my problem is that all the examples I see are trying to predicte continuous values, such as temperatures or stock values.
Are any models (and special layers) targeting this kind of problems?
Any good examples available online?

The first step is to pick a target variable. What are you trying to predict "problem name", "severity", "sub category", …?
Based on the target, it becomes either a conventional regression, ordinal regression, binary classification, multi-class classification, or multi-label classification problem.
Another issue might be how the date is encoded. "t1", "t2", … is not a conventional datestamp format.


Clustering Metrics for large data I have a dataset containing 150k rows and 10 columns. After clustering, I would like to get clustering metrics. Below are lists of metrics that I would like to use>

 "C_index", "Calinski_Harabasz", "Davies_Bouldin", "S_Dbw", "Silhouette"

I have a problem. I've tried getting values using clusterCrit (from R) and sklearn (from python), but when I tried to get each value by using clusterCrit, it took a huge amount of time, so I couldn't get it.

I think this is because of the large data (my RAM is 8GB). Is there any way for me to get these values?

Just use a subsample of your data.

There is little use in evaluating (nor clustering, usually) the entire data set. It's highly redundant. Trying to scale this to the entire data set is a waste of time.