Should I only use scalar labels with Keras LSTM? I have an array X_train = (1110,25,2) and a y_train = (1110,5,2). It means I use arrays with length of 25 for inputs and length of 5 for labels. But when I use:

model = Sequential()
model.add(LSTM(units = 25, return_sequences = True, input_shape = (25, 2)))
model.add(Dropout(0.2))
model.add(Dense(units = 2))
model.compile(optimizer = 'adam', loss = 'mean_squared_error')
model.fit(X_train, y_train, epochs = 100 , batch_size = 25)

It gives me this error in the last line of the code:

  ValueError: Error when checking target: expected dense_1 to have 2
  dimensions, but got array with shape (1110, 5, 2) [Finished in 5.1s
  with exit code 1]

The code works if I change the length of y_train to the 1, but I like to test longer y labels to train. What is the problem and how can I fix it?

EDIT:
I create X_train and y_train arrays with this code:

for i in range((len(training_set)%30) + 30 , len(training_set) - days ):
    X_train.append(training_set_scaled[i-30:i-5])
    y_train.append(training_set_scaled[i-5:i])
X_train, y_train = np.array(X_train), np.array(y_train)

This is the result of model.summary()

model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_45 (LSTM)               (None, 25, 25)            2800
_________________________________________________________________
dropout_45 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_46 (LSTM)               (None, 25, 25)            5100
_________________________________________________________________
dropout_46 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_47 (LSTM)               (None, 25, 25)            5100
_________________________________________________________________
dropout_47 (Dropout)         (None, 25, 25)            0
_________________________________________________________________
lstm_48 (LSTM)               (None, 25)                5100
_________________________________________________________________
dropout_48 (Dropout)         (None, 25)                0
_________________________________________________________________
dense_12 (Dense)             (None, 2)                 52
=================================================================
Total params: 18,152
Trainable params: 18,152
Non-trainable params: 0
_________________________________________________________________

EDIT2:
I have tried to solve my problem with RepeatVector() function in encoder-decoder approach with the following code:

model = Sequential()
model.add(LSTM(units = 25, return_sequences = True, input_shape = (25, 2)))
model.add(Dropout(0.2))
model.add(LSTM(units = 25, return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = 25)) #, return_sequences = True))
model.add(Dropout(0.2))
model.add(RepeatVector(5))
model.add(LSTM(units = 5 ,return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = 5 ,return_sequences = True ))
model.add(Dropout(0.2))
model.add(Dense(units = 2))

But I get this stupid result:

Can you post a model summary using:

model.summary()

Also, elaborate on how exactly the Y_train dataset works with the X_train?  It's not clear how the 25 time steps from X_train data correspond to the Y_train 5 outputs.


Tests to find which theory agrees better with observation I have three curves ( 1.> observation: yobs , 2.> theory-1: yth1 , 3.> theory-2: yth2 ). All of these curves are functions of a single variable (say variable x.) From a computational perspective, all these curves can be thought of as arrays with discrete values. To get an idea of error bars around  yobs, I use 100 numerical simulations. All arrays, yobs, yth1, yth2 and all 100 simulations have the same length (same number of values).

What statistical test can tell me which one of these two theories (yth1 or yth2)  is a more accurate representation of the observation ( yobs ) inside a given distance range ([x1, x2])?

Below is a schematic diagram of yobs, yth1 and yth2. The orange shaded region around yobs shows  error bar gotten from 100 simulations.

I have already tried  analysis using the formulae given underneath (here,  denotes covariance matrix obtained from the 100 simulations). But, I get very high values of / degree of freedom for comparisons of yth1 with yobs , and yth2 with yobs .  As such, I am looking for tests other than the  test.

Another test that I have tried is the method of fractional errors (shown in the two formulae given below). While the fractional tests tell me which theory is a more accurate model for observations, it does not use errors from 100 simulations. So, I am also trying to find an alternative to this.

I'm not sure about the best statistical way, but one practical approach would be to compare actuals v expected for each of the two hypothetical curves.

Calculating the sum of squared differences (i.e. sum((y_obs - y_theory)^2)) will give you a numerical estimate of goodness-of-fit for each curve. You could also use other metrics depending on your application (e.g. RMSE, RMSLE, SMAPE...).

Or, for each theoretical curve, you could look at the distribution of differences between the actual observation and the value predicted by the curve. It can be handy to view these empirical differences graphically.

I'm sure there are other statistical ways to evaluate goodness-of-fit.


R Programming rearranging rows and colums from timeline data I am new to R programming and data analytics. This is probably a trivial question but I could not find an answer. I'm also limited with my ontology of data science and data manipulation. So, I'm not sure if I conveyed my question right.

I have huge irregular time series data close to 1 Million records. They are all in the form of time stamps with different discrete attributes. I want to rearrange the data as shown below.

An Example of what I am trying to achieve FROM log entry of every car sold and the nature of the sale.

Date         Vehicle        Sale Type       Registration    Profit
1-1-2017    Toyota  Camry   Cash Sale          CA           2500
1-1-2017    Toyota Rav4     Lease              CA           8500
1-1-2017    Toyota Camry    Installment        AZ           250
2-1-2017    Toyota Corolla  Installment        NV           250
4-1-2017    Toyota Etios    Cash Sale          AZ           2500
4-1-2017    Toyota Camry    Lease              CA           8500
4-1-2017    Toyota Prius    Installment        CA           250
13-1-2017   Toyota Prius    Installment        VA           250

TO with rows having car names and columns containing cash sale, lease, installment

Vehicle Type    Cash Sale   Lease   Installment
Toyota Camry       1           1           1
Toyota Rav4                    1
Toyota Corolla                             1
Toyota Etios       1
Toyota Prius                               2

This would do the job:

#install packages
library(qdapTools)
library(dplyr)
#set working directory
setwd("C:/Users")

#read csv
test <- read.csv("test.csv")

#create dummy variable
#test.new <- dummy.data.frame(test$Sale.Type, sep = ".")
test.new <- cbind(test, mtabulate(test$Sale.Type))

#rename column
names(test.new)[names(test.new) == 'Cash Sale'] <- 'cashsale'

#making new columns
test.new1 <- test.new %>%
                group_by(Vehicle)

test.new2 <-  test.new1 %>%
                  mutate(sum_Cashsale = sum(cashsale)) %>%
                  mutate(sum_Ins = sum(Installment)) %>%
                  mutate(sum_Lease = sum(Lease))

#select few necessary colums
test.new3 <- subset(test.new2, select=c("Vehicle", "sum_Cashsale", "sum_Ins","sum_Lease"))

#select the distict rows
test.new4 <- dplyr::distinct(test.new3)

Just copy paste the code.
saved your sample data to test.csv
Give your working directory details, do let me know if you have any issues.


Scaling the data iteratively one by one vs batch scaling I have 2000 signals in a dataset of shape (2000, 400000) where each signal is recorded within the range -127, 128. I want to downscale each signal from (-127, 128) to (-1,1) to save memory space and also for better visualization. There are two approaches:
Approach 1:
Iteratively apply minmax_scale individually at each signal something like the following:
from sklearn.preprocessing import minmax_scale
data = read_dataset(...)
for i in range(data):
    minmax_scale(i, feature_range=(-1, 1))

Approach 2:
Fit the whole dataset using MinMaxScaler something like the following:
from sklearn.preprocessing import MinMaxScaler
data = read_dataset(...)
scalar = MinMaxScaler(feature_range=(-1,1))
scaled_data = scalar.fit_transform(data)

I use the first approach because the dataset does not fit the memory but I am worried if it is incorrect. I want to make sure if my choice is sound, in theory.
Thank you very much.

With the first approach, you are completely disregarding the global scale of the signal and only focusing on the relative scale. This will most probably hurt the performance of whatever system you train on that data (or any analysis you perform on it), compared to training on the globally scaled data, unless the relative values are the only important pieces of information in your signals.
I noticed that you already know the range in which the data is defined: (-127, 128). If you want to scale your data, why don't you scale the data with a fixed computation like $((x + 127) \cdot 2 / 255) - 1$ ?
Anyway, I don't see how a mere linear change in scale would save memory or help visualization, which seem to be your final goal.


What is noise-tolerant learning? I was reading this paper and came across the below paragraph. Can you please help me understand what does the highlighted term noise-tolerant learning or noisy-labeled training data mean with a simple example and how is it useful when we don't have labels in our dataset etc? I am learning ML and your inputs would be helpful

To address the scarcity of labeled training data, Chen et al used
active learning to intelligently select training samples for labeling,
demonstrating that classifier performance could be preserved with
fewer samples.16 Another trend is the use of “silver standard training
sets,” a semisupervised approach where training samples are labeled
using an imperfect heuristic rather than by manual review.17–22 The
intuition is that noise-tolerant classifiers trained on imperfectly
labeled data will abstract higher order properties of the phenotype
beyond the original labeling heuristic (so-called “noise-tolerant
learning”23).

Learning that can generalise well.
Take for example differnetial privacy. There you inject noise on Purpose to anonymise your data, and in the process of you losse accuracy. Goal is to find such algorithms, that will with smart noise injections, be able to generalise and Keep the good accuracy Level.


Forecasting via LSTM or XGBoost... is it really a forecast or I guess I understand the idea of predictions made via LSTM or XGBoost models, but want to reach out to the community to confirm my thoughts.

This tutorial does a nice job explaining step by step of what to do:
"How to Develop Multi-Step LSTM Time Series Forecasting Models for Power Usage"

However, when it came to forecasting, the author held out portion of the data and then used that data to predict future values... In my mind, it is not really a forecasting (something that is done via ARIMA, VAR, etc - where you specify number of time periods and then don't specify anything else and the model gives you some forecasting for the future)
You can see my comments to the author and basically the predictions via LSTM or XGBoost models will be based on the input values and not on the future data range, i.e. I would need to supply temp, humidity, wind, etc to get the forecast of the power consumption for the house hold.

That being said, in what extend LSTM or GXBoost are used in forecasting? Given these values predict what would be the final outcome? Hence if I need to forecast something in the future and have no clue about the other input values, just stick to traditional VAR model?

By looking at this question/answer by @Fnguyen, "If you have more input variables you need a way to forecast or impute these because to make a prediction your model needs all inputs that build the model. " it seems that I should use ARIMA to forecast other input values and only then use LSTM... makes me wonder if forecast would be that accurate now that it depends on ARIMA that would predict future values of the input of LSTM and then LSTM model that would make final prediction. Then again, accurate forecast is fairly complicated thing to do as we are trying to predict the future.

However, when it came to forecasting, the author held out portion of the data and then used that data to predict future values... In my mind, it is not really forecasting (something that is done via ARIMA, VAR, etc - where you specify number of time periods and then don't specify anything else and the model gives you some forecasting for the future)

Generally speaking, I think the term "forecasting" can apply to any problem where the goal is to predict future events/values.

You are describing a subclass of forecasting problems called "univariate time-series forecasting".  In the typical case of univariate time-series forecasting, a model is built using only historical observations of the target variable.  ARIMA and VAR models are commonly used for these problems. (Although there are almost always multivariate extensions of these models.)

There's also "multivariate time-series forecasting", where the time-series includes more than one time-dependent variable, and each variable might depend on both its past values and the past/present values of other variables.  Weather forecasting is a good example of this type of problem.

  [To what extent are LSTM or XGBoost ] used in forecasting?

As you have correctly pointed out, models like XGBoost are only useful in cases where you have additional inputs other than historical observations of the target.  (LSTMs can be actually used with or without additional inputs.)

It's hard to say for sure how common XGBoost or any other model is in industry, but there is a
pretty
huge
body
of
research
on forecasting with exogenous inputs.

I once worked on a project where the goal was forecasting solar irradiance at a solar farm.  Naturally, there's a significant amount of seasonal variation in solar irradiance.  But local weather conditions like cloud cover, fog, and temperature also have a big impact on irradiance, so a univariate model is not sufficient.

We trained XGBoost, Random Forest, SVM, and deep learning models to forecast future irradiance.  The input to these models were forecasted weather varaibles from one of NOAA's numerical weather models, and I think this is a very common approach.

  [If] I need to forecast something in the future and have no clue about the other input values, just stick to traditional VAR model?

Yep, if you don't have any information about other variables, then you're restricted to univariate time-series methods.  But even if you don't know the other inputs with 100% confidence, you may be able to find reasonable predictions that could be leveraged to build a more sophisticated model :)


sklearn NearestNeighbors returning misaligned results I'm using NearestNeighbors to do name matching and at a certain point the results become misaligned. My standardized list of names is 100s of millions.  My list of names coming in to be matched is considerably smaller but still could be in the 250k to 500k range.  After a certain point it appears the index begins to shift by 1 or more.

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values) # set used for increased performance
#matching query:
def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices

print('Getting nearest n...')
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list
print('Finding matches...')
matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

print('Building data frame...')
matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
print('Data frame built')

It appears that once my standardized list gets over 80k it begins to shift the results down.

The "messy name" of VITALI, ANGELO (has a comma)

  VITALI, ANGELO

The standardized list of names may include these(no comma)

  VITALI ANGELO
  SENSABLE TECHNOLOGIES INC

After running it through the above matching the result below shows that VITALI, ANGELO is a near perfect match to SENSABLE TECNOLOGIES INC, because the index has shifted down one...I think.

  0.00  SENSABLE TECHNOLOGIES INC   VITALI, ANGELO

Is it possible that the size or number of records is  exceeding that matrix limits and it somehow messes the indices?

You can refactor your code to make this issue easier to investigate.

Something like this:

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf)
orgs = list(set(names['VariationName'].values))

queryTFIDF_ = vectorizer.transform(orgs)
distances, indices = nbrs.kneighbors(queryTFIDF_)

matches = pd.DataFrame(columns=['Match confidence (lower is better)', 'Matched name', 'Original name'])

for distance, index, org in zip(distances, indices, orgs):
    match_confidence = round(distance, 2)
    matched_name = clean_org_names.values[index][0][0]
    matches.append(match_confidence, matched_name, org)


Using clustering and Lasso with cv I used clustering on my dataset. Now when I'm trying to use a LASSO with cv to predict a response, one of the variables it takes into consideration is which cluster a new point is classified into.(I included the cluster variable as a predictor to see if being in a particular group affects the response)
Since the information on all variables is already captured by the cluster variable,using it again in the Lasso model with some other variables,does it become redundant/biased?

Since the information on all variables is already captured by the cluster variable, using it again in the Lasso model with some other variables, does it become redundant/biased?

It does not become biased, at least not in the statistical sense. Depending on the clustering mechanism, the information may be redundant.

If you're using regression with the RMSE as the loss function and the squared-error as your loss metric for your clustering algorithm, then the information will work out to be redundant.

Regardless, it's harmless to throw into the model and test this. The LASSO should learn that this information is redundant.

Here's a simulation in R.

library(glmnet)
library(Matrix)

n <- 1e5
nclusters <- 5
set.seed(420)
rmse <- function(y, yhat){
  return (sqrt( sum( (y-yhat)**2 )))
}
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.-1,data=ls)

print(head(xs))

# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- runif(dim(xs)[2])

# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))

# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.-1, data = data.frame(cluster = factor(clusterpred[,1])))
# Concatenating the original features and the assigned clusters
totalxs <- cbind2(xs, cxs)

head(cxs)

# Setting alpha = 1 implies LASSO for the GLMNET Package
model <- cv.glmnet(y=ys, x=xs, alpha=1)
cmodel <- cv.glmnet(y=ys, x=cxs, alpha=1)

# Running on the clusters and the original features
totalmodel <- cv.glmnet(y=ys, x=totalxs, alpha=1)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
yhatt <- predict(totalmodel, totalxs)

# Looking at the difference RMSEs
print(rmse(ys, yhat))
print(rmse(ys, yhatc))
print(rmse(ys, yhatt))

# It seems to select most of the features and *one* cluster
print(coef(totalmodel, s='lambda.min'))

In this simulation, we see that the performance of the model slightly worse, but fairly similar, from including the clusters.


Cosine similarity vs The Levenshtein distance Cosine similarity vs The Levenshtein distance

I wanted to know what is the difference between them and in what situations they work best?

As per my understanding:

Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. The cosine of 0° is 1, and it is less than 1 for any angle in the interval (0,π] radians.

The Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits

My question is

When would one use Cosine similarity over The Levenshtein distance?

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between sequences of characters.

Nevertheless they both can be used in non-traditional settings and are indeed comparable:

the vectors compared with cosine can for instance contain frequencies of characters or characters n-grams, hence making it a string similarity measure
one can replace the sequence of characters with a sequence of strings or a sequence of n-grams, thus making Levenshtein a more general distance measure.

The main conceptual difference between Cosine and Levenshtein is that the former assumes a "bag-of-words" vector representation, i.e. compares unordered sets, whereas the latter takes into account the order of the elements in the sequences.

In the context of comparing sequences of words many combinations are possible. In case that's what you're looking for you might be interested in this paper: https://www.aclweb.org/anthology/C08-1075/ (full disclosure: I'm one of the authors).


what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule) describes the probability of an event, based on prior knowledge of conditions that might be related to the event.

wiki gives this example to illustrate Bayes' theorem

  Suppose that a test for using a particular drug is 99% sensitive and
  99% specific. That is, the test will produce 99% true positive results
  for drug users and 99% true negative results for non-drug users.
  Suppose that 0.5% of people are users of the drug. What is the
  probability that a randomly selected individual with a positive test
  is a drug user?

${\displaystyle {\begin{aligned}P({\text{User}}\mid {\text{+}})&={\frac {P({\text{+}}\mid {\text{User}})P({\text{User}})}{P(+)}}\\&={\frac {P({\text{+}}\mid {\text{User}})P({\text{User}})}{P({\text{+}}\mid {\text{User}})P({\text{User}})+P({\text{+}}\mid {\text{Non-user}})P({\text{Non-user}})}}\\[8pt]&={\frac {0.99\times 0.005}{0.99\times 0.005+0.01\times 0.995}}\\[8pt]&\approx 33.2\%\end{aligned}}}$

to make this discussion more concrete, assume there are 10,000,000 people in the computation.

0.5% of people are users of the drug means there are 50,000 people are drug users.

$P({\text{+}}\mid {\text{User}})$ represents that the test will produce 99% true positive results for the 50,000 drug users, that is, 4,950 people

in this context, what does "$P(\text{+})$" mean?

$P(+)$ represent the probability that the test will produce positive result.

This is computed using law of total probability in the denominator.


Adaboost with other classifier fitting There is the opportunity to fit decision trees with other decision trees. For example:
adaclassification= AdaBoostClassifier(RandomForestClassifier(n_jobs=-1))
adaclassification.fit(X_train,y_train)

I got better results with random forest, so improved the result from adaboost with the random forest classifier. However I dont understand what´s happening here? It sounds easy: adaboost uses a random forest to fit it´s classification. But what´s mathematically going on here?
Adaboost is made of the residuals as a sequence (boosting). Random forest (bagging) built a forest out of trees.

Your description is apt. There isn't anything especially "mathematical" happening here, aside from the AdaBoost algorithm itself.
In psuedocode, something like this is happening:
For n in 1 .. N_Estimators do
  Train classifier Tn on data X with weights W
  Compute weighted residuals E from Tn
  Update W based on E
  Renormalize W
end

In your case, Tn would be a Random Forest model, which is itself an ensemble based on bagging. So at each iteration of the "outer" AdaBoost model, an entire Random Forest model is being trained, i.e. several decision trees are fitted on random sub-samples of data points and features.
Of course, this is an unusual setup for a boosting model. But there's no conceptual or computational reason why you couldn't run the algorithm this way.
If you are curious about how exactly the weights are computed and updated, Scikit-learn uses the SAMME algorithm, which is based on but not exactly identical to the original AdaBoost. SAMME is described in "Multi-Class AdaBoost" by Zhu, Rhosset, Zhou, & Hastie (2006).


Perceptron - Which step function to choose I'm studying Perceptron algorithm. Some books use this step function

  1 if x>=0 else -1

where x is a dot product between the weights w and a sample x.

Other books use:

  1 if x>=0 else 0

What are the practical differences between these step functions?

They have the same meaning in this context although during training using Rosenblatt update rule, the former may have great changes during each update. Perceptron is used for binary classification which means there are two possible classes to classify. If the result of inner product, here dot product, is greater than or equal to zero, the class of inputs will be the first class and if it's smaller than zero the class of inputs would be the other class. Perceptron just has one neuron. It's a simple linear classifier. The value of threshold is just important. Means that if the product is greater than zero, the input belongs to e.g. positive class and if is negative it belongs to negative class. The step functions and Rosenblatt update rule are not used any more. They have so much oscillation. Today networks learn using gradient descending algorithms which uses the concept of derivative.

When you progress, you will see that neural nets which use other activation functions like Sigmoid or Tanh are different. the former has 0.5 expected value and the latter has 0 expected value which causes the second learn so much faster. Although now a days ReLU is more popular among other activation functions.


What are the tools to speed up the running time of machine learning algorithms? I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I run on my machine on RStudio. I am thinking another tool to reduce this expensive time-cost.

I am talking mainly about the distributed computing tools like spark regardless of the type of algorithm and the size of my data.

Any help is appreciated !

It all depends on the model you are running and your chosen sample sizing.

The sample sizing is obvious, so I won't go into that (or at least it should be obvious to you vis-a-vis how R handles data). But the chosen model can have an effect. For example, neural networks are one of the rare problems in computing that you actually can throw more hardware at to improve. That's going to be a while different experience than, say, gradient boosting or something along those lines.


Fitting a pandas dataframe to a Poisson Distribution I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using:

import numpy as np
from scipy.optimize import curve_fit
data=df2.values
bins=df2.index
def poisson(k, lamb):
    return (lamb^k/ np.math.factorial(k)) * np.exp(-lamb)

params, cov =  curve_fit(poisson, np.array(bins.tolist()), data.flatten())

I get the following error:

  TypeError: only size-1 arrays can be converted to Python scalars

I think the cause of the error is the np.math.factorial(k) function call, since curve_fit passes a numpy array as the first parameter to the poisson function, and if you try to run the code

np.math.factorial(np.array([1, 2, 3]))

You'll get the error

TypeError: only size-1 arrays can be converted to Python scalars

Try using scipy.special.factorial since it accepts a numpy array as input instead of only accepting scalers.

Thus, just change your poisson function to

def poisson(k, lamb):
    return (lamb**k/ scipy.special.factorial(k)) * np.exp(-lamb)

Hope this helps

EDIT:
Also I fixed the ^ to ** since that's how you use the exponential operator in python.


Clustering unique visitors by useragent, ip, session_id Given website access data in the form session_id, ip, user_agent, and optionally timestamp, following the conditions below, how would you best cluster the sessions into unique visitors?

session_id: is an id given to every new visitor. It does not expire, however if the user doesn't accept cookies/clears cookies/changes browser/changes device, he will not be recognised anymore

IP can be shared between different users (Imagine a free wi-fi cafe, or your ISP reassigning IPs), and they will often have at least 2, home and work.

User_agent is the browser+OS version, allowing to distinguish between devices. For example a user is likely to use both phone and laptop, but is unlikely to use windows+apple laptops. It is unlikely that the same session id has multiple useragents.

Data might look as the fiddle here:
http://sqlfiddle.com/#!2/c4de40/1

Of course, we are talking about assumptions, but it's about getting as close to reality as possible. For example, if we encounter the same ip and useragent in a limited time frame with a different session_id, it would be a fair assumption that it's the same user, with some edge case exceptions.

Edit: Language in which the problem is solved is irellevant, it's mostly about logic and not implementation. Pseudocode is fine.

Edit: due to the slow nature of the fiddle, you can alternatively read/run the mysql:

select session_id, floor(rand()*256*256*256*256) as ip_num , floor(rand()*1000) as user_agent_id
from
    (select 1+a.nr+10*b.nr as session_id, ceil(rand()*3) as nr
    from
        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)a
    join
        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)b
        order by 1
    )d
inner join
    (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
    union all select 6 union all select 7 union all select 8 union all select 9 )e
    on d.nr>=e.nr

One possibility here (and this is really an extension of what Sean Owen posted) is to define a "stable user."

For the given info you have you can imagine making a user_id that is a hash of ip and some user agent info (pseudo code):

uid = MD5Hash(ip + UA.device + UA.model)

Then you flag these ids with "stable" or "unstable" based on usage heuristics you observe for your users.  This can be a threshold of # of visits in a given time window, length of time their cookies persist, some end action on your site (I realize this wasn't stated in your original log), etc...

The idea here is to separate the users that don't drop cookies from those that do.

From here you can attribute session_ids to stable uids from your logs.  You will then have "left over" session_ids for unstable users that you are relatively unsure about.  You may be over or under counting sessions, attributing behavior to multiple people when there is only one, etc...  But this is at least limited to the users you are now "less certain" about.

You then perform analytics on your stable group and project that to the unstable group.  Take a user count for example, you know the total # of sessions, but you are unsure of how many users generated those sessions.  You can find the # sessions / unique stable user and use this to project the "estimated" number of unique users in the unstable group since you know the number of sessions attributed to that group.

projected_num_unstable_users = num_sess_unstable / num_sess_per_stable_uid

This doesn't help with per user level investigation on unstable users but you can at least get some mileage out of a cohort of stable users that persist for some time.  You can, by various methods, project behavior and counts into the unstable group.  The above is a simple example of something you might want to know.  The general idea is again to define a set of users you are confident persist, measure what you want to measure, and use certain ground truths (num searches, visits, clicks, etc...) to project into the unknown user space and estimate counts for them.

This is a longstanding problem in unique user counting, logging, etc... for services that don't require log in.


This seems intuitive.
When we have a single-label response (Multi-Class setting), our response prior to encoding looks like:
[1, 4, 2, 3, 5]

After encoding,it becomes
[[1,0,0,0,0], [0,0,0,1,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,0,1]]

Indicating the presence of ith Class. But only one.
Similarly, in a multi-label setting-
Input should be - [[1,2,5], [4,1], [2,3] [3], [5,4]]
Encoded value   - [[1,1,0,0,1], [1,0,0,1,0], [0,1,1,0,0], [0,0,1,0,0], [0,0,0,1,1]]
Indicating the presence of (i,j,k)th Classes. Can be many
There is no need to keep duplicate values in the data before encoding.
The inverse function will work in the same way.

The set of labels for each sample such that y[i] consists of classes_[j] for each yt[i, j] == 1.

But if you try to transform again the o/p of inverse, you will get the same encoding.
mlb.transform(zet)

array([
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0, 1, 0, 0, 0],
   [1, 1, 0, 1, 0, 1, 0, 0, 0, 0],
[1, 1, 0, 0, 1, 0, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0, 0, 0, 1, 0],
[1, 1, 0, 0, 0, 0, 1, 0, 0, 0],
 [1, 0, 1, 0, 0, 0, 0, 1, 0, 0],
 [1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 1, 0, 1, 0, 0, 0]])


Better code completion in python notebooks How come python code suggestion is awful in notebooks and spyder? Smetimes it shows classes methods and variables, sometimes not

If you are working locally (running e.g. starting up a Jupyter notebook from a terminal on localhost) code completiong should work just fine - I have never had any problems.

If you are running a Jupyter notebook attached to a Python process in a Docker container, there can be issues with code completion (the two-way communication may need to be activated).

You can try running the following ipython "magic command" that makes the interpreter greedy - this has worked for me in the past:

%config IPCompleter.greedy = True

Simply run that in the first Jupyter cell.

Here is a list of all built-in magic commands.

I am afraid I don't know of any differences in these cases between jupyter notebooks and Spyder.


What Are Stats Metrics To Analyze How Well One Column Predicts Another Column I have the following table with three columns:

Type    Category    Actual
A       1           47
A       1           90
A       1           74
A       2           64
A       2           25
A       3           55
A       3           55
A       3           19
B       6           80
B       6           28
B       2           60
B       2           49
B       2           7

Next, I'm given a static table that provides predictions by Type and Category.  I will be using the "Prediction" column values as the actual predictions.

Type    Category    Prediction
A       1           83
A       2           48
B       6           54
B       2           57

So once I combine the two tables by Type and Category, I have the following table:

Type    Category    Actual  Prediction
A       1           47      83
A       1           90      83
A       1           74      83
A       2           64      48
A       2           25      48
B       6           80      54
B       6           28      54
B       2           60      57
B       2           49      57
B       2           7       57

I want to evaluate my data by distinct Type and Category combinations.  So my output table could be something like this:

Type    Category    Avg_Actual  Avg_Prediction
A       1           value       value
A       2           value       value
B       6           value       value
B       2           value       value

My question is: What are different metrics I can use to evaluate how predictive the "Prediction" column is against the "Actual" values given my data set.

Please be specific with your metrics and the more you can provide (with details on how to derive them) the better.  So far, the only one I came up with was calculating RMSE.

I would like all the metrics at a distinct Type-Category level.  Additionally, is there a way to incorporate confidence intervals into this analysis?

Any help is greatly appreciated.

Other possible evaluation measures are:

Mean Absolute Error (MAE): $\frac{\sum_{i=1}^n{\| y_i - x_i\|}}{n}$
R^2: also known as coefficient of determination, the formula is rather lengthy, but a detailed definition, as well as a motivation, can be found here
Root Mean Squared Logarithmic Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(x_i + 1))^2}}$
Explained Variance Score: $1 - \frac{Var(y - x)}{Var \ y}$ more here and here.
Mean absolute percentage error (MAPE): see the wikipedia for the formula and major drawbacks of this measure.
Median Absolute Error (MAE2): $Median(| y_1 - x_1 | \dots | y_n - x_n |)$

These measures, as well as others, can be found here together with some Python code.


Is this a bug in Seaborn? I am using the seaborn violin plot feature however the size of the violins aren't what I would expect them to be. Does anyone know whether this is something I am doing wrong, or as I suspect whether this is a bug in seaborn.

The code I am using is the following;

 sns.violinplot(x = "Issue Type", y = "issue_adj_risk_score", data=subset, scale="count", cut =0)

and the chart I can see is like this

As I understand it, becuase I have used the scale ="count" parameter, the sizes of each of these violins should be proportional to an overall count of the number of things in each area. In 'category 1' however (pink) I have a total of 936 with an issue_adj_risk_score of 1 whilst in category 2 (orange) I have a total of 1212 with an issue_adj_risk_score of 1.

I am confused however as the pink is larger than the orange! :-(

Does anyone know why this might be?

EDIT

I am now seeing further problems where the range of a particular category is inconsistent. I have a dataset and used violinplot to plot the below

Here we can see that a certain category (which I have highlighted in red) has a range from 1-4. This surprised me so I looked into the underlying data and it doesn't contain any values at 4. I then took a subset that contained this category only and plotted a violinplot with the same data and got this

Which is what I would have expected to see in the initial chart!

To investigate this further I plotted a boxplot based on the original data once again, and got the following

Which is consistent with the first chart but doesn't reflect the actual data. How can all of these 3 charts come from the same data!!??

From the seaborn documentation:

  scale : {“area”, “count”, “width”}, optional

  The method used to scale the width of each violin. If area, each violin will have the same area. If count, the width of the violins will be scaled by the number of observations in that bin. If width, each violin will have the same width.

So the violins are scaled such that the width is proportional to the number of observations. You were expecting the area to be scaled with count, not the width, and that seems natural to me but that option does not seem to exist in seaborn. Here is an issue regarding this, with a handmade solution:

https://github.com/mwaskom/seaborn/issues/962


Should scaling be done for mixed data (categorical and numerical)? My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations

NumFeature1,NumFeature2....NumFeature10,CatFeature1,CatFeature2,CatFeature3, Target

All Categorical are non-ordinal and each have the following categories:

CatFeature1: 0/1

CatFeature2: 0/1/2

CatFeature3: 0/1/2/3

It is a binary classification problem where we have to predict the probability for each class of the target class.

I have 3 Questions for above dataset:

Q1- For the categorical Feature, Should I use LabelEncoder() or OneHotEncoder() or df.get_dummies() or should I just combine custom label encoder with one-hot encoder?

Q2- Should scaling be done for Numerical features only or it should be done for all the features including categorical after encoding

Q3- What should be the best model to get the probability of the binary classification. So far, I have tried kNN, LogisticRegression, and RandomForestClassifier with predict_proba, but log_loss score was 0.301 the best.

Ans 1: get_dummies() or (label encoder + one-hot encoder) would do the trick.

Ans 2: Scaling categorical dummy data does not make sense. It also loses out on interpretability.

Ans 3: Logistic regression might tend to overfit since you only have 180 observations. KNN might perform well with a small number of observations but it doesn't handle categorical variables well. Random Forest or Extreme Random Forest might be your best bet. However, there's no "one fits all" concept in ML. You'll have to try a variety of algorithms and see which fits your dataset best. Though there are Linear SVM and other algorithms, I'd suggest that you try simpler ones as otherwise it might be an overkill.

Good luck!


Torch tensor view and stride mechanics I want my Mx2xN float tensor to become MxN complex tensor.
In this minimal example I supply the the 10x2 matrix which should become the vector [0+10j,1+11j,2+12k,...]
x = torch.stack([torch.arange(10.),torch.arange(10.,20)])
torch.view_as_complex(x.T)

gives the error RuntimeError: Tensor must have a last dimension with stride 1 because x.T.stride() is (1,10)
Trying
>>> x.T.as_strided(size=(2,10),stride=(10,1)).T.stride()
(1, 10)

Didn't get me any further, how do efficiently convert these?

Found this as a hack, is this an efficient solution?
torch.complex(*x)
it unpacks the first dimension and feeds it into the real and complex dimension


Intuition behind the number of output neurons for a neural network I am reading Michael Nielsen's book on deep learning. In the first chapter, he gives the classic example of classifying 10 handwritten digits, and uses it to explain the intuition behind choosing the number of output neurons.

Initially, before reading the book, I thought it was intuitive that you would want 10 output neurons, each representing a given digit, but I'm starting to question why I thought so.

Anyways, here's what Nielsen explains:

  You might wonder why we use 10 output neurons. After all, the goal of the network is to tell us which digit (0,1,2,…,9) corresponds to the input image. A seemingly natural way of doing that is to use just 4 output neurons, treating each neuron as taking on a binary value, depending on whether the neuron's output is closer to 0 or to 1. Four neurons are enough to encode the answer, since 2^4=16 is more than the 10 possible values for the input digit. Why should our network use 10 neurons instead? Isn't that inefficient? The ultimate justification is empirical: we can try out both network designs, and it turns out that, for this particular problem, the network with 10 output neurons learns to recognize digits better than the network with 4 output neurons. But that leaves us wondering why using 10 output neurons works better. Is there some heuristic that would tell us in advance that we should use the 10-output encoding instead of the 4 -output encoding?
  To understand why we do this, it helps to think about what the neural network is doing from first principles. Consider first the case where we use 10 output neurons. Let's concentrate on the first output neuron, the one that's trying to decide whether or not the digit is a 0. It does this by weighing up evidence from the hidden layer of neurons. What are those hidden neurons doing? Well, just suppose for the sake of argument that the first neuron in the hidden layer detects whether or not an image like the following is present:

  It can do this by heavily weighting input pixels which overlap with
  the image, and only lightly weighting the other inputs. In a similar
  way, let's suppose for the sake of argument that the second, third,
  and fourth neurons in the hidden layer detect whether or not the
  following images are present:

  As you may have guessed, these four images together make up the 0
  image that we saw in the line of digits shown earlier:

  So if all four of these hidden neurons are firing then we can conclude
  that the digit is a 0. Of course, that's not the only sort of evidence
  we can use to conclude that the image was a 0 - we could legitimately
  get a 0 in many other ways (say, through translations of the above
  images, or slight distortions). But it seems safe to say that at least
  in this case we'd conclude that the input was a 0.

  Supposing the neural network functions in this way, we can give a
  plausible explanation for why it's better to have 10 outputs from the
  network, rather than 4. If we had 4 outputs, then the first output neuron would be trying to decide what the most significant bit of the digit was. And there's no easy way to relate that most significant bit to simple shapes like those shown above. It's hard to imagine that there's any good historical reason the component shapes of the digit will be closely related to (say) the most significant bit in the output.

  Now, with all that said, this is all just a heuristic. Nothing says
  that the three-layer neural network has to operate in the way I
  described, with the hidden neurons detecting simple component shapes.
  Maybe a clever learning algorithm will find some assignment of weights
  that lets us use only 4 output neurons. But as a heuristic the way of
  thinking I've described works pretty well, and can save you a lot of
  time in designing good neural network architectures.

I don't understand what he means in the second to last paragraph. I included the rest for context, but it's this second to last paragraph that's confusing me. Could someone clarify what he's talking about?

The reason 10 neurons work better than 4 is because it allows the network to encode all possible answers independently.

You could have 4 neurons and train the first one to encode the least significant bit, the second neuron the second-least significant bit, and so on. It would work.

Images of a 1 (bits: 0001) are quite like images of a 7 (bits 0111). Imagine an image where we know it’s a 1 or a 7, but we have no idea which one. With 4 outputs (as described) you would output (0, 0.5, 0.5, 1), meaning the correct answer is either 1, 3, 5, or 7. With 10 outputs, all classes would have 0 probability and 1 and 7 would have 50% each, which conveys more accurately what can be interpreted from the image.

You could have a single output that just outputs the value of the digit directly. But a 1 and 7 are now far apart in the output node (the whole range is from 0 to 9), so it’s impossible for this network to say “1 or 7”. It will probably say: 4 (midpoint between 1 and 7). That’s why it doesn’t work well.

Very poorly explained by “Nielsen’s book”, IMHO.


SGDClassifier - Why do I need to use argmax instead of argmin to find the lowest threshold satisfying given precision? I am an experienced programmer, but new to Python and data science. I am following Aurelien Gerone's book and I don't understand one thing.
I create SGDClassifier and calculate its precision_recall_curve(). Then I am trying to find the lowest threshold to satisfy precision equal to 90%:
precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)
threshold_90_precision = thresholds[np.argmax(precisions >= 0.90)]

Why on earth I am searching for argmax if I need to find the minimum threshold value? If I try to use argmin I get the wrong value, with precision equal to 0.1.
As I understand this:

precisions >= 0.90 creates an array with precision scores only above or equal to 0.90,
argmax returns an index, at which I find the highest value in the given array (so this should be as far from 90% as possible, but it's not!),
then I choose a threshold with returned index.

What am I missing?

Okay, I solved this myself.
precisions >= 0.90 doesn't create an array with precision scores only above 90%, but it transforms this array to the array of Booleans, where precisions below 90% are turned to False and the others are True.
argmax, if there are multiple identical, maximum values (and True is max here) returns the first index of this occurrence.
I sometimes hate this book, why he just doesn't use method like "array.first_equals(True)"?


How many examples needed for named entity disambiguation? If I want to build a named entity linking system for resumes using an ontology of occupations and skills about how many annotations would I need? The ontology has about 20,000 entities.

As a lower bound I'm guessing I would need about 10 examples per entity and maybe 3 different annotators to label each mention so ~600K annotations. Does that make sense?

It depends on the signal-to-noise in the dataset. The amount of data to perform named entity disambiguation will depend on the tf-idf score of the occupation and skills, rare occupations and skills will require less data to build a performant model.

For example, that the sentence "I am a cook that multitasks well." "Cook" is an occupation and "multitask" is a relevant skill. In a similar sentence, "I multitasked while I cooked." "Cook" is no longer an occupation and "multitask" is no longer a relevant skill. However, the phrase "saturation diver" is less frequent than "cook", thus much easier to build a model to identify as an occupation and find relevant skills.

Annotator performance is easier to measure. Cohen's kappa is a common method of judging inter-rater reliability. Again, the number of needed raters depends on their agreement on the task. If task performance is easy, the number of raters and the number of items per rater can be lower. It is best to benchmark your system and then decide how much data you need to raise the benchmark scores.

One way to automatically create ontologies from a text is the TextRank algorithm.


It seem that the textbook "Machine Learning - A Probabilistic Perspective" uses input and output in a opposite way, is it? Chapter 1 of "Machine Learning - A Probabilistic Perspective" by Kevin Patrick Murphy says

  We now consider unsupervised learning, where we are just given output data, without any inputs. The goal is to discover “interesting structure” in the data; this is sometimes called knowledge discovery.

post this post

  Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses.

I saw this kind of explanation more times than the one in that book.

It seems that the book uses the terms in a opposite way, where the dataset is viewed as the output of some input, is my understanding right?

No, it is not the case. I’m almost sure that it’s a typo and it should be changed to:

“We now consider unsupervised learning, where we are just given input data, without any outputs.”

It can be deduced by looking at the definition of supervised learning from the book:

“In this section, we discuss classification. Here the goal is to learn a mapping from inputs x to outputs y, where y ∈ {1,...,C}, with C being the number of classes.”


How to represent a user who hasn't churned in training data I am building a file with sample data that has a bunch of variables:

date, customer_id, amount_spent, number_of_transactions, time_since_last_transaction etc. that i am mapping against days_to_churn

I will train my model using Keras to map the emboldened variables to the italicised days_to_churn. However there are many instances where a user is an active subscriber and this value is blank.

How would I go about incorporating this to ensure that I am not excluding active people from my churn calculations?

In your setup, the only way is probably to set it to some large number, say 365. However, this will force you to discard all dates that are less than 365 days old, because you can't be certain a customer won't churn before he reaches 365 days (which is still in the future).

A better and more common way is to look at churn on a rolling window basis, e.g. will the customer churn within the next 30 days. This turns your problem into a classification problem. That way you only need to discard the most recent 30 days of data.


How does k fold cross validation work? You split the data in k subsamples. Train it on k-1 subsamples, test it on kth subsample, record the performance with some error merric.

Do it k times for each of the k subsamples, record the error each time.
Then choose the model with the lowest error?
Is it the same as ensemble technique?

Imagine you have 1500 labeled data points, and you want to estimate how well some classifier will work on new data. One (naive) method would be to train a model with all 1500 of your data points, and then check how many of the 1500 data points were classified correctly. This is not likely to give a good estimate of the performance on new data, because new data was not used to test the model. Some models like decision trees and neural networks will often be able to get 100% accuracy on the training data, but perform much worse on new data.

So you think to yourself that you will split the data into two sets - a training set which you will build a model with, and a testing set that you will use to evaluate the model. Lets say you decided to train the model with 1000 of your examples, and evaluate with 500. This should give a reasonable estimate of how well your model will perform on new data, but it seems a bit limited; after all, one third of your data has not been used for training at all! We only have predictions for the 500 test samples - if these ones randomly happened to be easier to classify correctly on average, then our performance estimate is overly optimistic.

Cross validation is a way to address this. Lets set $k=3$, so the data is split into three sets of 500 points (A, B and C). Use A & B to train a model, and get predictions for C with this model. Use B & C to train a model, to get predictions for A. Finally, use A & C to train a model, and get predictions for B. Now we have a prediction for every point in our labeled data that came from a model trained on different data. By averaging the performance of each of these models, we can end up with a better estimate of how well the model will perform on new data.

Note that you should then re-train your model using all 1500 labeled points if you want to apply it to new data. Cross validation is only for estimating the performance of this new model. Also if your data is large enough, cross validation is probably unnecessary and you could just make a single train/test or train/valid/test split.


Custom loss function Is it possible to apply a custom loss function in a regression model (or any other algorithm for predicting continuous variable) ? I'm working on a stock market prediction model and I need to maximize the following loss function: if [predicted] < [actual]  then [predicted] else [-actual]. Would that be possible ?
Thanks

The concept of a loss function comes from decision theory. Although there are some 'classic' loss functions the point is to be subjective, in the sense of being flexible enough to represent any particular problem conctext. So in that sense, yes, loss functions can be customised.
One of the main ways this has been achieved is via Bayesian regression, as the output of Bayesian regression is a probability distribution which can then the input for a loss function. This blog post shows the mechanics in a simple way (texts for Bayesian statistics and decision are often non-accessible), and also gives some examples of somewhat 'custom' loss functions in 'Examples: Part 3'. -> http://www.statsathome.com/2017/10/12/bayesian-decision-theory-made-ridiculously-simple/


sklearn.metrics.average_precision_score getting different answers for same data but different formats I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet.

Consider the following two snippet:

import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 1, 0])
y_scores = np.array([0.4, 0.4, 0.8])
average_precision_score(y_true, y_scores) # 0.3333333333333333

and

y_true = np.array([[1,0], [0,1],[1,0]])
y_scores = np.array([[0.6,0.4], [0.6,0.4], [0.2,0.8]])
average_precision_score(y_true, y_scores) # 0.45833333333333326

From what I understand, these are the same data but formatted in different ways. The first is only showing the true labels and predicted scores for the positive class, while the second is giving information on both classes.

But why are they giving different answers? In particular, how are these two results calculated? And which one is correct? I was reading this post, but I didn't understand how that precision-recall table in the answer is constructed. Can anyone can go through a similar calculation for my example?

By explicitly giving both classes, sklearn computes the average precision for each class.  Then we need to look at the average parameter: the default is macro:

  Calculate metrics for each label, and find their unweighted mean...

If you switch the parameter to None, you get

average_precision_score(y_true, y_scores, average=None)
# array([0.58333333, 0.33333333])

whose second entry agrees with the answer for the positive class, and whose average gives you the other output you're seeing.

As for hand calculations:
For the positive class, with a threshold in $(0.8,1]$ we get zero positive predictions, and so 0 recall and 1 precision (by convention).  With the threshold in $(0.4,0.8)$ we get one false positive, one false negative, and one true negative, so 0 recall and 0 precision.  With the threshold in $[0, 0.4)$ we get one true positive and two false positives, so 1 recall and 0.333 precision.  So the table from the linked question in this case is

    R     P
1   0.0   1.0
2   0.0   0.0
3   1.0   0.333

Finally, the average precision computation is
$$(0.0-0.0)\cdot 0.0 + (1.0-0.0)\cdot 0.333 = 0.333$$
Somewhat degenerate example, but it checks out.

For the other class:
$(0.6,1]$ gives zero "positive" predictions, so 0 and 1 again.
$(0.2,0.4)$ gives one true positive, one false positive, and one true negative.
$[0,0.2)$ gives two true positives, one false positive.  So

    R     P
1   0.0   1.0
2   0.5   0.5
3   1.0   0.666

and average precision is
$$(0.5-0.0)\cdot 0.5 + (1.0-0.5)\cdot0.666 = 0.58333$$


No result for scored labels in Azure ML Web Service I am trying to predict scored labels using regression. But when I am about to get the result from Azure ML Web Service in Excel 2016, there is no result appeared in the scored label column. How should I fix this? Below is all my process...

Here is my problem I always get. As you are seeing now, there is no result in scored label column when I try to predict.

Test out the web service in the web client under configure to see if it works ...
There might be something wrong with the client.


HMM and its competitive alternatives In Natural language processing, what are the major applications of Hidden Markov Chain (HMM), and what are the alternatives that usually can outperform HMM, is RNN and LSTM always the choice right now?

Transformer based architectures are some of the most popular in NLP right now. You can check this blog post for more information:
https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html
Other than performance, one major advantage of transformers is that operations can be parallelized, making it much faster than RNNs/LSTMs.


CNN Design for Counting on Simple Images This is the first CNN I'm designing following college examples and assignments. I'm working on a CNN that I'd like to use to classify images by the number of shapes on them. My basic problem is that I can't seem, to get the CNN to respond (accuracy and val_accuracy are flat) after n EPOCHS (I have varied n these along with Steps and Batch Size) The images are 98 x 150 pixels and look like this:

This is 10 data squares on a by background. The image set has been built by varying the position of the shapes on the image, changing the background color (4 colors), changing the shape color (3 colors), changing the shape type (square, circle, hexagon), flipping the images (vert/horiz). The images are stored in folders (1-26) with 1 having 1 shape per image, 26 having 26 shapes per image. There are 3300 images per folder totaling ~86k images (I have experimented with a lower number of shapes). Here's an images of the training/validation file counts:

The problem I'm seeing is the model never seems to "initialize" in other words the accuracy and validation accuracy do not change much while training the model:

The accuracy and val_accuracy are very different to any models I've worked with - normally I see a fast ramp rate which tapers off as the model trains. In this case both are flat and never increase. Even after larger EPOCHs.
Here's one of the models I've tried:
# Define the Model
model = models.Sequential([
  layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.flatten(),
  layers.Dense(512, activation='relu'),
  layers.Dense(num_classes)
])

I've tried several variants of this model along with an AlexNET all yield the same results.
I'm thinking that maybe the images are too simple/varied for the model to train? Maybe putting the shapes on a pixelated background would help make them more complex/varied? I'm also wondering whether initializing the model with random weights/biasses might help? I'm using Tensorflow 2.x and don't know whether this is done automatically.
Does anyone have any thoughts/advice on how I may improve the model and at least see accuracy/validation increase rather than it being flat? I could deal with overfitting later (if it occurs).
I've be very grateful for any advice - thanks in advance!

This doesn't seem like a Deep Learning problem, this could be solved by classic Computer Vision techniques, so maybe you could inspire your architecture by those.

Check is your model convolutional layers receptive field is large enough for the shapes you trying to count.

You probably won't be able to do counting without some kind of post processing

Check Yolo strategies, you probably will be able to do a counter with it.

This is a object detection problem, you will often have poor performance trying to use a architecture designed for image classification such as AlexNet while trying to face this as a regression problem.

Your output size should probably be a varying length vector from which the length is the desired count.


Purpose of visualizing high dimensional data? There are many techniques for visualizing high dimension datasets, such as T-SNE, isomap, PCA, supervised PCA, etc. And we go through the motions of projecting the data down to a 2D or 3D space, so we have a "pretty pictures". Some of these embedding (manifold learning) methods are described here.

But is this "pretty picture" actually meaningful? What possible insights can someone grab by trying to visualize this embedded space?

I ask because the projection down to this embedded space is usually meaningless. For example, if you project your data down to principal components generated by PCA, those principal components (eiganvectors) don't correspond to features in the dataset; they're their own feature space.

Similarly, t-SNE projects your data down to a space, where items are near each other if they minimize some KL divergence. This isn't the original feature space anymore. (Correct me if I'm wrong, but I don't even think there is a large effort by the ML community to use t-SNE to aid classification; that's a different problem than data visualization though.)

I'm just very largely confused why people make such a big deal about some of these visualizations.

First of all your explanation about the methods are right. The point is that Embedding algorithms are not to only visualize but basically reducing the dimentionality to cope with two main problems in Statistical Data Analysis, namely Curse of Dimentionaliy and Low-Sample Size Problem so that they are not supposed to depict physically understood features and they are not only meaningful but also necessary for data analysis!

Actually the visualization is almost the last usage of embedding methods. Projecting high-dimensional data into a lower-dimension space helps to preserve the actual pair-wise distances (mainly Euclidean one) which get distorted in the high dimensions or capturing the most information embedded in the variance of different features.


What does AR(max=1) mean? http://cocodataset.org/#detection-eval
"ARmax=1: AR given 1 detection per image"
"AR is the maximum recall given a fixed number of detections per image, averaged over categories and IoUs."

Above link, they described ARmax=1, but I didn't understand exactly. I understood general recall concept in machine learning or other areas. Could you explain ARmax=1 more detail ??

AR is the abbreviation for Average Recall.
max=1 only looks for 1 detection per image.
Thus, ARmax=1 is the average ability of the model to find a given category in an image. For example if there is a dog in the image, the model should predict a dog in the image. That performance is averaged across all categories.


Why neural networks do not perform well on structured data? I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / hidden layers with an objective to beat the decision tree classifier accuracy on the test set. But the best accuracy I could achieve with neural networks was that of 0.42 and decision tree was at 0.50.

I had asked a question here, as what could be the case and someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image). In the comment linked to the same answer, it was pointed out that :

  Well you can take a look at kaggle competition winners. In competitions containing structured data by far the most popular algorithm is xgboost (along with other similar algorithms lightgbm, catboost, etc.). On the other hand Neural Networks are rarely used in these competitions because they are not so strong with these types of data. This is also evident by the near 20-year disappearance of neural networks, until deep learning made them relevant again. During these years trees and SVMs on top.

That could generally be true but I do not have the intuition/reason as to why neural networks do not work well with the structured data? Could someone help me reason that? It will also be great if you could point me to some paper/post that explains this.

One feeling that I have is, it could be because of the less volume of data. Neural networks might not generalize well with fewer data points as compared to other classifiers like decision trees, svms etc. But then I am not really sure about this.

... someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image).

It's difficult to propose a universal analogy, but perhaps one moderately complex example which is easy to understand will suffice.

In the link you provided user JkBk mentions the "no free lunch theorem", let's add to that the "infinite monkey theorem". The fact is you can have one or more free lunches, just not as many as you like. Similarly you can not have infinite monkeys, but you can have a lot of them. Limits on each.

Take for example this data from the World Health Organization, let's only look at age versus height for boys aged 5 to 19. Let's look at a subset of the data in tabular form, I presume you understand the concept of "height-for-age".

Process this chart with your brain, what does it tell you. It tells you exactly everything averaged for a subset of the data. Can you draw any inferences from it, how about another several tables of data?

A neural network would have to process the data in all the tables and come up with a means to say "this age = this height" and "this height = this age" as an average.

Here is one of several tables:

Now let's look at the whole dataset in a chart:

See how at age 13 there is the start of a growth spurt, that continues almost to the age 15 when it starts to slow down. Between 15 and 17 it starts to level off, and from 17 to 19 there's almost no growth at all. From 5 to 11 a simple algorithm can easily solve the equation in either direction, beyond 11 the algorithm is not so simply but it still can be solved with a polynomial.

For a neural network once it gets on track and thinks it is coming up with a solution a monkey wrench gets thrown in to the works. Look at the "Alice and Bob example" (from 'no free lunch'), trying to solve the problem rationally requires knowledge and intelligence to avoid getting trapped.

Alternatively an Equation Solver can intelligently brute-force a solution, providing a relatively simple equation that can be used efficiently on large volumes of data.

Once you have a simple equation you could apply it to a huge dataset, like the Census, to check for errors such as age not matching height. A neural network would be better for discovering that age versus height was different than the average in a particular area and supply the reason (distance to inexpensive high quality food, versus earnings, and a sprinkle of education) whereas a simple polynomial wouldn't have the additional knowledge buried within it to make deductions.

A neural network doesn't comprehend (literally or figuratively) why a solution that was working later fails; it needs to continue, to learn to ignore it's success, to reach a working solution.

A non neural network solution doesn't care (literally or figuratively) whether it was on track, figured something out and learned to apply it, it only knows to minimize error and eliminate variables.

That doesn't mean that each couldn't switch roles and learn, or be programmed, to do the job of the other; but then the neural network would be pedantic about precision throughout (slow) and the solver would over analyze (calculate left or right with high precision) rather than simply weigh the outcomes with adequate precision for each step.


How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning? I try to create a machine learning model, linear regression, to predict a price of a diamonds.

All examples that I found online do not have a step with scaling of data, using MinMaxScaler or StandardScaler. But I personally think that this is an important step in ML. The Spark guide shows this feature, but separately from OneHotEncoding. Moreover, because the result of OneHotEncoding in Scala is different from Python (for instance, it looks like this: (4,[3],[1.0]) in Spark, while in Python it is a combination of 1 and 0), I am confused where to apply StandardScaler - after Indexing and before OneHotEncoder, or after OneHotEncoder or in some other step?

My code after I cleaned data a little bit. This code works, but predictions are far from being good. I use Scala dataframe and pipeline:

import org.apache.spark.ml.feature.{VectorAssembler,StringIndexer,VectorIndexer,OneHotEncoder}
import org.apache.spark.ml.linalg.Vectors

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

val cutEncoder = new OneHotEncoder().setInputCol("cutIndex").setOutputCol("cutVec")
val colorEncoder = new OneHotEncoder().setInputCol("colorIndex").setOutputCol("colorVec")
val clarityEncoder = new OneHotEncoder().setInputCol("clarityIndex").setOutputCol("clarityVec")

val assembler = (new VectorAssembler()
                  .setInputCols(Array("carat", "cutVec", "colorVec", "clarityVec", "depth", "table", "x", "y", "z"))
                  .setOutputCol("features") )

val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(false)

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,cutEncoder,colorEncoder,clarityEncoder, assembler, scaler, lr))

val model = pipeline.fit(training)

val results = model.transform(test)

The subset of data:

+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|label|carat|      cut|color|clarity|depth|table|   x|   y|   z|
+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|  326| 0.23|    Ideal|    E|    SI2| 61.5| 55.0|3.95|3.98|2.43|
|  326| 0.21|  Premium|    E|    SI1| 59.8| 61.0|3.89|3.84|2.31|
|  327| 0.23|     Good|    E|    VS1| 56.9| 65.0|4.05|4.07|2.31|

Thanks in advance!

Okay, I came up with a solution by myself. Applied OneHotEncoderEstimator that was added recently instead of OneHotEncode on every categorical feature. Then I used MinMaxScaler on features from VectorAssembler.

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

import org.apache.spark.ml.feature.OneHotEncoderEstimator

val encoder = new OneHotEncoderEstimator().setInputCols(Array("cutIndex", "colorIndex", "clarityIndex")).setOutputCols(Array("cutIndexEnc", "colorIndexEnc", "clarityIndexEnc"))

val assembler = (new VectorAssembler()
                    .setInputCols(Array("carat", "cutIndexEnc", "colorIndexEnc", "clarityIndexEnc", "depth", "table", "x", "y", "z"))
                    .setOutputCol("features_assem") )

import org.apache.spark.ml.feature.MinMaxScaler
val scaler = new MinMaxScaler().setInputCol("features_assem").setOutputCol("features")

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,encoder, assembler,scaler, lr))

val model = pipeline.fit(training)

val predictions = model.transform(test)


What is the proper proportion for train and test set for classification system? I have recently googled the best proportion for training and test set for classifying physiological data between normal and abnormal. Much of the source tells that the proportion should be 70:30 or 80:20 for train:test. One of the sources could be visited here.
I am a little unsure because I haven't found the ground scientific truth behind this proportion. Is that any science that explains this proportion?
Thanks

In short theres no best proportion or rule of thumb.
It's highly dependent on how much data you have, its distribution in relation to the number of labels, and if samples are related to each other in anyway or are they completely independent of each other.  You could look into k-fold training. Say you split your data into 5 sections each 20 percent of the size of the entire dataset. For each split of 20 percent of data use the remaining 80 percent of data as the training set. Go through each 20 percent of data and use that as the test and the rest as the training. Make sure your labels are evenly distributed between splits.
K-fold reference


Why BatchNormalization fails in Keras I try to test ResNet approach on cifar10 dataset with the following python code:

# load data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')/256
X_test = X_test.astype('float32')/256
y_train = keras.utils.to_categorical(y_train)
y_test = keras.utils.to_categorical(y_test)

# build a model
def res_unit(x):
    x_shortcut = x

    x = Conv2D(16, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(16, (3, 3), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(32, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Add()([x_shortcut, x])

    x = Activation('relu')(x)

    return x

X_input = Input((X_train.shape[1:]))

X = Conv2D(32, (3, 3), padding='same')(X_input)
X = BatchNormalization()(X)
X = Activation('relu')(X)

X = res_unit(X)
X = res_unit(X)
X = res_unit(X)

X = Flatten()(X)
X = Dense(32)(X)
X = Activation('relu')(X)
X = Dense(10)(X)

X = Activation('softmax')(X)

model = Model(inputs=X_input, outputs=X)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# run model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5)

which outputs the worst result for 10-class classification: 0.1 accuracy on train and validation.

But when I comment all lines with BatchNormalization, everything starts to be OK: I got 0.57 accuracy after first epoch and then it even rises up.

What is the problem with BatchNormalization in my case? Am I using it in a correct way?

I have played around with it and managed to get a working configuration.
Necessary changes:

axis=1 for BatchNormalization. Normally, this should be used when data_format="channels_first", but it worked for me despite TensorFlow backend
Standardize the input:

X_train = X_train * 2.0 - 1.0; X_test = X_test * 2.0 - 1.0
Avoid batchnorm after the last convolution in res_unit.

The working solution looks like this:

# build a model
def res_unit(x, prefix=''):
    x_shortcut = x

    x = Conv2D(16, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv1'))(x)
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv1'), axis=1)(x)
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv1'))(x)

    x = Conv2D(16, (3, 3), padding='same', name='{}/{}'.format(prefix, 'conv2'))(x)
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv2'), axis=1)(x)
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv2'))(x)

    x = Conv2D(32, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv3'))(x)
    x = Add(name='{}/add'.format(prefix))([x_shortcut, x])
    x = Activation('relu', name='{}/relu'.format(prefix))(x)

    return x

def main():
    (X_train, y_train), (X_test, y_test) = load_data()
    X_train = X_train.astype('float32') / 255
    X_test = X_test.astype('float32') / 255
    X_train = X_train * 2.0 - 1.0
    X_test = X_test * 2.0 - 1.0
    y_train = keras_utils.to_categorical(y_train)
    y_test = keras_utils.to_categorical(y_test)

    print(X_train.min())
    print(X_train.max())

    X_input = Input((X_train.shape[1:]))

    X = Conv2D(32, (3, 3), padding='same', name='conv0')(X_input)
    X = BatchNormalization(name='conv0/bn', axis=1)(X)
    X = Activation('relu', name='conv0/relu')(X)

    X = res_unit(X, prefix='block1')
    X = res_unit(X, prefix='block2')
    X = res_unit(X, prefix='block3')

    X = Flatten()(X)
    X = Dense(32)(X)
    X = Activation('relu')(X)
    X = Dense(10)(X)

    X = Activation('softmax', name='softmax')(X)

    model = Model(inputs=X_input, outputs=X)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

I've got 59.07% validation accuracy with it. However, sometimes training does not converge, it feels like it is very sensitive to the random init.
Still don't know why making batchnorm work here is so hard...


How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding? I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:

Data:
    8000 labelled news articles (Sports:Non-sports::15:85)

Parameters:
    embedding size = 128
    vocabulary size = 100000
    No. of LSTM cell in each layer = 128
    No. of hidden layers = 2
    batch size = 16
    epochs = 10000

Result:
    AUC on training set = 0.60
    AUC on testing set = 0.55

As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:

What would be the optimum data size required?
Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.

I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. If you have a vocabulary size of 100,000 then you really need to use some extra feature extraction.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 8000 instances!!! Yes. But, you also have 100,000 features for each instance. This is insufficient even for shallow machine learning models. For a neural network, i usually suggest to follow this very general rule of thumb,

$\#examples = 100 * \#features$.

So you will need a MASSIVE amount of data to properly train your neural network model. Moreover, if you have a skewed dataset then you should expect to be using even more training examples, to show sufficient examples such that the model is capable of distinguishing the two classes.

I would suggest a less intensive model. You should try to use: naive bayes, kernel-SVM or knn for text classification. These methods would do MUCH MUCH better than a neural network. Especially considering you have a skewed dataset!!

My Suggestion

I would start with bag of words and then use kernel-SVM. This should be a good starting point.

A recurrent neural network is 0% recommended for the amount of data you have.


Machine learning model with simultaneous function optimization Consider the following scenario. I am a sculpturer and customers ask me for what price I am willing to provide them with some statues. Their request for sculptures can vary in difficulty, quantity, material, size. In response, I reply with the price and subsequently they will either accept or reject (1 or 0) my proposal. I know of traditional machine learning methods that I can train to predict the probability of acceptance given all inputs.

In this case the price can be influenced by myself. Not let's assume that my expected profit is this function: (quantity * price * (probability_of_acceptance)). I would like to respond to the customers with the price where this expected profit is the highest. Of course you can input different prices in the machine learning model, which will give you some kind of probability acceptance curve that you can subsequently optimize against. But I am looking to do this in one go.

I am wondering:

What is the theoretical name for this type of problem?
What machine learning method is suitable here?
Are there any python packages that implement these models?

Have a look at this competition organized by the European Spatial Agency (ESA).
https://kelvins.esa.int/collision-avoidance-challenge/scoring/

Here they use a composed loss function. A binary classification whether two spatial are at risk of colliding or not. And in case that they collide a regression saying what is estimated risk between them.

This is done by constructing a loss function combining the Mean Squared Error (MSE) and the F_β score with β=2.

You could consider your problem similar. You will want to classify whether your offer gets accepted or not, and in case of acceptance the money in return that you will receive.

In the part of the discussion of the challenge, you will find some notes on how to do it. Also, I believe that they will publish a paper in the next weeks.


Default value of mtry for random forests It is argued that the default value of mtry for random forests is square root of total number of features (for classification) and number of features divided by 3 for regression. Can someone tell me the literature where it is specifically mentioned?

The textbook 'The Elements of Statistical Learning' by authors Trevor Hastie, Robert Tibshirani, and Jerome Friedman discuss how to tune the number of variables to sample as you build the trees in Chapter 15, Section 3. The pdf of the text is freely available and can be found here.

Note that the values you have listed are in fact default values, as you have mentioned. In practice, it is best to tune the value of 'mtry', as the best value will depend on the specific problem and dataset you are working with.


How to best handle imbalanced text classification with Keras? I implemented a text classification model using Keras.

Most of the datasets that I use are imbalanced. Therefore, I would like to use SMOTE to handle said imbalance.

I tried both on plain text, and once the text was vectorized, but I don't seem to be able to apply SMOTE on text data.

I use imblearn and received the following error:

Expected n_neighbors <= n_samples,  but n_samples = 3, n_neighbors = 6

How can I fix this error? And is SMOTE a good idea? If not, what other ways could I deal with class imbalance?

First of all, to reassure you, SMOTE should work on text data. SMOTE will work on any data type as long as there is a way to compute the distance between data points.

Based on the error message you receive, it seems that it's an implementation issue (adding part of your code or how much data you have would greatly help).

As the error states, you have only 3 samples but the method requires at least 6. My guess is that something went wrong and you should have much more than 3 samples.


Machine learning detect changes in components I am a student who will finish my studies next year and I want to analyze the job market in advance. I have found an interesting job where it says:

The department of assemblies and systems of the X deals with the simulation, testing and evaluation of components, assemblies and complete systems mainly from the automotive sector. In the future, machine learning methods will be used to detect changes in components or test benches at an early stage

I find the last sentence very interesting, but I don't quite understand it. If you expect a change to a component (e.g. the component starts to break), you can hard code it to detect this change. Could somebody explain to me with a simple example how machine learning can be helpful in this area?

This is an interesting question about how machine learning can be used to changes in components. In this case, I am going to infer that is maybe talking about detecting defect components, which of course should not be put into a new vehicle.
In this case, you might use images (maybe taken from video of the assembly floor) to the detect potentially defect components. So, hard-coding this would be extremely hard, because:

Firstly, you would need to the code to detect the component and separate this from the background (this task is known as semantic segmentation).
You then have to compare the component with images of a "good" component and a "bad" component (preferably multiple examples of each). This also isn't an easy task to hard-code, because you need might observe the component in different angles (bird's eye view or front on, or example), which are not discrete. Therefore, some level of generalisation is required to do this (hence machine learning)

I look forward to other's interpretations and answers to this question.


What is the risk of removing a feature with low correlation...? I'm running a linear regression model as baseline for a specific estimation problem. Based on the resulting R-squared, regressor coefficients and their respective p-values, I can conclude that specific independent variables can be removed from the model.

What is the induced risk of removing these variables from the feature set?
Can other models—that are better at modelling nonlinear relationships—suffer from this decision?
How can I be sure that I am not loosing valuable nonlinear information without running "nonlinear" regressors?

Variable selection in linear regression is based on partial correlations, not zero order correlations. The partial correlation is the one resulting after a conditioning variable effect is removed from both X and Y.

While a variable may be non highly correlated with Y, it may be highly correlated with an X variable in such a way that the X variable after being partialled out of the conditioning effect becomes highly significant.

This effect is known as suppression among other names.
HTH


How to download images and bounding boxes from imageNet? I am doing object detection for a specific class, say, chairs .

I want to download images of chairs from imageNet. I also want to download the annotation xml files (bounding boxes) from imageNet.

Both these things are provided on imageNet and I have successfully been able to download them using a tool called ImageNet_Utils

https://github.com/tzutalin/ImageNet_Utils

But the downloaded images and bounding boxes don't have matching names. So it is impossible to tell which xml file is for which image.

How do I download images and bounding boxes from imageNet such that corresponding image and annotation xml files have matching names?

But the downloaded images and bounding boxes don't have matching  ames. So it is impossible to tell which xml file is for which image.

I don't think that happens, image and annotation may have same name. But Bbox and image is odd.

Still if you meant image and annotation that would be odd.


What are good practices in reporting RMSE or MAPE estimates for a machine learning model? I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of the model. The average RMSE and MAPE of my model seem too high, with values of 3.4 million USD and 515 % respectively. However, looking at the RMSE and MAPE distribution in relation to the prices, I can see that there are a few, very low prices, that are difficult to predict and cause the very high mean error estimate. The scatterplot below shows price on the x axis and MAPE on the y axis

If I eliminate those 9 'outliers', the RMSE and MAPE go down to more reasonable values, namely 0.95 million USD and 26.7% respectively. My questions are:

should I eliminate those 'outliers' (i.e. very low prices) before training?
or should I train the model to predict all the data, and then exclude the outliers from the final RMSE/MAPE estimation?
or is there any other best practice in providing the most realistic estimation of the error
or, is there any other error metric that is robust to outliers?

Thank you very much

In general you should not attemt to make your model look better than it actually is. Always stick to the truth. Some ideas:

Outliers: It is not okay to simply drop the outliers. If you want to make a prediction or do inference, you have to ask yourself how relevant such outliers are in practice. Where do they come from (are data reliable or is it an error)? If they are part of the true bigger picture, you cheat if you just drop them. If they are a very special part of the market with specific attributes, you could try to control for these special types (e.g. by adding indicators or changing the parametric form of your model). This question essentially is data and method driven.
Your metrics should of course reflect the final model!
By definition of the RMSE it is the mean (not median). Naturally, you can describe the situation by discussing the influence of the outliers in your own way. But this will not make your model/fit better after all.

Finally: I think you should NOT ask about a metric that suits your approach and make it look better. You should ask yourself if you can improve your model. You did not specify what type of method you use and what data you work with, but I'm quite sure there are ways to improve the fit. The scatter plot looks like an L-shaped function. So this might be the first thing to start to work on!?


Machine Learning algorhtm that can be trained to learn Pig-Latin Pig-Latin, and other similair argots, were quite popular when i was young (but since i grew up in Sweden the biggest was Rövarspråket). These language games almost always consist of a given set of rules from which letters and words are translated into something else. For example in Pig-Latin the rules are the following (source):

  For words that begin with consonant sounds, all letters before the
  initial vowel are placed at the end of the word sequence. Then, "ay"
  is added, as in the following examples:[10]

  "pig" = "igpay"
  "latin" = "atinlay"

  When words begin with consonant clusters (multiple consonants that
  form one sound), the whole sound is added to the end when speaking or
  writing.[11]

  "cheers" = "eerschay"
  "shesh" = "eshshay"

  For words that begin with vowel sounds, one just adds "way" or "yay"
  to the end (or just "ay"). Examples are:

  "eat" = "eatway" or "eatay"
  "omelet" = "omeletway" or "omeletay"

  An alternative convention for words beginning with vowel sounds, one
  removes the initial vowel(s) along with the first consonant or
  consonant cluster. This usually only works for words with more than
  one syllable and offers a more unique variant of the words in keeping
  with the mysterious, unrecognizable sounds of the converted words.
  Examples are:

  "every" = "eryevay"
  "omelet" = "eletomay"

My question is if there are any Machine Learening algorithms that is suitable to be able to learn to translate a sentence to Pig-Latin by training on a large set of "english"="pig_latin"-pairs without explicity knowing anything about the rules? Or if anybody can motivate and point me to the right direction.

My goal is to be able to input an arbitrary word, which might not even be a real word, and get the "Pig-Latin"-translated word.

My initial idea was to set up a Neural Network (mainly because thats what I know best) where the input neurons should take the ASCII-code for each letter and the output should be the ASCII-code for each letter (always capital letters), i.e.

Input (for PIG):
080 073 071

Output (for IGPAY)):
073 071 080 065 089

A neural network can be trained to map for example 080 to 073, but how do i for example handle the aribtrary length of the input and output (if Neural Networks ar applicable to this problem).

Look at Neural Machine Translation models, you could either 'cheat' and tokenize both the English and the Pig-Latin words or you could make it character based which I think is more interesting and it allows to generalize for words out of dictionary. These models are called sequence to sequence models and there are plenty of implementations that can take pairs of sequences to translate. I think this task can be solved with accuracy close to 100%.

Here is an implementation in Keras: https://github.com/farizrahman4u/seq2seq

Here is an official TensorFlow tutorial: https://github.com/tensorflow/nmt


How to know the name of the predicted class? As we know all, after training a neural network, each neuron in the output layer presents one class. These class are in a numeric format for example 0 for sunny 1 for overcast and 3 for rainy. after the prediction phase, the model print for example 1, how to know that 1 corresponds to the class overcast?

You as the designer of the network specify each class in the training example. You set e.g. a car class to label $0$ and another class to $1$. During training your classifier tries to map the inputs to the corresponding class which are those numbers where you have specified in the dataset. This phase is called encoding. After outputting the label, you should decode the label which is a trivial task!


Machine learning model using keywords for binary decision I have a some experience from Uni with convolutional NN and edge detection, but haven't much explored the other types of machine learning models.

I was wondering if there might be one that is suited for being able to use certain, textual keywords (for example, words that were parsed from internet news sources) in order to arrive at some binary, yes/no decision?

The most obvious binary classification task coming from text data is sentiment analysis. Kaggle is plenty of datasets such as this one on movie reviews.

Sentiment classifiers can be either very simple, like logistic regressions on naive Bayes classifiers, or very complex such as the ones based on RNNs and/or part-of-speech tagging models.

If you are interested in news specifically, similar models can be used to estimate news polarization.