Keypoint matching using HoG and SIFT I have two images and I've found their keypoints using sift keypoint detector,
Now I have to match their keypoints with HoG features,
I know how to extract HoG description,
but I dont know how to combine it with SIFT and match the keypoints,
any ideas pls?
I am using opencv and python3

In general, you can use brute force or a smart feature matcher implemented in openCV.
Another approach is seeing the task as image registration based on extracted features.

The question may be what is the relation of HoG and SIFT if one image has only HoG and other SIFT or both images have detected both features HoG and SIFT.


Clustering based on similarity scores Assume that we have a set of elements E and a similarity (not distance) function sim(ei, ej) between two elements ei,ej ∈ E.

How could we (efficiently) cluster the elements of E, using sim?

k-means, for example, requires a given k, Canopy Clustering requires two threshold values. What if we don't want such predefined parameters?

Note, that sim is not neccessarily a metric (i.e. the triangle inequality may, or may not hold). Moreover, it doesn't matter if the clusters are disjoint (partitions of E).

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here).  For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters.  So you can probably use DBSCAN, although you may have to do some kind of nonstandard spatial index to do efficient lookups in your case.  Your version of epsilon-neighborhood will likely be sim > 1/epsilon rather than the other way around.  Same story with k-means and related algorithms.
Can you construct a metric from your similarity?  One possibility: dist(ei, ej) = min( sim(ei, ek) + sim(ek, ej) ) for all k ...  Alternately, can you provide an upper  bound such that sim(ei, ej) < sim(ei, ek) + sim(ek, ej) + d, for all k and some positive constant d?  Intuitively, large sim values means closer together: is 1/sim metric-like?  What about 1/(sim + constant)?  What about min( 1/sim(ei, ek) + 1/sim(ek, ej) ) for all k? (that last is guaranteed to be a metric, btw)
An alternate construction of a metric is to do an embedding.  As a first step, you can try to map your points ei -> xi, such that xi minimize sum( abs( sim(ei, ej) - f( dist(xi, xj) ) ), for some suitable function f and metric dist.  The function f converts distance in the embedding to a similarity-like value; you'd have to experiment a bit, but 1/dist or exp^-dist are good starting points.  You'd also have to experiment on the best dimension for xi.  From there, you can use conventional clustering on xi.  The idea here is that you can almost (in a best fit sense) convert your distances in the embedding to similarity values, so they would cluster correctly.
On the use of predefined parameters, all algorithms have some tuning.  DBSCAN can find the number of clusters, but you still need to give it some parameters.  In general, tuning requires multiple runs of the algorithm with different values for the tunable parameters, together with some function that evaluates goodness-of-clustering (either calculated separately, provided by the clustering algorithm itself, or just eyeballed :)  If the character of your data doesn't change, you can tune once and then use those fixed parameters; if it changes then you have to tune for each run.  You can find that out by tuning for each run and then comparing how well the parameters from one run work on another, compared to the parameters specifically tuned for that.


Automatically uses several cores on R I am using a library called MFE to generate meta-features. However, I am working right now with several files and I notice that I am using only 1 core of my machine and taking too much time.

I have been trying to implement some libraries as I saw in another question:
library(iterators)
library(foreach)
library(doParallel)

This one, but me being dumb could not implement it ='(.

I just would like to put this snippet running in all my cores so I can make it faster:

## Extract general, statistical and model based measures
metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

So how would I do this? MFE library

[EDIT]:

What I am doing right now:

library(iterators)
library(foreach)
library(doParallel)
library(mfe)

# foreach
split = detectCores()
eachStart = 25
# set up iterators
iters = iter(rep(eachStart, split))
# set up combine function
cl = makeCluster(split)
registerDoParallel(cl)
result = foreach(nstart=iters) %do%
  metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))
stopCluster(cl)

result

SAMPLE OF EXPECTED RESULT
What I am expecting is this following output and using all cores from my CPU.

First, you are not dumb. We are all learning here.

Second, I have looked at your code and I can see the problem closer now.

The code below works, but it doesn't give anything useful (nor does it lead to any speedup). I'll try to explain why.

library(iterators)
library(foreach)
library(doParallel)
library(mfe)

data = iris

# foreach
split = detectCores()
eachStart = 25

# set up iterators
iters = iter(rep(eachStart, split))

# set up combine function
cl = makeCluster(split)
registerDoParallel(cl)

result = foreach(nstart=iters, .packages = c("iterators", "mfe")) %dopar% {

  metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

}

stopCluster(cl)

This will return a list of length equal to the number of cores on your machine, but all of the elements in the list will be exactly the same. You are getting no speedup (in fact, your code will run slower if you use this) because all you are doing is running the exact same piece of code n = (number of cores on your machine) number of times, namely this:

metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

You aren't iterating over anything in your foreach loop to give any benefit to running in parallel. The overall idea of running stuff in parallel (in this context, anyway) is to run the exact same piece of code (i.e. a function) on each worker node, but with different inputs to the function's arguments in each call. In the context of a for loop, you can think of this as a single iteration of the for loop being sent to a single worker (splits, as you defined them in your code) on your computer. The benefit of course is that if you have many cores on your computer, different iterations of your for loop can be computed at the exact same time (vs. on a single core machine, you would have to work on a single iteration of the for loop one at a time). It is important to note that this all implies that each iteration in your for loop is independent of all other iterations and that the order in which you compute iterations does not matter (i.e., your algorithm is NOT sequential).

Here is one example that I can think of where you could consider running in parallel: good old k-fold cross validation. Each resulting training set receives the exact same pre processing but all you are changing is what folds make up the training and test sets.

This could be written like this:

library(foreach)
library(doParallel)
library(caret)
library(e1071)
library(tidyverse)

data = iris

cl = makeCluster(detectCores())

#Generate three folds from our dataset

folds = createFolds(y = data$Species, k = 3)

#Train a support vector machine in parallel using foreach. Returns the predictions from the fitted model.

results = foreach(i = 1:length(folds), .packages = c("caret", "e1071", "tidyverse")) %dopar% {

  training = data[-folds[[i]], ]
  test = data[folds[[i]],]

  #Pre processing, if desired

  #Train the support vector machine. Parameters are just for example, ripped from a webpage that just so happened to be using the exact same dataset.
  model = svm(Species~., data=training,
          method="C-classification", kernal="radial",
          gamma=0.1, cost=10)

  tibble(Row = folds[[i]], Predicted.Class = predict(model, newdata = test))

}

#Bind all the data frames in the list "results"

processedResults = bind_rows(results) %>%
  arrange(-Row)

Notice how we are iterating over different training/test splits (with these lines):

training = data[-folds[[i]], ]
test = data[folds[[i]],]

but the function itself remains the same. Each core will receive a different train/test set in each iteration of the for loop. In your code, you aren't changing arguments to your function and so there is no point in running your code in parallel.

Now, if I understand mfe correctly, it is generating some summary statistics from your dataset. This could maybe run in parallel (maybe some statistics can be computed in parallel? I doubt there is any speed to be gained here), but I highly doubt it is much faster due to the overhead associated with running stuff in parallel (and hence, why the package creator did not choose to implement such a feature).

You mentioned random forest and how you were able to run it in parallel easily. This is because a random forest is easily made parallel (it is "embarrassingly parallel"). Each individual tree can be fit on a single worker (because each tree is fit independently of all other trees in the forest on a bootstrapped training set). The order in which the trees are fit doesn't matter either, because we just average over all of the trees predictions to get our final predictions.

My two cents: running in parallel is only useful if the algorithm itself can be run in parallel. You can't take advantage of all of your cores if the algorithm itself cannot be "divided" and then "combined". And even then, if the algorithm is trivial to run on a single thread then the computational overhead associated with running stuff in parallel makes it generally worse in these cases.


What is the standard terminology for the output matrix of a hidden layer in a CNN? The output of a hidden layer $h_i$ in a convolutional neural network is (generally) a 3D grid of values. These values are the outputs of the neurons of layer $h_i$. Is there a standard way to refer to this matrix of values?

Not that I’m aware. The 2-d matrices indexed by the third dimension are often called channels. Other than that, “intermediate representations”, maybe? But since it isn’t really standard, you should explain whatever you decide to call it, to make sure it’s unambiguous.


Neural Network - distinguishing between several normalized values is impossible? It's a common practice to normalize inputs to the neural Network.

Let's assume we have a vector of activations.

One of techniques, the Layer Normalization simply looks at the vector's components,  re-centers this activated vector from $\mu$ to zero, then divides by the standard deviation $\sigma$

How is it then possible to distinguish activations [1,2,3,4] from [4,5,6,7]  if both will be re-centered to the same vector [-1.5, -0.5,  0.5, 1.5f]  and then divided by std deviation? Also, I can see such a problem when merely normalizing the input-state-vectors for any Neural Net.

Edit:

there seems to be a hint in first half of page 4 in the paper, however due to my weakness in Maths I can't comprehend it :(

Edit after accepted the answer:

guys, don't forget that Layer Norm (and Batch norm) both have Gain and Bias terms. If the network performs poorly, the gain is tweaked to undo division-by-std-deviation, and tweaking the bias is used to undo the shift (the re-centering). This allows some neurons to indeed pay attention to scaling and shifting, when really required.

The math you are talking about is in equation (7):

  Let x′ be a new data point obtained by re-scaling x by δ. Then we have,

  It is easy to see re-scaling individual data points does not change
  the model’s prediction under layer normalization. Similar to the
  re-centering of the weight matrix in layer normalization, we can also
  show that batch normalization is invariant to re-centering of the
  dataset.

It proves that the prediction over x' would be the same as in x. So what you say is right, they are indistinguishable.

And I think this is what we want because improves generalization in predictions!

For example in image classification, you are trying to detect orangutangs, if an image  I represent an orangutang, ( I x 5)+2 still is an orangutang!


How to map ground truth to prediction for UNet architecture I've gone through the paper describing the UNet convolutional neural network a number of times, but am still having trouble figuring out how to connect the output of the network to the ground truth targets.  Below is an image depicting the architecture of the network.

(source: uni-freiburg.de)
As can be seen from the figure, the output is a 388 x 388 x k matrix (where k is the number of classes).  The target segmentation in this case should be 572 x 572 (same spatial dimensions as the input image).  How do we match these up?  Are we suppose to perform some kind of interpolation on the output feature map to get it to match the dimensionality of the input?

It depends on how your UNet architecture is constructed.

In the example image you've shown, the output of the first few layers of convolutions is cropped to 392x392 before it's copied over to the input of the final convolutional layers. Then, it's decreased to 388x388 by convolution without padding. For the example, you're probably best off cropping your image to 392x392, then scaling down to 388x388, as the lost resolution due to the convolution is more analogous to scaling than cropping.

You could also construct the network to output an image of the same dimensionality to avoid this issue. The downside to this approach is that the network will produce some strange/poor results near the boundaries of the image, so you're spending computational power on pixels you will need to disregard anyway.


How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed? Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have managed to reduced the number of continuous features from 69 to 41. Now I am trying to reduce the number of categorical feature, but since my data is not normally distributed I can't use t-test or ANOVA. Which other tests can I use to see if the features are significant to predict the target?

If I understand your question correctly, You are asking how to reduce categorical features from a dataset. If yes then few of the approach I can think of -

1> Iterative Process - Build a model with all numerical features and one categorical feature then evaluate your improvement of the model by whatever metrics you are using and then add other categorical features and so on. So if you have N cat features you will be building N+1 models.

2> chi square test of predictor and target variables.

3> (what I use) Build a model with all the available features and measure it's performance and then use the feature importance functionality of that model to determine which feature is important. In case of linear regression, Higher the value of coefficient better the feature. Alternatively, you can use L1 regularization to check for non zero features. Do check for multi-collinearity before considering feature importance in linear regression.


Why is a lower bound necessary in proofs of VC-dimensions for various examples of hypotheses? In the book "Foundations of Machine Learning" there are examples of proving the VC dimensions for various hypotheses, e.g., for axis-aligned rectangles, convex polygons, sine functions, hyperplanes, etc.

All proofs first derive a lower bound, and then show an upper bound.  However, why not just derive the upper bound since the definition of VC dimension only cares about the "largest" set that can be shattered by hypothesis set $\mathcal{H}$?  Since all examples ends up with a lower bound matching the upper bound, is the lower bound just helpful/useful to set a target when trying to show an upper bound?

Reference:
From page 41 of this book pdf version
https://pdfs.semanticscholar.org/e923/9469aba4bccf3e36d1c27894721e8dbefc44.pdf

Lets use the quotes from the book.

  To give an upper bound, we need to prove that no set $S$ of cardinality
  $d + 1$ can be shattered by $H$

By doing this, we are proving that $\text{VCdim}(H) < d+1$, but it does not necessarily mean $\text{VCdim}(H) = d$. For example, we may go for an easier-to-prove failure like sets with $2d+1$ members, and consequently prove $\text{VCdim}(H) < 2d+1$, although we know $\text{VCdim}(H) = d$. Therefore:

  To give a lower bound $d$ for $\text{VCdim}(H)$, it suffices to show that a set $S$
  of cardinality $d$ can be shattered by $H$

We need to prove the lower bound as well to show that $d \le \text{VCdim}(H) < d+1$, which implies $\text{VCdim}(H) = d$. The last equality means that for each size $1$ to $d$, $H$ can shatter at least one set $S$ with that size.

Note that for the $2d+1$ example, we will fail to prove $2d \le \text{VCdim}(H)$, therefore $\text{VCdim}(H) \neq 2d$, and consequently we must try to prove a smaller (probably harder to prove) upper bound $d+1$.


MAE and MSE are Nan for regression with Neural Networks? I doing a simple neural network for Reggression, I didn't get any error but the MSE & MAE are Nan.  The code is:

dataset = pd.read_excel('data_Z.xlsx')
#df = pd.DataFrame(stock, columns= ['X1', 'X2', 'X3', 'X4', 'X5', 'Y1',
'Y2'])

#Variables
x=dataset.iloc[:,0:5]
y=dataset.iloc[:,5].values
y=y.reshape (-1,1)
scaler = MinMaxScaler()
print(scaler.fit(x))
print(scaler.fit(y))
xscale=scaler.transform(x)
yscale=scaler.transform(y)
X_train, X_test, y_train, y_test = train_test_split(xscale, yscale)
model = Sequential()
model.add(Dense(12, input_dim=5, kernel_initializer='normal',
activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()

model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
history = model.fit(X_train, y_train, epochs=150, batch_size=50,  verbose=1,
 validation_split=0.2)

I cannot understand why?

Possibilities:

There are missing values in your dataset.
You are introducing missing values with your scaling.  (In particular, were any of the features constant?  And you seem to be misusing the scaler: the fit method sets the max and min from the data, so running fit(x) then immediately after fit(y) probably isn't what you meant to do.)
If neither of the above is the problem, then presumably Keras is introducing the problem.  This is trickier to diagnose, but often seems to be from vanishing/exploding gradients?  Do you get real scores initially, or NaN's from the very start of training?  See also https://stackoverflow.com/questions/37232782/nan-loss-when-training-regression-network
and its Related questions.


design pattern for extracting features I am looking for a design pattern that is relevant to a module that extracts features.
I want to define a certain number of features over my data points, and then according to the performance and the feature selection, I may want to remove some of them and add others, and also I may want to consider any subsets of them to test.

What is a good design pattern to do that? Did I miss something obvious? I am neither an engineer nor a developer, so I never study such things but I understand that it could help me a lot!

Thanks for any help,

When I think about how to implement a certain ML or data mining process in an OOP (like Java), I usually go to see how smarter people than me designed their system.
In this case, I'd see how Weka, RapidMiner, JAVA ML or others decided to tackle this problem.

In your case of feature/attribute selection, i'm adding a link to Weka's API.
If you want to better understand how it's done, you should download Weka and play with the source code.

http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/AttributeSelectedClassifier.html

Without getting into specific implementations, I think that a linked list of any sort would do the trick, since you want to select a subset of all features.


Need help with how to diagnose training and validation metrics I'm currently learning to build a DNN binary classification model on some dataset. But when I analysed the results, I found it counter-intuitive that the training accuracy changed almost at the same rate as the validation accuracy. So did the losses. And they all reached a plateau at some point. How could I possibly improve the model performance at this stage? Here's a snippet of the results:

fold: 0 epoch: 0 batch: 0
training loss: 0.674389 validation loss: 0.67371
training accuracy: 0.656331 validation accuracy: 0.656968
Fold: 0 epoch: 0 batch: 500
training loss: 0.527997 validation loss: 0.527813
training accuracy: 0.734021 validation accuracy: 0.733639
Fold: 0 epoch: 0 batch: 1000
training loss: 0.526019 validation loss: 0.526277
training accuracy: 0.736067 validation accuracy: 0.735111
Fold: 0 epoch: 0 batch: 1500
training loss: 0.525466 validation loss: 0.525721
training accuracy: 0.736672 validation accuracy: 0.73532
Fold: 1 epoch: 0 batch: 0
training loss: 0.525416 validation loss: 0.525261
training accuracy: 0.735794 validation accuracy: 0.736488
Fold: 1 epoch: 0 batch: 500
training loss: 0.525751 validation loss: 0.525623
training accuracy: 0.735472 validation accuracy: 0.735887
Fold: 1 epoch: 0 batch: 1000
training loss: 0.525579 validation loss: 0.525459
training accuracy: 0.735538 validation accuracy: 0.736077

Generally, the fact that your training and validation performance are improving at the same rate is a good thing- this (usually) means that the algorithm is learning generalizable features of your problem space rather than overfitting to the noise of your training set.

Reaching a plateau in performance is also to be expected- it's very rare that a real-world machine learning problem can be perfectly solved, and a perfectly solved problem would be the only type that didn't reach a plateau in testing and validation performance (before it eventually did reach a plateau at 100% accuracy).

Think of the plateau as the maximum performance that can be achieved given the particular parameter values, features, and architecture of the solution. In order to achieve performance beyond your plateau values, one of these considerations will need to be adjusted.


UNK in ludwig learnning model I am using ludwig to train and test on the dataset. There is one independent variable that has a text data type. I used 'Parallel_CNN' as a text encoder for the independent variable. My dependent variable has the category data type and it has a binary value (i.e. 0 or 1). After the model finished I plotted the predicted and actual output in the form of confusion matrix which is shown as below:

I was expecting TP,TN,FP, and FN in the confusion matrix. However,  which is unknown also found in the confusion matrix. I tried to google it to see the meaning and impact of unknown on model learning but could not find any useful information. Could anyone help me why does 'Unk' comes into model? Does that mean model is not efficient in learning?

I don't know ludwig or even "parallel CNN" but there's a clear problem: this is not a confusion matrix, this is a heat map.

A confusion matrix shows the number of instances for every combination of predicted/true label. It's for categorical labels, you could indeed obtain TP/FP/FN/TN classification status from it.
A heatmap is for numerical values, as can be seen from the scale on the right side.

It's very likely that this heatmap is made of the probabilities of the labels for every instance, rather than the labels themselves (also the choice of colours is unusual but this doesn't matter).
The good news is that it appears that most instances which have 0 or 1 as true label are correctly predicted (high value for 1,1 and 0,0).
The bad news is that there's an issue in the labels themselves: apparently a large number of instances (at least a third I think) don't have any label and are considered 'UNKNOWN'. And apparently the system just considers these 'UNKNOWN' instances as a third class, so your labels are not binary actually. It manages to recognize this class quite well, but it's probably an issue you should investigate.


ANN on Pattern Recognition I have been trying to apply a simple neural network using keras to predict a sequence of numbers and the rule is if the input integer is odd it should be 4 and if its even it should be 2. Yet the neural network gets stuck at a 60% accuracy rate. Anyone know a solution to this?

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

from keras.layers.normalization import BatchNormalization

from sklearn.model_selection import cross_val_score
from keras.wrappers.scikit_learn import KerasClassifier

import numpy as np

def gen(x):
    if (x%2==0):
        return 2;
    else:
        return 4;

a = []
for i in range(1,100001):
    a.append([i,gen(i)])

a = np.array(a)

x = a[:,0:1]
y = a[:,1:2]

def MakeClassifier():
    network_classifier = Sequential()
    network_classifier.add(Dense(units=2,kernel_initializer="uniform",activation="relu",input_dim=1)) #Hidden Layer1 taking into account number of inputs(independant variables(x)
    network_classifier.add(BatchNormalization())
    network_classifier.add(Dense(units=1,kernel_initializer="uniform",activation="sigmoid"))#OutPutLayer

    network_classifier.compile(optimizer="adam",loss="binary_crossentropy",metrics=["accuracy"])#If multicategorical then categorical_crossentropy
    return network_classifier

classifier = KerasClassifier(build_fn= MakeClassifier , batch_size = 10 , epochs = 1000)

classifier.fit(x,y,epochs=100,batch_size=1000)

print(classifier.predict([[6],[7]])) #Should Predict 2 and 4

There are two possible reasons for this result:

Low number of training examples
Using dense layers without batch normalization

You have a relatively deep network and your training set size is small. In such cases if you run your model too many times you will certainly overfit the training data. The reason is that whenever you have a powerful model that can learn complicated functions and if you provide low number of training examples, it has capability to fit the data and not learn it. Suppose that you have a fitting problem in calculus and you have 4 points and you also have polynomials with degree of four. In such cases you can fit the data exactly without any error but the point is that you are fitting it, not learning it. In your case your model is powerful and it tries to fit the data, not to learn it. The cure is to provide enough training data.

The reason that your model can not learn is that you are stacking dense layers without batch normalization. each time you update the weights, the output of the deep layers change and this cause the covariat shift. To avoid this problem use batch normalization.

As a solution for you I recommend you to putting just two hidden layers and manipulate the number of units in those layers and provide more training examples. Your task can easily be learned by a two hidden-layer-network.

After struggling for about one day finally I want to express my opinion about the problem. Depending on the problems, they may be solved using machine-learning or other techniques. Machine learning problems are those which you can not define an appropriate function to map the input to output or it may be so much hard to do so. Whenever you have the function, it can be used as the hypothesis, the final goal of machine learning algorithms. I tried hard and put so much time on this code and I get the same result. Nothing is going to be learned at all. I have been trying for about one day and still no progress has been seen. To explain the reason, the data is so much hard to be learned! for imagining how difficult it is, I recommend you to write numbers from one to ten in a straight line and put a line between consecutive numbers. The numbers are endless, so you will have no generalization because the boundaries that are going to be found will work just for the two neighbor numbers. This means that if you use the current features, you can not separate, learn, your data. I tried to do, somehow, feature engineering and used the following code to solve the problem:

import keras
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers.normalization import BatchNormalization
from keras import regularizers
from keras.optimizers import Adam

def gen(x):
    if (x % 2 == 0):
        return 0; # represents 2
    else:
        return 1; # represents 4

a = []
for i in range(1,100001):
    temp = np.random.randint(0, 10000000)
    a.append([temp, temp ** 2, temp ** 3, temp ** 4, gen(temp)])

a = np.array(a)

x = a[:, 0: a.shape[1] - 1]
y = a[:, a.shape[1] - 1:]

mean_of_x = np.mean(x, axis = 0, keepdims = True)
std_of_x = np.std(np.float64(x), axis = 0, keepdims = True)
x = (x - mean_of_x) / std_of_x

n_classes = 2
y = keras.utils.to_categorical(y, 2)

percentage = 95 / 100
limit = int(percentage * x.shape[0])

x_train = x[: limit, :]
y_train = y[: limit, :]

x_test = x[limit: , :]
y_test = y[limit: , :]

x_train.shape

model = Sequential()
model.add(Dense(1000, activation='relu', input_shape=(x_train.shape[1],)))
model.add(BatchNormalization())
# model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(BatchNormalization())
# model.add(Dropout(0.5))

model.add(Dense(n_classes, activation='softmax'))

model.summary()

model.compile(loss = 'categorical_crossentropy', optimizer = keras.optimizers.Adam(lr = 0.0001, decay = 1.5), metrics=['accuracy'])

model.fit(x_train, y_train, batch_size = 128, epochs = 2000, verbose = 1, validation_data=(x_test, y_test), shuffle = True,
          class_weight = {0: 10, 1: 1})

As you can see, the above code uses high order polynomial. Surprisingly, no progress was seen here too. this model has the mentioned problem for the previous feature in higher dimensions. That's why the learning does not happen here too.

the point here is that although you can not learn using the current feature, number itself, or its high order polynomials, you already have a solution for the problem. Instead of passing the number itself to the learning problem, pass the modulus two of the current number. This feature is so much easy to be learned. you may need just one unit.


Highly Imbalanced dataset fro classes more than 200 I have a text dataset where I need to train a classifier to classify the titles into categories. The dataset shape is more than 575000. There are 256 target classes here. The problem is the dataset is highly imbalanced. For target X1 it has 171793 records, X2 has 101575,........Xn-1 has 2, Xn has 2 records. Consider the target value counts are in decreasing order.

To handle with the imbalanced dataset, oversampling and undersampling works for multiclass say 3 classes. But in my case, there are 256 classes. How do I sample my dataset in this situation? How do I sample the dataset in a way so my model is stable for all the targets?

Do I have to remove the classes which have value counts 2 - 100 from this dataset? and apply undersampling/oversampling. Is there any approach to handle these type of situation?

The situation you describe is a few-shot learning problem: you have a lot of classes and only a few examples for some of them.

Similarity metric learning with siamese neural networks is well suited for this task. The idea is to learn a general similarity metric between examples, then classify new examples as belonging to the class of the "closest" sample from the training set. It seems a bit complex but it's probably the best way to learn "across" classes.

The link I've shared above is applied to an NLP problem so should be able to reuse it fairly easily.


Does Decision tree classifier calculate entropies before transforming categorical features using OneHotEncoder or transformation should be done I am new to machine learning, and I've got to the point to drop out from it as online tutorials are pretty confusing as well.

Entropy and Decision trees

One of confusing tutorials was as the following:

Another tutorial was pretty straightforward and comprehensive in term of how entropies and information gain is being calculated, but he didn't split the data.

Where the instructor started the entropy calculations and ended up with the following tree:

I did understand the calculation as we used to calculate entropies since high school. The confusing part starts when I decided to program the same steps using python:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_excel('data.xlsx')
X = dataset.iloc[:, 0:4]
y = dataset.iloc[:, 4]

At this point, I tried to apply DecisionTreeClassifier directly on X and y. And BAM. Lots of errors appeared at the console. So I used LabelEncoder and OneHotEncoder:

from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()
X['Outlook'] = lb.fit_transform(X['Outlook'])
X['Temp'] = lb.fit_transform(X['Temp'])
X['Humidity'] = lb.fit_transform(X['Humidity'])
X['Windy'] = lb.fit_transform(X['Windy'])
y = lb.fit_transform(y)

from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

ohe.fit_transform(X)

I then figured it out, as there is categorical data that should splitted using OneHotEncoder. By that, the 4 columns (Outlook, Temp, Humidity, and Windy) will turn into:

Rainy   Overcast   Sunny   Hot   Cold   Mild     High   Normal    False    True

As independent features, and then the calculation should start from this point. So which concept is used by the model to calculate the entropies, the one that confused me even if user used OneHotEncoder or my logic is the true one.

It's not totally clear to me what is your main question but here are a few points which might help you:

Like any regular classifier, a decision tree classifier does not transform the features. It's up to the user if they want to apply any transformation before calling the classifier.
Standard decision tree classifiers can deal with features which have multiple values.

Afaik training a decision tree model using a library classifier is not going to help you much understanding how the model is calculated, it's just going to give you the result. You could try to code your own implementation if you want.


How to process natural language queries? I'm curious about natural language querying.  Stanford has what looks to be a strong set of software for processing natural language.  I've also seen the Apache OpenNLP library, and the General Architecture for Text Engineering.

There are an incredible amount of uses for natural language processing and that makes the documentation of these projects difficult to quickly absorb.

Can you simplify things for me a bit and at a high level outline the tasks necessary for performing a basic translation of simple questions into SQL?

The first rectangle on my flow chart is a bit of a mystery.

For example, I might want to know:

How many books were sold last month?

And I'd want that translated into

Select count(*)
  from sales
  where
   item_type='book' and
   sales_date >= '5/1/2014' and
   sales_date <= '5/31/2014'

Natural language querying poses very many intricacies which can be very difficult to generalize.  From a high level, I would start with trying to think of things in terms of nouns and verbs.

So for the sentence: How many books were sold last month?

You would start by breaking the sentence down with a parser which will return a tree format similar to this:

You can see that there is a subject books, a compound verbal phrase indicating the past action of sell, and then a noun phrase where you have the time focus of a month.

We can further break down the subject for modifiers: "how many" for books, and "last" for month.

Once you have broken the sentence down you need to map those elements to sql language e.g.: how many => count, books => book, sold => sales, month => sales_date (interval), and so on.

Finally, once you have the elements of the language you just need to come up with a set of rules for how different entities interact with each other, which leaves you with:

Select count(*)
  from sales
  where
   item_type='book' and
   sales_date >= '5/1/2014' and
   sales_date <= '5/31/2014'

This is at a high level how I would begin, while almost every step I have mentioned is non-trivial and really the rabbit hole can be endless, this should give you many of the dots to connect.


K Nearest Neighbour with different distance matrix to each datapoint I'm wondering if there is library support in python (such as sklearn) for doing KNN on a data set that has a custom distance matrix (positive definite) for each data point (x is a query point, $x_i$ is a data set point):
$$
d(x,x_i) = (x-x_i)^TQ_i(x-x_i)
$$

I know that for a fixed positive definite matrix for all data points, this is a metric that I can transform into
$$
Q = A^TA
\ \ \ \ \ \ \
d(x,x_i) = (Ax - Ax_i)^T(Ax - Ax_i)
$$
Which I can compute via normal KNN by first transforming the input space via multiplying $A$.

My problem of having a separate matrix for each data point came up because I have a covariance around the neighbourhood of each point. KNN can then be interpreted as what are the most likely neighbourhoods this query point lies in. If a neighbourhood doesn't vary along a dimension then we should penalize difference along that dimension highly in terms of increasing distance.

As pointed out by @Pratik Deoghare you can create a custom metric on sklearn kNN and you can see how in the link he provided.

But you want a function to that is different for each $x_i$, that is not a metric in the mathematical sense, but I can see how that could benefit the algorithm either way.

the function you pass as a metric (see how in the link) could be defined as

def creatmydist(AllA):
    Alist = AllA
    def mydist(x, y):
        nonlocal Alist
        if x[-1] == 0: i = y[-1]
        else: i=x[-1]
        A = Alist[i]
        x = np.dot(A,x[0:len(x)-1])
        y = np.dot(A,y[0:len(y)-1])
        return np.dot(x-y,x-y)
    return mydist

Where AllA should be a list with all $A = (A^T)^{-1}Q$ and every $x_i$  should have as last element it's index $i$


3d input for Dense Layer Keras Is there any example of how Keras Dense layer handles 3D input.
The documentation explains the following:
If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot).
But I could not understand the internal matrix calculation
For example:
import tensorflow as tf
from tensorflow.keras.layers import Dense
sample_3d_input = tf.constant(tf.random.normal(shape=(4,3,2)))
dense_layer  = Dense(5)
op = dense_layer(sample_3d_input)

based on the documentation for a 3D input of shape (m,d0,d1), the shape of Layer's weight_matrix (or) kernel will have the shape (d1,units) which is (2,5) in this case. But I dont understand how the op is calculated to have the shape (m,d0,units)

In a regular fully connected layer (Dense), the computation is done using the following Matrix operation : $$R = A*W + B$$
With all matrixes being vectors (if batch size = 1), exept $W$, which has size(inputsize, outputsize).
for a 3D input, the way TF computes the output is simply by applying this formula only to the last dimension, considering all other dimensions as similar to batch sizes.
So in your example with input of size (4,3,2), Tensorflow 'flattens' the vector into a (12, 2) matrix, computes the result as if it was a 2D input, this gives a (12, 5) vector, and then 'unflattens' the vector to give back its 3 dimensions (4,3,5).


How to create clusters of position data? I am asking this question because the previous one wasn't very helpful and I asked about a different solution for the same problem.

The Problem

I have lateral positions, xcoord, of vehicles over time which were recorded as the distances from the right edge of the road. This can be seen for one vehicle in the following plot:

Each point on the plot represents the position of the front center of the vehicle. When the vehicle changes the lane (lane numbers not shown) there is a drastic change in the position as seen after the 'Start of Lane Change' on the plot.
The data behind this plot are like below:

  Vehicle.ID Frame.ID   xcoord Lane
1          2       13 16.46700    2
2          2       14 16.44669    2
3          2       15 16.42600    2
4          2       16 16.40540    2
5          2       17 16.38486    2
6          2       18 16.36433    2

I want to identify the start and end data points of a lane change by clustering the data as shown in the plot. The data points in the plot circled in red are more similar to each other because the variation between them is smaller compared to the data points in the middle which see large variation in position (xcoord).
My questions are: Is it possible to apply any clustering technique to segment these data so that I could identify the start and end point of a lane change? If yes, which technique would be most suitable?
I use R. I have tried Hierarchical clustering before but don't know how to apply it in this context. Please help.

I doubt any of the clustering algorithms will work well.

Instead, you should look into:

segmentation (yes, this is something different), specifically time series segmentation
change detection (as you said, there is a rather constant distribution first, then a change, then a rather constant distribution again
segment-wise regression may also work: try to find the best fit that is constant, linearly changing, and constant again. It's essentially four parameters to optimize in this restricted model: average before and after + beginning and end of transition.


How to make multiple regression perform better for outliers? (without reducing effect of them) I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values.

So I wonder what technics exist to make algorithm perform better for high values that can't be dismissed as outliers. You see, these few high values make MSE very large, because the models tends to underestimate these high values, predicting them to be 2 or more times smaller.

I have an idea to generate fake data for outliers, but I haven't found how to do it the right way for regression. Is it right to generate fake data for high values based on their proximity by mixing their features? So it would be smth like SMOTE, but instead of classes we have nearby values?

Or, perhaps, the other idea is to cluster targets by density and then to generate balanced clusters by generating fake data with SMOTE or ADASYN?

P.S. Note that I don't want to reduce effect of outliers. On the opposite, these high values are extremely important, so I want model to perform well for them.

When it comes to outliers, one possible way of reducing the effects of the same is by means of a weighting mechanism.

When a standard regression is run on an entire dataset, all the observations in the dataset are being weighted equally.

Therefore, you could choose to run a robust regression where you use a weighting system, e.g. Huber, Bisquare, to adjust the weights so as to place a smaller weight on outliers in your dataset.

One way of detecting outliers in the first instance can be through the use of Cook's Distance, whereby if the distance is greater than 1 (as a rule of thumb), then the observation could be deemed to be an outlier.

You might find the following references helpful:

ROBUST REGRESSION | R DATA ANALYSIS EXAMPLES
Robust Regression — Reduce Outlier Effects


How to deal with missing values in the survey data to perform Paired sample t test I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it  providing efficient services to the patients?
We have two columns 'Expected service quality' and 'Perceived service quality'. We will perform this equation= Service quality=Expected-Perceived to see if the service quality is positive or negative. If the difference is negative, it states we need improvement in that area.
Now the problem is the two columns have many missing values and I have to perform paired t test on the data. In the first column there are 15 missing values and in 2nd I have 56 missing values. How do I deal with these missing values?

You simply cannot use these data points for this test, you have to use only the data points for which both variables have a value. This will make the sample smaller, so it's less likely to result in a significant difference.


pytorch convolution with 0-stride along one dimension For some square images, I'd like to use torch.nn.Conv2d with the kernel as a vertical block. As in, the kernel size is defined as max value of the first dimension by 1. Since the first dimension has no more room, I'd like to have 0 stride along that first dimension. I tried using the following definition:
self.conv1 = torch.nn.Conv2d(3, 32, (max_dim_0, 1),
                                    stride=(0, 1),
                                    padding=0,
                                    dilation=1,
                                    groups=1,
                                    bias=True,
                                    padding_mode='zeros')

However, this causes my training loop throws a RuntimeError: non-positive stride is not supported exception. Is there a way to define the kernel as a vertical block?

Setting stride to 0 is not necessary, torch will simply compute with respect to the input tensor sizes, so you can set stride to (1,1).
For x of size (batch_size, 3, max_dim_0, max_dim_0) (square image) the tensor output will be of size (batch_size, 32, 1, max_dim_0).


Are there any annotators or Named Entity Recognition for license plate numbers? Most vehicle license/number plate extractors I've found involve reading a plate from an image (OCR) but I'm interested in something that could tag instances of license plates in a body of text. Are there any such annotators out there?

There are a lot of pretty decent tools out there for text annotation in general, and given the broad nature of the task you're approaching (license plates are about as general as words), the annotation tools you are looking at should probably come from the more classical tools for annotation.

There was actually a pretty good discussion about annotation tools on this question, which should actually apply to this problem. The most relied-upon thing in annotation right now is probably brat. You can learn more about brat here.

Hope that helps! Let me know if you've got any more questions.


Is Loss not a good indication of performance? Im trying to segment 3D volumes using a 3D uNet network. Ive reached a stage where I am getting very good validation loss using CrossEntropy and BCE

idx:  0 of  53 - Validation Loss:  0.029650183394551277
idx:  5 of  53 - Validation Loss:  0.009899887256324291
idx:  10 of  53 - Validation Loss:  0.05049080401659012
idx:  15 of  53 - Validation Loss:  0.02019292116165161
idx:  20 of  53 - Validation Loss:  0.04724293574690819
idx:  25 of  53 - Validation Loss:  0.02810296043753624
idx:  30 of  53 - Validation Loss:  0.02642594277858734
idx:  35 of  53 - Validation Loss:  0.029894422739744186
idx:  40 of  53 - Validation Loss:  0.04158024489879608
idx:  45 of  53 - Validation Loss:  0.04574814811348915
idx:  50 of  53 - Validation Loss:  0.05406259000301361

I assumed my network is performing very well so i wrote a script to visualize my network outputs against their respective targets. What I get is something very different, not something that justifies this loss.
The samples are of depth 32 and I've outputted each z-plane as a single image.
Here is the target:

And the predicted output:

All samples are like this, not one of them accurately represent the target with the reported loss.. so I ask is my loss wrong? What should I look into to fix this?

Thanks

To answer the title of your question:

  Is Loss not a good indication of performance?

It is only a relative indicator of performance over the training session.

First, what is loss anyway? In general, the loss is some expression of the difference between the model's predicted output and the target output. Depending on the loss function used (e.g. if log is involved) or on the nominal values of the inputs themselves, the value of the loss can also be very small or very large.

People usually normalise data so the values are smaller, but the point is that you cannot always say that a validation loss of 0.0012345 is actually a good value. Or that 12345 is definitely bad!

Other possible loss functions

Other 3d segmentation models that I have seen it is common to use the Dice Coefficient for your cost/loss. Maybe give that a try.  The Dice coefficient is essentially the same as the F1 score; you are really finding a trade-off in how to penalise a model for its mistakes in classification e.g. of pixels or voxels. Do you want to strongly punich bad cases or rather a more averging approach. As that link points out, it is similar to the difference between the $L_1$ and $L_2$ losses.

There is also the Jaccard index, which is essentially the same as the Dice Coefficient.

The Tversky index is a generalisation of the two - it is an asymmetric similarity measure. :

$$
Tversky(A, B; α, β) = \frac{|TruePos|}{|TruePos| + \alpha |FalsePos| + \beta |FalseNeg|}
$$

The Dice coefficient is this with $\alpha = \frac{1}{2}$ and $\beta = \frac{1}{2}$
The Jaccard index instead has $\alpha = 1$ and $\beta = 1$

This might be a nice approach for your problem to tweak how the loss is computed.
There are plenty of source online to explain more about these well-defined measures and perhaps help you gain an intuition for their results.

Your results

With the losses you posted, it doesn't actually look as good as you explained. I am not sure exactly what your idx values mean, but the loss values are actually going up:

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: import matplotlib.pyplot as plt
In [4]: val_loss = np.array([0.029650183394551277, 0.009899887256324291, 0.0504908040
   ...: 1659012, 0.02019292116165161, 0.04724293574690819, 0.02810296043753624, 0.026
   ...: 42594277858734, 0.029894422739744186, 0.04158024489879608, 0.0457481481134
   ...: 8913, 0.05406259000301361])
In [5]: pd.Series(val_loss, index=range(0, 51, 5)).plot()
Out[5]: <matplotlib.axes._subplots.AxesSubplot at 0x7f4405326390>
In [6]: plt.show()

This suggests you might be overfitting. However, the differences between the predictions and ground truth in the images you show suggest there is a more fundamental problem.


What is the error rate for random guessing? I am studying for my Machine Learning exam. In sample exam questions, there is a specific one that I could not understand:

And the question is: "What is the error rate for random guessing?"

I could not understand the question, can anyone explain it to me?

Random guessing of value of whcih column? For example random guessing the gender of each row is: probability of it is male but guess female + vice versa which is $\frac{7}{10}\times\frac{1}{2} + \frac{3}{10}\times \frac{1}{2} = \frac{1}{2}$ and it means the error of random guessing the value of Gender is $50\%$.
Also you can compute for the others by the possible value for each column. For example for the Hair Length the error would be $\frac{2}{3}$ or $66.66\%$.


Classification procedure using Minimum Redundancy Maximum Relevance (MRMR) After selecting features with MRMR (by quantizing original feature space of training data), should we classify the test data using quantized values or original values?

If you are not using the quantized values, what was the point of using the MRMR? from what I understand, it helps you "select a feature subset set that best characterizes the statistical property of a target classification variable" (sounds like PCA). So after you have done that, it helps your classifier to focus on the more important features.


Correct normalisation terminology Take the given values: -2.998, 8.0, 2.5.

My normalisation function takes a sum of the absolute values (13.498), let's call that x.

Next, each absolute value is divided by x (essentially calculating the mean distribution) - here's the result: 0.222, 0.593, 0.185.

So our result values are spread between 0-1, (it's what I need), and works great. But what's the precise terminology for this type of normalisation? I want to be able to document and comment my code accurately.

This is a bit unusual because you are throwing away information on the sign: The vectors (-2.998, 8.0, 2.5) and (2.9998, 8.0, 2.5) yield the same result.

Therefore I do not think there is a standard term for this procedure.
A clear comment in your code could be 'Take the absolut value and normalize by the sum of all values' or, if you are feeling mathematical today, 'Take the absolute value and normalize by the $L^1$-norm'.


What are recommended ways\tools for processing large data from Excel Files? A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time data to the millisecond. The processing also has to be efficient in view of the large data volume. What is the recommended methodology and tool to handle this?

I've seen others convert excel to CSV, and then confining their analysis to the CSV itself. Is this method advantageous in the sense that CSV is faster to read in, and can be processed by a larger toolset? Is there any powerful tool\library that can do batch conversion, and even extract custom formatting data?

Thanks and Regards

pandas loads a csv upto 10x faster than excel. So, if you can, please convert these files to csv, which ever ones that are being loaded several times.

I have been using cython, which seems to speedup read functions and processing. Please follow this link for more details. It says, that using cython speeds up the processing 10x.

Suggestion: If you can, I would recommend to store data in database and use an orm like django/peewee for the data processing.

Please lemme know if you have any other queries.


Format for storing textual data For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a better format? Or does it not matter?

In general, use a storage method that allows you to quickly query it. If your collection is huge, you might need something Lucene-based, like ElasticSearch. If you are a SQL crack and your favorite DB supports it, a full-text index might do the trick. For small sizes like the 5000 documents, even Linux' LocateDb+grep or OSX' spotlight could be enough.

The important point is to be able to quickly verify assumptions about the content of your data - how many documents contain X and Y, does any document contain W but not V, etc.

This will be useful at both the whole set level as well as for analyzing your topic clusters. Finally, a few GNU tools or SQL mastery can also help you profile your document sets more efficiently (n-gram counts/ranks, collocations, concordances, etc)

Edit: that means, for the above reasons and given your collection size, good old plain text (in a file system or a database) might be more efficient than any "fancy" format.


Train and Validation Curve I'm new in DeepLearning. I'm not good at understanding and commenting on graphics.Can you help me with these graphs

The questions I ask myself when I see learning graphs are the following ones:

Is the loss decreasing and the accuracy increasing ? if yes, your network is learning and everything works fine, which is already a good new.
Have we reached a kind of plateau ? (in accuracy especially), which means learning is over. (Here maybe you could train your network a bit more, it doesn't seem to have reached a plateau yet).
What is the value of that plateau ? Final validation accuracy is the most important value we seek when learning.
Do the curve have an exponential shape ? This is a tricky one and i'm not sure everyone does this, but i like having exponential shape curves when learning (which is not possible with every optimizer), so in your case (linear curves) i would try increasing the learning rate to get a faster learning.
Is the validation accuracy decreasing at the end of the training ? in which case it means we are overfitting and need to stop learning earlier. (Not your case here)

Hope this helps, this is only my tips for analysing learning graphs and may not be what everyone ask theirself when facing these graphs.


How to select algorithms for ensemble methods? There is a general recommendation that algorithms in ensemble learning combinations should be different in nature. Is there a classification table, a scale or some rules that allow to evaluate how far away are the algorithms from each other? What are the best combinations?

In general in an ensemble you try to combine the opinions of multiple classifiers. The idea is like asking a bunch of experts on the same thing. You get multiple opinions and you later have to combine their answers (e.g. by a voting scheme). For this trick to work you want the classifiers to be different from each other, that is you don't want to ask the same "expert" twice for the same thing.

In practice, the classifiers do not have to be different in the sense of a different algorithm. What you can do is train the same algorithm with different subset of the data or a different subset of features (or both). If you use different training sets you end up with different models and different "independent" classifiers.

There is no golden rule on what works best in general. You have to try to see if there is an improvement for your specific problem.


Reduce dimension, then apply SVM Just out of curiousity, is it generally a good idea to reduce the dimension of training set before using it to train SVM classifier?

I have a collection of documents, each of them is represented by a vector with tf-idf weight calculated by scikit-learn's tfidf_transformer. The number of terms (feature?) is close to 60k, and with my training set that consist of about 2.5mil of documents, it makes the training process to go on forever.

Besides taking forever to train, the classification also wasn't accurate most probably due to the wrong model. Just to get an idea of what I am dealing with, I tried finding a way to visualize the data somehow. And I decomposed the document matrix into a (m, 2) matrix using SVD with scikit-learn (wanted to try other methods, but they all crashed halfway).

So this is what the visualization looks like

So is it generally a good practice to reduce the dimension, and then only proceed with SVM? Also in this case what can I do to improve the accuracy of the classifier? I am trying to use sklearn.svm.SVC and kernel='poly', and degree=3 and it is taking a very long time to complete.

I'd recommend spending more time thinking about feature selection and representation for your SVM than worrying about the number of dimensions in your model. Generally speaking, SVM tends to be very robust to uninformative features (e.g., see Joachims, 1997, or Joachims, 1999 for a nice overview). In my experience, SVM doesn't often benefit as much from spending time on feature selection as do other algorithms, such as Naïve Bayes. The best gains I've seen with SVM tend to come from trying to encode your own expert knowledge about the classification domain in a way that is computationally accessible. Say for example that you're classifying publications on whether they contain information on protein-protein interactions. Something that is lost in the bag of words and tfidf vectorization approaches is the concept of proximity—two protein-related words occurring close to each other in a document are more likely to be found in documents dealing with protein-protein interaction. This can sometimes be achieved using $n$-gram modeling, but there are better alternatives that you'll only be able to use if you think about the characteristics of the types of documents you're trying to identify.

If you still want to try doing feature selection, I'd recommend $\chi^{2}$ (chi-squared) feature selection. To do this, you rank your features with respect to the objective

\begin{equation}
\chi^{2}(\textbf{D},t,c) = \sum_{e_{t}\in{0,1}}\sum_{e_{c}\in{0,1}}\frac{(N_{e_{t}e_{c}}-E_{e_{t}e_{c}})^{2}}{E_{e_{t}}e_{c}},
\end{equation}
where $N$ is the observed frequency of a term in $\textbf{D}$, $E$ is its expected frequency, and $t$ and $c$ denote term and class, respectively. You can easily compute this in sklearn, unless you want the educational experience of coding it yourself $\ddot\smile$


Tensorflow simple neural network has very bad performance in regression (curve fitting) problem I'm trying to implement a very simple one layered MLP for a toy regression problem with one variable (dimension = 1) and one target (dimension = 1). It's a simple curve fitting problem with zero noise.

Matlab - Deep Learning Toolbox

Using levenberg-marquardt backpropagation on a MLP with a single hidden layer with 100 neurons and hyperbolic tangent activation I got pretty decent performance with almost zero effort:

MSE = 7.18e-08

Here's a plot of the fitting:

Edit:
This is the working matlab code. Please note that the "feedforwardnet(100)" function only produces a network object with one hidden layer with 100 neurons and tanh activation and output layer with linear activation:

net = feedforwardnet(100);
net.trainParam.min_grad = 1e-25;
net.trainParam.max_fail = 50;
net.trainParam.epochs = 500;
%net1.trainParam.showWindow = false;
net.inputs{1,1}.processFcns = {};
net.outputs{1,2}.processFcns = {};
net = train(net,Train_Vars,Train_Target);
Test_Predictions = net(Test_Vars);
Accuracy = msemetric({Test_Predictions},{Test_Target});

Python - TensorFlow - Keras

With the same network settings I used in matlab there's almost no training. No matter how hard I try to tune the training parameters or switch the optimizer.

MSE = 0.12900154

I can obtain something better using RELU activations for the hidden layer but we're still far:

MSE = 0.0582045

This is the code I used in Python:

#  IMPORT LIBRARIES
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras

#  IMPORT DATASET FROM CSV FILE, SHUFFLE TRAINING SET
#  AND MAKE NUMPY ARRAY FOR TRAINING (DATA ARE ALREADY NORMALIZED)
dataset_path = "C:/Users/Rob/Desktop/Learning1.csv"
Learning_Dataset = pd.read_csv(dataset_path
                          , comment='\t',sep=","
                          ,skipinitialspace=False)
Learning_Dataset = Learning_Dataset.sample(frac = 1)  # SHUFFLING

test_dataset_path = "C:/Users/Rob/Desktop/Test1.csv"
Test_Dataset = pd.read_csv(test_dataset_path
                          , comment='\t',sep=","
                          ,skipinitialspace=False)

Learning_Target = Learning_Dataset.pop('Target')
Test_Target = Test_Dataset.pop('Target')

Learning_Dataset = np.array(Learning_Dataset,dtype = "float32")
Test_Dataset = np.array(Test_Dataset,dtype = "float32")
Learning_Target = np.array(Learning_Target,dtype = "float32")
Test_Target = np.array(Test_Target,dtype = "float32")

#  DEFINE SIMPLE MLP MODEL
inputs = tf.keras.layers.Input(shape=(1,))
x = tf.keras.layers.Dense(100, activation='relu')(inputs)
y = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs=inputs, outputs=y)

#  TRAIN MODEL
opt = tf.keras.optimizers.RMSprop(learning_rate = 0.001,
                                  rho = 0.9,
                                  momentum = 0.0,
                                  epsilon = 1e-07,
                                  centered = False)
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=100)
model.compile(optimizer = opt,
              loss = 'mse',
              metrics = ['mse'])

model.fit(Learning_Dataset,
          Learning_Target,
          epochs=500,
          validation_split = 0.2,
          verbose=0,
          callbacks=[early_stop],
          shuffle = False,
          batch_size = 100)

#  INFERENCE AND CHECK ACCURACY
Predictions = model.predict(Test_Dataset)
Predictions = Predictions.reshape(10000)

print(np.square(np.subtract(Test_Target,Predictions)).mean()) #  MSE

plt.plot(Test_Dataset,Test_Target,'o',Test_Dataset,Predictions,'o')
plt.legend(('Target','Model Prediction'))
plt.show()

What am i doing wrong?

Thanks

I am not sure which network training function is your MATLAB feedforwardnet is using. Also, most likely the activation function in your MATLAB net is 'tansig'. Try using 'tanh' activation function in your python/keras code.


Image classification on stock data I am  trying to classify stock images into 4 categories:

Category 1: Image patterns which gives more than 10% profit next 3
day.
Category 2: Image patterns which gives less than 10% profit but
greater than 0% next 3 day.
Category 3: Image patterns which gives more than -0.01% but greater
than -10% loss next 3 day.
Category 4: Image patterns which gives more than -10% loss next 3 day

I have divided these Images in to 4 category, below are the Images from category 1 and category 2.

Image contains 10 day stock movement, and classification category are generate from next 3 day stock data.

Red Bar is volume Traded of that stock.

Pink line is RSI(14) (Relative strength Index)

Yellow line is stock price.

Blue and Green line are moving average 10, 20 respectively.

I know by seeing the image it is hard to differentiate between them but this is what I am trying to find out can I classify it using CNN.

My CNN:

  (layer1): Sequential(
    (0): Conv2d(3, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (layer2): Sequential(
    (0): Conv2d(64, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (layer3): Sequential(
    (0): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (drop_out): Dropout(p=0.5, inplace=False)
  (fc1): Linear(in_features=65536, out_features=1024, bias=True)
  (fc2): Linear(in_features=1024, out_features=512, bias=True)
  (fc3): Linear(in_features=512, out_features=128, bias=True)
  (fc4): Linear(in_features=128, out_features=4, bias=True)

Optimizer: Adam, Loss: CrossEntropyLoss.

But the problem is CNN ends up classifying same class for each image to obtain 25% accuracy.

Am I working in wrong direction, should I work on imporving data or try other technique like LSTM?

I like this question!

Potential pitfalls:

1. Architecture is just not sophisticated enough to catch all nuances in the time-series image. Take a look at here what they did for earthquake-time-eries-as-images approach or take a look at this paper. You could take some guidelines from there, but you notice right away that the architecture is bigger. Or this one

2. Data, you say its predicting only one class, but what part of the image is it focusing on. Use shap values for images to inspect. I suspect following: it will be some trend that is consistent only for one class, hence you can think about pre-processing steps that can point to properties on images of other classes. Or even add some hand-crafted numeric features. In any case increase your dataset if you havent already, you want to have enough information to catch all nuances.


How to determine irrelevant data in binary classification? Suppose I was looking at social media to find users' intents on trading stocks. I might have a binary classification model that predicted "buy" and "sell". However, it's obvious that most social media posts mentioning a company are not related to buying or selling its stock. Even if I was to look specifically at places in the Internet where the main topic of discussion is buying and selling stocks, there would be handfuls of posts that were in a sense "off-topic" (e.g. "I applied to Microsoft today." or "What does everyone here think about Alphabet?")

My question is, how would one go about recognizing when a social media post does not suggest a user would buy or sell the stock. I had three quick ideas:

Create rules that would be able to differentiate relevant from irrelevant posts
Create a second binary classifier, that differentiates between relevant and irrelevant posts, and then uses the main classifier on only the relevant posts
Change the binary classifier into a classifier that can detect buy, sell, and off-topic documents.

Is there a customary approach to this problem?

I agree that you could reasonably classify the stock that it relates to, and whether it concerns a trading intent or not, separately. Presumably, trading intent looks similar across stocks, so the two are not related much.

Your second and third ideas therefore sound relevant to me.

It's possible to consider a multi-task classifier too with deep learning architectures, one that tries to solve both at once. There might be small advantages to that, but it is more complex.


How do I use Machine Learning to analyze process statistics of automated test results? (I am newbieeee in machine learning)

At the end of every automated test (Python), I collect a lot of information from our product. The information includes CPU, Memory Usage, Disk Activity, Postfix Queue statistics, and various other stats.

I want to come up with a script/tool(Python) that leverages machine learning and automatically analyze the results for me, instead of me looking at the results manually every day. I want to train the tool on what acceptable levels are for various components like CPU, Memory, depending on product's specifications, and various inputs.

From what I read online, I think I need to implement supervised classification in my tool, build a model, train it for a few days. But what I don’t know is where to start. When I looked at Supervised learning, I see a bunch of stuff and don’t really know what is really relevant to me. I appreciate if I can get some suggestions on what I need to focus on and how to implement this.

All the data can be in memory before my original script completes, or I can get the data from a database to which my original script writes to.

I want to come up with a script/tool(Python) that leverages machine learning and automatically analyze the results for me, instead of me looking at the results manually every day.

This is the wrong approach. Choose your tools to solve tasks, not because they sound cool. Imagine a craftsman saying "I want to use a hammer". (However, machine learning might still suit to this task.)

  From what I read online, I think I need to [...] train it for a few days.

Depends on how much data, how much features, how much computing power and which algorithm you use. In many cases you can train a model within seconds on a normal computer.

The real question

The problem: You have some characteristics (features) like CPU usage, memory usage, ... for every automated test. For some instances (meaning a test and the associated features) you have the binary information "normal", "abnormal". Now, when you get the characteristics of a new test, you want to automatically decide if it was normal or abnormal.

This is a binary classification problem. If you have the label ("normal", "abnormal") for all instances, you can apply supervised learning methods. If you have the label only for some instances, it is semi-supervised learning. If you don't have any labels it is unsupervised.

The Answer

I assume you are in a supervised learning case. Then, I would suggest to use a decision tree (see sklearn.tree.DecisionTreeClassifier - sklearn is pretty easy to use and has many standard algorithms implemented) for this decision making. It basically automatically generates many nested if/else statements to get to the decision. Hence it is interpretable and you could manually adjust or improve it.


How to start analysing and modelling data for an academic project, when not a statistician or data scientist I have collected data for a PhD thesis, and need help understanding how to build a road map to do analytical and statistical analysis. The PhD is not itself in statistics or machine learning, but I would like to understand what are the steps and type of analysis that I have to follow for analysing data for an advanced degree? In general, how should I approach such a problem?
In the data I have collected, there are 623 observations including one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal) that are defined based on the researcher experience and literature review.
I considered planning to do several regression analysis to predict the dependent variable and study the effective factors (if they are positive, negative, and their magnitude) on it. I've tried multiple linear regression including different transformation on independent variables. On the other hand, I'm not sure if I should study each independent variables through the time and  forecast their values in the time horizon?
Here are the steps in my mind so far:

Plotting the scatter plots of different independent variables vs dependent variable to define outliers and check if the model is linear also with respect to coefficients

Removing the potential outliers

Splitting the data into two data sets to build the model and validate it after that.
If the model is linear then:

Performing the multiple linear regression

Performing the multiple linear regression including different transformations to enhance the model

Validating the model

Doing the quantile regression

Doing supervised learning machine etc.

If the model is not linear, I may instead need to use non-linear statistical techniques.
Any feedback would be highly appreciated. My goal is to build a clear and robust road map for this part of the work.

Typically, quantitative analysis is planned and performed, based on research study's goals. Focusing on research goals and corresponding research questions, researcher would propose a model (or several models) and a set of hypotheses, associated with the model(s). Model(s) and its/their elements' types usually dictate (suggest) quantitative approaches that would make sense in a particular situation. For example, if your model includes latent variables, you would have to use appropriate methods to perform data analysis (i.e., structural equation modeling). Otherwise, you can apply a variety of other methods, such as time series analysis or, as you mentioned, multiple regression and machine learning. For more details on research workflow with latent variables, also see section #3 in my relevant answer.

One last note: whatever methods you use, pay enough attention to the following two very important aspects - performing full-scale exploratory data analysis (EDA) (see my relevant answer) and trying to design and perform your analysis in the reproducible research fashion (see my relevant answer).


How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm? There's a popular solution to the CartPole game using Keras and Deep Q-Learning:
https://keon.github.io/deep-q-learning/

But there's a line of code that's confusing, this same question has been asked in the same article and many people are confused but there's no a complete answer.

They are basically creating a main network but also a target network to try to approximate the Q function.

In this part of the code they are replaying from the buffer to train the target network:

# Sample minibatch from the memory
minibatch = random.sample(self.memory, batch_size)

# Extract informations from each memory
for state, action, reward, next_state, done in minibatch:

# if done, make our target reward
target = reward

if not done:
  # predict the future discounted reward
  target = reward + self.gamma * \
           np.amax(self.model.predict(next_state)[0])

# make the agent to approximately map
# the current state to future discounted reward
# We'll call that target_f
target_f = self.model.predict(state)
target_f[0][action] = target

# Train the Neural Net with the state and target_f
self.model.fit(state, target_f, epochs=1, verbose=0)

What I can't understand is this line:

target_f[0][action] = target

In terms of code, the predict function is returning a numpy array of arrays, like this one for example:

[[-0.2635497   0.03837822]]

Writing target_f[0] to access the first predicted action is understandable, but why are they using the [action]?

Thank you very much for the help!

Hi David and welcome to the community! I think that the [0] is for accessing the array of actions as it is in double brackets. After this you need the action index to update appropriately.

The Q network would have as many outputs as the actions available (in your case 2). Then you want to update the weights of the network in the output layer that is responsible for estimating the Q that corresponds to the action selected.  So it seems that you have received a $r(s,action)$ and thus you assign it as target in order to use it for the MSE($Q(s,action)$, $r(s,action)$) between estimation and real reward.

The weights of the other outputs cannot be updated by using the collected $r(s,action)$ as the Q-learning update equation would be wrong so the target remains the same as the prediction (resulting a MSE=0). Think that each of the network's outputs is responsible for estimating the return by performing a particular action, at a particular state. For example the first output is responsible for estimating the return (given input state) for action=left ($Q(s,left)$) and the other for action=right ($Q(s,right)$). You train each network head (output layer) with MSE between real reward and estimation. Each time the reward that is sampled from the environment would be the result of a selected action. Thus you update only the corresponding head by assigning as target the reward sample.


How is the number of grid cells in YOLO determined? I'm now learning YOLO but I don't understand how the number of grid cells is determined. Some article like this one uses 13 x 13 while others use 19 x 19 (e.g. Andrew Ng's course in Coursera). It seems that the height and width should always be the same, but how is it determined?

Is there any general guideline regarding how many grid cells should be picked up over the others?

As far as I can tell, there is no specific rule. It will depend in part on how crowded your scene will become with items that you want to detect and locate separately. Creating a high granularity grid increases computational cost for training, and there is no reason to do so if it would only cover additional cases that are much rarer than the detection accuracy that the base algorithm achieves.

The choice can be driven by the ground truth data. The ground truth for YOLO needs to be expressed in the form of grid locations with classes and bounding rectangle sizes. If you don't find any training examples where you want to label two items with their centre inside the same grid square, then this is a good indication that your grid size is fine-grained enough.

Even if there are one or two examples with a clash like this, you may be able justify labelling just one item in the ground truth and be OK with a resulting model that is not able to cope with close overlap between two separate objects. And even with smaller grid squares, YOLO may not be able to learn to separate the objects, if such an overlap only occurs rarely.

I would expect a simple rule of diminishing returns applies. As datasets grow larger, and object detection can be trained on more powerful computers, we may see state of the art models still using YOLO but with more grid points.


svm optimization problem Suppose we have the dataset:

{(3,1),(3-1),(6,1),(6,-1)} {(1,0),(0,1),(0,-1),(-1,0)}
the first set represent the positive label, and de second the negative.

I want manually find the support vectors, solve the equations and determine the coefficients W and bias
w0 of the separating hyperplane.

I know that the SVs are {s1=(1,0),s2=(3,1),s3=(3-1)}

after i wrote a system of equations:

α1k(s1,s1) + α2k(s2,s1) + α3(s3,s1) = -1

α1k(s1,s2) + α2k(s2,s2) + α3(s3,s2) = -1

α1k(s1,s3) + α2k(s2,s3) + α3(s3,s3) = -1

Where k(x,y) is the dot product xy

the resolution of this system is impossible, can someone tell me where I'm wrong?

I think that this system of equation is incorrect. If you know that (3, -1), (3, 1) and (1, 0) are support vectors then you need to solve the next system:

3*w1 - 1*w2 - w0 = ±1
3*w1 + 1*w2 - w0 = ±1
1*w1 + 0*w2 - w0 = ±1

Support vectors on the same side of the separation line have the same sign on the right side of the equation (same sign here: ±1). In this example, (1, 0) locates on one side, while (3, -1) and (3, 1) are on the other side. Thus, the system will be the following:

3*w1 - 1*w2 - w0 = 1
3*w1 + 1*w2 - w0 = 1
1*w1 + 0*w2 - w0 = -1

The solution is: w0 = 2, w1 = 1, w2 = 0. You can plot all the points and check that this solution is correct.


Genetic algorithms: what connection to support vector machine / naive bayes I found the following list of seven classifiers:

Linear Classifiers: Logistic Regression, Naive Bayes Classifier
Nearest Neighbor
Support Vector Machines
Decision Trees
Boosted Trees
Random Forest
Neural Network

What are genetic algorithms, and why aren't they considered as part of the seven classifiers?

A fast search on Google for the term "Genetic Algorithms" will return you this answer:

"A genetic algorithm is a search heuristic that is inspired by Charles Darwin's theory of natural evolution"

So, the term is usually associated with a "search heuristic", for example the search for a local optima.

By that common definition: Genetic Algorithms are not listed among these 7 classifiers since it is not a classifier, it would be in a list of optimization methods like Gradient Descent (and it's variation), Grid Search, Random Search, etc.


Weights for unbalanced classification I'm working with an unbalanced classification problem, in which the target variable contains:

np.bincount(y_train)
array([151953,  13273])

i.e. 151953  zeroes and 13273 ones.

To deal with this I'm using XGBoost's weight parameter when defining the DMatrix:

dtrain = xgb.DMatrix(data=x_train,
                     label=y_train,
                     weight=weights)

For the weights I've been using:

bc = np.bincount(y_train)
n_samples = bc.sum()
n_classes = len(bc)
weights = n_samples / (n_classes * bc)
w = weights[y_train.values]

Where weightsis array([0.54367469, 6.22413923]), and with the last line of code I'm just indexing it using the binary values in y_train. This seems like the correct approach to define the weights, since it represents the actual ratio between the amount of values of one class vs the other. However this seems to be favoring the minoritary class, which can be seen by inspecting the confusion matrix:

array([[18881, 19195],
       [  657,  2574]])

So just by trying out different weight values, I've realized that with a fairly close weight ratio, specifically array([1, 7]), the results seem much more reasonable:

array([[23020, 15056],
       [  837,  2394]])

So my question is:

Why using the actual weights of each class yielding poor metrics?
Which is the right way to set the weights for an unbalanced problem?

Depending on your choice of accuracy metric, you'll find that different balancing ratios give the optimum value of the metric. To see why this is true, consider optimizing precision alone vs. optimizing recall alone. Precision is optimized (=1.0) when there are no false positives. Upweighting negative data reduces the positive rate, and therefore the false positive weight. So if you just want to optimize precision, give the positive data zero weight! You'll always predict negative labels and the precision will be ideal. Likewise, for only optimizing recall, give the negative data zero weight - you'll always get the ideal value of recall. These extreme cases are silly for real-world applications, but they do show that your "best" balancing ratio depends on your metric.

As you're probably aware, metrics like AUC and F1 try to compromise between precision and recall. In the absence of prior information, people often try to choose "equal balance" between precision and recall, as implemented in AUC. Since AUC is relatively insensitive to data balance, 1:1 data balancing is generally appropriate. However, in real life you may care more about precision than recall, or vice versa. So, you do need to select your metric in advance, depending on the problem you're solving. Then keep your metric fixed, vary your data balance, and look at your trained model performance on realistic test datasets. Then you can see whether your model is making the optimum predictions, from the point of view of your chosen metric and your real-world dataset.


Word list as a baseline for measuring a classifier's performance? I am working on a simple Naive Bayes classifier that categorizes text messages as either "positive" or "negative". I was told that the simplest baseline to measure the classifier's performance would be to compare it against a word list.
However, I have no clue as to how this might be helpful.
How can a word list be a baseline for a classifier?
P.S. I am new to machine learning and data science, so please don't hate on my question.

Features of your naive bayes classifier is most likely a bag of words where each feature represents a single word.
The naive bayes classifier also assumes that each feature (or word) contributes independently to the final prediction.
So if your sentence is composed of a single word - you get an idea of how the model reacts to that word.


How to do time-series prediction in the browser I've been trying to find a simple tutorial to take me through using TensorFlow.js to do predictions on a series of values, but I haven't been able to find something simple and based in JS.

I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.  I have over a million values in this set, and they're just simple values in sequence.

Any direction or helpful and simple tutorials would be really appreciated.

I've got a long list of values (to two decimal places), and I'd like to train on a subset of these and test to see if there's a way to predict the next value.

The first thing comes to mind is sequence models, I think this will be useful for you, see Real-time Performance RNN in the Browser


ValueError: Cannot feed value of shape (3,) for Tensor 'X:0', which has shape '(1, 3)' I am trying to do a multivariate linear regression and I am having some issues. Namely, I am getting the following error:

ValueError: Cannot feed value of shape (3,) for Tensor 'X:0', which has shape '(1, 3)'

I have 3 feature variables, which I call trainX and 1 label, which I call trainY. Their shapes are the following (they are numpy arrays):

trainX.shape:
(2500, 3)
trainY.shape:
(2500,)

The following piece of code defines the tensors that I use to compute the model:

X = tf.compat.v1.placeholder("float", [1, 3], name="X")
Y = tf.compat.v1.placeholder("float", [1], name="Y")

W = tf.Variable(tf.zeros([3, 1]), name="W")
b = tf.Variable(tf.zeros([1]), name="b")

I calculate the predicted label and the cost function and the optimizer by doing:

predicted_y = tf.matmul(X, W) + b
cost = tf.reduce_sum(tf.pow(predicted_y-Y, 2)) / (2 * n)
optimizer = tf.compat.v1.train.GradientDescentOptimizer(learning_rate).minimize(cost)

I am getting the error in the tensor-flow session, namely in the following piece of code:

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(training_epochs):
        for (_x, _y) in zip(trainX, trainY):
            sess.run(optimizer, feed_dict={X: _x, Y: _y})
        if (epoch + 1) % 100 == 0:
            c = sess.run(cost, feed_dict={X: trainX, Y: trainY})
            print("Epoch", (epoch + 1), ": cost =", c, "W =", sess.run(W), "b =", sess.run(b))
    # Storing necessary values to be used outside the Session
    training_cost = sess.run(cost, feed_dict={X: trainX, Y: trainY})
    weight = sess.run(W)
    bias = sess.run(b)

Any help would be greatly appreciated.

trainX has shape (2500, 3), so when you iterate over trainX you get values with shape (3,).  To match the shape of your placeholder "X", you need them to have shape (1, 3).  This can be accomplished with numpy.reshape:

# do this after loading trainX
trainX = trainX.reshape((-1, 1, 3))
# new shape: (2500, 1, 3)


How to make model for Multivariate timeseries using tensorflow probability structural model? I have done modelling for Univariate timeseries but while using multivariate time series ( independent features) not able to achieve result. please let me know if anybody have used .

please find reference link

You just need to ensure that the model you forecast with has a design matrix covering both the observed and forecasted timesteps. That is, you'd build a model including a component along the lines of

sts.LinearRegression(
  design_matrix=tf.concat([temperature_for_observed_timesteps,
                           temperature_for_forecast_timesteps], axis=-2),
  name='temperature_effect')

(ignoring any centering and reshaping logic) and then pass that model to the forecast method.

If you don't have access to future values of the external regressor when you first build the model, a useful pattern is to encapsulate model building in a method def build_model(observed_time_series, design_matrix) that returns a StructuralTimeSeries model object. Then you can build an initial model with just the observed time steps in order to fit parameters, and then remake the model later once you have the regressors for the forecast steps on hand.


predict_classes() returning only 0 or 1 for multiclass image classification I am trying to build a multi class image classifier but the only returns 0 or 1 .

Why is it not returning "Rock" , "Paper" , "Scissor" ?
and why only 0 and 1 but not 2?

CODE:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten
import numpy as np

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical',
    classes = ["Rock", "Paper" , "Scissor"]
    )

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)
filenames = validation_generator.filenames
nb_samples = len(filenames)
desired_batch_size = 1

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_classes(validation_generator, batch_size = None)

print(predict)

Output:

[0 1 1 ... 1 0 1]

I got the answer so I'm posting it for future reference for others:

The 3 classes defined in

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

in class_mode should be Rock , Paper , "Scissors"
those were the names of the folders i create but i made a mistake and spelled "Scissors" wrong in the code.

Thanks a lot to everyone who commented!


What is difference between detrend and normalization? matlab function detrend subtracts the mean from data. If data contains several data columns, detrend treats each data column separately.

One of the normalization technique is subtracting the mean and dividing it by standard deviation.

Since the normalization already subtract the mean from the data, in such case, is it essential to perform (before or after) detrend operation?

What is the significance of each operation?

You detrend data in order to get rid of the linear trend in your data, which might cause spurious regression, misleading evidence that there is some relation between variables.

Normalization means adjusting values measured on different scales to a notionally common scale.
Example you give: Standard score $\frac{X - \mu}{\sigma}$ is just one possible method of normalization . It allows you to see where value lies in comparison to mean.

In my experience you would only detrend data in order to create time series model. On the other hand, normalization is frequently used to compare previously not comparable statistics or detect anomalies (in case of standard score). As a result, usage really depends on the use case and you rarely use both at the same time.

Wikipedia is good enough source to clarify this unambiguity.


Make the ML model pass the Load Test I have made a model to predict some values. In real time, it works perfectly fine and yields prediction within a fraction of a second. The algorithm I am using is PassiveAggresiveClassifier to predict on multiple classes.

However, when I load tested it, it lagged in performance. I tried on passing 500 requests at a time and it essentially bogged down.

Is there a way to enhance this aspect of a model program-wise? Can python solve this? Is there another way?

Thank you.

Assuming the server is as at least as powerful as the first local machine, you should obtain the same time or shorter duration by request. If not, there's a problem worth investigating.

Then to improve further it could be useful to design the server side architecture so that:

the model is pre-loaded in memory, not read from file every time (huge bottleneck).
there are multiple instances of the program running in parallel so as to distribute the load.


Traditional ML Model or Deep Learning for ~200-300 samples? Good morning all!

I'm working on a resume parser that is integrated with an RPA (robotic process automation) platform.

The robot has OCR to extract text from a PDF resume, and it supplies the tokenized contents as well as the X and Y coordinates of each word.

My goal is to create a predictive model to identify key sections of a resume, and then use traditional resume parsing for the remainder of the workflow.

Some of the features included in the model are as follows:

| Word | X_norm | Y_norm | Length_Word | Num_Word_per_Line | Special Char? | Contains numbers? |

Where normalization is min-max.
We are also working on getting font size and normalized color (i.e boldness).

My gut feeling is that traditional ML model might be better as we only have the resources to label 200-300 resumes, not thousands. But I wanted to be open to deep learning (something I don't have much experience with). The RPA workflow is such that a Resume Specialist can give feedback to the model in real time, and something that can improve itself on the fly might be better for this use case.

Can traditional ML models also efficiently self-improve?

Thanks for your time!

The less data you have, the less complex your model can be. Otherwise you will overfit your data. There is not really a good way for me to judge what model is appropriate for you without knowing a lot about your dataset, but I doubt you will get anything sensible from 200 data points with a deep learning model. Try some simpler models like bag-of-words and see whether you overfit. If so, you will just have to collect more data. If not, try models of increasing complexity and see how accurate you can get before overfitting sets in.


Pig latin code error While running the below pig script I am getting error in line4: If it is GROUP then I am getting error. If I change from GROUP TO group in line4, then the script is running. What is the difference between group and GROUP.

LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );

WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;

WORDSGROUPED = GROUP WORDS BY ZIPS;

WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);

WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;

DUMP WORDSSORT;

When we do grouping of the data, Pig creates a new key named "group" and puts all the tuple matching that key into a bag and associates the bag with the key. So after the group operation schema of the grouped data will be something like

raw = load '$input' using PigStorage('\u0001') as (id1:int, name:chararray);
groupdata1 = group raw by (id1,name);
describe groupdata1;
{group: (id1: int,name: chararray),raw: {(id1: int,name: chararray)}}

The 'GROUP' in line 4 you are trying to access is one of the attribute of the schema from the last statement. These attribute name are case sensitive. It will produce the error saying it doesn't exist in the schema. So you need to use 'group' only to access it.