How to cluster categorical and numerical data in the same dataset? I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some unsupervised learning algorithm (K-Means clustering algorithm is currently used).

For example the Gender,None,Low,Medium,High,Breakfast,Lunch and Dinner columns are represented in binaries. eg: Gender represents 1 for male and 0 for female. While the rest of the above mentioned columns represents 0 for unavailibilty and 1 for availability. The Meal and the Exercise columns are also categorical but are not binary. For example in the Meal if it is breakfast then 1, lunch = 2, and dinner = 3. So how can we use this type of mixed dataset for clustering? Please ignore the Event column since it is the target column.

Further should we need to normalize the rest of the numerical data before adding to any sort of unsupervised learning algorithm? And how can we deal with those different types of categorical data? Your guidance is greatly appreciated.

Thanks.

You can use one-hot-encoding for all categorical features. Then normalizing numerical features (as one-hot is 0 and 1 then maybe normalizing your numerical data to [0,1] will bean intuitive starting poit). Then apply a dimensionality reduction technique as probably you will produce a sparse matrix with maybe considerable number of dimensions. Then do your clustering (try at least two different one. I would say k-means and DBSCAN).

A nice intuitive replacement to one-hot-encoding is to replace values with their fraction. For example in column gender you have 60 male and 40 female, then replace male with 0.6 and female with 0.4. The only drawback is when the number of categories are equal in a column! You need to be careful about those.


Preparing multiple training time-series for Keras LSTM regression model training I have training data organised in a numpy array in which:
* column is feature - last one is the target,
* every row is one observation.

The thing is that this 2D array consists of around 15 concatenated series of data of unequal length.I would like to use it for LSTM network training. Is it possible to train LSTM network in Keras using series of unequal length? If so, how am I supposed to transform my 2D training data array into valid 3D array?

The standard way to treat unequal lengths is to pad them.

However, if you have very long sequences, it is often handled by splitting them into short sequences - in this case, it is up to you, what sequence length to choose.

Then you will have to decide, whether you need stateful LSTM or no, but it is out of the scope of this question.


Downsampling and class ratios My target variable is whether an application is accepted or not.  It is a highly imbalanced target with 98.5% of applications accepted.  I am unclear about the concept of downsampling.  If I were to downsample the applications, do I have to maintain the current ratio of accepted to rejected applications while lowering the total number of applications in the training data or can I change the ratio of accepted to rejected apps to say 50% accepted, 50% rejected?  What are the benefits of doing either of these approaches?  Help is welcome.  Thanks!

Downsampling means you sample from the majority class (the 98.5%) to reduce the imbalance between majority and minority class. If you keep the ratio constant you simply reduce your number of trainings examples. This doesn't make sense. However, you don't have to sample down to a ratio of 50:50. If you have a ratio of 98:2, you can sample to 80:2 instead of 2:2.

The main goal of downsampling (and upsampling) is to increase the discriminative power between the two classes. Ideally, you would have a classifier that outputs a decision surface that is not simply binary (e.g. logistic regression (where you don't have to select a cut-off point of 0.5)) but gives you a continuous decision value. You can then order the data and set a decision threshold that gives you the best outcome.

Since downsampling (or upsampling) changes your training distribution from your true distribution, you only want to downsample (or upsample) so much that your classifier can start discriminating between the two classes. You then fine-tune the results by selecting an appropriate decision threshold. (Also, in my experience, upsampling is often a better choice over downsampling.)


Using cloud to create CNN model Are there any differences in the results of training the deep learning algorithm (CNN) using a device with a GPU and using any device with a cloud service such as AWS and Floydhub?

I think you might be a little confused with the terminology here.

Whether you train locally on your own computer or in the cloud (via AWS, Microsoft Azure, GoogleCloud, FloydHub, ...) there are only two three options:

using a CPU $\rightarrow$ whatever is in your computer!
using a GPU $\rightarrow$ probably an Nvidia card supportung CUDA
using a TPU $\rightarrow$ a type of GPU to accelerate Machine Learning)

Your deep learning framework of choice (Pytorch, Tensorflow, CNTK, ...) will make use of the hardware with which you provide it.

All cloud services will offer instances with or without GPUs (they will almost vertainly have a CPU).

Training on a GPU is of course much faster thatn a CPU, as it can parallelise many of the operations that are performed (especially for CNNs, compared to e.g. LSTMs). It is of course possible to offload certain parts on training onto each of them, so you would be using both together.

A GPU in your computer at home and GPU in a big server-rack in an AWS data-center would both perform as equally as can be hoped for. The only differences will be cost, but that is a different discussion.


How to deal with class imbalance problem in natural language processing? I am doing a NLP binary classification task, using Bert + softmax layer on top of it. The network uses cross-entropy loss.
When the ratio of positive class to negative class is 1:1 or 1:2, the model performs well on correctly classifying both classes(accuracy for each class is around 0.92).
When the ratio is 1:3 to 1:10, the model performs poorly as expected. When the ratio is 1:10, the model has a 0.98 accuracy on correctly classifying negative class instances, but only has a 0.80 accuracy on correctly classifying positive class instances.
The behavior is as expected as the model turns to classify most/all instances toward negative class since the ratio of positive class to negative class is 1:10.
I just want to ask what's the recommended way for handling this kind of class imbalance problem in natural language processing specifically?
I saw someone suggests to change loss function, or perform up/down sampling, but most of them are targetting computer vision class imbalance problem.

Disclaimer: this answer might be disappointing ;)
In general my advice would be to carefully analyze the errors that the model makes and try to make the model deal with these cases better. This can involve many different strategies depending on the task and the data. Here are a few general directions to consider:

Most of the time the imbalance is not the real problem, the real problem is why the model can't differentiate between the classes. Even in case of extreme imbalance if the classes are easy to discriminate a model can perform very well. The imbalance only causes the model to assign the majority class when it doesn't have enough indication to decide, so it resorts to the conservative choice.
If the minority class is really small in absolute terms it's likely that there's not enough language diversity in the positive instances (data sparsity). This will usually cause a kind of overfitting which can be hidden by the fact that the model almost always assigns the majority class. In this case the goal should be to treat the overfitting, so the first direction is to simplify the model and/or the data representation.
Sometimes it can make sense to consider alternative ML designs: in a regular classification problem a model relies on the distribution of the classes, by principle. Some alternative approaches might not as influenced by the distribution, for example one-class classification. Of course it's not suited for every problem.

Overall my old-school advice is not to rely too much on technical answers such as resampling methods. It can make sense sometimes, but it shouldn't be used as some kind if magical answer instead of careful analysis.


Info obtained from a confusion matrix I am new to data science and I am trying to understand the use/importance of accuracy, precision, recall, sensitivity and f1-score when I have a confusion matrix.

I know how to compute all of them but I cannot really understand which of them to use each time.

Could you give examples where for instance precision is a better metric that recall or where the f1-score gives essential information that I cannot get from the other terms ? In other words, in which cases should I use each of the aforementioned terms ?

First, let's be clear about the fact that all these measures are only for evaluating binary classification tasks.

The way to understand the differences is to look at examples where the number of instances is (very) different in the two classes, either the true classes (gold) or predicted classes.

For instance imagine a task to detect cities names among the words in a text. It's not very common, so in your test set you may have 1000 words, only 5 of them are cities names (positive). Now imagine two systems:

Dummy system A which always says "negative" for any word
Real system B (e.g. which works with a dictionary of cities names). Let's say that B misses 2 real cities and mistakenly identifies 8 other words as cities.

System A gets an accuracy of 995/1000 = 99.5%, even though it does nothing. System B has 990/1000=99.0%. It looks like A is better, that's why accuracy rarely gives the full picture.

Precision represents how correct a system is in its positive predictions: system A always says negative so it has 0% precision. System B has 3/11 = 27%.

Recall represents the proportion of true positive instances which are retrieved by a system: system A doesn't retrieve anything so it has 0% recall. System B has
3/5 = 60%.

F1-score is a way to have a single value which represents the harmonic mean of the precision and recall. It's used as a "summary" of these two values, which is convenient when one needs to order different systems by their performance.

The choice of an evaluation measure depends on the task: for instance, if predicting a FN has life-threatening consequences (e.g. cancer detection), then recall is crucial. If on the contrary it's very important to avoid FP cases, then precision makes more sense (say for instance if an automatic missile system would mistaken identify a commercial flight as a threat).
The most common case though is certainly F1-score (or more generally F$\alpha$-score), which is suited to most binary classification tasks.


Feature has a pattern in relation to class but does not enhance classifier to predict class I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the model accuracy. I want to know the reason, I thought that maybe there is a different distribution in my training and testing data, that lead to this, and checked this hypothesis, the result was that distributions are different. In general case any idea is appreciated.

There are multiple things at play here.

First of all, if your training and testing data are different then it becomes difficult to learn to generalize, because it will learn the relationships in your training data and if those relationships are significantly different then it will learn the wrong things. It could be that the data density is just different in your testing set which is not as much of a problem.

Second of all, if the relationship that you see is not linear, your logistic regression might not be able to pick up the relationship due to the underlying linear assumption.

Third of all, maybe other features are highly (co)correlated and adding the new feature does not add any signal but does add the potential for fitting additional noise, thus explaining the performance reduction.


i'm using GridSearchCV to find parameter C for SVC() classifier present in sklearn.svm . I'm not getting the optimal result desired this is a screenshot of my code. i used abc.best_estimator_ (my GridSearchCV model) to find out best results. As you can see grid has values of C=1 and C=100 along with other values. abc.best_estimator_ says C=1 is the best value. For cross checking i tried using different values of c and here i'm getting a better score for C=100. I was getting similar results while finding gamma also, but later on i commented out gamma so as to focus on C only.
Any idea why is this happening? Am i doing something wrong?

You didn't do any cross validation of data in GridSearchCV as you didn't mention any cv parameter. So what GridSearchCV did is it found the best estimator on the train set which possibly made the model overfitted on training data. You should use cross validation to find the best best estimator.
abc = GridSearchCV(clf,params,cv=5)


Max pooling has no parameters and therefore doesn't affect the backpropagation? I feel this is a question that has a lot of variations already posted but it doesn't exactly answer my question. I understand the concept of max pooling and also the concept of backpropagation. What i cannot understand is that why is it said that max pooling doesn't affect the process of backpropagation? Is it because max-pooling "doesn't have any parameters"? Also is this statement - max pooling has no parameters- right? Why or why not?

Pooling layers does not have parameters which affect the back propagation. Backpropagation is an algorithm to efficiently implement gradient descent in a neural network by using the chain rule and propagating the error from the final layers to the early ones. The parameters that affect backprop are often called trainable variables (in Tensorflow), and they are the weights and biases of the convolutional filters. Adding pooling layers does not change the number of them.

Pooling layers are a way of performing downsampling, and they are used for the following main reasons:

To decrease the computational load of the network: smaller outputs means less FLOPS
To save memory during training: less memory allocation in the GPUs
To increase the receptive field of the neurons in successive layers. You can check this link about this topic: https://medium.com/@santi.pdp/receptive-fields-in-convolutional-neural-networks-6368a699d838
Adding transalation-invariance to the feature maps: when pooling, small translations of a feature will result in this feature located in the same position in the following pooled layer

What is true is that pooling layers may have a great impact in the learning capabilities of a network


How to choose dimensionality of the Dense layer in LSTM? I have a task of multi-label text classification. My dataset has 1369 classes:

# data shape
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
(54629, 500)
(23413, 500)
(54629, 1369)
(23413, 1369)

For this task, I've decided to use LSTM NN with the next parameters:

# define model
maxlen = 400
inp = Input(shape=(maxlen, ))
embed_size = 128
x = Embedding(max_features, embed_size)(inp)
x = LSTM(60, return_sequences=True,name='lstm_layer')(x)
x = GlobalMaxPool1D()(x)
x = Dropout(0.1)(x)
x = Dense(2000, activation="relu")(x)
x = Dropout(0.1)(x)
x = Dense(1369, activation="sigmoid")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy']
batch_size = 32
epochs = 2
model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)

Question: Are there any scientific methods for determining Dense and LSTM dimensionality (in my example, LSTM dimension=60, I Dense dimension=2000, and II Dense dimension=1369)?

If there are no scientific methods, maybe there are some heuristics or tips on how to do this with data with similar dimension.

I randomly chose these parameters. I would like to improve the accuracy of the model and correctly approach to solving similar problems.

This is a very broad and difficult question, because these parameters depend completely on the complexity of your dataset. There is (currently) no "scientific method" to determine these parameters. The best solution is to apply a form of hyperparameter optimization.

See this similar post for more answers.


Polynomial Kernel Parameters in SVMs In SVMs the polynomial kernel is defined as:

(scale * crossprod(x, y) + offset)^degree

How do the scale and offset parameters affect the model and what range should they be in? (intuitively please)

Are the scale and offset for numeric stability only (that's what it looks like to me), or do they influence prediction accuracy as well?

Can good values for scale and offset be calculated/estimated when the data is known or is a grid search required? The caret package always sets the offset to 1, but it does a grid search for scale. (Why) is an offset of 1 a good value?

Thanks

PS.: Wikipedia didn't really help my understanding:

  For degree-d polynomials, the polynomial kernel is defined as

  where x and y are vectors in the input space, i.e. vectors of features
  computed from training or test samples,  is a constant trading
  off the influence of higher-order versus lower-order terms in the
  polynomial. When , the kernel is called homogeneous.(A further
  generalized polykernel divides  by a user-specified scalar
  parameter .)

Neither did ?polydot's explanation in R's help system:

  scale: The scaling parameter of the polynomial and tangent kernel is a
  convenient way of normalizing patterns (<-!?) without the need to modify the
  data itself

  offset: The offset used in a polynomial or hyperbolic tangent kernel (<- lol thanks)

The offset parameter is sometimes called "bias" in classification tasks, and its intuitive understanding doesn't have to do with what kind of kernel is used. It is basically used to compensate for feature vectors that are not centered around 0.

I will try to intuitively explain what the bias does with a toy example. Let's say you have some feature vector x who's parameters are always negative. The set of weights w you use in your SVM (let's say linear, just for clarity) will perhaps transform the features into the range [0, 1] - so they will always be negative. But, those elements that belong to class 1 fall in the range [0, 0.5], and the ones that are from class 2 fall into the range [0.5, 1].

To classify into a class, the SVM uses 0 as the threshold breakpoint - if greater than 0, it is an element of class 1, if less than 0, it is an element of class -1. But in this case, all the elements will be classified into class 2. However, with a bias of 0.5 (in this linear case), they will be classified correctly.

This geometric interpretation doesn't quite work for more complex kernels, but the idea is the same: the bias term attempts to compensate for features that are not centered around zero. In practice, if the features are centered around zero the bias term isn't always needed. To get around the bias issue, you can either:

center your features and forget about the bias term (doesn't always work)
augment your feature data to include a 1 at the beginning. i.e.:

[feat1 feat2 feat3] --> [feat1 feat2 feat3 1]

for all of your feature vectors. Then the bias term will just be learned as another SVM weight parameter.
Learn your weights, then calculate the bias based on the optimal prediction rule:

That is, given all the support vectors $S$, the learned weights $\alpha_i$, the training class $y^{(i)}$, training features $x^{(i)}$, and features-to-use-for-prediction $x$, the predicted class will be $y(x)$. The optimal $w_0$ or bias will then be calculated as the average distance between the correct and calculated predicted class:

Hope that helps. I think the easiest thing is to center your features ;)


Storing Big Matrix in DataBase I have a mysql database with the following format:

id      string
1        foo1...
2        foo2...
..       ...

There are >100k entries in this db.

What I want to do is for each string, compare it to each other string and store some metric of the comparison.  Doing this will essentially yield a 2D matrix of size NxN where N is the number of row in the db.

My initial thought was creating another db where each index corresponds to the string of the index in the first db and each column is the value from comparing the two strings. For example, id 1 column 2 in the second db would be the value outputted from comparing id1 and id2 in the first db.

The format of the second db:

id    col1    col2    col3    ....
1       1      0.4     0.5    .....
...    ...     ...      ...

This way of creating the second db would result in 100k rows x 100k columns, which is the issue at hand.  What is the best way to handle large data sets like this?  Is storing the data in a text file more efficient (say each text file corresponds to one row in the second db.)

It is usually more practical to compute the distances on the fly rather than storing $N^2$ values. If possible, you will want to use a heuristic to select only the items could be sufficiently similar so you don't waste time computing distance to irrelevant items. PostgreSQL can create an index on character trigrams in strings that enables efficient querying for other strings that are sufficiently similar (measured by the Jaccard similarity coefficient).

Storing an $N$ column wide table is not possible because the maximum number of columns allowed in MySQL is 4096. Even if that were not the case, using an RDBMS in such a way is considered bad practice and such a table would be difficult to work with.

If for some reason you must store a dense distance matrix, a format such as HDF5 would be more efficient than an RDBMS.


Statistical significance test in deep learning for regression problems I was reading the tutorial "Statistical Significance Test for comparing ML algorithms", where it suggests to use k-fold and apply the appropriate statistical test.

Suppose that I have a train set and a test set and two deep learning neural networks in a regression problem. Since the training in deep learning takes lots of time, the test procedure using k-fold would be very costly regarding computation resources.

I was wondering if it is possible to apply the statistical tests only in the results of the test set.

For example, if the test set has 1000 samples, the two neural networks would be trained in the same training set and then apply Wilcoxon in the results of the two networks over the 1000 points of the test set, is that correct? Or do I always need to perform the k-fold? Besides can I use the MSE in each test point and compare those results?

You could use either the k-fold or the test-only.

The k-fold is a possibility which gives you more robustness because you tried on every possible combination of train/test, but you are still avoiding overfitting if you use the train/test partition.

Just make sure you use the same dataset in all the comparisons you make.


Find outliers in Hive - SemanticException I'm trying to find some outliers on my database using HIVE and I'm using Standard Deviation technique. My query is:

SELECT ID
FROM data
WHERE ID < (AVG(ID) + STDDEV(ID))
  AND ID > (AVG(ID) - STDDEV(ID));

When I run this code I'm getting the following error:

 Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 3:12 Not yet supported place for UDAF 'AVG'

How to solve this problem? Many thanks!

Seems like Hive doesn't let you use avg in a where clause. You can solve this with a subquery.

SELECT id
FROM
    (SELECT id, AVG(id) as avg_id, STDDEV(id) as stddev_id FROM data)
WHERE id < avg_id + stddev_id AND id > avg_id - stddev_id


Different models for different time durations of a day I have hourly temperature and power consumption data of several days of a month. The pattern is almost similar across days like this:

Using this data I want to predict the usage of a coming day. I have features : 1) hour of the day 2) temperature; and the response variable, power. Looking at the data, I believe I should fit three separate models and not a single model

First model for data from Midnight to 10 AM as usage remains almost constant during this time, and temperature does not vary too much
Second model for data from 11AM till 6 PM. This portion follows a sharp increase and then almost constant usage
Third model for data from 7 PM till midnight. This portion shows constant decrease in power

To follow this intuition, I used three models accordingly and later combined predictions from these to output sequence of 24 numbers for coming day. Formula for each of these models is:
lm(power ~ time_hour + temperature, data = xxx), but each model is trained with the data corresponding to specific time duration of the day.

Instead of dividing the data manually and using three separate models, is there any other existing technique which will take care of our intuition and does not need manual division of data or creation of separate models.

During my search, I found that I can use GAM (Generalized Additive Models) and I came up with following formula

library(splines)
lm(power ~ ns(time_hour, knots =(9, 18)) + temperture, data = xxx)

Using above formula I think that I am placing knots at 9 AM and at 6 PM. Right?
I don't know how should I enforce knots in temperature feature exactly at these specific times, so that knots of temperature and time_hour will sync.

The above plots were plotted using following data:

dframe <- structure(list(time_hour = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24), temperature = c(22.5,
24, 23.5, 20.5, 22.5, 22.5, 19.5, 23.5, 23, 20.5, 26.5, 28.5,
30, 32, 33.5, 33, 30.5, 30, 29.5, 29, 28, 27, 28, 28.5), power = c(97.04319,
95.7225, 88.59191, 88.34882, 90.17179, 88.82062, 87.73833, 89.36342,
85.31775, 91.1292, 116.79035, 149.58614, 172.32438, 171.27931,
159.53858, 162.03544, 170.78468, 164.0275, 155.86717, 135.77197,
133.01235, 116.29253, 100.87483, 97.84942)), .Names = c("time_hour",
"temperature", "power"), row.names = c(NA, -24L), class = "data.frame")

and minimal code used is:

par(mfrow= c(1,2))
plot(dframe$time_hour ,dframe$power,type="l",xlab = "Hour of day", ylab = "power" )
plot(dframe$time_hour ,dframe$temperature,type="l",xlab = "Hour of day", ylab = "Temperature" )

This is a typical time series problem. The first step is to make sure your time series is stationary, see here for a good explanation why this is necessary.

So we first do,

dframe$power_diff <- c(NA, diff(dframe$power))
plot(dframe$power_diff, type = "l")

gives,

Next we want to consider an ARIMAX model. As part of this exercise we want to know whether there is a dependency in the time series, a so called auto regressive term. We can research this using,

acf(na.omit(dframe$power_diff))

which gives,

In this plot we see that there is a positive auto regressive term at lag 1 at a significance level of 95%, we conclude this based on the observation that the second line, which is lag 1, is above the blue dotted line.

Next we want to fit a ARIMAX model.

# Convert data to ts object
power_data <- ts(data = dframe$power_diff[-1], frequency = 8766, start = c(2016, 1, 1))
temp_data <- ts(data = dframe$temp_diff[-1], frequency = 8766, start = c(2016, 1, 1))

# Build the model.
m <- Arima(power_data, order = c(1, 0, 0), xreg = temp_data)

# Let's see how our fit is.
plot(power_data)
lines(fitted(m), col = "blue")

This gives this result,

You can play with the ARIMA(p, i, q) terms. We should leave $p=1$ since we found that the AR term is significant at lag 1. If we add a MA(1) we find a model that has a MAPE=260.46 versus a MAPE=272.62 for our initial model. So you can create the best model using,

m2 <- Arima(power_data, order = c(1, 1, 0), xreg = temp_data)
lines(fitted(m2), col = 'red')

Will lead to,

Of course we can automate the above step with auto.arima(). This function will search for the best arima model,

m3 <- auto.arima(power_data, xreg = temp_data)
lines(fitted(m3), col = 'green')

which will give you,

You see that each model has a poor fit in the spike. To improve this you could introduce a dummy variable,

xdata <- ts(data = as.matrix(cbind(temp_data, dframe$power_diff[-1] > 20)), frequency = 8766, start =c(2016,1,1))
m_dummy <- auto.arima(power_data, xreg = xdata)
lines(fitted(m_dummy), col = 'orange')

which gives us the following result,

We see that this model gives us a much better result. So if you can relate dummy to a point in time you could create a model that has a good fit.

According Occam's Razor you should try to keep your models simple. So I would suggest to build a single model for all your time slots instead of three separate models as you suggested in your question.

You can turn this back to the original time series with cumsum. Hope this helps.


LSTM model prediction scaling with loaded model I am deploying a LSTM pytorch model for production and I have issue with scaling the LSTM output correctly. While the model was tested the output was scaled with label data:
y_scaler = MinMaxScaler(feature_range=(-1, 1))
y_test_scaled = y_scaler.transform(y_test.reshape(-1, 1))

Now when I am loading the model from state_dict and making a prediction how I should inverse scale the data? I am not having the newest label data during the forecast. For the input data I am using QuantilieTransformer. I can make predictions this way:
y_test_scaler = MinMaxScaler(feature_range=(-1, 1))
y_minmax = label_data # For example past 40 days of the label data

y_test_scaled = y_test_scaler.fit_transform(y_minmax.reshape(-1, 1))

If I use the whole label-data as y_minmax the results are not good enough. But If I use less label data for scaling predictions are decent.
# Inverse transform predictions from LSTM model
y_actual = y_test_scaler.inverse_transform(y.reshape(-1, 1))

Should the output of LSTM be somehow inversed with x_scaler or how? :)
EDIT:
I got good results by making prediction for large amount of sequences. (Predicting 1 sequence to future and 300 sequences of history data). It is not a problem to load a lot of history data during the prediction, but I believe there is a smarter way to get the same result.

Answering to my own question:
One way to do it was to use joblib dumping the scalers that were used in the training process.
More about it is discussed in here: https://stackoverflow.com/questions/41993565/save-minmaxscaler-model-in-sklearn
Works the same way for LSTM.


Prepare JSON data from sentiment analysis to perform Logistic Regression I'm new to this field, so very sorry for this basic question.
I'm working on a text analysis project using Google's NLP API along with some other APIS. After performing the sentiment analysis I have its results in JSON format and now I need to apply the Logistic regression, I have no idea how can I prepare my JSON data to perform Logistic regress.

Here's a sample of my data:

{
   'documentSentiment':{
      'polarity':-0.3,
      'magnitude':0,
      'score':0
   },
   'language':'en',
   'sentences':[
      {
         'text':{
            'content':'{\\rtf1\\ansi\\ansicpg1252\\cocoartf1639\\cocoasubrtf100\n{\\fonttbl\\f0\\fswiss\\fcharset0 Helvetica;}\n{\\colortbl;\\red255\\green255\\blue255;}\n{\\*\\expandedcolortbl;;}\n\\paperw11900\\paperh16840\\margl1440\\margr1440\\vieww10800\\viewh8400\\viewkind0\n\\pard\\tx566\\tx1133\\tx1700\\tx2267\\tx2834\\tx3401\\tx3968\\tx4535\\tx5102\\tx5669\\tx6236\\tx6803\\pardirnatural\\partightenfactor0',
            'beginOffset':-1
         },
         'sentiment':{
            'polarity':-1,
            'magnitude':0,
            'score':0
         }
      },
      {
         'text':{
            'content':'\\f0\\fs24 \\cf0 This is the first text file.}',
            'beginOffset':-1
         },
         'sentiment':{
            'polarity':1,
            'magnitude':0,
            'score':0
         }
      }
   ]
}

How can I prepare this data to perform Logistic Regression?
I will use Pandas, Numpy and Sckikit-learn.

Help me, please!

Thanks in advance!

Reputation barrier hence mentioning this as an answer, instead of a comment. This article should lead you the right way, as author has used Twitch API and Python to create his dataset. You shall get a feel of entire data flow.

EDIT: This is a nice tutorial on understanding (basically read/write) JSON files using Python.


How to run R programs on multicore using doParallel package? I am running SVM algorithm in R.It is taking long time to run the algorithm.I have system with 32GB RAM.How can I use that whole RAM memory to speed my process.

I would add a comment but I do not have enough reputation points. I might suggest Using "R revolution open". It is a Build of R that includes a lot of native support for multi-core processing. I have not used it much as my computer is very old, but it is defiantly worth looking at. Plus it is free.


Is it possible to design a deep CNN model on a small size image dataset I have started a project of classifying a dataset using deep learning. I have tried transfer learning on pretrained models. Now, I want to design a CNN model which can do this work of classification but everywhere it says it needs large data. My question is: Is it possible to design a deep CNN model on an image dataset of size less than 1000 images?

According to your description of the data, it is highly probable that training any neural network of reasonable size will overfit the training data.

One option is to apply any possible regularization approach; these come to mind:

Use L1/L2 weight regularization.
Reduce the number of parameters as much as you can (less layers, less number of filters).
Use dropout.

Also, try any data augmentation trick: rotate images, mirror images, crop and scale images, etc.


How does personalized machine learning work? Many services (such as Netflix, Amazon, and Google Search, Apple's Siri) are said to get better by learning the 'habits' of their users. As I understand, they somehow create a customized machine learning model for each individual because a generic model would not work well. However, I would like to know how this can be achieved in practice. For the sake of discussion, suppose I am developing a new massage chair that changes its "massage recipe" based on the respiration pattern of its user. I suppose (and I might be wrong here) that the respiration pattern varies from person to another but that the variations in a person's respiration pattern indicate what kind of message the person would like to have (again, these suppositions are wrong but they are presented for the sake of discussion). I have questions on how these models are customized. Specifically:

How would I train the model for my hypothetical device?
How would customize the said model? Do big companies (Amazon et.al) create a person-specific model for every user? And if this is the case, wouldn't this be impractical (Google serves billions of people; therefore, having billions of models would be hard to maintain)
In the case of Google for example, when setting a new android phone for the first time, the user is requested to repeat three-times "Ok Google" to train their voice recognition model. Is this small sample of sound enough to create the voice recognition model? Or is the sample used to customize an existing generic model? And if so, how is this done?
Is there any references (blog, paper, etc...) that discuss this topic in details?

You should google recommendation systems. A good starting point is this place https://paperswithcode.com/task/recommendation-systems where they have some papers on recommendation systems with code.


Conditional logit in Stata Question: Can Stata present or output the actual betas/coefficients in a datasheet when it performs the conditional logit model?

I understand that Stata has a "predict" capability, but I need to store the betas/coefficients and use them later, after new factor values have been calculated.

I believe that the command you are looking for is outreg2. For other output options check out this page.


The learning_rate in TensorFlow code is the sum learning rate of a batch or the learning rate of a data? import tensorflow as tf
batch_size = 10
learning_rate = 0.001
tf.train.AdamOptimizer(learning_rate)

So the question is:
The learning_rate of a data is 0.0001 or 0.001 ?
The sum learning_rate of a batch is 0.01 or 0.001 ?

It is the learning rate that you set on the optimizer for your problem. It requires a fair bit of experimentation to figure out what learning rate works best for your problem. It is a more of an art than a science, though there exists heuristics on an optimal range of values to select from.
For most problems, the default learning rate should be a good starting point.


How to force DecisionTreeRegressor to use polyfit equation instead of mse at leaf level in python SKlearn I am using python SKlearn DecisionTreeRegressor for my data. As DecisionTreeRegressor defines constant data at leaf node, my prediction is looking like a step, instead I wanted to force polyfit Equation of my interest. I was looking some thing like to use Decision tree to some depth, then apply normal regression at leaf node to predict continuous data.
For my data set normal regression doesn't work as it looks for global error minimization, so i am looking for this.
I am a beginner, any response would be appreciated. Thanks.

This is an expanded answer based on my comment.

First of all, there are many interesting highly non-linear classifiers besides decision trees. Most widely used are neural networks and SVMs. (Nitpick but SVMs can be seen as neural networks.)

Within the world of decision trees, there are things like decision trees that have linear regressions in the nodes. However, these are very slow to train. The problem is that you can no longer take advantage of the fact that decision trees entropy or gini score can be calculated incrementally as the threshold $t$ is varied (where $t$ is the right-hand part of the node rule, $x_i < t$). This trick can not be used for these trees. There are of course many other rule induction algorithms besides decision trees, which are not as step-like.

Anyhow, usually ensembles are used. Ensembles are models made of several models, which then "vote" on the result. You can use them with any class of models, but they are particularly excellent when combined with decision trees because:

Decision trees can have very different results if the sample is slightly different. Ensembles fix that;
Decision trees decision boundaries are orthogonal to the features, which have "step" effects;
Decision trees have to be pruned, while some ensembles can be used without pruning;
Most winners of Kaggle and other data mining competitions have used decision trees ensembles;
Ensembles give you probabilities if you need them; probabilities from decision trees are highly dependent on how you prune them (and unpruned decision trees cannot give you probabilities at all).

See here the results of some decision trees ensembles:

There are two types of ensembles widely used:

Random Forest: where each decision tree is trained independently.
Gradient Boosting Trees: these are based on AdaBoost. The idea is that the next tree is trained by weighting more strongly the observations that the previous trees have not been able to qualify correctly.

Usually, Kaggle winners use gradient boosting trees, especially using the xgboost package. But they have more hyperparameters to tweak, so if you don't have much time to tweak them, just go with a random forest. Gradient boosting trees are more ticky to get right, and the results can be different if you change parameters like n_estimators. Random forests, on the other hand, tend to the expected behavior when n_estimators$\rightarrow\infty$.

Also, gradient boosting trees can overfit, albeit I think it's rare. (Random forests can also overfit, but it is because the underlying estimator is overfitting. They tend to the average behavior as you increase n_estimators. Tell me if this isn't clear.)

Note: n_estimators in sklearn is sometimes very low, and you should increment it. In real problems, people use thousands of trees.


Handling categorical variables in large df I have a df with nearly 40 million rows and ~20 columns (total size is 2.2+GB). 15 of my features are categorical. I figured that the most reasonable way to go about this problem without making the df any bigger would be fit/transform each with LabelEncoder and then convert each feature to category data type.

The only thing is that none of the categorical features are ordinal. Should I fit/transform them with StandardScaler or is that unnecessary?

No. If, as you said, the variables are categorical, performing a scaling does not make any sense. Plain LabelEncoder already does what you want.


Confusion matrix. "How close I am to the diagonal?". Is there such metric? I have a question regarding confusion matrices.
To start, we discuss the case of multi-class classification so the confusion matrix has dimension, for example 4 times 4, for classification task with 4 possible outcomes.
The correct predictions should bring all the answers on the diagonal. Is it possible to rate though how close are the results on the diagonal? For example, is much better to predict a class very close to the current one (so guess 3 instead of 4) in comparison to predict wrongly a further away class (predict 4 while it was 1).
Is there a metric that can take that into consideration? Rate kinder classes that are have missed the ideal target by a bit and penalize stronger misclassification that are further away from the real class?
I have been looking a lot about this on the internet but no luck so far.
Any suggestions on how to proceed?

The classes that define the columns/rows can be arbitrarily rearranged. Therefore, the "distance" of a misclassification to the diagonal has no meaning. So no, there is no such metric.
I like @Dave's comment: "Is it worse to call a dog a cat than it is to call a dog a horse?"

Maybe you'd ask yourself, "some classes feel closer together than others"
For example, we could create a confusion matrix like this:
$ \begin{matrix}
&&&&PREDICTED\\
&&Person & Woman & Man & Camera & TV
\\
T&Person & 33 & 5 & 3 & 0 & 1
\\
R&Woman & 10 & 50 & 2 & 22 & 0
\\
U&Man & 12 & 23 & 47 & 1 & 13
\\
T&Camera & 4 & 2 & 7 & 24 & 9
\\
H&TV & 3 & 5 & 8 & 13 & 11
\end{matrix}$
(Where $Person$ denotes non-binary of course).
It feels like misclassifying $Camera$'s as $Woman$ should be "more wrong" than misclassifying $Man$ as $Woman$. After all, women aren't objects.
However, in the world of unfeeling classifiers, a "woman" has no meaning, and neither does "object". Hence we call such classifiers "tools".

Edit to address your comment:
In situations where there's some notion of distance, you'd use regression rather than classification. You can use regression even for discrete dependent variables. In other words, you need either numeric or ordinal data for distance to make sense.
To use the example in your comment of "classifying" a variable that represents performance:

Continuous: If you measure performance as a continuous variable, then it's clear to use regression.

Ordinal: But even if you measure performance on, say, an integer scale from 1-10, you can still regress the data. (As an aside, in practice, all measurements can be considered are discrete if you consider that they are limited by resolution/precision). You can also map ordered concepts to discrete but numeric values. For example, the Likert scale (Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree) can be mapped to integers 1-5. However, the reason you can't directly determine distance without mapping to numerical values is because there's no intrinsic distance between the nominal values. Strongly Disagree could be two units away from Disagree, and maybe Neutral is a billion units away from Agree.

Nominal: If you measure performance using words like "good", "decent", "fine" where there's no clear ordering, then distance makes no sense.


PCA in visual Analytics I m studying visual analytics and i have a theoretical question about this topic.

My professor introduced this schema in him slide

For connect data to visualisation.
Some topic is very easy to understand and regards the rappresentation of data, plot and other tool, other topic instead regards the "data" in particular the data reduction using algorithm like PCA.

I m not very clear why using PCA over the data for reduction some data that we want rappresent.
Ad example, if i have dataset with all movie(it s just an example) with some attribute ad example

id,title,year,review, userReview ecc...

Where is the correlation of this data?Why i need apply a PCA for reduce big database and lose some of this components that I can visualize in some plot or schema that for me is important
For example could be interesting see all best movie that have high average review, using PCA we just cluster user in different movie (i saw example on Iris database here: https://medium.com/@kyasar.mail/pca-principal-component-analysis-729068e28ec8 )

Anyone can clarify me?

You have to see things for what they are. First, you have to figure out what PCA does. It's not clustering, but it can be used to realize that there could be clusters in the data.
PCA is not always the answer. It depends on the data. There are other dimensionality reduction techniques such as autoencoders, non-negative matrix factorization and so on...

For the movie example, it does not make sense to use PCA. The iris database contains another type of data, where it does make sense. For the movie example, you might want to have a look at singular value decomposition and collaborative filtering.

One of the reasons for doing PCA could be for clustering. If you have many dimensions, and the intrinsic dimensionality is low you will have the curse of dimensionality. Reducing the dimensionality by PCA and doing clustering on the embeddings solves, in this case, the curse of dimensionality.


Titanic Kaggle Data: Why am I getting lower accuracy on Kaggle submissions than on held-out data? I am going through my first solo machine learning project and would like to gain some insight into what I am doing wrong/what is going on here as I am a bit stuck.

I have been applying machine learning to the Titanic data set with SKlearn and have been holding out 10% of the training data to calculate the accuracy of my fitted models. I also use K-fold cross valdation with 10 folds to evaluate the model performance and choose hyper-parameters. I have so far applied logistic regression and a linear Kernel SVM and in both cases I get 78-80% accuracy on the K-fold validation sets and when applying the fitted classifiers to my held-back previously unseen testing data. However when I predict on Kaggle's test data and submit my predictions it comes back with values around 76% which is significantly less than I'd expect, and well outside the variance in the accuracy values I get with K-fold cross validation.

A link to the Jupyter notebook where I do this is provided below: http://nbviewer.jupyter.org/github/AshleySetter/Kaggle_Competitions/blob/master/Titanic_project/Titanic_machine_learning_clean.ipynb

Could anyone give me some insight into what is going on here and what I am doing wrong?

It could be because of the percentage of the different class.
Imagine your data is 30% survived and 70% died but in Kaggle's test data this ratio may change, i.e 50%-50%. So your model could not predict kaggle's survived part as well as your test data.
you may impute missing by the mean. if you use test data for calculating mean it could be a cheating for your model.


How to evaluate the clustering result when cluster numbers are not equal to data set class When apply clustering algorithm with the multi-class data set and class numbers are not equal to the result cluster numbers(For example , When we use K-Means algorithm by setting K = 3 apply with "yeast" data set that has 14 classes).

What is the proper way to evaluate the result in this situation ?

If no pair of your classes are overlapping and they do classify each and every element of the data set, then they constitute a partition of that set. So does your clustering.

In such a case, I would suggest to use a distance metric to compare two partitions, instead of any bona fide index, even if it's popular -unless that index happens to be a metric as well.

One such distances for comparing partitions is the $VI$ index, rediscovered by Maria Meila$\phantom{,}^\dagger$ a few years back, but its origin dating back to the 80's in a physics paper by Zurek$\phantom{,}^{\ddagger}$.

How to calculate distance between two partitions P and Q:

Determine $P\wedge Q, $the "intersection" (I always mix up the words join and meet here) of P and Q. This is simply the partition formed by taking the intersection of each cluster of P against each cluster of Q.
For all three partitions $P$, $Q$, $P\wedge Q$ calculate their entropy $H$ defined as follows: $$H(P)=-\frac{1}{N}\sum_{i=1}^Np_i\log p_i\;,\;p_i\equiv\frac{n_i}{N}$$ where $N$ is the total number of elements you clustered/classified and $n_i$ is the number of elements within cluster/class $i$ of a given partition $P$.
The distance $d(P,Q)$ between partitions P and Q is then given by $$d(P,Q)=2H(P\wedge Q)-H(P)-H(Q)$$

The reason for using a real metric is simple: a metric satisfy the triangular inequality and our intuitions of "close" and "similar" heavily relies on that. Any ad-hoc measure of dis-similarity (distance) that doesn't satisfy the triangular inequality will eventually give you results like this: A is "$d$ away" from B (d being concrete number), with $d$ smaller than a given bound, say $D$. Then you find A close to C by an amount $d'<D$. If this distance were not a metric, it could then happen that the distance $d''$ between B and C is say $2D$. For a sound metric distance, it should be
$d''\leq d+d'<2D$. In other words A would be "similar" to B and C, but the last two would be "different"/"not similar" from each other. This is counterintuitive.

The downside is that the VI distance may be computationally more expensive than calculating say a Rand index. The upside is that it's mathematically sound and makes sense.

You can use R to calculate the VI distance. You can also use Partanalyzer a program I wrote some time ago (https://github.com/MASantos/Partanalyzer). It needs a rewrite to make it more efficient, but it's open source, so you can modify it at will.

$\dagger$: Meila, M. Journal of Multivariate Analysis, 98, 873 (2007).

$\ddagger$: W.H. Zurek, Nature, vol.341, p.119 (1989).


How to get a dataframe values in one single column for the following dataset? m38     m78        alpha

4.4717  4.8745
4.4569  4.6491
4.5101  4.7262
4.4407  4.8234
4.1184  4.3862
3.8448  4.2816
3.7246  4.6183
3.2857  4.6744

For the above sample data (actually 8000 rows) i need to calculate alpha(column), using the log function.

In: alpha = (np.log([dataset.m38,dataset.m78])/np.math.log(38,78.7))

OUT: array([[1.79754463, 1.90105609],
       [1.7935659 , 1.8442364 ],
       [1.80780671, 1.86397626],
       ...,
       [2.06431358, 2.40416332],
       [2.08820691, 2.41635699],
       [2.09982107, 2.39551918]])

I converted this into dataframe by using the below code:

In: alpha = pd.DataFrame(data = alpha)

OUT: 2 rows × 8046 columns

Now, i used alpha.T to get this in 8046 rows and 2 columns.

Why i am getting 2 columns not one? and how should i convert it into one column?

You are not getting 2 columns. You are getting 2 rows.

The function alpha = (np.log([dataset.m38,dataset.m78])/np.math.log(38,78.7)) returns a 2-dimensional array. Putting in:

         m38     m78
      0  4.4717  4.8745
      1  4.4569  4.6491
      2  4.5101  4.7262
      3  4.4407  4.8234
      4  4.1184  4.3862

gives us: [[0.41174795 0.41083658 0.4140986  0.40983552 0.38912198]
           [0.43545842 0.42244323 0.42696487 0.43256132 0.40644075]]

This is due to the fact that np.log([dataset.m38,dataset.m78]) returns a 2-dimensional array.

Regarding the flipped columns/ row issue: Once your function doesn't return a 2-dimensional array, it will most likely return a Pandas Series which will have one column and 8000 rows.

If you want to turn that into a DataFrame, you can use
alpha = pd.DataFrame({'alpha':alpha})

Think about what your formula does. Putting in [dataset.m38,dataset.m78] will do the calculation for both columns seperately resulting in 2 different arrays.


Definition of 'feature coverage' I have heard the term 'feature coverage' in machine learning. However I found no relative infomation after I googled this term. Could some one give me a definition of 'feature coverage' and some references?

Depends on the context it may mean differently. One common usage of the term is when not all the data points have valid/reliable values for a specific feature. For example for a feature set of $x=\{x_1, x_2, \dots, x_p, \dots, x_n\}$ not all data points may have specific feature $p$.

It is possible that some data points lack a value for feature $p$ either because of data corruption, noise or etc. In this case the percentage of data point which actually have a reliable value for this feature is called the feature coverage of feature $p$ for this specific data set.


Calculate confidence score of a neural network prediction I am using a deep neural network model to make predictions. My problem is a classification(binary) problem. I wish to calculate the confidence score of each prediction. As of now, I use predict_proba of sklearn to get probability of a class and consider it as the model confidence for that prediction. Is it the right approach? What could be a better way to calculate the confidence score?
Kindly add relevant literature to support your approach.

One way to estimate the level of confidence we have about an ANN prediction is to use dropout perturbations. The idea was proposed in this paper: Dropout as a Bayesian Approximation. Representing Model Uncertainty in Deep Learning. The core idea is to use dropout as a perturbation method, and check how predictions change with varying levels of dropout. Once you sampled a sufficient number of "distorted predictions", you can estimate something analogous to a confidence interval around the initial model prediction. This technique works either for classifiers and regressors.

You can read an explanation of this approach here.

EDIT:

To be more precise:

In order to implement this technique, use Dropout() layers, they can be used in prediction phase too (not just during training). You can train your Neural Network, then transfer its weights into another ANN with the same architecture + dropout layers. Something like:

new_model.set_weights(original_model.get_weights())

Once the new ANN with Dropout() has its weights, run it and vary its dropout hyperparams, get the predictions and calculate the CIs. I know it's time consuming so do it only if you think it's really worth it. Another possible way to implement can be done with custom layers like:

model.add(Lambda(lambda x: K.dropout(x, level=0.5)))


General approach to work with text of phone calls (topics, promises, sentiments, etc.) I have an NLP task. There is a text (telephone conversations). Voice is already converted into text and is divided into agent and customer paragraphs. I need to understand what approach is the best one for the next tasks:

Who is the customer and who is the agent?
Customer Name
The topic of conversation
Promises made by the operator to the customer (for example, "I call back tomorrow")
Negative Sentiment (if there is something in the conversation that the subscriber is not happy with)

I am just trying to understand how to handle it. Is it possible to create some kind of general approach for this? If yes, for which packages/publications/books could I pay my attention?

Step by step.

Since you say its seperated into C-customer and A-agent I am going to assume its labeled (some of the questions could be solved unsupervised also)

Who is the customer and who is the agent?
Simple binary classification. Find suitable numerical model representation of text and perform classification. Starting reference
Customer Name Named entity recognition - NER example how to use it:

  import nltk

from nltk.tag.stanford import NERTagger

st = NERTagger('stanford-ner/all.3class.distsim.crf.ser.gz', 'stanford-ner/stanford-ner.jar')
text = "blablabla"

for sent in nltk.sent_tokenize(text):
    tokens = nltk.tokenize.word_tokenize(sent)
    tags = st.tag(tokens)
    for tag in tags:
        if tag[1]=='PERSON': print tag`

The topic of conversation Start with these two approaches: LDA and LSA
Promises made by the operator to the customer (for example, "I call back tomorrow") General approach (its plausible, needs a bit of work by you) But lets say you use word embeddings, all of the sentences that have promises will be vectors that are close to each other in this high-dimensional space. You can limit your system to return all sentences that fall into this cluster in the space.
Negative Sentiment (if there is something in the conversation that the subscriber is not happy with) Simply just look at this introduction to Emotion AI Nutshell: You already have pre-trained models that can model positive-neutral-negative sentiment in the given text. Use the recources.


In fact, you use 1D convolution.
Given that the dimension of the output of embedding layer is 100, that the kernel size is 5, and that the number of filters is 128,
You have 100x5x128 = 64000 weights. Add to this 128 biases and you get 64128 parameters.

Note that parameter sharing is used, so that there is only one set of weights and biases per filter, in depth.


Scikit-Learn - Learned model description? Is there a way I can "look inside" a model once it's trained? For example, if I train a spam filter with a multinomialNB, is there a way I can extract which words are most likely to make an email classify as spam? I'd like to see how the models determine the outcome once fitted.

For the particular case of the MultinomialNB you can look here.

However if want you want is to determine which features are the most important, the SelectFromModel for selecting the most important features for the model.


is neural networks an online algorithm by nature? I have been doing machine learning for a while, but bits and pieces come together even after some time of practicing.

In neural networks, you adjust the weights by doing one pass (forward pass), and then computing the partial derivatives for the weights (backward pass) after each training example - and subtracting those partial derivatives from the initial weights.

in turn, the calculation of the new weights is mathematically complex (you need to compute the partial derivative of the weights, for which you compute the error at every layer of the neural net - but the input layer).

Is that not by definition an online algorithm, where cost and new weights are calculated after each training example?

Thanks!

There are three training modes for neural networks

stochastic gradient descent: Adjust the weights after every single training example
batch training: Adjust the weights after going through all data (an epoch)
mini-batch training: Adjust the weights after going through a mini-batch. This is usually 128 training examples.

Most of the time, mini-batch training seems to be used.

So the answer is:

No, the neural network learning algorithm is not online algorithm.


getting error while scrapping Amazon using Selenium and bs4 I'm working on a class project using BeautifulSoup and webdriver to scrap
Disposable Diapers on
amazon for the name of the item, price, reviews, rating.

My goal is to have something like this where I will split this info in
different column:

 Diapers Size 4, 150 Count - Pampers Swaddlers Disposable Baby Diapers, One
 Month Supply
   4.0 out of 5 stars
   1,982
   $43.98
  ($0.29/Count)

Unfortunately, I get this message after the 50 data appears: message: no such
element: unable to locate element: {"method":"css selector","selector":".a-last"}

Here is my code:

URL = "https://www.amazon.com/s?
k=baby+disposable&rh=n%3A166772011&ref=nb_sb_noss"
driver = ('C:/Users/Desktop/chromedriver_win32/chromedriver.exe')
driver.get(URL) html = driver.page_source soup = BeautifulSoup(html, "html.parser")
df = pd.DataFrame(columns = ["Product Name","Rating","Number of
Reviews","Price","Price Count"])

while True:
for i in soup.find_all(class_= "sg-col-4-of-24 sg-col-4-of-12 sg-col-4-of-36
s-result-item sg-col-
4-of-28 sg-col-4-of-16 sg-col sg-col-4-of-20 sg-col-4-of-32"):
ProductName = i.find(class_= "a-size-base-plus a-color-base a-text- normal").text#.span.get_text
print(ProductName)
 try:
Rating = i.find(class_= "a-icon-alt").text#.span.get_text()
  except:
 Rating = "Null"
print(Rating)
try:
NumberOfReviews = i.find(class_= "a-size-base").text#.span.get_text()
 except:
 NumberOfReviews = "Null"
 print(NumberOfReviews)
 try:
Price = i.find(class_= "a-offscreen").text#.span.get_text()
except:
Price = "Null"
print(Price)
try:
PriceCount = i.find(class_= "a-size-base a-color-secondary").text#.span.get_text()
except:
PriceCount = "Null"
print(PriceCount)
df = df.append({"Product Name":ProductName, "Rating":Rating, "Number of
Reviews":NumberOfReviews,
"Price":Price, "Price Count":PriceCount}, ignore_index = True)
nextlink = soup.find(class_= "a-disabled a-last")
if nextlink:
print ("This is the last page. ")
break
else:
progress = driver.find_element_by_class_name('a-last').click()
subhtml = driver.page_source
soup = BeautifulSoup(subhtml, "html.parser")

Unfortunately, I hit a block road trying to figure out why it is not taking a_last.

Most likely the element you are trying to find (I suppose it is a link or a button with class a-last) is not on the page. When the error appears, you should look at your chrome window and see what it is showing you. Check if the element is on the page.

It could be that amazon.com just showed a captcha and therefore all the usual elements disappeared from the screen. For example, if you try to do while True without waiting time on Google it will show you a captcha after several requests.


Binary classification based on pairwise relationships I have a seemingly esoteric machine learning problem and I'm not sure where to start looking at literature.

I am trying to train a binary classifier that accepts/rejects each object in a collection of N objects. As input, it is given feature vectors describing each object, as well as feature vectors describing pairwise relationships between each object. The list of pairwise relationships is not complete, but not very sparse: approx 1/3 of the possible pairs are described.

The ground truth is the accept/reject label of each object, and I have a set of collections to train on. From domain knowledge I know that the global properties of the entire collection are important to determining the classification of the object, which is why I would like to use the pairwise relationships in classification.

Currently to classify the objects we are using a greedy algorithm that uses a rule-based score based on the relationship measure. (We start with an object that we trust to be very good, then we greedily add the other objects until no more objects have a positive total relationship score in relation to the already-accepted subset) So we know that the relationship measure is important and will be present in the test data. I am attempting to improve on this by using a machine learning approach.

My first attempt was to treat each pairwise relationship I have as a data point, and concatenate the two self-descriptor vectors to the relationship vector. Then, I labeled this as "1" if they were an accept-accept pair, or "-1" if it was an accept-reject pair. Then, during testing, I could put these predictions in one affinity matrix for each collection (if the pairwise relationship is unknown, it that matrix entry is left as 0) and use spectral clustering ("accept" and "reject" objects would have opposite signs in the first eigenvector). However, this performed worse than using only the self-descriptors for classification. I suspect this is because there is a considerable deficiency in this training protocol -- unlike the rule-based score we currently use, there is no indicator of how strong the agreement or disagreement between the two objects is.

Therefore, I would like to create a classifier that looks at the entire collection of self-descriptors and pairwise relationships, and uses the accept/reject labels directly as ground truth somehow.

The solution I needed was permutation-equivariant neural networks. The application in the paper is not classification, but nevertheless it has the correct properties to adapt to classification.


Custom conditional Keras metric I am trying to create the following metric for my neural network using keras

$$
s = \left\{
        \begin{array}{ll}
            \sum_{i=1}^{n} e^{\frac{-d_i}{10}}-1 & \quad d < 0 \\
            \sum_{i=1}^{n} e^{\frac{d_i}{13}}-1 & \quad d \geq 0
        \end{array}
    \right.
$$
where $d_i=y_{pred}-y_{true}$

and both $y_{pred}$ and $y_{true}$ are vectors

With the following code:

import keras.backend as K

    def score(y_true, y_pred):
            d=(y_pred - y_true)
            if d<0:
                return K.exp(-d/10)-1
            else:
                return K.exp(d/13)-1

For the use of compiling my model:

model.compile(loss='mse', optimizer='adam', metrics=[score])

I received the following error code and I have not been able to correct the issue. Any help would be appreciated.

  raise TypeError("Using a tf.Tensor as a Python bool is not
  allowed. " "Use if t is not None: instead of if t: to test if a "
  "tensor is defined, and use TensorFlow ops such as "

  TypeError: Using a tf.Tensor as a Python bool is not allowed. Use
  if t is not None: instead of if t: to test if a tensor is defined,
  and use TensorFlow ops such as tf.cond to execute subgraphs
  conditioned on the value of a tensor.

I'm assuming y_true and y_pred both are vectors. You are trying to apply a conditional on the value of a tensor. The library is telling you that it is not possible (tf.Tensor as Python bool).

You may want to check tensorflow control flow operations like tensorflow.equal


Stemmer or dictionary? I have recently ported a stemmer from Java to Python for a highly inflectional language.
The stemmer learns how to change suffixes from the dictionary of words and their inflected forms. It basically builds a stemming table with learned stemming rules. As I was porting the algorithm I decided to train it on a larger dictionary. As a result, the learned stemming table got bigger, and stemming accuracy got higher as well.
Then I thought this actually make no sense as the stemming table size gets closer and closer to the size of the dictionary.
Why build or train stemming algorithms if you can simply lookup a dictionary?
I can understand that in old times storing large files could be a problem, but now? And for some languages there might be no proper dictionary resources. But is there any other reason?

There is another reason: words which don't appear in the dictionary. Of course a dictionary approach will correctly stem all the forms which are known in the dictionary, and depending on the language this may indeed lead to better accuracy. However the dictionary approach cannot do anything about unknown words, whereas a generic stemmer can try to apply its generic rules. This can be particularly important with texts which are either very domain-specific (e.g. medicine), which often contain technical words which are not in a general dictionary, or recent user-generated texts such as social media posts where people may use neologisms or words borrowed and sometimes transformed from another language.


Is it possible that I conduct a supervised machine learning like random forest after creating a target variable through clustering? I create 4 clusters by using k-mean cluster analysis. After that, I try to classify this target variable with random forest in order to confirm if the clusters are properly grouped. Is it possible?

Generally speaking it is possible.
Somemore information would be helpful:
How do you plan to test/train? Do you want to use k-fold cross validation, or split the data? Are you going to use exactly the same variables as predictors in the Random Forest, that u used for the clusteranalysis? In this case you propably need to be careful with your interpretation since it is circular by nature.


Orange 3 - Feature selection / importance I am using (and loving) Orange 3 for some projects at my school and have a question:

When using Python and e.g. doing a RandomForest Classification, I can easily access the feature importances by feature_importances_

In Orange 3 there seems to be no feature to access that in the visual programming interface (would be a great add-on), so I tried to write my own python script in orange...

I linked the RandomForest Classifier to the Python script widget's in_classifier and have the following code:

print(in_classifier)
importances = in_classifier.feature_importances_

The result is however:

Random Forest Classification {'random_state': 0, 'n_estimators': 500, 'n_jobs': 1, 'min_samples_leaf': 1, 'max_leaf_nodes': None, 'max_features': 'auto', 'bootstrap': True, 'verbose': 0, 'max_depth': None, 'oob_score': False, 'min_samples_split': 2, 'class_weight': None, 'min_weight_fraction_leaf': 0.0, 'criterion': 'gini'}
    Traceback (most recent call last):
      File "", line 1, in
      File "", line 2, in
    AttributeError: 'RandomForestClassifier' object has no attribute 'feature_importances_'

=> I don't understand whey the RandomForestClassifier Object doesn't provide this array?

Anyone any idea?

many thanks!!

You can connect Random Forest widget into Rank widget, which I guess is meant for subset feature selection. See Orange screenshot #19.


Order SparseVectors by the closest distance to given SparseVector I have a Spark dataset containing a column of SparseVector types. Additionally, I have another SparseVector $X$ which is not a part of the dataset. I want to order my dataset according to the closest distance (or similarity) relative to $X$.
Can anyone help me with how to implement this?

it appears that problem was solved with BucketedRandomProjectionLSH. After fit && transformation, and approxNearestNeighbors resulted dataset contains distCol, which (as per LSH.scala):

  @param distCol Output column for storing the distance between each result row and the key.


Why continuous features are more important than categorical features in decision tree models? I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.

I have converted all categorical variables into dummy variables using one hot encoding (for better interpretation in my logistic regression model).

On one hand, I use LogisticRegression (sklearn) and rank the most significant features by using their coefficients. In this way, I see both categorical and continuous variables among the most important features.

On the other hand, When I want to rank the features by using Decision Tree models (SelectFromModel) they always give higher scores (feature_importances_) first to continuous features and then to categorical (dummy) variables. A completely different behavior in comparison with Logistic Regression.

Whilst the performance of Decision Tree models is much higher (about 15%) than the performance of Logistic Regression, I want to know which sorting of features (Decision Tree or Logistic Regression) is more correct? And why Decision Tree models give more priority to continuous features?

It could be the way that you encode categorical variables.

If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight.

Let's say per example, that you are trying to predict the condition of a patient at the hospital Alive==0, Dead ==1. Imagine that you have a feature is called Head_Shot[0,1], that is really rare, it only appears a few times in the dataset.

The linear model will assign a lot of weight to this coefficient since it is crucial for the target variable. If this happens the rest of the features has no meaning.

For the decision tree, it could do a split in just one of the tree and since it calculates the importance of a feature weighted time the number of times it appears it wouldn´t such a relevant weight.

I am assuming you are doing one-hot encoding. With other techniques, it will be different. And I also assume the way that you calculate the feature importance. So this is far from a scientific answer.

Continuous variables can have more importance in decision trees because each tree can do several splits along its way.

Sorry for the example, it is a bit drastic but I believe it makes the point.


Local minima in Stochastic Gradient Descent I am running a logisitic regression model using Keras on a dataset. However, I am often running in a problem in which the model settle at an validation accuracy value lower than a theoretical value.

The hypothesis is confirmed by the fact that if I run the model again (without changing anything), the model settles at a different value (often higher) for validation.

I presume such a problem is due to the gradient descent encountering a local minimum and when I run it again, due to random fluctuations, I sometimes end up skipping the local minima.

So any method, to prevent such a problem?

def base():
    model = Sequential()
    model.add(Dense(20,activation = 'linear', input_dim = 12288))
    model.add(Dense(7,activation = 'relu'))
    model.add(Dense(5, activation = 'linear'))
    model.add(Dense(1,activation = 'sigmoid'))
    model.compile(optimizer = optimizers.SGD(lr=0.0075), loss = 'binary_crossentropy', metrics= ['accuracy'])
    return model

model = base()
model.fit(train_x,train_y, epochs = 2400, batch_size =10,validation_data = [test_x, test_y], verbose = 2)

One of the major advantage of stochastic gradient descent (SGD) over batch descent is the ability to explore different local minima's. But due to stochastic nature, you might end up with different results every time you repeat.

In all regression the most important task is to perform regularization to achieve a generalized fit for better cross validation scores. Your issue might be of over fitting rather than stochastic nature of optimizer. In your model.add method, add an argument for kernel_regularizer=L1L2(l1=0.0, l2=0.15). Try with different values for shrinkage of L1, L2 norms.

Also, If you are looking for stability in accuracy you can try batch/ vanilla gradient descent. Also, I would recommend you to try different learning rates and other optimizer like Adam that make use of adaptive learning rates.


Python - Todoist API - Store data into Dictionary I've this Python code that allows me to collect data from Todoist API. Now I want to store this data into a Dictionary in order to make some data processing and some advanced analytics. Using this code how can I store this values into the structure ?

from pytodoist import todoist
user = todoist.login('####','""""')
projects = user.get_projects()
for project in projects:
    print(project.name)

tasks = user.get_tasks()
for task in tasks:
    print("Project: ", task.project.name, ", Task: ", task.content, ", Date: ", task.date_string, ", Priority: ", task.priority, ", Added in: ", task.date_added)

My key_pais: The strings that I've in For loop "Project", "Task", etc.
Values: the values from taks

Many thanks!

At first remember that a dictionary is just bunches of keys and value pairs...

So, if the key is $a$ and the value is $10$ , you can access the value by doing d[a] (d is the dictionary)

Beaware that there is no order to storing the values, you can use OrderedDict for that..

Also you can have two values for the same key in a simple dictionary.

Here's the complete answer


The model of LSTM with more than one unit In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew below:

But the picture above only has one unit, so I am wondering if it has more than one unit, for example memory unit=2, what will this model look like.

or

or can someone point out the wrong part, or give a sample of visualize architecture of LSTM model with multiple units, thanks!

perhaps my description was not clear enough, so i try to give a more clear description and graph of the question:
I try to illustrate the question as the picture below, and my question is: is the architect correct when LSTM model has 2_units,2_layers.

The last image that you draw, where (at a given timestep) each cell of the second layer receives input from all cells in the first layer, is the right one.

You can think of recurrent layers as fully connected layers which receive sequences of input and apply their transformations at each timestep. They just receive other inputs depending on their previous inputs but the connectivity between layer is exactly the same as a fully connected : each unit of a given layer performs a weighted average of the activations of all the units in the previous layer (or of the inputs if it's the first layer).


Handling Numerical Categorical Column in ML models in Python When I was exploring the titanic dataset to estimate the probability of a person of surviving using the Logistic Model, I realized there are two ways of handling numerical categorical variables :

Use them as is in the dataset ie don't convert them into dummy variables. See the PClass below:

Convert the numerical categorical variable into dummy variables as below:

I had assumed both these would bear the same result but the model results are different using both these approaches. So, I wanted to understand the logic working behind both these approaches and which one should be preferred?

If you don't encode numerical categories with dummy variables, some models will end up being trained to use an ordering of numbers (e.g. 1 < 2 < 3 < 4 < 5 < ...) in their predictions. Whether or not this is desirable or useful depends on the context, and in particular on the meaning of the numerical categories and the model and implementation being used.

In your example, if the numerical labels for the Pclass categories represent "poverty class", with lower numbers corresponding to less income and higher numbers corresponding to higher income, then the ordering of the integers is really related to a meaningful ordering of the categories ( "1 < 2 < 3" and correspondingly "poverty class 1 is less wealthy than poverty class 2 is less wealthy than poverty class 3"). In this case, using the numerical encoding as it is doesn't appear to do any harm, and may even help some models. For example, a decision tree could learn rules like

if (Pclass >= 2):
    prediction = "survive"

whereas if the categories were encoded with dummy variables, this rule would instead look like:

if ((Pclass_2 == True) or (Pclass_3 == True)):
    prediction = "survive"

For some implementations/models, the first rule may be more likely to occur than the second, be more cost effective (just storing dummy variable data is more costly, let alone the effect it might have on model performance), generalize to new data better, etc.

On the other hand, if the numerical labels for the Pclass category represented, say, location of the room on the boat in which a person was staying (with 1 representing right next to the iceberg hit, 2 representing the super fancy ballroom with it's own getaway boat, 3 representing near-ish to the iceberg hit, 4 representing mid-tier, etc.), then the ordering of the integers appears to have no meaningful relationship to an ordering of the actual classes (that I can see anyway!). If one were to use the encoding for the numerical categories as they are, many model implementations could learn patterns in a "bad" way. For example, a decision tree could learn that the people near the ice berg hit had poor survival rates and end up producing the rule:

if Pclass <= 3:
   prediction = "does not survive"

This is "bad" both because it assumes being near the iceberg hit has something to do with the numerical encoding being less than 3 (it may not, for example future data might include many new category labels greater than 3 all of which encode rooms near the iceberg hit), and because the rule seems likely to produce many incorrect predictions for those with Pclass=2. If the categories were encoded with dummy variables, the rule learned might instead be:

if ((Pclass_1 == True) or (Pclass_3 == True)):
    prediction = "does not survive"

This rule is "better" in this context, as it does not share the two issues the previous rule did.

In general, I suggest the following rule of thumb:

  Use dummy variables to encode numerically labeled categories if the natural ordering on the numerical labels has no meaningful interpretation in terms of the categories.

All else being equal, I tend to prefer using dummy variables by default when I have the space in memory, and then think on whether it makes sense to use integral encodings. In your situation, what does the Pclass represent, and is the ordering of the numerical categories actually meaningful to you? If not, I would recommend using dummy variables.


Which graph will be appropriate for the visualization task? I have some terminal charging values for US and CHINA
comes in a pandas DataFrame like the following,

       value country
0      550.0     USA
1      820.0   CHINA
2      835.0   CHINA
3      600.0   USA
4      775.0   CHINA
5      785.0   USA
6      790.0   USA

This is the sample data and I have in total 5K+ entries. The data is cleared for the outliers and needs to be visualized. What kind of visualization can I use to plot my data meaningfully?

Maybe you can try something like this?

df.hist(by='country', bins=50)

If you want to plot these histograms in the same figure, checkout this.

References

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html


long shape line in tSNE plot My question is similar to this post: What does the long curve-shape t-SNE mean?

but the problem is that my data is not time series, but I also get these long shape line in tSNE plot .

I don't know what that means or how I can get rid of that. I suspect some structure in the data causing that. But I could not find a useful resource explaining what structure in data could cause this.

My data is single cell data, each row is a cell, each column is a biology marker (e.g. CD3, CD4), so the data is the marker expression on each cell.

Welcome to the community Dalide!

There is a sequential pattern in your data. As a non-Bioinformatics guy, I simply googled and saw that biomarker are also vastly used for learning the Progression of a disease (e.g. Alzheimer). I hope I am right about example and if I am, then it is not a big surprise that you see those curves which might indicate the progress of something. If you need more precise answer you better give some more info about the origin of data (or a link to your data if it is publicly available).

From ML point of view, about which I fortunately know a bit, those curves in tSNE are caused by sequential patterns.

Fun: As an equally surprising example, once I found sequential pattern in characteristics of different Wines!! (each row a Wine and each column a wine property like Alcohole%, density, etc.)


Fast introduction to deep learning in Python, with advanced math and some machine learning backgrounds, but not much Python experience I've the following somewhat unusual background and I've managed (probably by luck) to get an industry job of a computer vision researcher using deep learning.

My background: I've a PhD in pure math, and have the following machine learning experience: linear and logistic regressions, support vector machines (SVM), linear and quadratic discriminant analyses (LDA/QDA), decision trees, PCA. I'm familiar with the mathematical theory of all the above, and implemented the LDA/QDA, SVM, PCA in matlab, where I'm proficient in.

However, my experience with Python and deep learning are very limited to zero. I don't know scikit learn, only somewhat familiar with NumPy arrays, but still have problems with the basics like concatenation with empty arrays. I don't know for examples the data structures like classes or lists in Python-just to give you an idea. I know what a neural network is, but don't know anything else, e.g. backpropagation or autoencoder. Hence I don't know the theory of deep learning as well.

Time constraint: So I'm in a situation where I'll have to gain some hands on experience in both Python as well as deep learning, possibly in 2 months (the company has a 4 months trial period and use only Python, but they'll get a sense of my learning in 2 months, hence I've to learn fast).

Question: Given my expertise and time constraints, could you mention a road map (if possible) so that I can get a fast introduction to deep learning and its Python libraries (e.g. Tensor Flow or Keras etc.)? I know there're tons of resources out there, but not all of them can be learnt within a short time span.

Thanks so much in advance!!!

Stanfords CS231n has a great python+numpy tutorial.
http://cs231n.github.io/python-numpy-tutorial/

As for Neural Networks and Keras + Tensorflow, I can recommend the Deep Learning specialization on Coursera. It is free for one week. If you do not want the certificates you can download all videos and stop.

If you want the certficates you can enroll in all courses simultaneously and finish them faster than advertised. They assume 3-6h per week for 16 weeks. If you have 6h per day, it is possible to go through a weeks content per day. Andrew Ng speaks clearly and the videos are understandable at 1.5x-2x speed.

As for the roadmap, i would suggest to

start with python + numpy until you have a basic understanding of array slicing and read up on unknown commands when you encounter them.
start a DL course of your choice but do not skip classes. That is I think it is important to know the basics even if you will only use high level frameworks later on.


Boruta Feature Selection package I am using Boruta feature selection (with Random forest) to decide the important features in the below data set.

   Gender Married Dependents     Education Self_Employed  ApplicantIncome  \
0   Male      No          0      Graduate            No             5849
1   Male     Yes          1      Graduate            No             4583
2   Male     Yes          0      Graduate           Yes             3000
3   Male     Yes          0  Not Graduate            No             2583
4   Male      No          0      Graduate            No             6000

 CoapplicantIncome  LoanAmount  Loan_Amount_Term  Credit_History  \
0                0.0         NaN             360.0             1.0
1             1508.0       128.0             360.0             1.0
2                0.0        66.0             360.0             1.0
3             2358.0       120.0             360.0             1.0
4                0.0       141.0             360.0             1.0

Property_Area Loan_Status
0         Urban           Y
1         Rural           N
2         Urban           Y
3         Urban           Y
4         Urban           Y

Please help me in clarifying the below doubts
1) whether I need to convert the all categorical variable into numeric variable (using one hot encoding) before applying Boruta?
2) Whether the NA values would be taken care by Boruta or do we need to remove NA values before feeding into Boruta ?

In case of Regression problem, whether the Boruta approach (Random forest classifier -> Boruta) remains the same as in classification problem ?

Thank you.

In R, Boruta relies on the ranger implementation of random forest. So:

Converting input variables from categorical to numeric is not necessary.
You will need to address NA values prior to running the algorithm.

Be aware that Boruta can be very slow!


How to handle missing data for machine learning I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.

As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:

dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility

While the last one would be the field I like to predict at some point.

There are three main approaches to handling missing data.

Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.
Ignore - some methods can just ignore missing data, and not use it in the model at all
Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output.


What is the correct way to call Keras flow_from_directory() method? In the following article there is an instruction that dataset needs to be divided into train, validation and test folders where the test folder should not contain the labeled subfolders. Instead it should only contain a single folder (i.e. Test_folder).

When I use the following code, I get the output message refering that no image were found.

Ver.1:
test_generator = test_datagen.flow_from_directory(
    "dataset\\test\\test_folder\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=1,
    class_mode=None,
    shuffle=False,
    seed=10)

Output message: "Found 0 images belonging to 0 classes.".

Instead, if I use the same folder structure (dataset\test\class_a\test_1.jpg etc) as in the train and validation folders, everything seems to be OK and I manage to evaluate my model.

Ver.2:
test_generator = test_datagen.flow_from_directory(
    "dataset\\test\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=32,
    class_mode='categorical',
    shuffle=False,
    seed=10)

Output message: "Found 1500 images belonging to 3 classes.".

I also tried the recommendation where 'classes' attribute is specified but still 0 images were found.

Ver.3:
test_generator = test_datagen.flow_from_directory(
    "dataset2\\test\\test_folder\\",
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=1,
    classes=['test'],
    class_mode=None,
    shuffle=False,
    seed=10)

Output message: Found 0 images belonging to 1 classes.

Thus, what is the correct way to call flow_from_directory() method and why am I getting the message that no files were found? Is my model not correctly evaluated when I use the Ver.2 solution?

Please find a working solution here.

The generators look like:

# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=batch_size,
        # Since we use categorical_crossentropy loss, we need binary labels
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

Make sure you have your images stored in the correct way. E.g.

...images/train/class1/
...images/train/class2/
...images/val/class1/
...images/val/class2/

The generator function does need this structure. So make sure there are subfolders for each class in the train/test directories.

You can also use the data generator function for prediction as described in this post.


How to extract paragraphs from text document? I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like:
Sample pdf file is here

FORWARD-LOOKING STATEMENTS

In this Annual Report, we have disclosed forward-looking information to enable investors to

comprehend our prospects and take investment decisions. This report and other statements written and oral – that we periodically share contain

forward-looking statements that set out ...//Content Continued like in this format until the paragraph ends//

HERO
O
FOREVER
Now if check out the actual report on above link the text under Notes to Account and Independent Auditor's Report follows the almost similar structure in all companies reports in India. Only the Chairman's message or Boards reports vary but usually talks about growth, performance, future, investments etc.
So is there any way to extract only the paragraphs/multiple paragraphs combines into single(if continuation of same information) which contains useful information. I have searched but i find most of work on paragraph/document summarization but donot find something like extraction of actual continuous blocks of text data from documents.
Note : There is lot of noisy data(actual data in pdf but they are like page number, if company added some text in header/footer apart from title of page etc) between paragraphs which is generated while converting from pdf to text like:

Multi-line paragraph
R

N

O

I

WE W DS REMA INDS

BRANVANT IN M

RELE MARKETSDES

AND SS DECA

ACRO

6

Next Multiline paragraph

It's not always possible to extract paragraphs from a pdf since sometime paragraph are split into multiple pdf frames so pdftotext split them into different paragraph even if there are actually linked. Similarly some frames ends collocated even they represent different information like the menu in the example pdf.

Here is a simple approach to split a text file into multiple paragraph using empty lines:

def txt2paragraph(filepath):
    with open(filepath) as f:
        lines = f.readlines()

    paragraph = ''
    for line in lines:
        if line.isspace():  # is it an empty line?
            if paragraph:
                yield paragraph
                paragraph = ''
            else:
                continue
        else:
            paragraph += ' ' + line.strip()
    yield paragraph


Given data that is labeled as outliers, how can I classify data as outliers? I have a dataset that is a mixture of sparse binary features and quantitative features. I only have definite outliers labeled. How should I approach trying to classify unlabeled data?

I considered using OSVM or other methods of one-class classification.

However, in my data the normal data points are clustered close to the mean. The outliers are generally points that deviate from the mean in any direction. My problem is that the outliers form a sort of high dimensional doughnut around the normal data.

Considering that the deviations occur in all directions, what algorithms would be best suited to the task? Keep in mind that I have significantly less normal labeled data points for training although the normal points will outnumber the outliers in the unlabeled data.

PS I posted this question on Cross Validated as well. Which site should this question be posted on?

EDIT:
Mahalanobis is able to work fairly well. However, I have the labeled outliers. Is there someway I could use them to improve accuracy?

If you are sure that your data are actually normally distributed and that your outliers actually form a high-dimensional ring around your "good" data, you simply require a distance-metric, e.g. Mahalanobis distance is suitable for normal data, and determine the threshold value for distance from the mean to consider all points beyond the high-dimensional ball or ellipsoid depending on the variance structure outliers according to your description.


i need a characters /letters dataset for matlab My final project is number plate recognition.i need a data set of A-Z characters and 0-9 letters. i donot find it on any website give me a data set or send me a link. i have to make a neural network in which i put that data set to train and test my data.

You can find car license plat Here  or you can collect them from this web site but it will take you  a while.