Convolutional neural network with 1 channel images/1 input channel I'm following a tutorial on tensorflow using a convolutional neural network for images, but I'm looking to do it with grayscale images. How would the code posted there be different if it was for grayscale images instead of colored images with 3 channels?
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))

The following will be the updated code for grayscale images:
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))

Why?
Because for an RGB image, there are 3 channels, 'R', 'G' and 'B'
So, the input shape will be (height, width, 3)
But since grayscale has only one channel, the input shape becomes (height, width, 1)
Note that if you are using Keras with Tensorflow backend, then the data_format is channels_last, which means that the input shape should be  (height, width, channels).
Otherwise, if you are using Theano as the backend, then the input shape should be (channels, height, width) since Theano uses the channels_first data format.
Hope this is of help.
Please upvote if helpful.


Compare Neural Network generalization results I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both).
Since in bayesian regularization I have not the validation set, how can I compare the generalization capability of the networks obtained with the two methodologies?
Thanks

As Neil said in the comments, split out a test set (data you don't use in training either model) and see how each trained model performs on the test set.


Xgboost in R: Dealing with extreme class imbalance My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result,

Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance?

I'm a newbie in the field. Any help would be much appreciated. Thanks!

The drop from 64 to 50 doesn't seem like overfitting, if your test set was properly held back from model development.  (And from the confusion matrix it looks like you haven't made the fairly common mistake of upsampling before splitting.)

If the "completely different test data" is not identically distributed to the training data, that can account for the discrepancy.  There aren't universally good ways to account for failure of the iid data assumption; you could try to underfit the model more, on the assumption that there's some underlying large trend that's constant across datasets and the model is finding (and overfitting to) some additional, more localized, trend in the train/validation set.


True positives and true negatives, F1 score: multi class classification I have 4 classes for an application of classification of animal kingdom: 1 --> invertibrates; 2 --> vertibrates; 3--> mammal; 4 ---> ambhibian. Given a mixture of images the objective is to identify mammals correctly. In the confusion matrix for this example will the TP denote the class 3 (mammal)?

Q1: Therefore, in general TP is the class category which is of utmost significance? What if all the classes are equally important, then how to denote TP and TN?

Q2: How to calculate the F1 score? should it be done separately for each class? If then there will be multiple TN's w.r.t each class !!

Can somebody please help clear these confusions? Thank you.

In a multiclass problem there is one score for each class, counting any other class as a negative.

For example for class 1:

TP instances are gold standard class 1 predicted as class 1
FN instances are gold standard class 1 predicted as class 2,3 or 4
FP instances are gold standard class 2,3 or 4 predicted as class 1
TN instances are gold standard class 2,3 or 4 predicted as class 2,3 or 4 (here errors don't matter as long as class 1 is not involved)

In other words, the problem is evaluated as if it was a binary classification problem for every class individually. Doing the same process for every class independently (since the status of an instance depends on the target class), one obtains a different F1-score for each class.

After that, one generally calculates either the macro F1-score or the micro F1-score (or both) in order to obtain an overall performance statistic.


Standardscaler() not standardscaling? I have following pipeline:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

Originally, the data looks like

After applying the estimator/StandardScaler()

X_train = prepare_data.fit_transform(X_train_raw)

Why are the values mostly/always negative now?

When I have a look at another variable in X_train it looks as it should (I guess):

Before

after

You got outliers. Standard scaler scales each attribute independently to center at zero and unit variance. Either deal with your outliers or use some more robust scaler. Try instead to plot each attribute with separate box plot.


Reason for having both low loss and same predicted class? I'm training a cNN for binary classification. I used a batch size of 128, and the loss is decreasing and accuracy is increasing over epoches. The accuracy reached over 0.99 eventually, and the loss reached below 0.3. But after a few more epoches, the model converges to loss 0.6 and accuracy 0.5. An inspection of the model shows that it always predicts 0.5. I'm using binary cross entropy as the loss function. For each epoch, all data points are shuffled. I'm using SGD, learning rate is 0.01.

Am I hitting a local minimum with low accuracy rate but fairly good loss value? What is the suggested approach to deal with this? Also, why is it possible to have a low loss function with one single predicted class?

From your description, it might be an issue of too high learning rate. When this happens, the weights cannot reach closer to the minimum and the loss does not go down.

Accordingly, I would suggest reducing the learning rate to 0.001 and using Adam.


Word Embedding or Hash? In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of IDs.

I am preprocessing the data for LSTM, and I will do word embedding on the text column.

My question is, should I add the follower IDs to the word embedding of the text column, or should I hash the follower IDs and add an extra LSTM input layer for the IDs?

Thanks in advance!

It depends…

The general rule of thumb is that there should be at least 40 occurrences of an item to train an embedding model to find a robust representation. If most follower IDs repeat then an embedding model can learn which ones co-occur. If follower IDs are sparse then hashing (which randomly assigns numbers) is a better choice.

Which method is better is an empirical question. You can create both models, benchmark, and then choose the data processing pipeline that is best for your task.


How exactly do Gaussian Processes (square dist kernel) enforce smoothness? (Aka how are they computed to do so?) From:
http://www.cs.cmu.edu/~16831-f12/notes/F10/16831_lecture22_jlisee/16831_lecture22.jlisee.pdf

"Gaussian Processes artificially introduce correlation between close samples in that vector in order to
enforce some sort of smoothness on the succession of samples."

But how is this computed? Is the function f(x) ~ GP(mu,k(x,x')) performed incrementally?
e.g.
the n'th calculated value f(xn) uses values f(x-1)...f(x-n) to compute its mean and variance?

It's by definition, when you fit a guassian process you specify the mean function m(x) and the covariance function (or kernel) k(x,x'). Often the mean function is 0 and the covariance is the radial basis function or squared expential kernel which is smooth (in fact infinitly differentiable).


Training Deep learning and validation loss I'm trying to replicate result of a paper. The paper is a U-net for De-noising of some images. So basically I have a simple U-net that I give noisy data as input and have denoised data as the wanted output (use l2/MSE loss) . So, in the paper and generally in most papers like this (deep learning applied to medical imaging), they say they run the model for something like 300 epochs or they say they run it like ~50 hours. My question is that aren't they supposed to put call backs for the validation loss, so once the validation loss stops improving they stop the training, otherwise the model will overfit? Also does the number of batches in an epoch matter. If because of memory constraints I use 2 batch sizes per epoch, would it be a bad practice?

Using validation loss to determine when to stop iterating is a good strategy when you have a lot of labeled training data. However, in medical contexts, it's often the case that labeled data is expensive and/or difficult to come by. You mention that you only have 1000 samples -- this is a pretty small amount of data to be training a deep net with! Setting aside 100 or 200 images to form a validation set will probably hurt the model. You can use a validation set to get an estimate of how many epochs you should train for, and then train on the whole dataset for that many epochs.

In regards to your batch size query: I would recommend working with smaller batch sizes of around 32-128 images. In my experience, the networks converge more quickly when using smaller batch sizes.


Advantages of monotonic activation functions over non-monotonic functions in neural networks? What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks?

Do they perform better than non-monotonic ones?
Is this mathematically proven?
Are there any papers/references that are related to this?

I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian kernel, symmetric around $x=0$ but slides off towards $f(x)=0$ if x strays away from 0 on either side. If we have a sample that we feed into our network that performs poorly when our activation is high, we want to change the input of our node to give a lower activation. In case of a non-monotonic activation, whether we want to decrease or increase the input depends on whether the input was positive or negative, and is mostly dependent on our weight initialization.

This makes learning more difficult, because if another sample also  needs it to be lower but was on the other side of the top, backpropogation will attempt to map the input to the other side. Most of the time the best solution will be to put everything on one side of the top, making it monotonic again. Another way of looking at it is that monotonic are somewhat one-to-one (not entirely true, for example ReLU). This means that two very different inputs don't map to the same output unless everything in between also maps there.

Here was a similar question with some links: (Why) do activation functions have to be monotonic?


Normalization vs standardization for image classification problem For day and night image classification, is it better to normalize or standardize images? In general, when should I use each method? I am interested in with example why one method is preferred over another one?
Here, by normalization, I mean dividing pixel values by 255. Standardization means subtracting the mean value of pixels and then dividing by standard deviation. See the code samples below
# normalization
datagen = ImageDataGenerator(rescale=1.0/255.0)

or
# stardartization
datagen = ImageDataGenerator(samplewise_center=True, samplewise_std_normalization=True)

To be able to answer your question, firstly we should take into account the differences between Normalization and Standardization.
Normalization: normalized_value=(raw_value-min)/(max-min)
Standardization: standarized_value=(raw_value-μ)/σ
Also, one additional famous method is there which is centering. Where you subtract the mean from the pixels.
In general, all of these serve one common and crucial purpose: to provide "fair conditions" for all features. I.o.w. since, for each feature, we have many shared parameters (e.g. learning rate) instead of custom parameters, we need to have similar ranges. Therefore, sharing of parameters would not be optimal because a parameter (e.g. learning rate) would not have the same power over features with different ranges. For example, converging to the global optimum value might be effective for one of the features, however, another one might require a smaller learning rate value because of its range.
Additionally, smaller values increase the calculation process, fastens the converging. Also, interpretation and comparison of weights would be easier if the features are standard. If ranges are dissimilar one of the coefficients might be very big while another one might be comparably very small.
However, I want to draw attention to your Normalization technique where you divide pixel values by 255, which just lowers the range from 0-255 to 0-1, but the spread of pixel values over the range is kept the same. Normalization is mostly used when we want to have an abounding range for features, however, RGB has already an abounding system of 0-255. However, theoretically, this would still help to fasten the training process.
It is hard to say that one of these (Normalization or Standardization) is better than the other because one might beat the other depending on the scenario. Commonly, both techniques are tried and compared to see which one performs better.
In many cases, to have a more accurate model, usage of these techniques is a must. Let's say you have two images where one of the images is dark where all pixels have values close to each other (narrow range, small deviations, high kurtosis), but another one is a complex image where parts of the image dissimilar to each other in terms of pixel values (wide range, large deviations, low kurtosis). In such a case, using normalization would help a model to better understand and capture the relations inside of the images.


Non trainable problems Before facing this question, I always thought non-learnable problems are those which the provided data for the problem has high amount of outliers, those which don't have sufficient features or those for which the Bayes error is large because of having same features with different labels. As you can see, it seems that the data is fine because the learning should be comparable with human level inference. A human can distinguish between even or odd numbers by just looking at them. I know that we as human begins, do modulus two operation in our mind to decide whether a number is even or odd, the feature extraction part, but we are doing that with just the number itself. It is clear that we can not find a decision boundary to be able to generalize because the inputs have alternative behavior. 1 is even 2 is odd, 3 is even 4 is odd and all the other numbers in this manner. I want to know this kind of problem ,which does not have the mentioned problems which may cause an algorithm not to learn, has any special name?

In this discussion, it is described how a neural network that distinguishes odd and even numbers can be constructed.
Your question can be rephrased more generally: Is there a function that cannot be learned by a machine learning algorithm. This question was discussed here and here. Both discussions refer to the Universal Approximation Theorem that basically states that any computable function on a given finite range can be approximated by a neural network. So, what is left is uncomputable functions or undecidable problems. These cannot be learned by a machine-learning algorithm.


Tool for test/train automation I need to test different datasets as well as different algorithm implementations. The current workflow looks like:

Perform feature extraction from train set
Train classifier on this features
Feed this classifier to production code
Run production code on a test set, feeding samples one by one
Take results of production code and convert to the same format as test dataset
Show statistics

How to automate this as much as possible? I currently use a bunch of bash/python scripts. I've looked through sklearn pipelines, but not sure they would give any benefit except for running functions in a line.

Maybe it is a bit too much just for this use case, but I have good experience with airflow.

It is an Apache project and quite helpful to automate some stuff.

Otherwise there are a lot of commercial platform helping you. Just google for data science platform.


Plot hurricane track Hello folks,
            Recently while going through a research paper I happened to come across a plot showing the hurricane track(figure attached). Can anyone say how it is plotted or anyone has any experience of plotting such figures? Also, I want to know what are such types of plots called.

.

The information for this plots is provided by a meteorological service in a Geospatial Format.  A common format for weather data is ESRI shape file format (SHP). I gave a presentation on an introduction to modelling hurricane losses a couple of years ago (YouTube). There is a link to the code here.  An example example storm track with intensity bands that was produced is:

Hope this helps.


Understanding Classifier performance on text data I am working on a multi-label text classification problem(Total target labels 90). The data distribution has a long tail and class imbalance and around 1900k records. Currently, I am working on a small sample of around 100k records with similar target distribution.I am using the OAA strategy (One against all). I have tried many algorithms on data.

Currently, each label has atleast 5000 data rows. The class imbalance is high with around 80k records for the most common label and the most rare with just one data row which I have not considered in the modelling. This dataset contains text from academic journals. It has Title and Abstract  columns.

I am using HashingVectorizer(number of features 2**20, char analyzer) to generate features and TSVD to reduce the dimensionality(n_components=200).

LinearSVC(class_weight='balanced') # Got many warnings that it could not converge. I came to know that it may due to data not scaled properly. How can I scale text data??
LogisticRegression(solver='lbfgs') # Converged very quickly
RandomForestClassifier(n_estimators=40,class_weight="balanced") # Train time ~2hr

I noticed that LinearSVC has good recall(less false negatives) while Logistic and RF has good precision(less false positives) scores. Can anyone help me in identifying the reasons behind these scores and how can I improve them.

Currently, I am not using deep learning/transformer models due to limited computation resources.

As the data is imbalanced and skewed towards few classes, that's why RF and Logistic results are biased causing high FP values, therefore high precision and low recall.
SVC on the other hand might have tried to create hyperplanes to get most out of the other side of curve, thereby producing different results.

To improve the results, try to use non linear kernels and also try to balance the input data (by scaling etc) before feeding to classifiers.


What is the meaning of likelihood? I am studying Bayes probability applied to machine learning, and I have encoutered the concept of likelihood, which I don't understand.

I have seen that the Bayes rule is:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

where $P(B|A)P(A)$ is the conditional probability, while $P(B|A)$ is the likelihood, but I don't understand what the likelihood is.

Can somebody please help me?

All of the answers here, including the accepted one, are conspicuously confused. I down-voted the accepted answer but downvotes of users who lack reputation in this "community" are not counted. I have a reputation of more than 200,000 (two-hundred-thousand) on math.stackexchange.com and I have a Ph.D. in statistics, but none of that counts here.

The question says $P(B\mid A)$ is the likelihood, but it ought to say instead that $A\mapsto P(B\mid A)$ is the likelihood. I.e. it's this probability as a function of $A,$ not of $B.$

It also says $P(B\mid A)P(A)$ is the conditional probability. That is not right. $P(B\mid A)$ is a conditional probability. $P(B\mid A)P(A)$ is $P(A\ \&\ B).$

Now suppose you have
$$
\Pr(A_1) = \frac 1 {20}, \quad \Pr(A_2) = \frac 2 {20}, \quad \Pr(A_3) = \frac{17}{20}
$$
and $A_1,A_2,A_3$ are mutually exclusive. This is a prior probability distribution.

Further suppose that
$$
\Pr(B\mid A_1) = \frac 9 {10}, \quad \Pr(B\mid A_2)= \frac 2 3, \quad \Pr(B\mid A_3) = \frac 1 2.
$$
Note well:

These three probabilities do not add up to $1.$ What is expressed here is not a probability distribution.
It is a function of $\text{“}A\text{''},$ a variable whose value may be either $A_1,$ $A_2,$ or $A_3.$
The likelihood is $\Pr(B\mid A)$ as a function of $A,$ not of $B.$

Bayes's theorem says: When one multiplies (pointwise) the prior probability distribution by the the likelihood and then normalizes, one gets the posterior probability distribution, i.e. the distribution conditional on the data. The "data" is the observed event $B.$

Thus
\begin{align}
& \left( \tfrac 1 {20}, \tfrac 2 {20}, \tfrac{17}{20} \right) \times\left( \tfrac 9{10}, \tfrac 2 3, \tfrac 1 2 \right) \\[8pt]
= {} & \left( \tfrac 9 {200}, \tfrac 2 {60}, \tfrac{17}{40} \right) \\[8pt]
\propto {} & \left( 27, 20, 255 \right) \\
& \text{(Here I multiplied all three components} \\
& \phantom{(} \text{by 600, which is the l.c.m. of the denominators.)} \\[8pt]
\propto {} & \left( \tfrac{27}{302}, \tfrac{20}{302}, \tfrac{255}{302} \right) \\
& \text{(Here I normalized, i.e. divided by} \\
& \phantom{(} 27+20+255 = 302 \text{ so that the} \\
& \phantom{(} \text{sum of the three components is $1.$)} \\[8pt]
= {} & \left( \Pr(A_1\mid B), \Pr(A_2\mid B), \Pr(A_3\mid B) \right) \\[8pt]
= {} & \textbf{the posterior probability distribution.}
\end{align}

Often one sees something like
$$
X\sim\operatorname{Binomial}(4,p),
$$
so that, for example
$$
\Pr(X=2) = \binom 4 2 p^2 (1-p)^{4-2}.
$$
Then if the data consists of the observation that $X=2,$ then the likelihood function is
$$
L(p) = \binom 4 2 p^2(1-p)^{4-2}.
$$
This is a function of $p,$ not of a variable whose values can be the possible values of $X,$ which in this case are $0,1,2,3,4.$


Pandas Data Frame: Calculating custom moving average I have a time series containing stock price data.
I would like to calculate the Money Flow Index (MFI) for each row.
Given that the MFI, uses the previous approx. 14 rows to calculate the current MFI, what would be the best approach to doing this?
The below calculates the current MFI for a given DataFrame, but I would like to implement this for each row in the data frame
# Typical Price =(high price + low price + closing price) / 3
tp=(hst['High']+hst['Low']+hst['Close'])/3
tp=tp.to_frame(name='Price')

# Raw money flow = typical price x volume
tp['Raw']=tp['Price'] * hst['Volume']

# Identify flow (upwards or downwards)
tp['Direction'] = np.where(np.nan_to_num(tp['Raw'].shift(1))>tp['Raw'], 'up', 'down')
mfr=0
# Money flow ratio = (14-day Positive Money Flow) / (14-day Negative Money Flow)
if tp.loc[tp['Direction']=='down']['Price'].sum(axis=0) != 0:
    mfr=tp.loc[tp['Direction']=='up']['Price'].sum(axis=0) / tp.loc[tp['Direction']=='down']['Price'].sum(axis=0)

# MFI = 100 - 100 / (1 + money flow ratio)
100-(100/(1+ mfr))

I managed to work this using the 'apply' function, but this function only allows for 1 argument, whilst I would need to pass more than 1 argument to compute the above.

Worked with the below to add a column that calculates the above function

quote['MFITEST']=quote['Date'].apply(lambda x: calc_mfi(quote,14,x))


Dealing with irrelevant features in dataset (Homework) I have a specific question pertaining to one of my machine learning homeworks.

Basically, we are required to build a model that takes a 5000*10000 dataset X (5000 examples each with 10000 features), and predict Z, which is a 5000*2 Matrix. The dataset Z is synthetically generated from X, however, it only depends on two of those features from X. That is, only 2 of the 10000 features are relevant. Z has only 3 possible classes, [0,0], [1,0] and [1,1], and is not balanced. All of the features of X are sampled equally from a normal distribution, the only distinguishing feature is that Z is calculated from just 2 of them.

I thought that using PCA might be a good idea, and tried this. However, I had two problems. Firstly calling np.linalg.eig on X@X.T or X.T@X takes an unreasonable amount of time. And secondly, when I used np.linalg.eigh, it was slightly less computationally prohibitive, but many of the eigenvalues were actually very large. This makes sense to me as well since nothing actually distinguishes X except their relationship to Z.

I then built a simple neural network, linear -> tanh -> linear -> sigmoid. This descended fine, however it just learn to mirror the underlying distribution of the dataset. That is, it didn't learn which parameters of X were relevant and instead learnt to predict such that the overall error based on it (seemingly) randomly guessing was minimized.

Can anyone suggest some techniques that might help me to solve my problem?

This could be many things.

Since you want to calssify the data into three classes, I would use one-hot-encoding, rather than the binary enumeration, because you kind of introduce the information to the model that [1,0] and [1,1] are "closer" than [0,0] and [1,1] for example. Unless this is actually the case, better encode the classes like [1,0,0], [0,1,0] and [0,0,1].
That the eigenvalues in your PCA are large can have two reasons: a) you need to divide XX^T by the number of data points, as the PCA is computed on an approximation of the covariance matrix, which in turn is based on an expectation value
b) not sure if you did, but make sure to center your data first
I think you discovered this yourself already, but if the PCA is useful on your problem depends actually on your data distribution and how you computet the Z. It sounds a bit as if there was no covariance between the features, so there are no primary components to discover. And in case you picked the features to compute Z randomly, you have no guarantee that the primary components tell you anything about Z.
Maybe do a PCA on the data and append Z to X. This might help with dimensionality reduction or identifying relevent features.
Since you have many features and use saturating activation functions, it is important that you normalize the data before passing it to the model and that you initialize the weights properly, otherwise the neurons will saturate and the gradient will be almost zero. Also training might be very slow anyway, as with all the "useless" features you introduce a lot of noise.

I hope this helps. If not, feel free to post some more information on your training data and the neural network you wanted to use.


NLP Library or routine for Quantitative relationships I'm searching for generalized methods and libraries to extract quantitative relationships between concepts in natural language.  For instance

Exam score greater than 90%
Exam scores of 90% or greater

Should extract to a structure like

Exam Score,>=,90%

I began with a simple regexp / dictionary method, but it feels very primitive.

It depends on how "generalized" you want your methods. If you have known and fixed set of quantitative relationships (=, <=, >=, …), then you can consider it a classification problem. Given some input string, which category does it most likely belong to? The same goes for entities. Are you assuming it will always be two entities?

If the system should handle an unlimited number of entities with unknown and possibly infinite number of relationships than the problem quickly becomes intractable.

If you assume there will always two entities, then you can apply Resource Description Framework (RDF). RDF goal is to find semantic entities and their relationships. RDF finds triples: a subject, a predicate, and an object. The predicate is "quantitative relationship" between the subject and object. There are a variety of frameworks for extracting RDF from a sentence. The most effective ones depend on building a parse tree of the natural language.


Problem using Anaconda I have installed Anaconda, but every time I open Terminal I have to go give the command:

export PATH=~/anaconda3/bin:$PATH

How can I fix this issue?

You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile.

Open a terminal and follow these steps:

open the terminal profile: gedit ~/.bashrc
at the end of the file, add: export PATH=~/anaconda3/bin:$PATH
save the file (control+s) and close it
load the changed profile: source ~/.bashrc

Now this terminal window (and any new ones) should find Anaconda! See which version of Python is now the default, using which -a python. First in the list should be something like /home/username/anaconda3/bin/python

The Anaconda setup/installation usually asks you if you want to prepend Anaconda to start of your path, so whoever installed it must have said no or skipped that step.


XGBoost: predictive, descriptive (or both) model? I have trained an XGBoost model for prediction. The algorithm is able to calculate variable importances.  I was asked why I have not analyzed these variable importances and I did not because as I understood XGBoost is rather a predictive than a descriptive model. I am not sure what extent theses importances can be used to capture real importance (I understand that it may be dependent on the context). Am I right that XGBoost is rather a predictive model?

You're right that it is a predictive model, but analysing feature importance, partial dependence and other metrics/plots are intended to allow you to gain an understanding of the classification process of models which are often very black-box in nature.

Yes, you can use this to gain some understanding of the underlying data, but I'd be more inclined to say that the primary reason for looking at feature importance is to understand your model rather than to understand your data.


Very simple real-valued time-series dataset for RNN prototyping Is there a simple real-valued time-series dataset on which a vanilla RNN model can be trained. With "very simple" I mean only two to four real-valued inputs per time step and a single real-valued output per time-step.

Background: I am doing research in the intersection of machine learning and formal methods. To test a new technique for formally verifying RNNs, we need to start with a quite simple setup.

Thanks!

You can take a look at some of these datasets in UCI:

https://archive.ics.uci.edu/ml/datasets.php?format=&task=&att=&area=&numAtt=10to100&numIns=&type=ts&sort=attUp&view=table

These are quite straight forward, although they might contain more columns that 4 or 5. But they are quite simple.

Or you can take a look at this:

https://www.kaggle.com/uciml/electric-power-consumption-data-set

This is also based on UCI data.

Overall you can take a look at datasets available on Kaggle, you can probably find something there.

https://www.kaggle.com/datasets

Hope this helps.


Simple implementation of Apriori algorithm in R I am preparing a lecture on data mining algorithms in R and I want to demonstrate the famous Apriori algorithm in it.

My question
Could anybody point me to a simple implementation of this algorithm in R?

(I am not looking for a package, e.g. arules, but for comprehensible source code of an implementation from scratch. It also does not help to look at the source code of arules: It just calls a C implementation of the algorithm!)

I found an implementation here: Learning Data Mining with R by Bater Makhabel

The source code can be downloaded here: https://www.packtpub.com/big-data-and-business-intelligence/learning-data-mining-r


How can I combine two datasets and assign factors to each? I have a very basic question (just starting with R).

I have two datasets (xlsx files) with one variable (x) and 11 observations.

I need to combine these two tables to one table with the variable x (ending up with 22 observations and 1 variable).

But I still need to be able to distinguish which observations belonged to group1 and which to group2 --> so I guess I have to add a Factor like A for group1 and B for group 2.

How can I add this factor so that it remains with the values after combining/stacking the datasets?

Thanks already in advance!

Hierarchical indices

Have a look at here (scroll a bit underneath for the code)


Representing similar states in reinforcement learning? Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state having 10 spades, 4 hearts, 6 clubs on the table and holding King and Queen of hearts had already been visited, I would like it to affect the weights of similar states, like the same cards with different suits. How do I accomplish this?

Define your biggest suits as suit1 and if lowest is different suit2. Then do the same with the ground.

In your example, it would be king and queen of suit1 in your hand and 4 suit1, and 10 suit2 and 6 suit3 on the ground.


Coefficient of determination of linear regression in Azure ML studio differs significantly from regression R square in Orange 3 for same dataset why? I tried exactly same dataset in Orange 3 and Azure ML Studio for linear regression. Coefficient of determination in Azure ML studio is .71 while Orange gives R square as .45. Why such a huge difference?

It could help if you have added screenshots of both models settings. However, what I can say is Orange and Azure ML Studio use different parameter settings. It might be possible that some are available in one of them but not in the other.

If we assume that you have exactly the same configuration in your two models, then the random parameter can also play a big role here. Even if you use the same seed, Azure ML and Orange, probably, don't use the same random generator algorithm. As a result, the outcome of your models is not replicable.

I would advise trying with cross-validation in both implementations and dig more on the evaluation metrics. If you need more help from the DS community, you might need to add more details as well.


SGD versus Adam Optimization Clarification Reading the Adam paper, I need some clarificaiton.

It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is different as learning rate is variable (adaptive), and can change during training.

Is this the primary difference why Adam performs (for most cases) better than SGD? Also, it states that it is computationally cheaper, how can this be given that it seems more complex than SGD?

I hope my questions are clear!

In many applications I've seen (e.g. GANs) $\beta_1$ is set to $0$, so $m_1=g_1$, i.e. the numerator of the update rule is the same as in SGD. This leaves two main differences, both related to the MA of the second moment:

$v_t:$ raw MA of the second moment serves as a gradient normalizer that divides the gradient by the square root of the moving average of squares of gradients
$1-\beta_2$. To reduce bias, $\sqrt{v_t}$ is also divided by $\sqrt{1-\beta_2^t}$. This follows the derivation of the expectation of the square of the gradient, $\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2]$ in Section 3 of the article. Essentially $\mathbf{E}v_t = (1-\beta^t_2)\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2] + \varepsilon, $ hence the exppression. Early in the training MAs are close to $0$, and division by $\sqrt{1-\beta_2}$ helps move away from it.

In probability and statistics, moments refer to the uncentered expressions of the form $\mathbf{E}X^k$, which moving averages estimates, hence the name. Normalization allows the gradient adjustment, and, hence, better parameter update


How to scale prediction back after preprocessing So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself.

The issue I'm having is that I'm using preprocessing to scale the data to train my classifier. However when I make a prediction, the answer is also scaled. When I comment out all the preprocessing, I get accurate results. Is there a way to scale the prediction back?

The outputs are rounded to 0, 1 or 2 with each number representing one of three species.

You can see my code below:

import pandas as pd
import numpy as np
from sklearn import preprocessing, model_selection
from sklearn.linear_model import LinearRegression

df = pd.read_csv("iris.csv")

# setosa - 0
# versicolor - 1
# virginica - 2
df = df.replace("setosa", 0)
df = df.replace("versicolor", 1)
df = df.replace("virginica", 2)

X = np.array(df.drop(['species'], 1))

y = np.array(df['species'])

# Scale features
# X = preprocessing.scale(X)

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)

clf = LinearRegression(n_jobs=1)  # Linear regression clf

clf.fit(X_train, y_train)

confidence = clf.score(X_test, y_test)

print("Confidence: " + confidence)

# Inputs
sepal_length = float(input("Enter sepal length: "))
sepal_width = float(input("Enter sepal width: "))
petal_length = float(input("Enter petal length: "))
petal_width = float(input("Enter petal width: "))

# Create panda data frame with inputted data
index = [0]
d = {'sepal_length': sepal_length, 'sepal_width': sepal_width, 'petal_length': petal_length, 'petal_width': petal_width}
predict_df = pd.DataFrame(data=d, index=index)

# Create np array of features
predict_X = np.array(predict_df)

# Need to scale new X feature values
# predict_X = preprocessing.scale(predict_X, axis=1)

# Make a prediction against prediction features
prediction = clf.predict(predict_X)

print(predict_X, prediction)

rounded_prediction = int(round(prediction[0]))

if rounded_prediction == 0:
    print("== Predicted as Setosa ==")
elif rounded_prediction == 1:
    print("== Predicted as Versicolor ==")
elif rounded_prediction == 2:
    print("== Predicted as Virginica ==")
else:
    print("== Unable to make a prediction ==")

Here is an example of my output with preprocessing enabled. I'll be using one of the lines from the CSV as an example (6.4 sepal length, 3.2 sepal width, 4.5 petal length and 1.5 petal width) which should equal the versicolor species (1):

Confidence: 0.9449475378336242
Enter sepal length: 6.4
Enter sepal width: 3.2
Enter petal length: 4.5
Enter petal width: 1.5
[[ 1.39427847 -0.39039797  0.33462683 -1.33850733]] [0.41069281]
== Predicted as Setosa ==

Now with preprocessing commented out:

Confidence: 0.9132522144785978
Enter sepal length: 6.4
Enter sepal width: 3.2
Enter petal length: 4.5
Enter petal width: 1.5
[[6.4 3.2 4.5 1.5]] [1.29119283]
== Predicted as Versicolor ==

It seems I'm either doing the preprocessing wrong, or there's an extra step that I've missed out. I'm sorry if I get some of the terminology wrong and thanks in advance for answering.

When you decided you have to scale your data, you usually have to follow these steps:

For training:

Scale / Standarize the training set
Store the scaling / standarization factors of the training set
Train the model

For predicting:

Scale / standarize the input data, but very important, with the scaling/standarization factors stored during the training process. You dont have to compute the min, max or mean values of the new data.
Predict

The reason is that you have to map the new data to the same feature-space used for the training process, so you have to scale/std it with the same factors, otherwise, you are changing the feature space.


R: reading big data files in R In R, there are libraries which help speed up the process of reading huge data files. Examples of these libraries include sqldf and ff.

What would be the disadvantage of using these packages to read a small file that one would typically read with read.csv?

If their is no disadvantage of using them to read small files, does this mean functions such as read.csv might not have much use in future given the emerging big-data-targeted readers?

In R many great improvements are typically made by creating new libraries rather than changing the actual functions in base R itself. datatable, read_csv in readr, tibbles, etc are very good examples.
read.csv vs fread as answered by Matt Dowle himself
read.csv coerces strings as factors by default which is not done by read_csv and fread. So if that is what you need, read.csv will work well. Now answering your question,there is no disadvantage using the fast readers to read small files apart from a small speedup that you might get.


score or cost function for AgglomerativeClustering I am learning AgglomerativeClustering using sklearn. It is fairly easy to use for example:

  # create clusters
    hc = AgglomerativeClustering(n_clusters=10, affinity = 'euclidean', linkage = 'ward')
    # save clusters for chart
    y_hc = hc.fit_predict(points)

In my case,n_clusters is dynamic within a range, say from 9-15. I would like to run some cost function or score function, so I can plot a chart and then pick one there.

However, seems AgglomerativeClustering doesn't have a score function, unlike Kmeans, which I can either use inertia_ or score to plot.

So is there a way I can plot a chart and justify my n_clusters choice?

Don't run agglomerative clustering with multiple n_clusters, that is just unnecessary.

Agglomerative clustering is a two-step process (but the sklearn API is suboptimal here, consider using scipy itself instead!).

Construct a dendrogram
Decide where to cut the dendrogram

The first step is expensive, so you should only do this once. It does not yet produce clusters, but the Dendrogram which can be helpful to decide on the number of clusters. In the -cheap- second step, clusters are then extracted. You can use the height value of the dendrogram in a similar way as inertia with k-means. In fact, the height with Ward linkage is very closely related to inertia. But the same drawbacks apply: the value is (for most linkages) monotone. So it's not too helpful for choosing upon the number of clusters naively.


Reason behind choosing Neural Network for classification Given a two class multi dimensional classification problem, what reason would you give to choose Artificial Neural Network for carrying out the classification instead of Support Vector Machine or other classification methods?

I wouldn't give any reason to make an choice a priori, based on such a broad description.

Instead I would build models for NN and SVM (or any other that you were considering), train them on the data then compare using cross-validation and a suitable metric for your problem - such as accuracy, F1 score, AUROC or log loss. You may also want to note characteristics other than performance metric if you care about costs such as time to train, amount of computing resource required etc.

Experience may tell you likely best choices from the problem domain. For instance, if you are classifying photographic or scanned images, then currently a convolutional neural network would be a very strong candidate, as these have repeatedly shown themselves as best performers with real-world image data.


Disparity between training and testing errors with deep learning: the bias-variance tradeoff and model selection I am developing a convolutional neural network and have a dataset with 13,000 datapoints that is split 80%/10%/10% train/validation/test. In tuning the model architecture, I found the following, after averaging results over several runs with different random seeds:

3 conv layers: training MAE = 0.255, val MAE = 0.340
4 conv layers: training MAE = 0.232, val MAE = 0.337
5 conv layers: training MAE = 0.172, val MAE = 0.328.

Normally, I'd pick the model with the best validation MAE (the trends are the same for the testing MAE, for what it's worth). However, the architecture with the best validation MAE also has the largest difference between training and validation MAE. Why is what I'd normally think of as overfitting giving better results? Would you also go with 5 convolutional layers here, or are there concerns with a large difference in training and validation/testing performance?

On what I imagine is a related note, I am familiar with the article "Reconciling modern machine-learning practice and the classical bias–variance trade-off" in PNAS, which has the thought-provoking image below. Is this something that's actually observed in practice -- that you can have minimal training error but good out-of-sample, generalizable performance, as shown in subpanel B?

Your question is, what model is better between one that seems more overfitted (larger difference between train and eval set) but it has also higher scores or one that has less variance between train and eval set but at the same time it has worst results. Everything assuming that you have done a correct train test split and there is no data leakage and distributions keep the same in every split (this is important to check).

There was a discussion about this some time ago. The answer seems to be kind of subjective since the quantitative analysis has been performed

Normally there is the following trade-offs:

Complexity: Occams razor and complexity vs interpretability. In your case, both models are almost with the same complexity (it's not a linear regression against DL, just a couple layers more) and the interpretability stays the same,
Generalization. You want your model to behave in the best possible way in production, an overfitted model in train seems to have more probable cause to fail due to a change of distribution in production.

You only have 3 data points so it's hard to say what it will be best. My suggestions will be that:

Add some more layers (6,7,8) just to see when your test results start to go down (you can still overfit much more) and then visualize the data and keeping both concepts defined before choosing what are the best architectures for your model
Investigate with more parameters (adding one more layer seems to be a high difference hyperpameter), like learning rate, layer size, activations functions and so on...
Consider using one of the famous architectures for your problem, they are developed in every framework and tested by a lot of people, they are there because the seem to be the best at their task, give them ago. There has been already a lot of electricity wasted in deep learning hyperparameter tunning.


sklearn KNN imputation Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ?
Is it allowed ? Or , Should I only fit and transform both on a single set ?

The short answer is that usually if you are performing data transforms, you fit on the training data and transform training and test data.
The long answer is that if you are doing purely supervised machine learning then you want to avoid "leakage".

Data leakage is when information from outside the training dataset is used to create the model.

https://machinelearningmastery.com/data-leakage-machine-learning/

I'd read the above link if you want more information. In your case, it just comes down to preprocessing only using information in your training data.


How to build News Tagging model(s) I am trying to build a news tagging system. Given a piece of news article, find 5-6 key terms from the news article that best describe the article. Refer to the image below from google news.

What are some approaches I can look at to get human readable tags ?

You can get the tags for the headline or news content using NLP. Tags are generally noun chunks, entities or the action mentioned in the headline. You need to perform standard steps of text processing i.e remove stop words, tokenize sentence into words, do stemming or lemmatization then parse it to extract information. Most of these are already implemented. Look into spaCy or use NLTK.

You can also look into SVO (Subject, verb, object model) to assign tags. SpaCy provides an easy way to do this, do check it out.


Selection of base model for transfer learning Is there a golden rule which gives intuition on which base model needs to be used for a give image classification problem.

Most of the articles gives the below details which says how to train the model based on the dataset.

However I was not able to find good reference for the selection of base model.

Thank you

There is no specific rule associated with the base model selection for transfer learning. It is generally a trade-off between model precision and resource allocation. As the number of layers increases, the number of parameters increases and the model becomes more and more resource heavy but deeper models tend to have better accuracy over shallower counterparts. Here's a comparison:

Apart from that also refer to this question: Which is the fastest image pre-trained model? if the speed of the model matters.


Multiple models in the same notebook Having working on data sets, sometimes we want to keep track of mtiple models with different architectures which work on the same data set on which we have made some transformations and preprecessing of data has been done.
So I would like to know what is the elegant way to work on multiple models which use the same data set?
Because having multiple models on the same notebook is cumborsome and recreating the same data preprocessing nd transformations on separate notebooks is also a lot of copy-paste which I think can be solved with some existing solutions which I am not aware of.
What is the industry standard when doing such tasks?
Any help is appreciated.

If I was in your situation, I would approach it this way:

Create a "preprocessing"/"data" module

That module can either be a simple data access layer that can be shared across notebooks, or it can also include preprocessing steps that you can add. This enforces a common data access layers across notebooks without necessarily duplicating code.
So you could do something like this:
from data_layer import Data, Preprocessor

preprocessor = Preprocessor(**kwargs)
data = Data(preprocessor = preprocessor, **kwargs)

In an ideal world, you should be able to define your data access layer as data itself. For instance, a JSON document containing the data source, and the preprocessing options.

Create a different notebook per model

And in this case, if your models share components, I'd also create modules they can all import from. The same way you'd use sklearn or torch component.


Video classification of birds I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked for my dataset.

Dataset

The dataset consists of 9 different classes with around 3 000 videos with labels:

Björktrast (Fieldfare)
Koltrast (Blackbird)
Nötväcka (Nuthatch)
Pilfink (Tree sparrow)
Mus (Mouse)
Blåmes (Blue tit)
Rödhake (Robin)
Katt (Cat)
Inget objekt (No object)

Here is a video of a sample dataset containing the different classes that my deep learning model should be able to predict:

Method

Currently I have used this codebase for temporal segment networks to train my model and the resulting accuracy for my own dataset was about 87% (only with RGB and not using optical flow). Although depending on weather condition and lighting, the accuracy can be very different.

I have expanded the dataset to around 30 000 videos using the following techniques:

Adding RGB noise to videos already in the training dataset.
Adding random distortions and warping to videos in the training dataset.

Problems

I have a few questions about doing video classification to predict bird species:

Are there better techniques for doing video classification compared to using temporal segment networks?
Are there any preprocessing steps that should be done to improve accuracy?
How to train the model to generalize better to be less sensitive to lighting and weather?
How can the model be trained to classify unknown species?
Are there other ways to expand the dataset except those mentioned above?

In short: What is the best deep learning model for video classification for bird species?

Update 2018/04/17: I discovered that the accuracy of the model is significantly lower with objects obstructing the camera which have similar colours to the classes of birds to be recognized. This often causes the model to label the video incorrectly and setting a high confidence.

Are there better techniques for doing video classification compared to using temporal segment networks?

I would suggest to train the network to detect bird species in image space without the temporal dimension. This will make the network have much less parameters to tune and you will likely get better results. Then for a novel video, you can detect the species for each frame and do a majority vote to decide what species was in the video.

Are there any preprocessing steps that should be done to improve accuracy?

If you train in image space then you can use a DataGenerator on the image and transform it in all kinds of ways, the position and rotation of the bird does not affect the species. This can be done through the Keras pipeline. You can check how to do this in general here.

How to train the model to generalize better to be less sensitive to lighting and weather?

You can try to pass your images through some pretrained segmentation network to identify background and foreground and then set that area to a static color. Perhaps black (i.e. all 0's). This will make the decision boundary for the network easier. However, I think you should focus on the other points, the network should inherently detect what features are the most salient for discriminating different species, if you have sufficient training samples the weather and background will be low information features and will thus not be used.

How can the model be trained to classify unknown species?

This would be called anomaly detection. This is a much harder problem. If you have birds of other species available in your dataset then you can add a class label for other. This is still standard supervised learning. If this is not available it will be very difficult. Most anomaly detection algorithms do not work well for high dimensional datasets. Then you will need a way to compress the information in your input images. This can be achieved with some transfer learning on a pretrained autoencoder network trained on natural images such as ImageNET, then you can use transfer learning using your bird images to further tune the network for your specific use case. You can then use this compressed feature space using one of the anomaly detection techniques I described here.


Similar algorithm to apriori to find unpopular sequential patterns I am working with a dataset that looks like similar to this one but it is way larger (approx. 30.000 arrays):
sequences = [[car, house, bike, train],
             [car, house, bike],
             [apartment, train, car],
             [building,flower, bike, train],...]

It is a bit difficult to explain but there are some items in the data, that occur pretty often and some dont. Nevertheless the items that occurs less often occur in sequences that I want to find as well. So in this case I would like to find all transportation possibilities that occur together, but I also want to find all housings that occur together or all plants ect. I know that plants and buildings dont occur very often in general so the support value of these itemsets in apriori are penalized by this. All my results somehow contain transportation possibilities as their support values are higher as they occur often within the data set.
Currently I am using MLxtend's apriori but I would be glad if there are algorithms that I could use that do not penalize less often occuring items in the data set.
Is there some library/algorithm I could use to solve this problem?
Thanks in advance!

The algorithms I would recommend in your case are Apriori-Inverse and Apriori-Rare. Disclaimer: I have not found a Python implementation (nor can I provide a reliable one due to time constraints), but there exists an open-source Java library SPMF with implementations of these algorithms (Inverse, Rare, and a whole bunch more) which can be used for reference -- the source code has informative comments and the algorithms are structured well enough to understand what is going on. Of course, this is only an option if you are willing to do some digging and have enough time on your hands. Maybe going through that Java code will provide you with insights on how to modify existing Python implementations to suit your needs, which should require far less effort, since Inverse and Rare differ very little from the classical Apriori. I hope this helps you at least somewhat. Good luck!


How to import statsmodels module to use OLS class? I am trying multiple Regression

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Importing Dataset
dataset = pd.read_csv(
    'C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 2 - Regression/Section 5 - Multiple Linear Regression/50_Startups.csv')
print(dataset)
X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, 4].values

# Categorical Data

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

labelencoder = LabelEncoder()
X[:, 3] = labelencoder.fit_transform(X[:, 3])
onehotencoder = OneHotEncoder(categorical_features=[3])
X = onehotencoder.fit_transform(X).toarray()

# Splitting the dataset into training set and test set

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)
print(Y_train)

# Fitting Multiple Linear Regression

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()
regressor.fit(X_train, Y_train)

# predicting the test result
Y_pred = regressor.predict(X_test)

This is the part where error is occuring

# Building the optimal model with Backward Elimination
import statsmodels.formula.api as sm

X = np.append(arr=np.ones((50, 1)).astype(int), values=X, axis=1)
print(X)
X_opt = X[:, [0, 1, 2, 3, 4, 5]]
regressor_ols = sm.OLS(endog=Y, exog=X_opt).fit()
print(regressor_ols.summary())

This is the error message

Traceback (most recent call last):
  File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/Muliple_Linear_Regression.py", line 39, in <module>
    import statsmodels.formula.api as sm
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\formula\api.py", line 15, in <module>
    from statsmodels.discrete.discrete_model import MNLogit
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\discrete\discrete_model.py", line 45, in <module>
    from statsmodels.distributions import genpoisson_p
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\distributions\__init__.py", line 2, in <module>
    from .edgeworth import ExpandedNormal
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\statsmodels\distributions\edgeworth.py", line 7, in <module>
    from scipy.misc import factorial
ImportError: cannot import name 'factorial'

Process finished with exit code 1

See https://stackoverflow.com/a/56284155/9524424

You need to have a matching scipy version (1.2 instead of 1.3)


custom tensorflow layer that inherits another class I am trying to make my own custom layer in tensorflow which looks like:

class custom_LSTM(RNN):
    # implementation of class

where RNN is another class.

However, to use this custom layer in model it needs to inherit tf.keras.layers.Layer. I am creating models using tf.keras.

Any idea how do I solve this problem?

Although I'm not an export of RNN's, the following should work given that RNN is inherited from Layer:

from keras.layers import RNN
class custom_LSTM(RNN):
   # implementation of class
   ...

It might be good to provide the smallest piece of code that can reproduce your error.


Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer
IterativeImputer(max_iter=<num_iter>,
                 initial_strategy = 'most_frequent',
                 verbose=True,
                 estimator=ExtraTreesRegressor(n_estimators=100,
                                               min_samples_leaf=1,
                                               min_samples_split=4,
                                               random_state=0,
                                               n_jobs=-1))
imp.fit(X_missing)
imputed = imp.transform(X_missing)

running on an 8 core (16 thread) 32G, the run completed with 1 iteration but crashed due to low memory with 2 iterations
running on a cloud machine with 16 cores 128G, when running with 4 iterations it uses up 115G of ram, anything higher than that crashes with not enough memory
Does anyone know how to reduce the memory footprint of imputer?

TL;DR - use the max_depth and max_samples arguments to ExtraTreesRegressor to reduce the maximum tree size. The sizes you pick might depend on the distribution of your data. As a starting point, you could start with max_depth=5 and max_samples=0.1*data.shape[0] (10%), and compare results to what you have already. Tweak as you see fit.

Apart from the fairly large input space, the data structure built by the ExtraTreeRegressor is the main issue. It will continue to expand the tree size until each leaf reaches your criteria, namely min_samples_leaf=1. This means every single data point of your input dataset must end up in its own leaf. Apart from probably overfitting, this is going to lead to high memory consumption.
See the Note: in the relevant documentation:

The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values.

Each ExtraTreesRegressor that you create looks like it might make a full copy of your dataset, according to the documentation for max_samples`:
    max_samples : int or float, default=None
        If bootstrap is True, the number of samples to draw from X
        to train each base estimator.
        - If None (default), then draw `X.shape[0]` samples.

To gain a deeper understanding of how you might tune your memory usage, you could take a look at the source code of the ExtraTreesRegressor.


What is the neural network architecture behind Facebook's Starspace model? Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace.

In their paper, they explain the loss function and the training procedure of the model, but they don't emphasize much on the architecture of the model.

Does somebody know what the NN behind looks like?

Calling StarSpace a neural model would be misleading I think. You could certainly think of the it as a neural network with a single layer and a linear activation function, but I think don't think that would be very illuminating. They didn't discuss the architecture much in that paper for a reason- there isn't really any in terms of layers of neurons, activation functions, latent variables, or anything else except the constraints on the number of dimensions.

In fact, the most helpful way to think about StarSpace is  that, at its core like many (maybe most) popular embedding techniques across natural language, graphs, etc it's a low rank matrix factorization. What the sampling procedures are doing is using the data in some way to produce a positive definite gram matrix. It doesn't appear that way initially because that is done solely through sampling- if you were to find the expectations of each input/target pair, however, you'd find that the optimization objective is maximizing the expected value of vector similarity over a joint distribution on pairs of items minus a marginal distribution (this is due to the negative samples). Essentially, the goal is to maximize the difference in similarity between items that are frequently sampled and the similarity between items sampled independently from the marginal distribution.

If this sounds familiar to SGNS implicitly factorizing a shifted PPMI matrix or GlOVe explicitly factorizing a relaxed variant of the same, good. The specifics are different and StarSpace has significantly more flexibility in the sampling distribution it works with, but the principle is the same. "Neural Word Embedding as Implicit Matrix Factorization" and "Improving Distributional Similarity with Lessons Learned from Word Embeddings" are fantastic papers by Levy, 2014 and 2015 if I recall correctly, that discuss the connections between neural embeddings and explicit matrix factorization techniques like PPMI-SVD and glove and the principles that make them successful.

Similarly, "Network Embedding as Matrix Factorization: Unifying
DeepWalk, LINE, PTE, and node2vec" is a great discussion of the connections between neural network embeddings and the same implicit objectives as the neural word embeddings.

In short- it doesn't sound like there's much going on in StarSpace as far as architecture because there isn't. It's quite literally adjusting the placement of points in the embedding space to make associated items more similar to each other than to unrelated items.


Cannot create new percent columns from CSV file I am fairly new to coding and cannot figure out how to create the AB_perc and CD_perc columns in the Desired Output table below.

The original csv file I am using has all sorts ot DNA sequencing data, so below is a simplified example of what I am trying to do.

CSV file:

Desired Output:

I was able to generate the AB_sum and CD_sum columns by using:

df['AB_sum'] = df['A'] + df['B']
df['CD_sum'] = df['C'] + df['D']
df.groupby('RunNumber', as_index=False).agg({'AB_sum': 'sum', 'Confusion C<->G': 'sum'})

But I cannot figure out how to create the AB_perc and CD_perc columns.

For example:

To calculate the percentage of AB for Run 1 (AB_perc: 79.94), the AB_sum value, 570, is divided by 713, the sum of all orange numbers in the csv file, and then multiplied by 100

I tried several things all day and nothing worked for me. I'm sure there's a simple solution, but I feel like if I try any harder my nose will start to bleed. Haha.

Any help would be much appreciated!

Try using groupby with a custom function and apply.

df = pd.DataFrame()
df['RunNumber'] = ['Run1','Run1','Run2','Run2']
df['A']=[500,60,20,30]
df['B']=[5,5,2,2]
df['C']=[5,10,2,6]
df['D']=[3,4,5,4]
df['E']=[65,56,56,44]

def calculate_sum_percent(input_df):
    output_df = pd.Series()
    output_df['AB_sum']=input_df['A'].sum() + input_df['B'].sum()
    output_df['CD_sum']=input_df['C'].sum() + input_df['D'].sum()
    output_df['AB_perc']=100*(input_df['A'].sum() + input_df['B'].sum())/input_df.sum().sum()
    output_df['CD_perc']=100*(input_df['C'].sum() + input_df['D'].sum())/input_df.sum().sum()
    return output_df

stats_df = df.groupby('RunNumber').apply(calculate_sum_percent)
display(stats_df)


How to use glob() output as input to os.listdir() I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following:

f = glob.glob(ct)
print(f)

['C:\\Users\\tennant\\Desktop\\RF WAVEFORMS\\SPRING 2018\\RF\\1L22\\2018_05_02\\133258.2']

which, if I try to use as input to listdir() gives the following error

test = os.listdir(str(f))

Eventually through enough trial and error, I was able to find a solution.

f = glob.glob(ct)
fnew = str(f).strip('[]')
test = os.listdir(fnew.replace('\'',""))
print(fnew.replace('\'',""), len(test))

C:\\Users\\tennant\\Desktop\\RF WAVEFORMS\\SPRING 2018\\RF\\1L22\\2018_05_02\\133258.2 7

However it's messy and I'm clearly not understanding something more fundamental about the output of glob() or strings in general. Anything that could clean this code up and help my understanding would be greatly appreciated!

Assuming you have a string to pass into glob that does wildcard matching, glob will return a list of matches. So you don't need to make a string out of that list and replace the square-brackets and so on. You can just iterate over that list and do something with each of the values, which are already strings.

results = glob.glob(your_pattern)

Based on your code, the results is actually a list of folder names for which you want to count the number of files in each. We can loop over results and print the counts:

for folder in results:
    n = len(os.listdir(folder))
    print("Folder: {0} --> \t {1} files".format(folder, n))


Orange release notes What is the best way to understand changes made in new releases? For example, Orange 3.21 came out recently and I’m eager to understand any prominent fixes or modified features.

All release notes for new versions of the Orange Data Mining software can be found by looking at the release page on its GitHub page here. Then just navigate to the most recent release and click on the latest branch. This will display a window similar to the following where you can view all of the changes that were made (both enhancements and bug fixes) in the most recent update.

Snapshot for Orange 3.21.0 below:


How to install rattle in centos While running rattle in my system I am getting this error

rattle()

Error: attempt to apply non-function
In addition: Warning message:
In method(obj, ...) : Unknown internal child: selection

I am using R version 3.1.0 (2014-04-10)

I got the answer.
I have to install some of the packages in terminal . I installed it and it works.

sudo yum install gtk+-devel gtk2-devel


Why do probabilities sum to one and how can I set optimal threshold level? I am working on a text classification use case.
The training data has two classes, so the XBBoostClassifier  and onevsrest model is classifying the test data into either of the two classes.
But my requirement is to classify either into given the classes or if no match found, then set it as 'undetermined" so that i can manually classify the data.

I tried the following OneVsRest Classifier

pl = Pipeline([
        ('vec', CountVectorizer(token_pattern = tks)),
        ('clf', OneVsRestClassifier(LogisticRegression()))
    ])
pl.fit(x,y)
predictions = pl.predict_proba(test.comment_text)

But the sum of the probabilities is one and moreover the probability is above 90 for the class to which the data belongs to.

Please clarify the following points
1. Why the probability is always one? whether it means that the data is mutually exclusive?
2. The probabilities are like this

CLASS 1           :CLASS 2
0.892993358265023 : 0.106808845640795
0.999999742528922 : 2.57685096542208E-07

Does that mean that in first case, probability is only 90% for class 1 and hence the classifier is not able to classify the data properly. However in other case, there is clear cut difference as the probability is around 99%

Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data?

Please provide your inputs

Question 1. Why do probabilities sum to 1: Probability theory. Probabilities sum 1 because that’s how we define them. It just so happens that, by forcing them to sum 1, they have an intuitive interpretation and also calculations end up being easier. But this is mere convenience. Probabilities (or, more specifically, probability measures) could have been defined to sum 12 or 100 or whatever number you prefer. It doesn’t really matter.

Further sub-questions:

Its not the data thats exclusive its the classes.
Classifier is able to classify just with lower probability ( I would actually argue that 99999% is to much maybe he is overfitting)

Question 2 Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data? Let the data tell you optimal cut-off level/threshold (from the ROC curve, read it up)

def optimal_cutoff(ground_truths: np.array,
                             predictions: np.array) -> float:
        """

        :param ground_truths: array of arrays (matrix) of all ground truths
        :param predictions: array of arrays (matrix) of all predictions
        :return: optimal cut-off level
        """
        optimal_thresholds = []
        for y_train, y_pred in zip(ground_truths, predictions):
            fpr, tpr, threshold = roc_curve(y_train, y_pred)
            optimal_idx = np.argmax(tpr - fpr)
            optimal_thresholds.append(threshold[optimal_idx])

        return sum(optimal_thresholds) / (len(optimal_thresholds))

So you would calculate cut-off on your available GROUND truths. Now if, when you specified optimal cut-off level your metric (lets say f1) falls behind certain level you can tell users to check it up.


Conflicting directions of weights gradients in gradient descent? In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 to eventually update them.

We calculate ∂Err/∂w1, ∂Err/∂w2, ∂Err/∂w3 and update the weights as wi = wi + ∂Err / ∂wi for each of the weights so that we move towards the direction where the loss function reduces in measure.

The problem that I see is that there might be situations all the time when some of the weight delta directions conflict in terms of the loss function. That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err. Isn't this the case? What am I missing?

That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err.
Isn't this the case?

Not exactly. Using back propagation, the weight gradients may be calculated precisely (for the given training data). It doesn't matter that there are many of them updating at the same time. The gradients don't "conflict" as such, they are 100% compatible. But they are only valid locally to the current values of $w_i$
An update to the all the weights at once in the opposite direction to the gradient is guaranteed to reduce the error value for that training data, with an important caveat that it is only fully guaranteed to make an infinitesimal small improvement, when the step size is also infinitesimal.
If you make an update step that is too large (for some value of "too large" which varies a lot depending on context), then the curve may not remain consistent over the step and your error value could increase.
This problem is not directly related to updating multiple weights at once. It can occur even when you have a single weight. However, when there is a more complex function, with more weights all changing at once, there can be more places where the function does this.
In practice, infinitesimal updates would take too long to train, so you need to find a larger value - but not so large it causes problems - as a compromise.
In addition:

The training data you have will usually allow you at best to create a rough approximation to the "true" function you are searching for. So you don't really want to find an actual global minimum in the error function for your training data set, as it would overfit.

Typically, using mini-batches, the gradient is also only a rough approximation, so updates only go roughly in the right direction (this can sometimes be good as it can help escape from local minima and saddle points)


Complete IPv4 Address Space I know this was asked before, but I didn't get a definitive answer. I am trying to download a simple 17GB .txt file that contains the entire IPv4 address space. There must be a file like this already on the web.

You can simply generate them yourself. They are simply representations of all integers up to $2^{32}$

Which format do you want the address? There are many ways to represent them according to your requirements, due to the fact that they are simply representation of a set of normal numbers:

standard dot-separated e.g. 192.168.0.1
unsigned int32 : e.g 3548551264
hex, dotted hex, octal bytes

You could generate them in Python like this:

def generate_ips():
    """Generate all possible IPv4 addresses, on by one"""
    for i in range(255):
        for j in range(255):
            for k in range(255):
                for l in range(255):
                    yield "{0}.{1}.{2}.{3}".format(i, j, k, l)

The you must loop on the generator function like this to actuall use the values:

for ip in generate_ips():
    print(ip)

# output
0.0.0.0
0.0.0.1
0.0.0.2
0.0.0.3
0.0.0.4
0.0.0.5
0.0.0.6
0.0.0.7

I used a generator (with the yield keyword instead of saving all values in a list) because it will just give you one at a time and not consume all those Gb of memory that you mentioned!

If you want to store all values in a list for some reason, you can alter the function to append each IP address to a list, then return that:

def generate_ips():
    """Return a list of all possible IPv4 addresses"""
    all_ips = []
    for i in range(255):
        for j in range(255):
            for k in range(255):
                for l in range(255):
                    ip = "{0}.{1}.{2}.{3}".format(i, j, k, l)
                    all_ips.append(ip)
    return all_ips


Support Vector Regression trained with data sets I am now searching for a long time on the internet and on papers for an answers of simple questions. Am I able to train a Support Vector Regression algorithm with different data sets? If yes, how is the approach called?

I have 10 times the same battery with different usage, temperature and capacity.

Usage and temperature are features (x_i,i) and capacity is the output (y_i,i).
Battery_1 till timepoint n: [x_1,1 y_1,1; ... ;x_1,n y_1,n]
...
Battery_10 till timepoint n: [x_10,1 y_10,1; ... ;x_10,n y_10,n]

Now I want to train my SVR with these sets, where the samples within a set belong together. I want give the algorithm a set of usage and temperature where I don't know the capacity and my SVR should predict it as such : x_d --> y_d.

Thank you very much for your help and input.

Why not concatenating the 10 datasets you have and simply creating one big dataset?
Alternatively you could train 10 seperate SVMs and let them vote (for examples: output = mean of the 10 SVMs).


Deconvolution vs Sub-pixel Convolution I recently read Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Wenzhe Shi et al. I cannot understand the difference between deconvolution (mentioned in Section 2.1) and the Efficient sub-pixel convolution layer (ESCL for short) (Section 2.2)

Section 2.2 defines the ESCL as performing convolution with a fractional stride. Isn't this what deconvolution is? Why is a distinction being made?

Thanks

In this paper, theauthors have provided a deep explanation, basically its the exact same thing. sub-pixel uses or^2 conv kernels all at res H/r,W/r while the regular transpose conv uses o conv kernels all at res H,W the authors prove that one can rearrange the feature map (Or^2,h,w) to a feature map (O,hr,wr) without losing information.


Sparse matrix in R based on the data frame Suppose I have book ratings in the form of data frame (where 0 means no rating):

$\begin{array}{|c|c|c|}
\hline
\textbf{User.ID}& \textbf{ISBN} & \textbf{Book.Rating} \\ \hline
 276725 & 034545104X & 0 \\ \hline
 276726 & 0155061224 & 5 \\ \hline
 276725 & 3257224281 & 7 \\ \hline
 ... & ... & ... \\ \hline
\end{array}$

In what easiest way can I get the form as below (I want to use it to create a realRatingMatrix object) ?

$\begin{array}{|c|c|c|c|c|}
\hline
\ & \textbf{034545104X} & \textbf{0155061224} & \textbf{3257224281} & ...\\ \hline
 \textbf{276725} & . & 3 & 7 & ...\\ \hline
 \textbf{276726} & 5 & 5 & . & ...\\ \hline
 ... & ... & ... & ... & ...\\ \hline
\end{array}$

Your data looks like this at the moment:

data <- data.frame(User_Id = c(276725, 276726, 276725, 276726, 276725),
                   ISBN = c("A", "B", "C", "A", "B"),
                   Book_Rating = c(0, 5, 7, 5, 3))

> data

  User_Id ISBN Book_Rating
1  276725    A           0
2  276726    B           5
3  276725    C           7
4  276726    A           5
5  276725    B           3

With the following commands you can create a dgCMatrix, required as input for a realRatingMatrix object.

library(Matrix)
data_sparse = sparseMatrix(as.integer(data$User_Id), as.integer(data$ISBN), x = data$Book_Rating)
colnames(data_sparse) = levels(data$ISBN)
rownames(data_sparse) = levels(data$User_Id)

> data_sparse

276726 x 3 sparse Matrix of class "dgCMatrix"
     A B C
[1,] . . .
[2,] . . .
[3,] . . .

..............................
........suppressing rows in show(); maybe adjust 'options(max.print= *, width = *)'
..............................

[276722,] . . .
[276723,] . . .
[276724,] . . .
[276725,] 0 3 7
[276726,] 5 5 .

After that you can call the command

new("realRatingMatrix", data = data_sparse)

Bear in mind that this matrix starts at one and it's only populated in two rows (276725 and 276726) but the rest of the columns from 1 to 276725 exist. If you don't want to use User_Id as indices you will have to create new indices and use those instead and have one User_Id correspond to a new index. That means if User_Id starts at, for instance, [11, 18...] then it'd become User_Id 11 = New_Index 1, User_Id 18 = New_Index = 2. Use New_Index instead of User_Id and you're done.

If that's the case then data should end up looking like this:

data <- data.frame(User_Id = as.factor(c(276725, 276726, 276725, 276726, 276725)),
                   ISBN = c("A", "B", "C", "A", "B"),
                   Book_Rating = c(0, 5, 7, 5, 3),
                   New_Index = c(1, 2, 1, 2, 1))

> data

  User_Id ISBN Book_Rating New_Index
1  276725    A           0         1
2  276726    B           5         2
3  276725    C           7         1
4  276726    A           5         2
5  276725    B           3         1

library(Matrix)
data_sparse = sparseMatrix(as.integer(data$New_Index), as.integer(data$ISBN), x = data$Book_Rating)
colnames(data_sparse) = levels(data$ISBN)
rownames(data_sparse) = levels(data$User_Id)

> data_sparse

2 x 3 sparse Matrix of class "dgCMatrix"
       A B C
276725 0 3 7
276726 5 5 .


problem loading data into R I have problem with loading data into R:

fileUrl <- "http://jadi.net/files/iran_it_status_1394_detail_data_jadi_net.tsv"

download.file(fileUrl , destfile="iran_it_status_1394_detail_data_jadi_net.tsv")

dev <- read.delim("iran_it_status_1394_detail_data_jadi_net.tsv",
              header=TRUE,sep="\t",blank.lines.skip = TRUE,
              na.strings="",fileEncoding="UTF-8",
              stringsAsFactors=FALSE,skipNul = TRUE)

I receive the following error:

Error in read.table(file = file, header = header, sep = sep, quote = quote,  :
  no lines available in input
  In addition: Warning message:
  In read.table(file = file, header = header, sep = sep, quote = quote,  :
  invalid input found on input connection    'iran_it_status_1394_detail_data_jadi_net.tsv'

Edit: The dataset has 1217 rows and 33 variables.

names(data) <- c("timestamp","age","sex","birth_province","work_province","experience","education",
            "certificate","learn","project","book","language","wish_language","db","desktop_os",
            "wish_os","mobile","env","theme","src_ctrl","tab_space","drink","items","device","title",
            "org_type","org_emp","income","perk","job_contract","job_type","hour_wage","happy")

for language variable I expect this output:

data[1:3,"language"]

C#, Javascript, R, SQL

Java, C#, Javascript, Objective C, Swift, SQL

C#, SQL

Python Solutions are also welcome

I am able to load the data set like so:

 dev <- read.table("iran_it_status_1394_detail_data_jadi_net.tsv",
                   header=TRUE, sep="\t", blank.lines.skip = TRUE,
                   na.strings="",
                   stringsAsFactors=FALSE, skipNul = TRUE, fill=T, quote="")

Note the removal of the encoding (so that the function "finds" lines in the file), the fill attribute (to allow for a ragged table with empty cells), and the elimination of quotes (apparently there is a misquoted line somewhere near line 585).

This results in a table full of encoded characters - you'll need to know more about the source data to figure out how to work with it, but if you open the file up in a raw text editor (e.g.: Sublime text) you might get some clues:


Is multi-device trajectory prediction possible using only classical machine learning? So I have a dataset of trajectories of multiple devices. Each device has trajectory IDs (multiple trajectories), where each trajectory consists of start point (x, y) and end point (x, y), start time, and end time. Basically, each device is said to stop when it doesn't move for some time. When they move again, it's another trajectory.
The task is to predict the END end point (the x and y for the last trajectory) for each device, so it's a multi-label regression problem. The problem is each device will travel inherently different path, so I think I have to train a model for each device. I don't think it'll work since I'll have to predict entirely different set of devices, but let's put that aside for a moment. The main problem I'm having is that in the training data, some devices only have ONE trajectory and almost half of the devices have a count of 5 trajectories, so I think my solution is out.
How would you suggest to approach this problem using classical machine learning, assuming it's possible? If it's not, what NN architecture(s) should I consider? I'm not yet familiar with NNs but if it's the only way to do it, I'll try to quickly learn about the necessary architectures.

Any machine learning algorithm could possibly learn a solution once the problem properly defined.
One technique that could aid in learning is to bin the locations. Instead of predicting a specific x and y (and being precisely wrong), predict a more general region. Based on the topological structure of the location, bin similar location points together.
One class of classic machine learning techniques to try would be tree-based, such as decision tree, random forest, or boosted trees.


specific feature engineering for specific algorithm Days ago,One AI financial service provider offered us a lesson and mentioned that you are supposed to perform specific feature engineering according to the specific algorithm you are using.For example,when using logistics regression,fitting more features（uncorrelated）like binning the continuous variable into discrete ones are often suggested.Because logistics regression is a simple algorithm and we try to raise dimension in the way that samples will be separated better.

I searched a lot(maybe not yet),most of materials are "why/what feature engineering important","scaling/standardization/binning continuous variable","dealing with null value" or some theoretical comments with no discrete manipulation.

why and how the specific feature engineering should work on specific algorithm.Or any advice on this saying,is it right or wrong?what do you think.any comments are appreciated.

（I am not good at English,sorry about that if I am not clear enough）

I am not looking forward a detailed answer,some deep thinking about this part is good.

In general, features are engineered so as to retain optimum relevant information present in the dataset with succinct representation and then features are adapted so that an algorithm can accept it as input.

Feature engineering generally involves methods like binning, PCA, etc. Adapting those features to pass to algorithm is another step which is a very small part of the feature engineering step. Adapting example: for an image, we may have to reshape the image like below so that x[0] points to the image

x = image.img_to_array(img)
normalise(x) // feature engineering
x = x.reshape((1,) + x.shape) // Adapting for algorithm

With this understanding, if features are generated once for a dataset, it may be used for any relevant algorithm with corresponding adaption.


Custom keras dataset generator not accepted by fit_generator I've got a problem trying to use generators in Keras. I've got TensorFlow 2.1 with Python 3.7. I'm using the Keras version bundled with TensorFlow

I have defined a class for my generator derived from tf.keras.models.Sequential:

class RandomFramesFromPathsToVideos(tf.keras.models.Sequential):
    def __init__(self, x_set, y_set, number_of_videos_per_batch, ort_session, frames_per_video=25, type_of_frame='cropped_frame'):
        self.x, self.y = x_set, y_set
        self.batch_size = number_of_videos_per_batch * frames_per_video
        self.number_of_videos_per_batch = number_of_videos_per_batch
        self.frames_per_video = frames_per_video
        self.type_of_frame = type_of_frame
        self.ort_session = ort_session

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.number_of_videos_per_batch)))

    def __getitem__(self, idx):

        bla, bla, bla...

        return (np.array(devol_x, dtype=np.float32), np.array(devol_y, dtype=np.float32))

and it works since I'm perfectly able to train my model item by item using a crafted loop like this:

ds_train = RandomFramesFromPathsToVideos(x_set = set_of_paths, y_set = set_of_labels,
    number_of_videos_per_batch=4,
    ort_session = ort_session)

for i in range(0:len(ds_train)):
    (x,y)=ds_train.__getitem__(i)
    model.fit(x,y)

but when I try to use the generator as it should be used:

model.fit_generator(ds_train)

(which is now equivalent in TF2.1 to model.fit(ds_train), I know), I get the error:

ValueError: Failed to find data adapter that can handle input: <class 'functions.RandomFramesFromPathsToVideos'>, <class 'NoneType'>

I've tracked the error up to this point in internal code of TensorFlow:

def select_data_adapter(x, y):
  """Selects a data adapter than can handle a given x and y."""
  adapter_cls = [cls for cls in ALL_ADAPTER_CLS if cls.can_handle(x, y)]
  if not adapter_cls:
    # TODO(scottzhu): This should be a less implementation-specific error.
    raise ValueError(
        "Failed to find data adapter that can handle "
        "input: {}, {}".format(
            _type_name(x), _type_name(y)))
  elif len(adapter_cls) > 1:
    raise RuntimeError(
        "Data adapters should be mutually exclusive for "
        "handling inputs. Found multiple adapters {} to handle "
        "input: {}, {}".format(
            adapter_cls, _type_name(x), _type_name(y)))
  return adapter_cls[0]

The contents of ALL_ADAPTER_CLS is:

result = {list: 7} [<class 'tensorflow.python.keras.engine.data_adapter.ListsOfScalarsDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.TensorLikeDataAdapter'>, <class 'tensorflow.python.keras.engine.data_adapter.GenericArrayLikeDataAdapter'>, <class 'tensor
 0 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.ListsOfScalarsDataAdapter'>
 1 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.TensorLikeDataAdapter'>
 2 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.GenericArrayLikeDataAdapter'>
 3 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.DatasetAdapter'>
 4 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.GeneratorDataAdapter'>
 5 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.KerasSequenceAdapter'>
 6 = {ABCMeta} <class 'tensorflow.python.keras.engine.data_adapter.CompositeTensorDataAdapter'>
 __len__ = {int} 7

I don't understand why my generator doesn't work. Has anyone any idea about what I'm doing wrong?.
 My dataset is a huge bunch of videos and is impossible to fit in memory by any means. Also, I'm doing non standard transformations like locating and cropping elements that appear in the frames so I can't use an standard generator that may be already implemented in Keras.

Thanks a lot

I was following the instructions on a tutorial but it seems that there was a mistake or that the internal code of TensorFlow has changed since it was written. I was defining my class like this:

class DataGenerator(tf.keras.utils.Sequence):

and later on I tried also this:

class DataGenerator(tf.keras.models.Sequential):

(which, by the way, is completely different and is totally wrong)

Inspecting the code for some of the adapters, I saw this:

class KerasSequenceAdapter(GeneratorDataAdapter):
  """Adapter that handles `keras.utils.Sequence`."""

  @staticmethod
  def can_handle(x, y=None):
    return isinstance(x, data_utils.Sequence)

and also for my other suspect adapter:

class GeneratorDataAdapter(DataAdapter):
  """Adapter that handles python generators and iterators."""

  @staticmethod
  def can_handle(x, y=None):
    return ((hasattr(x, "__next__") or hasattr(x, "next"))
            and hasattr(x, "__iter__")
            and not isinstance(x, data_utils.Sequence))

where data_utils is defined as:

from tensorflow.python.keras.utils import data_utils

So, I changed the definition of my class to this:

from tensorflow.python.keras.utils import data_utils
class RandomFramesFromPathsToVideos(data_utils.Sequence):

and now it works.

I've got, so, two options. Using a subclass from data_utils.Sequence is one of them. In that case, we are talking about a "Keras generator" (handled in TensorFlow by KerasSequenceAdapter) in which you need to define two methods: __len__ and __getitem__

The other one is building a new class NOT DERIVED from data_utils.Sequence, and defining the methods __iter__ and __next__ (or simply next). In that case we are defining an standard Python generator, which will be handled by GeneratorDataAdapter inside TensorFlow

Please, bear in mind that a Keras generator is not the same thing as a Python generator. At first, that confused me a lot.


What is max_trials and executions_per_trial in keras-tuner I am wondering what is the difference between max_trials & executions_per_trial in kerastuner.tuners.bayesian.BayesianOptimization function. Does execution_per_trial somehow relates to cross valdation? max_trails at first sounds like number of epochs to train but its not the case since there we have a another attribute for that.  I didn't get much from the original documentation found here

  You should specify the model-building function, the name of the
  objective to optimize (whether to minimize or maximize is
  automatically inferred for built-in metrics), the total number of
  trials (max_trials) to test, and the number of models that should be
  built and fit for each trial (executions_per_trial).

max_trials represents the number of hyperparameter combinations that will be tested by the tuner, while execution_per_trial is the number of models that should be built and fit for each trial for robustness purposes.

For example, let's imagine you have a shallow network (one hidden layer) with the following parameter search space:

Number of Hidden units: 16, 32, 48, 64
Choice of activation functions: ReLu, Sigmoid, TanH

You have a total of 12 combinations (4 times 3).

Then, at this point, when you set up max_trials = 40, you configure the tuner to find 40 random tuples of the # Hidden units and activation functions, e.g. (16, ReLu), (32, Sigmoid), (16, TanH)... up until 40 tuples. For each tuple you will run as many executions as you also set up in execution_per_trial variable, given that depending on how the model runs the optimization process, final results could be very different. For each trial and execution, the tuner will fit the model with as many epochs as you configure in the script.