ValueError: Error when checking input: expected the_input to have 3 dimensions, but got array with shape (14174, 1) hope you're all doing good !
I am working on Automatic Speech Recognition with Python with the LibriSpeech Dataset.
After preprocessing the audios data and applying an "MFCC featurizing" I append everything into a list and get a shape of (14174,). Knowing that each sample has a different length but the same number of features for example :
print(X[0].shape)
print(X[12000].shape)
>> (615, 13)
>> (301, 13)

Now when I feed the data into my network with an Input layer defined as
input_data = Input(name='the_input', shape=(None, input_dim)) # with input_dim = 13 MFCC features

I get the following error
ValueError: Error when checking input: expected the_input to have 3 dimensions, but got array with shape (14174, 1)

I tried reshaping with different shapes but I am still struggling.
This is the model
def final_model(input_dim, units, output_dim=29):
    """ Build a bidirectional recurrent network for speech
    """
    # Main acoustic input
    input_data = Input(name='the_input', shape=(None, input_dim))

    # =============== 1st Layer =============== #
    # Add bidirectional recurrent layer
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(input_data)
    # Add batch normalization
    batch_normalization = BatchNormalization(name = "batch_normalization_bidirectional_rnn")(bidirectional_rnn)
    # Add activation function
    activation = Activation('relu')(batch_normalization)
    # Add dropout
    #drop = Dropout(rate = 0.1)(activation)

    # =============== 2nd Layer =============== #
    # Add bidirectional recurrent layer
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(activation)
    # Add batch normalization
    batch_normalization = BatchNormalization(name = "bn_bidir_rnn_2")(bidirectional_rnn)
    # Add activation function
    activation = Activation('relu')(batch_normalization)
    # Add dropout
    #drop = Dropout(rate = 0.1)(activation)

    # =============== 3rd Layer =============== #
    # Add a TimeDistributed(Dense(output_dim)) layer
    time_dense = TimeDistributed(Dense(output_dim))(activation)
    # Add softmax activation layer
    y_pred = Activation('softmax', name='softmax')(time_dense)

    # Specify the model
    model = Model(inputs=input_data, outputs=y_pred)
    model.output_length = lambda x: x
    print(model.summary())
    return model

Thanks

Your inputs have different length so as suggested by @skrrrt, you should pad your data and apply a mask into your model.
The following pads all your input with 0. values so that all sequences have the same length.
from tensorflow.keras.preprocessing.sequence import pad_sequences
padded_inputs = pad_sequences(X, padding="post", dtype='float')

You can choose which value to use for padding with the parameter value=0.0 (documentation)
Then, add a masking layer just after your input layer in your model.
    # Main acoustic input
    input_data = Input(name='the_input', shape=(None, input_dim))
    masked_input = Masking(mask_value=0.0)(input_data)
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(masked_input)

Refer to this tutorial for more information.


Are there any deep learning models to forcast waveforms from waveforms? I want to forcast waveforms from waveforms. Now I have collected the dataset.
The input feature is a waveform, e.g.:
input feature
The out is also a waveform, e.g.:
output feature
Are there any avaliable models to solve this problem?

All sequence-to-sequence models will for this. If you want to do image to image translation, then, just convert your image into sequence using CNNs (see text recognition models).


How to determine the number of the training images in Keras after data augmentaion? I want to create a CNN model and I am using data augmentation. I want know the number of augmented images in Keras.

How to determine the number of the training images in Keras after data augmentation?

For example, if I have 2000 images for training and I am applying data augmentation, what is the number of the images after augmentation?

The images in the dataset are not used directly. Instead, only augmented images are provided to the model. So, your answer is 2000 images


Bayesian Neural net with non probibalistic Data? Is it possible to construct a Bayesian Neural Network without Probability Distributions as dependent Variable for purpose of predictive modeling?

I mean, if id like to Infer on a Specific Value, like y(e.g. y=5), with a vector of explanatory Variables X(e.g. X=[3,5,1.3,(.....)])
The Bayesian Neural Network infers on a distribution of ymean with standard deviation sigma(e.g. ymean =5, sigma =0.5).

Does it even make sense? Is the loss function of the Neural net able to work by comparing y with ymean, without taking simgma into account?

partial answer:
I think sigma is the result of the distributions in the weight matrices of the Neural net, and it should work. But I want to be sure and understand.

PS: I work in ecology, so getting a probability distribution as result would serve my goals.

It is possible to predict a single value from a Bayesian Neural Network. Given a set of input data, conduct the forward pass to generate the resulting probability distribution. Then convert that probability distribution to a single specific value in one of the following common ways:

Sample - Take a random sample. That random sample will automatically be weighted by the posterior probability distribution. This type of sampling is similar to Thompson sampling.
Use a measure of central tendency. Given that posterior probability distribution, calculate the most useful measure of central tendency (e.g., mean, median, or mode).


How to predict ETA using Regression? I have a data from GPS in the form

1.('latitude', 'longitude','Timestamp').
2.('latitude', 'longitude','Timestamp').
3.('latitude', 'longitude','Timestamp').

I am changing this data into the below form

'latitude_1', 'longitude_1', 'Timestamp_1', 'latitude_2', 'longitude_2', 'Timestamp_2, Timestamp_2-Timestamp_1.

With this format I am training a LinearRegressionWithSGD model of spark where label is Timestamp_2-Timestamp_1 and features are latitude_1, longitude_1, latitude_2, longitude_2.

But when I am giving Origin ( latitude and longitude ) and destination ( latitude and longitude ) the results are very bad.

Kindly guide me whether this approach is the right approach ? and if not then how to build a prediction model from given data to predict Estimated Time of Arrival.

I suggest to calculate the Haversine distance between two points, and fit a linear regression to find the relation between the Haversine distance and the trip duration.
So your regression will be
$duration_t = timestamp_t - timestamp_{t-1} = \alpha + \beta*d(point_t,point_{t-1})$
Where $d$ is the Haversine distance. $point_t$ is a lat/long pair at time $t$.
Note however that there's an assumption that the user drives at the same speed. If half of your data was gathered while walking and half while driving, then your relation between time and ETA is possibly not linear.


Predicting sequence element based on the previous M and the following N elements I have an array of sequences of equal length, each sequence contains 300 numbers (M=300). Each element in a sequence is a number from 1 to 9:
13571398...2455 # 300 numbers
33344467...1143 # 300 numbers
...
...
...
66118859...2121 # 300 numbers

My task is to build a model that predicts an element (number) at sequence positions from 180 to 190 based on the first 179 elements and the last 110 elements in a sequence.
In other words, given elements at positions from 0 to 179 and from 191 to 299 predict elements in a sequence at positions from 180 to 190.
I am thinking about the following steps to solve this task with Keras BiLSTM model:

Split all sequences into train / validation / test sets
Train BiLSTM on a train set to predict next number anywhere in a sequence
In test and validation sets randomly replace K elements at positions from 180 to 190 with 0 (a number that does not exists in original sequences).
Use pre-trained BiLSTM to predict true values of '0' elements in validation and test sets

Please help with the following questions:

How should I represent data and classes for BiLSTM in this case? It looks like my data and classes are the one and the same thing. Both 1...9 numbers are data and corresponding classes to BiLSTM.
What data structures, encodings in this case should I create to train and predict with Keras BiLSTM?
How to evaluate quality of this model on a train and test sets ?

Any other ideas of using other models, in particular Transformers (PyTorch, Tesnsorflow) are very welcome, thanks!

The framing of your problem is close to so-called language modeling task. Because your input data is fixed-length samples, you can use a seq2seq model with fixed-size context embedding.
What this means is you would essentially have an encoder, Bi-LSTM for example which encodes your input into a fixed representation (by concatenating the final output states of forward and backward LSTM), and a decoder, for example LSTM, which sequentially produces the output tokens.
Your objective function could be a mean of cross-entropy loss over each output token, or a more complex loss like CTC.
You can also simplify it by just predicting the masked tokens, instead of the whole sentence, as output of your neural network.
The fact that your tokens are integers makes no difference and actually simplifies the embedding. You can simply feed the data as is to an embedding layer in Keras or PyTorch.
If you use PyTorch, there is this tutorial that I would recommend, using transformer instead of LSTM.


What is the role of logistic function in logistic regression? After training Logistic regression on training data, getting each test sample to algorithm computes weighted sum of sample features. If this is greater than zero, we know the sample is from negative class and otherwise, sample is positive. I want to know why we need to calculate the logistic function that map our weighted sum to range zero and one. One reason can be that we want to interpret the results as probability. are there any other reasons?

Because it makes the network learn the desired output (usually 1 or 0) much easier. It is possible to omit the logistic function (aka sigmoid function) and achieve desired results, but the network has to learn to specifically map each input to one or zero which is hard when it has the ability to map it to any real number.
The logistic function makes it easier because the network just has to learn to map it to a positive or negative number.

Now the network can map an input to -6 if it wanted to and still produce the desired output -1, whereas before it has to output specifically -1.

Also, @Media logistic regression doesn't need a non-linearity for this purpose. It will still only be able to separate the data in a linear fashion.


How to represent the number of neurons in an LSTM for architecture schematic? I'm trying to visualise a neural network schematic and found a great tool for building schematics here http://alexlenail.me/NN-SVG/index.html. I've edited the SVG file to change one of the dense layers into a LSTM layer, and the input to time series instead of singular neurons.

At the bottom of the image there is some set notation detailing how many neurons is in each layer. I'm not too familiar with set notation. I'm not quite sure how to represent the LSTM layers the number of neurons in accordance with the set notation given?

Its a singular LSTM layer with hidden_size=8 and the input/output size is obviously just the previous/next layers (5 and 8).

Also, the same question applies for the initial time series inputs; how to represent 5 time series of (say 100 points long) in set notation?

Any additional advice on how it might be better to represent the architecture would also be appreciated! Thanks!

Within the recurrent cell, you can add the hidden units- often, we se this notated as:

Unit 1
  ...
Unit n

Where n is the dimensionality of the recurrent representation.


Using the GA R package to optimize the weights of a MLP neural network The neural network I am trying to evolve uses the tanh as an activation function in each neuron and has a topology of 1-5-1, so I need at least 5 weights. The solution of the GA is a real-number vector of length 5, which represents the weights of the network and each weight should take values between -5 to 5. I wrote an R function to use as a fitness function which returns the mean squared error (MSE) of the output data in comparison to the desired output. I want it to learn the cubic function. The input data I am using is

input<-seq(-1,1,0.02)

and the output data is its cubic function

des_out<-input^3

The evaluation function is the following

evalnn<-function(x){

mse<-0
for(i in 1:length(input)){

     nn.out <- tanh(x[1]*input[i]) + tanh(x[2]*input[i]) + tanh(x[3]*input[i]) + tanh(x[4]*input[i]) + tanh(x[5]*input[i])
     mse <- (des.out[i] - nn.out)^2 + mse

}
return(-(mse)/length(input))
}

I have set the return value to negative, because I want the smallest value to be thought as the best fit.

gann<-ga(type="real-valued", fitness=evalnn,min=c(-5,-5,-5,-5,-5),max=c(5,5,5,5,5),popSize=100,maxiter=150,pmutation=0.01,pcrossover=0.8)

What I always get back from the GA are weights that make up a linear function although I have been experimenting with the ga's parameters quite a lot, i.e. I have tried all crossover methods and mutations. https://cran.r-project.org/web/packages/GA/GA.pdf.

The cubic function plotted together with my output function.

I have used a linear function as training data before the cubic with this implentation and worked. It has trouble with the non-linear.

If anybody could figure out why do I get this, is there something I have missed.
Thank you

I finally got the results I was looking for.

Apparently a single hidden layer neural network cannot learn the cubic function. Instead of just evolving the weights, I included the topology of the neural network into the chromosome. My neural network can have maximum 5 hidden layers with maximum 5 neurons in each layer and the chromosome contains information for the structure of the network and the selection of the weights.

When I ran the GA with population of 200 and for 100 iterations I got a full topology of:

1(input) - 4-2-3-2-2 - 1(output), a mean squared error of 0.0051 and the following plot

I would be glad if anyone has any comments or questions.


How to classify a set of words into one of the given labels I have three labels: amusement, calm and energetic.
I get sets of words like:
Set1 = {Cloud
Sky
People in nature
Plant
Flash photography
Happy
Shorts
Grass
Leisure
Recreation}
Set2 = {Plant
Green
Natural landscape
Natural environment
Branch
Tree
People in nature
Shade
Wood
Deciduous}
I want to classify these group of words into one of the labels. What do you guys think? Set1 should be labelled energetic and Set2 should be labelled calm.

There are probably many variants but here are two simple approches:

Using pretrained word embeddings, you can calculate the semantic similarity between two words. For example you could use cosine to measure the similarity between the vector of a target word (e.g. "calm") and every word in the set (e.g. "cloud"). Then the mean across words in the set gives how much the set is associated to the target, and you can pick the target which has the max similarity.
Using WordNet to directly obtain a semantic distance/similarity between words. The method is similar to the one above.

Note that there are many improvements that can be done to these basic ideas, for example you could use a predefined set of words related to "calm" instead of just the word "calm" (you can get the most similar words from WordNet for instance). There also many options for the aggregation across the set of words.


How is data science applied in investment banking? What do data scientists do at investment banks? What tools are they using? What kind of analysis are they doing? Why are they doing it? Etc.

In investment banks, this role is called "Quant" (Quantitative analyst). A quantitative analyst is a person who specializes in the application of mathematical and statistical methods to financial and risk management problems. The occupation is similar to those in industrial mathematics in other industries.

https://www.phaidoninternational.com/industryinsights/the-growth-of-quants-in-investment-banks-71503010311421

https://www.investopedia.com/articles/active-trading/111214/quants-what-they-do-and-how-theyve-evolved.asp

https://www.investopedia.com/articles/financialcareers/08/quants-quantitative-analyst.asp

https://en.wikipedia.org/wiki/Quantitative_analyst


How to train neural network that has different kind of layers If we have MLP then we can easily compute the gradient for each parameters, by computing the gradient recursively begin with the last layer of the network, but suppose I have neural network that consist of different type of layer for instance Input->convolution layer->ReLu->max pooling->fully connected layer->siftmax layer,  how do I compute the gradient for each parameters ?

The different layers you describe can all have gradients calculated using the same back propagation equations as for a simpler MLP. It is still the same recursive process, but it is altered by the parameters of each layer in turn.

There are some details worth noting:

If you want to understand the correct formula to use, you will need to study the equations of back propagation using the chain rule (note I have picked one example worked through, there are plenty to choose from - including some notes I made myself for a now defunct software project).
When feed-forward values overlap (e.g. convolutional) or are selected (e.g. dropout, max pooling), then the combinations are usually logically simple and easy to understand:

For overlapped and combined weights, such as with convolution, then gradients simply add. When you back propagate the the gradients from each feature "pixel" in a higher layer, they add into the gradients for the shared weights in the kernel, and also add into the gradients for the feature map "pixels" in the layer below (in each case before starting calculation, you might create an all-zero matrix to sum up the final gradients into).
For a selection mechanisms, such as the max pooling layer, you only backprop the gradient to the selected output neuron in the previous layer. The others do not affect the output, so by definition increasing or decreasing their value has no effect - they have a gradient of 0 for the example being calculated.

In the case of a feed-forward network, each layer's processing is independent from the next, so you only have a complex rule to follow if you have a complex layer. You can write the back propagation equations down so that they relate gradients in one layer to the already-calculated gradients in the layer above (and ultimately to the loss function evaluated in the output layer). It doesn't directly matter what the activation function was in the output layer after you backpropagate the gradient from it - at that point the only difference is numeric, the equations relating deeper layer gradients to each other do not depend on the output at all.
Finally, if you want to just use a neural network library, you don't need to worry much about this, it is usually just done for you. All the standard activation functions and layer architectures are covered by existing code. It is only when creating your own implementations from scratch, or when making use of unusual functions or structure, that you might need to go as far as deriving the values directly.


Accuracy on Validation and Test set, Overfit? Just a quick question, I am building a ML model right now however I am receiving very similar (72.2 and 72.4 for example)% for both Accuracy and F1-Score on my Validation Dataset and my unseen Test Set respectively. This is occuring on most of the baseline models I have produced for my problem right now.
Is this showing that my model is completely overfitting or just acting completely random and getting lucky.
Thanks

If the training set, validation set and the unseen test set (as you put it) have the same score, but lower than you expected then the model has not overfitted.
An overfitted model would have higher scores for the training data at least, and depending on how you optimised the hyper parameters on the validation data but lower for the unseen test set.
The more likely outcome is that your model has underfitted i.e. low but consistent scores across all 3 sets of data.


What is the difference between lightgbm.LGBMModel and lightgbm.LGBMClassifier? https://lightgbm.readthedocs.io/en/latest/Python-API.html

I have compared both of them in the lightgbm documentation and could not figure out which one I would choose and why I would choose one over the other?

Glancing at the source (available from your link), it appears that LGBMModel is the parent class for LGBMClassifier (and Ranker and Regressor).  You should probably stick with the Classifier; it enforces proper loss functions, adds an array of data classes, translates the model's score into class probabilities and from there into predicted classes, etc.


How to choose a kernel function and a feature mapping function? Although, after extensive of reading, I know the concepts of support vector machines pretty well by now, I have trouble translating the concept of the kernel function $K$ and the feature mapping function $\phi$ to a simple example such as the following.

My example data $x \in \mathbb{R}^2$: $(1,0), (4,0)$ are from one class, $(2,0), (3,0)$ are from another.

So here are my two questions:

Would $\phi((x_1,x_2))=(x_1,x_2,(x_1-2.5)^2)$ be a wise choice for the mapping function $\phi:\mathbb{R}^2 \to \mathbb{R}^3$ ? If not, what $\phi$ would be a wiser choice?
What would be the corresponding choice for the kernel function $K$?

To answer your questions:

Yes, $\phi((x_1,x_2))=(x_1,x_2,(x_1-2.5)^2)$ is a good choice. You can also remove the second dimension $x_2$, as it's zero for all of your training examples. By doing so, you will have a mapping function $\phi:\mathbb{R}^2 \to \mathbb{R}^2$ which is: $\phi((x_1,x_2))=(x_1,(x_1-2.5)^2)$. (Also note that mapping into higher dimension in not necessary; mapping to a space where data becomes linearly separable is sufficient.)
Kernel function $K$ calculates the dot product of points in the new space, so we have:

$$K(x,y)=\left\langle \phi(x),\phi(y)\right\rangle=\left\langle (x_1, (x_1-2.5)^2),(y_1,(y_1-2.5)^2)\right\rangle=x_1y_1+(x_1-2.5)^2(y_1-2.5)^2 \to K(x,y)=x_1y_1+((x_1-2.5)(y_1-2.5))^2$$


Classifier runtime evaluation I am looking for an example or tutorial of a system predicting numeric values by the use of various classifiers like SVM, Decision trees, ANN or KNN which optimises its choice of algorithm and parameter settings at runtime. Assuming there is unlimited input data and the system can train itself nonstop, the system should be able to adapt to different input data and choose the most accurate prediction setting as possible. Examples in Python, C++ or JAVA would be preferred. Thank you in advance!

I don't have much hands-on experience in this, but nevertheless I think this kind of processing of streamingdata is called "online learning"; and finding optimal hyperparameters for any type of learning (online or batch) automatically such that the system is self-tuning is an area of active research.

Prof. George Hinton talks about it generally in this video "Lecture 16C : Bayesian optimization of neural network hyperparameters" in this playlist: https://www.youtube.com/playlist?list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0
(13min)

If you like audio, a listener asked a similar question in this  Episode of the Talking Machines podcast:
(about 8 minutes in)

Clever switching of the learning algorithm also has been explored, see "Bagging" or "Boosting" on Wikipedia, or Ensemble learning

For a relatively easy-to-use Java-based GUI tool you might try "MOA"- Massive ONline Analysis


Un-learning a single training example from a trained model I was going through the paper "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" by google on which suggests best practices for models in production. In a section about privacy controls in the data pipeline it says:

  Finally, test that any user-requested data deletion propagates to the
  data in the ML training pipeline, and to any learned models.

I understand about the data deletion from the data pipeline but is it even possible to "un-learn" a single training example without retraining on the new data? They have mentioned in the paper that the practices are being used in google at some point or other, so there might be an efficient way but I'm unable to get any information on this.

I am looking for any literature on this or any ideas about how one would go on solving this problem.

Edit:
On further research, I found this paper which focuses on the specific problem. Though making a lot of assumptions they propose a method for k-means too. Looks like this is an upcoming research area and would require time to develop!

is it even possible to "un-learn" a single training example without retraining on the new data?

To the best of my knowledge, the answer is no except in some very special cases.

The most obvious exception that comes to mind is instance-based learning, such as kNN: since the "model" itself consists only of the set of training instances, it's straightforward to remove an instance.

In general, supervised ML relies on generalizing patterns based on the instances from the training set. Any non-trivial model consists of multiple such patterns, with every pattern potentially resulting from a different subset of instances. Even if there was a way to trace which instance participated to which pattern (that would be extremely inefficient), removing any pattern would probably cause the model to fail.


Unbalanced dataset on image classification, is it better to lose samples and balance it? I am dealing with a binary image classifier. I'm using a CNN to predict if an image is positive or negative. The problem is that the positive class represents only the 2% of the total samples. In this case, I can get lots of images, so that is not a problem. My question is about what is the best option to set up the dataset.
One option is to have an unbalanced dataset, if it is extremely unbalanced like this case, the net won't learn, it'would only predict the major class. The other option is to artificially balance the data. What I've done is preparing a dataset with a 40% of the positive class and a 60% of the negative class. This dataset does not represent the real problem and is making lose a lot of samples. Is there a better option?
Thanks.

Instead of downsampling the dataset, you should try using the Upsampling technique. Sometime downsampling leads to loss of data. Use augmentation techniques to increase the size of your dataset.


Preprocessing to-be-predicted data in ML with R - "learn" and "apply" features I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem.

I apply the "usual" preparation for modeling (dummy variables, normalization, PCA etc., of course in the necessary cases) to the training data. So far, so good. But when I get the to-be-classified new data to make prediction the model constructed above, it's evident that I must apply these preparatory steps to this new data set as well.

And the problem arises here, because if I simply apply the preparatory steps for my new data in turn again, these doesn't take into consideration the characteristics of the training data. So, if I convert the new data factors into dummies, then it takes only the existing factor levels in the new data into account; if I min-max normalize the new data, it will be normalized according its own min-max values, disregarding the values in the training data; and if I use PCA, then the resulting components from the new data will be totally independent of the training data.

So essentially my point is that applying the same conversions separately to the train set and the new data set(s) (which could be only one observation as well), then the two resulting transformed sets will have nothing in common, so the prediction will be baseless.

I found some traces that in some cases there is some "learning" step in the training phase in these transformations as well and apply this "knowledge" to new data (caret and Sklearn, for instance, with "predict" could transform to the new data with characteristics learned from the training data), but generally speaking this inconsistency remains unmentioned otherwise.

What is the correct practice here?

In short, the correct practice is to use the scaling/preprocessing parameters you found and used on your training set, and apply them to your test set.  the motivation for this is that you should be able to deal with someone coming to you with one new observation they want a prediction on.  You will be able to preprocess that single observation using your training set preprocessing parameters.

Ill go through some of the issues you brought up:

Dummy Variables:
This might entail you using setdiff() in R to find what columns are in your training set but not in your test set, then make a column in your test set with a column of zeros.  This is the simplest way to do it, but it is very data/purpose dependent.  You may also want to change this or look for better options depending on what modeling technique you're using.

Min/Max Normalize:
When you scale your training set, save the min and max of each column, and scale your test set using those parameters (NOT the min/max of your test set).

PCA:
When you perform PCA on your training set, you are transform your data based on rotations that were calculated on your training set.  You want to perform those pre-calculated rotations on your test set.  Check out ?predict.prcomp.  This will show you how to apply your rotations from your training set onto your test set.


What's the point with neural networks if you can only predict linear test data? So, I have tried all the different activation functions listed on https://keras.io/api/layers/activations/. I can indeed approximate any nonlinear function in the training range perfectly well - but for any data outside the training range, I have a model that is limited to linear functions. For example, I tried to approximate the sinus function and had great results within the training range but was left with a linear function outside this range. I used a network with 3 hidden ReLU layers(with 16 units per layer) and an affine output layer. Here is the good approximation within the training range:

and the predictions outside the training range:

The same appends when trying to approximate any other nonlinear function. This is unfortunate because you would rather want your model to generalize well than to perform good exclusively on training data :(
I could of course use a sine explicitly as an activation function but that seems to ridicule(maybe a strong word but you get the idea) the "self-learning" of neural networks.
This seems to make neural networks look very limited in my eyes - am I missing something?
I really appreciate your time!

Neural networks can generalize and successfully predict outside their training data. This ability is hindered by overfitting, where the network memorizes the training data and does not perform well on unseen data.
As with any other problem faced with neural networks, it is key for the network to have an inductive bias that is appropriate for the problem data. From this SO answer:

Every machine learning algorithm with any ability to generalize beyond the training data that it sees has some type of inductive bias, which are the assumptions made by the model to learn the target function and to generalize beyond training data.

For instance, convolutional networks perform well in image data due to their spatial locality inductive bias.
The main problem in your examples is that the function you are modeling is periodic while the activation functions you are using lack a periodic inductive bias. This problem is studied in the article Neural Networks Fail to Learn Periodic Functions and How to Fix It presented at NeurIPS 2020. From the abstract:

[...] we prove and demonstrate experimentally that the standard
activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a “periodic” inductive bias. As a fix of this problem, we propose a new activation, namely, $x + sin^2 (x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations.


Why don't we want Autoencoders to perfectly represent their training data? From Ian Goodfellow's Deep Learning Book:

  If an autoencoder succeeds in simply learning to set g(f(x)) = x everywhere, then it is not especially useful. Instead, autoencoders are designed to be unable to learn to copy perfectly

I don't understand this part. g is the decoder, and f is the encoder. Why is it undesirable for the encoder and decoder to perfectly represent the input data x?

Another way to frame this question is - why do autoencoders require regularization? I understand in predictive machine learning, we regularize the model so that it can generalize beyond the training data.

However, with a sufficiently massive training set (as is common in Deep Learning), there should not be a need for regularization. To me, it seems desirable to learn g(f(x)) = x everywhere, and I don't understand why the author says otherwise.

The only way an autoencoder can to perfectly represent the training data is by having a hidden layer that is the same size as the input and output layer. Thus, there would be no compression of the training data. The data would be its own model (i.e., f and g are identity functions).

The goal of an autoencoder is to learn a compressed, lossy model of the data.


Is t-SNE just for visualization? I have used the t-SNE algorithm to visualize my high dimensional data. However, I was wondering if this is a practical method for inference?

It's a dimensionality reduction algorithm. Inference is the problem of determining the parameters, or labels, that best fit the model for a given input once the model parameters have been learned, or estimated.


Quadratic approximation of L1 regularized cost function I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't give me (7.23).

Thanks for your help!

The analysis becomes a little bit hairy because of the discontinuous derivative of the L1 penalty.  There may be a better way of phrasing the argument, but here's something.

Taking the derivative of $|w_i|$ gives $\operatorname{sgn}(w_i)$, unless $w_i=0$ which we will take as a special case.
In the main case, when $w_i\neq0$, taking the derivative w.r.t. $w_i$ equal to zero gives:
$$\begin{align*}
H_{ii}(w_i-w_i^*) + \alpha \operatorname{sgn}(w_i) & = 0 \\
\Longrightarrow w_i &= w_i^* - \frac{\alpha \operatorname{sgn}(w_i)}{H_{ii}}.
\end{align*}$$
Now for a trick.  From the original loss formulation, it's clear that the optimum $w_i$ will have the same sign as $w_i^*$ (or is zero): otherwise, switching signs decreases the first term and doesn't change the second term.  So we have
$$\begin{align*}
w_i &= w_i^* - \frac{\alpha \operatorname{sgn}(w_i^*)}{H_{ii}} \\
 &= \operatorname{sgn}(w_i^*) \left( |w_i^*| - \frac{\alpha}{H_{ii}} \right).
\end{align*}$$
And again, knowing that $w_i$ and $w_i^*$ must share sign, we must have that $|w_i^*|-\frac{\alpha}{H_{ii}}\geq0$.  If not, then this is not a location of zero-derivative, and we have to look to our other case.  In that case, simply $w_i=0$, and we can tack that on as a separate case here using the maximum given in the text.  (We should also confirm that $w_i=0$ is not the minimum when $|w_i^*|-\frac{\alpha}{H_{ii}}>0$, which is easy to do.)


Multiprocessing or Parallel Computing Python Code I am working on bioinformatics big datasets; training set and optimisation taking huge time to execute. I check and found that training and optimisation compiling on one core of cpu and because of that reason its taking huge time.

I  tried to get look for multiprocessing or parallel computing code but I didn't get any good relevant sample to code to execute on my script. I am new in multiprocessing, can anyone share the multiprocessing or parallel computing code in Python please.

If you're using scikit-learn, there is a parameter for most learners called n_jobs. This parameter can be set to -1 to utilize all processors. For more details, see scikit-learn n_jobs parameter on CPU usage & memory


error in running exe file after changing .py to .exe I have changed my python code to an .exe file using pyinstaller and anaconda. In my python script ("untitled.py") I import opencv which I have already installed from source using cmake. After creating exe file I run untitled.exe in cmd windows. However, I get the following error:

File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
File "cv2_init_.py", line 96, in
File "cv2_init_.py", line 58, in bootstrap
File "cv2_init_.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.

There is config.py in OpenCV folder, but it cannot be found. I tried this:
pyinstaller -F  --hidden-import ‘OpenCV’ untitled.py

But still does not work. I wanted to ask how I can solve this issue?

A quick google search of the error message reveals two issues describing the issue and potential solutions.
One of the issues is in the pyinstaller repo, and suggests adding the cv2 path to the pyinstaller invocation:
pyinstaller --onefile sample.py --paths "/root/opencv-4.3.0/release/lib/python3/cv2"

You can get the actual path by running a simple python program:
import cv2
print(cv2.__file__)

The other issue is in the opencv repo and may apply especially if you compiled opencv yourself.
UPDATE: with respect to the error ImportError: DLL load failed while importing cv2, according to the FAQ, you should install the Visual Studio C++ redistributable


What kind of research I can do with my data set? I have a data containing 200K ads (sale and rent property in the United Arab Emirates, or UAE). Here is a sample:

190459 obs. of  29 variables:
 $ id                  : chr  "tnydu.biz/DafdVO" "tnydu.biz/DafloP" "tnydu.biz/Dafmvy" "tnydu.biz/Dafuy5" ...
 $ AdType              : chr  "sale" "sale" "sale" "sale" ...
 $ AgencyFees          : num  NA NA NA NA NA NA NA NA NA NA ...
 $ Amenities           : chr  "CentralA/C&Heating,SharedPool,SharedGym,Security,BuiltinWardrobes,ViewofWater,ViewofLandmark" "Study,CentralA/C&Heating,Balcony,Security,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchenAppliances,Vi"| __truncated__ "CentralA/C&Heating,SharedSpa,Security,ConciergeService,MaidService,CoveredParking,BuiltinWardrobes,Walk-inCloset,BuiltinKitchen"| __truncated__ "CentralA/C&Heating,Balcony,Security,CoveredParking,BuiltinWardrobes,BuiltinKitchenAppliances,ViewofLandmark" ...
 $ AnnualCommunityFee  : chr  "9000" NA NA NA ...
 $ AreaDescription     : chr  "\r\n            \r\n                \r\n                    1.9 km from Najmat Reem Marina\r\n                \r\n            \"| __truncated__ "\r\n            \r\n                \r\n                    0.3 km from The Dubai Mall\r\n                \r\n            \r\n "| __truncated__ "\r\n            \r\n                \r\n            \r\n        " "\r\n            \r\n                \r\n                    0.7 km from The Dubai Mall\r\n                \r\n            \r\n "| __truncated__ ...
 $ Bathrooms           : num  1 NA NA NA 5 3 NA NA 3 3 ...
 $ Bedrooms            : num  0 3 0 1 3 2 2 2 3 3 ...
 $ Building            : chr  "HydraAvenueTowers" "BurjVista1" "TheAddressDubaiMall" "TheAddressDowntown" ...
 $ City                : chr  "AbuDhabi" "Dubai" "Dubai" "Dubai" ...
 $ PublishDate         : POSIXct, format: "2015-10-30" "2015-11-11" "2015-11-13" "2015-11-09" ...
 $ DealerCode          : chr  "599942" "604296" "604296" "604296" ...
 $ DealerName          : chr  "STARWOOD PROPERTIES BROKER" "BLUE PALACE REAL ESTATE BROKERS" "BLUE PALACE REAL ESTATE BROKERS" "BLUE PALACE REAL ESTATE BROKERS" ...
 $ Developer           : chr  "HydraProperties" "EMAAR" "EMAAR" NA ...
 $ Furnished           : chr  NA NA NA NA ...
 $ ListedBy            : chr  "Agent" "Agent" "Agent" "Agent" ...
 $ Location            : chr  "City of Lights, Tamouh Marina Square" "Downtown Dubai, Dubai" "Downtown, Cairo" "Downtown Dubai, Dubai" ...
 $ LocationGPSLatitude : num  24.5 25.2 30 25.2 25.1 ...
 $ LocationGPSLongitude: num  54.4 55.3 31.3 55.3 55.1 ...
 $ Price               : num  900000 3822888 2150000 3200000 5500000 ...
 $ PriceSqFt           : num  1129 2185 3909 3422 2431 ...
 $ PropertyReference   : chr  NA "BP9801" "DT-K20" " BP8958" ...
 $ PropertyType        : chr  "apartment" "apartment" "apartment" "apartment" ...
 $ ReadyBy             : POSIXct, format: "2015-04-30" NA NA NA ...
 $ RentIsPaid          : chr  NA NA NA NA ...
 $ ShortLink           : chr  "tnydu.biz/DafdVO" "tnydu.biz/DafloP" "tnydu.biz/Dafmvy" "tnydu.biz/Dafuy5" ...
 $ Size                : num  797 1749 550 935 2262 ...
 $ TotalClosingFee     : chr  NA NA NA NA ...
 $ VirtualView         : chr  NA NA NA NA ...

I'm looking for help figuring out what kind of questions I can answer using these data.
For example,
1. What the mean price per year for a 1-bedroom apartment to rent in different areas of Dubai?
2. How are supplies for apartments and villas distributed on the city map?
3. Does the number of bathrooms influence on sale/rent price?
4. Where is the best place for investing to get more profit from apartment cost and rent revenue?

I have already done some of this (plotted the map with mean price for studio in Dubai in thousands AED/year, 1USD = 3.66AED):

From your experience, what other questions can I ask? What I have to be aware in this data set? I can share my data set if anyone want to examine it (it is a 12mb zip).

There are several things that can be done, however as @Dawny33 mentioned, start with the problem you want solved.

As an example, If I was trying to solve: Which factors, influence the prices and how?

A simple analysis could be to run a linear regression model (lm) with PriceSqFt as a dependent variable and others as regressors. You can then select top 10 or 15 or how so ever many you like and look at their relationship.

This may help you answer some simple questions like:

How do prices vary by dealers, by property type, by location, by size?
Does move in ready date make a difference to rent / prices?
Do specific kind of properties have later than usual ready dates?
Is there a negative or positive correlation between size and pricesqft?
Are certain dealers over or under priced compared to other identical
properties and location?

You can also create a plots using pairs or other functions to look at pairwise relationship for initial analysis and building on top of what you see.

If you really want to take it a to the next level, you can create a prediction model to predict the price based on select parameters.


value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame How Can I save a DataFrame as a CSV in HDFS?

I'm trying with this:

df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")

This is not standard part of the API of DataFrames. You can either map it to a RDD, join the row entries to a string and save that or the more flexible way is to use the DataBricks spark-csv package that can be found here.

If it's just one column you can map it to a RDD and just call .saveAsTextFile(filename)


EfficientNet: Compound scaling method intuition I was reading the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks and couldn't get my head around this sentence:

  Intuitively, the compound scaling method makes sense because if the input image is bigger, then the network needs more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image.

In the case of a big image, why the network needs more layers to increase the receptive field ? What does increasing the receptive field mean? Increasing its width/height ? If so, we can do it directly without increasing the number of layer in the network no ?
is "fine-grained patterns" referring to noisy shape we can see after visualizing convolution output ?

I feel like I am missing / misunderstanding something evident.

Receptive field refers to the number of input pixels that a convolutional filter will operate on. There's a nice distill article about how to calculate receptive field size for your filters (with a nice visualization of receptive field size) and an interactive calculator here if you're only curious about how receptive field size grows with changes to depth and filter size.

Increases to receptive field size typically come from adding layers and from increasing the kernel size. A larger kernel operates on more pixels, which grows the receptive field. Increasing the depth of your network refers to adding additional convolutional layers. These downstream filters operate on the feature maps produced by the initial conv. layers of your net, which increases the receptive field for the filters in those additional layers (if this isn't clear, this is a good guide). The distill article also goes into detail about how other operations affect receptive field size.

As for the claim of a gain in number fine grain patterns captured, it's more in line with the intuition that more filters will give the network more ways to learn specific features of your data. See articles around visualizing convolutional filters for a sense of what types of features are captured (this tutorial on object detection links to a nice visualization).

Hope this helps!


How to find the right regression model for my project I tried several things to find the best regression model with the best parameters but i can't go higher than 40% right predictions.

So i have 67741 rows in an excel file. the data looks like this after cleaning (4 columns only , is it enough ?) :

and the target rows like this :

I'll try to explain my process .

I went to this website https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html.
From this graph the models that should suit my data are Lasso and ElasticNet,
i got very bad score with this code :

classifiers = [
ElasticNetCV(cv=5, random_state=0,max_iter=40000), # i added the max_iter cause i got a warning saying that i should increase it
linear_model.Lasso(alpha=0.1,max_iter=40000)] # i added the max_iter cause i got a warning saying that i should increase it

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.33, random_state=42)

for clf, param in zip(classifiers, param_grid):
    name = clf.__class__.__name_
    clf.fit(X_train, y_train)
    print("=" * len(name))
    print("{}".format(name))

    print(clf.score(X_test, y_test))

The scores:

============
ElasticNetCV
0.002404878871672067
=====
Lasso
0.0066801704903023396

Then i tried several other models and i finally got something with:

BaggingRegressor(base_estimator=DecisionTreeRegressor(),
              max_features=0.5, max_samples=0.5)

Score :

================
BaggingRegressor
0.3460147571634854

So i used GridSearch and then cross validation score to get the best parameters but every time i launch it i got a different result something like that :

BaggingRegressor(base_estimator=DecisionTreeRegressor(), bootstrap=True,
bootstrap_features=False,
              max_features=0.2, max_samples=0.7, n_estimators=10,
              n_jobs=None, oob_score=False, random_state=None, verbose=0,
              warm_start=False)

i used this that way :

param_grid = [{'max_samples': [0.1, 0.2, 0.5, 0.7, 1],
           'max_features': [0.1, 0.2, 0.5, 0.7, 1],
           'n_estimators': [5,10,15,20,25]}

def grid_search(clf, param_grid):
    grid_search = GridSearchCV(clf, param_grid, cv=5)
    grid_search.fit(X, y)
    print(grid_search.best_estimator_)
    print("=" * len(name))
    print(grid_search.best_params_)
    print("=" * len(name))
    print(grid_search.best_score_)

and cross validation like that : scores = cross_val_score(clf, X, y, cv=10)

Sorry if it's a bit long, with 67000 rows i can't get something more than 30% correct predictions ? what's the problem ?

Thank you

This answer is based on additional information regarding the data provided in a comment to the question:

  [...] So it's about delivery The
  first and second are Id corresponding to cities, the 3rd is a date,
  and the last is an id corresponding to the truck that is changing (i
  sorted it that's why its the same in the first 3 columns). And the
  target Y is delay.

If you just add the time column in a unformated way to the regression, the method(s) will not be able to digest the information in a good way. Think about your problem. Delivery may be delayed because of trafic conditions (time, or the day of the week). I suggest that you encode time e.g. in hour of day, day of week, week of year (or so) and add all these (possibly one-hot encoded) variables/features to your model (together with the remaining features). You can use regularization (e.g. lasso) to "shrink" unnecessary features. This should give you an okay fit.

Essentially you need to really make sure you get as much information as possible out of the "date" column. In Python you can encode dates by predefined functions, e.g.

import datetime
Todays date datetime.datetime.today()
Day of week datetime.datetime.today().weekday()
Week of year datetime.datetime.today().isocalendar()[1]

Find an example of how to apply lasso here.


How the vector of weights assigned to a neural network? You have set of input nodes, hidden nodes and output nodes although I have one output node in my case.

We assign weights to the connections between those nodes. So is there any pattern of assigning weights to those connections?

This is one pattern of assigning weights to neural network. See the another way of assigning the weights.

Please tell which one of the following is the correct way of assigning the weights to the neural network?

It looks like when you say "way of assigning the weights", that you mean "what order are weights counted in, so that I know which weights connect between which neurons".

There is no formal "correct" way of doing this for all neural networks. However, in practice for feed-forward networks like your diagram, you would choose to use a matrix, not a vector, to represent the weights connecting layers. That is how pretty much all standard libraries will represent weights.

A matrix uses two indices (call them $i,j$ in this case) to identify a single scalar value. If we call the weight matrix $W$, then an individual weight is $W_{ij}$.

To determine how the weights connect between neurons, then you index the input layer neuron with $i$ and the output layer neuron with $j$.

In math notation, looking at your diagram, call each input value $x_i$ and each hidden value $h_j$, then the formula for calculating a single $h_j$ would be:

$h_j = f(b_j + \sum_{i=1}^{N} W_{ij}x_{i})$

Where $f()$ is a transfer function (such as sigmoid), $N$ is the number of input features, and $b_j$ is the bias for hidden layer neuron $j$.

You will also often see this written using matrix notation:

$\mathbf{h} = f(W\mathbf{x} + \mathbf{b})$

. . . this is not only clear and simple notation, but using matrix maths like this to describe a neural network is what allows us to use high performance libraries on GPUs such as TensorFlow.


How to apply SelectKBest in azure machine learning I am very new to machine learning, I started to understand fundamental and using it in azure studio, I see that there is an algorithm in scikit for feature selection called SelectKBest, then I searched azure studio and I coudn't find it by name, I was wondering if is it supported there?

I have found there are other feature selection algorithms there

Please see the below screen shots


You can use a density based method such as Local Outlier Factor in order to do this.

If you want to identify outliers in each column independent of other columns, then you would apply this method separately to each column. (Note that this is different from identifying outlier data points considering all columns together.)


How is cross validation used to prune a decision tree As I understand it one can use cross validation to help find the optimal pruning of a classification or regression tree, for example, implemented here in Matlab.

I do not understand the following about this process. Since each cross validation fold has random data, it may be that the trees fit on each data set may be different and have differing depths. How can one find the best pruning level over all such trees then?

I don't know how this is implemented in matlab. I know that some packages use cross validation to decide whether to grow the tree or not. Quite simply, they decide whether to grow based on that evaluation.

(Notice what a lot of packages call pruning is: during training, they mark branches with some score criterium and then remove if the user wants afterwards.)

Personally, I use sklearn which does not have this feature. So, I just do a grid search for several values of max_depth and use whatever maximizes accuracy or whatever score I want.


Multiple Object recognition in image using deep Learning I am working on recognizing object classes in images using neuronal nets so I could make classifiers for cats, dogs... using Imagenet and some Conv nets famous architectures but my problems is if I had an image that contain a human and a cat and a car how can I detect and recognize 3 of them?

It falls under the multi label object detection problem. There have been lots of advancements done in this field and you should definitely look at implementations like :

Yolonet: https://pjreddie.com/darknet/yolo/
Squeezenet: http://songhan.github.io/SqueezeNet-Deep-Compression/


How to add more theta parameters into my logistic regression? I am a complete beginner in machine learning and coding in python. I have been tasked with coding logistic regression from scratch in comparison with using sklearn. My question is, with my code below I believe I have set the number of thetas I want with:

"X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]"

but I am unsure how to prove this is true with my code below and its definition of theta, if I added more parameters (e.g. all 31 variables of this dataset [https://www.kaggle.com/uciml/breast-cancer-wisconsin-data] which is for classifying tumours) would I just need to add them into this list above? Any help pointing me towards the right direction just to understand this better would be appreciated.

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)

X = data["diagnosis"].map(lambda x: float(x))

X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]
X = np.array(X)
X = min_max_scaler.fit_transform(X)
Y = data["diagnosis"].map(lambda x: float(x))
Y = np.array(Y)

def Sigmoid(z):
    if z < 0:
        return 1 - 1/(1 + math.exp(z))
    else:
        return 1/(1 + math.exp(-z))
def Hypothesis(theta, x):
    z = 0
    for i in range(len(theta)):
    z += x[i]*theta[i]
return Sigmoid(z)enter preformatted text here

def Cost_Function(X,Y,theta,m):
    sumOfErrors = 0
    for i in range(m):
        xi = X[i]
        hi = Hypothesis(theta,xi)
        error = Y[i] * math.log(hi if  hi >0 else 1)
        if Y[i] == 1:
            error = Y[i] * math.log(hi if  hi >0 else 1)
        elif Y[i] == 0:
            error = (1-Y[i]) * math.log(1-hi  if  1-hi >0 else 1)
        sumOfErrors += error

    const = -1/m
    J = const * sumOfErrors
    print ('cost is: ', J )
    return J

def Cost_Function_Derivative(X,Y,theta,j,m,alpha):
    sumErrors = 0
    for i in range(m):
        xi = X[i]
        xij = xi[j]
        hi = Hypothesis(theta,X[i])
        error = (hi - Y[i])*xij
        sumErrors += error
    m = len(Y)
    constant = float(alpha)/float(m)
    J = constant * sumErrors
    return J

def Gradient_Descent(X,Y,theta,m,alpha):
    new_theta = []
    constant = alpha/m
    for j in range(len(theta)):
        CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)
        new_theta_value = theta[j] - CFDerivative
        new_theta.append(new_theta_value)
    return new_theta

def Logistic_Regression(X,Y,alpha,theta,num_iters):
    m = len(Y)
    for x in range(num_iters):
        new_theta = Gradient_Descent(X,Y,theta,m,alpha)
        theta = new_theta
        if x % 100 == 0:
            Cost_Function(X,Y,theta,m)
            print ('theta: ', theta)
            print ('cost is: ', Cost_Function(X,Y,theta,m))

initial_theta = [0,1]
alpha = 0.01
iterations = 1000
Logistic_Regression(X,Y,alpha,initial_theta,iterations)

Your pass your initial_theta into Logistic_Regression where it defines how the cost function and its derivative are evaluated. Just make initial_theta the same width as X. If you want your code to be fool-proof, check variable sizes within your function like

def Logistic_Regression(X,Y,alpha,theta,num_iters):
    assert len(theta) == X.shape[1], 'theta should have one coefficient per each column of X'
    ....

A more convenient solution would be to define initial_theta inside the function, based on X:

def Logistic_Regression(X,Y,alpha,num_iters):
    theta = np.zeros(X.shape[1])
    ....

In this case, you can be sure that the shape of initial $\theta$ is correct.


Learning the task execution steps for a target goal I am trying to model a network which can take :-
Input - A Target Goal (e.g. Go To Cinema Hall)
Output - A sequence of steps (e.g. Go To Bus Stop -> Onboard A Bus -> Drop at Location X -> Walk 100 meters.
My training data is a similar kind of Objective - Task Sequence Mappings.
Can anyone please suggest a suitable approach using DL4J to address this problem and model suitable solutions?

Input - A Target Goal (e.g. Go To Cinema Hall)
Output - A sequence of steps (e.g. Go To Bus Stop -> Onboard A Bus -> Drop at Location X -> Walk 100 meters.

I assume that there's no other information available to the model, such as geographical information about the possible locations (i.e. the training data is made only of examples objective + sequence). As far as I can tell this implies that there's no way to use similarity between two objectives, since there's no way to know if similar places are in the same area.
Under this definition, the objective is a single categorical variable. So there's just nothing to learn except mapping an objective with its corresponding sequence, no need for ML at all: just store the list which maps any objective in the training data to its corresponding sequence in a map (dictionary).


How do I draw the following Keras model? I am new to deep learning models, and I am trying to figure out the best way to draw a visual representation of the following model. I found a few tools that allow you to draw a model, but I am confused about what the 'best' way to draw this particular model would be.

Any suggestions or recommendations would be great.
Thanks!

You should Go along with Tensorboard, it looks more professional than the rest and it is easier to understand the model.
May I ask your exact purpose for the visualization? You could check others material with the similar purpose and go along with the general choice! Many would go for draw_convnet or PlotNeuralNet if the final use-case was to publish a blog. It also depends on which libraries and softwares you are using!


Setting up network communication between R and Python I want to share data between R and python via network sockets (I expect the solution to also work for R-R and python-python). I am interested in server being both R and Python side. For now I am interested in a simpler case of synchronous connections.

What are the best approaches, packages and formats to use for this task?

Apache Thrift or Protocol buffers are common choices for serialization.

If you want to share data in-memory, there is the Apache Arrow format which interoperates between many data types. If you want to share dataframes on disk, there is Feather.


Detecting voice in a noisy environment I'm trying to detect voice in a noisy environment. This is a New Zealand suburban/bush soundscape. We're using low quality microphones (but I don't think this has an effect on my problem) to periodically record the soundscape in an effort to record birdsong. One of the main challenges is we need to remove all potential human voice for privacy reasons. I'm stuck on the problem of how to detect voice in a noisy environment.

Here's an example of a spectogram with a person speaking 8 steps away from the microphone. There is crackle in the recording (don't believe this has an effect) and a bird chirping (activity around 2000hz) in the background. You can see the voice at 256-1024hz from 8sec to 13sec. The activity around 128hz from 15 to 22sec is a vehicle.

I'm trying to find a way of recognising voice activity (with high degrees of precision) just like in this file.

I believe that using a Machine Learning algorithm will have the best success due to the variability of voice and these sound files. (I have also been working on a mathmatical method based on energy proportion in the voice freq ranges per sample but this just doesn't seem to work with the variability of the sound files).

We only have around 10minutes of voice recorded like in the above spectogram. We then probably have about 24hours of non-voice recordings.

Q1: Will this be enough data, if not, would using data from freesound.org be advisable?

Q2: What types of algorithms would work best? - processing time is not a worry, the data contains noise from a multitude of sources (vehicles, animals, people, weather), I don't have much voice to work with(but I expect I can just use voice files from freesound.org) and I want to achieve 100% precision with not so much of a worry about recall.

Q3: Does anyone know of any projects in a similar space to mine where I could see what they've done? (there has been work done on voice detection [VAD] but from a few experiments it all isn't suited to as noisy environments as mine ie. this)

Q4: Last question, does anyone know any efficient techniques to label discrete events in continuous audio data to use in ML algorithms in Python? (At the moment I expect I'll be listening to files on Audacity and recording where voice starts and stops via a spreadsheet)

Otherwise, any other advice would be awesome!

(FYI I have done courses looking at ML-regression, Neural Nets and SVM/tree's but wouldn't call my self 'ML proficient')

Also, are there any other places I can ask these questions? I realise DSP.stack is more suited to solving small problems in the dsp area but I don't know where else to ask this!

This is in fact a very well suited problem for machine learning and has been explored in the past as published here. It is often called the cocktail party problem, because of the way our human brains can focus on a very particular voice pattern despite being drowned by countless other voices all around us.

I can think of two ways by which this problem can be approached the first is to transform the spectrogram to only keep the birds chirping. The second, would be to extract the voice patterns from a sample and then substract the extracted voices from the original sample. This second approach would not remove the noise from the truck.

I would go with the former technique. I would get a few thousands clips of birds chirping. Then introduce some other sounds into it. The mixed sample would be my input $X$ and the target $Y$ would be the clean bird sample. This was you can simulate many instances. However, you will need enough bird chirping data to cover the wide distribution which exists for birds making noise.

Q1

For this to work well, you will need A LOT of data. Yes, you should definitely try to get as many samples from wherever you can. You can then randomly introduce these human voices with the bird clips in order to get the dirty signals, then use these to train your model.

Q2

You most likely will not achieve 100% accuracy with existing machine learning methods. However, if you get close to 90% that is already a miracle and will truly be great. If you are using spectrograms I would first try convolutional neural network (CNN). These have been shown to produce very good results here. The same paper then proposes an autoencoder to clean up the spectrograms. I think this has a high probability of working, but will introduce smoothing to the spectrogram which may distort the birds chirping. Lastly, I would suggest using a generative adversarial network (GAN).

Q3

I would look into the cocktail party problem. You can also get some insight as to what techniques to use by looking at the extensive work that has been done on spectrograms in MRI spectroscopy (MRSI).

Q4

Like I said above I would suggest going from a $n \times n$ sample containing a mixture of sounds to a clean $n \times n$ sample with only the bird chirping. I think this strategy is the one where you will find the most existing literature and many groups have shown it to work.


Text classification with thousands of output classes in Keras Task:

I have a dataset with job titles and descriptions. The task is to predict tags for job by job title and description.

There are several tags for each job posting. Therefore, the number of labels for the model will be measured in tens of thousands.

Number of job postings = 78042

Number of unique classes (tags) = 1369

Questions:

Could you advise working types of neural networks (desirable in Keras)?

Or maybe you know how to solve this problem with the help of classical machine learning algorithms?

Still I will be very grateful for links to articles where similar problems are solved.

The ratio of number of examples to number of classes is not large. There are few classes which have high number of occurrences (from the second graph) and the distribution seems to be following power law.

In such cases, I'll advise the following strategy,

Sort your tags by number of occurrences and discard tags which occur very few number of times. This will make the problem more tractable.
You can benchmark the accuracy that you get from classical machine learning techniques. Many classical methods support multilabel output, you can check the documentation of support in scikit-learn library here.
You can do a mix of unsupervised learning and nearest neighbours approach. For example, learn doc2vec embeddings on your data including tags and suggest tags from nearest matching document for a new input. The number of documents is small by doc2vec standards and you will need careful tuning of doc2vec parameters.
With neural networks, you can use more suitable loss functions like multilabel soft margin loss.

I'd advise you start with classical techniques to first benchmark the potential accuracy.


What is an inception layer? I'am reading an article called "FaceNet: A Unified Embedding for Face Recognition and Clustering". And there, they use something called "inception". I guess it's something about layers, but I can't find any information about it. Just found some code, which doesn't explain much. Can somebody give a clue, how can I search for this correctly, or maybe, kindly and briefly tell the concept idea?

The paper cited in the question "FaceNet: A Unified Embedding for Face Recognition and Clustering" is available at https://arxiv.org/pdf/1503.03832.pdf . Page no 5 of this paper lists the reference [16] for the Inception module architecture.

The original paper introducing and describing this Inception architecture is - "Going Deeper With Convolutions", it is accessible at https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf .

An overview of inception modules is given in the diagram on page 4, its included here -

The key idea for devising this architecture is to deploy multiple convolutions with multiple filters and pooling layers simultaneously in parallel within the same layer (inception layer). For example, in the architecture shown in this diagram (figure a) employs convolution with 1x1 filters as well as 3x3 and 5x5 filters and a max pooling layer. Figure b demonstrates how the use of 1x1 convolution filters can achieve dimensionality reduction (since no. of channels is reduced).

The intention is to let the neural network learn the best weights when training the network and automatically select the more useful features. Additionally, it intends to reduce the no. of dimensions so that the no. of units and layers can be increased at later stages. The side-effect of this is to increase the computational cost for training this layer. To address this, a number of solutions have been suggested in the paper such as to deploy parallel computations for this architecture.

As an illustration of these concepts, the table on page 5 shows an incarnation of this architecture termed as "GoogLeNet" which was used for the ILSVRC 2014 competition.


Computing a cumulative distribution function in Python I'm trying to compute the distribution function of any of the usual distributions in Python... However, all the methods I've seen involve first drawing N samples from said distribution, and then order them somehow, and then do a cumulative sum.
In Mathematica, I can just do CDF[ChiSquaredDistribution[df],quantile]. If I want another distribution, I just substitute ChiSquaredDistribution for the name of that other distribution.
Is there a simple way, like in Mathematica, to compute a cumulative distribution function in Python?

If I understand you correctly, these can be found in scipy.stats. scipy has a long list of different distributions that you can use, both continuous as well as multivariate and discrete. All distribution functions have an underlying cdf method which allows you to calculate the cumulative distribution functions of that specific distribution. Using the Chi-squared distribution from your example would look as follows:
from scipy.stats import chi2

chi2.cdf(x=30, df=50)
# 0.011164780271550276

Using other distributions is as simple as importing that distribution and using the cdf method as shown above.


What variance threshold to consider in feature selection? Consider a numerical dataset with continuous variables, that has been scaled to end up with values in the [0,1] range. How can I compute a reasonable variance threshold for all the variables?

Usually 3 is taken as a threshold Value by standard deviation

Using z-score we can look for value above or below threshold like in code below:

threshold=3
mean = np.mean(data)
std =np.std(data)

for x in data:
    z_score= (x - mean)/std
    if np.abs(z_score) > threshold:
        ...


Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution? I have a categorical variable which has thousands of values, for a dataset which has millions of records. The data is being used to create a binary classification model. I am in the early steps of feature selection, but I am trying out Random Forest, Boosted Trees, and Logistic Regression to see what works.

If I find the frequency of each category and sort that, I see that about 50 values make up the top 80%. Is it valid to condense this feature as a binary on whether the value is in that set of values or not. By 'valid', I mean is it likely that this sort of transformation retain any useful information for a model? I have a concern that sorting these categorical values which do not have any order to them creates some incorrect assumptions.

The frequecy distribution looks a little like this:

A;10%
D;5%
E;1.2%
B;1.1%
...
Z;0.004%
W;0.0037%
...

Going one step further, is it valid to profile each class in my dataset and do the same? Say Categories A-F comprise the top 80% of class 0 and Categories D-H are the top 80% of class 1. I would convert:

data_id;cat_var
1;B
2;F
3;H
4;Z

to

data_id;cat_var_top80class0;cat_var_top80class1
1;1;0
2;1;1
3;0;1
4;0;0

Adding picture to hopefully clear up this idea. In yellow are the pre-calculated distributions of cat_var (***_id in the picture) for classes 0 and 1 based on the training set. On the right shows how the transformation would be applied:

(Edited after @D.W. suggestion).

To the best of my knowledge, there is nothing wrong with what you have in mind; thus it is certainly valid.. As you said, you have to try out all possible ways you can think and see which one works better. The most important point adopting a particular encoding is to be aware of you have certain amount of information getting lost and it varies one case to another depending on the problem at hand. For example in the case of binary coding based on high/low frequency sublevels, there will large of information (details) lost which could help the algorithm to do classification. I liked your idea of percentile coding based on cumulative density distribution. Maybe you want to look at Quantile-based discretization which is available in pandas.qcut.

The rest is my earlier answer as it was (below). I intended to suggest trying other techniques as well on top of what you have had in mind; but apparently the message was not clear. Please note that I do not seek to get my answer marked as final answer, as I aware that still it does not fully answer your question; it is simply to brainstorm and exchange ideas in length. ;-)

Perhaps you have already digged out enough about ways to convert categorical variables into continuous data. In case you did not and missed checking this answer, check it out. As discussed, they are many ways to convert cat-to-num, and your problem is one of the hardest yet the most common across many domains. You have a high cardinality in your categorical variable, and as far as I understood imbalance distributions of those sublevels and you are not sure whether messing up the order of those sublevels matters or not. You may need to try Ordinal Encoding (if order really matters) or weight of evidence (WoE) transformation (see this blog post for instance) that I have heard but not tried or even going beyond mixing them in a meaningful way to represent your categorical data properly.

What I have learned, despite all efforts in the field, that this problem is still an open challenge in data science and machine learning. Thus there is no best solution, or well-established method as far as I checked. Please do let me know if you come across one.


Google prediction API: What training/prediction methods Google Prediction API employs? The details of the Google Prediction API are on this page, but I am not able to find any details about the prediction algorithms running behind the API.

So far I have gathered that they let you provide your preprocessing steps in PMML format.

If you take a look over the specifications of PMML which you can find here you can see on the left menu what options you have (like ModelTree, NaiveBayes, Neural Nets and so on).


what is difference between the DDQN and DQN? I think I did not understand what is the difference between DQN and DDQN in implementation.
I understand that we change the traget network during the running of DDQN but I do not understand how it is done in this code.

We put the self.target_model.set_weights(self.model.get_weights())
In implementation of DDQN and this is added when action is finished for DQN  https://github.com/keon/deep-q-learning self.target_model.set_weights(self.model.get_weights()) is added to DQN in order to change the DQN to DDQN! But this happens when we are going out from running by break! There for there is no difference between them!

What is wrong in my mind? (Maybe the difference will be at test? Is this code for train and the test is done with setting the exploration rate=0  and then run this for just one episode with new weight we found? Is it right?

Therefore, what is difference between the presented DQN and DDQN of this link.

In particular, DQN is just Q-learning, which uses neural networks as a policy and use "hacks" like experience replay, target networks and reward clipping.

In original paper authors use convolutional network, which takes your image pixels and then fit it into a set of convolutional layers. However there are a couple of statistical problems:

DQN approximate a set of values that are very interrelated (DDQN
solves it)
DQN tend to be overoptimistic. It will over-appreciate being in this state although this only happened due to the statistical error (Double DQN solves it)

$$Q(s,a) = V(s) + A(s,a)$$

By decoupling the estimation, intuitively our DDQN can learn which states are (or are not) valuable without having to learn the effect of each action at each state (since it’s also calculating V(s)).

We’re able to calculate V(s). This is particularly useful for states where their actions do not affect the environment in a relevant way. In this case, it’s unnecessary to calculate the value of each action. For instance, moving right or left only matters if there is a risk of collision

As @emilyfy said self.target_model.set_weights(self.model.get_weights()) - the updating of the target model.


To One-Hot-Encode or not to One-Hot-Encode? I have been struggling to find proof for that but I couldnt

Every time I prepare dataset I face the same issue

when a column is a classification such as CountryCode or TaskType in this dataset

TaskType  CountryCode  Target
1         61           Red
1         962          Yellow
2         1            Yellow
6         61           Yellow
4         81           Red
2         1            Blue
1         61           Red
2         962          Green
4         61           Blue

if I applied the dataset as to different models such as linear regression, SVM, KNN, etc.

will these model consider CountryCode and TaskType as numeric fields and treat them as continuous data?

Shall I One Hot Encode these features before using them?

what is the best way to handle this scenario?

An intuitive explanation, why we should encode categorical features, is that otherwise there will be absolutely another sense of "closeness" between features of the same type. The model will treat this features as continuous and as a result if you have two points in your feature space (p1 with CountryCode 61 and p2 CountryCode 962) and then you add the third point p3 with CountryCode 81, then model will miss out, that it can't evaluate distances between 61, 962 and 81.


Difference between sequence length and batch size in time series forecasting I am using Keras for time series forecasting and I am trying to understand the tutorial on the offical site of keras about time series forecasting that you can find here (https://keras.io/examples/timeseries/timeseries_weather_forecasting/).
They use one keras-method called keras.preprocessing.timeseries_dataset_from_array and it has the following parameters (here is a documentation https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/timeseries_dataset_from_array):
    dataset_train = keras.preprocessing.timeseries_dataset_from_array(
    x_train,
    y_train,
    sequence_length=sequence_length,
    sampling_rate=step,
    batch_size=batch_size,
)

So my question is what is the difference between the sequence length and the batch size. I think the sequence is the size of the sliding window (x-features and one target y-value). But what is the batch size? Unfortunately I can't have a look at the output of this method as
   print(dataset_train) or
   print(dataset_train.head())

does not show me the data and I do not know any other function how I could have a look at the output of the method.
Has anyone of you had expercience with this method or generally with sequences and batches? I'd appreciate every comment.

Let's take a TS data = [ 1, 2, 3, 4, 5, 7, 8, 9, 10 ]
Call the function with these parameters
sequence_length=5, sampling_rate=1, sequence_stride=1, shuffle=False, batch_size=2
 shuffle, batch_size has no role in TS data creation. It will come into effect when you iterate on the returned Dataset.
 In this case, we will have the following data points,
[ 1, 2, 3, 4, 5 ]
[ 2, 3, 4, 5, 6 ]
[ 3, 4, 5, 6, 7 ]
[ 4, 5, 6, 7, 8 ]
[ 5, 6, 7, 8, 9 ]
[ 6, 7, 8, 9, 10 ]
batch_size
When you will iterate on this dataset, you will receive 2 records in each iteration.
If shuffle=True, records will be shuffled before batching.
for batch in dataset:
  inputs, targets = batch

In the above snippet, inputs will be a batch of records, not just one record. You may have the batch_size=1 if required.
targets

Targets corresponding to timesteps in data. It should have same length as data. targets[i] should be the target corresponding to the window that starts at index i (see example 2 below). Pass None if you don't have target data (in this case the dataset will only yield the input data)

This is a general-purpose function.
It is not deciding the Target on some logic i.e. Autoregressive approach. It expects that the targets will be provided, otherwise, it will just return the Predictors.


Estimating class prevalence in unlabelled data after predicting labels with a binary classifier I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.

These are the steps I'm hoping to perform:

Train a binary classifier on labelled training data.
Use a labelled test set to estimate the specificity and sensitivity of the classifier.
Use the classifier to predict the label for the unlabelled records in the dataset.
Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.

So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.

Thanks in advance!

There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.

Basically, the idea of AC is:

1.1) Learn a binary classifier from the train dataset

1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation

2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)

That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).

The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)


Is it possible to (de)activate a specific set of cells in jupyter? I have a jupyter notebook and I would like to perform several runs, the code I want to run depending on the results of the previous runs. I divided my notebook into several cells, and for each run I would like to select which cell should be executed and which one should not. Is there a functionality which looks like "run all cells except those I explicitly deactivate"?

Welcome to DataScience.SE!

This is not currently possible. You could change the cells to Raw.


Is datawarehouse considered as datalake in big data environnment? Suppose I have a datawarehouse (DWH) and now I would like to add many other bigdata sources of information most of them are not structured. I still keep the DWH with no architectural change. The only thing I do is to enrich the bigdata with the data resides in the DWH using connectivity. Can the DWH defined as datalake in this case?

Check out this article: http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html

Looks like the answer is yes, your DWH can be defined as datalake because you are adding unstructured data.  Here is an excerpt:

  “A data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed.”


Can I use more features for my training data than my test data will supply? I am pretty new to the data science game so pardon me, if the answer to my question should be a no-brainer.

We are looking at manufacturing / quality data where products are labeled 'okay' or 'not okay' based on different measurements. In our data set, we also have information on the product type, which machine was used to build it, at what time it was build, temperature, etc.

We are trying to train a classification model that will correctly classify 'okay' or 'not okay' products. At the point where the classification should be done, we do not have the measurements of the product, but all the other data I mentioned. Someone suggested using all data to train the model incl. the measurements, even though the test data will not have this information.

So the question is: is it feasible and does it make sense to train a model with more features than it can test on the test data? What ever the answer - please explain because I just can't seem to wrap my head around it.

Kind regards

Julia

The final purpose of a model is not the training but the actual application of it. If some features are not available during the final application and you cannot derive those features from other features, it is counter-productive to include these features during training.

Many classification methods will not even work if you don't give them the same set of features as they were trained on.


What models are used to get the Next Best Action to convert a physician from a non writer to a write? Assume that a sales rep has actions A, B, C, D.... X that he can perform to influence a physician to write his drug

I want to predict what the next best action a sales rep should take to have the highest impact on a physician

For example, if a sales rep has already performed A, D, F then i want to predict the next best action for the sales rep to perform to convert the physician to a writer

What models/algorithms are suitable for this type of a problem?

Expanding on my comment, the simplest predictive model would be to just compute the conversion rates Pr(conversion | action) for each action {A,B,C,D, ... }.

A solution in R:

library(dplyr)
library(tidyr)

set.seed(100) # for reproducibility
action <- sample(letters[1:5], 100, replace = TRUE) # dummy up sales actions (independent variable)
converted <- as.logical(rbinom(100, 1, prob = .5))  # dummy up physician conversion (dependent variable)

(df <- data.frame(action = action, converted = converted))

# Compute conversion rates for each action
conv_df <- df %>%
  group_by(action, converted) %>% # unique combinations of action by converted
  summarize(freq = n()) %>% # compute freq of unique combinations
  spread(converted, freq) %>% # spread rows to cols
  mutate(rate = `TRUE`/sum(`TRUE`,`FALSE`)) # compute conversion rate

conv_df[order(-conv_df$rate),] #reverse sort

##Source: local data frame [5 x 4]
##Groups: action [5]
##
##  action FALSE  TRUE      rate
##  (fctr) (int) (int)     (dbl)
##1      d     8    13 0.6190476
##2      e     7    11 0.6111111
##3      a     5     6 0.5454545
##4      c    13    10 0.4347826
##5      b    17    10 0.3703704

The solution above assumes the application is to be some sort of guide for sales reps... if you are wanting to model the distribution of actions in sales contexts, you would need to compute the prevalence of each action as well.


Perform PCA on columns of different length I have about 20-30 columns, all with different lengths. The first column has 25000 rows, the second column has 19000 rows, and it's different for all the columns.
All are the survey data with 0 (No),1(Yes), and 666 (for missing data) values. I want to perform PCA on these data. As each column is of unequal length, I am having a hard time doing PCA due to 'NaN' values for most of the columns that are shorter in length.
I don't want most of the information to be lost, so I am not willing to delete the rows of longer length columns and make every column equal in length.
Please advise me the possible solutions.

According to my understanding PCA requires that you have the column of equal length, so you either need to

shorten the longer columns (basically just skip the incomplete observations) or

fill in the gaps in the shorter columns.

If you choose the second option, you'll need to learn about the concept of imputation (see the following link for reference Missing data imputation).
Please note, that you need to get rid of the dummy values representing missing response (666s) in your data before performig PCA, because these values are arbitrary (they also could be 9999 or -1) and would have significant impact on the results.
Furthermore, it's arguable whether PCA can be applied to binary data, see e.g. discussion under related SE question.


How to estimate the accuracy on a large dataset? Given that I have a deep learning model(handover from former colleague).
For some reason, the train/dev set was missing.

In my situation, I want to classify my dataset into 100 categories.
The dataset is extremely imbalanced.
The dataset size is about tens of millions

First of all, I run the model and got the prediction on the whole dataset.

Then, I sample 100 records per category(according to the prediction) and got a 10,000 test set.

Next, I labeled the ground truth of each record for the test set and calculate the precision, recall, f1 for each category and got F1-micro and F1-macro.

How to estimate the accuracy or other metrics on the whole dataset? Is it correct that I use the weighted sum of each category's precision(the weight is the proportion of prediction on the whole) to estimate?

Accuracy has a specific meaning classification -  the data points with predicted labels must exactly match actual labels over the total number of data points.

In order to calculate accuracy, you need the actual labels for each data point. If you do not have actual labels for a data point, those data points can not be used in the analysis.


Place a marker on a plot I want to place a Call Marker on a plot. Call should be "buy" whenever the smaller moving average (21) crosses over longer moving average (34) AND the Call should be "sell" whenever smaller moving average crosses under longer moving average.

I have a column average price. I have calculated the rolling mean for 21 and 34 days using the rolling() function and plotted the line plot of all three columns: average price, sma_21 and sma_34 using matplotlib. I want to place a marker on the plot.

If the smaller moving average (21) crosses over longer moving average (34) I need to place this maker "^" AND if smaller moving average crosses under longer moving average I need to place this maker "v".

averageprice = [
    2352.6, 2410.26, 2443.31, 2525.78, 2506.58, 2530.69, 2530.49, 2545.01,
    2605.4, 2593, 2577.65, 2554.74, 2549.69, 2552.85, 2568.84, 2577.2,
    2693.18, 2624.95, 2543.44, 2513.28, 2487.48, 2464.89, 2469.41, 2427.94,
    2402.96, 2430.5, 2427.14, 2412.24, 2403.02, 2388.78, 2357.33, 2345.89,
    2342.52, 2361.01, 2368.46, 2366.9, 2354.42, 2348.75, 2343.49, 2426.54,
    2478.13, 2453.34, 2449.5, 2396.18, 2402.63
]
avg_p = df['Average Price']
sma21 = avg_p.rolling(window = 21).mean()
sma34 = avg_p.rolling(window = 34).mean()
avg_p.plot()
x = df.index
f = sma21
g = sma34
plt.plot(x, f)
plt.plot(x, g)
idx = np.argwhere(np.diff(np.sign(f - g))).flatten()
plt.plot(x[idx], f[idx], '^')
plt.show()

I used this code to get the intersection point and place the marker

idx = np.argwhere(np.diff(np.sign(f - g))).flatten()

and got a plot like this:

I need to get markers that look something like this:

I was able to produce the following with a marker for each point, based on some criteria:

I made a dummy Pandas Dataframe, using the sine and cosine to get lines that cross each other nicely; also making the index itself a column (with reset_index) just to make plotting easier later on.

df = pd.DataFrame.from_dict(dict(sin=np.sin(np.linspace(0, 2*np.pi, 50)),
                             cos=np.cos(np.linspace(0, 2*np.pi, 50))))
df.index.name = "index"
df.reset_index(inplace=True)
df.head(5)

  index sin        cos
0   0   0.000000    1.000000
1   1   0.063424    0.997987
2   2   0.126592    0.991955
3   3   0.189251    0.981929
4   4   0.251148    0.967949

I make a new column which contains the criteria you want, so if one line is above or below another line:

df["criteria"] = df.sin > df.cos    # where sin values are higher than cos values

This will allow us to select which points to plot with which style (e.g. which marker).

The final plot itself is made up of two parts (note that we re-use the same ax object, so all plots appear on the same graph)

The underlying data itself can be added to create the simple lines (these are your moving average columns).

ax = df.sin.plot(c="gray", figsize=(10, 6))    # plots the sine curve in gray
df.cos.plot(c="cyan", ax=ax)                   # re-uses the ax object - important!

The markers, based on our criteria column. We filter the points that satisfy our criteria and plot them with a certain marker, then filter the points that do not have the criteria (using the negation selector: ~). Whereas we used the basic line plot above, we now need to use the scatter kind of plot:

df[df.criteria].plot(kind="scatter", x="index", y="sin",
                     c="green", marker="^", ax=ax, label="higher")

df[~df.criteria].plot(kind="scatter", x="index", y="sin",
                     c="red", marker="v", ax=ax, label="lower");

Note that we continue to use the same ax object.

We can also add the legend, which uses either the Pandas column names by default or the label that was passed in each plot method:

plt.legend()="lower");    # adds legend for all plots

You can see in the matplotlib documentation that there is a marker argument, but it seems you cannot simply specify a column of a Pandas dataframe to give the marker for each individual point.


How Dummy Variables Should Be Modeled In A Linear Regression Model? I've a cross sectional model where I want predict number of users that take specific service, to make it I've many variables but have specifically two nominal: isWorkday(0 or 1) and weeday(1,2,3,...,7). When I make the model, taking into account the two variables, generates high multicollinearity. So I've delete one of them, so what's better have many dummies (weeday) or less dummies (isWorkday).

Since your task is to predict something, the better variable is the one that gives you a higher prediction accuracy. So you can simply test both and choose the one with which your model performs better.

However, I would suggest considering to engineer your own feature that incorporates information of both variables. For example, you could create three dummy variables: workday, weekend and holiday and include two of them into your model (to prevent falling into the dummy variable trap). Another option would be to only include the interaction terms between isWorkday and weekday.


How does L1 regularization make low-value features more zero than L2? Below formulas, L1 and L2 regularization

Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero value. Could you please explain the reason why L1 has more tendency to make zero value? (It's gonna be great if you let me know this reason by using formula(like above equation)!

The penalty on coefficients of L1 is more aggressive on values close to zero than it is for L2.

With L1, when a weight value comes closer to zero, it tends to get even closer, because of the $\epsilon\lambda$ penalty which stays constant.

With L2, the $\epsilon\lambda w^{(t)}$ term gets smaller, so the regularization gets smaller as a weight comes closer to zero, with the update depending only on $\epsilon\Delta E(w)$.

So the main argument for this to my knowledge is that
$$
\lim_{w\rightarrow0}\lambda = \lambda\\
\lim_{w\rightarrow0}\lambda w = 0
$$