Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater learning rates for the features that are less common (or more sparse) when using Adagrad.
Now, why do we even want to have lower learning rates for the features that are most common?
Why do we even care if one feature is more sparse than any other feature?

Intuition behind learning rates
With Adagrad, parameters that model the influence of the features in our problem tend to be updated at the same rhythm, something that is accomplished by what you explained in your question $\rightarrow$ the parameters are updated with different learning rates/ steps which may depend on how frequent (or sparse) the features are.
But why the parameters that we are learning should be updated with different learning rates? $\rightarrow$ For example, if our cost function was $f(x,y,z) = 20x^2+y^2+z^2 \rightarrow$ and $x,y,z$ were our parameters, it is clear that having a point far away from the global minimum, the function will be more sensitive to changes on $x$ than on $y$ or $z$. But the same parameter step applied to $y$ and $z$ will lead to the same change on our cost function $f(x,y,z)$.
In the general case, when we don't know exactly the terms of our cost function, one conservative rule for updating would be to use smaller learning rates on the directions with "big" gradients. This would prevent us from overshooting on these directions if the decision of moving along them with a big step was wrong.
To visualize the previous reasoning we can plot how would be the paths followed by our parameters applying different learning rates to them. In this case, the hypotetical cost function that I have used is $f(x,y)=20x^2+y^2$ and the number of iterations made are the same in all of them ($20$). With this plots we can see that:

Left plot $\rightarrow$ Applying a "big" learning rate to both $x$ and $y$ directions lead to overshooting on the $x$ direction, which causes that the path doesn't reach the minimum.
Center plot $\rightarrow$ Applying a "small" learning rate to both $x$ and $y$, solves the issue of overshooting, but causes the learning to be very slow.
Right plot $\rightarrow$ Applying a "small" learning rate to $x$ than to $y$ not only solves the issue of overshooting on the $x$ direction, but also makes faster the optimization.

Note that the optimization used here is not exactly like Adagrad optimization, but with it I pretend to show the effects of different learning rates.

Adagrad
With Adagrad we are able to take this previous intuition of why the learning rates should be different for each parameter but with another point of view.
For understanding why Adagrad behaves in the way described by the question, we should take a look at the practical formula that is commonly used for applying it:
$$ \mathbf{w}_{t+1}= \mathbf{w}_t - \frac{\alpha}{\sqrt{\epsilon I + \text{diag}(G_{t+1})}}\,\,g_{t+1} $$
Where $\mathbf{w}$ is the vector of parameters that we want to update, $g$ is the gradient of the cost function w.r.t. these parameters $\mathbf{w}$, $t$ is the number of the iteration taking place and $G_t$ is given by:
$$G_{t+1}=\sum_{\tau=1}^{t+1} g_\tau\,g_\tau^T$$
So, if we have a look at one parameter $w^j$ from the vector $\mathbf{w}= [w^1,w^2,...,w^n]^T$, we have:
$$ w_{t+1}^j = w_t^j - \frac{\alpha}{\sqrt{\epsilon + G_{t+1}^{jj}}}\,\,g_{t+1}^j \,\,\,\,\,\, \leftrightarrow \,\,\,\,\,\, G_{t+1}^{jj}=\sum_{\tau=1}^{t+1}(g_\tau^j)^2$$
What does this mean? $\rightarrow$ As $t\uparrow\uparrow$ the learning rate related to the parameter $w^j$ will tend to decrease. This is because in the denominator we have the cummulative sum of the absolute value of its previous gradients.
Thereby, using Adagrad we could also solve the problem presented in the previous plots. Due to the fact that the plotted function was more sensitive to changes on the $x$ direction (hence the x-component of the gradient was bigger), Adagrad will automatically give a lower learning rate to the updates in $x$ than on $y$.
But what is the relation of the above with the more/less common features?
Common features here take the meaning of features that during a significant number of updates their influence on the problem has been poorly analysed. This can happen because during these iterations the components of the gradient vector related to these features have had significant lower magnitudes.
Let's look at an example using Stochastic Gradient Descent $\rightarrow$ we will be updating the parameters for every single sample that we have.
Let's consider that our model of parameters is linear, and have two parameters $(w^{(1)},w^{(2)})$ that have to be trained $\rightarrow$ so we can have a predictive function like this:
$$\hat{y} = w^{(1)} x^{(1)} + w^{(2)} x^{(2)}$$
where $x^{(1)}$ and $x^{(2)}$ are the features of our problem.
So if we make use of a cost function ($J$) that represents the mean squared error of the prediction ($\hat{y}$) given by our parameters, the gradient at iteration $t$ will be given by a quantity proportional to the features:
$$ g_t = \left(\frac{\partial J}{\partial w^{(1)}_t}, \frac{\partial J}{\partial w^{(2)}_t}\right)^T \propto (x^{(1)}_t, x^{(2)}_t)^T $$
Thereby, using SGD the quantity that we use to update each of our parameters $\delta w_{t+1}$ is also proportional to the associated feature:
$$ \delta w_{t}^{(j)} \propto x^{(j)}_t$$
With this already set up, lets imaging that our training is done in three samples (just for visualization purposes). During these samples the $x^{(1)}$ feature is more frequent (or less sparse) than the other feature $x^{(2)}$, so we have a dataset, $\mathbb{X}$, similar to the next:
$$ \mathbb{X} = \begin{pmatrix}
1 & 0\\
5 & 0\\
3 & 3\\
\end{pmatrix}$$
Where each row represents a sample of $x^{(1)}_t$ and $x^{(2)}_t$, with $t\in\{1,2,3\}$.
Now it is clear that the parameter $w^{(2)}$ won't be updated until the last sample ($t=3$), because during the first two samples ($t=1, 2$) its gradient component has been zero (becase its proportional to $x^{(2)}_t$).
But what would have happened at this point to $w^{(1)}$? $\rightarrow$ it would have been updated during the first two samples (non-zero gradient component), and because of it, its associated learning rate will be smaller than the one of $w^{(2)}$ in spite of having the same $x^{(1)}_3 = x^{(2)}_3$!
So now we understand why the parameters associated with sparse features can be updated with greater learning rates using Adagrad.
Conclusion
Adagrad allows us to give more importance to updates in parameters that have associated features which are sparse, or more generally, to give more importance to parameter updates that have experimented a record of relatively lower gradients (in magnitude).
Why is this useful? To answer this, we can cite the original authors (see paper: "Adaptive Subgradient Methods for
Online Learning and Stochastic Optimization "):
> "Our procedures give frequently occurring features
very low learning rates and infrequent features high learning rates, where the intuition is that each
time an infrequent feature is seen, the learner should “take notice.” Thus, the adaptation facilitates
finding and identifying very predictive but comparatively rare features"


What are the best ways to use a time series data for binary classification I have large number of csv files and each of them are timeseries based csv files sampled at Avery 5 seconds for 2-3 mins. I have 20k such files with 200-300 variables in each file.
I am aggregating the data by mean over the entire 2-3 mins window are using it for binary classification.

Currently I am using mean of each column in the .CSV file to represent that file, so basically I am summarizing the csv's using one scalar value per column.
 so each file is one sample represented by its respective mean value.
Could anyone suggest me some better ways to summarize the timeseries data.

Thanks for your time.

From your comment, I understand that you are trying to solve the binary classification problem using your aggregated data and you are getting very poor results when you simply use the mean.

Depending on specifics of your data and the shape of your time series, there are several alternatives that you could try. Note, that you might need (significantly) more than just a single number per time series to solve your problem.

In addition to the mean, you could use the quantiles or some other summary statistic, like standard deviation, min, or max.
You could try to sample the data, i.e. instead of taking the entire time series, pick only the values that are minutes, hours or days a part. Or pick only mid-day values. The frequency of the sampling depends on your data.
Or just pre-aggregate by calculating averages for every hour, day, month, etc.
Additionally, you could calculate the periodicity of your time series and use it as a new feature.
Or calculate some trends.
Try to fit some standard time series models to your data, e.g. ARIMA and use the coefficients as informative features.
Last but not least, use the domain knowledge re what could be relevant feature for your classification problem: the biggest jump (max first order difference), change of regime, etc.

Edit
I’d pick at least 10-20 features per time series generated as described above and apply logistic regression with LASSO or even xgboost.

After selecting 10-20 features per time series you also could try PCA to reduce the dimension.


Read csv file and save images from the output I have created some code that reads my CSV file and converts the dataset to a grayscale image. I want to know if there is any possible way to read through each row in the dataset and save each of the images created from the rows?
So far, I have got this code that reads the CSV files and creates an image using .imshow
import pandas as pd
import numpy as np
from sklearn.datasets import load_digits
from keras.preprocessing.image import array_to_img
import matplotlib.pyplot as plt

data_path = "dataset_malwares.csv"

data = pd.read_csv(data_path);

label = data.Malware.values

data = data.drop("Malware", axis=1)
data = data.drop("Name", axis=1)
data = data.values
data = data.reshape(data.shape[0], data.shape[1], 1)
data = np.tile(data, (1, data.shape[1]))

plt.imshow(data[1], cmap="gray")

plt.title("label: {0:}".format(label[1]))
plt.show()

print(data[0].shape)

I want to go through the dataset and save each image but not too sure where to start. Any suggestions would be great. Thanks :)
Rows/format of the data - I've provided a shared version of the dataset:
https://1drv.ms/x/s!AqFNg8FC48SSgtZSObDmmGHs3utWog

Are you looking to simply save the file produced by plt.imshow? If yes, then you should be able to use plt.savefig as follows:
plt.imshow(data[1], cmap="gray")
plt.title("label: {0:}".format(label[1]))
plt.savefig("output_image.png")
plt.show()

You can either remove or keep the plt.show() call depending on whether you want the image to still be shown or just save the image.
EDIT: If you want to do this for all rows you can just loop through the numpy array as follows:
for i in range(data.shape[0])):
    plt.imshow(data[i], cmap="gray")
    plt.title("label: {0:}".format(label[1]))
    plt.savefig(f"output_image_{i}.png")
    plt.close()


extracting information in presence of noise Assume I have a vector of displacements, when I calculate the derivative I obtain velocity. The problem is, there are so many noisy points and I am looking for one number as a velocity. What do you think is the best technique to obtain the velocity ? I am taking derivative and then take the mean, this gives me good result when there is not much noise however this technique is not really robust to noise also taking the median is not giving good result. Can anyone suggest a better technique to handle this problem?

Usually removing noise is a process depends on the type of noise.
Noises can be high or low frequency , you are using low frequency filters ( average and median) .
1. You did not describe the source of noise, we can not tell if it is a high frequency or low frequency .
2. Try to use a higher order low pass filter


Deep learning: Training in batches How does training in batches help in obtaining a better deep learning model? What should one keep in mind while splitting data into batches?

The point in using batch training is that you can't take a step using all data due to the size of the training data which is really big. Consequently, you may want to use batch optimisation techniques which take steps which are near to the best step. The reason is that in each batch, the distribution of data points is similar to the whole training data or at least it is not very different. If you have a small size for your batch, your steps may oscillate and may not be very perfect but for bigger batches they are better.

Batch techniques also facilitate the number of steps that you can take. They also can be helpful for not being stuck in local minimum points.


Why do we need cross validation set? I know we need to test our model on onseen data, but isn't that test set are for? Also what will happen if we increase K value in kfold?

For simplicity, I will talk about the classification problem, but regression follows the same rules.
You are given some data: part of it with known labels and others with unknown labels. The "classical" goal is to determine the missing labels based on known labels. The data is naturally split into two chunks:
(a) data without labels -- we call it a test set.
(b) data with labels -- in some textbooks, it is called the train set, but the train set is used for another thing as well (see below), so I would call it labeled data to avoid confusion.
You want to train a selected model on some data and apply the model to the test set. The data which is used for training is called the training set. So, a naive scheme is

Train the model on the train set
Predict the labels on the test set.

However, before making predictions on the test set, you need to measure how good the model is. We modify our approach to

Train the model on the train set
Verify that the model performs well
Predict the labels on the test set

To verify the model performance, we do the following: given a set called a validation set, we predict the labels on the validation set, and then check the accuracy. Note that the validation set must contain labeled data only. Why not to use the train set for validation? This approach is prone to overfitting:

deep enough random forest will predict each entry in the train set accurately
polynomial with a high enough degree will approximate all the points with zero error
KNN with k=1 will have each node guessed accurately on the train set because the closest node is itself.
etc.

To fix this issue, we make a train set and a validation set disjoint. Usually, we randomly split the labeled data into two chunks. Note that the test set contains another data.
In your question, you refer to the validation set as a test set. Some authors do this, but it is very misleading.
For increasing K-value in K-fold, we manipulate the bias-variance tradeoff. It is another concept, and you can read about it, for example, here: https://medium.com/@karenovna.ak/part-ii-evaluating-a-predictive-model-cross-validation-and-bias-and-variance-tradeoff-9874b836cd2e


Which of the 180 algorithms in R's caret package are feasible? R's caret package works with 180 models. The author warns that some of the package may be intractably slow or less accurate than top-choice models.

The author is not wrong about this. I've tried to train Boruta and evtree models and had to give up after they ran > 5 hours on a cluster.

The author links to a set of machine learning benchmarks, but those only cover performance of a small number of algorithms, comparing different implementations.

Is there some other resource I can turn to, for guidance on which of the 180 models are worth trying, and which will be very inaccurate or unreasonably slow?

Benchmarking mlr (default) learners on OpenML

The entire openml database of ML results.

Test from RStudio suggests SVM.

Mlmastery suggests LDA and Trial and Error.

Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fern ́andez-Delgado et al.

Paper concludes  parallel random forest (parRF_t) is best followed by  random forest, LibSVM with Gaussian kernel (svm), extreme learning machine with Gaussian kernel,    C5.0  decision  tree  and  multi-layer  perceptron  (avNNet).

  The best boosting and bagging ensembles use LibSVM as base classifiers
  (in Weka),  being slightly better than the single LibSVM classifier,
  and adaboost R (ensemble of decision trees trained using Adaboost.M1).
  The probabilistic neural network in Matlab, tuning the Gaussian kernel
  spread (pnn m), and the direct kernel perceptron in C (dkp C), a very
  simple and fast neural network proposed by us (Fern ́andez-Delgado et
  al.,2014),  are  also  very  near  to  the  top-20.

Wainer, Jacques (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets  Based on Fernandez-Delgado et al. (2014). "We have shown that random forests, RBF SVM, and gradient boosting machines are classification algorithm that most likely will result in the highest accuracy"

Rich Caruana & Alexandru Niculescu-Mizil () An Empirical Comparison of Supervised Learning Algorithms (classification) concludes with Platt-Calibrated Boosted Trees as best followed RF BagT Cal.SVM NN.

Many other studies include comparisons of models used. Some papers prefer SVM  others SVM with radial-basis or polynomial kernel for classification. (maybe same thing)

From my own regressions on generated data I recommend earth(MARS) Cubist SVMlinear.

Manisha Thesis first runs tests on UCI Machine Learning Repository then soil fertility which is the focus of the thesis. Best models on UCI were :"elm-kernel is the ELM neural network but with Gaussian kernel", "svr is the support vector machine for regression, with Gaussian kernel using the Lib-SVM library with the C++ interface", extraTrees and cubist. The thesis includes great descriptions of each model and links to more papers."extraTrees achieved
the best RMSE for 7 of 10 soil problems". Paper is definitely worth a read.


Why can't I use data augmentation with a pretrained convnet? Reading Deep Learning with Python by François Chollet. In section 5.3.1, we've instantiated a pretrained convnet, VGG16, and are given two options to proceed:

  A) Running the convolutional base over your dataset, recording its
  output to a Numpy array on disk, and then using this data as input to
  a standalone, densely connected classifier similar to those you saw in
  part 1 of this book. This solution is fast and cheap to run, because
  it only requires running the convolutional base once for every input
  image, and the convolutional base is by far the most expensive part of
  the pipeline. But for the same reason, this technique won’t allow you
  to use data augmentation.

  B) Extending the model you have (conv_base) by adding Dense layers on
  top, and running the whole thing end to end on the input data. This
  will allow you to use data augmentation, because every input image
  goes through the convolutional base every time it’s seen by the model.
  But for the same reason, this technique is far more expensive than the
  first.

Why can't I use data augmentation to generate more training data from existing training samples then go with option A? Seems like I can run the VGG16 base over my augmented dataset and use the output as the input to a standalone densely connected classifier.

Found the answer in stats.stackexchange.com. Hopefully this helps anyone else with the same question.

feature extraction: freezing convolutional base vs. training on extracted features


Is there an R package which uses neural networks to explicitly model count data? Ripley's nnet package, for example, allows you to model count data using a multi nomial setting but is there a package which preserves the complete information relating to a count?
For example, whereas an ordinal multinomial model preserves the ordering of the integers that make up the count, a fully developed model of count data as a GLM such as Poisson or Negative Binomial Regression includes how large the integer counts are in relation to each other.

Another phrasing might be, 'What kind of models come closest to combining the advantages of neural networks, in terms of, as an example, easily modelling non-linearity in the predictors, and count data GLMs, which are good at taking into account that the data is in fact a count?'

I skimmed over a paper recently that aims to use Neural Networks as Poisson regression. The method they propose is basically a standard Multi-Layer Perceptron where they use a different loss function, namely:

$$E = -\sum_{n=1}^N[-t_n + y_nlog(t_n)]$$

This is a version without regularization to prevent the overfitting, they use regular weight decay.

They mention that they wrote it in R and Matlab but I don't have a clue if it's available online somewhere, but any neural network package where you can pass your own loss function should suffice.

http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&%20Applic.pdf


Meaning of equation for CNN probabilities So the first equation above refers to a CNN (rather a committee of CNNs) for image classification. I am unable to understand exactly what the author is trying to do in the first equation.

So far, I think they're calculating the index of max likehlihood probabilities for all committees, then adding up the probabilities for all committees for those indices, and finally taking the maximum index.

But this seems overly convoluted and I'm not really sure. Could someone clarify this?

I agree the equation might not be clear, but you can decompose it into something like the following:

First, the term  $\operatorname{argmax}_k p^i (y=k|\mathbf{x})$ tells you which label has the higher probability from model $i$ given the input object $\mathbf{x}$.
Then, this "iterates" over all models in the Committee, computing for each the label that is most likely.
Finally finding which label is the most common one (that $\operatorname{argmax}_j$) at the end.

Also, it helps to think about it in pseudo-code

def get_label(CNNs, x):
    labels = [0, 0, 0, 0, 0]  # each position refers to that last $j$
    for pCNNi in CNNs:
        predictions = pCNNi(x)
        label_i = predictions.index(max(predictions))  # this is the argmax_k
        labels[label_i] += 1
    return labels.index(max(labels))  # this is the argmax_j


reliability of human-level evaluation of the interpretability quality of a model Christoph Molnar, in his book Interpretable Machine Learning, writes that

Human level evaluation (simple task) is a simplified application level
evaluation. The difference is that these experiments are not carried
out with the domain experts, but with laypersons. This makes
experiments cheaper (especially if the domain experts are
radiologists) and it is easier to find more testers. An example would
be to show a user different explanations and the user would choose the
best one.

(Chapter = Interpretability , section = Approaches for Evaluating the Interpretability Quality).
Why would anyone pick/trust a human-backed(not an expert) model over, say a domain-expert backed model or even a functionally evaluated model(i.e. accuracy/precision/recall/f1-score etc. are considerably good)?

This is specifically for interpretability of outcomes, i.e. a task where non-expert humans outperform machines.
There is a problem in collecting labels in machine learning, whereby labelling datasets is very expensive and time consuming (due to size of datasets & cost of experts' time).
So it's less about trust, its more about practicality. Consider hiring a data scientist to develop an algorithm to automatically label a dataset based of expert heuristics (e.g. "label the data as cancerous if it looks red"), it might take 6 months to collect data, plan, develop & test - therefore for certain use-cases hiring 10 non-experts and telling them the heuristic might be cheaper and faster.
The book uses an example "show a user different explanations and the human would choose the best." in the context of radiology, it could be something like:
"Look at the images of the patient, and compare it to this dictionary of images and diagnoses, combine multiple sources and then report what the diagnosis is"
Of course if you have an algorithm which outperforms non-experts, you might just want some expert labels to validate your algorithm, and forget the non-experts.


Interpret clustering results after variable transformation since some time I have a question to which I have not found the proper answer yet.

My doubt concerns the interpretation of the results of a clustering algorithm which was run on features to which a log-transformation was applied.

Specifically, let's assume we want to run a k-means algorithm on 3 interval variables. Unfortunately, these three interval variables are extremely bad distributed and the k-means gives the worst result we have ever seen.
However, let's imagine that by applying a log transformation to each variable, we obtain three incredibly perfect normal distribution.
Then, we run again the k-means and we obtain perfect clusters.

Now, my doubt concerns the interpretation of this cluster obtained by running a k-means on three log-transformed variables: it is not clear whether our interpretation of the clusters obtained should be made on the original variables or it should be made on the log-transformed variables?

Clearly, my example is related to log-transformation but we can talk about z-score or min-max normalization or any other kind of transformation that we apply in order to improve the quality distribution before running the clustering algorithm.

To clarify, what I mean by interpretation is the profiling of the cluster, which means try to describe which are the characteristics common to the individuals belonging to that cluster.

Very interesting!
What you did to your data is simply a feature mapping/transformation. So how this affects the clustering results?

Clustering is not a clearly defined problem but at least we know something about it: It's about internal similarities (patterns) so these similarities should be maintained through the feature transformation. In your example if you found clusters in transformed space, it shows that you have had clusters in original space as well. You just couldn't see them according to the algorithm you used in that space!

For instance if you use Kernelized versions of algorithms you easily find that what they do is nothing but what you did as transformation. They first use a kernel to map the data into new space and then use the algorithm in that space (of course with a bit of theoretical differences/constraints).

To summarize, no transformation produces a fake pattern in the data. In worst case it vanishes the original pattern and in the best case it reveals the pattern which was not visible originally (which is your case).

I mentioned Fake Pattern above so let me say a bit more on it. I think there is a fundamental issue concerning your question:

You assume that there is a Right clustering that you got after transformation. Actually there is no right clustering!

We do not have fake pattern! If there is a pattern in a feature space, then that is true! i.e. you found an interesting representation of your data. If it does not match with the labels then either the data is very noisy or wrong features have been chosen to represent classes (maybe more reasons. just these two came to my mind now). If there is no label (your case) be sure there is a correlation between features of those cluster members.


How to combine rows after Pandas Groupby function May I know how to combine several rows into one single row after I used Pandas groupby function?

In below example, I would like to to group the data by Employee ID, Customer Last Name and Customer First Name. Then I want all his dependents' data listed in the same row.

Thanks a lot!

You should use the function drop_duplicates :

Define the columns you take into account:

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")

But it seems to me that the column Employee Id is enough, as you have only one person per Employee Id so you can define :

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")

Then you can write:

import pandas as pd

unique_columns = Seq("Employee Id", "Customer Last name", "Customer First Name")
df = df.drop_duplicates(subset=unique_columns)


Which book is a standard for introduction to genetic algorithms? I have heard of genetic algorithms, but I have never seen practical examples and I've never got a systematic introduction to them.

I am now looking for a textbook which introduces genetic algorithms in detail and gives practical examples how they are used, what their strengths are compared to other solution methods and what their weaknesses are.

Is there any standard textbook for this?

Most of the "standard textbooks" (e.g., Goldberg, Mitchell, etc.) are pretty dated now. If you just want to have some confidence that you understand how the basic algorithms work, they're fine, but they tend to emphasize material that's doesn't necessarily match the more modern way of understanding and talking about things like theoretical issues.

I've used the Eiben and Smith (https://www.amazon.com/Introduction-Evolutionary-Computing-Natural/dp/3642072852) book to teach evolutionary algorithms to my students over the past few years. It's a bit more concise than some of the others, but if you're OK with that, I think it does a better job of covering the stuff that's relevant about evolutionary computation today.


Instead of one-hot encoding, can I store the same information in one column using a single value? Rather than creating 15 additional columns full of sparse binary data, could I:

1) use the first 15 prime numbers as indexes for the 15 categories

2) store data by multiplying the prime numbers of the categories that otherwise would have a value of 1 in one-hot encoding

3) retrieve data by factorizing the value generated by multiplying unique prime numbers

Ex: 1914 would yield the list [2, 3, 11, 29] which would let you know that the user with the 1914 value has property 2, 3, 11, and 29 but nothing else.

I understand this is limited because BIGINTs can only hold the product of the first 15 prime numbers, but would it not still be useful in some situations and save time when searching the database? The entire table would be 14 columns smaller. I guess this is less about machine learning algorithms and more about storing and retrieving data.

I suppose you could do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the prime factors of the stored value, why don't you just store the flags as a bit string? Your example of 15 different possible values could be stored in a single SMALLINT (2-byte) SQL column. After retrieving the value, you would just need to extract the bits of interest for your record with some basic bitwise arithmetic.


Perceptron Primitive Boolean Functions Thanks for reading.

I'm currently reading Tom Mitchell's Machine Learning (I'm a beginner into ML), and I'm on chapter 4 about perceptrons. I'm really confused about this paragraph:

I understand the AND but I'm still not so sure how OR can be represented.

For example, if the inputs are:

$X_1 = +1$

$X_2 = -1$

In that case, using his logic for OR we would end up with the perceptron actually outputting -1, because:

$1*0.5  + -1*0.5 - .3 = -.3$

The function would give us a negative number. Is he wrong about the OR function? How would a single perceptron act as an OR?

Thank you!

I interpret it as the input takes binary value $0$ (false) and $1$ (true) while the output take $-1$ (false) and $1$ (true).

The map is $$sign\left(\sum_{i=1}^nw_ix_i+w_0\right).$$

If we let $w_i=0.5, i \ge 1$, then we have the mapping rule to be

$$sign\left(\frac12\sum_{i=1}^nx_i+w_0\right).$$

For the AND operator, if we set $w_0=-0.8$, for it to take positive value both $x_i$ has to be $1$.

For the OR operator, if we set $w_0=-0.3$, we just need one of them to take positive value for it to be true.


"Invalid value" in RMSprop implementation from scratch in Python Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * (np.square(gradients[f"dW{l}"] + reg_term))

Edit 1: I see that S[f"dW{l}"] contains some negative values. How is this possible when np.square(gradients[f"dW{l}"] always contains positive values?
I have implemented a neural network from scratch which uses mini-batch gradient descent. The network works well. Unfortunately, I can't get my RMSprop implementation to work. I have verified that the network works well with momentum.
I get a RuntimeWarning when training the network with RMSprop: "invalid value encountered in sqrt". This happens in the "RMSprop" update step.
My implementation of update parameters:
def update_parameters(parameters, gradients, V, S, batch_size, t, learning_rate, reg_param):
    L = len(parameters) // 2
    beta1 = 0.9
    beta2 = 0.999
    epsilon = 1e-8

    for l in range(1, L+1):
        reg_term = (reg_param / batch_size) * parameters[f"W{l}"]

        # RMSprop gradients
        S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * (np.square(gradients[f"dW{l}"]) + reg_term)
        S[f"db{l}"] = beta2 * S[f"db{l}"] + (1 - beta2) * np.square(gradients[f"db{l}"])

        # RMSprop update
        parameters[f"W{l}"] -= learning_rate * (gradients[f"dW{l}"] / (np.sqrt(S[f"dW{l}"])) + epsilon)
        parameters[f"b{l}"] -= learning_rate * (gradients[f"db{l}"] / (np.sqrt(S[f"db{l}"])) + epsilon)

This is how I initialize the parameters:
def init_params_V_and_S(activation_layers):
    params = {}
    V = {}
    S = {}
    L = len(activation_layers)

    for l in range(1, L):
        params[f"W{l}"] = np.random.randn(activation_layers[l], activation_layers[l-1]) * np.sqrt(2 / activation_layers[l-1])
        params[f"b{l}"] = np.zeros((activation_layers[l], 1))

        # RMSprop params
        S[f"dW{l}"] = np.zeros((activation_layers[l], activation_layers[l-1]))
        S[f"db{l}"] = np.zeros((activation_layers[l], 1))

    return params, V, S

Any ideas what's causing this?

The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * np.square(gradients[f"dW{l}"] + reg_term)


Abstraction in Neural Networks In my nonlinear dynamics class in college, we discussed a simple perceptron with two input neurons and one output neuron that is trained on the patterns

pattern | Input | Output
   1    |  1,0  |   2
   2    |  3,1  |   6
   3    |  0,0  |   0

Solving the system of equations to determine the weights

$$w_1(1) + w_2(0) = 2$$
$$w_1(3) + w_2(1) = 6$$
$$w_1(0) + w_2(0) = 0$$

gives $w_1 = 2$ and $w_2 = 0$, which means the perceptron has made an abstraction. In more advanced neural networks, (a) how do you test for abstractions in your network and (b) how do you interpret what that abstraction means?

Since the weight of each feature (column) determines how important that column is in determining the output value, the abstraction means the most efficient and correct weights. For instance, here the features for pattern 1 are 1 and 0, and each of them are given a weight multiplying with which the output is calculated:

feature1 * weight 1 + feature2 * weight2 = output

So based upon the current pattern, the best values for weights to give the already extant outputs, are the ones given.

The test for the abstraction can be done via the compile method. First initial wights are assigned to features, then a single record or a patch of data passes through network and a cost function is calculated, then the error is backpropagated and new weights are set for the features (based on how much responsible each of them is for the error) and this continues.


How CNN applies backpropagation to update its weights and biases? I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer.

However, I do not understand how CNN updates its weights and biases using backpropagation.

I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Any reference is appreciated.

the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer

Not necessarily in fact. Conv layers are the only layers that you absolutely need in order to implement a CNN. All the other are not strictly necessary. For example, I trained a CNN on Fashion MNIST dataset using ELU activation (you can check my Notebook here). Moreover, if the size of the input data is not too big and you have enough computing power, you can get away with Pooling layers and use only the Convolutional ones. (As computing power increasese, Pool layers will be progressively abandoned since they loose too much information).

Coming to your question:

  how CNN updates its weights and biases using backpropagation. I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Backpropagation works this way: it calculates how much each weight is contributing to the final Loss value (this computation is done finding the first partial derivative with respect to each weight). This trick works in the same way for "normal", Dense layers as for Convolutional layers. The only difference is that Conv layers are not fully connected, i.e. not all the nodes of one layer are connected with all the nodes of the following layer. The difference is that only the nodes that fall under each sliding filter are connected. That is a trick to process large amount of data (such as pixels).

Conv layers have a different architecture from Dense layers, but their training works exactly the same as for Dense layers.

Pool layers, on the other side, do not contain any learnable parameter. They are just mechanical operations that don't require any training.

EDIT: This is a great explanation of how backprop works.


Transition from clustering to classification? To date, I have done several ad-hoc text clustering projects which use combinations of topic modeling, k-means, and other algorithms. Basically, the point of these projects was to produce themes for different events based on associated text. The themes were named manually after the appropriate levels of clustering were determined, and are now stored in a csv in the following format:

event_id    majortheme    minortheme         majortheme_id    minortheme_id
12          Job Failure   TWS Issue          1                major1minor1
14          Job Failure   TWS Issue          1                major1minor1
15          Job Failure   Job Abend          1                major1minor2
16          Access Issue  Unable to Login    2                major2minor1
17          Access Issue  Unable to Connect  2                major2minor2

I want to transition from clustering to classification (from descriptive to prescriptive analytics), i.e. being able to take new events (with new event_ids) and classify them based on previous clusterings. This would be sort of an iterative training dataset, as new classifications would be added to previous clusterings, after verifying that the model hasn't gone completely awry. Using Python, what is the best approach to implement this sort of classification pipeline? Is it as simple as saving my initial clustering results and then just using that data as a training set going forward? Then saving the test predictions to the original training dataset, and so on?

In order to build a model to make predictions you need a labeled training set, that is, a training set in which each training example is assigned a class label. Training sets are usually labeled by human experts that use their domain knowledge to manually classify the examples in the training set. You have already done that, as described in your first paragraph.

However, sometimes this process is expensive. In order to decrease cost sometimes semi-supervised learning is applied. In semi-supervised learning, and under some assumptions, a small amount of labeled, together with a large amount of unlabeled data, are used to build a predictive model. The unlabeled data are assigned labels during the training process.

This seems to match your idea in your last paragraph, and it is already implemented in Python. I think that if you found clear clusters in your data it may work very well. Worth a try at least.


Which recommender system approach allows for inclusion of user profile? I wanted to enhance a recommendation engine with information relying not only on past purchases or ratings but also on behavioral and demographical variables like sex, age, location, service usage frequency or hours. This information may be sparse (e.g. the user may not have provided me with his age).
Can you suggest an approach that would allow for it?

My first approach:
After screening the general approaches to recommender systems, I think the one that is able to implement my idea is custom(?) item-based collaborative filtering.
To be more precise: I would include user profile information in the same way item ratings are introduced, probably with two alterations to raw data:

Normalize scale to be compliant with the rating scale.
Add a parameter in my algorithm that would put a weight (e.g. 20% or 50%) on user profile rows, as there may be 10 user-profile
variables and a million product items.

Try matrix/tensor factorization methods and data fusion.


Linear Regression Optimization I am learning linear regression right now.
In the most of the examples of implementation of this method, which I found, gradient descent is used.

Is there a better way to optimize linear regression than gradient descent?

Is there a better way to optimize linear regression than gradient descent?

If by better you mean finding the better separator, no you can't due to the fact that the cost function for linear regression is convex which means there is just one optimal point.

If you want to optimize using different algorithms, there are different kinds of solutions. Gradient-based algorithms like Adam and RMSProp are of those. You also can use normal equation.


Apache Spark Understanding I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/deep learning when should be the case that it be must to use Apache Spark?

If you are just starting out, it is pretty unlikely that you come across such big datasets that require massive parallelization either in computing power or data storage means.

When you start pushing the limits of what a single machine can do, that's when you deploy a big data solution like Spark and Apache HDFS storage and start writing parallelizable models that scale well with bigger data.

In a small scale it might be actually way slower to run some jobs at Spark, because it adds some overhead to the calculations. Spark is best used for large enough, well parallelizable and iterative problems.

It is also worth mentioning that it requires some patience to configure the whole cluster to work well.


Using Machine Learning to play the game SET I recently tried to make a CNN which could play the game Set: https://en.wikipedia.org/wiki/Set_(game)

However, I quickly learned after using my own convolutional neural network and trying to retrain Inception-V3 and AlexNet to classify pictures of three cards into "set" and "notSet", that a CNN is not useful for this task.

Which machine learning algorithm would be useful for playing this game? Would an SVM work?

Here are links to my last question (with some code) and the Github repo which includes sample images of the dataset.

Keras val_acc unchanging when training (same label assigned to all images)

https://github.com/Keyan-Rhm/MLWithSet

So you would like to play this game using images?

If that's the case, I suggest dividing your problem into 2 sub-problems: Image classification (supervised learning) and reinforcement learning.

The first sub-problem will detect the right card and the second sub-problem takes that as an input.

When it comes to game, it's definitely worth trying reinforcement learning. You will only need to create a reinforcement learning environment for this game and use some off-the-shelf algorithms (like OpenAI baselines).


Is there an NLP corpus that contains common medical terms? I am trying to use the NLTK library to extract keywords denoting medical symptoms from medical reports of patients. For example, I have a medical report as follows:

s:a 33 year old female crystallographer presents with mild spells of
vertigo, mild headaches particularly at the back of the head and in
the morning x 2 weeks. pt also reports chronic mild occasional
lightheadedness. o:Height 160 cm, Weight 53.8 kg, Temperature 37.3 C,
Pulse 76, SystolicBP 146, DiastolicBP 93, Respiration 15, Heart = 2/6
systolic murmur at base of heart, Chest = clear to auscultation B/L,
no rales or wheezing, Extremities = no edema or clubbing, Heart =
normal S1, S2, RRR a:Hypertension p:performed E/M Level 2 (established
patient) - Completed, and prescribed Hydrochlorothiazide - 50 mg po
qd, and ordered Cholesterol.

Here, I would like to find all the keywords or bigrams that represent medical symptoms. In the text above, these keywords are 'mild spells of vertigo', 'mild headaches', 'lightheadedness' etc.
For this, I need some kind of collection of terms that represent symptoms, so that I can detect similar terms in the medical reports I have. Is there any NLTK corpus associated with medical terminology? If I find a list of words that denote medical symptoms, I can tokenise and lemmatise the words I have detected in the medical reports and compare them to the words found in the corpus.
Thank you.

Welcome to the biomedical domain, one of the few domains in NLP where there are too many resources to choose from :)

Data resources:

Medline is a database corpus of 30 millions abstracts.
Each Medline abstract is annotated with Mesh descriptors, Mesh being a structured hierarchy of medical concepts.
PubMed Central (PMC) is a database of around 6 millions full biomedical articles (not only abstracts).
UMLS is a database of millions of medical terms grouped by concept, themselves grouped by semantic type (e.g. disease, gene, etc.)
PubTator is a resource which provides all the Medline and PMC documents fully annotated with a combination of Mesh and other ontologies.

Software tools:

MetaMap is the venerable annotator system which annotates any medical text with UMLS labels.
cTakes is another annotator system which is more specialized with clinical texts.
SciSpacy is a Spacy variant specialized for biomedical text. It can also annotate medical terms with UMLS labels.

I think that's all the main ones that I know of... so far.
From your description it looks to me like you probably just need cTakes or SciSpacy. In case you're going to start working with Medline or PMC, be aware that these datasets are massive (a few hundreds GBs).


how to find the best candidate I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know that higher values are better but I do not know about their weights. I am looking for an approach to find the best candidates (there exist some training data, the suitable candidate of the entity is available).
Could you please advise me which one should be used: classification, regression or otherwise?

Are both the candidates and values to be estimated? Or is it only the candidates.
If it's only the candidates, then try using a classification model. You can then use the predicted class probabilities as weights on the candidate  values to arrive at a final values for your entities.


How to best read large dataset from disk I want to solve a task using a ResNet in keras and tensorflow. My Dataset is big, and right now I'm considering my data loading options and trying to determine which one suits the task best.
About the Dataset:

x: arrays of 200x700 cells in range -1.0...1.0, I don't want to downsample them; they are currently saved as matlab or npz file
y: the label consists of two floats per x.
I have 1.2Million of these (x, y) which are currently saved in 1000 npz files, each with 1GB, totalling to 1TB of data.

Problem: I don't have 1TB RAM in my system, so I can't keep all the data in memory. Thus I need a suitable solution to read my data from disk while training my neural network.
Solutions that I found so far:

save these files as images and use keras dataset io "load_images_from_directory", downside: I need to save the images on disk which would probably take even more than 1TB. And what about the labels? Plus probably additional preprocessing to from range 0..1 to -1..1
tfrecords which feels like an overkill, since my dataset is not really a structured one but it's just (array, label)
hdf files which is also more for structured/hierarchical data.

Things that I also want to take into account:

Do I save my data as is, or do I need to save shuffled batches? But according to this I should also shuffle the mini-batches in each epoch new. This would mean that the order and filesizes (e.g. one file is one mini-batch) whith which I save my files is not important - the mini-batches should be shuffled anyways.
Later, I will most likely also need to transfer the whole project to pytorch, so a data storage which is supported by both (kears/tensorflow and pytorch) can save me some time later.
If I store each (x,y) sample as one small .bin file, this file is smaller than the block size of my disk, thus using more disk size than necessary.

So the question is: What are the pros and cons specific to my dataset/task, thus which dataloading should I use? Are there more options that I haven't discovered yet?

A common way is to create a class inheriting tf.keras.utils.Sequence. This class implements a function __getitem__ which is called when you use model.fit() method. In this method, you can simply load one batch at a time, so no need to load the whole dataset. See the documentation. You can also directly use the .npz files when calling __getitem__.
Shuffle data

Do I save my data as is, or do I need to save shuffled batches?

You would implement the method on_epoch_end() that does the shuffling of a list of indices. Then when you load a .npz file, use something like data[indices[i]] which load the index i in the shuffled list of indices.
on_epoch_end is called at the end of each epoch by .fit() method, and you can also use it in __init__ of your Sequence class to initialize the shuffle.
From Keras to Pytorch
Pytorch has similar module called torch.utils.data.Dataset. The conversion is straightforward. See this tutorial.


What is the difference between bootstrapping and cross-validation? I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see the main difference between them in terms of performance estimation.

As far as I see, bootstrapping is also producing a certain number of random training+testing subsets (albeit in a different way) so what is the point, advantage for using this method over CV? The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this. But this aspect seems to be a very little nuisance.

Both cross validation and bootstrapping are resampling methods.

bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original data set). Due to the drawing with replacement, a bootstrapped data set may contain multiple instances of the same original cases, and may completely omit other original cases.
cross validation resamples without replacement and thus produces surrogate data sets that are smaller than the original. These data sets are produced in a systematic way so that after a pre-specified number $k$ of surrogate data sets, each of the $n$ original cases has been left out exactly once. This is called k-fold cross validation or leave-x-out cross validation with $x = \frac{n}{k}$, e.g. leave-one-out cross validation omits 1 case for each surrogate set, i.e. $k = n$.
As the name cross validation suggests, its primary purpose is measuring (generalization) performance of a model. On contrast, bootstrapping is primarily used to establish empirical distribution functions for a widespread range of statistics (widespread as in ranging from, say, the variation of the mean to the variation of models in bagged ensemble models).
The leave-one-out analogue of the bootstrap procedure is called jackknifing (and is actually older than bootstrapping).
The bootstrap analogue to cross validation estimates of generalization error is called out-of-bootstrap estimate (because the test cases are those that were left out of the bootstrap resampled training set).

  [cross validation vs. out-of-bootstrap validation] However, I cannot see the main difference between them in terms of performance estimation.

That intuition is correct: in practice there's often not much of a difference between iterated $k$-fold cross validation and out-of-bootstrap. With a similar total number of evaluated surrogate models, total error [of the model prediction error measurement] has been found to be similar, although oob typically has more bias and less variance than the corresponding CV estimates.

There are a number of attempts to reduce oob bias (.632-bootstrap, .632+-bootstrap) but whether they will actually improve the situation depends on the situation at hand.

Literature:

Kohavi, R.: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Mellish, C. S. (ed.) Artificial Intelligence Proceedings 14$^th$ International Joint Conference, 20 -- 25. August 1995, Montréal, Québec, Canada, Morgan Kaufmann, USA, , 1137 - 1145 (1995).
 Kim, J.-H. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , Computational Statistics & Data Analysis , 53, 3735 - 3745 (2009). DOI: 10.1016/j.csda.2009.04.009
Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

  The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this.

Yes, there are fewer combinations possible for CV than for bootstrapping. But the limit for CV is probably higher than you are aware of.
For a data set with  $n$ cases and $k$-fold cross validation, you have

CV  $\binom{n}{k}$ combinations without replacement (for k < n that are far more than the $k$ possibilities that are usually evaluated) vs.
bootstrap/oob $\binom{2 n - 1}{n}$ combinations with replacement (which are again far more than the, say, 100 or 1000 surrogate models that are typically evaluated)


How does the validation_split parameter of Keras' fit function work? Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :

  validation_split: Float between 0 and 1. Fraction of the training data
  to be used as validation data. The model will set apart this fraction
  of the training data, will not train on it, and will evaluate the loss
  and any model metrics on this data at the end of each epoch. The
  validation data is selected from the last samples in the x and y data
  provided, before shuffling.

Please note the last line:

  The validation data is selected from the last samples in the x and y
  data provided, before shuffling.

Does it means that validation data is always fixed and taken from bottom of main dataset?

Is there any way it can be made to randomly select given fraction of data from main dataset?

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training.

The function as it is designed ensures that the data is separated in such a way that it always trains on the same portion of the data for each epoch. All shuffling is done within the training sample between epochs if that option is chosen.

However, for some datasets getting the last few instances is not useful, specifically if the dataset is regroup based on class. Then the distribution of your classes will be skewed. Thus you will need some kind of random way to extract a subset of the data to get balanced class distributions in the training and validation set. For this I always like to use the sklearn function as follows

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_valid, y_train, y_valid = train_test_split(data, labels, test_size=0.33, shuffle= True)

It's a nice easy to use function that does what you want. The variables data and labels are standard numpy matrices with the first dimension being the instances.


How to specify scale_pos_weight value at runtime in Hyperopt? I want to use LighgbmClassifier for a binary Classification. for Hyper Parameter tuning I want to use Hyperopt. The Dataset is imbalanced. Using Sklearns class_weight.compute_class_weight as shown below
        clas_wts_arr = class_weight.compute_class_weight('balanced',np.unique(y_trn),y_trn)
        self.scale_pos_wt = clas_wts_arr[0] / clas_wts_arr[1]

The following is the space parameter that I am passing to the objective function
        space = {'objective' : hp.choice('objective', objective_list),
                 'boosting' : hp.choice('boosting', boosting_list),
                 'metric' : hp.choice('metric', metric_list),
                 "max_depth": hp.quniform("max_depth", 1, 15,2),
                 'min_data_in_leaf': hp.quniform('min_data_in_leaf', 1, 256, 1),
                 'num_leaves': hp.quniform('num_leaves', 7, 150, 1),
                 'feature_fraction' : hp.quniform('feature_fraction', 0.5, 1, 0.01),
                 'min_gain_to_split' : hp.quniform('min_gain_to_split', 0.1, 5, 0.01),
                 'lambda_l1' : hp.uniform('lambda_l1', 0, 5),
                 'lambda_l2' : hp.uniform('lambda_l2', 0, 5),
                 'feature_pre_filter': False}

My question will the following set scale_pos_weight properly in the space dictionary
        #set scale pos weight explicitly
        space['scale_pos_weight'] = self.scale_pos_wt

If that is wrong then what would be the correct way to set scale_pos_weight at runtime in the space dictionary that is passed to the Objective fn that is in turn passed to the fmin of Hyperopt.
Thanks for your help and answers.

Use Sklearns class_weight.compute_class_weight, which will return an array of weights.
My problem here is binary hence i had two elements in the array. We need to increase the weightage of the minority class. The target variable 1 was my minority class, thus I set the maximum scale_pos_weight so to the value from the second element in the array.
 clas_wts_arr = class_weight.compute_class_weight('balanced',np.unique(y_trn),y_trn)
 self.scale_pos_wt = clas_wts_arr[1]

As part of the space dictionary passed to the objective fn defined within fmin we include scale_pos_weight like the following
'scale_pos_weight' :hp.choice('scale_pos_weight',scl_ps_wt)

if the fmin object has been defined as follows
            best = fmin(fn=objective,
                    space=space,
                    algo=tpe.suggest,
                    max_evals=self.NUM_EVALS,
                    trials=trials)

Best will have the index of the best value in the array . Using this index we get the actual value of scale_pos_weight . We use this value to set the scale_pos_weight in best .
 best['scale_pos_weight'] = scl_ps_wt[best['scale_pos_weight']]

Now, use can set best as parameter to you classifier.


What should be 'y_train' in Keras LSTM? I refer to the example given at the Keras website here:

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
       input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))

# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))

model.fit(x_train, y_train, batch_size=64, epochs=5, validation_data=(x_val, y_val))

For a real-world example, what should be y_train and y_val? Should they be the same as x_train and x_val respectively, since they come from the same sequence?

Also, how should I understand data_dim and num_classes?

'y_train' and 'y_val' should be whatever it is you are trying to predict. They can be values, classes, or they can be a sequence. The form of what you are trying to predict will influence how you structure a RNN in Keras: Many to one and many to many LSTM examples in Keras.
'data_dim' is the number of features in the dataset. If you are predicting the weather, the features may be 'pressure', 'temperature, 'precipitation', and so 'data_dim' = 3.
'num_classes' is the number of classes that the outcome variable can assume. If the outcome variable is 'type of animal', and the classes the outcome variable can assume are 'dog', 'cat', 'bird', then 'num_classes' = 3.


Changing default values of ANNIE resources in GATE from Java code In GATE, default values for ANNIE are set during initialization, but sometimes based on requirements they have to be changed.
My Requirement : I want to extract English sentences without considering the "nextline character" but considering "full stop" which gives correct sentences. For that, I need to change the default value of transducerURL in SentenceSplitter in ANNIE.
This can be done in two ways:

Using ANNIE_with_defaults.gapp - changing initparams value in Sentencesplitter and accessing from java:
 Gate.setGateHome(new File(Configuration.GATE_HOME));
 Gate.init();
 // Load ANNIE with defaults from the plug-ins folder
 File pluginsHome = Gate.getPluginsHome();
 File anniePlugin = new File(pluginsHome, ANNIEConstants.PLUGIN_DIR);
 File annieGapp = new File(anniePlugin, ANNIEConstants.DEFAULT_FILE);
 annieController = (CorpusController) PersistenceManager.loadObjectFromFile(annieGapp);

I am not being able to find where to change its value in gapp file.

After initialising GATE, accessing ResourceData using CreoleRegister and changing the default value of parameter transducerURL as shown below:
 String resourceClassName = "gate.creole.splitter.SentenceSplitter";
 ResourceData resData = Gate.getCreoleRegister().get(resourceClassName);
 //System.out.println(resData.getParameterList());
 ParameterList params = resData.getParameterList();
 List<List<Parameter>> param =  params.getInitimeParameters();
 System.out.println(param);
 //System.out.println(param.get(0));
 for(List<Parameter> plist : param)
 {
     for(Parameter pm : plist)
     {
         System.out.println("Name : "+pm.getName()+", "+pm.getDefaultValue());
     }
 }

But Parameter does not have setDefaultvalue method.
How can I fix this?

For any of these situations the best approach is to construct the application as you need it using GATE Developer, "save application state" to create your own custom gapp file, then pass your new gapp to the PersistenceManager instead of loading the standard ANNIE_with_defaults.


What features are generally used from Parse trees in classification process in NLP? I am exploring different types of parse tree structures. The two widely known parse tree structures are
a) Constituency based parse tree and
b) Dependency based parse tree structures.

I am able to use generate both types of parse tree structures using Stanford NLP package. However, I am not sure how to use these tree structures for my classification task.

For e.g If I want to do sentiment analysis and want to categorize text into positive and negative classes, what features can I derive from parse tree structures for my classification task?

By using a parse tree, you divide your sentence into parts. Suppose, in the example of sentiment analysis, you can use those parts to assign a positive/negative sentiment to each part and then take the cumulative effect of those parts.

This image will help you understand more. The first half has a negative sentiment(mainly because of the word "dry") but because of the word "but" and the usage of the word "enjoyed", the negative sentiment is turned into a positive sentiment.

As for using them, you can simply generate a word vector representation of the individual words in the sentence and use neurons in place of the parent nodes. Each neuron should be connected to another neuron through weights. All the leaf nodes will be the word vector representations of words of the sentence. The top parent neuron(in this case the top blue + symbol) should generate a positive/negative sentiment according to the sentence. This tree structure can be trained in a supervised manner.

Read this paper for a more through understanding.

Image credits: cs224.stanford.edu


Fuzzy Inference system in R I want to use R for implementing a fuzzy inference system. There are 4 input variables and one output. Each rule is dependent on all input variables and based on there membership the output class is decided.

Can anyone suggest a good library and source examples to begin with? The library that have good visualisation support also.
Should I go with 'frbs' R package?

you can use "frbs", "sets" and fugeR packege for fuzzy logic model building.


Why does the transformer positional encoding use both sine and cosine? In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.

I am wondering why it needs to use both sine and cosine though instead of just one or the other?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.
In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you need to know the length of the sequence beforehand.
This is why the authors propose a cyclic solution. Sin and cos functions can benefit from this idea and thus are the choice here. But why using both instead of one?
It is not clearly answered in the original paper, but if you go though this article(which I urge you to) the reason is that you can use a linear transformation to go from the two functions, to the same functions with an offset:

And if you wonder why having this property available is useful, if you dig into the comments below the article you will find the explanation you seek:

I suppose that this encoding framework enables the model to attend to relative positions by simply generating a transformation matrix, which only depends on the k. To be clear, I hypothesize that given any input PE(pos), the model can create the attention query matrix Q that targets PE(pos+k) by multiplying the PE(pos) with a weight matrix T (the transformation matrix). The weight matrix T, which could be parameters of a single feed-forward layer, can be learned during the training process.

Basically the model can learn a matrix not dependant on t as part o the trining to generalize all the range of encoding values.
Can we just use one of the other?
You could encode the values of the position, yes. But you would not have the property of having this linear transformation available, which seems to be a very important part for the learning process to succeed.


Using appropriate algorithm I went through basic concepts of ML algorithms but still had few queries on their advantages over each other.

When should we use linear regression vs Neural Networks for regression?
When should be use Logistic Regression vs SVM vs Neural Networks classification vs Decision Trees?

If you need something interpretable but accuracy is less important, use the methods on the left side of the question (linear regression, logistic regression). If accuracy is more important, use the methods on the right side of the question (neural nets). SVM is a somewhere in the middle.


0.8 is the threshold value meaning you want to exclude columns(features) with at least 80% of values is 0 or 1: sort of unbalanced dummies. This is only applied well to dummies, but not necessarily general numerical values.
For numerical, you can test RFECV function in feature_selection package from sciki-learn, the syntax as follows:

feature_folds = ms.KFold(n_splits=10, shuffle = True) # cross-validation folds

selector = fs.RFECV(estimator = your_model, cv = feature_folds,
                      scoring = 'roc_auc')

selector = selector.fit(your_features, your_labels)

selector.support_


How to interpret shapley force plot for feature importance? I am trying to practice and learn shapley value approach to explain my predictions on a binary classification problem. However am having difficulty in understanding the below plot.

1) Does it indicate the day_2_balance influences prediction to 1? or does blue values leads to prediction 1

2) What about the axis scale? (-4.357 to 5.643)

3) What does base value mean?

4) When I hover around the pink color , I see few more column names with some values. What do they indicate?

5) does the size of features represent their importance? Meaning PEEP_min=5 has a larger size than other features?

6) What does higher to lower and lower to higher indicate?

Can someone help me with this?

I've never practiced this package myself, but I've read a few analyses based on SHAP, so here's what I can say:

A day_2_balance of 532 contributes to increase the predicted output. In this area, such a value of day_2_balance would let to higher predictions.
The axis scale represents the predicted output value scale. The actually predicted value is in bold font (-2.98). I don't know if the min and max values of the scale represent true min and max of the model predicted values. They more likely result from automatic scaling of the plot.
Base value should be the mean value of the estimator over the whole input space.
Those indicate the other features of which value influence the prediction to a higher value. Features are sorted by local importance, so those are features that have lower influence than those visible.
Yes, but only locally. On some other locations, you could have other contributions
higher/lower is a caption. It indicates if each feature value influences the prediction to a higher or lower output value.

From the example plot, you can draw the following interpretation: "sample n°4100 is predicted to be -2.92, which is much lower than the average predicted value (~0.643), mostly due to the specific values of features PEEP_min(5), Fi02_100_max (50), etc., and although day_2_balance is 532".


In recommender systems, how to avoid recommending a product that the user has just bought? Suppose I'm running an online store that sells many products, but from only a couple of categories, say: $A$, $B$, or $C$.

Let's say a user has bought a product in the A category, and there's no sense of recommending to him products from that category, at least for some time. This might be in a form of a "you might also like" panel, or indeed a modification to the recommendation engine itself.

Even better: let's suppose that I know the probability density $P(\Delta t | A)$ of a product being re-bought after time $\Delta t$ from within the $A$ category (same for other categories).

My question is: how to choose the time threshold $\tau_X$ for each category $X\in \{A, B, C\}$ such that once a product in category $X$ was bought by a given user, other products from that category are not going to be recommended to that user for time $
\tau_X$?

The main "gain" from not showing a user products from that category is that we can use the recommendation slots for other categories, thus increasing the chance of the user buying new products (from other categories).

Does this set up resemble any well-known problem? What keywords should I google to find a more formal problem statement and/or possible solutions?

Interesting, so yeah I assume you can find out the time between purchases within category from your data. Some random ideas:

The simplest answer I can think of is to find the mean time per category and use that as the threshold. Empirically, it's the point at which customers are more likely than not to have bought again.

That's fairly binary, just removing all recommendations from a category. You could imagine, say, keeping 20% of the top recommendations in category A when the cumulative probability of enough time having elapsed is 20%. Or factoring it into a scoring function, and multiply by this probability and then rerank. (That is, there's an 80% chance not interested at all, and a 20% chance interested as usual.)

To do that you'll probably need to fit a distribution, and Weibull seems attractive. As a generalization of exponential, which can be thought of time between events like purchases where the rate is constant, it can accommodate a rate that increases over time. That is, the rate of repurchase starts off low and increases.

Both of those are too pessimistic as you more want to assess when they're as willing as ever to buy again, not so much buy again, which may be pretty infrequent in your data. Perhaps you model the rate of purchase per category among those who have and have not purchased in the category and empirically figure out when the rate goes back to similar.


How much is the Class Imbalance Problem rates? I'm working on a data set and wanted to know is there a standard rate about Class Imbalance problem or not?

I have 47 samples in Class A and 150 Sample in class B , should I use Class Imbalance Technique or these rates are normal?

There is no general rule but you better to use such techniques (SMOTE, sampling etc) and try to obtain 50:50 if you can.
You can utilize my another answer (just ignore multiclass part).

My suggestion is you should create more samples by synthetic data generators like SMOTE. You have a model with very less observations (197). The model may not fit well thus data generation may help your for that case.

Hope it helps!


Object Detection scratch implementation or high level API? I am at an intermediate level in Deep Learning. I wanted to implement object detection from scratch. Is the scratch implementation possible by a single person or will i have to use an high level API to do so. I am confused and don't know where to start.

As you are looking to implement already known networks with a framework such as pytorch/tensorflow, I would definitely suggest reading into this post : https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98

Given that you wanted to implement this from scratch, its important you understand what the following:

Bounding boxes in images
Loss function - Intersection over union
Network architecture - You can use nettron to learn about the network architecture once you have written it.
Visualize the learning of your model after every 'x' epochs, to gain an intuituve understanding of what your network is trying to learn given the training dataset.


Are Decision Trees Robust to Outliers I read that decision trees (I am using scikit-learn's classifier) are robust to outlier. Does that mean that I will not have any side-effect if I choose not to remove my outliers?

Yes.
Because decision trees divide items by lines, so it does not difference how far is a point from lines.


Why does Transfer Learning works better on smaller datasets than on larger ones? This question is not about the utility of Tranfer Learning compared with regular supervised learning.
1. Context
I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. In health-monitoring, a major issue is the low amount of failure examples (one can't afford to perform thousands of run-to-failure tests on aircraft engines). This is why Transfer Learning has been studied to solve this, in Transfer Learning with Deep Recurrent Neural Networks for Remaining Useful Life Estimation, Zhang et al, 2018. My question is about the results presented in this article.
2. Question
C-MAPSS dataset is composed of 4 subdatasets, each of which has different operational modes and failure modes. The article cited above performs transfer learning between these subdatasets. Particularly, when training a model on a target subdataset B using the weights of a trained model on a source dataset A, they don't train on all the B dataset. They conduct an experiment in which they test various sizes for the target dataset B : they try on 5%, 10%, ..., 50% of the total dataset B.
The results are presented in page 11. A few cases excepted, the have better results on smaller target datasets. This seems counter intuitive to me : how could the model learn better on less examples ?
Why does Transfer Learning works better on smaller datasets than on larger ones ?

In the article you provide, from page 11 results, I think one cannot conclude that transfer learning works better on smaller datasets than on larger ones.
If you look at the results of transfer learning score values (or RMSE) vs size of learning, it is also getting better while dataset size is increasing (for instance E2 or E5 or E8). So transfer learning does not work better on small datasets.
However, you might be looking at the IMP index which is based on the mean score (or RMSE) of learning with and without transfer learning.

IMP= (1−(WithTransfer)/(NoTransfer))×100

The index is based on two curves.

WithTransfer which will have good performances even at the beginning because when using relevant transfer learning, the model could already extract pertinent information from a very small testing dataset.
NoTransfer which will start with poor performances (difficulty to generalize) and  then increasing with the size of testing data.

The IMP index has then the expected curve you're pointed out, for example with E2 and E5.


tensorflow saved model to java Hello data scientists (I'm not),
I adapted a multi-class-text-classification-cnn from https://github.com/jiegzhan/multi-class-text-classification-cnn (thanks Jie Zhang)
It works ! 90% accuracy !
Now my issue ;-)
I really need to load my model with java in order to allow my existing java service to predict categories.
According to documentations, I need to use SavedModelBuilder which is the new way of modele saving. I tried for hours but I misse a lot of things... About signature, assets.
If someone could help me with that ? => replace the old saver.save by tf.saved_model.builder.SavedModelBuilder into https://github.com/jiegzhan/multi-class-text-classification-cnn/blob/master/train.py
He or she will have my infinite gratitude

train.py code:

builder = tf.saved_model.builder.SavedModelBuilder("/home/datam/cnn-text-classification-tf/model/20180423")
builder.add_meta_graph_and_variables(
    sess,
    [tf.saved_model.tag_constants.SERVING],
    clear_devices=True
)
builder.save()

Java Coding:

SavedModelBundle model = SavedModelBundle.load(SAVED_MODEL_PATH+"/20180423", "serve");
Session s = model.session();
Graph g = model.graph();


How to perform a running (moving) standardization for feature scaling of a growing dataset? Let's say that there is a function $r$
$r_n = r(\tau_n)$,
where $n$ denotes a so-called time-step of a system with an evolving state. Both $\rho$ and $\tau$ should equally influence $r$, and should therefore be scaled. The problem is, the sequence $(\tau_1, \tau_2, \dots, \tau_n)$ grows in time because $n$ grows.
How to perform a running standardization of $(\tau_1, \tau_2, \dots, \tau_n)$. Running mean is relatively simple to express:
$\text{mean}(\tau)_{n+1} = \frac{1}{n+1}\left[\tau_{n+1} + N \text{mean}(\tau)_n\right]$
where $\text{mean}(\tau)_1 = \tau_1$.
The standardization requires
$\tilde{\tau}_n = \dfrac{\tau_n - \text{mean}(\tau)_n}{\sigma(\tau)_n}$
where
$\sigma(\tau)_n = \sqrt{\dfrac{1}{n-1}\sum_{i=1}^{n}[\tau_i - \text{mean}(\tau)_n]}$ (1)
is the standard deviation of $(\tau_1, \tau_2, \dots, \tau_n)$.
Question: is there an expression for a running standard deviation? Online I've only found links on stack overflow and Matlab functions, but I am not sure which algorithm is best suited for feature scaling. By running (moving) I mean not having to store $(\tau_1, \tau_2, \dots , \tau_n)$ to calculate (1), instead update it incrementally.

I think you want
$$S_{n}=S_{n−1}+ (x_{n}−μ_{n−1})(x_{n}−μ_{n})$$
where S, x and μ are respectively the variance, value and mean.
See https://fanf2.user.srcf.net/hermes/doc/antiforgery/stats.pdf for explanation and derivation.


Calculating image text orientation in scanned documents I'm trying to calculate the text orientation in scanned documents in order to de skew them. I'm aware of some of the common methods which include using hugh transform to do the same. However I recently read an article which I can't find.
If I recollect correctly, it basically said that find the nearest neighbours for the connected components in the image. This should connect the adjacent alphabets in the words, connect the centroid of those components and build a histogram of those vectors. The histogram should give you the skew angle of the vectors.
Does anybody know such an approach for identifying document skew?
Any other approaches are appreciated as well.
Thank you.

You might be thinking of this article, which mentions some other approaches too!


How to build a supervised artificial neural network? I am trying to build and train a machine learning data science algorithm that correctly predicts what president won in what county. I have the following information for training data.

Total population Median age % BachelorsDeg or higher Unemployment rate Per capita income Total households Average household size % Owner occupied housing % Renter occupied housing % Vacant housing Median home value Population growth House hold growth Per capita income growth Winner

I am new to the field and this is my first time building an artificial neural network and I don't know what to do to start (so forgive me if my question is extremely broad). What I have done so far is read the wikipedia page for artificial neural networks. What I want to do next is implement supervised learning using the training data.

I would appreciate any help getting started. Please let me know if you have used any good tutorials or libraries to help build something similar. I was thinking of using the library Lasagne.

A few specific questions I have as I tried to use Lasagne - What is the error I am to calculate when building an algorithm using training data? How many layers do I need and what does each layer signify?

Note that I am cross-posting this answer because the question is cross-posted, and one of the questions will likely be deleted. If this is a breach of etiquette, please let me know. Cross-post is here.

This question has two parts, at least implicitly.  The second part is, generally speaking, how to train and predict from neural networks (in Python).  The first part is (implicitly) how do we build a model that predicts (correctly) which person won a presidency based on demographics.

The answer to the second question is quite easy to state, but harder to do.  It boils down to the rather unhelpful statement:  learn about neural networks, and use that knowledge.  Google for tutorials on neural networks. I would make recommendations, but I don't know your mathematical, statistical, and computer science background, and thus cannot really make suggestions.

The first question, however, I can answer, although likely not in the way you'd like.  Just using the data you've listed, it's highly unlikely you'll get any sort of accuracy out of your neural networks, or any other model you fit.  Why?  For several reasons.

The first is that unless you can always classify the president into one of two (or more) distinct groups, as you can in the US (Democrat, Republican), your data are changing all the time.  You can't classify anything if it doesn't have a class, and that's difficult in some countries.

Secondly, even if there are distinct classes, demographic data is likely not a set of good predictors.  Demographic data changes depending on how good the local governance is, as well as the overall governance.  You might, might be able to code that in somehow.  But demographic data also changes due to a lot of other things. When it comes down to voting, people choose to vote based on a bunch of things, and their demographic status is only one factor.

The third reason is more a caution than anything.  Neural networks and all kinds of machine learning/statistical modelling methods can give good predictions with the right kind of data.  Something like predicting a president is relatively difficult, especially in smaller countries.  (Nate Silver did it well in the US, but there was a lot more data there.)  And even if you've got great data and a great model, there's invariably something you miss.  George Box said it right:  "All models are wrong, but some are useful."  When you mention predicting correctly, all sorts of alarm bells go off in my head.  The problem with trying for "correct" prediction is that there are no guarantees your prediction will be correct, often even in simple cases.  No model is perfect.


Finding best cluster after running K-Means clustering I have bunch of text which I want to segregate based on semantic similarity. Running through K-Means, I was able to divide the complete text into different clusters.

However, I still need to find which cluster contains most similar text. For e.g :-

Cluster 1 :-
'Hey there, how are you'.
'How are you dear'.
'I haven't seen you for past three days. How are you'.

Cluster 2 :-
'As per survey conducted, smoking kills more people than ...'.
'Avoid smoking in public places.'.
'People more likely to avoid smoking if they ...'.
'As per survey conducted, people most likely chose democrats'.

Cluster 1 should rank above Cluster 2 as it has more semantic similarity among text contained within.

Generally, library that you use for k-means clustering will report "within cluster sum-of-squares" value after running k-means procedure. A smaller value means the cluster is tightly bound (most messages are similar to each other). You can use this value to order the cluster IDs.


Pyspark code is not performant enough when compared to pure python alternative I transformed the existing code which was in python pasted below was in pyspark.

Python code:

import json
import csv

def main():
    # create a simple JSON array
    with open('paytm_tweets_data_1495614657.json') as str:

        tweetsList = []
        # change the JSON string into a JSON object
        jsonObject = json.load(str)

        #print(jsonObject)

        # # print the keys and values
        for i in range(len(jsonObject)):
            tweetsList.insert(i,jsonObject[i]["text"])

        #print(tweetsList)
    displaySentiment(tweetsList)

def displaySentiment(tweetsList):
    aDict = {}

    from sentiment import sentiment_score

    for i in range(len(tweetsList)):
        aDict[tweetsList[i]] = sentiment_score(tweetsList[i])
    print (aDict)

    with open('PaytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for key, value in aDict.items():
            writer.writerow([key, value])

if __name__ == '__main__':
    main()

Converted Pyspark Code:

import json
import csv
import os
from pyspark import SparkContext, SparkConf
from pyspark.python.pyspark.shell import spark

os.environ['PYSPARK_PYTHON'] = "/usr/local/bin/python3"

def main():
    path = "/Users/i322865/DeepInsights/bitbucket-code/ai-engine/twitter-sentiment-analysis/flipkart_tweets_data_1495601666.json"
    peopleDF = spark.read.json(path).rdd
    df = peopleDF.map(lambda row: row['text'])
    print(df.collect())
    displaySentiment(df.collect())

def displaySentiment(tweetsList):
    from sentiment import sentiment_score

    aDict = sentiment_score(tweetsList)

    #
    with open('paytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for i in range(len(tweetsList)):
            writer.writerow([tweetsList[i], aDict[i]])
            print([tweetsList[i], aDict[i]])

if __name__ == '__main__':
    conf = SparkConf().setAppName("Test").setMaster("local")
    sc = SparkContext.getOrCreate(conf=conf)
    main()

I ran both programs but didn't see any significant performance improvement. What am I missing? Please could you shed some thoughts?

You have instantiated a sparkContext object with "local" mode configuration. It means you have allocated ressources for a single multi-core Java Virtual Machine on your computer. In this configuration, you can't reach better performs than with python. Because :

With local mode, you have less ressources (cpu, memory...) with Spark than with python (you can't have a JVM with more ressources than your computer).
PySpark code is converted into Scala code before execution.

Spark benefits arise when you use Spark on multiple nodes. In this configuration, spark master is yarn, mesos or standalone. In that case, a spark job would be separated in multiple tasks and each node would be dedicated to different tasks.

For example, if you have 3 nodes and 6 tasks, each node can handle 2 tasks. If you have 6 nodes and 6 tasks, each node will handle a task. With simple python, think that you have only 1 node for "6 tasks". So, for large tasks (big enough datasets), Spark will results in better latencies than simple python.


Modulo as activation function in neural network Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our input data.
By modulo function $f(x)$ I mean

\begin{equation}
f(x) = \begin{bmatrix}
\vdots \\
  x+a \ \ \  \if \  -a < x < 0 \\
 x  \  \ \ \ \ \ \  \if\  \ \ \ 0 < x < a\\
 x-a \  \ \  \if  \ \ \ \ a < x < 2a \\
\vdots \\
\end{bmatrix}
\end{equation}
where a is a positive constant number and could be treated as hyperparameter for simplicity.

I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution (, $\sigma^2$) where , $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network

I am new to Latex, sorry for not using a good formating.

The modulo function is not monotonic.
Even though it has a positive derivative except at points where it is not differentiable.
It is generally not recommended to use a non-convex Activation function. Hence, it is better to avoid it or change the structure of problem.


How to manipulate this column of less than/greater than? I have a dataset in which one column is as given in the picture.
What will be the best way to handle such a column?

This is a categorical variable. There are multiple ways to handle categorical variables in the literature -

Dummification (mostly for nominal variables)

The variable is split and converted into multiple binary variables (the number of variables should be 1-(number of classes) ).

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k',
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])

print(pd.get_dummies(series).iloc[:, 1:])

Output

>50k
0      0
1      0
2      0
3      0
4      0
5      0
6      0
7      1
8      1
9      1
10     1
11     1
12     0

Notice how the information that was string has been now encoded in numbers. The reason was dropping a column is so that there is no perfect multicollinearity.

Ranking(in case of ordinal variables)

These are the cases where you can clearly rank the variables. In your case it would make more sense to rank them as we know then >50k is actually more than <50k.

Use the ranking method in you salary case and the dummification method for nominal cases (like location variables - where it can't be ordered)

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k',
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])
uniq = series.unique()
ranks = { uniq[i]:i+1 for i in range(len(uniq)) }
print(ranks)
series_ranked = series.apply(lambda x: ranks[x])
print(series_ranked)

Output

0     1
1     1
2     1
3     1
4     1
5     1
6     1
7     2
8     2
9     2
10    2
11    2
12    1
dtype: int64

Again, in no way is this an exhaustive answer, but rest assured these are the most popular methods used to deal with categorical variables. Hope that helps. Cheers!


How to represent audio data in a format that can be used for preprocessing and modelling? I have a project that I am working on currently. The project is to classify audio data. The data is in two folders train and test. In the train folder, there are multiple folders like cat, dog, elephant etc and these folders contain multiple audio files. For example, there are multiple audio files inside the folder dog, cat etc. The folders are also the class names representing the class the audio belongs to. The audio is in .WAV format. How to represent this data in a format on which I can use Machine learning models?

To explain my question a bit: let's take a .csv file. We have rows and columns. Rows are data points and columns are features and training machine training models on this data makes sense.

So, How do I represent this audio data that I can work with and fit ml models?

Audio .wav codec file has a 44 byte header which will give you critical data like bit depth ( CD quality audio is 16 bits per sample), sample rate ( CD quality uses 44,100 audio samples per second ), number of channels, etc ... the balance of bytes in a .wav file is the payload which is the audio curve stored as a set of integers which define the height of the audio curve across a succession of instances in time. The beauty of .wav is its not compressed so all bytes after the header only store the audio curve.

Many audio libraries will allow you read a wav file and return back a floating point array of samples per channel.  Visualize a dog bark, this sound propagates from the dog and lands on the membrane of a microphone.  Similar wobbling different mediums.  This wobbling recorded over time can be represented as a time series curve ... as its digitized its transformed from analog to digital ( ADC ) the original analog audio wave is ~sampled~ X number of times per second to generate a simplification of that analog curve during this analog to digital conversion.  Each audio sample represents the height of that audio curve at a specific instant in time.

If you only devote 1 bit to record the audio curve, for a given sample, this will transition from 0 to 1 as the audio curve wobbles above and below the stationary silence flat line mark.  Whereas a bit depth of 16 bits which can store 2^16 distinct integers allows far greater granularity and so more accurately models the height of the audio curve.

Unlike image processing where conceptually the entire dataset can be ~consumed~ in an instant of time, audio has an inherent dimension of time.  Audio is a time series.  Another attribute of audio is the simplicity with which any arbitrary audio curve can be represented by a set of pure sin curves, each with an amplitude, frequency and phase shift.  Joseph Fourier outlined the theory whereby time domain data like audio can be represented equally well in either the time domain ( wav codec or analog audio curve ) or in the frequency domain.  It does not go unnoticed that this symmetry implies a conservation of information.

To an arbitrary degree of precision a window of audio samples ( in the time domain ) can be transformed using a Fourier Transform into its frequency domain representation with a controllable degree of information loss or lack thereof.  Be aware that as you increase the number of samples in this window you decrease the temporal specificity of the resultant spectrogram ... meaning the ability to pin point when a frequency occurred is aided by using the lowest number of audio samples in your window send into the FFT call ... although the frequency resolution in the freq domain is increased by feeding in a larger number of samples to your FFT call ... there is no such thing as a free lunch

So now you have options ... perform your analysis of audio while in its native time domain or send it into a FFT call and have the same information as a set of frequency bins where each bin has values for ( frequency , amplitude, phase shift )

This should give you ideas to tease apart and dig into - take care