RL's policy gradient (REINFORCE) pipeline clarification I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly non-conventional):

$$\omega = \omega + \alpha \cdot \nabla_\omega log\ \pi(A_t|S_t) * V_t$$

My questions I am unsure about are the following:

Do I calculate the gradient values at each time step $t$ (like in
SGD fashion) or averaging gradient over all timesteps of the episode is a better option?
Do I care about getting the gradient values of the selected action
probability output only (ignoring the outputs for other actions, in
a discrete case)? In other words, do I consider the $V_t$ term for non-selected actions to be 0, which make the gradient values equal 0 as well?
In a discrete case the cross-entropy (the loss) is defined as:
$$H(p,q) = -\sum_x P(x) * log Q(x)$$

(source: wikipedia)

Does that mean that if I substitute the labels (denoted as $P(x)$)
with the $V_t$ terms (non-zero for selected action only) in my
neural network training, I will be getting the correct gradient
values of the log-loss which fully satisfy the REINFORCE definition?

For notation and visualizations please take a look at this excellent tutorial Policy Gradients.

For your questions:

The second is correct. In PGs we try to maximize the expected reward. In order to do this we approximate the expectation with the mean reward over samples of trajectories under a parametrized policy. In other words sample actions and get their respective rewards over a period of time within an episode. Compute the discounted return from last step back to the first step (this is your $V_t$ in your notation which usually you will find it as $R_t$ and is the discounted return). Multiply returns with logits and sum. Take a look at slides 8 and 9 of the (1) to see how the REINFORCE is being implemented along with this code examples (lines 59-75).
As you may have already realized, no. It's a return over an episode (multiple timesteps) and is calculated as a discounted sum of all the rewards that you collected. Even if you get 1 at the end and 0 everywhere else the reward is propagated back to the first step (doing it by hand helps a lot!) so all rewards at every timestep are converted into returns from that state and timestep.
Look at slide 13 of (1). Your intuition is correct. The maximum likelihood equation for a classification problem (cross-entropy) multiplied by the return equates the policy gradient loss. If you use a simple Neural Network with REINFORCE with two actions to perform a task you will notice that the gradients propagated back are the same as in a classification task but here are multiplied with their respective return (line 69 2) instead of the class label (0/1).

A bit of intuition for the 3: This is with accordance with PG methods considered as Model-free methods and map states to actions. If you use a clustering technique to cluster the hidden representations of the last layer of your network (after dimensionality reduction) and color the data points according to which action your network chose, you will find out that the representations are naturally clustered into separate groups depending the action. At a high level, you can state that the REINFORCE performs a type of classification having as label the reward signal.

Hope it helps!


Auto.arima with xreg in R, restriction on forecast periods I am using the forecast package and implement auto.arima with xreg. Here I want to forecast only for 1 year ahead but I am unable to use h parameter in the forecast function. Below is the reason for that:

  Definition is given in manual(F1 check):

  h = "Number of period of forecast but if xreg is used 'h' is ignored and the forecast period will be number of rows"

Please suggest me an alternate way to use h for the specific period forecast.

Using xreg suggests that you have external (exogenous) variables. In this, a regression model is fitted to the external variables with ARIMA errors.

When forecasting you need to provide future values of these external variables. In practice, these are often forecasts or could be known. For example, if you're trying to predict Sales and you use Advertising spend as an external variable, you may know the advertising spend for the upcoming year.

auto.arima then produces forecasts for the length of xreg, therefore disregarding h.

Based on your comments below, I've provided an example script demonstrating this based on the Sales example above.

library(forecast)

# Generate sample data
sales <- sample(100:170, 4*10, replace = TRUE)
advertising <- sample(50:70, 4*10, replace = TRUE)

# Create time series objects.
sales_ts <- ts(sales, frequency = 4, end = c(2017, 4))

fit <- auto.arima(sales_ts, xreg = advertising)

# If we pass external_regressor into the forecast, h will be disregarded and we will
# get a forecast for length(external_regressor)

wrong_forecast = forecast(fit, h = 4, xreg = advertising)

length(wrong_forecast) # Will be 40

# To forecast four quarters in advance, we must provide forecasted external regressor data
# for the upcoming four quarters, so that length(new_regressor) == 4.
# In reality, this data is either forecasted from another forecast, or is known. We'll randomly generate it.

upcoming_advertising <- sample(50:70, 4, replace = TRUE)

correct_forecast <- forecast(fit, xreg = upcoming_advertising)

length(correct_forecast$mean) # Will be 4

The key things to note are:

If we forecast with the same regressors as we did when generating the forecast, h will be disregarded and a forecast will be generated for the length of xreg in your case, 10 years.

As such, we must provide new data for xreg for the length of time we wish to forecast - in your case, 4 quarters.


How we can Identify Specific Feature from a larger amount of Dataset? In Machine Learning, we need to play with any kind of datasets.
In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called columns),
So the main problem for a data scientist is to know the behavior of the dataset and get meaningful insights from the dataset,
Let's take one example from Kaggle platform, There's a dataset for house price prediction to know what the price of a house, based on there features,
here is the link of the datasets
House Price Prediction Advance Regression Machine Learning Problem

So, The question is how to identify the meaningful feature from the dataset?

I dont think there is one correct way, but what you can do is

Use PCA if you have many features. This will reduce some number of features based on the amount of variance in each feature. You may use other dimensionality reduction techniques.
You can use models like Lightgbm or random forest and know which feature are important. 3. You may use Lasso Regression for feature selection.
You may use intuition to see if some features are just does not make any sense.

These are some of the methods to understand important features.
You may read this article:
https://towardsdatascience.com/the-5-feature-selection-algorithms-every-data-scientist-need-to-know-3a6b566efd2


implementing an algorithm that mixes data clustering and linear regression i have the following dataframe available in the link as a csv, it conveys information about stars.

more specifically -
column ID represents arbitrary ID of sample.
column z represents my target variable (response).
the other columns represent the attributes available for each sample (predictors) and their corresponding measurement errors.

i used the following code to reduce the 11D data to 3 principal components and plotted the scatter of the data in principal space (with color indicating of the target variable Z)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

#first we remove the target z and ID from the dataset and standredize it (mean=0 and std=1)
pca = PCA(n_components=3)
data_for_pca=data_clean.iloc[:,2:13]
data_for_pca=StandardScaler().fit_transform(data_for_pca)
#now we perform the pca and get the amount of variance, or relative information that each new component holds.
principal_c=pca.fit_transform(data_for_pca)
pd.DataFrame(pca.explained_variance_ratio_).transpose()

import matplotlib.cm as cmx
from mpl_toolkits.mplot3d import Axes3D
def scatter3d(x,y,z, cs, colorsMap='jet'):
    cm = plt.get_cmap(colorsMap)
    cNorm = matplotlib.colors.Normalize(vmin=min(cs), vmax=max(cs))
    scalarMap = cmx.ScalarMappable(norm=cNorm, cmap=cm)
    fig6 = plt.figure()
    ax6 = Axes3D(fig6)
    ax6.scatter(x, y, z, c=scalarMap.to_rgba(cs))
    ax6.set_xlabel('pc1',fontweight='bold')
    ax6.set_ylabel('pc2',fontweight='bold')
    ax6.set_zlabel('pc3',fontweight='bold')

    scalarMap.set_array(cs)
    fig6.colorbar(scalarMap)

    plt.show()
scatter3d(principal_c[:,0],principal_c[:,1], principal_c[:,2],np.array(data.iloc[:,1]))

i attached the dataframe and code so anyone can reproduce and observe the 3d plot from all directions, my main intention with this question is to check if my intuitive analysis of the results is good and how to implement my idea on predicting z from the predictor data.

i see that the dots have somewhat of a smooth spherical gradient centered - roughly speaking - at (-2.5,-1,-0.25).
perhaps i should implement some kind of gaussian kernel? if that's a good idea, how can i implement it?

another observation is that the data is slightly clustered in "plates" (as can be seen  .

maybe i should perform seperate linear regression for each cluster/plate. and let the algorithm classify each point to a cluster/plate, then for each cluster/plate i can infer the target with the more sensitive linear regression coefficients.

if you think that might work, how should i implement it?

maybe there's a more rigorous approach to the further analysis of the PCA? (i mean that i'm kinda using my eyes to decide what is best to do but i'm sure there's a computational approach to the task).

would love to hear opinion and advice, this is a solo project that is part of an attempt to get a better understanding of data science after a BSc in physics.

thanks in advance!

A lot of the values from your data seem to be of the same size, or very small difference, also it seems it is important to deal with noise. The same problem is true for image compression. I think a good approach would be to use Haar-Wavelet Transform which compresses the information by a very huge amount and reduces the noisy data, like it does for image compression. After the transofrm, you need to apply quantization on the resulted data, here you can simply set all values to zero which have the smallest absolute value up until some bound, the remaining values you can also quntize into groups, if their difference is less than some bound. By doing this, you get rid of all noise. From here you would have to specify more precisely, what you want to achieve next. You want to predict z?
For this task you can simply use Decision Tree learning methods, using the maximum information gain to decide on which attribute to make the first splits. Random Forests, which use a lot of different Decision Trees, where each one can be seen as an expert on some concrete task, should give very good results on this data, a good quantization method is very important for trees. You can also implement them all by yourself, without using any black box libraries, you learn much more from implementier it yourself. "McGrawHill_-_Machine_Learning_-Tom_Mitchell" Has very well explanations on those algorithms, explaining the mechanics behind all those black box like approaches.


How to convert a text to lower case using tm package? I am using the below R code to convert text to lower case:

movie_Clean <- tm_map(movie_Clean, content_transformer(tolower))

However I end up getting the below error:

  Error in FUN(content(x), ...) :    invalid input 'I just wanna watch
  Jurassic World í ½í¸«' in 'utf8towcs'.

Please help how to overcome this error.

This seems like an encoding error. Try adding the line

Encoding(movie_Clean)  <- "UTF-8"

before you lowercase the data. Check out this answer for a little context: https://stackoverflow.com/a/28340080/4539807


Probability vs recall for time series classification task I am working on the time series classification task that focuses on predicting a fault. I framed the problem as a multi-step forecasting problem, where my goal is to predict to the class at y(t+1 hour) = f(X_lagged). My target y is either 1, or 0, and the problem is essentially a binary classification problem. In the papers I have read, recall/precision/f1 metrics are used as a model evaluation metric. However, if I want to state the problem as "What is the probability that the fault will happen in 1 hour?", what metric do I need to use?

[edited]

The Area Under the Curve (AUC) is a good fit for this kind of problem, but it's not exactly a standard case. The ROC curve is built by ranking your instances by their predicted probability of fault, for example:

p(faulty)   gold
0.03        0
0.14        1
0.19        0
0.23        0
0.31        1
0.32        0
0.65        1
0.78        1
0.83        0
0.90        1

Each level of probability corresponds to a possible threshold for the binary classification problem. The number of True/False Positive/Negative can be calculated for every possible level of threshold:

p(faulty)   gold TP TN FP FN
                 5  0  5  0
0.03        0    5  1  4  0
0.14        1    4  1  4  1
0.19        0    4  2  3  1
0.23        0    4  3  2  1
0.31        1    3  3  2  2
0.32        0    3  4  1  2
0.65        1    2  4  1  3
0.78        1    1  4  1  4
0.83        0    1  5  0  4
0.90        1    0  5  0  5

At this stage one obtains exactly the data required for a standard ROC curve: from these values precision and recall scores can be computed for every level and plotted on the ROC curve. The AUC is used in the usual way.

This method takes into account the continuity of the probabilities while matching predictions against binary classes. An example of problem where this evaluation method is used: https://pan.webis.de/clef15/pan15-web/author-identification.html


Regularization hyperparam tuning during training I have an idea for a regularization-hyperparam selection method, which I haven't encountered before and can't find on Google, but I'm sure someone has already tried it and I'm wondering what are the best practices.
The most common method for hyperparam selection is to select different hyperparams (e.g some value for L2 regularization), train NNs with them, and test the NNs on some validation set - and select the best one.
My idea is to train a single NN and test the NN on a validation set between epochs, and then auto-adjust the regularization hypeparam between epochs - if we see that the accuracy on the validation set is decreasing between epochs, then we should increase the value of the L1/L2/dropout.
Naturally, this can be more efficient than training multiple NNs.
It's still a basic idea, and I'm sure it can be developed further. Is there research and best practices in this field?

As mentioned above you can try a couple of things, depending on which framework you are using and how you want to go with hyperparameter optimisation.
sklearn (with Keras wrappers):

GridSearchCV: slow but sure to find optimal hyperparams
RandomizedSearchCV: faster and almost as good as GridSearch

Keras (with Keras Tuner):

BayesianOptimization: tuning with Gaussian process
Hyperband: Variation of HyperBand algorithm (Li, Lisha, and Kevin Jamieson. Journal of Machine Learning Research 18 (2018): 1-52.)

Pytorch (with RayTune):

Population Based Training: trains a group of models (or agents) in parallel (https://deepmind.com/blog/population-based-training-neural-networks)
HyperBand: as above
ASHA:Compared to the original version of HyperBand, this implementation provides better parallelism and avoids straggler issues during elimination
Median Stopping Rule: implements the simple strategy of stopping a trial if its performance falls below the median of other trials at similar points in time.
FIFOScheduler: Simple scheduler that just runs trials in submission order


2nd, 3rd, Nth closest guesses I have used the KMeans algorithm to create an engine that can guess the cluster that a particular set of input data will fall into.

Can I use it to guess the 2nd closest cluster, 3rd closest, and so on?

Currently, I am using the sklearn.cluster.KMeans library if that's any help - it doesn't seem like the API provides this functionality already.

You could simply compute the distance to each cluster center provided in the cluster_centers_ attribute (once the KMeans instance is fitted). The predict method actually does that for the closest cluster center.


Which is more important - stable training results or good test results? Which is more important - stable training results or good test results?

For instance, is obtaining an unstable training accuracy in different epochs, but good test accuracy better? Or is obtaining a stable training accuracy in different epochs, but test accuracy is not good.

Which one is the best choice?

I would personally prefer the scenario where the training/testing accuracy is high/low respectively, because it would mean that your model was at least well-fit in the training data but unfortunately performed poorly on unseen data. It is also a conservative scenario, because it can't get worse than that, so you have a good feeling that your model needs improvement.

On the other hand, if your training accuracy is low, you shouldn't even test the model on unseen data, because you are certain that it is not trained well enough. However, if you insist on testing it, even though the training accuracy is low, and the testing results are better than the training results, then this probably means that you were lucky during testing. Under no circumstances should you make any inference of the model's quality in such a scenario.

The solution to your problem would be to use k-fold validation, meaning that you should randomly split your dataset multiple times into different training/test sets and evaluate its training/testing accuracy over all the splits to have a better feeling of how the model behaves on your dataset.


How to use multiple encoders(one-hot and numerical) together for PCA I want to implement PCA on a dataset(retail) but the data is categorical. One-hot encoding on some columns like Gender, Fabric, Brand makes sense but on other features like price range, size, I would like the encoded values to have some numeric significance, i.e. higher value actually means something. Any suggestions on implementing both these encodings together for PCA?

There is a method that can handle multiple types of data simulataneously called Generalized Low Rank Models - actually one paper that deals with it is called PCA on a Data Frame. GLRMs in Python (and not only Python) are implemented in H2O.

Other than that you could try encoding your categorical data as numeric. There are multiple approaches to this. One example is mean encoding - see this answer for details. For implementation see Category Encoders. BTW if your task is totally unsupervised (you don't have any target) you can choose any continuous feature you have for mean encoding (so you can produce many columns from each categorical column).


regarding lasso.score in lasso modeling using scikit-learn I once saw the following code segment of using lasso model based on scikit-learn
lasso = LassoCV(normalize = True, max_iter=2000)
lasso.fit(X_train, y_train);
lasso.score(X_test, y_test)

y_pred = lasso.predict(X_test)

I can understand lasso.fit and lasso_predict, but what does lasso.score generally offer? According to the scikit-learn, it Returns the coefficient of determination R^2 of the prediction. I am not quite clear how to use this kind of information?

R^2 is a statistical measure of how close the data are to the fitted regression line. It does this by seeing percentage of the variance of dependent varible that's explained by independent variable.
To know more about R^2 score refer this video.
So, basically it is a metric to see how well model fits the data but it is not adequate.
Refer this discussion on why R^2 score is not good for Lasso.


How discriminator loss generated? The images generated by generator has no labels, then how do Discriminator loss is generated on the basis of classification of generator generated images.

We already know that the images from the generator are fake and those from our training set are real. So, we assign them labels manually, real images as $0$ and fake ones as $1$, which gives us labelled data. Check the definition of loss function in any implementation of a GAN and you'll find that we assign the labels ourselves. For example, take a look here.


Why does it not need to set test group when using 'rank:pairwise' in xgboost? I'm new for learning-to-rank. I'm trying to learn the Learning to rank example provided by xgboost. I found that the core code is as follows in rank.py.

train_dmatrix = DMatrix(x_train, y_train)
valid_dmatrix = DMatrix(x_valid, y_valid)
test_dmatrix = DMatrix(x_test)

train_dmatrix.set_group(group_train)
valid_dmatrix.set_group(group_valid)

params = {'objective': 'rank:pairwise', 'eta': 0.1, 'gamma': 1.0,
               'min_child_weight': 0.1, 'max_depth': 6}
xgb_model = xgb.train(params, train_dmatrix, num_boost_round=4,
                           evals=[(valid_dmatrix, 'validation')])
pred = xgb_model.predict(test_dmatrix)

Group data is used in both training and validation sets. But test set prediction does not use group data. I also looked at some explanations to introduce model output such as What is the output of XGboost using 'rank:pairwise'?.

  Actually, in Learning to Rank field, we are trying to predict the
  relative score for each document to a specific query.

My understanding is that if the test set does not have group data, no query is specified. How does the model output the relative score to the specified query?

And I've tried adding test_dmatrix.set_group(group_test). The output results of the two methods are in good agreement like:

[ 1.3535978  -2.9462705   0.86084974 ... -0.23594362  0.712791
 -1.633297  ]

So my question as follows:

Why does it not need to set test group when using 'rank:pairwise' in xgboost?
How can I get label to the specified group query based on the forecasting score results?

Can anybody explain it to me? Thanks in advance.

The output is a score that can be used to rank the samples, and the point in this sort of ranking problem is that you'll only care about ranking samples within the same group (which you think of as being results from a given query).

But that can be safely left to you on the testing set.  (Indeed, you might as well only run the prediction for each group separately.  You might think about the output in your case as assuming that the test set is all from a single query.)  For scoring on the test set, it might matter what the specified groups are, but not for just making predictions.

For training, the group data is needed so the algorithm knows not to calibrate the rankings for intergroup comparisons.

See also:
How fit pairwise ranking models in xgBoost?
https://github.com/dmlc/xgboost/blob/master/doc/tutorials/input_format.rst#group-input-format


Is there a cost associated with converting Koalas dataframe to Spark dataframe? I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated with switching back and forth between Koalas and PySpark dataframes?

#convert to pyspark dataframe
df.to_spark()

#convert to kolas frame
koalas_df = ks.DataFrame(df)

Edit: With cost I mean, does it ks.Dataframe(ks) create additional overhead? For example, toPandas() results in the collection of all records in the DataFrame to the driver program. Therefore we can only do toPandas() on a small subset of data.

Since I am switching between Koalas and Spark I am wondering if there is any such overhead or if Koalas "interprets" Spark dataframes without collecting records on the driver. At the moment I am working with a small subset of the data, but I am interested in any drawbacks when using larger amounts.

As you said, since the Koalas is aiming for processing the big data, there is no such overhead like collecting data into a single partition when ks.DataFrame(df).

However, the overheads are occurred when creating a default columns for creating the _InternalFrame which internally manages the metadata between pandas and PySpark.

Koalas is internally using immutable frame named _InternalFrame, so you can refer the /koalas/databricks/koalas/internal.py if you want to more detail.

Here is a short code examples from internal.py that creating the default index if given Spark DataFrame has no index information.

https://github.com/databricks/koalas/blob/a42af49c55c3b4cc39c62463c0bed186e7ff9f08/databricks/koalas/internal.py#L478-L491

        if index_map is None:
            assert not any(SPARK_INDEX_NAME_PATTERN.match(name) for name in spark_frame.columns), (
                "Index columns should not appear in columns of the Spark DataFrame. Avoid "
                "index column names [%s]." % SPARK_INDEX_NAME_PATTERN
            )

            # Create default index.
            spark_frame = _InternalFrame.attach_default_index(spark_frame)
            index_map = OrderedDict({SPARK_DEFAULT_INDEX_NAME: None})

        if NATURAL_ORDER_COLUMN_NAME not in spark_frame.columns:
            spark_frame = spark_frame.withColumn(
                NATURAL_ORDER_COLUMN_NAME, F.monotonically_increasing_id()
            )


Which is the best method for Neural Network Layers in Keras In keras we can create neural network layers in many ways.
1. Sequential API.
for example
model=sequential()

2. Functional
for example
x1=Input(shape=(2,)
x2=Dense(2)(x1)

3. Subclassing
for example
class Mymodel(keras.layers.Layers):
    def __init__(self,output):

My Question is:
1. Whether which one to use and their pros and cons.

The sequential API is the simplest.  It allows you to declare a sequence of layers with an single input and a single output.  Naturally, this API is a good choice for sequential networks - those where data flows through each layer in sequence.
Pros: straightforward to read and write.
Cons: Difficult to accommodate multiple inputs/outputs.  Cannot define arbitrary acyclic networks.  No layer reuse.
The functional API allows you more freedom in how you connect inputs and outputs of different layers.  You can route the output of any layer to any other layer, which allows you to define arbitrary acyclic graphs.  Use this API when you need a network architecture which is not sequential.
Pros: More flexibility.  Can define arbitrary networks with multiple inputs/outputs, residual layers, etc.
Cons: More effort to read/write.  Higher potential for bugs (since you have to connect layers manually)
Subclassing is actually a different mechanism than the sequential or functional API.  The APIs above are used to build custom networks by linking together pre-existing layers.
By contrast, subclassing keras.layers.Layer is used to build your own custom layers.  This is useful if you are doing cutting-edge research or if you are trying to implement some technique from the literature that is not part of the standard Keras library.  After you have created a custom layer, you can use it with either the sequential or functional APIs to include your layer in a network.
Pros: Ultimate flexibility - you can make your layer do whatever you like.
Cons: Massive effort compared to using pre-built layers.


What's the math for real world back-propagation? Considering a simple ANN:

$$x \rightarrow
f=(U_{m\times n}x^T)^T \rightarrow
g = g(f) \rightarrow
h = (V_{p \times m}g^T)^T \rightarrow
L = L(h,y)
$$

where $x\in\mathbb{R}^n$, $U$ and $V$ are matrices, $g$ is the point-wise sigmoid function, $L$ returns a real number indicating the loss by comparing the output $h$ with target $y$, and finally $\rightarrow$ represents data flow.

To minimize $L$ over $U$ and $V$ using gradient descent, we need to know $\frac{\partial L}{\partial U_{ij}}$ and $\frac{\partial L}{\partial V_{ij}}$, I know two ways to do this:

do the differentiation point wise, and having a hard time figuring out how to vectorize it
flatten $U$ and $V$ into a row vector, and use multivariate calculus (takes a vector, yields a vector) to do the differentiation

For the purpose of tutorial or illustration, the above two methods might be suffice, but say if you really want to implement back-prop by hand in the real world, what math will you use to do the derivative?
I mean, is there a branch, or method in meth, that teaches you how to take derivative of vector-valued function of matrices?

There is Matrix Calculus, (and I would recommend the very useful Matrix Cookbook as a bookmark to keep), but for the most part, when it comes to derivatives, it just boils down to pointwise differentiation and keeping your dimensionalities in check.

You might also want to look up Autodifferentiation. This is sort of a generalisation of the Chain Rule, such that it's possible to decompose any composite function, i.e. $a(x) = f(g(x))$, and calculate the gradient of the loss with respect to $g$ as a function of the gradient of the loss with respect to $f$.

This means that for every operation in your neural network, you can give it the gradient of the operation that "consumes" it, and it'll calculate its own gradient and propagate the error backwards (hence back-propagation)


Large scale Autoencoder for friendship recommendation I have a user friendship graph for about 30 Million users. I am trying to use an auto encoder [30 Million, 512, 512, 1024, Dropout(0.3), 512, 512, 30 Million]. But I am not learning anything.
Has anyone used Autoencoder for Friendship recommendation on a large scale graph?

Autoencoder will not predict potential friendship. Autoencoders learn a lossy compression of the data. A lossy compression of existing friendship relationships will not predict new relationships.


Bias-variance tradeoff in practice (CNN) I first trained a CNN on my dataset and got a loss plot that looks somewhat like this:

Orange is training loss, blue is dev loss. As you can see, the training loss is lower than the dev loss, so I figured: I have (reasonably) low bias and high variance, which means I'm overfitting, so I should add some regularization: dropout, L2 regularization and data augmentation. After that, I get a plot like this:

Now we see that the variance has decreased and the bias has increased. The model is overfitting less, is this correct? However, I would actually select the first model because it has lower validation loss.

My question is: in most literature, for the bias-variance tradeoff, they show the validation loss going up, but in my experiments this is not the case, so are these models actually overfitting? Are you overfitting as soon as training loss goes below validation loss, or only if validation loss goes back up? And is it okay to choose a model with high variance if the validation loss is lower?

I found this answer on a similar question, but what if your problem is so complex that you can't find an architecture that can overfit and then properly regularize the architecture? I can find an architecture that gets a training loss close(r) to zero, but then I can't really add enough dropout to make sure the variance is low. Also if I add augmentation my validation loss also goes up. Finally the answer confuses me, the answerer is talking about variance on the training set? But isn't bias always related to training loss and variance to dev loss?

Or am I just misinterpreting information and should I plot in function of dataset size rather than number of epochs to find out if I am overfitting?

Normally, the training loss is lower than the validation one. This does not indicate any overfitting. Indeed, it is even suspicious when you training loss is higher than the validation loss. From other hand, worsening of the validation accuracy while improving on the train set definitely tells you that you overfits.

Generally speaking, overfitting means bad generalization, memorization of the training set rather than learning a generic concepts behind the data. Besides the metrics during the training you can find it out by trying your model on external datasets from a similar but not the same domain/distribution. Very poor accuracy will indicate the overfitting that might be hidden by a validation set very similar to the training set.


Why is the accuracy of a LinearSVC not the same as the SDGClassifier? I'm fine tuning parameters for a linear support vector machine. There are multiple ways to do it, but I wanted to compare LinearSVC and SDGClassifier in terms of time. I expected the accuracy score to be the same but, even after fine tuning with GridSearchCV, the score of the LinearSVC is lower. I tried changing up parameters many times, but the maximum with LinearSVC I can get is 41.176 versus 41.503 of SDGClassifier. Why?
The code:
    class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import SVC, LinearSVC
    from sklearn.pipeline import Pipeline

    self.pipeline = Pipeline(
        [
            ('vect', CountVectorizer()),
            ('tfidf', TfidfTransformer()),
            #('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC( loss='hinge',
              penalty='l2', max_iter = 10,

             #SGDClassifier(
                #loss='hinge',
               # penalty='l2',
              #alpha=1e-3,
              #  random_state=42,
             #max_iter=100,
                #learning_rate = 'optimal',
                #tol=None

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    from sklearn.linear_model import SGDClassifier
    from sklearn import svm
    from sklearn import preprocessing
    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.svm import LinearSVC
    train_df = self.read_data(train_file, lower_case)
    param_range = [0.001, 0.01, 0.1, 1, 10, 100]
    parameters = {
        # 'vect__ngram_range': [(1, 1), (1, 2)],
        'tfidf__use_idf': (True, False),
        #'clf__alpha': [0.0001, 0.001, 0.01, 1, 10, 100],
        'clf__max_iter': [10, 100, 1000],
        'clf__tol': [0, 0.0001, 0.001, 0.01],
        'clf__loss':['hinge'],
        'clf__penalty': ['l2'],
        'clf__C': param_range

    }

    parameters1 = {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['linear'], 'clf__tol' : [0, 0.01]

                  }
    lr = LinearSVC()
    print(lr.get_params().keys())
    gs_clf = GridSearchCV(self.pipeline, parameters, cv=5, n_jobs=-1)
    gs_clf = gs_clf.fit(train_df['text'], train_df['truth'])
    print(gs_clf.best_score_)
    for param_name in sorted(parameters.keys()):
        print("%s: %r" % (param_name, gs_clf.best_params_[param_name]))

    # estimator_svm.best_score

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

When used with loss="hinge" The SGDClassifier gives a LinearSVM, so they should be the same. This is matter of choosing the same hyperparameters for both. Can you check that you using the exact same parameters?
As a side note (I don't know your dataset), 41.1 and 41.5 looks pretty similar, this also might be about splitting the training/testing data exactly the same and stopping the training at the correct time.


WEKA Random Forest J48 Attribute Importance I have been using WEKA to classify very long duration audio recordings. The best performing classifiers have been Random Forest and J48. The attributes used to classify the audio are acoustic indices. This process of generating these indices is quite resource intensive.

I would like to determine the importance of the various attributes. Is there a way for these classifiers to report this?

I see J48 produces a decision tree, is it safe to day the attributes used at the root of the tree are most important?

If you must use WEKA, I would suggest looking in the Attribute Selection panel. In this panel, you can rank attributes by information gain, as well as look at which subsets of attributes perform best. To my knowledge, you can't get this information directly from the classifier in the WEKA Explorer, but it is quite easy to get in other frameworks such as scikit-learn: see this example in the docs.


Reduction of multiple answers to single variable The questionnaire for the data is here

The first question takes multiple entry for the same question, I want to reduce this to a single variable. How do I do it?

The clean data is available here.
NB: The Column CompuPlat has missing values.

part of dataset

CMFam CMHobb  CMNone  CMOther CMPol   CMProf  CMRel
0   0   1   0   0   0   0
0   0   0   0   0   0   0
1   1   0   0   0   1   0
0   0   0   1   0   0   0
0   0   0   0   1   1   0
1   0   0   0   0   1   1
Community Membership_Family
Community Membership_Hobbies
Community Membership_None
Community Membership_Other
Community Membership_Political
Community Membership_Professional
Community Membership_Religious
Community Membership_Support

I want to club all of them in a variable CM

The variable represents the answer to the first question.

One straightforward way is to allow for all possible categories in this variable. For example, if there are 5 options in this answer, you will have to treat it as a categorical variable with 2^5 = 32 categories.

However, the number of categories increase exponentially with the number of options (check boxes) provided for the answer. In that case, it might be better to restrict the number of categories to, for example, 5. This can be done by leaving the top 4 choices/ options (by count) as they are and treating every other choice as "other".


Can I apply Clustering algorithms to the result of Manifold Visualization Methods? Some methods related to manifold-learning are commonly stated as good-for-visualization, such as T-SNE and self-organizing-maps (SOM).

I understand that when referring specifically to "visualization" means that the non-linear dimensionality reduction can provide good insights of data in its low-dimensional projection, but that most commonly this low-dimensional projection cannot be used in machine learning algorithms, since some of information of the high-dimensional structure is lost (roughly).

However, and here the question, If "clusters" are being observed in the visualization is it acceptable to apply a clustering algorithm to the low-dimensional transformed data and analyze the clusters or groups separately?

For example, I'm applying T-SNE to rather-high dimensional data (40 features) and obtaining this representation:

Disregarding the colors you observe in the picture, I would like to apply a clustering algorithm and separate the data from the found clusters (let's say 6 or 7 clusters), and then analyze the characteristics of each cluster using the high-dimensional representation of each point.

This is in synthesis: using the low dimensional to find clusters, and analyze (exploration) each cluster separately using the high dimensional representation. If I'm not able to do this, I don't see the actual point of visualizing in the low-dimensional space, in a practical sense.

I understand T-SNE preserves well local structures and less accurately global structures, is this a drawback to do why I want? Is this low-dim clustering approach more suitable for other manifold learning methods?

EDIT:
Probably a more direct way of asking this is: Can I use observed clusters in low dimensional representation to label or tag examples, and use these labels for discrimination using the original high dimensional representation?

You can do anything you want in the low dimensional space, and can try to validate as well. By clustering the above, you are in effect assigning features/tags to your data points in higher dimensions. Remember, tSNE tries to preserve distances, so that points in high dimensions will remain close to each other in low dimensions.

With that in mind, don't forget that no two instances of tSNE will be the same, which means that your clustering centers will be different each time you run tSNE.


Saving a different model for every epoch Keras As part of a project I am currently running a CNN for classifying images. Right now I am training with 50 epochs. I have noticed that after around 20-30 epochs I reach the highest accuracy and then the loss starts to increase significantly. I thought about using Early Stopping but my model loss has a lot of zig-zagging so I think it would be difficult to find the right patience argument to use. I think it would be easier to just save a model for every epoch so I can manually select the one with the highest accuracy. Is this possible in Keras?

Saving many models definitely possible.

Please find below code snippet:

# serialize model to JSON
model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("model.h5")
print("Saved model to disk")

# later...

# load json and create model
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model.h5")
print("Loaded model from disk")

For detailed reference:Save&Load Keras model

Note: Please append epoch value with the file name, so that you have all different trained model.s


How to automate ANOVA in Python I am at the dimensionality reduction phase of my model.  I have a list of categorical columns and I want to find the correlation between each column and my continuous SalePrice column.  Below is the list of column names:

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

Because its categorical vs continuous, I've read that ANOVA is the best way to go but I have never used it before and couldn't find a concise implementation of it in Python.  I want to loop through and output the correlation between each element in the list and the SalePrice column.

I am not sure ANOVA is the best and easiest way to find correlation between these categorical features and your target. You may see this great post where they propose many other methods along with ANOVA. If you persist to use ANOVA test or Kruskal-Wallis H Test, you need to know how it works to give you that notion of correlation (variation of variance among groups of categoricals). It is nicely explained in that post:

  ANOVA estimates the variance of the continuous variable that can be
  explained through the categorical variable. One need to group the
  continuous variable using the categorical variable, measure the
  variance in each group and comparing it to the overall variance of the
  continuous variable. If the variance after grouping falls down
  significantly, it means that the categorical variable can explain most
  of the variance of the continuous variable and so the two variables
  likely have a strong association. If the variables have no
  correlation, then the variance in the groups is expected to be similar
  to the original variance.

Once you understand how it works, implementing it and automating it is not difficult. In fact scipy and statsmodels have ANOVA. Check this post out, where they demonstrate in details how to perform ANOVA test on an actual dataset and estimate the correlation between categorical variable and continuous target. It is just a matter of putting these pieces together and change a bit to make it work for your own dataframe.


Coefficient of determination is close to 1 but the value of RMSE is large. What does it mean? I am working with the DecisionTreeRegressor and trying to understand how well the data fits the model. I calculated both RMSE and coefficient of determination. At a certain depth,  coefficient of determination has the value equal to 0.9918744073066561 but the value of RMSE is equal to 75.0025. I cannot understand this. The value of RMSE is quite large but the value of the coefficient of determination is close to 1.0. What does it really mean? Is the model/fit good enough?

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.metrics import mean_squared_error

def rmse(y_true, y_pred):
    return math.sqrt(mean_squared_error(y_true, y_pred))

sample_depth = np.linspace(1,40, num = 40, dtype=int)
dt_score_list = []
for index, depth in enumerate(sample_depth):
    boosted_regressor = AdaBoostRegressor(DecisionTreeRegressor(max_depth=depth), random_state=1)
    boosted_regressor.fit(X_train, y_train)
    dt_score_list.append(boosted_regressor.score(X_test, y_test))
    print(boosted_regressor.score(X_test, y_test), rmse(y_test, boosted_regressor.predict(X_test)), depth)

Compared to $R^2$ $RMSE$ is dependent on the variance of values. Even if $R^2$ is close to 1 but the standard deviation is high the value of $RMSE$ will also be high.
$$RMSE=\sigma_{y_{true}}\sqrt{1-R^2}$$
where $\sigma$ is a standard deviation

As can be seen from the formula $RMSE$ and $R^2$ have a strict mathematical relationship that shows that $RMSE$ changes according to standard deviation ($\sigma$) and the higher the standard deviation of the true values the higher the value of $RMSE$.

$RMSE$ shows the deviation of the error in units of the values, while $R^2$ shows the share of variance that is explained by the model. You can explain 99% of the variance ($R^2$) but your numbers are in millions of dollars and your error still varies in thousands of dollars ($RMSE$) Whether thousands of dollars is ok for your error or not depends on the case. If you predict profit plus or minus 1000\$ is ok. If you predict revenue then plus or minus 1000\$ could be a difference between a profitable and a non-profitable company.

Proof of the formula:

According to sklearn the formula for $R^2$ is

$$R^2 = 1 - \frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}$$

RMSE is
\begin{align}
RMSE &= {\sqrt{\frac1n\sum_i^n(y_{true} - y_{pred})^2}}
\\RMSE^2 &=\frac1n\sum_i^n(y_{true} - y_{pred})^2
\\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\frac1n\sum_i^n(y_{true} - y_{pred})^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=1-\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}
\\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=R^2\\1-\frac{RMSE^2}{\sigma_{y_{true}}^2} &=R^2
\\RMSE&=\sigma_{y_{true}}\sqrt{1-R^2}
\end{align}

where $\sigma$ is a standard deviation and $\overline{y}$ is an average


Available data about 'likers' as a page on Facebook I am hoping to model the characteristics of the users of a specific page on Facebook, which has roughly 2 million likes. I have been looking at the Facebook SDK/API, but I can't really see if what I would like to do is possible. It seems that the users share quite different amounts of data so I probably discard a lot of users and only use the ones with a quite open public profile. I would like to have the following data:

1) See the individuals that have 'liked' the page.

2) See the list of friends for each person that have 'liked' the page.

3) See gender for each person (optional)

4) See other pages that each person has liked (optional)

Could anyone tell me if it is possible to get this data? As mentioned earlier it is okay if I discard data for users that don't like to share this data.

Want to wish you good luck. Some time ago faced with the same problem, but didn't find any satisfying solution.
First of all, there is no way to get list of users, who "liked" a particular page. Even, if you are an administrator of this page (I was). One only can get list of last 3 or 5 hundred users.

Friendships data for most of the users is also inaccessible. Looks like gender is the only thing from your list, that you can get.

Data about pages, that exact user "likes", should be available (as it's written in docs), but in reality, through API you can collect something only for friends and FoF. Even though this data is available through web interface. So the only way is to try dirty trick with parsing and scraping (but remember, that I didn't advise it ;) ).


Automatically categorize parts of a piece of writing Suppose I had a piece of writing. The document contains aspects like questions, assertions, examples, and explanations. Is it possible to use Natural Language Processing to categorize each sentence of the writing into such aspects, thus creating a code like 1.1:assertion, 1.2:explanation, for instance, meaning paragraph 1, sentence 1 assertion, and paragraph 1, sentence 2, explanation?

If that is possible, I would appreciate if someone would share their knowledge about known algorithms for performing this task.

As far as I know there's no specific task for this, it's general text classification.  It's also related to text segmentation. There are certainly existing systems for similar tasks, but probably specific to a certain type of data.

In general this would be a supervised process, you would need a sample containing many documents in which the parts are pre-annotated. I think sequence labeling algorithms would be the standard approach.


Is there any work done on reconfigurable convolutional neural networks? Convolutional Neural networks are used in supervised learning meaning models are always "set in stone" after training (architecture and paramters) so this might not even be possible, but is there any research done on playing around with the data paths, model size (number of layers) and architecture during runtime, i.e after training is done, for instance creating a model that can be modified online to use less or more recources, skip layers or use portions of the model.

There is some work done recently on creating flexible frameworks for training and designing networks, but that's always "offline". There is also an interesting paper on training two models, "big" and "little" for the same application and using an accuracy/power trade-off policy to deploy one of the two.

It's a great question and one I have thought about a lot too.

Are you are talking about a network structure that evolves/mutates during training, or one that simply chooses whether or not to use all available resources? While I would never give a straight-up No to such a questions (especially the former), I can't say I have really seen it in publications. The cloest thing that comes to mind is a recent paper from DeepMind, which utilises a combination of "model-free and model-based aspects"; however, this more high-level than network architectures specifically, as you refer to them.

Perhaps you are considering a model, which is forced to choose between one resource and another? An example of the latter would be whether or not to use a skip connection or a dense layer (assuming shapes all match). Yann LeCun says (the video linked below) that such paths in a network not only provide more flexibility/complexity, but additionally ease the optimisation problem and provide a source of regularisation.

In any case, I am sure people are doing that, making use of dynamic networks created using PyTorch, the control flow operators in Tensorflow, the new tensorflow eager library (which aims to replicate dynamic networks like PyTorch produces).

One could also argue that providing a monolithic network to a model affords it the flexibility to use the resources (i.e. the weights) that it chooses. It doesn't have to use them all. Just as batch normslisation layers allow the model to parametrically adjust the extent to which they are used.

There are great discussions on what is required in the long run, heading toward Artificial General Intelligence (a recent overview in the context of RL part1 & part2. If we should be training models end-to-end, how much information should we give an algorithm learning to play a game and

The tools that would allow people to do that, like you mention, are coming along, but I haven't yet seen a paper that explicitly outlines how that is done. It would be great is someone else gives you a more satisfying answer! ;-)


Principal Component Analysis, Eigenvectors lying in the span of the observed data points? I have been reading several papers and articles related to Principal Component Analysis (PCA) and in some of them, there is one step which is quite unclear to me (in particular (3) in [Schölkopf 1996]).

Let me reproduce their reasoning below.

Consider the centered data set $D = \{\textbf{x}_k\}_{k=1}^M$ with $\textbf{x}_k \in \textbf{R}^N$ and $ \sum_{k=1}^M \textbf{x}_k = 0$. PCA diagonalizes the (sample) covariance matrix

$$
C = \frac{1}{M} \sum_{j=1}^M \textbf{x}_j \textbf{x}_j^T. \tag{1}
$$

To do this we find the solution to the eigenvector equation

$$
\lambda \textbf{v} = C \textbf{v} \tag{2}
$$

for eigenvalues $\lambda \geq 0$ and eigenvectors $\textbf{v} \in \textbf{R}^N\backslash \{{0}\}$. As

$$
\lambda \textbf{v} = C \textbf{v} = \frac{1}{M} \sum_{j=1}^M (\textbf{x}_j^T \textbf{v}) \textbf{x}_j, \tag{3}
$$

all solutions $\textbf{v}$ with $\lambda \neq 0$ must lie in the span of $\textbf{x}_1, \dots, \textbf{x}_M$, hence (2) is equivalent to

$$
\lambda(\textbf{x}_k^T \textbf{v}) = \textbf{x}_k^T C \textbf{v}, \qquad \text{for } k = 1, \dots, M \tag{4}
$$

In (4), doesn't $\lambda(\textbf{x}^T \textbf{v}) = \textbf{x}^T C \textbf{v}$ hold for $\textbf{any}$ value of $\textbf{x}$? Why does (4) only hold when $\textbf{x} \in D$? I do not understand how their end up with (4).

Thanks.

The statement says, that (2) and (4) are equal. That means (2)$\Rightarrow$(4) and (4)$\Rightarrow$(2). The first implication is trivial, as you correctly pointed out.
$$\lambda v=Cv$$
implies
$$\lambda x^Tv=x^TCv$$
for all $v$, not just those from $D$. But the second implication is a bit more tricky, but that is what the proof is about. It tells you, that if you want to check, whether a vector is an eigenvector of $C$, you don't have to check if (2) is satisfied. It tells you, that when (4) is satisfied, (2) is satisfies as well.

Imagine you have 2 points in 3D space. Those 2 points are
$$x_1=(-1,-1,0)$$
$$x_2=(1,1,0)$$
(please excuse me for not making this "dataset" centered) Note, that both points lie in the $xy$ plane. Now the correlation matrix is
$$C=\begin{bmatrix}
       1/2 & -1/2 & 0           \\[0.3em]
       -1/2 & 1/2           & 0 \\[0.3em]
       0           & 0 & 0
     \end{bmatrix}$$
Now you want to know, whether $v=[1\ -1\ 0]^T$ is an eigenvector with eigenvalue 1. The statement tells you, that you can either just check, if (2) is satisfied (3 equations) or if
$$\frac{1}{2}x_1^Tv=x_1^TCv=0$$
and
$$\frac{1}{2}x_2^Tv=x_2^TCv=0$$
which are only two equations.


Is it ok to concat train and test sets to for fit_transform before splitting them? I'm currently trying to use fancyimpute KNN to impute some data separate train/test sets. However, I'm running into the problem of not being able to do fit_transform on train data then using only transform on the test set because it appears that:

  transform not implemented! This imputation algorithm likely doesn't
  support inductive mode. Only fit_transform is supported at this time.

Is it alright to concat the two sets for imputing before splitting them after? Or is my approach even correct by calling only fit on the train set before applying that to impute the test set? Should I be calling separate fit_transform on both imputation?

Thanks!

They way you tried to do it is correct and no, you should not concatenate your train and test set to get around this.

Unfortunately it seems that the transformer you are trying to use just does not implement the functionality you need. From the docs:

  There are some imputation algorithms that are inductive, meaning they
  can be applied to new data after a call to solver.fit or
  solver.fit_transform. Currently only IterativeImputer supports the
  full scikit-learn API: fit, fit_transform, and transform, but we are
  actively looking for contributions that extend other imputers to
  support induction. At least the KNN and SimpleFill imputers can be
  extended in a straightforward manner.

Getting around this by concatenating your train and test set together will cause a data leak. This happens when your model gets information from the test set during training time, which invalidates the test evaluation.

And data leaks aside, what imputer will you use when you want to make predictions on new data?

As I see it your options now are:

Find another KNN imputation implementation that supports transform on new data
Use another inductive imputer
Implement KNN imputation yourself


Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.) I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/  . I would like to have the values described in the bar charts numerically into, say an Excel or pandas file. Is there a way of scraping or using an API that would turn the bar chart values into number values associated to each country? I looked at similar questions but the closest I found was transforming categorical data given in a file format and did not involve scraping/APIs to change categorical into numerical.
Thanks.

The data is available in json format through this link, which contains the data for all countries that can be selected from the list. I found this link by going to the link you provided,  opening the Network tab in the Chrome developer tools and reloading the page to see  all the resources that are loaded by the webpage.


Does mini-batch gradient descent nullify the effect of stratification on the training data set? In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset.
Mini-batch gradient descent uses random shuffling to ensure randomness in the mini-batches.
My doubt is- Why should we implement stratified shuffle on our dataset if it is going to be shuffled in a random manner later during training?

It doesn't, the workflow when training a model is like that:

Create 10 evenly distributed splits from the dataset using stratified shuffle
train set = 8 splits;
validation set = 1 split;
test set = 1 split
Shuffle the train set and the validation set and create minibatches from them
Train for one epoch using the batches
Repeat from step 3 until all epochs are over
Evaluate the model using the test set

If we skip the stratified shuffling in step 1 the classes of the train set, validation set and test set wont be evenly distributed.
If we skip the shuffling before each epoch in step 3 the mini-batches in each epoch will be the same.
The proportions of the train set, validation set and test set can of course vary.


How to group every data point with HDBSCAN to some group to have no noise? TASK

I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it

GOAL

The goal is when users come on our site and I can show similar products to what they viewing.

QUESTION

How to get all data point to be part of a group, so the goal is to not to have any noise?

CODE
clusterer = hdbscan.HDBSCAN(min_cluster_size=10,#smallest collection of data points you consider a cluster
                            min_samples=1 #LARGER this value -> more points will be declared as NOISE
                           ).fit(data)

color_palette = sns.color_palette('Paired', 2000)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusterer.labels_]
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, clusterer.probabilities_)]
plt.scatter(*projection.T, s=20, linewidth=0, c=cluster_member_colors, alpha=0.25)

labels = clusterer.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print('Estimated number of clusters: %d' % n_clusters_)

DBSCAN will always mark noisy points according to epsilon and min_samples parameters, so there is no way to avoid that unless you have very compact and "well defined" clusters, what seems unlikely.The short answers will be to use another clustering algorithm such as gaussian mixture, k-means, or Birch
If your problem really needs you to use DBSCAN, you can try performing a quantile tranformation (uniform) prior clustering
Example:
scaler = Pipeline([("imputer", SimpleImputer(strategy= "constant", fill_value= 0)),("transformer", QuantileTransformer(output_distribution = "uniform"))])

model = Pipeline([("scaler", scaler), ("cluster",HDBSCAN(min_cluster_size=10, min_samples=1))]).fit(data)


Why does not log transformation make the data normalized? Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following codes

  import numpy as np

  import pandas as pd

  from sklearn.preprocessing import MinMaxScaler

  df=pd.read_csv("Data.csv")

  scaler=MinMaxScaler(feature_range=(0,1))

  df['vBMD (mgHA/cm3)']=scaler.fit_transform(np.array(df['vBMD
  (mgHA/cm3)']).reshape(-1,1))

  df['vBMD (mgHA/cm3)']=np.log(np.array(df['vBMD (mgHA/cm3)']))

After the transfromation, I have got the following result.

While I am waiting that the feature will be normalized, its skewness increased. Thus, what am I doing wrong?

Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.

EDIT:

As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.


Are stopwords helpful when using tf-idf features for document classification? I have documents of pure natural language text. Those documents are rather short; e.g. 20 - 200 words. I want to classify them.

A typical representation is a bag of words (BoW). The drawback of BoW features is that some features might always be present / have a high value, simply because they are an important part of the language. Stopwords like the following are examples: is, are, with, the, a, an, ...

One way to deal with that is to simply define this list and remove them, e.g. by looking at the most common words and just deciding which of them don't carry meaning for the given task. Basically by gut feeling.

Another way is TF-IDF features. They weight the words by how often they occur in the training set overall vs. how often they occur in the specific document. This way, even words which might not directly carry meaningful information might be valuable.

The last part is my question: Should I remove stopwords when I use TF-IDF features? Are there any publications on this topic? (I'm pretty sure I'm not the first one to wonder about this question)

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it.

However, working without the stopwords in your documents will make the number of features smaller, which might have a slight computational advantage.


Seaborn Violin plot error No loop for unfunc add I'm trying to plot a violin plot with a split based on Sex ( like in the fourth example in the doccumentation but with Sex)

I can produce a categorical scatter plot and split it by Sex. However, when i attempt the same but as a violin plot; it throws an error.

Traceback (most recent call last):

  File "<ipython-input-868-0599b976fd6c>", line 1, in <module>
    sns.catplot(x="Batch", y="Age", hue = 'Sex', data = ages,  kind='violin')

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 3755, in catplot
    g.map_dataframe(plot_func, x, y, hue, **plot_kws)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py", line 820, in map_dataframe
    self._facet_plot(func, ax, args, kwargs)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py", line 838, in _facet_plot
    func(*plot_args, **plot_kwargs)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 2387, in violinplot
    color, palette, saturation)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 564, in __init__
    self.estimate_densities(bw, cut, scale, scale_hue, gridsize)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 679, in estimate_densities
    kde, bw_used = self.fit_kde(kde_data, bw)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/seaborn/categorical.py", line 719, in fit_kde
    kde = stats.gaussian_kde(x)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 208, in __init__
    self.set_bandwidth(bw_method=bw_method)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 540, in set_bandwidth
    self._compute_covariance()

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/scipy/stats/kde.py", line 551, in _compute_covariance
    aweights=self.weights))

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 2427, in cov
    avg, w_sum = average(X, axis=1, weights=w, returned=True)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 419, in average
    scl = wgt.sum(axis=axis, dtype=result_dtype)

  File "/home/tasty/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py", line 36, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial)

TypeError: No loop matching the specified signature and casting
was found for ufunc add

My code is:

>>> print(ages.head())
            Age     Sex  Batch
PassengerId
852          74    male  Train
86           33  female  Train
161          44    male  Train
812          39    male  Train
837          21    male  Train

>>> sns.catplot(x="Batch", y="Age", hue = 'Sex', data = ages,  kind='violin')

Removing the kind argument produces the following scatterplot:

How do I get rid of the error to display the data as a violin plot?

Thanks in advance

edit:
Seaborn version: 0.9.0
Numpy version: 1.16.2
Python version: 3.7.3

I had tried the suggestion from @foxthatruns's answer to no avail. I found that changing my numeric column to float64 solved the problem (reference).

df['my_column']=df['my_column'].astype('float64')

This was done with Python 3.7, seaborn 0.9.0, numpy 1.16.4.


I still don't know how deconvolution works after watching CS231 lecture, I need help https://www.youtube.com/watch?v=ByjaPdWXKJ4 This is the time-stamped video of "deconvolution". I can understand normal convolution but not so much with upsampling convolution. In the video he explained that you plop down the filter and use each individual scalar as the weight to apply to each value in the filter. I am having a hard time understanding how he got an output shape of 4x4? Is there a special formula for calculating the output for "upconvolution"? I am also confused on the padding and how it affects upconvolution?

My crappy paint image

OP here:

https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d
This has helped me understand how deconvolutional layer or transposed convolution works! Note to self don't google deconvolution or upconvolution google transposed convolution instead.

Update 2: http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html

It turns out "deconvolution" is just convolution but with different arithmetics. You can take the transpose or add enough padding so that

1) You can upsample instead of downsampling.

2) Keep the previously linked relation, what I mean is you need to make sure each upsampled filter still contains a symbolic link with the smaller input.


predict_generator gives "Found 0 images belonging to 0 classes." I have trained my model on the very big dataset (approx 1200000 train images, and 15000 classes) using Keras' flow_from_directory method. I have successfully trained my model using fit_generator. But, now I am facing the problem for predict_generator.

I got the following error while executing predict_generator:

  "Found 0 images belonging to 0 classes."

However, I have converted my train image dataset into the subdirectory based on class labels. But, for test dataset, I don't have any labels.
So how can I overcome this issue?

Make a folder Structure.

For instance:

Training_Data
       |
        ---Cat
       |
       -----Dog

Because keras expect your folder in catag


What should I start with first? Python for beginners or applied data science? I am new to data science and I would like to go deep into it.
I decided to start working with advanced competitions on kaggle, yet I should refine my knowledge in python and ML.
I am an engineer, with 4 years of programming (basically php vanilla JS, and javascript based frameworks)
I took a coirse on Udemy about ML with python. As I am already a programmer, I know most of the basics, I just need to familiarise with the script and how its written in python.
I am desperate for a help from someone, who can help me with if I should take the Python for everyone or simply just dig in directly into Applies data science with Python from University of Michigan.
I know sometimes I struggle writing a code with python but with a basic google search I can find an answer or figure out one of my own.
I know that I can skip lots of weeks on coursera as I already knows the basics. But still confused.

TLDR: No, it is not required for someone with your background to take "Python for everyone" beforehand.
I have taken the "Applied Data Science with Python Specialization" by UofM and it provided a very soft start in terms of Python for Data Science. Also, see the description of the first course of the specialization "Introduction to Data Science in Python":

This course will introduce the learner to the basics of the python programming environment, including fundamental python programming techniques such as lambdas, reading and manipulating csv files, and the numpy library. The course will introduce data manipulation and cleaning techniques using the popular python pandas data science library and introduce the abstraction of the Series and DataFrame as the central data structures for data analysis, along with tutorials on how to use functions such as groupby, merge, and pivot tables effectively. By the end of this course, students will be able to take tabular data, clean it, manipulate it, and run basic inferential statistical analyses.

As they say, the course walks you through the important libraries, such as Numpy and Pandas, and covers some key methods. Moreover, the second course of the specialization "Applied Plotting, Charting & Data Representation in Python" provides an intro to Matplotlib. So this one is covered too.
Finally, if you would like to go for competitions I highly recommend the course "How to Win a Data Science Competition: Learn from Top Kagglers" (as a next step). It is more advanced than the UofM specialization and provides great insights in the mechanics of and approaches for competitions.


Data scaling before PCA: how to deal with categorical values? I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal with them (basically I created dummy variables). Now, in order to apply PCA I have to scale the data matrix such that I have mean equal to 0.
My question is: does it make sense to normalize also the categorical values (which now are numbers, but they are actually categorical values)? I think it doesn't, but in that case how do I proceed? I do the PCA without scaling these variables?

Thanks!

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation:

  All your variables are numeric: use principal component analysis (prince.PCA)
  You have a contingency table: use correspondence analysis (prince.CA)
  You have more than 2 variables and they are all categorical: use multiple correspondence analysis (prince.MCA)
  You have groups of categorical or numerical variables: use multiple factor analysis (prince.MFA)
  You have both categorical and numerical variables: use factor analysis of mixed data (prince.FAMD)

Check also this question/answers in stats.stackexchange or discussion in researchgate.


Why does Ensemble Averaging actually improve results? Why does ensemble averaging work for neural networks? This is the main idea behind things like dropout.

Consider an example of a hypersurface defined by the following image (white means lowest Cost). We have two networks: yellow and red, each network has 2 weights, and adjusts them to end up in the white portion.

Clearly, if we average them after they were trained we will end up in the middle of the space, where the error is very high.

I think there is a missunderstanding in your question. In your question you imply that you take the average of the weights of the networks, and you should not do this. Instead, what you average are the predictions of different networks. For this reason, if you average two correct predictions the result will be a correct prediction, and the problem you were thinking about does not exist.


Higher frequency of time series benefits We are setting up an experiment for a model that is able to predict the evolution of a time series in different horizons. One of the parameters to decide is the granularity of frequency of our samples (every 1/5/10 mins). My question is: does higher frequency sampling provide better result as a general rule?

You are limited by the Nyquist frequency, which is a theoretical frequency. Let's suppose you want to capture a phenomenon that occurs at $60\,\text{Hz},$ such as standard electrical power from your wall output. The Nyquist frequency says that, theoretically, you should sample at $120\,\text{Hz}$, twice the phenomenon frequency, in order to recreate your signal.

However, in theory there's no difference between theory and practice, but in practice, there is. The usual rule-of-thumb in the data acquisition world is to sample at ten times the frequency of the phenomenon you want to capture. In the example above, therefore, you'd really want to sample at $600\,\text{Hz}$ to get a good picture for your signal.

Higher sampling rates will generally produce a better representation of your phenomenon. The flip side, of course, is that you produce more data, which makes analysis slower. There's definitely a law of diminishing returns going on here, and sampling the living daylights out of your system is unlikely to help you much unless you're in the RF world.


Differentiate between positive and negative clusters I have applied k-means clustering on my dataset of Amazon Alexa reviews.
model = KMeans(n_clusters=2, max_iter=1000, random_state=True, n_init=50).fit(X=word_vectors.vectors.astype('double'))

Now I want to check which cluster is positive and which is negative, can anyone suggest me some way to do that?
Also, is there any way to check is a particular word belongs to which cluster. E.g, the word 'bad' belongs to which cluster - 0 or 1

Maybe you don't have a positive and a negative class.
Your input are word vectors. Unless you trained your word vectors before with explicit positive and negative labels, it is very unlikely that your KMeans learned that difference.
If you used pre-trained word vectors, your KMeans could have learned an arbitrary difference between cluster 0 and cluster 1. Maybe it learned which reviews are from males and which from females, maybe which have the word "parachute" and which don't have the word "parachute", the options are endless.
What you can do, is access which labels your KMeans learned (model.labels_) and filter your input X per cluster. Then, count the occurence of each word in each cluster and order which words happen the most in each of them. This might help you understand the difference between cluster 0 and cluster 1.
Note: if the top words you get are words like: a, the, of, if, etc. Use a stop-word list, or filter those word with a max document frequency threshold.


Is over-fitting a matter of features engineering or a matter of parameters set? I am using sklearn package to make models.

I tried randomly to set some paramater to a sklearn.ensemble.RandomForestClassifier in order to have ideas about parameters to search for a future grid search.

Well, here the results of this attempt:

Accuracy :
Training: 0.9209427371993345 , Test: 0.7035561005518087

Clearly, I know it means the model is over-fitting because it's not able to generalize to new data. I know a cross-validation would be more accurate because it is possible the test-set is unfortunately too different from training-set just by random, but that's not what I am looking for in the current problematic of my topic.
Besides, the two classes I try to target are imbalanced. Class 1 is more present than Class 0.

Class 0: 34% of test set/training set, Class 1: 66% of test set/training set.

So because it's imbalanced I checked about precision and recall metrics:

On test set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.60 |   0.38 |     0.47 |     3326 |
|       1 |      0.73 |   0.87 |     0.80 |    6460  |
+---------+-----------+--------+----------+----------+

On training set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.98 |   0.81 |     0.89 |   31265  |
|       1 |      0.91 |   0.99 |     0.95 |  59492   |
+---------+-----------+--------+----------+----------+

Then, on training set I see both classes are well predicted. Intuitively I think: if it's well predicted on the training set, it means features are good enough to split the two classes. So, it's just a matter of parameters set.
But intuition is not as valuable as good experience is. So I am asking for more experienced people from this community if my intuition is wrong, and if so why it is?

UPDATES:

After checking variables, it shows distribution are the same between the training-set and testing-set through histograms.

A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing.

The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here.

Imo the problem come from your learning process, or more exactly : when does it stops ?

If you are going to explore your parameters space by hand, I would suggest to go look at the parameters of your learner : https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, and for each one of them ask yourself what would be the impact of increasing / decreasing them on the learning capacity of your algorithm.

It may be a bit counterintuitive but you have to actcually limit the capacity of your learner, so that it does not learn too much.

Then you can do some grid search on the main parameters driving the learning / performance.


Can 2 different OOV words get the same vector in FastText? Since FastText sums up the vectors(order is not considered) of an OOV word's subwords, is it possible for two different OOV words to get the same vector ? If so, then can you give an example?

TL;DR
Theoretically it is possible, but it is unlikely.
1) Uncommon subwords
word1 = 'iiii'
word2 = 'jjjj'

word1_subwords = ['<ii', 'iii', 'iii', 'ii>']
word2_subwords = ['<jj', 'jjj', 'jjj', 'jj>']

In this example, there are basically 6 subwords: ['<ii', '<jj', 'iii', 'jjj', 'ii>', 'jj>'], but these are not common subwords in general. So, there is a possibility that the embedding for all the subwords is the same (e.g. [0,0,...,0,0]), making their sum all the same.
2) Homographs
word1 = 'lie' # meaning: tell something untruthful
word2 = 'lie' # meaning: to rest on a horizontal position

In this example, there are two homograph words. These are different words but they have the same spelling. Since FastText only take syntax into account, they will have the same subword embedding sum.


Clustering mixed data: Find optimal number of clusters What do you say about this plot to find the number of cluster for kmean or kproto for mixed data. Where is the elbow to identify? I would say 5? I have 11 feautures.

You should select 9 as you can see from plot that for the WSS value there is a dip.

It doesn’t matter if you have 2 features or 9 features or n features. Clustering is on Data present in those features(it might depend on the amount of data).


Comparing files (text sources) in Orange What is the best method to compare text files in Orange?
How do I find the difference or similar between two texts and relations between main nodes?

The similarity hashing widget can compute document similarity within a certain corpus.


How to sum values in a column grouped by another column I have this CSV file:
ID,NAME,CITY,COUNTRY,CPERSON,EMPLCNT,CONTRCNT,CONTRCOST
00000001,Breadpot,Sydney,Australia,Sam.Keng@info.com,250,48,1024.00
00000002,Hoviz,Manchester,UK,harry.ham@hoviz.com,150,7,900.00
00000003,Hoviz,London,UK,hamlet.host@hoviz.com,1500,12800,10510.50
00000004,Grenns,London,UK,grenns@grenns.com,200,12800,128.30
00000005,Magnolia,Chicago,USA,man@info.com,1024,25600,512000.00
00000006,Dozen,San Francisco,USA,dozen@dozen.com,1000,5,1000.20
00000007,Sun,San Francisco,USA,sunny@sun.com,2000,2,10000.01

What I want to do is to find the COUNTRY with the largest number of CONTRCNT. Some countries appear more than once in the dataframe so I need to find the country with the largest sum of CONTRCNT.
I thought about summing up the CONTRCNT for all countries and then finding the largest one but I want to do this in a way that is not bruteforce. I actually want to know how I can use Pandas' groupby function to solve this problem.

Answering the last part of your question:
df.groupby('COUNTRY')['CONTRCNT'].agg(sum)


Vectorizing equation in MATLAB I am working on collaborative filtering using matrix factorization in MATLAB. I am using Gradient Descent for parameter learning. The cost function to optimize is :

$ J = {\left\| I \odot (R - U V') \right\|}_{F}^{2} + \lambda_1 {\left\| U \right\|}_{F}^{2} + \lambda_2 {\left\| V \right\|}_{F}^{2} $

I have the following equations for updates :

$ \nabla J_{u(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') v_j $

$ \nabla J_{v(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') u_i $

The dimensions of the matrices are :

$I  -> n * n$

$R -> n*n$

$U -> n*k$

$V -> n*k$

I am not able to come up with vectorized form of the gradient, so I have to loop over i's and j's explicitly, which is slowing down the code a lot.

Is your question how to vectorize the equations for updating?  You have a vector of i and j, and for each update you are summing over j.  Calculate the vectors of each of the components and then sum the vector over j.

Providing actual code might help your question.  As is, it seems vague.


R: substituting only whole statements (not parts of them) I have a data set DS.
I want to substitute the statements contained with others following a pattern-replacement schema like the following one:

I have used the followng function:

from_distincts_to_origins <- function(patternST, replacementST,originalFile){

  #---------------------------------------
  patternST<-paste0('\\b',patternST,'\\b')
  #---------------------------------------
  # we have to point back to the original data set
  file_origin_DF <- as.data.frame(originalFile)

  file_output <- file_origin_DF
  for(i in 1:length(patternST))   {
    where <- file_output[[1]]#vector
    file_output <- apply(file_output,
                         2,
                         function(x) sub(patternST[i],
                                         replacementST[i],
                                         where,
                                         #ignored.case=FALSE,
                                         perl=FALSE,
                                         fixed=FALSE, useBytes=FALSE)
    )

    file_output <- as.data.frame(file_output, stringsAsFactors =F) # in order to work
  }

  return(file_output)
}

What will happen is that R will start substituting all the "I like football" with "football". When it will pass to the second row it will look for "I like football, W Lazio" but it will only find "football, W Lazio", so those rows won't be substituted with "Lazio".

One possible solution is to order the Pattern such to have the longest strings at the top, but it would only reduce the substitution missings.

Is there another method in R to do that?

I believe it will help if you add the exact regex anchors to the pattern you are trying to match. In your case adding "^I like football$" as the pattern you are trying to match will look for strings starting and ending exactly like that. That way you have:

>grepl("^I like football$", "I like football, W Lazio")
[1] FALSE

This will eliminate at least the problem you are facing now.
 I am not sure if I understand the rest of the question though...as I get it, you have some other character vector called text for example, and inside you are trying to look for elements of the vector pattern and, if matched, replace them with the corresponding vector element from replacement right?


Train/Test/Validation Set Splitting in Sklearn How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, sklearn.cross_validation.train_test_split is only capable of splitting into two, not in three...

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this:

 X_train, X_test, y_train, y_test
    = train_test_split(X, y, test_size=0.2, random_state=1)

 X_train, X_val, y_train, y_val
    = train_test_split(X_train, y_train, test_size=0.25, random_state=1) # 0.25 x 0.8 = 0.2


Papers presenting results that are worse than random chance Is it me or has there been an increasingly large amount of object detection papers describing models that are performing worse than chance.  Here is an example (an extract so not to name names):-

AP represents the average precision.  No mention of recall.

The paper goes on to say that YOLOv3-Lite reaches state of the art performance in detecting the specified object. This table would suggest to me that the models are worse just flipping a coin. What am I missing here?

Not all probabilities are 50/50.

I'm assuming that the paper you are looking at is eYOLOv3-Lite: A Lightweight Crack Detection Network.

Under the evaluation metrics section, I see

  For each image, the intersection over union ($I_{oU}$) between the bounding box of the detected crack
  and ground truth can be calculated as: $I_{oU}$ = $\frac{A_o}{A_u}$
  , where $I_{oU}$ is the intersection over union, $A_o$ is the
  area of overlap, and $A_u$ is the area of union.
  When the $I_{oU}$ of the predicted bounding box and ground truth is greater than a certain threshold
  value (e.g., 0.5), it is considered to be a true positive; otherwise, it is a false positive.

It looks like the problem is to assign a bounding box to each image such that the bounding box contains the crack (or the ground truth box associated with the crack). It's not an easy probability to calculate exactly, but it's quite clear that assigning a bounding box completely at random will yield a success much lower than 50%.


Which Format is Faster, Matlab (.mat) or NumPy (.npy)? I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.

I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").

It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.

From my research:

both can store data in binary format
both store the data type of the data
I am unsure about compression ratios and load time, which seems to be the subtext of your question.

One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.

That said, I found this post useful and thought it may be helpful to you if you have not seen it already.  https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.

Have you considered the HDF5 format?  If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.

Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?


I Have Issues Installing Basemap I tried to install Basemap and it gives me this:

preparing transaction: done

verifying transaction: done

executing transaction: failed

ERROR conda.core.link:_execute(507): An error occurred while uninstalling packag

e 'defaults::conda-4.5.12-py37_0'.

PermissionError (13, Access is denied)

Attempting to roll back

Rolling back transaction: done

PermissionError (13, Access is denied)

Question: What should I do next? I will appreciate your response as I have been on this for some time now.

Thanks.

NOTE: I have also tried to install cartopy but I ran into the same problem.

NOTE: I use Windows 7 32 bits. I use python 3.7.1

Assuming you are talking about Basemap Matplotlib Toolkit, you have to manually download the Windows binary installers.


Dropout in Deep Neural Networks I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In the test phase, all units are present, but we multiply each of them with the probability.

Now, is it like, let we have 4 input units originally named a,b,c,d. In the training stage, after applying dropout, with a dropout rate of $0.5$, we are left with units a and c. So, As in the test stage, all the units are present, so, is it like, we multiply each of the units with $0.5$? Also, Is $p$ defined for each of the units in the network, or for the entire Neural Network?

Also, In doing so, how is the result same for training and test stage?

I guess you have not figured out the concept of dropout very well. First, the reason we apply it is that we add some noise to the architecture in order not be dependant on any special node. The reason is that it was observed that while training a network, after overfitting, the weights for some of neurons increases and cause the network to be dependant on them. By exploiting dropout, we are not dependant on any node anymore due to it is possible to drop it while training.

Now, answers to your question. First, you have to bear this point in mind that the probability shows the chance of dropping a node in a layer. Consequently, chance 0.5 does not mean you, for instance, will have those two nodes. It just means after employing dropout, the chance of dropping for each node is half. Dropout is used for layers. It is customary to use it in fully connected layers. You set the hyper-parameter and it is the chance of keeping the nodes in the layer. While testing, you don't drop any node. We don't multiply neurons to the probability. The probability specifies the chance of existence of that node.

Okey-doke! I update the answer. As you can read in the paper,

  At test time, it is not feasible to explicitly average the predictions from exponentially
  many thinned models. However, a very simple approximate averaging method works well in
  practice. The idea is to use a single neural net at test time without dropout. The weights
  of this network are scaled-down versions of the trained weights. If a unit is retained with
  probability p during training, the outgoing weights of that unit are multiplied by p at test
  time as shown in Figure 2. This ensures that for any hidden unit the expected output (under
  the distribution used to drop units at training time) is the same as the actual output at
  test time. By doing this scaling, 2n networks with shared weights can be combined into
  a single neural network to be used at test time. We found that training a network with
  dropout and using this approximate averaging method at test time leads to signicantly
  lower generalization error on a wide variety of classication problems compared to training
  with other regularization methods.

I guess the easiest way to understand it is to watch this video. As you can see there are different implementations for that but the reason it is multiplied is that for any hidden unit the expected output (under the distribution used to drop units at training time) is the same as the actual output at test time. To be concise, it is done in order not to change the distribution that the outputs of the layer have.


Transfer value of one column to another column into a new column based on condition I have a dataframe with 2 relevant columns.

+-------------+---------------+
| Bezeichnung | Artikelgruppe |
+-------------+---------------+
|     A       |       1       |
|     B       |       2       |
|     C       |       3       |
|     D       |       4       |
+-------------+---------------+

I want to paste the value of Bezeichnung into a new column (new_col) for all values of the column Artikelgruppe that are equal to 0.

This can be solved using a number of methods. One of the method is:

df['new_col']=df['Bezeichnung'][df['Artikelgruppe']==0]

This would result in a new column with the values of column Bezeichnung where values of column Artikelgruppe are 0 and the other values will be NaN. The NaN values could be easily replaced at any time of point.

+-------------+---------------+---------+
| Bezeichnung | Artikelgruppe | new_col |
+-------------+---------------+---------+
|     A       |       1       |    NaN  |
|     B       |       2       |    NaN  |
|     C       |       0       |     C   |
|     D       |       4       |    Nan  |
+-------------+---------------+---------+


Hellinger Distance in Gensim I have set of documents as follows where each document has set of words that represents the content of it.

Doc1: {fish, moose, wildlife, hunting, bears, polar}
Doc2: {energy, fuel, costs, oil, gas}
Doc3: {wildlife, hunt, polar, fishing}

So, if I look at my documents I can deduce that Doc1 and Doc3 are very much similar.

I want distance metrics for bag-of-words. I followed some tutorials in Gensim about how to do it. However, as I understand, initially they train a model and then use that model to calculate the Hellinger Distance. However, in my case, I do not have any training data. Hence, please let me know how to achieve this with no training data.

As u said that one has to train a model in order to calculate Hellinger distance. I am not sure that which model u are talking about but for now I assume that u might have thinking about the latent model so according to me there is not mandatory to train a latent model to produce a document vector. one can create a document vector from the directly word-doc matrix and can calculate Hellinger distance. One simple trick is that I used is first calculate a bag of word representation after getting doc-word matrix by gensim utilities. kindly review it and let me know your opinion.


Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set? Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

Particularly, I still have over 88k variables (features) left and one cannot do much with them without massive amounts of memory. But does doing models on "blocks" lose the interactions occuring between blocks or are there some technique for "aggregating" these?

If "variables" refers to training examples:
You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example.
Or you could use Mini-Batch Gradient Descent where each iteration uses a partition of the training set.  SGD is Mini-Batch Gradient Descent where the partition size is one training example.

If "variables" refers to features:
You should use dimensionality reduction to reduce your number of features.  For instance, you can use Principal Component Analysis (PCA) to reduce your feature vector size while maintaining high variance.  This would also help your models train significantly faster.