error when input function called with shape (2,2) I am new to Tensorflow and machine learning.

I am trying to use high level API from Tensorflow.

Please tell me what i am doing wrong.

import tensorflow as tf
import numpy as np

features = np.array([[-1,-2],[1,2]],dtype='int32')
label = np.array([0,1],dtype='int32')

feature_columns = [tf.feature_column.numeric_column('features',shape=[2,2])]

model = tf.estimator.LinearClassifier(feature_columns=feature_columns,n_classes=2)

model.train(input_fn= tf.estimator.inputs.numpy_input_fn(x=features,y=label,shuffle=True))

I am getting an error

ValueError: features should be a dictionary of `Tensor`s. Given type: <class 'tensorflow.python.framework.ops.Tensor'>

x, y that goes in tf.estimator.inputs.numpy_input_fn() must be either an array or a dictionary of arrays. Quoting from the link (https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/numpy_input_fn) - x: numpy array object or dict of numpy array objects. If an array, the array will be treated as a single feature.

So in your case it is treating 'feature' as a single feature. You can try this:

x1 = np.array([-1,-2])

x2 = np.array([1,2])

features = {'x1': x1, 'x2': x2}


SVM using scikit learn runs endlessly and never completes execution I am trying to run SVR using scikit-learn (python) on a training dataset that has 595605 rows and 5 columns (features) while the test dataset has 397070 rows. The data has been pre-processed and regularized.
I am able to successfully run the test examples, but on executing using my dataset and letting it run for over an hour, I could still not see any output or termination of the program. I tried executing using a different IDE and even from the terminal, but that does not seem to be the issue.
I also tried changing the 'C' parameter value from 1 to 1e3.
I am facing similar issues with all SVM implementations using scikit.
Am I not waiting long enough for it to complete?
How much time should this execution take?
From my experience, it should not require more than a few minutes.
Here is my system configuration:
Ubuntu 14.04, 8GB RAM, lots of free memory, 4th gen i7 processor

Kernelized SVMs require the computation of a distance function between each point in the dataset, which is the dominating cost of $\mathcal{O}(n_\text{features} \times n_\text{observations}^2)$. The storage of the distances is a burden on memory, so they're recomputed on the fly. Thankfully, only the points nearest the decision boundary are needed most of the time. Frequently computed distances are stored in a cache. If the cache is getting thrashed then the running time blows up to $\mathcal{O}(n_\text{features} \times n_\text{observations}^3)$.

You can increase this cache by invoking SVR as

model = SVR(cache_size=7000)

In general, this is not going to work. But all is not lost. You can subsample the data and use the rest as a validation set, or you can pick a different model. Above the 200,000 observation range, it's wise to choose linear learners.

Kernel SVM can be approximated, by approximating the kernel matrix and feeding it to a linear SVM. This allows you to trade off between accuracy and performance in linear time.

A popular means of achieving this is to use 100 or so cluster centers found by kmeans/kmeans++ as the basis of your kernel function. The new derived features are then fed into a linear model. This works very well in practice. Tools like sophia-ml and vowpal wabbit are how Google, Yahoo and Microsoft do this. Input/output becomes the dominating cost for simple linear learners.

In the abundance of data, nonparametric models perform roughly the same for most problems. The exceptions being structured inputs, like text, images, time series, audio.

Further reading

How to implement this.
How to train an ngram neural network with dropout that scales linearly
Kernel Approximations
A formal paper on using kmeans to approximate kernel machines


How to plot an histogram from a dictionary data? This is my data which is recorded in a dictionary. A sample of my dictionary looks like this: (id-s:{id-des:value})

{'1': {'1': 2.5039}, '2': {'2': 8.993242, '1': 8.972473}, '7': {'5': 6.4600, '7': 9.16026}, '3': {'5': 6.46004, '3': 8.8472, '6': 8.8266, '1': 8.806035}, '5': {'5': 8.70111}, '6': {'6': 8.8889}, '4': {'4': 9.55662}}

Firstly,I want to plot them in histograms. Every figure presents values of an  id-s. Every bar presents values (2.5039 is the value in the first example).

I have tried with this code.

plt.subplots(len(param_dict), 1, sharex=True)
    plt.subplots_adjust(hspace=0.8)
    for key, src in param_dict.items():
        plt.title('Source : {}'.format(key), fontsize=9)
        print(key)
        for attribute, value in src.items():
            print('{} : {}'.format(attribute, value))
            plt.hist(value, bins=20, alpha=0.5, label='my label')

But it takes only the last data.

Secondly, I want to plot all of them in one histogram, all values whatever their id-s.


How to use reduced dimensions (of a PCA) for detection purposes? A general question aiming at the application of a PCA: I want to detect abnormal data points and therefore I want to use a PCA for it at first. The next step is to try several distance functions or quantils of distributions or something similar to check if it is working.
My problem is: How do I know this when I operate on the dimensionally reduced data set? Or in other words: How do I transform it back to see if the certain marked data points match the ones from the original non-transformed data?

I expect something to do like here:

How do I know that this outlier in the transformed/reduced data is sensefull? I think I have to cross-check it somehow in the original data?

Well, mathematically speaking, applying PCA on a bunch of data points usually means (there are some variants) removing the mean and rotate to get uncorrelated features. Then you reduce dimensionality by discarding some of the uncorrelated features.

You may get the corresponding points in the original feature space by doing the inverse rotation and adding the mean back again. Of course, this points now will be contained in a shape of less dimension than before. For example, if you went from 3 original features to 2 uncorrelated features, then perhaps your original data points had some 3D shape, say a football. Then your new points (transformed-inverse transformed) will have the shape of an ellipse.


Confusion about Decoder labels for training seq-to-seq models So in seq-to-seq models for say NMT, the decoder is a sequence model for the right-shifted intended output.  My question is, during training, are the inputs and outputs of the decoder supposed to be the desired labels? Or are just the outputs the desired labels and the inputs are the actual predicted output from the last timestep?

It can be both.

If you input the desired label and predict the next desired label, it's called teacher forcing.

But using only this technique might hurt the performance at test time. So using the actual predicted output from the last time step is also a good idea.

It's possible to do both :
For each batch, with X% chance you use teacher forcing, otherwise you don't.


Geometric and harmonic means in ensembling methods When using ensembling methods for regression, a common approach is to average (using the arithmetic mean) the outputs of the weak learners in order to obtain the output of the ensemble. Is there a reason why the average that people take is the arithmetic mean? Why not performing a geometric or harmonic mean?

We can answer this overarching question by exploring a couple sub-questions:

What are the properties of popular averaging formulae?

Geometric:
$ \space $
$ \space $
$ \begin{equation}
\bigg(\displaystyle \prod^N_{i=1} x_{i}\bigg)^{\frac{1}{N}}
\end{equation} $

We can see that any set that contains a zero has a geometric mean of zero.

Also, processing negative numbers is awkward as they can become imaginary.  Even if negative numbers do not result in an imaginary mean (e.g. the set has an odd number of elements), the result is often not what we expect an "average" to be.  For instance, given a set, let's take some averages:

$ \{-1,2,4\} $

Arithmetic mean: $ 1.\overline{6} $

Geometric mean: $ -2 $

The geometric mean is clearly not what we would expect for an "average".  This is because the geometric mean is primarily used for understanding the multiplicative differences in a set, which is useful for averaging data on differing scales.

Let's say we only consider the absolute value of the elements.  Then, given a set below, we still get undesirable results:

$ \{2^0,2^1,...2^{10}\} $

Arithmetic mean: $ 186.\overline{09} $

Geometric mean: $ 32 $

From this example we can see that where the arithmetic mean is linear, the geometric mean is logarithmic.

Harmonic:
$ \space $
$ \space $
$ \begin{equation}
N \cdot \bigg({\displaystyle \sum^N_{i=1} x^{-1}_{i}}\bigg)^{-1}
\end{equation} $

Like the geometric mean, the harmonic mean of a set containing zero is uninformative.  In fact, it is undefined.

A more interesting property of the harmonic mean is that it discriminates against outliers.  For instance, given a set, let's again take some averages:

$ \{ 1e^{-5}, 1e^{-5}, 1e^{10}\}$

Arithmetic mean: $ 3.\overline{3}e^9 $

Harmonic mean: $ 1.5e^{-5} $

An easier way of thinking about the averages is like this:

Where arithmetic is linear, geometric is logarithmic, and harmonic is reciprocal.

Should every learner have an equal representation in the final vote?

If you are using the arithmetic mean, the weak learners are treated equally.  This is generally desired as you do not know which learner is closer to the correct answer.  Using the geometric or harmonic mean would imply that you want to "weight" the set of weak learner outputs according to their respective additive structures.

As a side-note, you can (and probably should) use machine learning algorithms, such as a MLP or SVM, to combine the outputs of your ensemble, which would probably yield superior results to simple averaging.


What is the difference between Slow Feature Analysis (SFA) and a Moving Average? I have started to read more about Slow Feature Analysis and I was wondering how SFA differed from simply taking a moving average?

The linked article suggests, "SFA is an unsupervised algorithm that learns (nonlinear) functions that extract slowly-varying signals from their input data.".

I understand that it is certainly more complicated than just taking a moving average on a signal, but I don't really understand what the benefit or purpose is of the learned non-linear functions.

What would be some typical applications of SFA?

Mathematically, SFA differs from moving average by the fact that the instant output $y_t$ can only be a function of instant input $x_t$. On the other hand, the output of moving $y_t$ is a function of the past history of $x_{t'}$ with $t' \leq t$.

One immediate result due to this difference, is that if you have a flat constant input, with only a pulse at time $t$, then the pulse will be smoothed if you use a moving average, but its shape will be kept better if you use SFA.

You can always try both moving average and SFA for your task, although the moving average requires some input parameters such as window width while SFA does not.


Time series classification, without the time dimension Edit
Thanks to the answer of @pcko1, I understand that I should use data augmentation to make my model resilient to order of data points.
Clarification after the answer of @Icrmorin :
My problem is actually more complex than just finding bullets. I also need to find title, order text appropriately (think 2-columns PDF), find header/footer, etc...
I am currently trying a rule-based approach, and it was working well so far. But as the number of PDF format I need to handle grow, the complexity of the code grow as well, and I'm reaching a point where PDF formats have opposite features and can't be handled by the same code.
I was hoping the machine learning approach might solve this issue and works well for any format.
According to the answer of @mariq vlahova, it seems there is no name for this kind of task ? Just use LSTM ?
end of edit

I'm looking for the name of a task, in order to search more literature about the subject.
The best I can describe this task is given in the title itself...
Basically I have data points with several features each, and I need to classify each of these data points, but not independently. However the order does not (or should not) matter.

Example
For example I have 3 data points [D1, D2, D3], and I want to classify is as [True, False, False].
These data points are dependent, ie changing 1 data point might alter the result of other as well :
[D1, D2', D3] might be classified as [True, True, True].
Also, the order does not matter (that's why I wrote "without the time dimension"):
If [D1, D2, D3] is classified as [True, False, False],
then [D2, D3, D1] should be classified as [False, False, True].

Context
A bit more context... Basically I want to classify PDF content, as being a bullet point or not.
So I want to parse my PDF file, extract chunks of text along with additional information (font size, position, etc...), and classify these chunks as bullet or not bullet.
But we can't classify each chunk independently. Consider the following example :
...End of previous paragraph

1. This is a title

Beginning of next paragraph...

In this case, 1. This is a title should not be classified as bullet. But :
1. This is a title

2. This is a second title

3. This is a third title

In that case, 1. This is a title should be classified as bullet.
I need to find literature about this problem but I don't even know the name of the task...

Honestly it seems you are quite far from what would need a supervised vision approach. I suggest you to try a simple non-ML approach first : extract text with a standard library then just label what would count as a 'bullet' then check if there is more than one in a row. This might just work and if it doesn't it will help you understand why.
Going the whole OCR way would imply a lot of pdf annotation for a gain that is not really clear without trying a simple benchmark.
Ps: if you want to deal with text, the relevant field is Natural Langage Processing not time-series.


Compare two tends with big difference in absolute value I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners.
So far i retrieved my data, and plotted it.

Obviously, since the various nations have different numbers in population, spotify users ecc... this graphic doesn't tell much.
So i decided to scale every curve dividing its values by its peak.  So for example, the US has a maximum value of 3.50 million streams, i normalized that curve with that value and i did the same for all the other states (with their maximum value) and I obtained this:

Could this be a good approach? In general which approach should I use if i want to compare different curves which have a really big difference in absolute values?
EDIT : In the end i normalized my data using a zscore normalization for each single line. So basically I computed the mean and standard deviation for each state and then I normalized the single state with its mean and std.
This is the resulting plot:

Is this a good approach? Can I now compare the different trends and conclude that there's an overall decrease in the period between 10 and 15 weeks?

I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using sklearn.
Why?
At least you should be aware that dividing by the maximum could hide smaller effects. In the case you have an outlier that has a very high value, you would loose the small changes in the corresponding curve. Moreover, you might not compare the same changes between all the curves.
Edit
On the question when to use standard scaler vs minmax scaler, I think a good start is the sklearn preprocessing page that deeply explains both. Then in this post, @Sammy pointed "Python Machine Learning" book by Raschka. The author provides some guidance on page 111 when to normalize (min-max scale) and when to standardize data that I requote here:

Although normalization via min-max scaling is a commonly used technique that is useful when we need values in a bounded interval, standardization can be more practical for many machine learning algorithms. The reason is that many linear models, such as the logistic regression and SVM, [...] initialize the weights to 0 or small random values close to 0. Using standardization, we center the feature columns at mean 0 with standard deviation 1 so that the feature columns take the form of a normal distribution, which makes it easier to learn the weights. Furthermore, standardization maintains useful information about outliers and makes the algorithm less sensitive to them in contrast to min-max scaling, which scales the data to a limited range of values.

Here are some pictures to go with the Why? section above.


Importance of feature selection for boosting methods While it is obviously clear that features can be ranked on basis of importance and many machine learning books give examples of random forests on how to do so, its not very clear on which occasions one should do so.

In particular, for boosting methods, is there any reason why one should do feature selection. Wouldn't the boosting methods themselves eliminate the low importance feature.

Isn't is just always better to add more features (if one didn't have the practical problem of time limitations).

There is a difference between boosting and features selection.
It is very important to understand that the original boosting algorithm or bagging algorithm have been modified and augmented with many features selection and/or data sampling ( over/ down/ synthetic) to improve the accuracy.
Let us talk about the difference between bagging and boosting :
Booth of them are random subspace based algorithms, the difference is in bagging we used uniform distribution and all the samples have the same weight , in boosting we use non- uniform distribution, during the training the distribution will be modified and difficult samples will have higher probability. The second difference is the voting. In bagging it is average voting , in boosting it is an weighted voting.

Features selection algorithms try to find the best set of features that can separate the classes. But there is no explicit consideration for difficult or easy samples and what is the used training algorithm.
In boosting, the algorithm selects the feature that minimize the error , the error is the sum of prob "weights" of samples that miss classified, since the difficult samples have higher weights , the selected feature will be the one that better distinguish between the difficult samples.

FE ( Features, data) --> feature set
Boosting ( features, data, base learners type, initial distribution, difficult samples) --> feature set


pandas groupby.count doesn't count zero occurrences I am using groupby.count for 2 columns to get value occurrences under a class constraint. However, if value $x$ in feature never occurs with class $y$, then this pandas method returns only non-zero frequencies. Is there any solution or alternate method?
The script is like:
combine = np.vstack([X_train[:,-1], y_train]).T
combine_df = pd.DataFrame(combine, columns = ['feature','class'])
class_count_groupby = combine_df[combine_df['class'] == 2]['class'].groupby(combine_df['feature']).count()

Use crosstab instead:
pd.crosstab(combine_df.class, combine_df.feature)


Predicting Soccer: guessing which matches a model will predict correctly I took on a project to predict the outcome of soccer matches but it turned out to be a very challenging task.
I tried out different models but I only got 50-54% accuracy on my test dataset. Some of the models were created in such
a way that a certain model would predict if a team will win, draw, or lose a match. That same model would also predict if
the opponent of that team will win, draw, or lose the match. Each model predicting with an accuracy of about 50% on each team distinctively. The second set of models I tried, takes the combination of data from both teams and predicts which class the match belongs to (home win, away win, draw). In the system,
only 10 matches are given everyday to be predicted. Meaning, if I predict the 10 matches using the second model, I have a chance of predicting 5
correctly. In this project, I only need to predict 3 matches correctly out of the 10 matches given in a day. Is there a system
of knowing the 3 matches which my models have the best chance of predicting correctly? I only need to get 3 correct predictions, I usually
get 5 correctly but I don't know how to select my 3 best matches.

Note: The first type of models uses about 50 features for prediction while the second uses 101. I've tried ensembles, they still give
me ~50% accuracy. I'm still about to set up a system that selects matches where the prediction for the home team does not
contradict the prediction for the away team using the first type of models.

This sounds like an interesting project. I recently worked on an almost case study. In order to get only 3 most accurate predictions, I think you may wanna sort those correctly predicted 5 matches by probability of event (win, draw or loose) and then select the first three matches.I hope your models are able to give you probabilities of events.
I hope this helps :-)


Predict next series of numbers based on previous data I have the following list of numbers (1 – 5  categories, first row) that I am trying to predict the next sequence for each column (total column sum = 6). There are over 1,500 lines of data in total. I have tried to make a column of the accumulated sum of each line and then use Linear Regression and Time Series Forecasting to predict the next accumulated sum and then subtract the previous number to predict the number (0,1,2,3,4) but I am not getting the results that I am after.
How best can I forecast the next series of numbers using a non-Linear approach ?

You could use LSTMs with a sequence to sequence model perhaps (using e.g. Keras).

For example:
x lines are input, the next line is output.

Just an idea, your question is not that clear.


How to interpret partial dependence interaction plot for binary classification? The plot below is an example from pdpbox library https://github.com/SauceCat/PDPbox/blob/master/tutorials/pdpbox_binary_classification.ipynb. The what does 0.900 color saturation to the right mean with respect to age and fare in the below plot? The data is taken from Titanic dataset and target variable is survived or not.

The partial dependence plot shows an increase in survival probability at ages between 0 and 10 with any fare range.

For more on how to interpret partial dependence plot:

https://christophm.github.io/interpretable-ml-book/pdp.html


How do I fix mis-rendered matplotlib? How do I correct my data or format it so that it is presentable, and fix my graphs?

Dataset is 345551 rows × 7 columns.
I am using numpy, pandas, seaborn and matplot lib.
It seems that my pricing data is being displayed in scientific notation.

When I fit a linear regression model I get the following coefficients

property_type= -3.096186e+05
new_build= -1.909146e+04

When I use a train/test split and check my predictions they don't make sense.

index=246862
actual=440000
predicted=4.252606e+05

Try using numpy's log or log1p. The math module also offers logarithmic functionality.

You can use sklearn to transform your variables - see MinMaxScaler, StandardScaler, etc.

Other transformations like Box-Cox can be found in scipy.stats


Does classification of a balanced data-set lead to any problem? So I came across a bioinformatics paper, where I found a line which says:

  One potential problem with using a training set with
  equal numbers of positive and negative examples in cross-validation
  is that it can artificially inflate performance estimates
  because the number of false-positive classifications is
  proportional to the number of examples classified. So applying
  these methods to all proteins in an organism may
  result in a large number of false-positive identifications.

I am unable to understand how classification of balanced dataset is a problem. Can someone please explain this to me?

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't work well because of changing the distribution of each class. In cases like a disease prediction where the number of each class varies for different places, a model that is trained in the U.S won't work in African countries at all. The reason is that the distribution of classes has been changed. So in such cases that usually the negative and positive classes are not balanced in practice, balancing them will cause the problem of distribution changes. On these occasions, people usually use the real data-set which is not balanced and use F1 score for evaluation.


Data science projects explained step by step? I am looking for a website or book where several practical examples are given step by step, explaining how they choose the relevant features, the model selection procedure, etc...

If you want an application-oriented book, consider Christopher Bishop's Model-Based Machine Learning. He has more technical books that are well regarded.
If you are looking for lots of code, Probabilistic Programming & Bayesian Methods for Hackers is an option.
Another introductory book with a more statistical bent is An Introduction to Statistical Learning with Applications in R. Again, the authors have a well-regarded technical version of the book.


What is the best way to optimize the parameters in a Sklearn classifier, when I have little data? What is the best way to optimize the parameters in a Sklearn classifier when I only have a data set with 684 rows and 177 columns, and the column I want to predict has 3 labels?

I know I should split my data into training, validation, and test set, and then find the parameters to train the training set that maximizes a metric in the validation set and use this optimized classifier in the test set. However, when I did this using a decision tree classifier, the parameters that worked best for the validation set showed a worse result in the test set than the default parameters. So what is the best way to find the best parameters in this data set? I do not know if it helps, but my data set is very sparse.

You can't get around the test/train thing. I guess in your case, the problem is the choice of method. When I read your data description, I suspect that you might achieve better results with regularization, e.g. Lasso. See some examples here.

Basically you would apply a multiclass logit with lasso penalty in your case. Alternatively you could also check boosting with lasso (aka $l1$ penalty). Here is some playcode with the iris data.

Since your data has many columns and not so many rows, I might go for lasso (not ridge or elastic net), because lasso can shrink features to zero if they don't make a meaningful contributin to the classification. This is especially helpful with high dimensional data.


How to write a LSTM model with 3 dimensional X_train and Y_trains? I have X_train and Y_train with [2160,24,3] dimensions. But when I try a simple LSTM like this:

model = Sequential()
model.add(LSTM(24 , return_sequences = True))
model.add(Dropout(dp))

It gives me this error message:

  ValueError: Error when checking target: expected dropout_192 to have
  shape (24, 24) but got array with shape (24, 3)

And when I try to change my code like this:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))

model.add(RepeatVector(24))

I get:

  ValueError: Error when checking target: expected repeat_vector_55 to
  have shape (24, 24) but got array with shape (24, 3)

Also:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))
model.add(RepeatVector(24))
model.add(TimeDistributed(Dense(1)))

Or:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))
model.add(RepeatVector(24))
model.add(LSTM(24 , return_sequences = True))
model.add(Dropout(dp))
model.add(TimeDistributed(Dense(1)))

Gives me:

  ValueError: Error when checking target: expected time_distributed_56
  to have shape (24, 1) but got array with shape (24, 3)

I am confused and don't know how to fix it? What I want is train a LSTM model with X_train and Y_train of size [2160,24,3]. How can I do that?

In all the above cases, you are not setting the output shape properly. The last layer should create 3 features.
Try this for example:

model = Sequential()
model.add(LSTM(24, return_sequences = True))
model.add(Dropout(dp))
model.add(Dense(3))


Predicting the next occurrence based on binary I have no experience in statistics or machine learning. I have a True/False binary array describing occupation of open public spaces

+---------------------+
|  index   |  Value   |
+---------------------+
| 0        |  True    |
| 1        |  True    |
| 2        |  False   |
| 3        |  False   |
| 4        |  False   |
| 5        |  True    |
| 6        |  False   |
| 7        |  False   |
| 8        |  True    |
| ...      |  ...     |
+---------------------+

Without getting into dependent variables and domain specific heuristics, is there (or maybe more than one) a simple method to do infer the next False in python?

Ideally in pure python or using packages written in pure python.

My question is somewhat similar to this one, but I have more of a time series (i think).

Do you have more features ?
If not, you can try to find a filling pattern, for example, probably some places are filled first because, for example, are closer to the entrance, then another group etc.
Try to plot as a time dependent problem. I cannot tell how much bigger the error will be.
If you have at least the x, y of the place and the timestep of the value, then you have a pattern.


What is a logworth statistic, and how useful is it? My teacher mentioned it today, and there is nearly zero good search results for it, other than one mention each in the SAS and JMP documentation.

It says it is -log10(p-value), but there is almost no explanations of this online. Also it seems like this statistic is widely ignored. It isn't mentioned in any tutorials or in any of the data science papers I have read in the past two years.

Question

So, what is this statistic, and where does it come from? How useful is it in data science?

logworth is a p-value transformation based on Pearson Chi-Squared test.

Pearson Chi-Squared test evaluates the probability of having a split caused by chance. The higher the Pearson Chi-Squared value, the higher the chance that the split is caused by dependence. A p-value is given according to the Chi-Squared score and the degree of freedom.

Because some data can have high dependence relationship, the p-value can be extremely small. Taking a logworth of p-value (-log(p-value)) allows us to mitigate this extremely low value.


How does the inertia cython implementation in scikit-learn for kmeans work? Specifically, what do the & symbol stand for? and why is the column index always 0?
    cpdef floating _inertia_dense(
        np.ndarray[floating, ndim=2, mode='c'] X,  # IN
        floating[::1] sample_weight,               # IN
        floating[:, ::1] centers,                  # IN
        int[::1] labels):                          # IN
    """Compute inertia for dense input data
    Sum of squared distance between each sample and its assigned center.
    """
    cdef:
        int n_samples = X.shape[0]
        int n_features = X.shape[1]
        int i, j

        floating sq_dist = 0.0
        floating inertia = 0.0

    for i in range(n_samples):
        j = labels[i]
        sq_dist = _euclidean_dense_dense(&X[i, 0], &centers[j, 0],
                                         n_features, True)
        inertia += sq_dist * sample_weight[i]

    return inertia

& is the "address-of" operator in c, and that appears to be how it's being used here.  See these two SO posts.
Note the signature of _euclidean_dense_dense:
cdef floating _euclidean_dense_dense(
        floating* a, # IN
        floating* b, # IN
        int n_features,
        bint squared) nogil:

The first two inputs are pointers. So you need to pass the address, not a copy of the data.
Notice too that only the address of the element in the first column gets passed. If you look at the definition for _euclidean_dense_dense that becomes clearer: that function actually loops over the addresses of the rest of the columns in its computation.


What does it mean for an activation function to be "saturated/non-saturated"? For context, in this paper

  Several RNN variants such as the long short-term memory (LSTM) [10,
  18] and the gated recurrent unit (GRU) [5] have been proposed to
  address the gradient problems. However, the use of the hyperbolic tan-
  gent and the sigmoid functions as the activation function in these
  variants results in gradient decay over layers. Conse- quently,
  construction and training of a deep LSTM or GRU based RNN network is
  practically difficult. By contrast, ex- isting CNNs using
  non-saturated activation function such as relu can be stacked into a very deep network (e.g. over 20 layers using the basic convolutional
  layers and over 100 lay- ers with residual connection

Please see this answer.

An activation function is considered non-satured if

$$
\lim_{z \rightarrow \infty} f(z) = \infty
$$

A saturated activation function has a compact range such as $[-1,1]$ for $\tanh$ or $[0,1]$ for the sigmoid.


What is semantic gap? Why it exists in AI The semantic gap characterizes the difference between two descriptions of an object by different linguistic representations, for instance languages or symbols. The semantic gap can be defined as "the difference in meaning between constructs formed within different representation systems".
I need to find a clear answers to the second part

Any ML model assumes a particular representation of the data, and connects bits of information only within this representation.
For example a linear regression model assumes a linear relation between the features and the target variable. A Naive Bayes model assumes that the features are independent of each other. And these are only the most obvious kind of simplifications made by ML models.
Naturally this results in different models representing data differently. Any semantic interpretation based on some model outcome is potentially biased by the assumptions made by the model.


Why does an L2 penalty prefer smaller and more diffuse weight vectors? For instance, why is it that it is more favourable for a weight of [0.25, 0.25, 0.25, 0.25] (for which the L2 penalty is 0.25) instead of simply [1, 0, 0, 0] (for which an L2 penalty is 1)?

In this case, both weights would give the same dot product when using W.T * X

You answered this in your question. "Prefer" means "produces a smaller penalty", and you've identified that the penalty in the first case is smaller. Why would this be a good thing? It amounts to preferring an explanation based a bit on many features, rather than one based entirely on one features. That's often a good bet to avoid overfitting.

These two weight vectors do not produce the same dot product with other vectors in general. If the vector X contained all identical values, they would.

If the 4 input features were regularly identical or nearly so, then it means they're redundant, and you may prefer to use just 1 of the features (your second case), instead of a bit of each. In this case, an L1 penalty would at least be indifferent to the two, not penalize the second one.


conceptnet api documentation for /wn/ part of the URI for url http://api.conceptnet.io/c/en/music/n/wn/communication

/c/en/music/n is explained the doc (https://github.com/commonsense/conceptnet5/wiki/URI-hierarchy) as:

"the initial /c to make it a concept, a part that indicates its language (usingthe BCP 47 language code for that language), and a part with the concept text. An optional fourth component gives the part of speech (as a single letter, following the convention of WordNet)."

But couldn't find documentation for the part /wn/communication. Could you point to the documentation for this or explain this part of URI .. Thanks

This is the finer-grained word sense information that was added in ConceptNet 5.7. WordNet defines "music" in the topic area of "communication". We keep this information in the word sense. The idea is to distinguish WordNet senses, but not to make the extremely fine distinctions that WordNet makes.

I didn't describe it on the "URI Hierarchy" page yet, sorry about that. It's described in the changelog for now: https://github.com/commonsense/conceptnet5/wiki/Changelog

"music" is a tricky example, because WordNet has finely-distinguished senses of it that appear to be in different WordNet topic areas, even though they all seem to relate to the same meaning. Its different senses are listed in topic areas including "communication", "act", and even "music" itself.


Feeding R agnes object into cutree I'm using agnes to group terms that frequently appear with one another in a set of documents. I get a dendrogram which verifies that things are working as expected. From this, I'd like to to retrieve the cluster of each member. I'm using cutree to cut the tree at a specified height, but it is returning one extra cluster, I'm not sure why.

length(flex75$order.lab)
[1] 132
length(cutree(flex75, h = 1.5))
[1] 133

It occurs for varying heights, even up to h = max(flex75$height) and all clusters = 1. So I cannot bind the labels to their clusters.

EDIT:

I found the issue to be when I reorder flex75$height to be in ascending order (I was previously receiving an error that: the 'height' component of 'tree' is not sorted (increasingly)). I was also reordering the $order and $order.lab components using the $height order, and turns out that height is of length n-1 and order/order.lab is of length n.

My question now is: What is the relationship between the agnes object's $height vector, and the $order/$order.lab vectors? Can I permute the heights, as I have, and feed that object into cutree? Should any of the object's other properties be correspondingly changed?

I found an answer in a related post here. Another user was getting a similar error about ordering height, but theirs came with the suggestion to apply as.hclust() first. I converted the agnes object to an hclust and passed that directly into cutree. That seemed to solve the problem.


What kind of data (in context of trends in data) is Logistic Regression appropriate for? I'm not able to visualise what kind of 'trends' I would have to observe in multi-featured data to be able to say 'Logistic Regression would work well here'.
For example if I have only 1 feature and if the data is something like all negative data is class 0 and all positive data is class 1. Then I can clearly say that Logistic Regression works well here.
So what kind of 'analysis' of the data (multi-featured) would I have to do to decide whether Logistic Regression would work well or not?

As you would be aware that, logistic regression is a standard linear model, following these steps would be helpful :

Plot the input data, just observe how would probably the decision boundary look like.
If it's completely non-linear then other models will give you a good result than logistic regression.
If there is a high chance of linearity possible then you can go with logistic regression


Class weights for imbalanced data in multilabel problems I am trying to train a CNN for a multiclass - multilabel classification task (20 classes, each sample can belong to 1+ labels) and the dataset is highly imbalanced. In single-label cases I would use the compute_class_weights function from sklearn to calculate the class weights in order to help the optimizer to account for the minority class. However, for the multilabel case I feel its not working as supposed to, because it considers as number of samples the number of times all classes occur, while the actual number of samples are less (since its multilabel). Is anyone familiar with a function, or even a formula, to calculate the class weights in this case?

Thanks

I think what you are looking for in this case is "cost-sensitive classification" - you can look it up on google scholar to find some papers. You'll likely have to define "costs" for each type of misclassification.

Alternatively, if you are doing one-vs-rest classification, you can reweight/upsample/downsample differently for each class.

If your dataset is small, you could also try to set weights as the objective of some constrained optimization objective that would make them more balanced.


Data Science conferences? This is a similar question like the Statistics Conferences question at CrossValidated

What are the most significant annual Data Science conferences?

Rules:

Include a link to the conference
Please include links for the talks (be it youtube, the conference site or some other video streaming site)

PyData - talks about Python Data tools
Link: http://pydata.org/events/

There is one PyData conference on the east coast and one on the west coast each year.

NIPS - Neural Information Processing Systems (NIPS)
Link: https://nips.cc/

This is one of the hardest / most prestigious academic Machine Learning conferences to get an abstract / poster accepted.

The 5th International Workshop on Parallel and Distributed Computing for Large Scale Machine Learning and Big Data Analytics (IEEE IPDPS 2016)
Link: http://parlearning.ecs.fullerton.edu/

This one is also an academic conference with paper submission.

Note:
I am not sure if you want academic or nor academic conferences (have conference proceedings / papers associated with the conference).
Some conferences are not about new data science methodologies  but the tools and libraries (e.g. PyData) that implement existing methodologies.
Also, data science is very broad and includes Stat, Machine Learning and data warehousing / mining etc.


DL model to assess quality of image I have an idea but I am not certain that it can be modeled in a DL architecture.

Let's say we have images of different qualities based on color patterns and their assessment as labels in a range from 0-1.
E.g. Image 1 has 0.25 quality, Image 2 has 0.5 quality and so on.

Could this be implemented in a standard Resnet50 architecture with 1 output and sigmoid?
Is there anything in literature that you could point me up to? I wasn't able to find anything, maybe I am searching it wrong.

EDIT: I have found this https://github.com/idealo/image-quality-assessment, but the implementation is different that I suggest. I know that this can work if I add five different quality classes.

I want to know if I can train this with 1 class output with a rank, in order to let the model understand how those qualities are linked. For instance quality 2 is the next best quality than quality 1.

I found the answer I was looking for in this Google paper:

However, in the case of ordered-classes (e.g.
aesthetic and quality estimation), cross-entropy loss lacks the
inter-class relationships between score buckets. One might
argue that ordered-classes can be represented by a real number, and consequently, can be learned through a regression
framework. Yet, it has been shown that for ordered classes,
the classification frameworks can outperform regression models [21], [31]. Hou et al. [21] show that training on datasets
with intrinsic ordering between classes can benefit from EMDbased losses. These loss functions penalize mis-classifications according to class distances


Normalization of CT scans I trained an infection segmentation models on a large dataset of CT scans, and want to extend it to other datasets to show the ability of the model to generalize. What I found though, is that CT scans look very different. For example,

CT scan from the first datasets,

Image from another dataset

I tried three things:

Global mean/standard deviation (subtract from each image the same vector $m$ and divide by $s$),
Local mean/standard deviation (from each image subtract its $m$ and divide by its $s$, os that every image is normally distributed with mean $m$ and std $s$,
Gamma correction ($\gamma=0.01$), see the image at the bottom.

None of which worked. My question is: is it possible to 'normalize' other CT scans, 'normalize' in a sense make them more similar to the one on which I trained the model?

There is an extensive literate and established software to normalize CT scans. It is required for medical research to conduct group level analysis. The most common process involves transforming to standard template. The paper "Age-specific CT and MRI templates for spatial normalization " goes into greater detail.


Optimal proportion between the amount of Class = 1 and the amount of Class = 0? I am quite new machine learning methods, so I may not write proper technical formulas.

My question is about the optimal proportion between sample size in Class = 1 and Class = 0 in a binary classification problem. (I might use different models, but I will start with decision tree).

I am trying to develop a risk model for a population of 300.000 by using some risk factors, and each risk factor has a risk-weight. So basically it is a accumulative model

  Risk_amount = Risk_factor_1 * weight_1 + Risk_factor_2 * weight_2 + Risk_factor_3 * weight_3 ...

My big question is if a Risk_amount really belongs to  Class = 1 or not.

I want my ML-model to find the most optimal weights to those risk factors.

I already know that 1000 out of 300.000 are risky in real. So the amount of target data with Class = 1 is 1000.

So the total amount of target data with non-risky population Class = 0 is 299.000

The question is what is the optimal proportion between the amount of Class = 1 and the amount of Class = 0?

1000 Class = 1 vs. 1000 Class = 0

or

1000 Class = 1 vs. the whole Class = 0 as 299.000?

The balance between two classes in a classification is very important, as you do not want your model to overfit for a particular class.

This is where you use metrics apart from accuracy to really evaluate how good your model truly is. In case you are not able to balance the dataset, there are multiple ways to work with imbalanced data. They are as follows:

You could use certain techniques like SMOTE to generate more samples of the undersampled class
You must split your dataset in test and train with stratification so that you have a balance in your evaluation.
You can sub-sample the large class and balance the two classes, and this multiple times by taking random sub samples of the larger class.

Thorough analysis of the result is very important to understand how to proceed towards the solution to the problem. Please look at f1 score, precision and recall apart from accuracy. Also read about micro/macro averaging of these metrics.

There are a lot of conversations on datascience stackexchange and stackoverflow on how to work with imbalanced data for classification. Here is a link : https://stackoverflow.com/questions/40568254/machine-learning-classification-on-imbalanced-data

Have fun with machine learning :)


How Adaboost calculates error for each weak learner in training? I am studying the Adaboost classification algorithm because i would like to implement it from scratch. I understand how it works, but i am not able to understand where some steps are placed.
I will describe the Adaboost training steps in my understanding (sorry for any incorrect formalism):

Initialize a weak learner $k$
Define a weight for each sample in the dataset equally $w =\frac{1}{N}$
Fit $k$ to the dataset
Calculate error $e = \sum_{i=0}^{N}e_iw_i$
Calculate importance $\alpha$ of $k$, i.e. $\alpha = \frac{1}{2}log(\frac{1-e}{e})$
Recalculate weights for the correct classified samples: $w_{t+1} = w  e^{\alpha}$
Recalculate weights for the incorrect classified samples: $w_{t+1} = w  e^{-\alpha}$
Normalize new sample weights: $w_{normalized} = \frac{w}{\sum_{i=0}^N w_i}$
For all sequent learners, select samples based in weighted random choice until get a dataset with same size as the original and do the same proccess.

My question is: how error is obtained? Regarding the implementation, should i firstly fit the dataset and then get the error from predicting the same dataset? This doesn't seems correct.
I've tried to read different sources about this and even a great explanation from the Statquest channel wasn't able to make this clear.
Thanks!

When I started learning about ensemble methods, this youtube video help me greatly.
The idea is that you fit the model on the data, calculate the error and then work with the error. So yes you should use a weak learner to predict your data and then, use another weak learner.
There are some more videos in the same channel that give a great explanation to have an intuition.


Normally distribute occurence or counts I am creating a mock of sales data. One of the columns is salesperson_id where each id can occur more than once (a salesperson can have multiple sales). I want to generate this column in such a way that if I create a chart of distribution of sale count (not total sales) per salesperson_id, it would be a normal curve.

Let's say I have an array of unique salesperson_id called salespersons, I want to generate n number of records. For example, I have 6 salespersons and I want to generate total of 14 records. This would be the distribution of their counts:

id  count
---------
A   1
B   2
C   4
D   4
E   2
F   1

If arranged in a bar chart, this would look like a normal distribution. I also need some way to control the 'flatness' of the chart.

Here is one way to do it:

from collections import Counter

from numpy.random import binomial

# Generate mock sales data by randomly sampling if a salesperson makes a sale
p = 0.5 # Control "flatness" of distribution
n_records = 14
salesperson_id = binomial(n=100, p=p, size=n_records).tolist() # Binomial is a reasonable approximation for a discrete normal distribution
count_per_salesperson = Counter(salesperson_id)
id, count = zip(*count_per_salesperson.items())


Should I connect my two GPUs with SLI or not? (for Keras + TensorFlow) I have two GPUs, NVIDIA GTX 1070 Ti. For using Keras with TensorFlow back-end, should I connect them with SLI or not? If not, then they will be treated separately, and one model will be trained on one card. These are the two options from what I understood so far. Thank you.

You need not connect GPUs via SLI.  Keras and TensorFlow will take care of distributing batches across GPUs

https://keras.io/utils/#multi_gpu_model

Instead of SLI, if you use NV-link, Keras can use use GPU for merge as well.
https://www.nvidia.com/en-us/data-center/nvlink/

From the documentation :

  Specifically, this function implements single-machine multi-GPU data
  parallelism. It works in the following way:

  Divide the model's input(s) into multiple sub-batches. Apply a model
  copy on each sub-batch. Every model copy is executed on a dedicated
  GPU. Concatenate the results (on CPU) into one big batch. E.g. if your
  batch_size is 64 and you use gpus=2, then we will divide the input
  into 2 sub-batches of 32 samples, process each sub-batch on one GPU,
  then return the full batch of 64 processed samples.

  This induces quasi-linear speedup on up to 8 GPUs.

  This function is only available with the TensorFlow backend for the
  time being.


Does Mixup requires two loss functions? I create a neural network with multi-label classification using MSE.
Now I want to use Mixup. Do I need two loss functions (for each target one) or is the result the same if I just combine the two targets like this?
 target = t * target1 + (1-t)* target2

In their paper, mixup: Beyong Empirical Risk Mininization, the authors provide a piece of code how to train an epoch in PyTorch:
# y1, y2 should be one-hot vectors
for (x1, y1), (x2, y2) in zip(loader1, loader2):
    lam = numpy.random.beta(alpha, alpha)
    x = Variable(lam * x1 + (1. - lam) * x2)
    y = Variable(lam * y1 + (1. - lam) * y2)
    optimizer.zero_grad()
    loss(net(x), y).backward()
    optimizer.step()

As you can see, they only use a single loss function defined on the new combined target variable y or, in your terminology, target.


Reading a json-file from an API, getting "AttributeError: 'Response' object has no attribute 'data' "error Title says it all.
Im trying to read a json file im accessing via an API. But when I try, I get the error:
'AttributeError: 'Response' object has no attribute 'data'
I don't understand why, as most (not all) of my fellow classmates do not get this error.
The code I'm trying to run:
[1] import folium
[2] import geopandas as gpd
[3] import pandas as pd
[4] import numpy as np
[5] import urllib3
[6] import requests
[7] import json
[8] import os

[9] dmi_api = 'f8654190-89bb-43d2-9e08-416346b79e7b'
[10] url = 'https://dmigw.govcloud.dk/v2/metObs/collections/observation/itemsstationId=06031&parameterId=temp_dry&datetime=2021-01-01T00:00:00Z/2021-03-01T00:00:00Z&api-key='+str(dmi_api)
[11] r = requests.get(url)
[12] print(r)
[13] print(url)

[14] data = json.loads(r.data.decode('utf-8')) #This is where the error is referring to (which leads back to line 11)
[15] data

What I don't understand is why the json.loads isnt working, as the link is obviously full of data.
I'm coding in Python 3.2 using the Jupyter Notebook accessed from Anaconda, if that's any help.

This is not a data science question and you should try programming stack-exchange instead. But this is happening because of an API mismatch in Python, specifically the type of the GET method of the requests object. It's likely the code snippet you got is from a version that is not python 3.2 or a requests module that is not up to date. I would try to install a compatible version of the requests module from PyPI or make your python version compatible with the snippet.


Simple EDA: what to visualize? I want to visualize a data set for EDA having 8 parameters and 2 class labels.
I am confused which parameters to select and on what basis? Also is I want to know whether it would be better to apply dimensionality reduction techniques before visualization?

Faraz!
Since you have just 8 parameters it is not time consuming to observe all you parameters and how they are represented for both classes.

You can start with simple information about distribution of parameters over your 2 classes. Check this notebook where an author has explored Titanic data. You can start your explanation in the similar way.

For more tips from a community you need to provide us with more data like what is the size of your data set, are the parameters categorical or numerical, etc.
And actually what is the purpose of your EDA? What is your next step going to be?


Performace of Fischer projection as dimension reduction compared to other LDA methods How is the performance of Fischer projection compared to other LDA methods of dimension reduction? I thought that Fischer projection was a great method of dimension reduction by maximizing class separation, but when I looked at the LDA methods in scikit learn, Fischer projection wasn't even in the list. This got me thinking, is it any good compared to the other methods out there?

Edit (answer): My bad, they are the same. Fischer projection is a 2 class special case of LDA. Projection using LDA can be performed using the fit_transform and transform methods in sklearn

I didn't get what you meant by "other LDA Methods". To the best of my knowledge, Fisher method is just one.

Sklearn has the implementation and you can find it here.

Is it good or bad? When it comes to dimensionality reduction you don't know as dimensionality reduction methods are usually unsupervised. So if PCA works better? No one knows. At least in the context of two-class classification, Fisher method is still brilliant.

Hope it helped!


How would you teach a machine how to count? var digits = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"];
var numbers = [digits[0], digits[1]...digits[9], digits[1] + digits[0], digits[1] + digits[1]...];

What would you use to build a model that can predictNextNumber(19)=20 by understanding how counting works?

I suggest using supervised learning and employing a linear model: linear regression.

This is a perfectly linear system (y=x+1), so linear regression will work just fine i.e. perfectly. Further, you have an infinite amount of data you can use to train the system, so it should be easy to train ;-)  I jest... I think that two data points will be sufficient, again, since it is perfectly linear!

Pedagogically, the triviality of this extremely simple linear system gets a little more interesting when you try analogue based methods like support vector machine (SVM) - which should also be able to provide a perfect result, decision trees or random forests, and even naive Bayes regressors.

Though its useful to start learning with very simple systems, I suggest quickly moving on to something more complex, i.e. don't get too stuck in your own head for the trivial linear model. Don't forget that the power of data science and machine learning lies in its statistical nature, so try to find a test case that includes some statistical variation in the input.

Hope this helps!


AUC and classification report in Logistic regression in python I have been trying to implement logistic regression in python. Basically the code works and it gives the accuracy of the predictive model at a level of 91% but for some reason the AUC score is 0.5 which is basically the worst possible score because it means that the model is completely random. Also the classification report returns error: "UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)". Does anyone know what should I change so it works properly?

 import numpy as np
 import pandas as pd
 from sklearn.cross_validation import train_test_split
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import accuracy_score
 from sklearn.preprocessing import StandardScaler
 from sklearn.metrics import roc_auc_score
 from sklearn.metrics import classification_report

 data_file = pd.read_csv('loan.csv', delimiter=',')

 # variable preprocessing

 data_file['loan_status'] = np.where(data_file['loan_status'].isin(['Fully
 Paid', 'Current']), 1, 0)
 loan_stat=data_file['loan_status']
 loan_stat=loan_stat.astype(np.float64)

 m = {
    'n/a': 0,
    '< 1 year': 0,
    '1 year': 1,
    '2 years': 2,
    '3 years': 3,
    '4 years': 4,
    '5 years': 5,
    '6 years': 6,
    '7 years': 7,
    '8 years': 8,
    '9 years': 9,
    '10+ years': 10
 }
 emp_length=data_file.emp_length.map(m)
 emp_length.astype(np.float64)

 annual_inc=data_file['annual_inc']
 delinq_2yrs=data_file['delinq_2yrs']
 dti=data_file['dti']
 loan_amnt=data_file['loan_amnt']
 installment=data_file['installment']
 int_rate=data_file['int_rate']
 total_acc=data_file['total_acc']
 open_acc=data_file['open_acc']
 pub_rec=data_file['pub_rec']
 acc_now_delinq=data_file['acc_now_delinq']

 #variables combined into one dataframe

 X=pd.DataFrame()

 X['annua_inc']=annual_inc
 X['delinq_2yrs']=delinq_2yrs
 X['dti']=dti
 X['emp_length']=emp_length
 X['loan_amnt']=loan_amnt
 X['installment']=installment
 X['int_rate']=int_rate
 X['total_acc']=total_acc
 X['open_acc']=open_acc
 X['pub_rec']=pub_rec
 X['acc_now_delinq']=acc_now_delinq
 X['loan_stat']=loan_stat

 X=X.dropna(axis=0)
 y=X['loan_stat']
 X=X.drop(['loan_stat'], axis=1)

 scaler=StandardScaler()
 X=scaler.fit_transform(X)

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
 random_state=42)

 model=LogisticRegression(penalty='l2', C=1)
 model.fit(X_train, y_train)
 score=accuracy_score(y_test, model.predict(X_test))
 roc=roc_auc_score(y_test, model.predict(X_test))
 cr=classification_report(y_test, model.predict(X_test))

Here is the link to the data: https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip

In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function:

roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])

This will give you the probability for each sample in X_test having label 1.


What is the actual output of Principal Component Analysis? I'm trying to understand PCA, but I don't have a machine learning background.  I come from software engineering, but the literature I've tried to read so far is hard for me to digest.
As far as I understand PCA, it will take a set of datapoints from an N dimensional space and translate them to an M dimensional space, where N > M.  I don't yet understand what the actual output of PCA is.
For example, take this 5 dimensional input data with values in the range [0,10):
// dimensions:
// a  b  c  d  e

[[ 4, 1, 2, 8, 8],      // component 1
 [ 3, 0, 2, 9, 8],
 [ 4, 0, 0, 9, 1],
 ...
 [ 7, 9, 1, 2, 3],      // component 2
 [ 9, 9, 0, 2, 7],
 [ 7, 8, 1, 0, 0]]

My assumption is that PCA could be used to reduce the data from 5 dimensions to, say, 1 dimension.
Data details:
There are two "components" in the data.

One component has mid a levels, low b and c levels, high d, and nondeterministic e levels.
The other component has high a and b levels, low c and d levels, and nondeterministic e levels.

This means that the two components are most differentiated by b and d, somewhat differentiated by a, and negligibly differentiated by c and e.
Outputs?
I'm making this up, but say the (non-normalized) linear combination with the highest differentiating power is something like
5*a + 10*b + 0*c + 10*d + 0*e

The above input data translated along that single axis is:
[[110],
 [105],
 [110],
 ...etc

Is that linear combination (or a vector describing it) the output of PCA?  Or is the output the actual reduced dataset?  Or something else entirely?

Have you tried reading the Intuition section of the PCA page on wiki? Also, the Further Considerations section, I think, explains what the various components represent and addresses your questions.

A short summary of what those sections describe is that the original components (i.e., axes) of the data do not properly represent the relationship inherit in the data. Instead, the components can be combined to yield a new set of components, or axes, that better describe how the data is distributed.

The wiki link states that "PCA can be thought of as fitting an n-dimensional ellipsoid to the data". In that line of thinking, the output of PCA is the eigenvectors and eigenvalues that define the orientation and length of the ellipsoid that best fits the data.

What you do with that ellipsoid is up to you and your application of PCA.


Regression on multiple datasets with a per-dataset variable I have 10 datasets, each with the same variables (e.g., age and income) but different numbers of observations.
Let us now consider a categorical variable $X$ that can only take values $0$ and $1$ per dataset, meaning that it keeps the same value for all observations. For 5 datasets, $X=0$; for the other 5, $X=1$.
How do I create a regression model for a variable of these datasets (e.g., age) that takes into account this "meta-variable" $X$?
A simple solution would be to append a new column for $X$ to each dataset, where the same value is repeated for all observations, and then concatenate the datasets. However, I think there are better ways.

I believe adding the meta-variable X to you dataset and combining them is a good option.
However - why do you have 10 different datasets ? and not a single dataset to start with ?
Is there a special meaning attached to each individual dataset - which is why they are not combined in the first place ? Does that sound like an additional categorical variable you need so as to differentiate between individual datasets ?


What is the proper way to use early stopping with cross-validation? I am not sure what is the proper way to use early stopping with cross-validation for a gradient boosting algorithm. For a simple train/valid split, we can use the valid dataset as the evaluation dataset for the early stopping and when refitting we use the best number of iterations.

But in case of cross-validation like k-fold, my intuition would be to use each valid set of each fold as evaluation dataset for the early stopping but that means the best number of iterations would be different from a fold to another. So when refitting, what do we use as the final best number of iterations ? the mean ?

Thanks !

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model's ideal number of estimators compares to those of the cv iterations.

For example, if your validation performance rises sharply with additional estimators, then levels out, and finally decreases very slowly, then going too far isn't such a problem but cutting off early is.  If instead your validation performance grows slowly to a peak but then plummets with overfitting, then you'll want to set a smaller number of estimators for the final model.  And then there are all the other considerations for your model aside from straight validation score; maybe you're particularly averse to overfitting and want to set a smaller number of estimators, say the minimum among the cv iterations.

Another wrench: with more data, your model may want more estimators than any of the cv-estimates.  If you have the resources to experiment, also look into this.

Finally, you may consider leaving an early-stopping validation set aside even for the final model.  That trades away some extra training data for the convenience of not needing to estimate the optimal number of estimators as above.


Implication of Central Limit theorem and its application in machine Learning According to central limit theorem, irrespective of the distribution of the original/population dataset, the sampling distribution will follow normal distribution.
My Question is if the means of n samples of the population data follow normal distribution, what is this helping us in ? I tried to find examples in the context of Ml but found no good example. It would be of great help if you could help me understand the significance of CLT ( basically application of knowing that sampling distribution will have bell curve) in the context of ML with an example.

Are you familiar with hypothesis testing? Confidence Intervals ? Statistical Significance ? All of these concepts typically rely on the central limit theorem.
One concrete example I can think of relating to ML: Regression
Most approaches to regression typically assume that the error term that appears follows a Gaussian distribution. How do we justify this assumption ? Well the idea is that we can think of the error term as a sum of multiple different error terms.
Now even if these individual error terms do not follow a Gaussian distribution, we can model their sum as a Gaussian because of the Central Limit Theorem.


Similarity of search results using Jaccard I have a set of search results with ranking position, keyword and URL. I want to make a distance matrix so I can cluster the keywords (or the URLs). One approach would be to take the first n URL rankings for each keyword and use Jaccard similarity.

However, I also want higher position ranks to be weighted more highly than lower position ranks - for example two keywords that have the same URL in positions 1 and 2 are more similar than two keywords that have the same URL ranking in positions 39 and 40.

I saw suggested that I could add in higher ranks multiple times, for example:

keyword_1 = [1,1,1,1,1,2,2,2,2,3,3,3,4,4,5]

where the number is the id of the URLs that rank in positions 1-5 of keyword_1.

This means that I can't use for example sklearn Jaccard implementation because sets are assumed.

I had a go at implementing this myself and intuitively the results seem to make sense, but I would like it to run faster, as I could use data for rankings up to 100. There is a lot of looping involved - is there a way of using numpy better to make this code more efficient?

Alternatively, is there a different approach that I haven't found to use already built algorithms?

def jaccard_sim_with_dupes(item1, item2):
        intersection = 0
        for i in item1:
            if i in item2:
                intersection += 1
        for j in item2:
            if j in item1:
                intersection += 1
        union = len(item1) + len(item2)
        return intersection / float(union)

def make_jaccard_distance_matrix(X):
    shape = (len(X), len(X))
    jaccard_distance_matrix = np.zeros(shape)
    for i in range(shape[0]):
        for j in range(shape[1]):
            jaccard_distance_matrix[i][j] = 1 - JaccardDistanceMatrix.jaccard_sim_with_dupes(X[i], X[j])
        if (i % 100 == 0):
            print ("\r{0:.0%}".format(float(i) / shape[0]))
    return jaccard_distance_matrix

Here X is a list of each keyword, where each keyword is represented as a list of URLs, with higher ranked keywords added in multiple times.

X = [[1,1,1,1,1,2,2,2,2,3,3,3,4,4,5], [6,6,6,6,6,1,1,1,1,7,7,7,8,8,9]]

make_jaccard_distance_matrix(X)

returns

array([[ 0. ,  0.7],
   [ 0.7,  0. ]])

I realised if I used the format

X = np.array([[5,4,2,3,1,0,0,0,0],[4,0,0,0,0,5,3,2,1]])

and a sparse matrix from the scipy module, I could use efficient matrix calculations. This ran in about 2 seconds for 7k samples and 400 features.

from scipy.sparse import csr_matrix
import numpy as np

def jaccard_sim_matrix(X):
    """X is an integer array of features"""

    sparseX = csr_matrix(X)

    # make a binary version of the matrix
    binX = sparseX
    binX.data[:] = 1

    intersection = ((sparseX * binX.T) + (binX * sparseX.T))

    rowwise_sum = np.sum(sparseX, axis=1)
    union = np.repeat(rowwise_sum, intersection.shape[0], axis=1) + np.repeat(rowwise_sum.T, intersection.shape[0],
                                                                              axis=0)

    return intersection / union

Then of course I got the distance by 1 - similarity


How are the findings learnt from the data set are generalized compared to Statistics? I am new to data science/ machine learning world. I know that in Statistics we assume that a certain event/ process has some particular distribution and the samples of that random process are part of some sampling distribution. The findings from the data could then be generalized by using confidence intervals and significance levels.

How do we generalize our findings once we "learn" the patterns in the data set? What is the alternative to confidence levels here?

Bias-Variance decomposition is one way, and V-C dimension/bound is another...

Both of these are metrics you can use to get a feeling of how confident you should be that your training results will generalize to out-of-sample.

V-C dimension focuses on the results of this learning algorithm outcome.
Bias-Variance focuses on the expected outcome of the algorithm itself.

Pick your poison - I sincerely hope that this was helpful.


Understanding Exclusive-OR predictions in Elman network I have been reading Elman network paper, which can be found Here. in page 185, under Exclusive-OR section it was written as follows.

  Notice that, given the temporal structure of this sequence, it is only sometimes
  possible to predict the next item correctly. When the network has
  received the first bit-1 in the example above-there is a 50% chance that
  the next bit will be a 1 (or a 0). When the network receives the second bit (0),
  however, it should then be possible to predict that the third will be the XOR,
  1. When the fourth bit is presented, the fifth is not predictable. But from
  the fifth bit, the sixth can be predicted, and so on.

So, to give a context, author was explaining how we can use networks with memory to form a XOR Gate.

What i don't understand here is this sentence

  "When the network receives the second bit (0),however, it should then
  be possible to predict that the third will be the XOR,1"

How can we be sure that the third element is 1, given second element is 0. And again, why can't we predict 4th element.

I have watched This video and then realized that i missed the way the data is represented. So, in elman networks, the data is represented by 2-bit inputs at a time followed by 1-bit output. So, the first element is always unpredictable(so, 50% chance) and so is the 2nd term, but, once we get both 1st and 2nd term, we can for sure predict the third element as its a XOR gate.


Date transformation for KNN to get distance I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.

For example:

f1 | 1 | 2 | 3 | 4 | 01/01/2019

f2 | 10 | 3 | 12 | 1 | 14/01/2019

Does anyone have any recommendations?

What do you consider a meaningful distance? One possibility would be days since some reference date.

If seasonality also matters, you could additionally (or also) have calendar date or some sine or cosine functions with period 1 year/n for integer n - the idea being to get a periodic spline. The same kind of idea (or flagging weekday vs. weekend etc.) could apply to day of the week, if that makes sense for your problem.


How do I combine predictions from classifiers for two different problem? I am working on a classification problem for predicting whether the shipment is going to be late or not.
I would say the classifier is mediocre at predicting the positive class at the moment. But the ambition is to improve it.
However, after doing some analysis, I have found out that there is an important component (Customs) that has appeared as the cause of shipment delay in majority of FN.
Currently, I don't have a feature that would directly be associated to customs and be used in the model. Moreover, I think because of the product we are shipping, custom process may vary.
My original problem was at a shipment level, because of which I had to exclude the products in the shipment. But, now I want to include the products. It is a  many to many relation -> Shipment can have multiple products and Vice a versa.
Following is my thought:
To have a separate predictor in addition to the original one that would predict if the product in the shipment is going to be late/not based on planned days for customs.
This is where I am struggling, if this is a right approach how do I consolidate the predictions for both the models to come up with a single prediction as Late or No Late?
In addition to this I need to understand if there is another way to tackle this?

My intuition would be to try to integrate the information about the products directly in the original model. Typically the possible products in a shipment can be represented as boolean features (one hot encoding), but this part might need some feature engineering if there are too many different products:

simple option: only a small set of features representing types of products (I'm assuming that it's not the specific product which causes custom delays, it's the type of product)
advanced option: feature selection/extraction to reduce the number of features

Generally a joint model (a single model which deals with all the information at once) tends to perform better, in particular because in the other option errors in the first model propagate to the second one. Also the two models option doesn't allow the second model to leverage any specific feature from the first one.
Note that this is just my intuition, I could be wrong.
Side note: probably this is already taken into account but I guess that the value of the shipment is also an important factor for customs delays.


Which machine learning approach/algorithm do I choose for path validation? I apologize for lack of terminology, I'm no computer scientist.
I have a problem of validating paths in a directed graph with complex nodes.

The full description is the following:

I have a decent set (about 1K) of directed graphs;
Each node contains a complex data structure (it is a hierarchical data structure, not a picture or sound);
I have some of paths in those graphs known as "correct" paths (based
mostly on data in nodes);
And I have some of paths in those graphs known as "incorrect" paths
(with classification why it is incorrect).

I'd like to predict given a graph with those complex nodes and a path, is this path "correct".

Which machine learning algorithm will suit me best?
In general, what approach I should use?

Edit:

Each full graph is either have app paths processed
(correct/incorrect) or completely blank (no path is processed);
Correctness depends on both position of node in a graph AND data in
the node;
Humans would need heuristics to decide or guess which paths are
correct;
Most of the paths are "correct";
I hope to convert human heuristics to some kind of "correctness"
recognition.

This is a good question but it's rather complicated. I can suggest two approaches:

Graphical models; specifically Bayesian networks since your graph is directed.
Recurrent neural networks. Here's a talk on popular recent model: Sequence to Sequence Learning with Neural Networks.


Methods of disaggregating data to smaller units? I have a relatively straightforward question that I know poses some difficult challenges.
Let's say I have a state-level rate of X. I would like to disaggregate the state-level rate to the county-level. I realize this is can be dangerous (ecological fallacy), but I have seen some studies use the technique with a set of assumptions.
For example, if I know that each county is a certain proportion of the entire state population, I could take that proportion and multiply it by the state-level rate of X to get an (incredibly) naive county-level rate of X.
I'm trying to find more information on ways to make this approach 'less' naive, but I can't seem to get any momentum. I've tried using the terms 'disaggregating' and 'weights', but I can't seem to tap into the right body of literature.
Does anyone know of any methods/body of work that have attempted to handle this problem?

From a statistical point of view this is impossible if one doesn't have any data at the fine-grained level. Any statistical inference must be based on a sample from which specific patterns can be observed.
If there is no data at the fine-grained level, any calculation is based on assumptions. For example one may assume that a variable is proportional to population (linear relation). But why not assume that the variable is a polynomial function of the temperature? Or that it is related to the prevalence of a particular gene? The main issue is that without any data there's no way to test any of these assumptions, so no there can be not reliable conclusion.


Classifying Car Data By Year I have huge car photos.

I want to predict car's "brand-model-body type and production year"

First, I splitted data into train and validation, and I categorized them like this.

Every category has about 1000 train and 900 validation images.

My plan was: I train my keras model with these categories after training, model can predict labels like below:

audi a3 sedan 2008  => %25

audi a3 sedan 2009  => %25

audi a3 sedan 2010  => %25

audi a3 sedan 2011  => %25

And I can tell user that: "This car is Audi A3 Sedan 2008-2011"

My problem is, some of these categories have very similar photos. For example: audi a3 2009 and audi a3 2010 have same body type and there is not much difference between photos (No difference in reality).
Because of that, train accuracy has improved to about 0.9 but validation accuracy hasn't improved above 0.55

When I try some predictions, it usually gives same label, "Ford Focus sedan 2009" :)

Here is my output:

epoch,          acc,              loss,              val_acc,       val_loss
27,    0.7965514530544776, 0.56618134500483, 0.5192149643316993, 1.729015349846447

28,    0.8058803490480816, 0.5408204138258657, 0.5176764522193236, 1.778763979018732

29,    0.8167710489770164, 0.5116128672937693, 0.523258489762041, 1.7806432932022545

30,    0.8256544639818643, 0.4872381848016096, 0.5207534764479939, 1.8059904007678271

31,    0.8355546238309248, 0.4629556378035959, 0.5237253032663666, 1.8191414148756815

32,    0.8424464767701014, 0.4444190686917562, 0.5242512903147193, 1.8496954914466912

33,    0.8508739288802705, 0.422022156655134, 0.5303593149032422, 1.8565427863780883

34,    0.8576819265745635, 0.40545297008116027, 0.5262894901236571, 1.909881308499735

My train code is here:

Image_width, Image_height = 224, 224
num_epoch = 5000
batch_size = 16
learning_rate = 0.0001
model = ResNet50(weights='imagenet', include_top=False, input_shape=(Image_width, Image_height, 3))
fc_neuron_count = 1024
output = model.output
output = GlobalAveragePooling2D()(output)
output = Dense(fc_neuron_count, activation='relu')(output)
predictions = Dense(num_classes, activation='softmax')(output)
model = Model(inputs=model.input, outputs=predictions)

model.compile(optimizer=opt.Adam(lr=learning_rate), loss=losses.categorical_crossentropy,
              metrics=['accuracy'])

history_transfer_learning = model.fit_generator(
    train_generator,
    epochs=num_epoch,
    steps_per_epoch=num_train_samples // batch_size,
    validation_data=validation_generator,
    validation_steps=num_validate_samples // batch_size,
    class_weight='auto',
    callbacks=callbacks_list)

Am I doing something wrong? How can I achieve this result?
Should I change validation accuracy calculation, or should I give more photos per category?

I want to say how I solved my problem for anyone who is looking for a similar question.

My categorization was a mistake. I realized later that, I gave the same photos to my model and waited for the different results. For example, I had nearly same photos in Audi A3 Hatchback/5 2009 and Audi A3 Hatchback/5 2010. When the model starts training, first it learns data. Then it predicts and validates output itself. If the output is wrong, it tries a different way to success. But wait a minute, there was no mistake. I gave you the same photos and waited for different results! My categorization failed here.

I categorized my cars by body changes like Audi A3 Hatchback 2008-2013. Except for categories that have the wrong photos, my results are great for now.

Now, we will work on better photos and better year categorization.


A/B testing randomization step Let's say I want to measure a medicine’s impact on height. So I randomly broke down my user base into two groups (control and experiment, obviously). I calculated the average height for both groups before experiment and found out there’s a 2 cm difference. Then, after the medicine is applied to the experiment group, I found a 4 cm difference.

If I recall correctly, the t-test doesn’t handle selection bias, is there a test I can do to put the initial 2 cm difference between group into consideration?

As far as I understand you simply don't have to worry about that.

So, you have a sample split randomly in control and treatment. You measure something before treatment, and after, for the same individuals. Because you measure on the same individuals than you have paired measures: delta = before - after. You are interested to measure if the mean of delta for control sample is significantly different than the mean of delta for control sample. This is done with paired sample test known also as dependent test.

If you assume a normal distribution you can use paired t test. If you can't reasonably assume the normal distribution than you can use Wilcoxon signed rank test.

The idea of pairings in test is to eliminate the effect of confounders. What you computed can be the effect of a confounder, but if the sample was split randomly in control and treatment you should not worry about that, unless you have strong reasons to doubt the randomization procedure.  In the later case you perhaps should include confounders in the equation and take the route of random effects modeling.


spaCy - Text Preprocessing - Keeping "Pronouns" in text I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
    nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
    stops = spacy.lang.en.stop_words.STOP_WORDS

    if lowercase:
        text = text.lower()
    text = nlp(text)
    if remove_punctuation:
        text = [t for t in text if t.text not in string.punctuation]
    lemmatized = list()
    for word in text:
        lemma = word.lemma_.strip()
        if lemma:
            if not remove_stopwords or (remove_stopwords and lemma not in stops):
                lemmatized.append(lemma)

    return " ".join(lemmatized)

Consider below input string:

Input: You're very beautiful!

If I clean that text, using my method:

test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)

It will return: -PRON- beautiful

Is this a "good" way to clean the text? I notice that -PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).

Should the -PRON- also be removed from the text? And should I complete even more text preprocessing?

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes:

probably best for speed to load your "nlp" object outside of the function call
"-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the correct logic. Though, you should investigate whether your model improves if you change your preprocessing logic

if you use SpaCy to create your classification model, I'd recommend experimenting training a model without using any preprocessing. As far as I know, SpaCy handles some of this implicitly through feature engineering


Why are only 3-4 features important in my random forest? I am running a random forest regression with Python's Scikit-Learn, code's below (X - features, y - to be predicted).
# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 1)

# Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test  = sc_X.transform(X_test)

# Random forest
from sklearn.ensemble import RandomForestRegressor
rf =RandomForestRegressor(max_depth=2, n_estimators = 100, random_state=0)
rf = rf.fit(X_train,y_train)
pred_train = rf.predict(X_train)
pred_test = rf.predict(X_test)

I am running this code for randomly sampled 100k dataset, that has 60+ features. Each time when I check feature importance I get 3 to 4 variables as important (with one of them holding over 80% of importance), and others' importance is set to 0. It is not reasonable to me that only these are important for prediction and the rest is rubbish.
var_num = X_train.shape[1]
plt.barh(range(var_num), rf.feature_importances_, align='center')
plt.yticks(np.arange(var_num), variable_names)
plt.xlabel('Variable Importance')
plt.ylabel('Variable')
plt.show()

Is it possible that I am missing something? That some other parameters needed to be defined? Could this be caused by a high correlation between variables themselves? Or is it really that the rest of my features are useless..?

RandomForestRegressor has a parameter called max_features, which is the number of features to consider when determining the optimal split. You haven't explicitly specified this, so Python will use the default (auto) and consider all features.

Given that your trees are very shallow and you are considering all of the features to split, it would not surprise me that the strongest 3-4 are consistently popping up (the bagging process in random forests will cause some variation within this).

Decreasing max_features and/or increasing max_depth may yield a greater variety of "important" features.


Trainer Component Error in Tensorflow Extension : AttributeError: module 'user_module' has no attribute 'trainer_fn' I want to build a tfx pipeline on own dataset.
I could use some advice on the transform code that I wrote and understand better.

I've done the ExampleGen, StatisticsGen, SchemaGen, ExampleValidator, Transform, and having an error in Trainer components.

How I could fix these issues?

ERROR:

c:\lib\site-packages\tfx\components\trainer\executor.py in _GetFn(self, exec_properties, fn_name)
    128     if has_module_file:
    129       return import_utils.import_func_from_source(
--> 130           exec_properties['module_file'], fn_name)
    131
    132     fn_path_split = exec_properties[fn_name].split('.')

c:\lib\site-packages\tfx\utils\import_utils.py in import_func_from_source(source_path, fn_name)
     66       user_module = types.ModuleType(loader.name)
     67       loader.exec_module(user_module)
---> 68       return getattr(user_module, fn_name)
     69
     70   except IOError:

AttributeError: module 'user_module' has no attribute 'trainer_fn'

CODE:

def get_model(show_summary=True):

#one-hot categorical features
num_A = 4,
num_B = 3,
num_C = 2,
num_D = 8,
num_E = 12,
num_F = 4,
num_G = 16,
num_H = 26

input_A = tf.keras.Input(shape=(num_A,), name="A_xf")
input_B = tf.keras.Input(shape=(num_B,), name="B_xf")
input_C = tf.keras.Input(shape=(num_C,), name="C_xf")
input_D = tf.keras.Input(shape=(num_D,), name="D_xf")
input_E = tf.keras.Input(shape=(num_E,), name="E_xf")
input_F = tf.keras.Input(shape=(num_F,), name="F_xf")
input_G = tf.keras.Input(shape=(num_G,), name="G_xf")
input_H = tf.keras.Input(shape=(num_H,), name="H_xf")

fl = keras.Input(shape=(75,))
dense = layers.Dense(35, activation = "relu")
x = dense(fl)
x = layers.Dense(15, activation="relu")(x)
outputs = layers.Dense(1, activation="sigmoid")(x)

_inputs = [input_A, input_B, input_C, input_D, input_E, input_F, input_G, input_H]

model = keras.Model(inputs=inputs, outputs=outputs)

model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])

if show_summary:
    model.summary()

return model

The reason is tfx doesn't support Keras sequential model, but function API. After migrating to use function API, it works in fine in the trainer pipeline.

def get_model(show_summary=True):

#one-hot categorical features
num_A = 4,
num_B = 3,
num_C = 2,
num_D = 8,
num_E = 12,
num_F = 4,
num_G = 16,
num_H = 26

input_A = tf.keras.Input(shape=(num_A,), name="A_xf")
input_B = tf.keras.Input(shape=(num_B,), name="B_xf")
input_C = tf.keras.Input(shape=(num_C,), name="C_xf")
input_D = tf.keras.Input(shape=(num_D,), name="D_xf")
input_E = tf.keras.Input(shape=(num_E,), name="E_xf")
input_F = tf.keras.Input(shape=(num_F,), name="F_xf")
input_G = tf.keras.Input(shape=(num_G,), name="G_xf")
input_H = tf.keras.Input(shape=(num_H,), name="H_xf")

inputs_con = tf.keras.layers.concatenate([
input_A,
input_B,
input_C,
input_D,
input_E,
input_F,
input_G,
input_H])

dense_1 = tf.keras.layers.Dense(50, activation = 'relu')(inputs_con)
dense_2 = tf keras.layers.Dense(25, activation = "rely") (dense_1)
output = tf.keras.laters.Dense(1, activation = "sigmoid") (dense_2)
model = keras.Model(inputs=inputs, outputs=outputs)

_inputs = [
input_A,
input_B,
input_C,
input_D,
input_E,
input_F,
input_G,
input_H]

model = tf.keras.models.Model(_inputs, output)

model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])

if show_summary:
    model.summary()

return model


Percentage of smaller dataset with respect to bigger dataset I have two datasets, which are lists of multidimensional real-valued vectors. One dataset (call it $A=\{x_1, x_2, x_3, ..., x_n\}$ is of a big size, the other (call it $B=\{x_1, x_2, x_3, ..., x_m\}$). Furthermore, the other is far smaller and is a subset of the bigger one ($B \subset A$). The smaller one $B$, comes from some sampling process and what I want to do is to calculate, what fraction of the smaller (obtained from sampling) is in the bigger.
Additionally, since those are real-valued vectors, I can't compare them directly one by one, so a clustering algorithm may be employed. Also the size of one dataset is bigger than the other $|A| >> |B|$.

Naive approach: define a similarity or distance function, say for instance cosine similarity.

Calculate the similarity score between any pair $(x_i\in A, y_j\in B)$
Define a precision level, say $\epsilon=0.000001$. The assumption is that it's extremely unlikely that two vectors would be this close by chance in $A$.
For every $y_j\in B$, find the set $c(y_j) = \{ x_i\in A\ |\ sim(x_i,y_j)\geq 1-\epsilon \}$
Obtain the union: $C(B)=\{x_i\in A\ |\ \exists y_j\in B: x_i\in c(y_j) \}$

The proportion of elements of $A$ which are "equal" to an element in $B$ is:
$$\frac{|C(B)|}{|A|}$$


mini batch vs. batch gradient descent In batch gradient descent, it is said that one iteration of gradient descent update takes the processing of whole entire dataset, which I believe makes an epoch.On the other hand, in mini batch algorithm an update is made after every mini batch and once every mini batch is done, one epoch is completed. So in both cases, an epoch is completed after all  the data is processed.I do not quite get what makes mini batch algorithm more efficient.

Thanks,

In short, batch gradient descent is accurate but plays it safe, and therefore is slow. Mini-batch gradient descent is a bit less accurate, but doesn't play it safe and is much faster.

When you do gradient descent, you use an estimate of the gradient to update your weights. When you use batch gradient descent, your gradient estimate is 100% accurate since it uses all your data.

Mini-batch is considered more efficient because you might be able to get, let's say,  an ~80% accurate gradient with only 5% of the data (these numbers are made up). So, your weights may not always be updated optimally (if your estimate is not so good), but you will be able to update your weights more often since you don't need to go through all your data at once.

The idea is that you update your weights more often with an approximation of your gradient, which often is good enough. The utility of mini-batch becomes more obvious when you start dealing with very large datasets.


Does the performance of neural networks depend on the method used to unroll weights ? Lets say we have weights(theta1 and theta2) of neural net as:

theta1 =[1, 2, 3]
theta2= [4, 5, 6]

If we unroll these weights into a single dimension array in matlab/octave ,we get:

theta = [theta1(:);theta2(:)]
%theta = [1, 4, 2, 5, 3, 6]

If we unroll these weights in a slight different way , for example consider this python code:

theta = np.array(theta1,theta2)
  theta = theta.ravel()
  #theta = [1,2,3,4,5,6]

I have implemented a neural network using gradient descent in octave and it worked fine but the same implementation is not working in python(% accuracy = 10%). The only thing different the python implementation is the way the weights are unrolled.
So, does the performance of neural network really depend on the way the weights are unrolled ?

I don't really understand your notation and don't have a lot of experience with unrolling, but from my understanding there can only be one way of transforming your 2 weights (theta1 & theta2) to a single one (theta).

If you adapt your python code to something like:

theta = np.array(theta1,theta2)
theta = np.ravel(theta, order='F')
#theta = [1, 4, 2, 5, 3, 6]

Does it work now? If so, you can't just choose how to "unroll" your weights.


How can I do a train test split for an unbalanced panel data set in Python? I have an unbalanced, panel pandas data frame.
I would like to split this data into a training set and a testing set. Python's train_test_split method will not work because it does a random split, and so, it will likely places observations from t + 1 into the training set, and observations from t into the test set.
Which, of course, makes no sense, because the future cannot predict the past.
TimeSeriesSplit will also not work because this function does not take into consideration the panel dimension of my data set.
Is there an easy way to do a train test split on unbalanced panel data sets? This split should (1) take into consideration the panel dimension of the data set, and (2) place earlier observations in the training set and later observations in the testing set.

Not sure what do you mean by panel dimension, I think would be best to clarify it more. Nonetheless, I can tell you for your 2nd question Stratified Sampling takes into account the imbalance. And if you want to have one group t on the training and t+1 on the test you can use group sampling. Sklearn has both implementations [1] [2].