What will be the input_shape of tf.keras.layers.Conv3D be for these inputs I have many videos, and each video is made up of 37 images (there are 37 frames in the whole video). And the dimension of each image is (100, 100, 3).... So the shape of my dataset is [num_of_videos, 37, 100, 100, 3]
If I want to pass these videos thorugh the tf.keras.layers.Conv3D()... what will be the input shape for the conv3D be:
Conv3D(32, 3, padding='same', activation='relu', input_shape=[What will the input shape here be])

From Keras document,

Input shape
5+D tensor with shape: batch_shape + (channels, conv_dim1, conv_dim2, conv_dim3) if data_format='channels_first' or 5+D tensor with shape: batch_shape + (conv_dim1, conv_dim2, conv_dim3, channels) if data_format='channels_last'

Assuming data_format='channels_last' and num_of_videos may be used to decide batch_size(which is not required while defing input_shape),
input_shape=[ 100, 100, 37, 3 ]


If I have got different routes - a series of (lat, lng) points, how to get the similarity of different routes? It is a real world use case. For example, a route from place A to place B can be different series of lat, lng points - the different trips though they are exactly the same sequence from Street x and then Road y then High way z.

The differences are the locations are reported in different time (for example each trip reports the location 1 minute) and the Vehicles appear in the different lane of the same street.

So, do you have some ideas about how to compute the similarity of two different trips belong to the same route (kinds of mapping different trips to the same route ).

I would go about this problem using the following approach:

Create a poly-line for the baseline trip.
Create a buffer zone (closed polygon) around this poly-line using a small radius, say 20 meters.
Using the points in the second trip, calculate the fraction of points that lie outside the buffer zone.

A fraction of zero means that all points from the second trip lie inside the first trip's buffer zone, while a fraction of one means that all lie outside.

You can use this fraction as a measure of dissimilarity, where 1 means that both trips are completely dissimilar and 0 means completely similar (within the given buffer radius). For completeness, you might want to reverse the poly-line roles and calculate the "reverse" similarity. Your final similarity score could then be the product of both.

I usually implement my algorithms in C# using .NET and for these purposes I use Microsoft's System.Spatial NuGet package. Here you can find methods like STBuffer and STContains that will greatly help to implement this.


In model validation (log regression), can AUC for the test sample be higher than that for the train sample? I have a relatively simple model (with 8'000 cases, 5 predictor variables) predicting a dihotomised outcome. It has an AUC of 0.82 (95%CI 0.82-0.83). Prediction in temporal (N=240) and external validation (N=170) datasets had AUCs of 0.85 (95%CI 0.81-0.88) and 0.76 (95% CI 0.68-0.84%). A reviewer told me it is impossible (can't happen, theoretically) for the temporal validation dataset to have higher AUC. Is this true?

This is just not true. Let's say that your future data is way easier to classify, then it can happen than temporal splits imply higher AUC. It often happens that future data is harder to classify because of shifts in the distributions of the variables, but if distributions are stable I don't see why not.


Datascience ML Guide I have started learning ML and I am stuck at finding a problem solution. I need the steps to follow up:
Below are the formats of data sets:
    cust_Demographics

Customer_ID         0
Nationality         0
Income_Range        0
Job_Type            0
Marital_Status      0
Gender              0
State               0
Language            0
Loyalty_Status      0
Age                 0
Points              0
recent_tran_date    0
Recency             0
custSince           0
Frequency           0
Monetary            0
90999

Customer_Transaction

Year                0
Week                0
Store_Code          0
City_Name           0
Transaction_Type    0
Customer_ID         0
Invoices            0
Item_Count          0
Revenue             0
Discount            0
Units_Sold          0
500000

Customer_Store

Region              0
Region_Code         0
Store_Code          0
Store_Name          0
Mall_Name           0
Sales_Per_Day       0
Store_Size_Sq_Ft    0
Customer_Count      0
Total_Revenue       0
38

Test
Customer_ID    0
Store_Code     0
400000

Problem: Need to predict whether a customer goes to a store(new stores) or not.(0/1).

So , I merge Demographics,Transaction and store table.

Q1 : I have total rows= 500000 and added one target column as 1(represents they went for shopping) so all the values are 1's for this column so how should I train to predict the target value.

Q2 : If I merge Test,demographics,Store it is ok but how can I use the features of transaction table. customer->store (A Customer shops for that store)

Or can it be related to the recommendation based problem?

Please go through mentioned link to explore the Recommendation rule.

https://medium.com/recombee-blog/machine-learning-for-recommender-systems-part-1-algorithms-evaluation-and-cold-start-6f696683d0ed

https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/

https://www.datacamp.com/community/tutorials/recommender-systems-python

https://www.kaggle.com/gspmoreira/recommender-systems-in-python-101


What is the upscaling factor in super resolution with deep learning? I have been reading papers on single image super resolution (SISR) and I frequently encounter X3 upscaling factor, X4 upscaling factors.
Example: SRGAN mentioning x4 upscaling factor
It would be wonderful if anyone could explain it in simpler words. Thanks

The upscaling factor simply denotes how much the original will be upscaled. In the case of SRGAN, which uses 4x upscaling, this simply means that the output image will be four times as large as the input image. An input image of size 720 by 1080 pixels will be of size 2880 by 4320 pixels after being upscaled using SRGAN.


How to find the input variables for a classification problem? I am working on a  classification problem. I have 1000+ features in this dataset. I don't know how to select the right variables/ features that can actually contribute to predicting the output. What are the different methods through which I can identify the important variables that can be used out of these 1000+ variables.

Two approaches come to mind:

Use all of them and perform feature selection to identify indicative features. Algorithms include Information Gain and Mutual Information.
Hand-pick some features you deem indicative intuitively, or delete the ones you deem non-indicative.

Generally, you have some sort of hypothesis that you are testing where you attest the capability of (a subset of) your features (and the algorithm you'll use) to uniquely identify the label/class they belong to. The features you use are one of the most important things in the classification task. So it is good to spend time to intelligently find out which features contribute.

For this task, you do have a lot of possible features so the automatic feature selection approach might be better.


How word2vec understands the relationship between numbers? I am using word2vec to train a big corpus of textual information. While analyzing the scatter plot of words I found that it has not only grouped numbers together but also ordered them based on their value. I am able to understand that it groups together words sharing similar context but not able to understand how it got the values also correctly. Can someone please give an intuitive explanation of how this happens? I am attaching the image where I have zoomed in on the numbers.

Attempting an answer from the understanding I have of word2vec and deep learning through my personal experience and some live presentations of Yoshua Bengio and Yann Le Cun that I had the chance to attend.

As mentioned in the question, I think too that numbers are grouped together as they are have similar functions in the sentences where they are found.

For the reason of how they get ordered, as said in my comment:
Intuitively I would say that it will be true for small numbers and less true for big numbers (the way 7-9-8 are ordered seems to go that way). When we speak we often say things like: "I ate 1 or 2 apples" or "I will buy 2 or 3 things" but rarely "I will buy 3 or 2 things". I think this is how it ends up ordering the numbers. Bigger numbers will appear less in a corpus so they will most probably not be ordered as precisely.

When looking at tutorials about word2vec, an example that is often given is that, if we see the relationship between 2 words as a vector, the final representation of the words by the model will give that the vector linking "man" to "woman" is approximately the same as the one linking "king" to "queen". Or that the vector linking "Paris" to "France" is approximately equal to the one linking "Rome" to "Italy". This even allows to do operations such as: France + Italy - Paris = Rome! The clear mechanics of how these relationships are so accurate (compared to other approaches) are, as of today, still obscur to the scientific community.

If these kind of operations works for countries and main cities, it seems logical that the vectors "3-2" and "2-1" point in the same direction, i.e. that 1-2-3 are ordered (given that, as said previously, consecutive numbers are more likely to be used together in a sentence). Of course, to achieve this, the numbers have to frequently appear in the training corpus.

I am very happy that someone showed this result with numbers because the way word2vec has been trained in most of the models that are available online did not includ digits. They were replaced by some symbols i.e., 12.34 becoming ##.##, and such observations cannot be made.

Any suggestion to improve this answer is welcome!


Training a regression algorithm with a variable number of features I need to train a regression algorithm with multiple features and a single label (predicted value). The problem is that this algorithm has to be able to do on-line learning and the number of features it will receive will vary. Let me give a clear example:

The algorithm is trained on a dataset of shape:

[--------Features-----------------] [Label]
[-- context11 -- | -- context12 --] [label1]

Then, for the next training example, one of the contexts might be missing, so the training example might either be:

[--------Features-----------------] [Label]
[-- context21 -- ]                  [label2]

or

[--------Features-----------------] [Label]
                  [-- context22 --] [label2]

How can I deal with this situation? So far, I thought about two possibilities:

Replace the missing part of the features with zeros. I am not sure how this will affect the algorithm though.
Use a decision tree or random forest? Do these have a more natural way of dealing with a variable number of features?

Any other ideas?

I think in the specific case that you have given as an example, you can use approach 1 which you have mentioned above i.e.

either replace the missing features with a zero value

or otherwise you can also

implement your algorithm in a sparse manner so that it doesn't need to always have all the features for an instance.

Using a sparse format will not affect the runtime (or any other aspect) of your algorithm. Whereas, if you use the zeroing features approach, then your algorithm will have to do the unnecessary multiply by zero operations and this will increase your runtime ( How much? that depends on how sparse is your actual data).

Hope that helps.


What is the optimal value of a Markov Decision process with Single actions at each state? I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:

There are two states ($a$ and $b$) stepping to $a$ is terminal.
All rewards are zero, discount for stepping from $b \to b$ is $1$, and discount for all other steps is zero
The possible scenarios are: $a\to b$ (probability $1$), $b\to a$ (probability $p$) and $b\to b$ (probability $1-p$).

The first question I have is whether or not its true that the optimal values for each state here are zero? If not how did you derive this?

Second question I have is if we want a parameter $\lambda$ and have a feature $\phi$ such that

$\lambda \times \phi(a)$ and $\lambda \times \phi(b)$

approximate the optimal values at the states a and b and we attempt to approximate such a $\lambda$ by means of TD(0) starting with $\lambda_0 = 1$ how can I find the expected value of $\lambda$ after one episode of updating in terms of $p$? ($E[\lambda_T]$ where $T$ is a random number representing the duration of the episode)

By an episode I mean if we go back from state b to a the episode is over.

Hi guys thanks for your comments and sorry for the slow reply. I cannot reply to them directly because I am not allowed.

@Constantinos, to your point regarding b: b-->b, b-->a being different actions.  I think you misinterpreted what I am asking. There is only one action (what you call it is arbitrary) but two possible outcomes.

@Neil Slater, Yes I think the question is badly written. An MRP is defined as an MDP with only one possible action in each state (or no action dependant on how you look at it). Reward is a real value that is the output of a function that maps a pair of states (so say (a,b) or (b,b) etc ) to a real value. Discount is defined similarly.

Really the crux of the question is: find a linear function approximation parameter $\lambda$ such that $\lambda* \phi(s) \approx v(s)$ where $v(s)$ is the value associated with a certain state $s$. I believe that such values are defined as $v(s) = \sum_{s'}P(s,s')(R(s,s') + \gamma v(s'))$ where $R(.,.)$ is a reward transition function and $P(.,.)$ is a function mapping state pairs to probabilities.

I think the main reason I am lost with this is because the question itself is pretty ambiguous so I am trying to fill in the gaps heuristically.

Thanks Guys


Could data from a test set 'leak' into predictor during PCA? After reading this article I have got a question about PCA.

Author was talking about whether to use test set while computing PCA.

  But, few important points to understand:

  1) We should not combine the train and test set to obtain PCA components
  of whole data at once. Because, this would violate the entire
  assumption of generalization since test data would get ‘leaked’ into
  the training set. In other words, the test data set would no longer
  remain ‘unseen’. Eventually, this will hammer down the generalization
  capability of the model.

  2) We should not perform PCA on test and train
  data sets separately. Because, the resultant vectors from train and
  test PCAs will have different directions ( due to unequal variance).
  Due to this, we’ll end up comparing data registered on different axes.
  Therefore, the resulting vectors from train and test data should have
  same axes.

Author mentioned "Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set. In other words, the test data set would no longer remain ‘unseen’."

As I understand it... test data can only affect which PC(s) we will choose ( because of changed variance) but data can't leak really( let's say only metadata leak). For each observation we still have data only from it's own previous variables(predictors) just in lower dimensionality(projected). Am I right? or during calculations of matrixes data from test set can leak into?

The author is right.  You are wrong.  There's no such thing as "only metadata leak" or "can't leak really".  It's like saying "not pregnant really" -- either you're pregnant, or you're not.  Same here -- either data can leak, or it can't.  In this case, data can leak.  Maybe only partial data, but it's hard to know just how partial and just how bad the impact of that might be.  It might be very bad, or it might not; but since you can't know, testing with a methodology where you know the results might be meaningless isn't a good idea.


Output data from scikit learn logistic regression I have run a logistic regression using scikit learn in python.  I know want to output the results to put into a csv and then load into Tableau.  To do that I need to combine the y_test, y_actual, and X_test data.  I was wondering if there's a way to output the y_test, y_actual, and X_test data?  I know I can use to_csv, but when I try that I can extract each of these data and concat together, but I'm afraid its not matching correctly since there's no identifying to join on.

Alternatively, is there a way to keep a unique id (uid) with the logistic regression so then it's easy to see what the regression predicts for a specific person?

Each line is treated independently at prediction, so you can be sure that the data is kept in the same order.

For simplicity, you can keep your data in a pandas dataframe. Here's a short working example:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100, n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = LogisticRegression()
clf.fit(X_train, y_train)

data_test = pd.DataFrame(data=X_test, columns=['f{}'.format(i) for i in range(1, 11)])
data_test['y_test'] = y_test
data_test['y_pred'] = clf.predict(X_test)


You can approach that problem as time-series prediction problem.
Below is one of articles or tutorials about time-series forecasting/prediction.

https://machinelearningmastery.com/time-series-forecasting/


Python SkLearn Gradient Boost Classifier Sample_Weight Clarification Using Python SkLearn Gradient Boost Classifier - is it true that sample_weight is modifying how the algorithm penalizes errors made on that particular class, rather than feeding more data into the trees by oversampling from that class. If you have any links to code that confirms this that would be ideal. Thanks

The sample_weight parameter is used both to weigh the splits as well as to weigh the errors.

Errors

For example, using the LeastSquaresError, the implementation uses the sample weights thusly:

def __call__(self, y, pred, sample_weight=None):
    if sample_weight is None:
        return np.mean((y - pred.ravel()) ** 2.0)
    else:
        return (1.0 / sample_weight.sum() *
                np.sum(sample_weight * ((y - pred.ravel()) ** 2.0)))

So if we provide large weights to certain samples, their individual error will have a bigger influence on the overall error for that sample set. The squared prediction error for each sample is scaled by by the corresponding weight, and the sum is normalised by the sum of the sample weights.

If no weights are provided, (leaving the default sample_weight=None), then they are initialised to a vector of ones, example here.

Splits

In the case that you set the parameter min_weight_fraction_leaf upon class instantiation, requiring each leaf to obtain a minimum fraction of the total sum of weight to indeed become a leaf, then sample_weight is also used to scale that value.  You can see it here:

    # Set min_weight_leaf from min_weight_fraction_leaf
    if self.min_weight_fraction_leaf != 0. and sample_weight is not None:
        min_weight_leaf = (self.min_weight_fraction_leaf *
                           np.sum(sample_weight))


What is the current state-of-the-art within aspect-based sentiment analysis? I've been reading Bing Liu's book on Sentiment Analysis. He mentions all of these slightly different approaches seen in research since 2004, but doesn't talk much about efficacy at all.

That leaves me - someone who has not done any sentiment analysis before - wondering what approaches are seeing the best results currently. And it specifically needs to be an approach that can extract the sentiments of individual topics found in the text, not just if a document/sentence is positive or negative.

In case it makes a huge difference: the object of analysis will be reddit comments, not tweets or reviews which seem to be the most common source of data.

Basically, there are two types of sentiment analysis.

Symbolic approach - the sentiment of the sentence is classified based on the kind of words used in the sentence. Involves a lexical database such as wordnet and some rules based on the grammatical structure of the language as knowledge representation.
Vector models - rather than detecting the sentiment in a sentence, the query sentence is classified into one of the category based on the training dataset and the features are represented as word vectors. Distributed and Distributional representations are types of it.

Whether it is coarse-classification (positive - negative) or fine-classification (types of emotions in it), a training dataset is needed if you are going to approach it via vector models as it is much more easier to play with.
The skip-gram model and continuous bag of words model developed in Google based on Recurrent neural network language model made a breakthrough in Natural language processing seems to be the state-of-art model in Distributional representation.
Since the code has been open-sourced, the c-code is available here and the python library is maintained here.
It doesn't matter if you are going to use reddit comments, tweets or reviews. What matters is how well the sentences are classified for the training data. Because garbage in is garbage out.


How can we use machine learning to distnguish between similarly looking images How can I build a model which can distinguish between Milk and Phenyl? I want to predict whether a given item is edible to eat or not. If I train a model with thousands of photos of Milk and Phenyl which are labelled, Won't the model get confused between them because they look very very similar? No matter how much I train, I will be getting poor results, won't I? Another such scenario is in the case of salt and sugar.

I want to know if there is a way to tackle this situation. Please let me know if I can achieve better results with any other approach.

Also Please let me know If I'm unclear.

From the images alone, assuming the links you posted are representative, I doubt a model will ever be able to predict better than random guessing.

There is a chance the model could effectively learn to read the label, but it seems like a long (and somewhat pointless) path to go down, unless you will only be predicting on labelled bottles in the future!

You will need to introduce different information with each sample, perhaps the price, amount of liquid and other such feature. At this point, you will likely want to ditch your CNN for images and instead use a standard classifier, such as Support Vector Machines or a simple feed-forward neural network.


Which openCv function can be used to compute BEV perspective transformation given a point coordinates and the camera extrinsics/intrinsics? I have the 3x3 intrinsics and 4x3 extrinsics matrices for my camera obtained via cv2.calibrateCamera()
Now I want to use these paramenters to compute the BEV (Bird Eye View) transformation for any given coordinates in a frame obtained from the camera.
Which openCv function can be used to compute the BEV perspective transformation for given point coordinates and the camera extrinsics and/or intrinsics 3x3 matrices?
I found something very related in the following post: https://deepnote.com/article/social-distancing-detector/ based on https://www.pyimagesearch.com/2014/08/25/4-point-opencv-getperspective-transform-example/,
they are using cv2.getPerspectiveTransform() to get a 3X3 matrix, but I don't know whether this matrix represents the intrinsics, the extrinsecs or something else. Then they are transforming the list of points using such matrix in the following way:
#Assuming list_downoids is the list of points to be transformed and matrix is the one obtained above
list_points_to_detect = np.float32(list_downoids).reshape(-1, 1, 2)
transformed_points = cv2.perspectiveTransform(list_points_to_detect, matrix)

I really need to know if I can use this cv2.perspectiveTransform function to compute the transformation or if there's another better way to do this using the extrinsics, the intrinsics or both, without having to reuse the frame, since I already have the coordinates of people detected in the frame saved into an array.
Somebody just told me I can reach this if I have at least 3 3D points by using the cv2.findHomography function. I'm trying this approach now. Does anyone else have another input?

After a deep investigation, I found out a good solution:
The projection matrix is a multiplication between theextrinsic and the intrinsic camera matrices

https://medium.com/analytics-vidhya/using-homography-for-pose-estimation-in-opencv-a7215f260fdd
Since the extrinsic is a 4x3 matrix and the intrinsec is a 3x3 matrix, but we need the projection matrix to be a 3x3 matrix, then we need to convert the extrinsic into 3x3 before performing the multiplication.

cv2.getPerspectiveTransform() gives us the Projection Matrix when we don't have the camera params:

https://towardsdatascience.com/a-hands-on-application-of-homography-ipm-18d9e47c152f

cv2.warpPerspective() transforms the image itsef.
For the problem above we don't need these two functions since we already have the extrinsics, the intrinsecs and the coordinates of the points in the image.
Considering the presented above, I wrote a function to transform into BEV a list o points list_x_y given the intrinsics and the extrinsics:
    def compute_point_perspective_transformation(intrinsics, extrinsics, point_x_y):
    """Auxiliary function to project a specific point to BEV

        Parameters
        ----------
        intrinsics (array)     : The camera intrinsics matrix
        extrinsics (array)     : The camera extrinsics matrix
        point_x_y (tuple[x,y]) : The coordinates of the point to be projected to BEV

        Returns
        ----------
        tuple[x,y] : the projection of the point
    """

        # Using the camera calibration for Bird Eye View
        intrinsics_matrix = np.array(intrinsics, dtype='float32')
        #In the intrinsics we have parameters such as focal length and the principal point

        extrinsics_matrix = np.array(extrinsics, dtype='float32')
        #The extrinsic matrix stores the position of the camera in global space
        #The 1st 3 columns represents the rotation matrix and the last is a translation vector
        extrinsics = extrinsics[:, [0, 1, 3]]

        #We removed the 3rd column of the extrinsics because it represents the z coordinate (0)
        projection_matrix = np.matmul(intrinsics_matrix, extrinsics_matrix)

        # Compute the new coordinates of our points - cv2.perspectiveTransform expects shape 3
        list_points_to_detect = np.array([[point_x_y]], dtype=np.float32)
        transformed_points = cv2.perspectiveTransform(list_points_to_detect, projection_matrix)
        return transformed_points[0][0][0], transformed_points[0][0][1]


Convert categorical data in numeric preserve euclidean distance I m looking how to preserve Euclidean distance with categorical attribute.

Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "female" ad "male" for gender, how can i do for analysis?

I seen that i can trasform in 0 and 1, but for me dont have more sense. Why i can't choose 10 and 20 like number for male and female?
I Wish that this value in my analysis take a sense.

Sameone have to suggest or explain a great tecnique?

If i understand your question correct, you are misusing the word categorical. Categorical is always a 0 or 1 in their respective indices.

for example:

Data - [M, F, M, M]

Categorical Data: [[1, 0], [0, 1], [1, 0], [1, 0]]

If three types of classes are there, then it would be a 3 arrayed input for each datapoint.

If you feel that having the numbers 10 and 20 for Male and Female is meaningful to you, then you can go ahead and use it. There's nothing wrong in that. But when you want to finally train on the Data, say LSTM, it prefers taking in the categorical data.

But if you are talking about the input attributes, then you need not worry about the 0-1 problem, Just use as they are.

Vote up, if you find this helpful ;)


Could principle component decomposed coordinates value be correlated to each other? I am wondering if we have a A= n*p matrix of samples and we run a PC decomposition on it.

Say the eigenvector matrix is E, so the samples in the eigenvector space should be

  B= A*(E)^-1

So, I am wondering if there is any rule that the columns of B to be correlated? That is, the first PC loadings and the second PC loadings to be correlated?
 Thanks!

PCs are calculated based on eigenvectors (e1 nad e2 in your case) of correlation or covariance matrix. Eigenvectors are orthogonal, so PCs are uncorrelated. If you would like to check the proof, you can find it on page 432 in Applied Multivariate Statistical Analysis 6th edition written by Richard A. Johnson and Dean W. Wichern.


GridSearchCV with MLPRegressor with Scikit learn I'm trying to apply automatic fine tuning to a MLPRegressor with Scikit learn. After reading around, I decided to use GridSearchCV to choose the most suitable hyperparameters. Before that, I've applied a MinMaxScaler preprocessing. The dataset is a list of 105 integers (monthly Champagne sales).

The problem is that for some reason the GridSearchCV isn't operating (at least correctly, I think). When I print the parameters used by the model, appear some values out of the range defined in param_list.

Furthermore, i know that the dataset is too small for a MLP, the idea is program the model now and use it later in a larger dataset. Although, the final dataset is not very large, so I'll be very thankful of hear any idea to improve the accuracy of the model in a small datasets!

Thanks!

Code:

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import TimeSeriesSplit
from sklearn.model_selection import GridSearchCV
from matplotlib import pyplot
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error
import pandas as pd

dataset = pd.read_csv('champagne.csv', header=None)

scaler = MinMaxScaler()
scaled_dataset = scaler.fit_transform(dataset)

mlpr = MLPRegressor(max_iter=7000)

param_list = {"hidden_layer_sizes": [1,50], "activation": ["identity", "logistic", "tanh", "relu"], "solver": ["lbfgs", "sgd", "adam"], "alpha": [0.00005,0.0005]}
gridCV = GridSearchCV(estimator=mlpr, param_grid=param_list)

splits = TimeSeriesSplit(n_splits=3)

pyplot.figure(1)
index = 1

for train_index, test_index in splits.split(scaled_dataset):

    training_set = scaled_dataset[train_index]
    testing_set = scaled_dataset[test_index]

    train_index_array = train_index.reshape(-1,1)
    test_index_array = test_index.reshape(-1,1)

    gridCV.fit(train_index_array, training_set)
    predicted = gridCV.predict(test_index_array)
    parameters = mlpr.get_params()

    test_mse = mean_squared_error(testing_set, predicted)

    pyplot.subplot(310 + index)
    pyplot.plot(predicted)
    pyplot.plot([None for i in training_set] + [x for x in testing_set])
    index += 1

    train_index.flatten()
    test_index.flatten()

It would be helpful to get the ouput of the program (or at least the error thrown)

However, MLPRegressor hidden_layer_sizes is a tuple, please change it to:

param_list = {"hidden_layer_sizes": [(1,),(50,)], "activation": ["identity", "logistic", "tanh", "relu"], "solver": ["lbfgs", "sgd", "adam"], "alpha": [0.00005,0.0005]}

https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html


In CNN (Convolutional Neural Network), does the combination of previous layer's filters make next layer's filters? I know that the first layer uses a low-level filter to see the edge information. As the layer gets deeper, it will represent high-level (abstract) information. Is it because the combinations of filters used in the previous layer are used as filters in the next layer? ("Does the combination of the previous layer's filters make the next layer's filters?) If so, are the combinations determined in advance?

I guess you are making mistake about the filters. After applying filters to the input of each layer, the output will be used as the input of the next layer. The first layer's filters try to find the edges in the image and their output show whether those edges exist in a specified position or not. Next layer filters try to find patterns in the outputs of the previous layer which shows the existence of edges. Due to the point that each filter is a window and specifies a receptive field on the input, it finds patterns in the input which are more abstract and more complicated than the previous layers' activations.


Can I use an ANN to translate image output from one sensor to simulate output from another sensor? Say, for instance, if I had image data from one high resolution digital camera and wanted to make it look like it was taken from another, lower resolution, digital camera?  Would training input/output pairs of overlapping images be a good way to do this?  What is this technique called?

For example, say I wanted to be able to count benches in parks in LOW resolution imagery.  Could I go through these sample images and create an appropriate dataset of high and low resolution pairs to train a network to learn what a low resolution bench looked like?  Would I be able to discern low resolution benches if my training set was incredibly diverse (image chips if entire city parks vs individual objects like fountains, trees and statues)?

I like this example because the images come from different sensors as well as being different resolutions. Some of my research has led me to super resolution, which is kind of the opposite of what I'm trying to do.

As for the amount of data, it would be painstaking but not technically difficult to get overlapping high and low resolution imagery.

This is very much possible. There is a function which can map the images from the higher resolution pictures to the lower dimensional ones; and a neural network can be trained to learn that function.

However, to train a neural network to do this you will need thousands of images from both cameras. Then you can feed the pictures taken with your higher resolution camera as the input to the network and then compute the loss of the network at the output with the corresponding lower resolution images.

If you do not have so many images, there has been work on taking images and applying some sort of filter to change their appearance. These techniques are often called style transfer, you can find some tutorial here and code which I have tried and can confirm works here. It might be hard to get a representative image to use as the style image using your old camera. You can try an average of a few pictures, or a picture of a white background, you would have to try things out, I do not know what would work in this case.

If you share examples of your data we can help you more.


Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3? I'm studying Pattern Recognition and Machine Learning by Christopher Bishop. What I realized is, he defines values of λ as ln λ. For example:

  We see that, for a value of lnλ = −18, the over-fitting has been suppressed and we
  now obtain a much closer representation of the underlying function sin(2πx). If,
  however, we use too large a value for λ then we again obtain a poor fit, as shown in
  Figure 1.7 for lnλ = 0

What is the reason for this? Why he doesn't just use λ?

I can just suppose, that it's because then we could consider regularization together with a Log Likelihood function, for instance, and that is why it's more convenient to have such a representation. It's easier to make computations, using log, for example, when we want to minimize some functions (ln a + ln b = ln(a*b), (ln a)' = 1/a etc)


How does "one-to-many" RNNs work? I recently came across an article about RNNs here. Which describes different types of RNNs like:

The first figure makes sense. A regular feedforward network.
The second is a big question for me. Is it one timestep cloned three times to make it recurrent with 3 timesteps?
The third is returning just the last timestep's output.
The fourth is an even bigger question. Is it getting the last timestep's output, clone it three times, and put another RNN layer on it? Or it's two RNN layers, where the output of the first layer is the input of the second (returning all timestep's outputs)?
The fifth makes sense as well, it returns the output of all timesteps.
So am I missing something, or the second, and fourth cases can only be made by cloning inputs/outputs as specified above?

I think this will help.

My understanding is that one-to-many and many-to-many(like in 4th case of your pictorial) are in a way similar to autoregressive networks, where you utilize the prediction you've made and use it predict further ahead.


Keep word2vexc/fasttext model loaded in memory without using API I have to use Fasttext model to return word embeddings. In test I was calling it through API. Since there are too many words to compute embeddings, API call seems to be expensive. I would like to use fasttext without API. For that I need to load the model once and keep it in memory for further calls. How can this be done without using API. Any help is highly appreciated.

Surely, I will good idea to write embedding on some database like SQLite,Postgres or RocksDB. I highly appreciate using RocksDB as it is store data in the form of a key-value pair which is the best suit for your use case. Maybe it will take little more time but your main issue will get resolve through it. If u directly deal with python than there is support of SQLite and Postgres. If u work with java or .NET than RocksDB or LevelDB is supercool. I hope this explanation will help you. Please have review my answer for better reach.


Can I use the training rows multiple times while training with different labels attached to it? If I have a data set where for every text message,  same but two labels are given. It could be that only one label has been filled. To visualise this scenario in real life, one may classify the accent to belong to 'US English' and 'UK English'.

Here is an example where fro every text, atmost 2 sentiments' values are given.

text - Sentiment - sentiment

Ron is very amicable person - 5 - 4
That is so not the place to enjoy - 1
Rita is not so caring - 1 - 2

Now, I have to predict SINGLE sentiment column for the given text column.

Can I create the training dataset as below for training purpose? What could be difficulties in doing so?

text - Sentiment

Ron is very amicable person - 5
Ron is very amicable person - 4
That is so not the place to enjoy - 1
Rita is not so caring - 1
Rita is not so caring - 2

Edit:
My focus is not on choosing a single class for given text but on the decision to include that text two times with different class attributed to it.

The problem you are describing is a perfect use case for Fuzzy Logic.

Contrary to boolean logic where things are either black or white, fuzzy logic allows things to be grey.

Take the example of weather. Given a temperature, some people might say it is cold whereas other would say it is warm. There will be situations where all say it is cold, but there exists a mid-term where the classification isn't $\{0,1\}$ but rather $[0,1]$.

Therefore, it won't be a problem given that your chosen algorithm allows for probabilistic output.

So given the temperature example. You have the classes [cold, warm, hot].
If your algorithm can only output binary classifications (e.g. [0, 1, 0]), then better only input data points with only 1 label. However, if your algorithm can output probabilistic classifications (e.g. [0.2, 0.7, 0.1]) then you can input the same datapoint with multiple labels.

In the end, you can pick the class with the largest probability, but you need to understand that this might not give the output you are expecting. If you get a classification of [0.49, 0.51, 0.0], you might say is class 2 where in truth the model is not actually telling you that.


Correlation feature selection followed by regression I have quarterly results data for a company with around 100 variables. Total 60 quarters results are available (total records 60).
sample data: (only few columns & 10 rows)

I would like know following,

which ML algorithms / methods to be used to get the information about most important variable(s) affecting the price movement

how much each variable is contributing towards price movement (+ve or -ve)

predict, if a variable value is changed then how it'll affect the price movement

Thanks!

I think this is a case for linear regression with a lasso/ridge penalty. The lasso/ridge does „shrink“ features/variables, so that it is easy to see which features are important. Since you have 100 variables, you could opt for lasso, since lasso can also „automatically“ exclude features. Here is a lasso example in Python: https://datascience.stackexchange.com/a/53639/71442.


Deep Learning Binary Text Classification I am looking to build a model for specific news and blog articles which merge fashion with patterns in biology. I have 35 websites that I read daily (it is exhausting). I am wondering how to approach creating such a model so that I can send articles to it daily and it can predict whether or not they deserve reading (ie relevant vs irrelevant). For example, in a perfect world I send it 230 articles of which 12 are returned as relevant based on previous training and testing articles. Assuming all 12 are relevant I save them to the training dir and recompile  to strengthen the model.

After reading Deep Learning with Keras I was hoping Chapters 5 and 6 on Word Embeddings and RNNs(simple, LSTM, GRU) would point me to how to develop such a model. I was also reading how to implement a CNN for text classification but cannot seem to construct a basic conceptual framework for how to start with a few articles and adding to the training data every day to strengthen the model.

Is there a term for this type of strategy? Has it been done in some form on Kaggle, Github, etc?

In a broader sense, what you're referring to here is known as text mining. This is where information is extracted from text according to certain criteria in order to form specific patterns and meanings, etc.

How you would go about implementing this depends on the criteria you have in mind.

For instance, let's suppose you have a blog article in text format (let's call it filepath.txt for this example).

You decide that you wish to sort words into specific categories. For instance, you could categorise the keywords hotel, flights, countries, as travel, i.e. each incidence of the three words would be replaced with data. Let's assume data is a relevant category of interest. Any article that would fall under a travel category would be included in your articles of interest.

Using Python, you can then scan your file and use filedata.replace to replace your keywords with categories:

# Read file
with open('filepath.txt', 'r') as file :
filedata = file.read()

# Replace keyword
filedata = filedata.replace('How ', ' ')
filedata = filedata.replace('Why ', ' ')
filedata = filedata.replace('of ', ' ')
filedata = filedata.replace('to ', ' ')
filedata = filedata.replace('you ', ' ')
filedata = filedata.replace('all ', ' ')
filedata = filedata.replace('and ', ' ')
filedata = filedata.replace('be ', ' ')
filedata = filedata.replace(' a ', ' ')
filedata = filedata.replace(' for ', ' ')
filedata = filedata.replace(' in ', ' ')
filedata = filedata.replace(' is ', ' ')
filedata = filedata.replace(' the ', ' ')
filedata = filedata.replace(' about ', ' ')
filedata = filedata.replace(' an ', ' ')
filedata = filedata.replace('Data', ' data ')
filedata = filedata.replace('Python', ' data ')
filedata = filedata.replace('R', ' data ')
filedata = filedata.replace('machine', ' data ')
filedata = filedata.replace('Linux', ' data ')
filedata = filedata.replace('technology', ' data ')
filedata = filedata.replace('flights', 'travel')
filedata = filedata.replace('countries', 'travel')
filedata = filedata.replace('hotel', 'travel')
filedata = filedata.replace('analytics', 'data')
filedata=  filedata.replace('CNN', 'news')
filedata=  filedata.replace('weather', 'news')
filedata=  filedata.replace('Trump', 'news')
filedata=  filedata.replace('market', 'business')
filedata=  filedata.replace('entrepreneur', 'business')
filedata=  filedata.replace('financial', 'business')

# Write to file
with open('filepath2.txt', 'w') as file:
file.write(filedata)

You could then use a library such as stringr in R to see how frequently the category (which would have the keywords replaced with the category) appears:

require(stringr)
WordList <- str_split(readLines("filepath2.txt"), pattern = " ")
searchqueries<-sort(table(WordList),decreasing=TRUE)[1:100]
searchqueries

e.g. if the category appears X number of times or more, then the article is deemed relevant.

This is just one example of how you could use text mining. Again, much of it hinges on your specific criteria and how you would go about determining relevancy. But that's one way you could approach it.


TensorFlow: how to print out all elements in a tensor I have a tensor that is of the size (19,19,5,80). I want to print out all the elements in it but if I use print(XXX.eval()), most of the elements will end up as ellipsis. How can I print all of them out?

Basically what you want to do is showing the full numpy array in jupyter notebook. Use the following code:

import numpy
numpy.set_printoptions(threshold=numpy.nan)

Then run your code and see the entire activation map.


Which F1-score is used for the semantic segmentation tasks? I read some papers about state-of-the-art semantic segmentation models and in all of them, authors use for comparison F1-score metric, but they did not write whether they use the "micro" or "macro" version of it.
Does anyone know which F1-score is used to describe the segmentation results and why it is so obvious that authors do not define it in papers?
Sample papers:
https://arxiv.org/pdf/1709.00201.pdf
https://arxiv.org/pdf/1511.00561.pdf

I looked very quickly and only at the first paper so I might miss something but it looks to me like the task is a binary classification problem. If this is correct then there's no need for averaging the F1-score.
Also in this paper the authors even give the formula of the F1-score (!), so I'd say that they are quite thorough in their description of the evaluation measures they use. I would take this as an additional indication that there's no averaging,  since it's unlikely that they wouldn't mention it.


Role of decoder in Transformer? I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation task: Je suis ètudiant -> I am a student
The encoder receives Je suis ètudiant as the input and generates encoder output which ideally should embed the context/meaning of the sentence.
The decoder receives this encoder output and an input query (I, am, a, student) as its inputs and outputs the next word (am, a, student, EOS). This is done step by step for every word.
Now, do I understand this correctly that the decoder is doing two things?

Figuring out relationship between the input query and encoder embedding i.e how is the query related to the input sentence Je suis ètudiant
Figuring out how is the current query related to previous queries through the masked attention mechanism. So when the query is student, the decoder would attend to relevant words which have already occurred (I am a).

If this is not the right way to think about it, can someone give a better explanation?
Also, if I have a task of classification or regression for a time series, do I need the decoder? I would think just the encoder would suffice as there is no context in the output of the model.

Yes, you are right in your understanding of the role of the decoder.
However, your use of "query" here, while somewhat technically correct, seems a bit strange. You are referring as "query" to the partially decoded sentence. While the partially decoded sentence is actually used as query in the first multihead attention block, people normally do not refer to it as "query" when describing stuff from the conceptual level of the decoder.
About needing the decoder in classification or regression tasks: the decoder is used when the output of the model is a sequence. If the output of the model is a single value, e.g. for a classification task or a single-value regression task, the encoder would suffice. If you want to predict multiple values of a time series, you should probably use a decoder that lets you condition not only on the input but also on the partially generated output values.


Assigning numerical IDs to variable values in a data file I have the problem that I got a huge source data file which is showing text for all variable values instead of numerical IDs. So for example, I would like to have the variable gender coded as 1 and 2 instead of "female" and "male" written out. And equally the same for 200 other variables of which some have up to hundreds of distinct variable values.

Therefore, doing this manually is not really an option here.

Could anybody please point me to a solution or hint within R, SPSS or Python how I can assign numerical IDs to each distinct variable value?

I thought this would be a problem other people face more commonly as well, but I have found nothing of this kind at all.

Thank you for any help!

You can use the Python sklearn preprocessing LabelEncoder. Here is some example code from this page with my comments:

# Make a Label Encoder instance
le = preprocessing.LabelEncoder()

# Show it the data it has to encode, so your column
le.fit(["paris", "paris", "tokyo", "amsterdam"])

# Get a ordered list of all classes it found
list(le.classes_)

# Transform a column/list
le.transform(["tokyo", "tokyo", "paris"])

# Transform encoding back to original
list(le.inverse_transform([2, 2, 1]))


How to find what values are assigned to labels that where encoded using LabelEncoder? places = ['India','France','India','Australia','Australia','India','India','France']

  Here places are the DataFrame Series, now how can I find that which
  label was encoded with values like India = 0 , Australia = 1 ,France =
  2.
  This is ok for few labels what if there are 100's of labels available
  in a huge dataset.

Use the classes_ attribute of your LabelEncoder. For example:

le = preprocessing.LabelEncoder()
le.fit(places)
print(le.classes_)

The index of the label in le.classes_ is the encoded value of the label. See another example here.


converting column from a factor to date format YYYY-MM-DDThh:mm:ss.SSSZ i want to convert the whole column from a factor to date.

The str of the dataset I am using is given below:

> str(dataset)
'data.frame':   2538 obs. of  5 variables:
 $ X        : int  1 2 3 4 5 6 7 8 9 10 ...
 $ SessionID: int  13307 21076 27813 8398 23118 12256 28799 11457 7542 19261 ...
 $ Timestamp: Factor w/ 2532 levels "2014-04-01T03:02:33.088Z",..: 2064 905 1086 1027 2419 1327 2035 1206 481 1354 ...
 $ ItemID   : int  214684513 214718203 214716928 214826900 214838180 214717318 214821307 214537967 214835775 214706432 ...
 $ Price    : int  0 0 0 0 0 0 0 0 0 0 ...

How  can I convert timestamp variable from factor to specific date format:

  YYYY-MM-DDThh:mm:ss.SSSZ (2014-04-06T18:42:05.823Z)

One way to do this is to use the PIPE operator to pass the column of the dataframe into your function, then assign it back to the dataframe.

I created a dataframe with one column holding your example date string five times:

> x <- "2014-04-06T18:42:05.823Z"
> DF <- as.data.frame(c(x, x, x, x, x), colnames="original")
> colnames(DF) <- "original"
> str(DF)
'data.frame':   5 obs. of  1 variable:
 $ original: chr  "2014-04-06T18:42:05.823Z" "2014-04-06T18:42:05.823Z"

I use this PIPE symbol, which allows you to chain methods together, passing the output from one into the input of the following. Using a . symbolises the output from the previous method.

To make the PIPE operator available, load in the magrittr package (I believe it is also included in the tidyverse, so perhaps via the dplyr package:

library(maggritr)

Now I apply the as.POSIXct function to the original column, and save it back to the dataframe in a columnn called new:

DF$new <- DF$original %>% as.POSIXct(., tz = "UTC", "%Y-%m-%dT%H:%M:%OS")
> DF
                  original                     new
1 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
2 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
3 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
4 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
5 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822

> str(DF)
'data.frame':   5 obs. of  2 variables:
 $ original: chr  "2014-04-06T18:42:05.823Z" "2014-04-06T18:42:05.823Z" ...
 $ new     : POSIXct, format: "2014-04-06 18:42:05.822" "2014-04-06 18:42:05.822" ...

You can remove the original column if you like, using:

df$original <- NULL


Linear regression incorrect prediction using Matlab In the plot below the red crossed line is the actual curve and the crossed blue line is the predicted curve. I am using least squares for linear prediction. I have used 1:79 examples in training and the remaining for testing. The test data points are never seen during training. What is my mistake? Why am I getting such a weird prediction? I want to see the sine curve as the predicted output which should be very close to the original data.

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;

% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);

You are calculating a linear regression in a nonlinear environment, which means you will need nonlinear exogenous variables.

You are calculating $Y=\beta_0+\beta_1x$ (a linear equation) when you need at least a polynominal equation. The following code calculated this equation: $Y=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3$ (maybe not the best equation but a working one)

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

hold on
X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X X.^2 X.^3];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y;

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples test_samples.^2 ...
test_samples.^3];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;

% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);


Get Gate Activation of GRU-Cells in TensorFlow I implemented a standard RNN in Tensorflow (0.12) using the classes

tf.python.ops.rnn_cell.GRUCell
tf.nn.dynamic_rnn

I am interested in the gates and which values they take during. Unfortunately the function dynamic_rnn doesn't support this and just gives the 'output' and 'state' (The final hidden state). In the code of the GRUCell the gates are called 'r' and 'u' for reset and update.

Is there a clever way to save the values of the gates over time or do I have to write my own GRU cell?

Thanks!

I am trying to figure out the same problem as well. I don't have the full solution but maybe we can help each other out.

If you do: names =[n.name for n in tf.get_default_graph().as_graph_def().node]

This will give you a list of all the names of the nodes in the graph. For Rnn the out put for names should look some what like this

.
'RNN/Shape',
'RNN/strided_slice/stack',
'RNN/strided_slice/stack_1',
.
.
.
'RNN/transpose'
.

Then you can use get_operation_by_name to get the operation you want to look at.

 rnn_transpose=tf.get_default_graph().get_operation_by_name('RNN/transpose').outputs[0]

From that you can run the eval() with your regular sess and feed_dict

 rnn_transpose.eval(session=sess,feed_dict =feed_dict)

The problem here is now we have to find which operations in the graph correspond to the proper gate operations.


Logbook: Machine Learning approaches In the past, when trying different machine learning algorithms in order to solve a problem, I used to write drown the set of approaches on a notebook, keeping details such as features, feature preprocessing, normalization, algorithms, algorithm parameters... therefore, building a hand-written logbook.

However, currently I'm concerned about using a 'more professional' tool, so that I can keep more details and even share it with other team-members, who are also able to stamp their approaches.

It would be great an automated and collaborative tool that keep track of the work done, considering details like: features, algorithms, algorithms parameter, data pre-process, data, metrics... beyond a collaborative Google Drive Spreadsheet for instance.

How are you solving this? How are you keeping track of the work done? What's your logbook tool?

Thank you very much in advance.

How are you solving this? How are you keeping track of the work done?
  What's your logbook tool?

This might not be the best approach. But, this is how my team does it. We believe that for pulling off an end-to-end data science experiment, proper conscience is very important. So, we use Slack for the same for our discussions and the meetings.

In addition to them, we have Rmd (R markdown) files for documenting the planning and the analysis parts.


How to determine if a string is 'food' or 'drink'? My data exists of several restaurant menus (in text form). I would like to detect the category of an item on the menu which can be 'food' or 'drink'.

How do I call this kind of machine learning? Text classification?

I know javascript isn't ideal for this kind of thing but I would like to use brain.js or tensorflowjs to implement a proof of concept.

Thanks in advance!

Menu is generally a list of items served in restaurant. Items broadly divided in to two sections:

Food: Where all food items are listed
Drinks: Where all drink items are listed.

If all restaurant menus that you have, listed items in similar way, I'm bit confused how you will use ML in this case.

But if restaurant menu includes recipe/ ingredients of the food/drink items, then yes I can apply machine learning in order to identify the category of an item.
In such a case we can use NLP or simple text classifier(by dividing the recipe in some key word tokens and segregate between food and drink)model for prediction.


What is a lower bound on the vocabulary size for generating word/sentence embedding using word2vec or skip thought vectors? I am working on a NLP related task. I have about 150 documents, each few pages long (5/6 pages long on average). After removing stopwords and other unnecessary symbols and digits, I have about 104,000 unique words. The task at hand probably require some kind of word embedding (such as word2vec) as simple bag-of-words type approach aren't working properly. However, I am concerned about the size of the data I have. I have looked at pre-trained word embedding (GloVec), however, due to the narrow focus of the domain (manufacturing) of our texts, I am hesitating to use these pre-trained vectors. That leaves me with training our own. However, the size of our data set concerns me. Hence I am just throwing this question out there: What should be the lower bound on the size of the vocabulary that we need in order to train a word embedding model (word2vec) that will be reasonable.

Any response would be greatly appreciated.

Thanks

It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much.

Another approach is  more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can set N to for example 10 or 20. This is common practice as you need a certain number of repetitions of the same word to have some meaningful results.


Best way to represent data as features vectors in Python I have a list of events topic retrieved from a tweets collection.
A set of features have been extracted and their values normalized between 0 and 1. An example of an event:

"paris_attack_news-20150107_100842-20150107_112852": {
    "ages": 0.5557594006583049,
    "density": 0.0012022814250710345,
    "followers": 0.1144661871115895,
    "friends": 0.13507755010659472,
    "hashtagCount": 0.033270950301517985,
    "lifespan": 0.29613227044582224,
    "mediaCount": 0.1095890410958904,
    "mentionCount": 0.020275919732441472,
    "objectivity": 0.2850584551023736,
    "polarity": 0.2963684492294102,
    "retweetCount": 0.21431767337807606,
    "status_count": 0.09222093073720204,
    "truth": 1.0,
    "tweetCount": 0.01300578034682081,
    "urlCount": 0.29494007989347537,
    "verified": 0.3392857142857143
}

Now I need to represent each event as an array of its features:

paris_attack_news-20150107_100842-20150107_112852 = [0.5557594006583049, 0.5557594006583049, 0.1144661871115895, 0.13507755010659472, ...]

After that, I need to manipulate/aggregate the array values in some ways to get a specific events sorting based on the results.

The data are already in a Python Pandas DataFrame (event name as index, features as columns).

Which is the best way (data structure for further storing or libraries such as NumPy, sklearn or similar) to build the arrays starting from this?

P.S.: Then I'll need to apply some machine learning alghoritms to detect if an event is TRUE or FALSE, using the feature named "truth": 1 or 0 as label classification.

Numpy is best.  You can just take

np.array(dict.values()).

Eg,  dict = Paris_attck

Most ML libraries operate on numpy arrays.


Checking if english sentences have impact catpured in them using NLP I am looking for high level approach to identify if a given sentence has impact captured in it. For example in below two examples, sentence 2 has impact captured.

Example 1

Sentence 1: I learnt speaking spanish this year.

Sentence 2: I learnt speaking spanish this year which helped expand my business to Mexico.

Example 2

Sentence 1 : I taught machine learning to my students this year.

Sentence 2 : I taught machine learning to my students earlier this year which resulted in increasing base salary of students by 20,000 Dollars.

My initial thought is below approach.

think of all the english words like helped, resulted etc.,
Apply these words as filters on a public domain data set to get sentences
Curate the sentences to see if they fall in the "Sentence 2" category.
Apply classification/modelling to categorize into sentences with impact and without impact

Question:

Is the above approach good or there are better approaches?
What is the best publicly available dateset to tackle this problem.

Any inputs are greatly appreciated. Thanks alot in advance!

The approach you propose is ok but it can be improved in my opinion:

Find a corpus (or several corpora), chosen to be representative of the kind data you expect to process eventually. Ideally it would be directly a sample of the target data.
Label all the sentences in the corpus or a random subset. This is because you need to preserve the distribution as much as possible, especially the proportion of positive/negative instances. If you start from the set of filtered sentences you'll have two problems:

a proportion of positive cases higher than in regular text, and this is likely to make your model over-predict positive cases;
sentences containing only the selected trigger words, and this will cause your model to predict as negative any sentence which doesn't have a trigger word (that's a problem since your list of trigger words can't be exhaustive).

Ideally you would manually label all your corpus, but that's probably not realistic. That's why you could try to use your idea of trigger words in a slightly different way in order to label all the instances efficiently:

Filter the sentences which contain any trigger word and label these; this is your initial training set
Use a semi-supervised approach to label the rest of the sentences (maybe you could consider active learning).


Association Rules - Data Mining - Train and Test approach? Does it make sense to use the train, test, and validation model using the Association Rules Technique?

Finding association rules is an unsupervised learning task (or exploratory task). You don't actually know which rules you want to find before you actually find them, so there's nothing to test against. Validating against a separated instance set is usually done on supervised learning tasks such as classification and regression.


Modeling Encoder-Decoder according to instructions from a paper I am new to this field and I was reading a paper "Predicting citation counts based on deep neural network learning techniques". There the authors describe the code that they implemented if someone wants to reproduce the results. I tried to do this but I am not sure if I succeeded.
Here is their description:
-RNN module - SimpleRNN
-Output dimension of the encoder - 512
-The output layer - Dense layer
-Activation function - ReLU
-Overfitting prevention technique - Dropout with 0.2 rate
-Epochs - 100
Optimization algorithm - RMSProp
Learning rate - 10^{-5}
Batch size - 256

And here is my implementation. I am not sure if the model I created is sequence to sequence.
epocsh = 100
batch_size = 256
optimizer = keras.optimizers.RMSprop(lr=0.00001)
model =  keras.models.Sequential([
    keras.layers.SimpleRNN(512, input_shape=[X_train.shape[0], X_train.shape[1]],
                           activation='relu', return_sequences=True, dropout=0.2),
    keras.layers.Dense(9)
])

model.compile(loss='mse', optimizer=optimizer, metrics=[keras.metrics.RootMeanSquaredError()])

The summary of this model is:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
simple_rnn (SimpleRNN)       (None, 154521, 512)       266240
_________________________________________________________________
dense (Dense)                (None, 154521, 9)         4617
=================================================================
Total params: 270,857
Trainable params: 270,857
Non-trainable params: 0
_________________________________________________________________

Any help would be appreciated. Thank you!
Update: Here is the dataset.
year  venue  c1  c2  c3  c4  c5  c6  c7  c8  c9  c10  c11  c12  c13  c14
1989    234   0   1   2   3   4   5   5   5   5    8    8   10   11   12
1989    251   0   0   0   0   0   0   0   0   0    0    0    0    0    0
1990    346   0   0   0   0   0   0   0   0   0    0    0    0    0    0

I need to give as an input all the columns until c5, and try to predict the other c's (which are citation count for the upcoming years).

What is the input and output? Assuming your input is a sequence of documents, you would need to convert it into some sort of embeddings (TF-IDF, any other). After that comes the RNN - which looks ok to me. The output of the RNN is equal to the number of timesteps (num of words in each doc?). For each timestep we predict using a Dense final layer. You have 9 outputs. Is it some sort of a classification across 9 different types of categories or levels of citation counts?
Overall looks ok with the above assumptions other than the fact that you are feeding raw data instead of using an embedding layer


Using LSTM for binary text Classification, getting almost same accuracy at each epoch I am doing Twitter sentiment classification. For that I am using LSTM with pretrained 50d GloVe word embeddings(not training them as of now, might do in future).

The tweets are of variable lengths ranging from 1 to 250. Hence I sorted the tweets and divided them into batches of almost similar length. Further I zero padded them with maxLen equal to the highest length of tweet in that particular batch, also in embedding layer I have set 'mask_zero = True'.

I am using adam optimizer with default values, and fit generator to train the model.

The first issue I observed was that for each epoch initially the accuracy starts increasing and goes till 80, but  when its around halfway through the training set the accuracy starts decreasing and fall to 74 point something. This happens for each epoch.

To solve this I shuffled the batches and then fed them to model, as I thought the model might be adjusting itself to smaller length tweets as those are being fed to it first and hence not generalizing to longer tweets.

After doing this, the accuracy randomly fluctuates and again lands on 74 something at the end of each epoch.

I don't know why I am getting stuck at local minima, ANY HELP IS MUCH APPRECIATED...:)

I will attach some code if required...

First of all, this is a binary classification problem (positive sentiment / negative sentiment), correct? And the dataset is roughly balanced? What were you trying to achieve by sorting them by length? Why are you using a generator and not just running fit on the dataset?

Don't worry about local minima. What you're looking at is a model/optimizer combination that converges to around 74% on this dataset. The one being fed sorted data has a higher accuracy at the start of the epoch because short tweets are easier to classify than long ones. And I wouldn't even worry about the batch-by-batch accuracy, just the epoch-by-epoch curve.


Binary classifier using Keras with backend Tensorflow with a Binary output I am trying to build a binary classifier with tensorflow.keras
Currently unable to identify a solution to having the model generating only 0s and 1s.

The code for compiling my tensorflow model.

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

def build_model(train_dataset):
  model = keras.Sequential([
    layers.Dense(10, activation='relu', input_shape=[len(train_dataset.keys())]),
    layers.Dense(1, activation='sigmoid')
  ])
  optimizer = 'adam'

  model.compile(loss='binary_crossentropy',
                optimizer=optimizer,
                metrics=[tf.keras.metrics.Precision(), tf.keras.metrics.Recall(), tf.keras.metrics.Accuracy()])
  return model

model = build_model(X)

After calling model.fit(X,y) and model.predict(X_test) an array of numbers are produced as prediction values:

array([[8.3726919e-01],
       [9.1233850e-04],
       [8.3726919e-01],
       ...,
       [4.6819448e-05],
       [1.5565268e-08],
       [0.0000000e+00]], dtype=float32)

My current solution to the problem is to get the output as a number in the range (0,1) from the sigmoid activation function; and then transforming the values to 0 or 1

reset_predictions = []
for p in predictions:
    if p >= 0.5:
        reset_predictions.append(1)
    else:
        reset_predictions.append(0)

However this is not ideal; what I hope to achieve is for the model to classify 0 or 1 for me without an external function.

I don't believe that's possible, in order for the model to return 0 or 1, your activation function on the output layer would have to return 0 or 1, which would mean that the activation function is non-differentiable, and you cannot do that.

Also you can simplify your transformer function to:

In [20]: predictions = np.array([[0.1], [0.9], [0.3], [0.6]])

In [21]: (predictions[:, 0] > 0.5).astype(np.int8)
Out[21]: array([0, 1, 0, 1], dtype=int8)

So it is very little extra work, and it gives your more flexibility if you want to see how confident the model is about the prediction, and you can change the threshold if you like.


The distribution of dataset train and test are the differents, how to fix this? I am new in data science and like some help to understand my problem. For instance, I have two signals non-stationary for the same condition (figure 1).

I acquisition them at different times(in the morning and in the afternoon), when applying the Kolmogorov-Smirnov test, the null hypothesis was rejected, I don't understand why distribution is different if I no change any parameters in my system of acquisition.

This is the main trouble in my analysis because of this no get model any algorithm of machine learning to classification (overfitting).

I read something about the Covariate Shift (*I saw this post), and Kullback_Leiber Importance Estimation Procedure, but I don't know iff will really work out.

*'Different Test Set and Training Set Distribution'

how does it compare to longer timespans? Can you still observe noticeable drift?
to some extent this is usual behaviour
here is a guide for KL divergence, It's just a tool you can use to compare probability distributions.
exploration is equally important as hard stats. For me those plots look alright. There's no defined point, telling you when you can or cannot do things. That's whole point of ML, you don't have a priori knowledge. Best run some models and check validation metrics.


How does C have effects on bias and variance of a Support Vector Machine? The minimization problem for SVM can be written as-
$$\overset{\text{min}}{\theta} C\sum_{i = 1}^{m}{[y^icost_1(\theta^Tx^i) + (1-y^i)cost_0(\theta^Tx^i)]} + \frac12\sum_{j = 1}^n{\theta_j}^2$$
Now, how can the choice of $C$ lead to underfitting or overfitting?
As I understand, parameters are chosen to make $C\sum_{i = 1}^{m}{[y^icost_1(\theta^Tx^i) + (1-y^i)cost_0(\theta^Tx^i)]}$ part $0$. And we concern ourselves with the second part.
And Andrew Ng says that a large $C$ leads to lower bias and higher variance.
How does this happen? What is the intuition behind this?

The C being a regularized parameter, controls how much you want to punish your model for each misclassified point for a given curve.
If you put large value to C it will try to reduce errors but at the same time it may happen that it would not perform better on test dataset hence cause overfitting.
To get to know more about effect of C in svm. Refer this.


Output landscape of ReLU, Swish and Mish I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681).

I understand that this figure describes how the loss is being changed, if the change is smooth or not. But the paper don't describe how it was generated, what does the color on the figure mean (no colorbar), or what is the plotted along x and y axis. Can anyone help in this regard?

From the article you cite, I have found this figure which looks similar to yours.

On the same page of the article, they say:

Having a smooth profile also plays a role in better gradient flow, as
shown in Fig. 3, where the output landscapes of a five-layered randomly
initialized neural network with ReLUand Mish are visualized.  The
landscapes were generated by passing in the co-ordinates to a
five-layered randomly initialized neural network which outputs the
corresponding scalar magnitude. The output landscape of ReLU has a
lot of sharp transitions as compared to the smooth profile of the
output landscape of Mish.

It seems your figure (and the one above) does not show the loss landscapes (which is the figure 4 in the article) but the landscape of the global neural network using different activation functions. I guess the neural network they used to produce these figures has 2 inputs (the coordinates of the plots), 3 hidden layers and one scalar output. The color in the plots corresponds to the scalar output value.
The authors present these figures because :

Smoother output landscapes suggest smooth loss landscape [...]

This is my understanding of the article, hope it helps.


better confussion matrix higher LogLoss ? Is that possible> I have tried a 2 different versions of a gbm in a multinomial classification problem.  The second model results in better confusion matrix but in worse Log Loss value (at the test sample). How is that possible.

Further are the results of the two models.

I thought that it could be because the Class A is much more oversampled and the small decrease of that class could lead to such a deterioration of hte logloss?

Any ideas?
Thank you

Log loss is a measure of confidence of the model in its predictions. Lower log loss implies higher confidence and vice-versa. Better confusion matrix with worse log-loss implies that few of the misclassifications, but with higher probability score. Since the accuracy is calculated with a fixed probability threshold, you might not observe this directly. Plot the ROC curves for both the models with is based on varying the decision threshold. This gives you the idea about which model is better.


What is the difference between a (dynamic) Bayes network and a HMM? I have read that HMMs, Particle Filters and Kalman filters are special cases of dynamic Bayes networks. However, I only know HMMs and I don't see the difference to dynamic Bayes networks.

Could somebody please explain?

It would be nice if your answer could be similar to the following, but for bayes Networks:

Hidden Markov Models

A Hidden Markov Model (HMM) is a 5-tuple $\lambda = (S, O, A, B, \Pi)$:

$S \neq \emptyset$: A set of states (e.g. "beginning of phoneme", "middle of phoneme", "end of phoneme")
$O \neq \emptyset$: A set of possible observations (audio signals)
$A \in \mathbb{R}^{|S| \times |S|}$: A stochastic matrix which gives probabilites $(a_{ij})$ to get from state $i$ to state $j$.
$B \in \mathbb{R}^{|S| \times |O|}$: A stochastic matrix which gives probabilites $(b_{kl})$ to get in state $k$ the observation $l$.
$\Pi \in \mathbb{R}^{|S|}$: Initial distribution to start in one of the states.

It is usually displayed as a directed graph, where each node corresponds to one state $s \in S$ and the transition probabilities are denoted on the edges.

Hidden Markov Models are called "hidden", because the current state is hidden. The algorithms have to guess it from the observations and the model itself. They are called "Markov", because for the next state only the current state matters.

For HMMs, you give a fixed topology (number of states, possible edges). Then there are 3 possible tasks

Evaluation: given a HMM $\lambda$, how likely is it to get observations $o_1, \dots, o_t$ (Forward algorithm)
Decoding: given a HMM $\lambda$ and a observations $o_1, \dots, o_t$, what is the most likely sequence of states $s_1, \dots, s_t$ (Viterbi algorithm)
Learning: learn $A, B, \Pi$: Baum-Welch algorithm, which is a special case of Expectation maximization.

Bayes networks

Bayes networks are directed acyclical graphs (DAGs) $G = (\mathcal{X}, \mathcal{E})$. The nodes represent random variables $X \in \mathcal{X}$. For every $X$, there is a probability distribution which is conditioned on the parents of $X$:

$$P(X|\text{parents}(X))$$

There seem to be (please clarify) two tasks:

Inference: Given some variables, get the most likely values of the others variables. Exact inference is NP-hard. Approximately, you can use MCMC.
Learning: How you learn those distributions depends on the exact problem (source):

known structure, fully observable:  maximum likelihood estimation (MLE)
known structure, partially observable: Expectation Maximization (EM) or Markov Chain Monte Carlo (MCMC)
unknown structure, fully observable: search through model space
unknown structure, partially observable: EM + search through model space

Dynamic Bayes networks

I guess dynamic Bayes networks (DBNs) are also directed probabilistic graphical models. The variability seems to come from the network changing over time. However, it seems to me that this is equivalent to only copying the same network and connecting every node at time $t$ with every the corresponding node at time $t+1$. Is that the case?

From a similar Cross Validation question follows @jerad answer:

HMMs are not equivalent to DBNs, rather they are a special case of DBNs in which the entire state of the world is represented by a single hidden state variable. Other models within the DBN framework generalize the basic HMM, allowing for more hidden state variables (see the second paper above for the many varieties).
Finally, no, DBNs are not always discrete. For example, linear Gaussian state models (Kalman Filters) can be conceived of as continuous valued HMMs, often used to track objects in space.
I'd recommend looking through these two excellent review papers:

An Introduction to Hidden Markov Models and Bayesian Networks by Zoubin Gharamani
Dynamic Bayesian Networks by Kevin Murphy


Error while using predict After splitting into test and train the glm function is used on train set. For example

m1 = glm(target ~ ., data = train, family="binomial")

Then

P = predict(m1,newdata=test, type="response")

Here I am getting warning message :

  newdata has 60000 rows but variables found have 100000 rows

And the next step:

table(test$target, P>0.5)

Here I am getting error:

  all arguments must have same length

What is the solution for this?

Works find for me. Maybe you have some error in the definition of the train/test set?

library(ISLR)
library(glmnet)
# Load data
iris = iris[iris$Species=="Setosa" | iris$Species=="versicolor",]
iris$Species = as.factor(iris$Species)
# Make a test/train split
set.seed(123)
smp_size <- floor(0.75 * nrow(iris))
train_ind <- sample(seq_len(nrow(iris)), size = smp_size)
train <- iris[train_ind, ]
test <- iris[-train_ind, ]

m1 = glm(Species ~ ., data = train, family="binomial")
p = predict(m1,newdata=test, type="response")
table(test$Species, p>0.5)


How to get sentence embedding using BERT? How to get sentence embedding using BERT?
from transformers import BertTokenizer
tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')
sentence='I really enjoyed this movie a lot.'
#1.Tokenize the sequence:
tokens=tokenizer.tokenize(sentence)
print(tokens)
print(type(tokens))

2. Add [CLS] and [SEP] tokens:
tokens = ['[CLS]'] + tokens + ['[SEP]']
print(" Tokens are \n {} ".format(tokens))

3. Padding the input:
T=15
padded_tokens=tokens +['[PAD]' for _ in range(T-len(tokens))]
print("Padded tokens are \n {} ".format(padded_tokens))
attn_mask=[ 1 if token != '[PAD]' else 0 for token in padded_tokens  ]
print("Attention Mask are \n {} ".format(attn_mask))

4. Maintain a list of segment tokens:
seg_ids=[0 for _ in range(len(padded_tokens))]
print("Segment Tokens are \n {}".format(seg_ids))

5. Obtaining indices of the tokens in BERT’s vocabulary:
sent_ids=tokenizer.convert_tokens_to_ids(padded_tokens)
print("senetence idexes \n {} ".format(sent_ids))
token_ids = torch.tensor(sent_ids).unsqueeze(0)
attn_mask = torch.tensor(attn_mask).unsqueeze(0)
seg_ids   = torch.tensor(seg_ids).unsqueeze(0)

Feed them to BERT
hidden_reps, cls_head = bert_model(token_ids, attention_mask = attn_mask,token_type_ids = seg_ids)
print(type(hidden_reps))
print(hidden_reps.shape ) #hidden states of each token in inout sequence
print(cls_head.shape ) #hidden states of each [cls]

output:
hidden_reps size
torch.Size([1, 15, 768])

cls_head size
torch.Size([1, 768])

Which vector represents the sentence embedding here? Is it hidden_reps  or cls_head ?
Is there any other way to get sentence embedding from BERT in order to perform similarity check with other sentences?

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT.
They also have a github repo which is easy to work with.


Range to define emotions We are capturing emotions as survey responses. We need to assign values for the responses(emotions) for analysis purposes. Is there an optimum range that can be assigned to achieve this? (like from -100 to 100). An example of a question and a set of answers are as follow.

Question: "How are you feeling today?"
Answers: Terrible, Sad, Ok, Good, Great

A suitable approach we could think of is to assign values from 1 to 100 with equal distance. Is this statistically valid? What are the things that we should consider when achieving this? In this case, only positive integers are assigned because we need to calculate statistics like weighted average. Can't we assign negative numbers as well?

The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents.

Asking if equally distancing the emotions is statistically correct does not make sense. This entirely depends on your use case and opinions. Also, there is absolutely no reason why you can't use negative numbers in a weighted average.

If you mentioned what you are doing and how you are evaluating emotion, there might be more to say.


Is there any clear tutorial for how to use AutoEncoders with text as input I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. It contains only numbers. I want to generate the same input using AutoEncoders but I didn't find any useful link that I can use for that. The ones that I used had some dimension problems when I use my data. Does Anyone recommend any helpful tutorial ?

Concerning Encoding, this link helped me a lot. If you try the code in part 'One Hot Encode with scikit-learn', you will get your encoded vectors. You just have to feed it a list of all your tokens. So I have extracted the fields of the register into a list of markups.

As an output of the one hot encoding part, you will get a n dimensional array, you  feed it to the script in part 'Let's build the simplest possible autoencoder' of the AutoEncoder link

In the Input vector you need to put how many classes you have (instead of the value 784) and in our case, you specify the number of columns of the nd array which represent the unique tokens.

Then to compare the predicted output, you have to use decoding in order to compare visually the text that was regenerated.


Non IID variables and SVM Classifier I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a conversation with a friend.
This made me rethink about my approach and I have a few questions.

Why does SVM assume IID data in the first place?

By IID, does it mean that all the features should be linearly uncorrelated or should there not be any non-linear relationships as well?

The features include basic stock data (Open, High, Low, Close, Volume) and technical indicators (derived from basic data). Of course, the technical indicators will have dependencies with basic data. So I removed the basic data from the feature space. However, this does mean that the technical indicators are not related. Also,

How does one make sure that the data is identically distributed? Does scaling (Normalization or Standardization) help in this case?

Lastly, I have read some papers wherein people have used NN and SVM (both assume IID data) for stock trend prediction, but nowhere did I see any mention of the IID assumption and the fact that stock data is actually not uncorrelated and to some extent exhibits some autocorrelation as well. So,

How does one justify the use of non IID data as input to such algorithms which assume that the data is IID?

Thank you :)

The assumption regarding IID variables is to ensure a unique solution. By IID, it means they should be uncorrelated. You can't make sure data is identically distributed. Scaling and standardization are the obvious for you to find the solution. One justifies the use by getting satisfied by the solution found by the SVM algorithm (which is just one of the solution it has found).
But, you can the parameters for SVM and use a different kernel altogether. Remove redundant features yourself.
Or if you are looking at stock price prediction, look at Jane Market Street Prediction on Kaggle. I have seen heavy use of trees, mixture models and NNs.


Is a neural network able to learn to map a completely different feature vector to the same class Is a neural network (for example a MLPClassifier in Python) able to learn to map a completely (or very) different input feature set to the same output class? Or is it better to work in this case with more than one output class and map these recognized output classes afterwards to the same class manually?

Yes, neural networks excel at finding highly non-linear decision boundaries. For example, take a look at this demo (don't forget to click the play button to make the network learn). Even this very simple network is able to learn that the opposite corners of the input space belong to the same class.


The Kaplan Meier curve is a summary statistic, similar to the average. Therefore, it is an unconditional statistic. If you are interested in the conditional survival, we can use the Kaplan Meier curve to do that too. Some notation first. Let $T$ be the unknown time of death, and $S(t) = P(T > t)$ the unconditional survival curve.

Given a subject has lived past some time $s$, we'd like to know $P(T > t | T > s)$. We can expand this conditional probability:

$$
\begin{align}P(T > t | T > s) &= \frac{ P(T > t \text{ and } T > s)}{P(T > s)}\\
&= \frac{ P(T > t)}{P(T > s)} \text{   ,since t > s}\\
&= \frac{ S(t)}{S(s)}
\end{align}
$$

That is, we scale that entire survival curve by $S(s)$.

In practice, if you have a censored subject and you'd like to know their conditional survival after, you just scale by the survival function at their censored time.

In lifelines, there is the property KaplanMeierFitter.conditional_time_to_event_ which computes the median remaining survival time given survival up to time $t$. This can be used as a conditional prediction.

kmf = KaplanMeierFitter().fit(t, e)
censored_times = t[~e.astype(bool)]
predicted_life_remaining = kmf.conditional_time_to_event_.asof(censored_times)


DIfferent learning rates converging to same minima I am optimizing some loss function using Gradient Descent method. I am trying it with different learning rates, but the objective function's value is converging to same exact point.

Does this means that I am getting stuck in a local minima?, because the loss function is non-convex so it is less likely that I would converge to a global minima.

This is the expected behavior. Different learning rates should converge to the same minimum if you are starting at the same location.

If you're optimizing a neural network and you want to explore the loss surface, randomize the starting parameters. If you always start your optimization algorithm from the same initial value, you will reach the same local extremum unless you really increase the step size and overshoot.


Multivariate Polynomial Feature generation I don't quite seem to understand the rules used to create the polynomial features when trying to find a polynomial model with Linear Regression in the multivariate setting.

Let's say I have a two predictor variables a and b. When generating polynomial features (for example using sklearn) I get 6 features for degree 2: y = bias + a + b + a * b + a^2 + b^2

This much I understand.
When I set the degree to 3 I get 10 features instead of my expected 8. I expected it to be this: y = bias + a + b + a * b + a^2 + b^2 + a^3 + b^3

What is the general formula of generating multivariate features? How does this look like in the 3rd degree?

General Formula is as follow:
 \begin{equation} N(n,d)=C(n+d,d) \end{equation} where n is the number of the features, $d$ is the degree of the polynomial, $C$ is binomial coefficient(combination).

Example with vector (2,3) to 3rd degree :

x = np.array[[2,3]]
pf = PolynomialFeatures(degree=3, include_bias=True)
pf.fit_transform(x)

returns :

array([[ 1.,  2.,  3.,  4.,  6.,  9.,  8., 12., 18., 27.]])

wich is : [1, $\:$ $x_1$, $\:$ $x_2$,
 $\:$ $x_1^2$,$\:$ $x_1x_2$, $\:$ $x_2^2$,
 $\:$ $x_1^3$, $\:$ $x_2x_1^2$, $\:$ $x_2^2 x_1$, $\:$ $x_2^3$]

You can see this as expanding this equation and throwing out the coefficients :

\begin{equation}1 + (x_1 + x_2) +  (x_1 + x_2)^2 +  (x_1 + x_2)^3 \end{equation}


When forecasting time series, how does one incorporate the test data back into the model after training? When you build a classification or regression model, you typically split the data into a train data set and a test data set. The test data is a randomly selected subset of the overall data.

Once you are done with the training, you discard the test data, and apply the model you built to new unknown data.

But in the case of time series, this wouldn't work: You can't just randomly choose any subset of the data for your test data set, since the data is ordered. The test data set has to be composed of later observations of the data, while the train data set is made up of earlier observations of the data.

Say for example that you have data for 11 months of the year, and you want to forecast the values for December. You would train your model with the data from January through September, and then test it on the data from October and November.

But after you've successfully trained your model, what do you do with the test data?

On one hand, it doesn't make sense to forecast the values for December using a model built with Jan-Sep data. Such a model would miss any important trends that occurred in Oct and Nov.

On the other if you bring the data from Oct and Nov back into the model, the parameters of the model will change, and so we're no longer sure that we are going to get the same accuracy that we got when we trained the model with just the Jan-Sep data.

There seems to be a dilemma here.

So how is this problem approached when using machine learning models, especially non parametric models like Neural Networks for time series forecasting? Do they incorporate the test data into the model or do they discard it?

I agree with what Emre has commented on your question.

If you have enough data, I would try cross-validating your model by training it on different time sections of your data. For example, train your model on Jan-March data and then make predictions on April's data. Then you could train the model on Feb-April data, and predict May's data. Essentially creating a few training and testing sets to see how well your model generalizes. Once you feel you are getting good results, you could train your final "production" model on all of your available data and then keep an eye on how it does when real data comes in and make adjustments on what you see.

I've used a similar approach in the past on accelerometer data to classify exercises from a specific workout, and it worked well. Maybe something similar will work for you.


How to do node classification without deep learning? I have a dataset of coordinates with labels, a set of those coordinates (a graph) follows globally the same pattern.
ie : I have coordinates of eyes,mouth and nose on thousands of images. A graph can be formed from those coordinates on a single image and I want to use that structure to learn from that pattern to predict the class of the nodes (point on image).
I'd like to create machine learning model to accurately predict the class of each element.
the task is pretty simple but all references I come across are about fancy research Graph Neural Networks.
How can I do it efficiently?

That problem is commonly called graph classification. Common non-deep learning approaches for graph classification are spectral and kernel based.


Building Stacking machine learning model using three base classifiers I did a stacking using three base classifiers RF, NB, KN N and metamodel random forest or SVM using sklearn library

But which is strange each time i change the metamodel i got the same results. Is it normal ?????

No, in generally speaking, even minor changes should affect your performance. Changing your meta-model should normally have a visible impact in your model's performance.

Two things you can try:

Check for any problems in your code.
Maybe your test set size is really small. For example if you have 5 test samples, it isn't difficult for all models to get 4/5 (i.e. 80% accuracy). As your test size increases so should the variance of your models' performance.