Transformation of categorical variables I have a data with continous variables and categorical variables. I am using Random Forest and have made my continues variables Gaussian by transformation and have standardized it. Should categorical variables be done the same?

You can just convert them into factors if your random forest algorithm is accepting that .Usually one hotting the categorical variables are not suggested
Here is the an article gives information about categorical variables in Random Forest


Extracting encoded features after CatBoost I have a dataset containing numerical as well as categorical variables.

After I've fit my dataset to a CatBoostClassifier, I want to extract the entire feature set, with the categorical variables encoded in whatever method the classifier decided to encode them.

How can I extract the fully transformed (encoded) features? (similar to what a fit_transform method would return)

I don't believe this is possible, CatBoost does target encoding per split, so you end up with different values of encoding at different trees.

Before each split is selected in the tree (see Choosing the tree
structure), categorical features are transformed to numerical. This is
done using various statistics on combinations of categorical features
and combinations of categorical and numerical features.

https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html
However, if you just want to use the CatBoost encoding algorithm. You can use CatBoostEncoder


How does PV-DBOW (doc2vec) work? The explanation from the official doc2vec paper "Distributed Representations of Sentences and Documents" for PV-DBOW is as follows:

  Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.

According to the paper the word vectors are not stored and PV-DBOW is said to work similar to skip gram in word2vec.

Skip-gram is explained in word2vec Parameter Learning. In the skip gram model the word vectors are mapped to the hidden layer. The matrix that performs this mapping is updated during the training. In PV-DBOW the dimension of the hidden layer should be the dimension of one paragraph vector. When I want to multiply the word vector of a sampled example with the paragraph vector they should have the same size. The original representation of a word is of size (vocabulary size x 1). Which mapping is performed to get the right size (paragraph dimension x 1) in the hidden layer. And how is this mapping performed when the word vectors are not stored? I assume that word and paragraph representation should have the same size in the hidden layer because of equation 26 in 2

I haven't read the paper you linked, but I have followed a lecture notes by Richard Socher.

So, basically that mapping matrix is called weight matrices. There are two weight matrices W1 and W2 for input and output mapping. Thus for each word, vectors in both the matrices are updated via backpropagation.

To answer your question, a word is represented by one-hot sparse vector which has the size of Vx1 (V is size of vocabulary), with a value of 1 in one of the position. and when this vector is multiplied with the weight matrix W1 with size NxV, the corresponding embedding vector of size Nx1 (N is size of the required embedding vector) is used in the hidden layer.

Document has no one-hot represented vectors. So, basically there is a document matrix D of size Nxd (d is number of documents) where each column represents a document. In other words, the matrices W1 and W2 need the one-hot representation only for mathematical steps, other than that they are representing each word in each column just as the document matrix D.


How to plot clusters in nice a way? I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot the clusters in an n-dimensional space. Which options do I have?

T-SNE is another dimensionality reduction algorithm not mentioned in the article in the other answer. Used for VERY high dimensional data, if you have trained some embeddings for your dataset. Reference Here . Python standard library here.

cheers


How to do k-fold validation with classifiers? I want to cross-validate a model that plays the card game below (see image).

I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the training dataset, but I want to k-fold validate it, to ensure that it generalises well.

How can I apply k-fold validation in this case?. This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

So I cannot calculate the MSE of validation folds, there is no MSE
  here.

The question that you firstly need to ask is "how do I evaluate my model".

Since the model is a classifier, you can use the accuracy (number of correct classifications over the overall attempts) or the F-score as measures of your model's performance. Based on the chosen measure, you will thereafter evaluate the results of kfold-validation.


How multi-head attention mechanism in Transformers is able to attend to information from different representation subspaces at different positions Question partially inspired by this post about the need of multi-head attention mechanism.
For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(so that they potentially can - as stated in the Attention is all you need paper - attend to information from different representation subspaces at different positions) and most importantly preserve this diversity during the training process.

I think, what you are looking for is $d_k = d_v = d_{model}/h$ [1] where $h$ number of heads and $d_{model}$ dimensions of keys, values and queries for single attention version of the model. In relation of the model architecture and its embedding specifically, the above translates to
$Query Size = Embedding Size / h$
Model input
According to the above, at the input embedding layer weights for each head are stacked together in the single embedding matrix. Stacking together, hmm.. (stacking, ensembling).

Per head scores
As in the normal self-attention, attention score is computed per head but given the above, these operations also take in place as a single matrix operation and not in a loop. The scaled dot product along with other calculations take place here.
Multi head merge
As a final step, the attention score of each head is merged by simply reshaping the full  attention score matrix so that the per-head attention scores are concatenated into a single attention score.

Summing it up
With a clearer view of what the architecture of the model is and how its computational graphs looks like, we can go back to your original questions and say:
The multi-headed model can capture richer interpretations because the embedding vectors for the input gets "segmented" across multiple heads and therefore different sections of the embedding can attend different per-head subspaces that link back to each word.
In a more general sense, one can argue that running through the scaled dot-product attention multiple times in parallel and concatenating is a form of ensembling.

I hope it helps. For more details about the several operation taking place in the transformer architecture, I suggest you have a look at this post from which my answer is heavily influenced from:
https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853
[1] 3.2.2 https://arxiv.org/pdf/1706.03762.pdf


When should ordinal data be represented catigorically and when as integer? I am doing the Kaggle competition House Prices: Advanced Regression Techniques to learn more about data analysis. I would like to apply multiple models to the data(Regularized LR, Random Forests, Neural Networks, and ensemble methods).

When inspecting the data, I found many fields were ordinal data represented as categorical data. Two examples:

HeatingQC: Heating quality and condition

   Ex   Excellent
   Gd   Good
   TA   Average/Typical
   Fa   Fair
   Po   Poor
LotShape: General shape of property

   Reg  Regular
   IR1  Slightly irregular
   IR2  Moderately Irregular
   IR3  Irregular

I was wondering whether I should keep the fields like this, or whether I should turn encode them as integers(i.e. give each class in the category a number like 1,2,3 or 4) . Since the question probably is 'it depends', I hope you could give me some more general insight in when I should keep this data ordinal, or when to transform it into integers.

You shouldn't keep it like this any way. One option is one-hot encode, but since your variables are ordinals, there's no point in one-hot encoding, you can just transform them into natural numbers.

One-hot encoding will increase numbers of independent variables significantly, that's why one-hot will be worse for your model. But your variables are clear ordinals, because they have clear order for all the values.

So, it this case transforming into natural numbers is the best option.


Dealing with missing data in several features at once Good day,
What are the approaches for handling missing data in several features (categorical and continuous) at once? I look through each feature and plotted several histograms of the distribution of them, and I think that a simple substitution of value by some constant value (mean, mode or something else) is not the best choice. I want to build to several models (classifications for categorical and regressions models for continuous values) in order to impute data, but I don't know the right approach for doing this. If I train my models only using non-null rows, then I won't be able to predict values with null values. On the other hand, If I simply drop columns(features) with null values, then I got bad results because features with null values are valuable for prediction. I have 615739 rows * 49 columns data set (after one-hot encoding procedure). If I drop all rows that contain at least one null value, then I will get 451063 rows. Thanks for your advice in advance.

Imputing one or more Features at once you still Need to Diagnose.
Generic, mean or constant imputations rarely work (good).
To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR).

MCAR means that there is no relationship between the missingness of the data and any of the values.

MAR means that that there is a systematic relationship between the propensity of missing values and the observed data, but not the missing data.

MNAR means that there is a systematic relationship between the propensity of a value to be missing and its values.

Given what you have told its likely that its MCAR. (assumption is that you already tried to find this propensity yourself (domain knowledge) or build a model between the missing columns and other features and failed in doing so)
Some other techniques to impute the data, I would suggest looking at KNN imputation (from experience always solid results) but you should try different methods
fancy impute supports such kind of imputation, using the following API:
from fancyimpute import KNN

# Use 10 nearest rows which have a feature to fill in each row's missing features

X_fill_knn = KNN(k=10).fit_transform(X)

Here are different methods also supported by this package:

•SimpleFill: Replaces missing entries with the mean or median of each

column.

•KNN: Nearest neighbor imputations which weights samples using the

mean squared difference on features for which two rows both have

observed data.

•SoftImpute: Matrix completion by iterative soft thresholding of SVD

decompositions. Inspired by the softImpute package for R, which is

based on Spectral Regularization Algorithms for Learning Large

Incomplete Matrices by Mazumder et. al.

•IterativeSVD: Matrix completion by iterative low-rank SVD

decomposition. Should be similar to SVDimpute from Missing value

estimation methods for DNA microarrays by Troyanskaya et. al.

•MICE: Reimplementation of Multiple Imputation by Chained Equations.

•MatrixFactorization: Direct factorization of the incomplete matrix

into low-rank U and V, with an L1 sparsity penalty on the elements of

U and an L2 penalty on the elements of V. Solved by gradient descent.

•NuclearNormMinimization: Simple implementation of Exact Matrix

Completion via Convex Optimization by Emmanuel Candes and Benjamin

Recht using cvxpy. Too slow for large matrices.

•BiScaler: Iterative estimation of row/column means and standard

deviations to get doubly normalized matrix. Not guaranteed to converge

but works well in practice. Taken from Matrix Completion and Low-Rank

SVD via Fast Alternating Least Squares.

MICE was deprecated and they moved it to sklearn under iterative imputer
Another Option is old kaggle trick:
A trick I have seen on Kaggle.
Step 1: replace NAN with the mean or the median. The mean, if the data is normally distributed, otherwise the median.
In my case, I have NANs in Age.

Step 2: Add a new column "NAN_Age." 1 for NAN, 0 otherwise. If there's a pattern in NAN, you help the algorithm catch it. A nice bonus is that this strategy doesn't care if it's MAR or MNAR (see above).


How to associate array of class probabilities with corresponding image in keras? I have a very basic convolutional neural net built in Keras with a TensorFlow backend. The model is based of this Kaggle kernel full model. The input training data are 256x256 images with a corresponding set of class labels in a csv file. For example, here are the class labels in the csv file:

image_name  |  tags
----------------------------------------
train_0     |  class1 class2 class3
train_1     |  class2
train_2     |  class3
and so on ...

The following code shows how I am building, training, and predicting results.

# Build model
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=(32, 32, 3))) # Originally (32, 32, 3)

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='sigmoid')) # Originally 17

model.compile(loss='binary_crossentropy', # We NEED binary here, since categorical_crossentropy l1 norms the output before calculating loss.
              optimizer='adam',
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train,
          batch_size=128,
          epochs=4,
          verbose=1,
          validation_data=(x_valid, y_valid))

# Predict
p_valid = model.predict(x_valid, batch_size=128)

You can see the predicted results are an array of class probabilities:

p_valid
Out[29]:
array([[2.4571007e-01, 1.4929530e-02, 9.3575776e-01],
       [2.6206359e-01, 1.2429485e-02, 9.5172155e-01],
       [3.3679003e-01, 2.8344743e-02, 8.5209453e-01],
       ...,
       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],
       [8.2605546e-03, 2.0513092e-07, 9.9999821e-01],
       [8.2605667e-03, 2.0513131e-07, 9.9999821e-01]], dtype=float32)

How do I know which probability array is corresponding to which image?

Keras Model.predict() method doesn't shuffle the data, so each row in p_valid corresponds to the row in x_valid. For instance, p_valid[2] holds the probabilities for the image x_valid[2].

By the way, note that the values in each row don't necessarily sum up to 1, because they correspond to the probability of class i being present, thus are not exclusive.


Using CPU after training in GPU I am using tensorflow-gpu 1.10.0 and keras-gpu 2.2.4 with a Nvidia gtx765M(2GB) GPU, OS is Win8.1-64 bit- 16GB RAM.

I can train a network with 560x560 pix images and batch-size=1, but after training is over when I try to test/predict I get the following error:

ResourceExhaustedError: OOM when allocating tensor with shape[20,16,560,560] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: conv2d_2/convolution = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](activation_1/Relu, conv2d_2/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

I suppose it's a memory issue.

So my question is, is it possible to first use GPU for training and then switching CPU to predict some results in one Jupyter notebook?

Can we free up GPU-memory in windows inside a script?

I found these two topics but I think we should use at the beginning of the script. What I want is switching after training.

https://github.com/keras-team/keras/issues/4613

Switching Keras backend Tensorflow to GPU

Any help will be appreciated,
Thanks

For prediction step, are you using the same batch size as training ?  batch prediction should bring down memory usage.


Using Image Data along with CSV file data for CNN model I have image data along with csv file where each row of csv file contains attributes for corresponding image.

I want to use images as well as csv file data to build CNN model using Keras.

What is preferred way of doing it?

I can append the attribute data with numpy array of image and train the model.
I can create 2 different models and use average/voting for prediction.

I think, in 1st appraoch csv file data which has around 50 columns would hardly matter as image shape is (128*128*3). Is it correct? and how can I give more importance to csv file (like 60% weightage to image data and 40% to csv file).

And again how CNN would work with that CSV file data as CNN is preferred on Image Data?

Kera's functional API allows arbitrary models to be combined.

One option is to define a "main input" as the image that feeds into a CNN and then define an "auxiliary input" that feeds into a MLP. Both of those sub-models can feed to the same higher layers. The weights of the entire model will be updated during training. The model will learn how to weight the data from csv compared to the image data (in contrast to you picking the relative weights).


How to create 3D images from .nii file What are .nii files, and how is data stored in them? I have some of these and I want to know how can I create 3D image of MRI scan from them.

I can load the file in my python script using nibabel. Where to go from here next?

The most common way of processing images in python through numpy arrays. Since you have already loaded your image through nibabel, you need to get the data from the image object and then cast it as a numpy array.

import nibabel as nib
import numpy as np

# Get nibabel image object
img = nib.load("path/to/image.nii")

# Get data from nibabel image object (returns numpy memmap object)
img_data = img.get_data()

# Convert to numpy ndarray (dtype: uint16)
img_data_arr = np.asarray(img_data)

By default this should be a 3D numpy array with a shape of (height,width,image). If you want to access one image you can always use PIL. For example to save the first image of the .nii file:

from PIL import Image
img = Image.fromarray(img_data_arr_norm[:,:,0], 'L')
img.save("image.jpeg")


Are there any machine learning techniques to identify points on plots/ images? I have data for each vehicle's lateral position over time and lane number as shown in these 3 plots in the image and sample data below.

> a
   Frame.ID   xcoord Lane
1       452 27.39400    3
2       453 27.38331    3
3       454 27.42999    3
4       455 27.46512    3
5       456 27.49066    3

The lateral position varies over time because a human driver does not have perfect control over vehicle's position. The lane change maneuver starts when the lateral position changes drastically and ends when the variation becomes 'normal' again. This can not be identified from the data directly. I have to manually look at each vehicle's plot to determine the start and end points of lane change maneuver in order to estimate the duration of lane change. But I have thousands of vehicles in the data set. Could you please direct me to any relevant image analysis/ machine learning algorithm that could be trained to identify these points? I work in R. Thanks in advance.

A first derivative, on the surface, would do it.  However, the data you show have a great deal of noise in them, so we need a way to evaluate the first derivative in a somewhat noiseless way, or at least within a frequency domain that eliminates the jitter and preserves the major derivative change.

Wavelet analysis could achieve this for you, especially if you use the first derivative of a Gaussian as your mother wavelet.  R has some decent wavelet packages (see r-project.org for starters).  If you do the wavelet transform at short scales, this will identify the locations of the bits of jitter in the steering.  If you do it at larger scales (i.e. lower frequency), you can likely find just the lane shifts and not the little jitters.

If you train the transform up with a reasonable data set, you should be able to identify a scale or range of scales that corresponds to lane changes.  But note that if you don't figure that out, this is something like O(n^2), so try to narrow the scale range down a bit to save compute time.


Is vanilla RNN suitable for time series prediction? I read this document: https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/
It was pretty simple, but I don't understand how to use it for predict the next sequence (for example) in trading systems.

In the above article it "loops" and returns each cycle a "bit", but if do I need only ONE output (like in trading systems)?

I suppose that I have to feed "outputs" also/only, is this right?

Thank you

Absolutely yes, Vanilla RNNs can be very good predictors for time series data. You can check this detailed official TensorFlow on time series forecasting, for example.

As they said in part 1:

  the model will be given the last 20 recorded temperature observations, and needs to learn to predict the temperature at the next time step

This also gives you information about how to structure your data for training. Given an input sequence of given length, predict the next n steps of the series - where n can be 1 or more. The choice is very task-specific.

Assuming you are working with univariate time series (i.e. single series with no other variables) then you can shape your X_train as:

( number of observations , length of input series , 1 )

If you are working with multivariate datasets, then change 1 with your number of explanatory variables.

At each training iteration, the model will look at input data, and the actual outcome, and try to "learn" a mapping between x and y.


Matrix based Visualization Meaning - Assocation Rules I didn't found any good resource that explains me very well what type of information I can extract from a Matrix based Visualization like this:

I do't uderstand what RHS and LHS represents

Anyone can explain me the meaning of this chart?

Many thanks!

An association rule is (usually) of the format X -> Y, meaning that if X happens then Y is likely to happen. In the traditional example of supermarket, a possible rule would be something like Bread -> Butter. In this case, X is called the antecedent, or LHS (left-hand side), and Y is the consequent, or RHS.

This specific chart has the number of items in the LHS as it's horizontal axis, and the number of items in the RHS as the vertical axis. The colors represents the metrics of support and confidence, as can be seen in the legend.

Example information that can be read from this chart is that the support is higher for lower number of items in the LHS, because the left side of the chart looks red-ish and the right side looks blue-ish, matching our intuition (more items in the LHS is more restrictive and thus less likely to happen). The confidence doesn't seem to be related to any of the axes in particular.


Feature importance of random forests I have a dataset with 11 features, I noticed that manipulating these features (eg dropping one or some of them) doesn't affect the error scores of training and testing data, so I had to check the importance of these features. Here's the following:

As noticed the first feature has a very high contirbution. However, the rest have insignificant importance. Thus I tried to run the model using only the first feature. It was expected that the results scores will not decrease significantly as the rest 10 dropped features have very low feature importance. However, after running the experiment with only the first feature, the abs error percentage of the testing data increased significantly from 14.13010% to 22.96036%. why is this happeneing? I expected that the error will be almost near to the base testing results as I train using the feature which dominates the feature importance?
Also, some of these features are correlated (no more than .62 correlation), is this the reason why the scores can't be so reliable? if so, what mertic can I use to test the feature importance for correlated features

I can´t give you an perfect answer because there is no code, dataset and the target what you want to achieve.
Because the feature importances from random forest, is calculated based on the training data given to the model, not on predictions on a test dataset. That means, that is not the true prediction power. You should check, if there are difference on training and test results, when you run a random forest model. Another oportunity is the permutation feature importance.
But I made a similar experience and solved this on another way.

check correlation (shows linear dependency)
check with the pps (shows polynomial dependency)
https://datascienceplus.com/correlation-vs-pps-in-python/
check feature and permutation importance

With these 4 options, I got a better view into my dataset.
Hope I could help you a bit.


Good model but bad confusion matrix? I am trying to understand the code here.
The output [12] shows that the model accuracy is above 90% even for the validation set, but the confusion matrix in [16] ca not even achieve 50%  accuracy, and it is also on the validation set, so I do not understand this low accuracy on the confusion matrix. I think it may be due to data augmentation, but I would be thankful if someone could explain it to me and tell me how I could then have a confusion matrix in adequacy with the learning curves. Thanks in advance.

I found the answer in https://stackoverflow.com/questions/55868975/keras-how-to-evaluate-model-accuracy-evaluate-generator-vs-predict-generator/57630536#57630536
I just had to add shuffle = False in the test_datagen.flow_from_dataframe, and the confusion matrix was good


Feature selection for regression Suppose I have a response variable y and and a set of feature variables (x1, x2 ... xn). I wish to find which of x1...xn are the best features for y in a regression problem (the relationship might not be linear).
Is there any way I can do this kind of feature selection without using any correlation measure or regression function in the process (i.e. I cannot use any filter or wrapper methods)?

You can train a LightGBM Regressor. LightGBM Regressor feature selection methods embbed in it. You can direct plot them to see which features are important.
See this link.
LightGBM Plot Importance


Compute gradients in parallel Here is part of my code:

class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(2, 1,  bias=False)
        self.linear2 = nn.Linear(1, 2,  bias=False)

    def forward(self, x):
        z = self.linear1(x)
        y_pred = self.linear2(z)

        return y_pred, z

model = SimpleNet().cuda()

for epoch in range(1):
    model.train()
    for i, dt in enumerate(data.trn_dl):
        optimizer.zero_grad()
        output = model(dt[0])

        loss2 = 0
        for j in range(0,len(output[0])):
            l1 = torch.autograd.grad(output[0][j][0], output[1], create_graph=True)[0][j]
            l2 = torch.autograd.grad(output[0][j][1], output[1], create_graph=True)[0][j]
            loss2 = loss2 + abs(torch.sqrt(l1**2+l2**2)-1)
        loss1 = F.mse_loss(output[0], dt[1])
        loss = loss1+loss2
        loss.backward()
        optimizer.step()
    if epoch%100==0:
        print(loss1,loss2,loss)

So I need the gradient of the output layer with respect to some node (this is a simple example, the real one has more layers in between), which I calculate using torch.autograd.grad(output[0][j][0], output[1], create_graph=True)[0][j]. However the way I do it now requires that for loop, over each element of the batch which is very slow. Is there a way to take this gradient all at once for a batch? Thank you!

The developers docs for torch specify that the arguments can be tensors, so you can just pass the tensors directly, but you need to reshape your output so that j is the last index. First, rewrite your code so that j is the last index and in your for-loop you have

l1 = torch.autograd.grad(output[0][0][j], ...)

And then drop the for-loop and remove the j-indices.


Dimension-Hopping in Machine Learning What is the dimension hopping problem in machine learning (occurring in convolutional neural networks and image recognition)? I have googled about it but all I get is information on the Physics of material shape deformation. It will be more helpful to me if some one explain it with an example related to machine learning. Can anyone help me out with this or point me toward resources that can?

Welcome to DataScience.SE! I'd never heard of this problem so I looked it up. It is explained on the third slide of this presentation by Geoff Hinton:

  More things that make it hard to recognize objects

  • Changes in viewpoint cause changes in images that standard learning
  methods cannot cope with.

  –  Information hops between input dimensions (i.e. pixels)

  • Imagine a medical database in which the age of a patient sometimes
  hops to the input dimension that normally codes for weight!

  –  To apply machine learning we would first want to eliminate this
  dimension-hopping.

In other words, it is about conceptual features migrating or hopping from one input feature dimension to another while still representing the same thing. One would like to be able to capture or extract the essence of the feature while being invariant to which input dimension it is encoded on.


python - Will this data mining approach work? Is it a good idea? I need to extract fields like the document number, date, and invoice amount from a bunch of .csv files, which I believe are referred to as "unstructured text." I have some labeled input files and will use the NLTK and Python to design a data extraction algorithm.

For the first round of classification, I plan to use tf-idf weighting with a classifier to identify the document type - there are multiple files that use the same format.

At this point, I need I way to extract the field from the document, given that it is X type of document. I thought about using features like the "most common numbers" or "largest number with a comma" to find the invoice amount, for example, but since the invoice amount can any numerical value I believe the sample size would be smaller than the number of possible features? (I have no training here, bear with me.)

Is there a better way to do the second part? I think the first part should be okay, but I'm not sure that second part will work or if I even really understand the problem. How is my approach in general? I'm new to this kind of thing and this was the best I could come up with.

I am not sure if using a classifier is the best way to approach this problem.  If it is something which can be easily extracted using regex, then that is the best way to do it. If however, you want to use classifiers, here are two questions you need to ask yourself.

One, what does the unlabelled data look like and can you design good features from it? Depending on the kind of feature vector you design, the complexity of the classification task may range from very easy, to impossible. (A perceptron cannot solve XOR usually, except when you provide it with specific linear combinations of the input variable).

Two, what does the labelled data look like? Is it representative of the entire dataset or does it only contain very specific types of format? If it is the former, then your classifier will not work well on files which are not represented in the labelled data.

If you just want to test run a classifier first, you can solve the problem of having more features than training samples by using Regularization. Regularization forces the training algorithm of the classifier to accept the simplest possible solution (think occam's razor).

Almost all Machine Learning related packages in Python will have regularization options you can use, so enjoy.


Train a model for unstructured data I am new to Machine Learning. I want to develop Curriculum Vitae recommender system. I want to determine how similar 2 CVs are, and given a random CV, it suggest which cluster of CVs it belongs to?

This is what I've already done, following a blog post:

I have a folder containing lot of CVs or resume text documents in plain text format (.txt).
I have done pre-processing on this data, like tokenization, stop words removal, stemming.
I extracted the Candidate's name, email-id, contact number, education and experience.

I am confused with how to train the data and how do I create a model for that. More specifically, I have the following questions:

Now how to create a model on text data?
Which algorithm I should apply on this data?

Please anyone answer. Your help will be appreciated.

Thanks.

I have worked on a similar project with JDs, we basically created a word2vec model for words in JD, the result were good as we had lots of JDs.
Basically, what word2vec does is convert a word to vectorial representation which signifies context.
You can check the documentation here: https://radimrehurek.com/gensim/

You may extract skills or other stuff from CVs, and do a semantic similarity based on w2v model. You may use a custom formulae for comparing similarity. Other things could be education, experience, similar projects etc


Grad_CAM for time series I am new to deep learning and trying to build a Grad-cam from time series data. Shape of my input sample  is (188,1), its an ECG signal and I have a cnn-1D model for classification. Keras provides function  Grad-cam for images(2D). I am trying to convert this function for 1D time-series data. Could anyone please help me how to do it. Thank you in anticipation.

check out the following paper. It purposes a 2 stage network and then used the output of the first stage to do grad CAM.
https://ieeexplore.ieee.org/abstract/document/8970899/


How to properly train your Self-Organized Map? I stumbled recently upon the Self-Organized Map, an ANN architecture used to cluster high dimensional data, while simultaneously imposing a neighbourhood structure on it. It's trained through a competitive learning approach where neurons compete to respond to a given input. The strongest responding neuron / best matching unit (BMU) is rewarded by being moved closer to the given input in the data space, as well its neighbours. However, within the literature and implementations, I find some deviations in how this training is implemented. Specifically, the influence of the BMU on its neighbors are mitigated using a neighborhood function

where d is the distance of the BMU to the input and σ(t) is a radius which is decreased during the training. Effectively, resulting in the influence of the readjustment of the BMU on its neighbourhood shrinking during training. The difference I find concerns the implementation of the shrinking of σ(t) . Most explanations and blog posts describe an exponential decrease

where λ is a decay constant which can be tuned arbitrarily. Alternatively, I find that some implementations do not really use this exponential decay, but instead use a linear interpolation of the form

where n is the number of training epochs and r is radius which is altered depending the training phase. These implementations further explicitely between a 'rough' training phase where

with e.g. SOM.dims=(100,100) being for a 100x100 sized SOM, and a 'fine-tuning' training phase where

My problem is that I do not quite understand why there seems to be this disagreement and what the 'canonical' way of training a SOM is. It certainly makes sense to divide the training into a 'rough' and a 'fine-tuning' phase, but why most newer descriptions neglect this without further discussion and only consider a single training phase with exponential decay is baffling me a bit.

An answer from Kohonen, inventor of the self-organized map himself:

"The true mathematical form of σ(t) is not crucial, as long as its
value is fairly large in the beginning of the process. Say, on the
order of half of the diameter of the grid, whereafter it is gradually
reduced to a fraction of it in about 1000 steps."

From: Kohonen, T., 2013. Essentials of the self-organizing map. Neural networks, 37, pp.52-65.


How to treat data transformation choices as hyperparemeters? While reading the book hands-on ML by Aurelien Geron, I came across this line-

Treat your data transformation choices as hyperparameters, especially
when you are not sure about them (e.g., if you’re not sure whether to
replace missing values with zeros or with the median value, or to just
drop the rows).

How exactly do I do that? Is there a way to do it via sklearn or do I have to manually keep several datasets (each with a different transformation) and then fit models onto all of them?

So, the question talks about how to treat transformation choices as hyper parameters.
How I would go about it is the following:
Use one baseline model architecture for the data and then repeat the following:

Instantiate the baseline model (effectively make sure all of the weights are initialised)
Create the transformed dataset
Train the model
Compute generalisation performance measures (AUC, precision, recall, whatever).

Then compare the generalisation performance across all of the data transforms to find the "best" transformation which improves a generalisation metric which is appropriate for your task.


The model performance vary between different train-test split? I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data.

What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.

To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.

Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation.


Hyperparameters in Gaussian Process My academical background is in physics and analysis (PDE's), but now I'am reading about data science.

I'm reading about Gaussian Process implementation in Sci-Kit Learn

I could not find a simple explanation about those hyper-parameters theta0, thetaL and thetaU. Can someone explain to me what those parameters are, and how to choose then?

As you probably know, the Gaussian distribution is fully specified by its mean and covariance. theta0, thetaL, and thetaU are, respectively, initial values, lower bounds, and upper bounds on the covariance. You can set them based on your knowledge of the particulars of the problem. To learn more about Gaussian processes, I suggest visiting this great Web site. It even offers a free ebook.


Is it a Problem if Training Data and Evaluation Data are Very Similar? I'm working on a group project where we have decided to use this Microsoft WIP repository as our starting point. It's a project which compares different frameworks and models and their ability to do NER in order to filter personal data.

We have several out of the box models we're evaluating from frameworks like SpaCy and Flair. We're also training additional models based on data generated with a data generator that is included in the project.

Recently I came to realize that the data generator actually just takes a file which includes templates, for example the line below:

The name in the account is not correct, please change it to [PERSON]

Then the [PERSON] part would be replaced with a name from another file including several names.

I came to realize that there are only 125 templates. These 125 templates are then used to generate 213 training samples, 59 test samples, and 28 validation sets. Thus of the 300 samples used for training our models over half are near duplicates, for example:

The name in the account is not correct, please change it to John Doe
The name in the account is not correct, please change it to Jeff Peters

We then have an additional step where we compare all the models on a different set of data. However this "different" set of data is just 300 more samples generated from the same 125 templates.

I believe this lack of diversity in data, and similarity between training and evaluation data makes all scores obtained so far invalid.

However my project partner does not believe this would have a negative effect. While I feel strongly that the data will create a problem I'm unable to provide strong citation or references to prove this to him.

He is saying we should "deconstruct the algorithm to show that sentence structure isn't as important to a machine as it is to humans".

Can anyone confirm that what I'm describing will be an issue and invalidate our results, and potentially provide references to back the claim?

Our aim is to compare models and their general performance for NER, not just their ability to perform on this specific data-set.

In general, in Machine Learning, if the train set and test set are very similar, then it leads to a small ability to generalization. In other words, the model not performing well on new data. It's called overfitting.

In your case, could be happened more, if some rows are the same in train and data set, then we could say that it's data leakage.


Ranking of feature sets which will be used for binary classification I have some feature sets say X1 and X2 ...
Each feature set have some variable amount of features and there is no intersection between different feature sets
Say X1 have 100 features and X2 have 500 features and none of the features they had is same .
Though the file from which these features are extracted is same for both x1 and x2.
Each feature set can be used individually for binary classification.
Like X1 can also be used for classification and X2 can also be used for classification. They can also be used together combined i.e X1 U X2 which will have 600 features
But I want to compare the feature sets and hence want to know if there is any statistical method which can be used to rank these feature sets other than classification accuracy
Like X1 is better than X2 and so on ...

You could use some statistical method to rank these feature sets, such as extracting in each features set a discriminative score per feature (kolmogorov smirnof, mutual information, ...) and then take the mean, the median, the p95, ...
But, depending on the classifier that will be used, this score may be useless, because some classifiers act as features selectors (decision trees, neural networks, ...) and others don't (Gaussian NB, kNN,...).
Example
If you have 10 features in your 500 that are really good to discriminate your classification and the other 490 that do not contain information. Using a decision tree based classifier will lead to good performance as probably only the 10 features will be used. kNN in the other hand will lead to bad performance...
However, your features set score would remain the same.


Represent time-series data in much compact form I have time series data of one month plotted day-wise as

Notice every day follows a different pattern. Now, I want to show this "diversity in pattern" of each day in much compact form in a research paper. What are the different ways/options of representing this in compact form using R.

Simulate some data:

library(ggplot2)
library(purrr)
library(ggthemes)

days <- seq(as.Date("2015-08-01"), as.Date("2015-08-31"), by="1 day")
hours <- sprintf("%02d", 0:23)

map_df(days, function(x) {
  map_df(hours, function(y) {
    data.frame(day=x, hour=y, val=sample(2500, 1), stringsAsFactors=FALSE)
  })
}) -> df

Check it:

ggplot(df, aes(x=hour, y=val, group=day)) +
  geom_line() +
  facet_wrap(~day) +
  theme_tufte(base_family="Helvetica") +
  labs(x=NULL, y=NULL)

Since you're only trying to convey the scope of the variation, perhaps use a boxplot of the values of hours across days?

ggplot(df, aes(x=hour, y=val)) +
  geom_boxplot(fill="#2b2b2b", alpha=0.25, width=0.75, size=0.25) +
  scale_x_discrete(expand=c(0,0)) +
  scale_y_continuous(expand=c(0,0)) +
  coord_flip() +
  theme_tufte(base_family="Helvetica") +
  theme(axis.ticks=element_blank()) +
  labs(x=NULL, y=NULL)

That can be tweaked to fit into most publication graphics slots and the boxplot shows just how varied each day's readings are.

You could also use boxplot.stats to get the summary data and plot it on a line chart:

library(dplyr)
library(tidyr)

bps <- function(x) {
  cnf <- boxplot.stats(x)$conf
  data.frame(as.list(set_names(cnf, c("lwr", "upr"))), mean=mean(x))
}

group_by(df, hour) %>%
  do(bps(.$val)) %>%
  ggplot(aes(x=hour, y=mean, ymin=lwr, ymax=upr, group=1)) +
  geom_ribbon(fill="#2b2b2b", alpha=0.25) +
  geom_line(size=0.25) +
  theme_tufte(base_family="Helvetica") +
  theme(axis.ticks=element_blank()) +
  labs(x=NULL, y=NULL)


What's beyond topic modeling? I tried topic modeling (LDA, NMF) to extract insights from the data.

I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context?

(Aside) Are there any methods to show similarity of a topic or documents from topic?

You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical clustering, or K-SOM.

The doc2vec model you create will be able to compute cosine similarity between two documents and also find the n most similar documents to a given document and provide a similarity score for each.


How to determine the best way to replace empty records in the dataset? I am doing a project related to predicting the next glucose value based on his/her past records. But in some patients, some of the recordings are missing. There are 2 scenarios of how the blood glucose values are missing in my dataset. So I'll label them as Scenario 1 and Scenario 2 in my examples.

Scenario - 1

A patient has recorded the glucose values thrice a day basically before breakfast, lunch and dinner on day 30. But on day 31 he has only recorded only the breakfast and dinner. Lunch is not recorded on day 31. So how can we replace the 0 value in lunch with another value?

Scenario - 2

Another patient has recorded the blood glucose values continuously from day 1 to day 40, and then he hasn't recorded the blood glucose values for another 2 days (day 41 and day 42 recordings are not there for before breakfast, before lunch and before dinner). Again he started recording the values in the day 43. So what is the best approach to tackle this kind of scenario?

I went through many articles and majority explained on replacing the mode, median or mean values for empty records. But I think mode, median is not suitable for this kind of dataset. I highly doubt whether I can use the mean also to replace values with empty records in the Scenario - 1. Can we actually use mean for replacing the empty records in Scenario - 1 or is there any other good approach?

From my knowledge I think I can't use the above three methods in replacing the values in Scenario - 2 since about 6 records are missing. If I am correct what is the best approach for the Scenario - 2.

Thank you!!!

Replacing missing values with mean, median and mode is feasible when the number of missing data is not small enough. In your cases, only a handful of data are missing. We can use the rest of the data to come up with estimates better than mean, median and mode. Let's see the possible approaches:

Scenario 1: using the reading for breakfast and lunch for day 31, try to find the nearest neighbours in day in terms of breakfast and lunch values. You can use this nearest neighbour day's dinner value as an estimate of the 31st day's dinner.

Improvement to above approach:

1) Instead of just choosing the single nearest neighbour, you may also use a kNN model.

2) If you have the dates or can figure out the day of the week (like its Sunday, Mon, Tues etc) then instead choosing the nearest neighbour among all the days, you can choose only among the same day of the week. (Nonetheless, you can figure it out by simply subtract or adding 7 to the day you want to find the values of.) Say 31st was a  Sunday, then its likely that the routine would be similar to others Sundays on the data rather than a working day like Monday.

Scenario 2: You can use the values of the same days of the week to estimate the missing values. Simplest ones would be to take a median. A more powerful estimate would be by using a small neural network to estimate the next day's values given the values of the past few days (here, the number of days to consider as input would be a hyperparameter).


Python and Titanic competition how to get the median of specific range of values where class is 3 I am trying to solve Kaggle's titanic competition. In the test set, there is only 1 row having Fare value as null. It's so easy to replace it with median or average of all Fare values.

But I am thinking of getting a graph showing relation between fares and classes (1, 2, 3) and fares with Embark field, to check some patterns and to narrow down the range of fare values before doing the calculations to replace null values.

The Embark field having 3 main values: S, Q and C.

The class field is having: 1, 2 and 3 as classes.

Fares are changing based on the class of course.

What I have done so far is:

fig, axs = plt.subplots(ncols=2, nrows=2, figsize=(20,20))
sns.scatterplot(test['Embarked'], y='Fare' , data=test,  ax=axs[0][0])
sns.scatterplot(test['Age'], y='Fare', data=test,  ax=axs[0][1])
sns.scatterplot(test['Pclass'], y='Fare', data=test,  ax=axs[1][0])
sns.scatterplot(test['Sex'], y='Fare', data=test,  ax=axs[1][1])

And the result was:

There is no patterns at the Embarked, Age and Sex plots.

But at the Pclass plot, we can clearly see that there is stricly defined range for class 3.

I tried to check it at the train set:

So the are nearly the same distribution.

How can fill the null value with median of values range of people having Pclass as 3?

the regular expression would be:

test['Fare'] = test['Fare'].fillna(test['Fare'].median())

But I really need to specify the range of values where Pclass=3

If I understand the problem correctly, you want to fill all missing values in the Fare column by the median value of the Fare column where Pclass=3. This can be achieved by putting the extra row filter test['Pclass']=3 on median of the fare column, see below.

test['Fare'] = test['Fare'].fillna(test.loc[test['Pclass']=3,'Fare'].median())


Predict correct answer among ten answers for a given question I have a case study to solve where I am given a dataset of questions and its answers, there are ten answers for a particular question.

It's a classification problem where correct answer is having class_label = 1 and all other nine answers having class_label = 0.

Which deep learning model would fit best for this type of case study and how should I proceed?

It's not DL but I suggest you start with the following approach: for every question $Q$ and its set of candidate answers $(A_1,..,A_{10})$, represent each pair $(Q,A_i)$ as an instance with its label 0 or 1. You could start with a few simple features such as:

number of words in common
similarity score, e.g. cosine TFIDF
... other indicators of how well question $Q$ and answer $A_i$ match

Train a regression model on this (e.g. decision tree, SVM,...). When the model is applied to a new question+answers, it returns a score (mostly between 0 and 1) for each of the 10 answers; finally select the answer which has the highest score.

You can certainly improve on this idea, e.g. with sentence embeddings.

Note: a simple baseline system based on the same idea would be to select the answer which has the maximum number of words in common with the question.


Intensity image to RGB for transfer learning My goal is to use a pre-trained model with an intensity-based image.
Most pre-trained models expect RGB (int) format as the input image.
An easy workaround is to duplicate the intensity channel 3 times in order
to obtain a greyscale image.
However, my intensity image has a large scale (not 0 - 255), so this solution will not work. I could squeeze the range, at the cost of information.
Fortunately, RGB space is 256256256. So it is the space to contain a very large intensity range. But what is the "best" way to do so?
Actually, I use matplotlib to apply color map on my image, which kinda does what I want but I'm not sure if that's correct, nor which colormap is the best.

If you mean images classification networks like AlexNet (and I suppose images segmentation ones too), then at first layer they detect color-agnostic features as well as color-dependent ones:  (https://arstechnica.com/science/2018/12/how-computers-got-shockingly-good-at-recognizing-images/3/). Here blue color mean that blue component is necessary for the filter to fire stronger, but (I believe) that G and B and not important for it. For magenta (R+B) red and blue are necessary anb green is not important. If the filter is white on the left, black on the right, it will fire when intensity raises from right to left.

Transfer learning has to work if your image has the same nature as the train photos (at least partially). So you want filter detecting legs to detect something similar to legs on your images. And it will detect if lower-level filters detect parts of leg (say rectangle - increase of intensity when background is darker than the leg and then decrease back – background-leg-background). If your leg is detected as a set of petals connected with dogs’ noses instead, there will be no sense in this, even if source net detect this as flowerbed with confidence. This will be a noise.

So you of course could pack your color intensities into RGB triples. Say for simplicity in the form of C letter on the R+G plain. You will get 256*3 values. But this would break the correspondence with the nature of your image. Since the net will see the packed values. Even if everything is good when you move from RG (0, 0) -> (255, 0) -> (255, 255), instead of intensity raising, it will decrease on the way -> (0, 255). This will likely rotate inside out filters that should detect this exact raising of intensity. Sometimes maybe it’s not important, there are pairs of opposite filters or somehow else… But I think far from always so still…

I would simply map your 0-N to gray (0, 0, 0)-(255, 255, 255), loosing of precision is so small and typical problem in real-life tasks... If you really need this resolution, maybe take windows of maximally appropriate size (e.g. v=0 for input 0-1000, linearly to 255 for input 1000-2000, 255 for bigger input values). Everything is MHO and as a first idea. More information about your task could help.


I need to find Simplest exercises available for learning neural networks I'm learning neural networks, and I'd like to give myself simple exercises and try to train neural networks to solve them.

I know that classifying hand-written digits is a classic problem, but I'm hoping for something much simpler than that. No image recognition, no thousand of input neurons.

One exercise I thought of, is to get a number and find its modulo when dividing by 7. But it was explained to me that this is actually a difficult task for a neural network beginner for various reasons.

Does anyone have ideas for a very simple exercise/training idea? Or a link to an existing list of exercises?

I'm learning myself too and I found kaggle.com and excellent resource for learning and practice, they have lots of data sets available for you to try your own models.

There is also a learn section with lots of topics for beginners

Hope that helps.


Why to exclude features used for label generation during modeling? I have a dataset like below without labels

But with the help of experts opinion, we generate labels based on the below 3 rules (all 3 rules has to be met to label it as 1)

So now the dataset looks like below

As you can see that my final dataset has the labels.

Now I can run a ML model for classification. Am I right?

But I read that during model building process, features that were used to create the labels will have to be excluded because they might result in perfect separation of classes and model might fail. what does it mean by fail? Aren't we aiming for separation of classes through classification algorithms?

May I know why do we have to exclude these features (Ex: RG, FG and BP features which were used to derive labels)?

It's basically my model will be built on below dataset. But aren't we losing the predictive power? why do we have to build a model by excluding those features (that were used to derive labels)?

What you are doing is right. You can build a ML model using this. In your case your input and output are correlated. Consider the salary of an employee wrt his experience. These both are related and sometimes used to derive salary based on experience.

What you might have read is if one feature is used to derive other feature and dont use both features, as both will yield same results. You can point to that writing for further clarification.

Additionally, what is the significance of variables T1,T2 and T3? If these are nowhere correlated to the output label then actually you would have to eliminate them.


Tensorflow skewed cost function I am writing a TensorFlow program which is trying to categorize a heavily skewed dataset between two categories. One category is represented at 30x the rate of the other.

Category 0: 800
Category 1: 30000

The label for each record is a one-hot vector of length 2, representing the two categories. Using a cross entropy cost equation doesn't work very well since a very high accuracy can be attained just by rating everything as category 1.

I then made a balanced dataset that has the following data:

Category 0: 800
Category 1: 800

This dataset has about 67% accuracy using softmax, but there is a real-world cost when falsely categorizing a category 1 item as category 0, which is blown way up when mistakenly categorizing many more items as category 0 mistakenly.

I wanted to fix this by making the cost equation penalize false categorization into category 0 more than other penalties. However, my cost equation is not producing any different results than the original cross entropy equation. I think I may have programmed it in wrong, but I am unsure where the mistake is.

In theory, the skewed equation will add 1 to the normal cost when the label for a record is [0,1] and the predicted value is [1,0], which represents a miscategorization of a category 1 item into category 0.

The skewed cost equation is in the condition where FLAGS.cost == 'skewed':

x = tf.placeholder(tf.float32, shape=[None, data_length])

W = tf.Variable(tf.zeros([data_length, 2]))
b = tf.Variable(tf.zeros([2]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, shape=[None, 2])

if FLAGS.cost == 'cross_entropy':
    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))
elif FLAGS.cost == 'skewed':
    positive_cutoff = tf.constant([0.5, 0.5])
    desired_tensor = tf.constant([False, True])
    true_tensor = tf.fill(tf.pack([2, tf.shape(y)[0], 2]), True)
    cost = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y,1e-10,1.0)))+tf.to_float(
        tf.reduce_all(
            tf.logical_and(
                tf.pack([
                    tf.logical_and(tf.greater_equal(y_, positive_cutoff), desired_tensor),
                    tf.logical_and(tf.less_equal(y, positive_cutoff), desired_tensor)
                ]),#shape [2,x,2]
                true_tensor#shape [2,x,2]
            )#shape [2,x,2]
        )#shape [1]
    )

train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

The equation I ended up using is listed in the link below, provided by Emre:

Linear regression with non-symmetric cost function?


How do I get Twitter Dataset for Visualization I am a newbie. I would like to do visualization on twitter data : top trends based on country (over map) and time variations (for each months in 1 year or for each year) . Can someone tell me where can I get the twitter data set and any advice on how to start proceeding would be really help full.

Thanks

(Fresh and complete) Twitter data is not easy to get nowadays, as Twitter is actively monetizing it (through GNIP). There is still a large Twitter graph collected by Kwak et al. for their "What is Twitter, a social network or a news media?" WWW'10 publicly available, but as to tweets, most nice datasets have disappeared from the Internet due to complaints from Twitter. There are still tiny datasets of tweets available online (e.g.), but they are small and stale. Thus, I see two "legal" ways to go about getting tweets now: 1) crawl Twitter using the new version of Twitter API (the obvious limitation of this approach is the limit on the number of requests you can send; this approach is unlikely to give you a representative picture of what is trending if you collect raw tweets, just because you won't have enough tweets); 2) buy data from Twitter (GNIP), which will cost an awful lot of money. Alternatively, you can 3) study Twitter API for other means of getting "summarized" information from it, not tweets directly. For example, they may provide a method to retrieve, say, 10 most popular hash tags at the moment, maybe, even filtered by geo-location -- you cannot do much data science with the result, but at least you can use it in your visualization application.


How to convert DNA sequences in FASTA format to OneHot Encoded Pandas Dataframe for Neural Networks? DNA sequences in FASTA format look like:
CATGCATTAGTTATTAATAGTAATCAATTACGGGGTCATTAGTTCA...

I am trying to convert them into one-hot encoded data in a Pandas dataframe so that I can use various neural networks to analyze them. This has probably been done many times. Can someone point me to references or Python packages for it?

I am not sure about any python package but you can simply do integer encoding using labelencoder() and then do one-hot encoding.
label_encoder = LabelEncoder()
integer_encoded_seq = label_encoder.fit_transform(seq)
onehot_encoder = OneHotEncoder(sparse=False)
integer_encoded_seq = integer_encoded_seq.reshape(len(integer_encoded_seq), 1)
onehot_encoded_seq = onehot_encoder.fit_transform(integer_encoded_seq)

Refer this.


image based example for Bayes error/irreducible error I have a hard time intuitively understand the Bayer error in the context of supervised learning. We have an input X and an output Y. We want to find the function f(X) = Y.

I feel like I don't understand why we model X as a random variable in the first place. I think we construct the function f not based on the distribution of X but rather on actual values of X. Why should we care if X is stochastic or not?

Example I:
Let's say X = [1,2,3], Y = [1,2,3] and f(x) = x.
When X=1 => Y=1, X=2 => Y=2, X=3 => Y=3, I will never make an error.

Example II: Why should it be different if X = [picture of dog, picture of cat, picture of house] and Y = [dog, cat, house]. I can still find a function which does the mapping. It is obviously more complex but doable.

Where did the Bayes error get lost in my examples?

I am looking for an intuitive explanation of the Bayes error preferably in the context of image classification.

As an MNIST dataset, see the mispredicted images on this figure below,  you notice for some samples, we as human don't know for sure what is the true label.

Maybe there are other factors that are not visible to us to predict the correct output.

Assume I want to predict customer will leave our company, maybe there are tons of useful features to start with, yet he may simply have some other problems that we are not aware of, and it may affect his decision in leaving the company.


Neural network accuracy for simple classification I am trying to develop a NN for a very simple classification model with keras/tensorflow:

Ex:

input: "Do" => class output: "Dog"
input: "Ca" => class output: "Cat"
input: "Mo" => class output: "Mouse"

I train the model with many "Do", "Ca", etc. (as dictionary), indexing the input to categorical arrays ([0,1,0] for example is "Cat", [1,0,0] is "Dog", [0,0,1] is "Mouse" in case of three classes).

I know that for this type of problem every other traditional classification algorithm should be used, and not DL, but I'm doing it with a purpose (I need to merge it with an image classifier).

My question is: since the classification is easy ("Do" is always "Dog", "Ca" is always "Cat") a Decision tree would have an accuracy of 100% always. Why do Neural Networks need tons of data and epochs to get an acceptable accuracy? With 10 classes, it takes 10 epochs and 8 thousand entries in order to get an accuracy higher than 90%. I'm using sparse categorical crossentropy as loss, and SGD as the optimizer. (2 Dense layer, relu - softmax). Also, I am a bit lost on how to choose the number of neurons, I guess trial and error is the way.

I will add my 2 cents at the end of this answer. However, this is how it can be done using a neural network. Firstly, yes, you should expect to need more data to train even a simple neural network because their are more parameters that need tuning. Think of them like little faucets that you need to tune in order to get the right output volume based on an input. If you have millions of these faucets you an imagine that this is an arduous process.

You will need some of the following imports

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

But, in your case you know what your data should be so you can simulate it. I do this as follows and make a training and testing set.

import numpy as np

n = 10
m = 2

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
    label = np.random.randint(0,m)
    y_train[i] = label
    x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
    label = np.random.randint(0,m)
    y_test[i] = label
    x_test[i, label] = 1

Now we will have a training set which contains $n$ instances and a testing set with a third as many. $m$ is the number of possible inputs. For cat vs. dog this would be $m=2$. You will be using your more general case where $m=10$. Each entry in the matrix $x$ has the vector with one-hot encoded vector where the index in accordance with the label is 1.

We need to reshape the data for it to fit with the Keras structure.

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)
input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

We then build our model

model = Sequential()
model.add(Dense(64, activation='relu', input_shape = input_shape))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

We then train our model

epochs = 4
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  Epoch 4/4 10/10 [==============================] - 0s 251us/step -
  loss: 0.6247 - acc: 0.6000 - val_loss: 0.5311 - val_acc: 1.0000

Voila, now you have perfect classification with this network. You can play around with the model and see a summary of the model using.

model.summary()

For $m = 10$

Due to the higher complexity that this set has you will need to increase the number of instances in your training set. I will also 2 layers to our model and make them less wide. Furthermore, we will add more epochs so we will train longer.

import numpy as np

n = 1000
m = 10

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
    label = np.random.randint(0,m)
    y_train[i] = label
    x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
    label = np.random.randint(0,m)
    y_test[i] = label
    x_test[i, label] = 1

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)

input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Dense(32, activation='relu', input_shape = input_shape))
model.add(Dense(32, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  Epoch 10/10 1000/1000 [==============================] - 0s 49us/step
  - loss: 1.5977 - acc: 1.0000 - val_loss: 1.5235 - val_acc: 1.0000

My suggestions

I would not use a NN for such a case. Most of the frameworks allow you to add information throughout your model. Such that if you have images you can run a CNN over them, then when you are ready to convert your layers to a densely connected layer you can in additional information, such as vectorized text.

You can thus use a random forests approach or something even simpler to get your 100% classification even faster. Then you can feed the output of this model to your deep learning framework which has already "extracted the features" from the images and concatenate these additional features to that tensor. Then you will pass this larger tensor through the subsequent Dense layers to get your final output.


Parking Prediction based on Mobile application I am quite new to predictive modelling but have knowledge of GIS, R, python, SQL, etc.

I am currently doing a project in work trying to predict when parking spaces will be available based on data received from a mobile phone application.

I have 2 sql tables

ParkingTickets

ParkingTicketId  (Integer)
ParkingAreaId    (Integer)
Latitude (Integer)
Longitude    (Integer)
Location (Text)
ParkingDate  (DateTime)
DurationInMinutes    (Integer)
ExpiryDate   (DateTime)
Day (Text)

ParkingAreas:

ParkingAreaId (Integer)
MaxSpaces (Integer)

Main assumption is that only app users can park in the 18 parking lots (both on-street and off-street lots (no multi store or underground)). I do not take in to account user preference, weather, events, etc., etc. This is purely presence and absence based on data at hand. I have researched techniques and was looking into using a birth/death model but struggling to find examples of it in use.

Any help or pointers on models to use would be brilliant!

sample data:

 ParkingTicketId    ParkingAreaId   Latitude    Longitude   Date    DurationInMinutes   ExpiryDate  Day
60465   302 42.56246869 -70.91313754    2014/03/07 16:36    5   2014/03/07 16:41    Friday
60466   302 42.57139883 -70.91906364    2014/03/07 16:36    23  2014/03/07 16:59    Friday
60467   302 42.54419925 -70.9417496 2014/03/07 16:36    24  2014/03/07 17:00    Friday
60472   302 42.57576595 -70.92876607    2014/03/07 16:36    16  2014/03/07 16:52    Friday
60477   302 42.55573294 -70.92912634    2014/03/07 16:36    9   2014/03/07 16:45    Friday
60479   302 42.55711998 -70.91200458    2014/03/07 16:36    19  2014/03/07 16:55    Friday
60480   302 42.58008043 -70.91559081    2014/03/07 16:37    5   2014/03/07 16:42    Friday
60485   302 42.55161223 -70.9240808 2014/03/07 16:37    21  2014/03/07 16:58    Friday
60492   302 42.58437849 -70.92764527    2014/03/07 16:37    6   2014/03/07 16:43    Friday

ParkingAreaId   MaxSpaces
302 8
304 50
306 95
308 30

If this data is coming in real-time then you don't need a model -- simply check how many spots have an ExpiryDate greater than now (i.e. when you need to provide a prediction) and subtract this from the total capacity of spots.
If the data is not coming in real-time, then you could use time of day and day of week as predictors. You might even want to make them into interaction terms. You would also need to decide how often you want to call your model and group your data so that each row represents how many tickets were active during that timeframe; this will define your target variable (what you're trying to predict).
By the way, I think you're referring to a survival model. I would recommend gradient boosting instead; it's much more powerful. Gradient boosting models (GBM) are part of caret in R and scikit-learn in Python by the way.


How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system? My Named Entity Recognition (NER) pipeline built with Apache uimaFIT and DKPro recognizes named entities (called datatypes for now) in texts (e.g. persons, locations, organizations and many more).

I have a gold corpus and a result corpus which I want to calculate precision, recall and F1 score on. As of now, I calulate these metrics like this:

1. Calculate precision, recall and F1 score from TP, FP and FN per datatype per document
2. Average precision, recall and F1 score per datatype for all documents

In the table you can see the results of step 2 in the corresponding datatype rows.

Regarding step 2: I think the way of calulating the F1 score is neither macro nor micro. I calculate precision and recall in the macro way (like explained here). But I don't calculate the F1 score as the harmonic mean of the average precision and recall (macro way), but as the average F1 score for every datatype for all documents. I am getting higher results for the macro-way F1 score compared to the way I am doing it at the moment.

Question: What is the right way to calculate the average F1 score for every datatype? Both ways seem to be correct to me. Please name sources for your answers.

There is a quite detailed comparison with references here: https://towardsdatascience.com/a-tale-of-two-macro-f1s-8811ddcf8f04

Basically the two definitions are used and both can be considered valid. For the sake of clarity I would recommend mentioning which definition you are using when you report your results.


Calculate future GDP % using machine learning I need to estimate the GDP % of a country three years into the future, based on historic data.

I have 30+ years of the following monthly data that includes features such as inflation and unemployment rate:

Year  Month  Inflation  Unemployment %  Other Features  GDP %
1990    1       1.1           6.2           ....         2.3
1990    2       1.3           6.1           ....         2.4

...
...
...

2019    6       0.8           4.8           ....         3.1
2019    7       0.9           4.9           ....         3.3

Using Random Forests (Python's scikit learn library), I can use the data to calculate the GDP of next month, 2019/8:

Once calculated (let's say GDP % = 3.2) I use this value and rerun the entire Random Forest process, including in the history data the value I obtained for 2019/8.

I continue iterating until I calculate 2022/8, and that is my result.

I have two issues:

It's a tedious process to recalculate everything on each iteration, if the historic data is large then this would take too long.
The 2022/8 result is obtained with just one path, I would need to rerun the overall process N times and take the average to have a more precise number.

I need this to work with machine learning (not necessarily Random Forests), any ideas how to improve/change this process?

I think you should consider time series modeling instead of observation based classification models. In the letter, you are propagating error in each prediction year.

I would use ARIMA, LSTMs, maybe semi-supervised models and motif discovery techniques.


How does sigmoid saturate with large weights? In cs231n course , it is mentioned that

  If the initial weights are too large then most neurons would become
  saturated and the network will barely learn.

How do the neurons get saturated? Large weights may lead to a z (output of saturation) which is not cery close 0 or 1 and so doesn't let z*(1-z) to saturate

The sigmoid function $$ \theta(z) = \frac{1}{1+e^{-z}}$$ looks like this :

where $$z=w_i a_i + bias$$
for activations $a_i$ from the previous layer, and weights $w_i$ of the current neuron.

When the weights $w_i$ are too large (positive or negative), $z$ tends to be large as well, driving the output of the sigmoid to the far left (value 0) or far right (value 1). These are saturation regions where the gradient/derivative is too small, slowing down learning.

Learning slows down when the gradient is small, because the weight upgrade of the network at each iteration is directly proportional to the gradient magnitude.


Minimize correlation between input and output of black box system I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve.

Let's say that I have a black box with multiple inputs and a single output. I know one of the inputs and the output and I have multiple example recordings of both. This known input modifies the output in a way that it is not desired, therefore, I would like to get rid of this "noise" caused by the known input. The transfer function for this input can be safely assumed as linear.

What I am doing right now, it is to loop through the example recordings, creating a linear regression model to predict the unwanted outcome and subtracting it from the real measured output signal, for each example. Afterwards, I compute the average of all the fixed output signals to reveal meaningful data beyond noise.

This strategy seems to work according to the following plot:

X axis is the known input signal, Y axis is the output signal, blue and green dots represent the averaged data before and after applying the linear regression algorithm, respectively. Lines are the best fit for each data set.

You can see that the green line ("cleaned" dataset) has the smallest slope, meaning that the output variable is considerably less linearly correlated with the input than it was previously. Therefore, I assume that the regression technique explained before is working as expected.

My question, looking at the plot, is there any mathematical procedure to directly "project" the original dataset in a way that the correlation between the input and output variables is minimized? Is there any math trick to avoid the use of the regression technique on all the example datasets to obtain a similar result?

My written expression is not the best so please feel free to comment the question if you need further explanations.

Any code is welcomed but python (pandas, numpy, etc.) and Matlab are preferred. Theoretical explanations are also very welcomed.

Please allow me to paraphrase your question to make sure I get your question right.
Suppose you have 1 million data entries. Each entry consists of three inputs $X_1$, $X_2$, $X_3$ and one output $Y$. One of the three inputs, $X_1$, is a noise. You want to find the relation between $X_1$ and $Y$. So you can remove the impact of $X_1$ on $Y$.
Let's use a simplified example, $Y = 2X_1 + 3X_2 + 4X_3$
If you update each of the 1 million entries with:
$Y_{new} = Y - 2X_1$
Now you can examine the updated 1 million entries to find the relation between input $X_2$, $X_3$ and $Y_{new}$:
$Y_{new} = 3X_2 + 4X_3$
The relation between $X_1$ and $Y$ can be computed with different algorithms, such as regression, decision tree. How to measure which algorithm generates the best result, i.e. truly reflects the relation between $X_1$ and $Y$ hence removes the impact of $X_1$ on $Y$ as much as possible?
The problem you're solving is an interesting one. I would use a free tool such as Knime to find out which algorithm generates the best result in modeling the relation between $X_1$ and $Y$. It's faster than coding in Python or Matlab.
Details: to put the problem intuitively, suppose the price of a house is determined by location, size of the house, condition of the house. Assume these three factors are independent from each other. You want to remove the impact of "condition of the house" on "price" as much as possible. In essence, you want to find the relation between "condition of the house" and "price" which may not be linear.
You can let Knime read the 1 million entries of "condition of the house" and "price", try Decision Tree, SVM, or even Ensemble Learning, and see which algorithm generates the best model of $X_1$ and $Y$. In other words, try and error :)


Accuracy keep changing by changing randomState of classifier I try to classify car sound samples. Using MLPClassifier from Scikit. I'm getting very different and confusing test results between 2 different test sets, and I am stuck:

Training is done with the first data set of 1500 samples, splitted as 70/30 train/test. Second set is isolated, I use only for the final testing, 700 samples completely unseen data.
First test set is stable and always around %90 test, %99 train set accuracy.
Second test is completely random, it changes while I change the randomState variable of the classifier. It can be %20 or %80 accuracy.

This is extremely frustrating. The difference between two sets is first one is mainly from sounds that come with UrbanSound data collection. And second set is more real world, I recorded with an iPhone. I checked that they all have the same duration, sample rate, and bit rate.

So my question is:

if your accuracy changes randomly by changing your randomState of your classifier, on a certain test set, what does this tell you about your data?
And second what would be my approach now? Totally lost.

ps: My features are first 20 of the mfcc coefficients. OR 60 band of mel spectogram. I try different things.

I am also wondering if all these code examples, academic papers about sound classification which uses the sound samples from UrbanSounds and ESD50 sets, did they ever test their accuracy with completely random real world sounds, recorded and processed with different tools ?!

Below is when I plot these 2 different sound sets(only the positive class) with:

  plt.plot(car_features_1,'.')
  plt.plot(car_features_2,'.')

They look quite different to me, they are both car sounds one is recorded by iphone other is coming from Urbansounds .etc

This is the code for 60 features(60 from mel spectogram ):

clf = MLPClassifier(activation='relu', solver='adam', alpha= 1, hidden_layer_sizes=(60, 60, 60), random_state=None, max_iter=2000)

clf.fit(X_train, y_train)

And this is how i extract features

 mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0) #128 array
 mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=20).T,axis=0) #20 array

Pal, you have set alpha to 1. Alpha is a L2 regularization term, its value is normally around 0.0005 ... 0.0001. By setting it to 1 you force the optimizer to make your model's weights almost zero.

P.S.
After setting l2 reg term to its default value, you might get better accuracy, but if not, pay attention to the following. The ratio between the amount of data size (1500 samples, how many feature vectors?) and the number of weights (20x60x60x60xN, so more than 4M weights) is too small. It might well be that the model simply memorize the training set and does not generalize for sound classification.

Here are some suggestions that might help in training a sound classifier:

Start with a simpler model. Sounds are known to be well classified with Gaussian mixture models, for instance. Unlike deep learning models, those are easier to train.
Use data augmentation to increase the amount and diversity of your training data. You can mix the sounds with some light background noises.
Try to reduce the amount of weights. Three layers with 60 neurons each is too much. Usually, layers become smaller with going up, something like 64->32->16. Also, convolutional layers can be very useful here, as they share the weights across their neurons. I used them quite successfully for sound classification.
scikit-learn is quite unusual choice for training deep learning models. I would try Keras, it works perfectly with numpy arrays.


Getting stock data in a discipline manner from Yahoo finance I used the below code for downloading stock data from yahoo finance:-

import yfinance as yf
import datetime

    stocks = ["AXISBANK.NS", "HDFCBANK.NS", "ICICIBANK.NS" ,"INDUSINDBK.NS",
    "KOTAKBANK.NS",
    "SBIN.NS",
    "YESBANK.NS"]
    start = datetime.datetime(2018,1,1)
    end = datetime.datetime(2019,7,17)
    data = yf.download(stocks, start=start, end=end)
    data

I get the data in a below manner:-

I saved the data using panda:-

import pandas as pd

df = pd.DataFrame(data)

# saving the dataframe
df.to_csv('BANKING STOCK.csv')

I got the data in this format:-

But I ant my data in this format:-

Because this format is more convenient for analyzing data through SQL. How can I change the format of the data?

Use:

data.stack().reset_index().rename(index=str, columns={"level_1": "Symbol"}).sort_values(['Symbol','Date'])

Output:

           Date       Symbol  Adj Close  ...     Low    Open     Volume
0    2018-01-01  AXISBANK.NS     564.80  ...  560.50  563.80    6943234
7    2018-01-02  AXISBANK.NS     558.81  ...  556.35  567.00    6292268
14   2018-01-03  AXISBANK.NS     559.75  ...  555.75  561.30    3990149
21   2018-01-04  AXISBANK.NS     559.21  ...  558.00  564.85    2766664
28   2018-01-05  AXISBANK.NS     562.55  ...  556.70  561.05    3249056
35   2018-01-08  AXISBANK.NS     564.75  ...  562.45  565.80    3770339
42   2018-01-09  AXISBANK.NS     564.60  ...  563.00  564.00    4607504
49   2018-01-10  AXISBANK.NS     562.10  ...  560.45  566.15    2411625

The basic operation is stack that stacks the column headers into a multiindex. This multiindex is then converted to columns (reset_index()) which then get proper names. Finally the data are sorted by Symbol and Date as required.

Remark: You don't need df = pd.DataFrame(data), yf.download already returns a DataFrame.


How many features do you generally use for your ML Model? I am working on a certain kaggle competition and users there say that they are using >5000 features and training a XGBoost or Random Forest on it.

The mentioned post is here: https://www.kaggle.com/c/walmart-recruiting-trip-type-classification/forums/t/17258/feature-counts

I tried doing so myself after exploding the feature space and creating interaction features, but it takes forever to run on my 16GB macbook.
For example: (96000,1000) data took around 5 hrs to finish to train an RF with n_estimators=1000,max_depth=40, cv=3.

Tuning such a model would take weeks and it only uses 1000 predictors....

Now here are some of my doubts:
1. Is my python scikit environment properly loaded. I use anaconda distribution. Are these running times normal?

The number of features should take into account the number of examples you have in the training data.

If your model training takes too long, have you considered principal components analysis or feature selection?  Both techniques can cut down the number of features into a more manageable size without throwing away information which describes the target variable.


Which models can handle null values? Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle null values as input. The only one I've found currently is XGBoost, which is a gradient descent algorithm. I believe tree based algorithms should in theory handle null values as well, but I'm not sure if that generalizes to all tree based algorithms or if some work better than others.

Some background, I have created multiple datasets, including ones where the NaNs ARE replaced. Those can be used to train models that can't handle null values, like neural networks.

Due to the sparseness and nature of the data, imputing with mean or median is going to introduce a LOT of bias and doesn't make sense. Just want to get that out there since that's the first thing everyone suggests. Also, the missing data actually represents a situation where a patient didn't get a specific lab test done, which is useful information of its own.

So back to the original question, besides XGBoost, are there other models that handle null values as inputs? Does that generalize to ALL gradient descent algorithms or tree algorithms or just specific ones? Is there a preferred model for a situation where you want to keep null values?

LightGBM and XGBoost Libraries can handle missing values

LightGBM: will ignore missing values during a split, then allocate
them to whichever side reduces the loss the most
XGBoost:   the  instance  is classified into a default    direction (the optimal default directions  are learnt from the data    somehow)

Finally, it is NOT a general property of gradient descent algorithms or tree algorithms. Only specific implementations of these algorithms have this property.

Personally, I believe that LightGBM and XGBoost can handle missing values effectively, in case one wants to keep them


Which ML algorithm should I use for following use case for classification and Why? I have data in table format having total 3 columns. One column for label, other two columns are features. So, such 30 rows(1 row contains 2 feature and 1 label) make one set of data with all 30 rows having same. In these 30 rows there could be some patterns of features. There could be more such sets of 30 rows(for 30 rows label will be same). So there will be 6 to 7 labels or class. I want algorithm to predict one label from 30 rows(one set). Which algorithm should I choose and how should I pre-process data?

You can check https://autogluon.mxnet.io/
from autogluon import TabularPrediction as task
predictor = task.fit(train_data=task.Dataset(file_path=TRAIN_DATA.csv), label=COLUMN_NAME)
predictions = predictor.predict(task.Dataset(file_path=TEST_DATA.csv))


Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn Reference
I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning?
Also, what is learning decay and learning offset, in this context and in general?

With batch you feed the entire data through each EM iteration. In the online implementation you feed only some of the data through each EM iteration (a "mini-batch"). From the sklearn user guide:

  While the batch method updates variational variables after each full pass through the data, the online method updates variational variables from mini-batch data points.

From the paper about online LDA:

  We then update $\lambda$ using a weighted average of its previous value and $\tilde\lambda$.
  The weight given to $\tilde\lambda$ is given by $\rho_{t} = (\tau_{0} + \tau)^{-\kappa}$, where $\kappa$ ∈ (0.5, 1] controls the rate at which
  old values of $\tilde\lambda$ are forgotten and $τ_{0} \ge 0$ slows down the early iterations of the algorithm.

In the sklearn implementation:

weight = np.power(self.learning_offset + self.n_batch_iter_, -self.learning_decay)

So learning_offset is $\tau_{0}$ which slows down early iterations, and learning_decay is $\kappa$ which controls rate at which old weights are forgotten.


How to cluster some text using TensorFlow Apologies for any inaccuracies due to the infancy in this field.

I'm trying to learn on how to return a dataset with three classes to be clustered by TensorFlow.

At this stage, I've read a lot and experimented with several Colabs, but none of them points to the clustering technique when it comes to text (just classification).

The whole point is I don't have any clue of how the text could be classified, and I though the recursive process used by TensorFlow would have offered a better approach and results of what I normally get by using K-Means.

I drafted a Colab myself, but it's a real mess of examples now, so I better not to share at this stage.

In a nutshell, I tokenised the keywords, created a dictionary, converted my text into a dense matrix and then started using the models.
However, when it comes to using the fit function, I end up getting a series of error with the most recent one being "ValueError: No gradients provided for any variable".

I would appreciate some support and hints also on materials to read if any.
The release of TensorFlow 2 makes things even more complicated as there are even less examples and trying to adapt the ones for TF1 not always work.

Thanks

UPDATE:

I've cleaned up the project, inferring from another example taken from here.
Clearly the original example was working, but not mine.

In my version I'm trying to train the model to learn three different classes.
My questions at this stage are:

How can I create a suitable sets of labels when the Dense output is greater than 1?

Hot encoding is not an option apparently as the label shape is not inferred from the size of the array as I initially thought, but from the values itself.
How can I predict a new unseen keyword/text?

Here, do I need to train all the time the model to "see" the new keywords. I would expect so, but how?
How can I print/export the output of my prediction?

I do not see many benefits of using TensorFlow here. TensorFlow (and PyTorch) are great tools if you need to compute gradients of a loss function and use it in an optimizer. Clustering does not give the kind of training that would allow you to train an RNN or a Transformer that would give you a reasonable representation.

In your case, I would try:

represent the by average word embedding of content words
represent the text using BERT

And then do use a clustering algorithm of your choice. You can indeed implement k-means or hierarchical clustering in TensorFlow.


Scale out to between 0 and 1 before computing loss fucnction in keras Is there a way to scale the output of a CNN before computing the loss function in keras. It would have to work for batches as well. Specifically I am training images and the output is a probability map of each pixels for the entire batch. Then I compute the loss function with the ground truth data. The problem is that the predicted output during training at some intermediate stage lies between 0.02 to 1e-4 and the ground truth pixels are between 0 and 1 so the loss function doesnt penalize pixels which dont look similar to the ground truth. I get this by visualizing the squared error between the output at some intermediate stage and the ground truth data.

Use Lambda layer
def scale(tensor):
    tensor = (tensor - K.min(tensor)+1e-20)/(K.max(tensor)-K.min(tensor)+1e-10)
    return tensor

scaled_density_pred = Lambda(scale, name="scale_layer")(density_pred)