Low accuracy on MNIST Dataset Hi downloaded the MNIST dataset images and labels and I am trying to train but I am getting low accuracy which is very low . not Even 50 %.

below is my code.

import glob
import imageio
import cv2
import pandas as pd
import numpy as np
images=[]
for image_path in glob.glob(r"C:\\Users\\Downloads\\tmnist_bundle_rgb\\tmnist_bundle_rgb\\imgs\\*.png"):
    im = imageio.imread(image_path)
    a= cv2.imread(image_path)
    gray_img = cv2.cvtColor(a, cv2.COLOR_BGR2GRAY)
    images.append(gray_img)

images = np.asarray(images,dtype=np.float32)

lables = pd.read_csv('C:\\Users\\Downloads\\tmnist_bundle_rgb\\tmnist_bundle_rgb\\index.csv',encoding='utf-8')

lables=lables.iloc[:, 0] # slicing the column

lables = np.asarray(lables) #Array

lables = np.asarray(lables,dtype=np.float32)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(images, lables, test_size=0.1) # split

X_train = X_train.reshape(54252,28,28,1)
X_test = X_test.reshape(6028,28,28,1)

X_train = X_train/ 255
X_test = X_test/ 255

from keras.utils import to_categorical

number_of_classes=10
y_train = to_categorical(y_train,number_of_classes)
y_test = to_categorical(y_test,number_of_classes)

#Model Buidling
from keras.models import Sequential
from keras.layers import Dense, Convolution2D, Flatten
from keras.layers.convolutional import MaxPooling2D
from keras.layers import Dropout

model = Sequential()
model = Sequential()
model.add(Convolution2D(32, (3, 3), activation='relu', input_shape=(28,28,1)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(X_train, y_train,batch_size=32, nb_epoch=5, verbose=1

Output:

  Epoch 1/5 54252/54252 [==============================] - 36s
  663us/step - loss: 2.3024 - acc: 0.1093

  Epoch 2/5 54252/54252 [==============================] - 36s
  664us/step - loss: 2.3017 - acc: 0.1111

  Epoch 3/5 54252/54252 [==============================] - 39s
  721us/step - loss: 2.3016 - acc: 0.1115

  Epoch 4/5 54252/54252 [==============================] - 40s
  733us/step - loss: 2.3018 - acc: 0.1110

  Epoch 5/5 54252/54252 [==============================] - 50s
  912us/step - loss: 2.3015 - acc: 0.1115

Your model have an accuracy of 0.10 so he is correct 10% of the time, a random model would do the same. It means your model doesn't learn at all. Even a bad model learn a little. So the problem come from your dataset.

I tested your model and got 97% accuracy.

Your problem probably come from how you import your dataset.
Here is how i imported:

import idx2numpy
import numpy as np
fileImg = 'data/train-images.idx3-ubyte'
fileLabel= 'data/train-labels.idx1-ubyte'
arrImg = idx2numpy.convert_from_file(fileImg)
arrLabels = idx2numpy.convert_from_file(fileLabel)

the dataset come from: http://yann.lecun.com/exdb/mnist/

You should test if you imported an processed your dataset correctly. For example :

import matplotlib.pyplot as plt
image = np.asarray(arrImg[1000])
plt.imshow(image)
plt.show()
print(arrLabels[1000])


Sliding window Algorithm and its convolutional implementation I want to know why the convolution implementation of the sliding windows is equivalent to the sequential step-by-step sliding window? Why are they the same thing?

I'm following Andrew NG for this:
What I understand is the first row in the picture is supposed to be a Convolution Network which is trained on 4 classes.

The second row is supposed to be 3 channel 16x16 image that is convolved by the trained convolution network from the row one (which is 16 5x5 filters -> 2x2 Max Pool -> 400 5x5 filters -> 400 1x1 filter -> 4 1x1 filter), the tricky part as I understood is that the output is the equivalent to sliding 14x14 window to on 16x16 image which has only 4 positions, they are 2x2x4 outputs, each cell in the 2x2 grid corresponds to a position of the 14x14 window on the 16x16 image.

The Last row is another example but with bigger 28x28 image with the same window size 14x14.

Consider the second row ie. the 16x16x3 example, if you do the sliding window approach you first take the 14x14x3 area marked in red and pass it as input to your trained CNN. What happens then, 5x5x3 filters then convolve the first 5x5x3 of the input ie. starting from top left and then it convolves on the next 5x5x3 section of the same input according to the stride, and so on, just as Convolution happens.

Now, consider the 14x14x3 section marked in green, (this will be the second window in Sliding Window Approach) and it like the first red area, goes into your trained CNN and hence is convolved by the same filters that convolved the red one, so far so good.

What will be the first part of this green input to be convolved? It would be the same as the second 5x5x3 part to be convolved on the first 14x14x3 green area, so what happens is that the same 5x5x3 area has undergone the same operation ie. same convolution in this case twice, that means you've basically done the same computation again on the same 5x5x3 input, once on the red input and then on the green input.

What's the use of doing the same calculation again, that's the reason we use the Convolutional Sliding Window Approach which will do every computation only once and since you aren't doing the same stuff again and again you will save that time and hence the Convolutional Sliding Window Approach will run faster and obviously does the same computation as the normal Sliding Window Approach.

Note: Modern implementations of CNN are very deep and involve lots of filters, so you are doing a lot of redundant work which slows down your implementation.

Hope it helps.


Can numerical discrete finite data be always treated also as categorical? In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or infinite.

I want to establish if a numerical, discrete and finite data can also be treated as categorical data.

I know that it depends on 'the meaning' of the data and requires some common sense analysis but I want to establish if the following statement is always true:

  "Numerical, discrete and finite data can also be a categorical data"

In the classification of data the numerical data is said to have 'mathematical meaning as a measure of something'. But 'technically', without assessing the meaning of the data, it does also make them capable of being a categorical data (ordinal or not), if we strip it from mathematical meaning.

Example can be a following array of items:

Energy

15
15
20
25
25

Every observation has 'Energy' characteristic it can be treated as mathematical discrete and finite numerical value which can be a measurement of energy an item has. But also it can be treated as a category: two items are in 15 category, one in category 20 and two in category 25.

Thanks for confirming this.

I would separate value with representation in this case.

Energy as you mentioned, in the real world, holds a very continuous value.  However, we may choose (for various reasons) to represent this value in different forms.

We can take values as they are (15.21252, 23.76535), we can round them into integers (15, 24), we can even decide to represent this data by clusters ("UNDER 20", "OVER 20").

Technically, all data can be represented as categorical data, we need to consider what value does this data represent, and what are we losing/gaining from using different representations.


Using LSTM to clear up corrupted text files LSTM can be used to generate text, can they be used to fix corrupted text files?

Say that my original was:

Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do

Now it has become:

Alice0004 XGFT was beginning to get 900io ALPHA very tired 00New York 333 of siing XXXYYY by her siter on THIS IS 898 the
bank, and of &*& HHH GTY DOG AND CAT having nothing to do.

Basically, a lot of crap was added (some meaningless words but also meaningful words, and some letters may have been erased).

Is there any way LSTM would be able to fix this? (Maybe by cleaning some beforehand and then let LSTM do the rest)

You should use a Seq2Seq (which uses LSTM/GRU/RNNs) architecture for the task of cleaning text. The encoder network would take in the "noisy" sequence and the decoder would generate the "cleaned" sequence.

Seq2Seq models is often used for Neural Machine Translation (NMT).

The next thing is you will need a very large data set. Fortunately this shouldn't be too hard to generate noise injections (of real words, symbols, fake words, letter omissions etc) into sequences of text.

A good guide for a similar task is in the following post. The implementation uses Keras and has all of the code. https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8


MinMaxScaler when LSTM predictions fall outside of training range? I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set may have values between 1-200, but in the most recent time steps, the values are toward the high end of that range.

How can I handle future time-steps that should be predicted to land above 200 (since activation function outputs values between 0-1)? I guess my question is, is there a way to manually (or through some “rule of thumb”) specify a maximum value for the training set? So in this case, would it be possible to use MinMaxScaler to use 0-500 as the min-max values to transform to 0-1, that way values in my test set or predictions can be inverse_transformed accordingly? Is this an out-of-the-box scikit-learn feature? What’s the most efficient way to implement this, especially if this has to be done independently for multiple features?

Inverse-transforming with MinMaxScaler should be capable of producing something outside of the training data's range.  It seems that, in your use case, using a final activation that lands in $[0,1]$ might not be appropriate.  Even if you transform the training data to land in, say, $[0,0.7]$, applying a sigmoid or some-such on the final layer seems to lack motivation.

As to the question of whether it can be done: yes, just with a little roundabout thinking.  You can't specify what you want the scaler to think your data's max and min are, but you can specify the output range you want (parameter feature_range), which amounts to the same thing.


Can someone explain to me how to use a predictive model to predict something other than the training set So let's say I create a logistic model to predict who will open a loan based on a based email list that includes who opened and who didn't that's 90% accurate. The model says age, income, bank engagement are three key variables that decide who opens a loan. Is there a way to apply this model to a different email list with the same variables to predict who will open a loan? Or what percentage of the people will open the loan? Or will I just need to analyze the data on the list myself to determine this.
Sorry, probably a dumb question, but it's one thing I've struggled to figure out on my data scientist journey.

Yes, you can predict who will open a loan with this model. You have to keep several things in mind. The model expects the new data to be from the same distribution as the training data, meaning the variables have the same distribution and targets as well.
You can not train on one set where 3% opened a loan and then predict on a set where you expect that 90% open a loan. This data is then too different for the model to accurately predict on.
Remember that your model only has 90% accuracy, so it is not guaranteed that a specific costumer will open a loan when the model predicts so.


Tableau: Clustering based on value-range for map coloring Is there a possibility to cluster coloring for certain statistical ranges?

This is what I have been able to achieve so far.

You can, as an example, create a binned field for the measure. The value range can be specified in the tooltip. I used a single color based continuous palette since (population) total is a continuous field.

[Edited]
Based on your comment, I tested with a fixed dimension based binned field for the continuous measure. At least in my example file, the values make more sense now with the new binned field. I see China at 16M population, India at 14M and so on.

If this solution is relevant to your use case, you will need to be careful with dimensions included/excluded in the fixed calculation.


Geting batch size in keras custom loss I am implementing a custom loss in keras, for example, a sum:

def custom_loss(y_true, y_pred):
    K.sum(y_true, y_pred)

Now, I want to normalize it by the batch size. Is it possible, to retrieve the batch size from y_true or y_pred?

I would say this highly depends on how you have your code set up.  What type of model are you creating?  Are you not setting batch-size as a global parameter?  Can you not just use that?  Also, assuming y_true is a numpy matrix, can you just use:
y_true.shape[dimension_representing_batch_size] as a means of getting the size?


Deriving Confidences from Distribution of Class Probabilities for a Prediction I run into this problem from time to time and have always felt like there should be an obvious answer.

I have probabilities for potential classes (from some classifier).  I will offer the prediction of the class with the highest probability, however, I would also like to attach a confidence for that prediction.

Example: If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1: 50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1 should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}.

Reporting that I predict class C1 with 60% probability isn't the whole story.  I should be able to derive a confidence from the distribution of probabilities as well.  I am certain there is a known method for solving this but I do not know what it is.

EDIT: Taking this to the extreme for clarification:
If I had a class C1 with 100% probability (and assuming the classifier had an accurate representation of each class) then I would be extremely confident that C1 was the correct classification.  On the other hand if all 5 classes had almost equal probability (Say they are all roughly 20%) than I would be very uncertain claiming that any one was the correct classification. These two extreme cases are more obvious, the challenge is derive a confidence for intermediate examples like the one above.

Any suggestions or references would be of great help.

Thanks in advance.

If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1:
  50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1
  should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2,
  C4: 1, C5: 2}.

Assuming that those probabilities are accurate, this isn't true. In your second case you can be a lot more confident that the ground truth is one of C1 or C2, but in terms of absolute confidence about C1 the probability is the same across both examples. To illustrate this with a more clear example, if you had a 100 sided die that had 50 sides labeled with "C1" then the labels on the the other 50 sides are irrelevant to the likelihood that you would roll a "C1".

Now with that said, your probabilities from your model are most certainly not perfect so there may be a way to use the intra-class correlations to improve them. Can you provide some more details about your specific problem and modeling workflow that you have used to get your probabilities?


Micro-F1 and Macro-F1 are equal in binary classification and I don't know why I have a binary classification problem which in the test set, the number of data in both classes are equal (the test number of class 0 and class 1 are equal). Since we know  that the number of samples from every class are equal, I use median on the probabilities of logistic regression output (probabilities of samples for class 1) and map probabilities to zero and one and then calculate f1-micro and f1-macro. But they are absolutely equal and I don't know is it odd or not and why is this happening. I would be grateful if you have any idea what's going on and what is wrong.

The difference between macro and micro averaging for performance metrics (such as the F1-score) is that macro weighs each class equally whereas micro weights each sample equally. If the distribution of classes is symmetrical (i.e. you have an equal number of samples for each class), then macro and micro will result in the same score.

As an example for your binary classification problem, say we get a F1-score of 0.7 for class 1 and 0.5 for class 2. Using macro averaging, we'd simply average those two scores to get an overall score for your classifier of 0.6, this would be the same no matter how the samples are distributed between the two classes.

If you were using micro averaging, then it would matter what the distribution was. Say that class 1 made up 80% of your data, the formula would then be 0.7*80% + 0.5*(100%-80%) which would equal 0.66, since each sample is weighed equally and as a result the score is representative of the data imbalance. If class 1 made up 50% of your data, the formula would shift to 0.7*50% + 0.5*(100%-50%) which would be 0.6, the same as the result from macro averaging.

If your data was perfectly balanced, then macro and micro averaging will both result in the same score. If not, there's still a chance that they result in the same score depending on the exact distribution of scores (or if your estimator has the same performance for all classes involved).


Mathematical algorithms to generate labels for training data What are the possible approaches when we need to train a model, but the training dataset is really small? (Assuming we have a lot of data, just not many data are labeled)

I know a library from Stanford: https://hazyresearch.github.io/snorkel/ that can generate training labels based on a some pre-determined experts rules. (Side question, anyone happen to know what's the underlining mathematics/statistics of this library?)

However, I am wondering in the scenarios when the snorkel packages can not be used, what are the approaches to label more data for training? Could Maximum Likelihood Estimator be used here? How would one implement such algorithm for labelling the data for training?

BTW, I am looking for a mathematical approach, not a brute-force one like using Amazon Mechanical Turk.

Thank you!

Assuming we have a lot of data, just not many data are labeled.

Try to tackle the problem first by using an unsupervised approach, following that, use the learned features for any downstream task.
see Deep Generative Models.

  generate training labels based on some pre-determined experts rules

If you have some function that can generate the data distribution then why you bother yourself with such finding yet another function that approximate the former.
you could use this approach for evaluation purposes, e.g. compare different models regarding the problem being solved.

  What are the approaches to label more data for training

you could aggregate (average) different models that tackle the same problem but with a different dataset.

  Could Maximum Likelihood Estimator be used here.

As I said, we could maximize the likelihood of the data, giving high probability to samples that are very similar to the training data is very useful at exploiting the internal structure (learning the data manifold) that could potentially disentangle the factors that are responsible for generating the data which presumably part of them could be a direct signal for predicting the target y that you are trying to predict.


Pull Random Numbers from my Data (Python) Let's imagine I have a series of numbers that represents cash flows into some account over the past 30 days in some time window. This data is non-normal but it does represent some distribution. I would like to pull "new" numbers from this distribution in an effort to create a monte-carlo simulation based on the numerical data I have. How can I accomplish this? I've seen methods where you assume the data is normal & pull numbers based on some mean and standard deviation - but what about non-normal distributions? I'm using python so any reference including python or some python libraries would be appreciated.

If what you want is to generate random numbers with the same distribution as your cashflow numbers I recommend you using Python's Fitter package
It is powerful and very simple to use.
You can in this way use it to find the distribution of your data and then generate random numbers with the same distribution.
From documentation:
from scipy import stats
data = stats.gamma.rvs(2, loc=1.5, scale=2, size=10000)

from fitter import Fitter
f = Fitter(data)
f.fit()
# may take some time since by default, all distributions are tried
# but you call manually provide a smaller set of distributions
f.summary()

Also useful resources might be found in stackoverflow


What more does TensorFlow offer to keras? I'm aware that keras serves as a high-level interface to TensorFlow.

But it seems to me that keras can do many functionalities on its own (data input, model creation, training, evaluation).

Furthermore, some of TensorFlow's functionality can be ported directly to keras (e.g. it is possible to use a tf metric or loss function in keras).

My question is, what does TensorFlow offer that can't be reproduced in keras?

Deep Learning frameworks operate at 2 levels of abstraction:

Lower Level: This is where frameworks like Tensorflow, MXNet, Theano, and PyTorch sit. This is the level where mathematical operations like Generalized Matrix-Matrix multiplication and Neural Network primitives like Convolutional operations are implemented.
Higher Level: This is where frameworks like Keras sit. At this Level, the lower level primitives are used to implement Neural Network abstraction like Layers and models. Generally, at this level other helpful APIs like model saving and model training are also implemented.

You cannot compare Keras and TensorFlow because they sit on different levels of abstraction. I also want to take this opportunity to share my experience of using Keras:

I do not agree that Keras is only useful for basic Deep Learning work. Keras is a beautifully written API. The functional nature of the API helps you completely and gets out of your way for more exotic applications. Keras does not block access to lower level frameworks.
Keras results in much more readable and succinct code.
Keras model Serialization/Deserialization APIs, callbacks, and data streaming using Python generators is very mature.
Keras has been declared the official high level abstraction for TensorFlow.


How to calculate the elapsed time of a flag status per day? I'd like to figure out the elapsed time between flag status changes.

Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed.

I'm using common storage and a script to loop the records, which is very expensive. I'm looking for a general best practice. I'm open to use any technology in combination or as a replacement. I'm new to data science and probably miss the right terms. Related advices are very appreciated.

Index: user_mood_log

 |    changed_at    |  mood  | user |
 ------------------------------
 | yyyy-mm-dd 12:00 | sad    | John |
 | yyyy-mm-dd 15:00 | happy  | John |
 | yyyy-mm-dd 18:00 | happy  | John |
 |        ...       |   ...  |  ... |

Index: user_mood_log_daily

 |    date    |  sad  | happy | user |
 -------------------------------------
 | 2019-05-22 |   3   |   21  | John |
 | 2019-05-21 |   5   |   19  | John |
 |    ...     |  ...  |   ... |  ... |

Challenges:

The last record of yesterday (or even older) needs to be considered. Means I can't simply group by day.
A mood might not change for several days.
Duplicated mood logs are possible.

Any advice? Thanks in advance!

Python's pandas package has the capability to answer all those questions, assuming the data can fit into the computer's memory.

Pandas can process timeseries data, find time deltas, and remove duplicates.


Stochastic gradient descent in matrix factorization, sensitive to label's scale? I'm trying to figure out a strange phenomenon, when I use matrix factorization (the Netflix Prize solution) for a rating matrix:

$R = P^T * Q + B_u + B_i$

with ratings ranging from 1 to 10.

Then I evaluate the model by each label's absolute mean average error in test set, the first column is origin_score, the second(we don't transform the data, then train and its  prediction error), the third(we transform the data all by dividing 2, train, and when I use this model to make prediction, firstly reconstruct the matrix and then just multiply 2 and make it back to the same scale)

As you see, in grade 3-4 (most samples are label from 3-4), it's more precise while in high score range(like 9 and 10, just 2% of the whole traiing set), it's worse.

+----------------------+--------------------+--------------------+
| rounded_origin_score | abs_mean_avg_error | abs_mean_avg_error  |
+----------------------+--------------------+---------------------+
| 1.0                  | 2.185225396100167  |  2.559125413626183  |
| 2.0                  | 1.4072212825108161 |  1.5290497332538155 |
| 3.0                  | 0.7606073396581479 |  0.6285151230269825 |
| 4.0                  | 0.7823491986435621 |  0.6419077576969795 |
| 5.0                  | 1.2734369551159568 |  1.256590210555053  |
| 6.0                  | 1.9546560495715863 |  2.0461809588933835 |
| 7.0                  | 2.707229888048017  |  2.8866856489147494 |
| 8.0                  | 3.5084244741417137 |  3.7212155956153796 |
| 9.0                  | 4.357185793060213  |  4.590550124054919  |
| 10.0                 | 5.180752400467891  |  5.468600926567884  |
+----------------------+--------------------+---------------------+

I've re-train the model several times, and got same result, so I think it's not effect by randomness.

The larger your target scores, the larger latent variables should be (well, it's not only a magnitude that matters, but also a variance, but it still applies to your case). There's no problem with larger coefficients of latent vectors unless you use regularization (and, likely, you do). In case of regularization your optimal solution will tend towards smaller values, and'd sometimes prefer to sacrifice some accuracy for lower regularization penalty.

Gradient Descent doesn't suffer from problem of large coefficients (unless you run into some sort of numerical issues): if the learning rate is tuned properly (there are lots of stuff on it, google), it should arrive to equivalent parameters. Otherwise nobody guarantees you convergence :-)

The common rule of thumb when doing regression (and your instance of matrix factorization is a kind of regression) is to standardize your data: make it having zero mean and unit variance.


Group similarity by high dimension vector comparison I have a dataset of 256 rows with 61 columns/variables. Each row should be considered a vector of dimension 61. If I randomly split it, by rows, in 2 groups, how could I prove that the 2 groups are similar? The origin of the data is biomedical and nonlinear approaches should be preferable.

You can't actually prove that the two groups are similar but you can establish a confidence level/threshold. Furthermore, it is possible that the two groups won't be similar (depending on your threshold for similarity) if, for example, only one of the two groups contains strong outliers.

That said, you can make comparisons based on assumptions regarding the underlying data distribution. For example, if the data can be assumed to be distributed as a multivariate normal distribution, you can use Hotelling's two-sample T-squared statistic (a multivariate generalization of the Student's t-test) to test your confidence interval.


Numpy ndarray holding string with unknown data type I ran the following code in a jupyter notebook cell

ndarrs=np.array(["1.2","1.5","1.6"], dtype=np.string_)

print(ndarrs.dtype)

It returned |S3 as shown below.

Can someone help me understand the meaning of this symbol?

In Python 3, you shouldn't really specify the np.string_ dtype, as it is left there for backwards compatibility with Python 2. The S type you see using np.dtype is a map to the bytes_ type, a zero-terminated string buffer, which shouldn't be used.
The S just means string and the number gives the number of bytes.
In [1]: s = ""        # start with an empty string

In [2]: for i in range(5):    # make the string larger
            s += str(i)
            a = np.array([s], dtype=np.string_)
            print(f"{a}\t{a.dtype}")

    [b'0']  |S1
    [b'01'] |S2
    [b'012']    |S3
    [b'0123']   |S4
    [b'01234']  |S5
    [b'012345'] |S6

For python 3 you should instead use np.unicode:
In [1]: a = np.array(["hello", "world"], dtype=np.unicode)

In [2]: type(a)
Out[2]: numpy.ndarray

In [3]: b.dtype
Out[3]: dtype('<U5')

< means little-endian
U means a unicode string
5 relates to the number of bytes used to hold the string. If you had really long strings, that number would then increase.

Have a look at this documentation, which states:

Note on string types:
For backward compatibility with Python 2 the S and a typestrings remain zero-terminated bytes and np.string_ continues to map to np.bytes_. To use actual strings in Python 3 use U or np.unicode_.


Using SVM Classifier with C=0 and C=infinity ,what would be the effect on classifying this data? I want to know the effect of applying different C values on SVM Classifier margin

C value is like lambda, the L2/L1 regularization hyper parameter, but in reverse manner. Whenever C is large, means there is high probability that your model overfit the data in hand. Whenever it is small, your model endure some errors in order to avoid producing small margin and consequently you will have a model less prone to overfitting. The first one is called hard SVM and the latter one is soft SVM. In your case, I guess there will be no difference because your model is linearly separable. If X4 was between X2 and X3 you would have different result for each hard and soft. Soft and hard differ in cases which the data is not linearly separable or maybe they are separable but the margin would be narrow or wide if SVM discards some samples. Soft tries to make the separator space, margin, as large as possible by discarding those minority data which hinder a large margin to be crated but hard tries to separate the data to reduce the error as much as possible. In your case, you will get same result.

For illustrating more, look at the left figure. It tries to discard the sample which hinders the large margin to be created but large margin, considers all the samples.


Details of the k-means++ algorithm that is used to seed k-means Regard to K-Means++ algorithm, which is an algorithm for choosing the initial values (or "seeds") for the k-means clustering algorithm.

K-Means++ algorithm in Wikipedia

The exact algorithm is as follows:

Choose one center uniformly at random from among the data points.
For each data point x, compute D(x), the distance between x and the nearest center that has already been chosen.
Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)2.
Repeat Steps 2 and 3 until k centers have been chosen.
Now that the initial centers have been chosen, proceed using standard k-means clustering.

I dont understand step 3

"Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)^2."

What is probability proportional  ?

If I do not misunderstand  . . .

The next centroid x I choose must the distance

D(x) = D(x)^2 / summation of all distances from all data points square

Is that right ?

I still wonder about implementation. I try this in java but it does not work , the chance is very low and it make the selection distort.

public static double euclidean(Data a, Data b) {

    double accumValue = 0;
    double res;

    for (int i = 0; i < 72; i++) {

        res = a.features[i] - b.features[i];

        res = Math.pow(res, 2);

        accumValue += res;

    }

    double finalRes = Math.sqrt(accumValue);

    return finalRes;

}

public static double accumeratedSqrDistanceCal(ArrayList<Data> dataList, ArrayList<Data> centroids) {

    double[] squareDistanceCollection = new double[dataList.size()];

    for (int i = 0; i < dataList.size(); i++) {

        double minDistance = minDistanceFromClosetCentroidsCal(dataList.get(i), centroids);

        squareDistanceCollection[i] = Math.pow(minDistance, 2);

    }

    double accumerateDistance = 0;

    for (int i = 0; i < dataList.size(); i++) {

        accumerateDistance += squareDistanceCollection[i];

    }

    return accumerateDistance;

}

public static double minDistanceFromClosetCentroidsCal(Data d, ArrayList<Data> centroids) {

    double minDistance = 100000;

    for (int i = 0; i < centroids.size(); i++) {

        double distance = euclidean(d, centroids.get(i));

        if (distance < minDistance) {

            minDistance = distance;
        }

    }

    return minDistance;

}
public static void main(String[] args) {

for (int i = 0; i < CENTROIDS_SIZE; i++) {

            for (int j = 0; j < dataList.size(); j++) {

                double accumerateDistance = accumeratedSqrDistanceCal(dataList, centroids);

                double rand = Math.random();

                double distance = minDistanceFromClosetCentroidsCal(dataList.get(j), centroids);

                double distanceSquare = Math.pow(distance, 2);

                double chance = distanceSquare / accumerateDistance;

                if (chance > rand) {

                    centroids.add(dataList.get(j));
                    break;

                }

            }
        }

}

You want to spread out the initial clusters, the article argues that this gives (on average) faster convergence and lower error. This means that points with a higher distance to the closest center are likely to be better candidates for initial clusters. So the probability distribution for a new initial clustering point is:

$\mathbb{P}(C = c) = \frac{D(c)^2}{\sum_{\forall x\in X}D(x)^2}$

Where $X$ is the collection of all candidate points and $C$ the chosen point distribution.

This means the higher the distance, the higher the chance it will be picked as initial point.


Algorithm for user profiling without distinct profiles I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide what software and hardware this new user would need.

I am rattling my brain thinking how I could get this to work -
I could use a supervised learning approach and train a model with a dataset of already employed users and the software and hardware they use, however, the variables in this dataset would be extensive as, through one-hot encoding, each user object would have to have variables that represent each possible software or hardware a user could own. Or can my dataset have a collection of user objects but not each object has the same software/hardware variables?

I'm very confused about how to go about this so any insight would help. I'm probably looking at going about this the wrong way.

It is not uncommon for machine learning models to employ thousands of features. I doubt your software / hardware repository contains significantly more than that so it is unlikely an issue.

Having said that, machine learning is useful if you need to generalize. You say that your input variables are going to be department, location and job_role. In such case you have three categorical and likely independent and always known features so I expect you can just look up someone with same department, role and location and create default hardware/software profile based on that. I do not see any value added by machine learning.


Perceptron weight vector update I read about the Rosenblatt Perceptron Learning Algorithm. Often there is an explicit note:

  It is important to note that all weights in the weight vector are being updated simultaneously

But why are all weights updated simultaneously? I tried another approach where I iterated over all weights and updated them in different iterations. It also worked on some simple test cases.

Could someone explain to me, why they are updated simultaneously and why should this approach be better?

The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the right direction, which could mess with the convergence of the procedure.

The idea is that in the weight space every input vector is a hyperplane. You need to find a weight vector that is on the correct side of all the hyperplanes of your data inputs. The correct weight therefore is in a convex cone. If you observe a misclassification, that means your weight vector is on the wrong side of the hyperplane and therefore outside the convex cone of possible solutions.

Now, by adding/subtracting the input vector to the weight vector you make sure that this data input vector is now correctly classified. You also make sure, that you decrease the distance of your weight vector by a sufficient margin (at least the length of the input vector) towards the cone of all possible solutions.

If you only add parts of the vector (i.e. not update all weights) at each iteration, you cannot be sure that you make sufficient progress or even move in the right direction of the solution space.


How to cluster histograms or density distributions? I have no idea where to start when it come to cluster distribution and finding out similar the similar one.
Is there a package in R that does the job?

There are quite a few ways to measure the difference between two distributions. Take a look at this overview article on wikipedia.

One very common way that is used often in Machine and Deep Learning is the Kullback-Leibler (KL) Divergence. It is most commonly used in minimising the cross-entropy between the distribution of your training data and the expected (generalised) distribution of the problem you are analysing. In general, a value close to 0 indicates that two distributions are expected to show similar behaviour, while a large value indicates the distributions behave very differently - so knowing the first distribution doesn't help you know anything about the second.

A more general class of measures of dissimilarity between distributions is the so called $f$-divergence, "an average, weighted by the function $f$ of the odds ratio given by [probability measures] $P$ and $Q$". The Wikipedia article contains a formal definition and a table with functions $f$ that give some popular dissimilarities/distances, including the KL-divergence.

  The following table lists many of the common divergences between probability distributions and the $f$ function to which they correspond (cf. Liese & Vajda (2006))

KL-Divergence is combined with the entropy itself, to define the cross-entropy. Take a look here, under the Information Theory View, for a bit more info.


Help with creating dimensions/features It is quite hard to name the title properly as I just started to learn ML, will try to explain here. I want to practice ML by creating Movie suggestion algorithm. I came up with the following list of dimensions/features:

Rating
Number of votes
Genre
Actors
Directors
Writers
Year
Combination of Actors/Directors/Writers working in the team
Combination of Year and Actor or Director or Writer
Actor or Director or Writer working in some genre

There is no problem for me for numeric fields, but in case of actors I have multiple values. How to create feature for the actors?

You could create a "flag" variable for particular actors. Each actor would have their own column in your data and the column would have a 1 if the actor was in that film and a 0 if the actor was not in that film.

Of course, this creates lots of columns and can be tedious to code, but binary indicator variables can be really useful if you want to build a predictive model - worth it in my opinion.

Example:

       actor1  actor2 actor3
movie1   1       0      0
movie2   0       1      1
movie3   0       1      0

Edit Re: Director/Actor

It is probably unrealistic to create a variable for every permutation of director/actor combinations. If you want to see how the variables interact you can code interactions into your model when you build it. If you were to do that in R it would look like this:

movie.predictor <- lm(suggestion ~ actor1 + actor2 + director1 + director2 +
                      actor1:director1 + actor2:director1 + actor1:director2 +
                      actor2:director2,
                      data = movies.data)

The ":" in R asks the lm(...) function to consider the interaction of those two variables in the regression problem, assuming you are doing regression. Don't worry the same ":" technique works across several model functions. Just read the documentation for your preferred language and package and look for how to code "variable interactions"

Of course, now you are typing out all of the permutations of variable interactions. You might want to also consider a clustering algorithm to group actors and directors together. I am not a movie expert but it seems like directors have favorite actors (and vice-a-versa) so I would expect there to be clusters of directors and actors with a little overlap here and there.


Categorise sentences based on their semantic similarity I have a set of unique sentences. For each sentence I calculate a semantic similarity score (between 0 to 1) with the remaining sentences as mentioned in the below example.

E.g.,

Dataset = {sen1, sen2, sen3, sen4,..., senN}

For sen1 I calculate pairwise semantic similarity scores as follows.

sen1 and sen2 = 0.3
sen1 and sen3 = 0.7
sen1 and sen4 = 0.9
...
...
...
sen1 and senN = 1.0

Likewise for all the sentences I calculate pairwise semantic similarity scores.

Since, I am getting a pairwise value, is it possible to cluster these sentences? Also what is the most appropriate clustering technique in my situation?

(I want to cluster sentences based on the similarity value I have and also I consider values above 0.5 as semantically similar sentences.)

There are several techniques that you could apply in order to cluster data if your input is a matrix of pairwise distances between elements. As usual, the best option depends on your specific data, so it is hard to answer to the question of what is the best one, but you could try any of the following ones:

The k-medoids algorithm is similar to the well-known k-means algorithm. After randomly choosing k of your sequences as initial cluster centers (initial medoids) and assigning each sequence to the closest medoid, you randomly reassign sequences to different clusters as long as the value of the cost function decreases.
Hierarchical clustering is another example of clustering algorithm whose input is a matrix of pairwise distances between sequences. In this case the output is a dendrogram.
Another option is to apply multidimensional scaling, a dimensionality reduction technique which input is a matrix of pairwise distances between sequences, to project your sequences into a 2D plane. Once you do that, you can apply any cluster algorithm you can think of, like for instance k-means.

As I said, there are many other options, but these ones are the simplest ones I can think of, and the ones I would start from.


Muti-Output Decision tree with classification and regression in output I'm working on a project and want to use decisions tree (because I have both catgorical and numerical values in my input and don't want to transform the categoricals variables) to predict an output varaible , but the problem is that I don't have only one output variable but 4 !

that picture is a look for my output dataset:

The first one will can be predict with a regression decision tree .

the the rest if can be predict with a classification decision tree but with more than 2 classes so a multi-class.
So I want to know what is the best solution to approach this problem:

Using 4 differents decisions trees one tree for each variables ?

Or using 1 multi-output decision tree?

What are the avantages and inconvenients of each approach?

I check and found this in scikit-learn  documentation and I'm sure that is the answer I was looking for!

  Multioutput classification support can be added to any classifier with MultiOutputClassifier. This strategy consists of fitting one classifier per target. This allows multiple target variable classifications. The purpose of this class is to extend estimators to be able to estimate a series of target functions (f1,f2,f3...,fn) that are trained on a single X predictor matrix to predict a series of reponses (y1,y2,y3...,yn).


neural network function approximation with constraints I would like to approximate a function $f(\cdot)$ by means of a neural network given a finite set of observations $f(x_i)$ where $x_i\in\mathbb{R}^n$ and $i=1\dots,N$. However, I have some prior knowledge on how this function should behave, for example that it is monotonic in the first coordinate.

Are there methodologies accounting for this type of shape constraints when training a (D)NN?

One option is to synthetically generate that data and provide it as training data to the model. Deep Neural Networks are quick at learning patterns. If the data is always monotonic in the first coordinate, the model will learn that.
Another option is using something like TensorFlow Lattice to add "common sense" constraints.


Is LSTM or pretrained BERTForMasked LM usable for predicting changed word in a sentence using a small dataset? (2000 samples) I have a small (2000 samples) dataset of newspaper headlines and their humorous conterparts where only one word is changed to sound silly, for example:
Original headline: Police <officer> arrested for abuse of authority
Humorous headline: Police <dog> arrested for abuse of authority
I want to train a model to predict changed sentence by the original. I am planning to implement two models for this task: one for binary tagging of input sequences (whether a word in a sentence needs to be changed) and one for predicting sentences with changed words.
Example of Model 1 input: Police officer arrested for abuse of authority
Example of Model 1 output: <no-change> <change> <no-change> <no-change> <no-change> <no-change>
Example of Model 2 input: Police <...> arrested for abuse of authority
Example of Model 2 output: Police dog arrested for abuse of authority
I am going to use a RNN/LSTM model for sequence tagging. As for the changed word prediction task, I am thinking of either using LSTM (concatenation of two parallel LSTM layers - one to run forwards on left context of word and the other to run backwards on right context) or fine-tuning BERTForMaskedLM from huggingface/transformers.
The question is whether it would be appropraite considering the small number of data, or should I switch to some other models?

Both bidirectional LSTM (like ELMo) and BERT seem appropriate for this kind of task. Whether one or the other performs best can only be known by testing.
If you use BERT, ensure to apply the typical measures to avoid overfitting. If you apply LSTMs, you will probably need regularization measures.


How to handle images of different sizes that are smaller than the input layer of a deep learning model? I am performing human awareness detection and have trained my model using transfer learning with MobileNetV2. This model expects a tensor of dimension [Null,224,224,3].
I have applied face detection using BlazeFace which uses an input of [128,128,3] on the input video stream and cropped the detected faces in order to send the cropped faces to my custom model but I am not sure what to do as the cropped images are all of varying sizes and smaller than what my model expects.
Example of a cropped face tensor
Array [
  1,
  43,
  111,
  3,
]

The issue was fixed by resizing the Tensor to fit into the model. I was reshaping them instead of resizing them.


Logistic Regression Independent Samples I'm studying Logistic Regression for the first time.

My book explains how to fit the weights $w$ using the likelihood function $L(w)$, assuming that the individual samples in our dataset are
independent of one another.

I know what independent events are. But I don't understand the meaning of the individual samples in our dataset are
independent of one another in the Logistic Regression case. What does this mean? And why?

Context / big picture:

Two events are independent if they have no influence on each other's outcomes. For example, if event A is "I go get coffee" and event B is "it's raining outside", then events A and B are independent, because I'm a coffee fiend and I don't care whether it's raining - I'm getting that coffee anyway!

Logistic regression example:

Say you have blood pressure samples from 50 different individuals at a hospital. You want to classify these as high or low blood pressure. In this example, your X (input variable) is the blood pressure from the 50 individuals, and your Y (binary response) is yes/no to the question 'is this blood pressure high?'. Each person's blood pressure is independent (if we assume they are unrelated strangers).

If you were to sample the same person's blood pressure twice, like before and after physical exercise, then you would no longer have independent samples.

Why:

We need independent samples in logistic regression because otherwise the degrees of freedom of the model are not what we expect them to be, and this affects further calculations. For example, in your dataset, you have 50 independent samples. You use 2 to estimate your intercept and slope, which leaves you with 48 degrees of freedom to work with. If you didn't have independent samples, then you would not have 48 degrees of freedom.

EDIT: a bit more detail on why independent samples are important.

The assumption of independence ensures that each sample contributes the same amount of information to the experiment. With independent samples, this is the case. A blood pressure measurement from one individual is not dependent on the blood pressure of another individual - they're independent. However, if you measure the same person's blood pressure before and after exercise, a different amount of information is added. You can no longer ask the same questions as you could with independent samples.

Let's pick a simpler model to describe what's going on. We will follow standard hypothesis testing principles to illustrate the practical difference between independent and dependent samples. I will round to two decimal places for ease of reading.

Suppose we have two groups of people: those with blood type A and those with blood type O. (Aside: notice people cannot have both blood types at once). We want to investigate whether those with blood type A have a higher resting heart rate than those with blood type O. We have 12 people with blood type A and 10 people with blood type O. To answer this question, we'll conduct a two-sample hypothesis test.

Null hypothesis: The mean resting heart rate of those with blood type A is the same as the mean resting heart rate of those with blood type O

Alternate hypothesis: The mean resting heart rate of those with blood type A is different from the mean resting heart rate of those with blood type O.

We measure each person's resting heart rate, and we get the following results:

Type A: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

Type O: 61, 64, 70, 72, 59, 62, 66, 78, 69, 75

We then calculate the sample mean and sample standard deviation of each group:

mean(Type A) = 69.67; standard deviation(Type A) = 8.35

mean(Type B) = 67.6; standard deviation(type B) = 6.28

We can then calculate a test statistic, which will follow a t-distribution. We may decide to use either Welch's method or a pooled variance test - in this case, I'll show Welch's method, which doesn't require the population variances to be equal (unlike the pooled variance test).

The test statistic is calculated via:

standard error: sqrt((8.35^2/12) + (6.28^2/12)) = 3.02

test statistic = (69.67 - 67.6)/3.02 = 0.69

The degrees of freedom here are difficult to calculate, but a generally accepted approximation is "the lesser of the degrees of freedom in the two groups", that is min(df_1, df_2). In this case the degrees of freedom of the first group is 12 - 1 = 11 and the degrees of freedom of the second group is 10 - 1 = 9. We subtract 1 here from each sample size, because we have estimated one parameter for each group (the sample mean).

If we wanted to continue this example, we could look at a t-distribution on 9 degrees of freedom and decide based on the test statistic whether to reject or not reject the null hypothesis. In this case, we're more interested in the degrees of freedom than the result, so we'll move on.

Dependent samples

Suppose we have a that same group of people who all have blood type A. Instead of measuring their resting heart rate just once, we’ll measure each Type A person’s resting heart rate twice. This introduces dependence between the samples.

Null hypothesis: the mean resting heart rate of people with blood type A is the same as the mean resting heart rate of all blood types

Alternative hypothesis: the mean resting heart rate of people with blood type A is different from the mean resting heart rate of all blood types

We measure each person's resting heart rate before exercise, and we get the following results:

Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

Then we measure their resting heart rate after exercise, and we get the following results (notice that person 1 in the previous list of heart rates is the same as person 1 in this list):

After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93, 84, 85

Now we have two observations for each individual. We can’t ask the same question, unless we want to throw out half of our data! Instead, we’ll ask whether there is a difference in the resting heart rate before and after exercise. We’ll use a paired-difference t-test.

To conduct this test, we have to take both observations from each individual into account. We calculate the differences between each person’s before and after measurement, and treat this set of differences as our actual data set:

60-90, 65-85,70-70, 62-72, …

Then we can calculate a sample mean, a sample standard deviation, etc. Our degrees of freedom would be 12 -1 = 11.

What would have happened if we had assumed that those dependent samples were independent?

Notice that when we had dependent samples, our degrees of freedom was 11; when we had independent samples, our degrees of freedom was 9. If we had assumed that the dependent samples were in fact two independent samples, then we could have used the wrong degrees of freedom.

** What if we had had only 10 measurements instead of 12 in the ‘dependent samples’ section? Wouldn’t we have had the same degrees of freedom? **

Yes, but it still wouldn’t have meant the same thing. We would have had:

Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93

And we would have had to ‘throw away’ the last two ‘Before exercises’ measurements and use only the first ten, as the last two had no partner measurement. The meaning has still changed, though. With dependent samples, you cannot ask the same kinds of questions as with independent samples.


Merge two datasets based on date ranges. R My goal is to merge two datasets using date ranges. Dataset1 contains patients stays in a hospital overtime. Dataset2 contains room information overtime. My goal is to identify what type of room the stays were in my Dataset1. It can get complicated since room type in some hospitalizations can change. For example patient 101 second hospitalization was part ICU and part Emergency.
Dataset 1
  PatientID Hospital Room           StartDate             EndDate
1       101     ODCC  4SW 2020-06-04 16:21:47 2020-06-22 15:12:39
2       101     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34
3       101     ODCC   1N 2020-09-08 14:03:35 2020-10-02 06:50:24
4       101     ODCC   1W 2020-10-02 06:50:25 2020-10-05 14:25:54

Dataset 2
  Hospital Room      Type    StartDT      EndDT
1     ODCC  11A     Other 2020-01-01 2021-05-12
2     ODCC   1W       ICU 2020-06-01 2020-07-30
3     ODCC   1W Emergency 2020-08-01 2021-05-12
4     ODCC   1N Emergency 2020-11-05 2021-02-07

My goal
  Patient.ID Hospital Room           StartDate             EndDate      Type    StartDT      EndDT
1        101     ODCC  4SW 2020-06-04 16:21:47 2020-06-22 15:12:39      <NA>       <NA>       <NA>
2        101     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34       ICU 2020-06-01 2020-07-30
3        102     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34 Emergency 2020-08-01 2021-05-12
4        101     ODCC   1N 2020-09-08 14:03:35 2020-10-02 06:50:24      <NA>       <NA>       <NA>
5        101     ODCC   1W 2020-10-02 06:50:25 2020-10-05 14:25:54 Emergency 2020-08-01 2021-05-12

Below you can find codes to replicate my datasets.
stays <- structure(list(
  PatientID = c(101, 101, 101, 101),
  Hospital = c("ODCC", "ODCC", "ODCC", "ODCC"),
  Room = c("4SW", "1W", "1N", "1W"),
  StartDate = structure(c(1591287707, 1592838760, 1599573815, 1601621425),
                        class = c("POSIXct", "POSIXt"), tzone = "UTC"),
  EndDate = structure(c(1592838759, 1599573814, 1601621424, 1601907954),
                      class = c("POSIXct", "POSIXt"), tzone = "UTC")),
  class = "data.frame", row.names = c(NA, -4L))

type <- structure(list(
  Hospital = c("ODCC", "ODCC", "ODCC", "ODCC"),
  Room = c("11A", "1W", "1W", "1N"), Type = c("Other", "ICU", "Emergency",
                                              "Emergency"),
  StartDT = structure(c(1577836800, 1590969600, 1596240000, 1604534400),
                      class = c("POSIXct", "POSIXt"), tzone = "UTC"),
  EndDT = structure(c(1620777600, 1596067200, 1620777600, 1612656000),
                    class = c("POSIXct","POSIXt"), tzone = "UTC")),
  class = "data.frame", row.names = c(NA, -4L))

Thank you!
Marvin


Normalize data from different groups I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything to the same scale. For example, I want to normalize to a 0-1 scale or 0-100, that way I could say something like 43rd percentile and it would have the same meaning across groups. If I were to just, say, standardize each individually by subtracting the mean of each and dividing by standard deviation work? Would I have to calculate the mean/st. dev of all of the combined data or do each of the 27 groups individually?

You can normalize each criteria independently in values between 0 and 1 without taking into account the other criterias, it will work better for most classification methods k-nearest neighbors, random forest, neural network, etc.
$$x^*_{i,j}=\frac{x_{i,j}-x^{min}_j}{x^{max}_j-x^{min}_j}$$


Clustering efficiency in a discrete time-series Is it possible to identify the point in time where the cluster separation is at its most in a discrete time series clustering?

Say I have 4 clusters of discrete time series and I want to pick a point in time where I can tell with the least bias which cluster it belongs to after a kmeans clustering, what other criteria than classification success can U use to identify my cluster separation performance?

Let me start with an important point; what other criteria than classification success can U use to identify my cluster separation performance?:

Classification as an indicator for clustering performance has an internal paradox. If you have the classification the clustering question does not apply anymore. These two concept are coming form two totally different philosophies so I would say be careful if you have already understood the concepts (what you say may make sense in semi-supervised learning which is not in your tags so I assume there is a misunderstanding).
Clustering has no performance evaluation! this is the problem I see most of data scientists struggling with. In practice you may define a good criterion (but honestly, what is good?!!) and deliver your solution, but in research schema there is no evaluation for clustering as the question itself is not well-defined i.e. you never have label to be sure who is who so you need to define closeness of points from which the problem starts; how close is called closeness?!!
Be careful about Curse of Dimensionality while using k-means for time-series clustering (I'm not sure how you do it).

After these points let's have a look at your question.

What are time-series? If time-series are pretty non-stationary or simply speaking is the dynamic behind variation is complicated enough, then there is not a one-to-one map for time-series characteristics and time points (imagine a simple ECG signal. It's pretty simple but if you explore research community you will find super sophisticated methods for feature extraction on ECGs. I'm pretty confident finding a time point at which ECGs differ is almost impossible.). You may extract features from your time series or embed it into some n-dimensional manifolds and look at it. In best case you may find some time-related features which describe your time-series and you may find some time-related criteria at which time-series differ (however I'd say it's not likely to find them).

Assuming time-series are pretty well-behaved (!!) with a simple dynamic (should be super simple). Then a solution might be to define a distance function of time-series which outputs the pair-wise distances of all time-series as a single score. Then the maximum of this function returns the time-point at which these time-series are pretty distinguishable.

If you provide more information on your data I might be able to give a more detailed precise answer.

Good Luck!


can i make model that can take N shape input? as far as i try all data that model take are fixed size or fixed shape, i had problem that i give data to model but it has no fixed shape and i can not cut part of it as all of it are important,so can i make model that have unshaped data as input then take unshaped data to predict ..
like have small problem that can be represented by N like famous problem travel sales man as in it N refer to number of cities

The first thing, which comes to mind speaking about unfixed input, is a recurrent neural network. Such a model adresses dynamic behavior. Maybe, it's possible to represent data as a sequence? I don't know nature and specificity of your data, but if each instance in the dataset consists of a different number of some "similar events", it might be reasonable to think in this direction.

The other way which might help is to compute statistics on the data. Then it will be another feature space, but in some cases it could be enough to achieve high accuracy of predictions. For example if there are k features of the same nature (say, temperature in different days), then average, standard deviation, min, max, etc. could be considered. And statistics don't depend on the length of each instance in the dataset, so this number k could differ.


What is Orange all about? I read a lot about Orange here in the forum. I know that it is a software developed at the University of Ljubljana. I guess it has some kind of GUI which makes it a "cheap and easy" way to implement ML models.
My question: What is special about Orange? What are the advantages of the software (especially when compared to R or Python)? Is it merely some kind of GUI or does it come with functionality that goes beyond what R or Python can do?
I work with both R and Python and I wonder if it is worth it to have a look at Orange.

As you mentioned, Orange is a data mining software developed by the University of Ljubljana. It can be used for developing and testing machine learning models as well as conducting exploratory data analysis and visualization.

One of the unique features that makes Orange "special" is its simplicity and ease of use. This is because components in Orange are drag and drop and widget based. This means instead of needing to write python code to import a data file and pass it through an algorithm, this can be done by connecting two widgets. Another feature is its specific widgets for things like Bioinformatics and Spectroscopy. It has built in modules for a wide range of fields which make it easy to work with various types of data from different applications.

Personally, I think some of the benefits of Orange include its rich visualizations, interactive models, and speed. I code mostly in Python but often use Orange to get a quick look at what the data looks like or cross validate a model's performance I am developing elsewhere very quickly. I also think its interactive KMeans and Gradient Descent widgets are great tools for understanding what is happening with these algorithms under the hood.

One of the drawbacks is that it is not enormously robust for working with large datasets. Datasets which may work well in Python may make Orange crash. This is why it is best suited for smaller projects, pedagogical purposes, or exploratory data analysis.

Below is an example snapshot of what an Orange workflow would look like:

Bottom line: I think every data scientist should have some familiarity with Orange to add more tools to their toolkit. Easy and quick to learn, worth the effort.


Scikit-learn average_precision_score() vs. auc score of precision_recall_curve() I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the precision_recall_curve(), I get a different macro-metric than when I use the builtin average_precision_score() function. As per the docs, this makes sense, due to auc() using a trapezoidal approximation, and thus average precision being more accurate. However, when I compute the micro-metrics, the values are the same. I'm trying to understand this.

Here's an example:

yhat_raw = np.array([[0.511, 0.62, 0.3],[0.11, 0.76, 0.945],[0.002, 0.671, 0.452],[0.9102, 0.783, 0.2]])
y = np.array([[0, 1, 1],[0, 0, 1],[0, 0, 0],[0, 1, 1]])
yhatmic = yhat_raw.ravel()
ymic = y.ravel()
prec["micro"], rec["micro"], _ = precision_recall_curve(ymic, yhatmic,
pos_label=1)

auc(rec["micro"], prec["micro"])
returns 0.58305555555555544.
average_precision_score(ymic, yhatmic) returns 0.62222222222222223.

Upon actually deploying the model, these metrics are coming to the same thing. I think this makes sense, considering that at a larger scale (e.g. along a dimension of thousands of datapoints rather than len(rec['micro']) = 11 datapoints in this example), the trapezoidal approximation is much closer to the true value.

My question is, why do the metrics then differ in the macro case? Is it fair to say that in a real-world deployment, the micro-evaluation has enough datapoints for the trapezoidal approximation to converge to the true value, but the macro- does not (computed across ~3000 test examples, with a label space of ~2000 labels)?

The metrics differ in the macro case because macro in scikit-learn ignores class imbalances. micro in scikit-learn looks at global rates. The data has class imbalances, thus the macro argument will return different values compared to micro.


Categorical Variables in Decision Tree I was going through the Andrew Ng's notes for Decision Trees. It has one section explaining the usage of categorical variables using Decision Trees in which I am not able to understand this part

" A caveat to the above is that we must take care to not allow a variable
to have too many categories. For a set of categories S, our set of possible
questions is the power set P(S), of cardinality 2^|S| . Thus, a large number of categories makes question selection computationally intractable. "

Q1 -> How is the possible set of questions is $2^{|S|}$ ? It should be equal to the non-leaf nodes in the decision tree .

Link to Andrew Ng notes on Decision Tree : http://cs229.stanford.edu/notes/cs229-notes-dt.pdf

Can you please give some clarification on it .

The $2^{|S|}$ number of POSSIBLE questions is defined by the number of different combinations of subsets of $S$ that could be part of the first partitions, then the second and so on until the tree is finished. That being the worst case.

Imagine that: $loc \in{} N$ is the first separation, so it's divided into Yes/No. Then $loc \in{} S$ is divided into Yes/No.

Then $loc \in{} E$ is the third partition and the answer is the same (Yes/No), you have 8 (2^3) questions asked. There are many combinations of use/no use/which to combine in which step, etc, but ultimately you will be asking the tree to consider all possibilities, which are $2^{card(S)}$.


Mean Absolute Error in Random Forest Regression I am new to the whole ML scene and am trying to resolve the Allstate Kaggle challenge to get a better feeling for the Random Forest Regression technique.

The challenge is evaluated based on the MAE for each row.

I've run the sklearn RandomForrestRegressor on my validation set, using the criterion=mae attribute. To my understanding this will run the Forest algorithm calculating the mae instead of the mse for each node.

After that I've used this: metrics.mean_absolute_error(Y_valid, m.predict(X_valid)) in order to calculate the MAE for each row of data.

What I would like to know is if the logic I'm following is sound. Am I making a fundamental mistake or missing something here? Should I have used the default MSE based Regressor and then calculate the MAE of each row using the mean_absolute_error function?

Let me clarify few fundamental things:

In sklearn, RandomForrest Regressor criterion is:

  The function to measure the quality of a split

It's a performance measure (by default, MSE) which helps the algorithm to decide on a rule for an optimum split on a node in a tree.
Kaggle is giving you a metric, i.e. MAE (again a performance/ quality measure) but to evaluate the performance of your ML model, once finalized.

To come back to your question: while both MAE/ MSE are performance measures, they are being used at two different stages of a modeling process and might not be related. So, while it makes sense to evaluate your final model on MAE as you would be judged on it, you can choose any of MAE/ MSE for criterion (i.e. for RandomForest) depending on performance at validation stage.

While the above being said, keep in mind that you might want to evaluate the validation errors (i.e. for finalizing a model) on the same metric (i.e. MAE in this case), to keep error measure consistent with the test set evaluation.


Input and output feature shapes in CNN for speech recognition I am currently studying this paper and are trying to understand what exactly the input and output shape is.  The paper describes an acoustic model consisting of using cnn-hmm as the acoustic model.  The input is a image of mel-log filter energies visualised as spectograms. The paper describes a method  for phone recognition in which (as far I understand) applying a CNN on these spectograms with a limited weight sharing scheme should be beneficial for phone recognition.

The input shape as far i understand, is 9-15 frames, which seem a bit confusing, as they don't consider number of phonemes a utterance may have, or the length of them, but simply just "choose" a number of frames to operate with.. The number doesn't seem to be connected with the output in any way - or am I misinterpreting something?

For the output

  We used 183 target class labels, i.e., 3 states for each HMM of 61 phones. After decoding, the original 61 phone classes were mapped to a set of 39 classes as in [47] for final scoring. In our experiments, a bigram language model over phones, estimated from the training set, was used in decoding. To prepare the ANN targets, a mono-phone HMM model was trained on the training data set, and it was used to generate state-level labels based on forced alignment.

So the output is divided in to 183 classes, being mapped into HMM's with 3 states for each 61 phonemes, and the ANN target (As I see it target = posterior probability) by training a monophone hmm with forced alignment. I am not sure I understand this process.  If the ANN targets are those the CNN should aim/regress to and at the end classify the state based on, Why then process the input?.. why not make a simple DNN that does the regression/classification?

It looks like the improvement lies in the use of forced alignment here, and only on monophone? where is the improvement?

And again how I am supposed to link the input shape and the output shape based this? This would require the audio files to have a certain length, the length of the audio is never specified, so I am assuming that this is not the case.

DNN/CNN prediction(training) is done for 1 frame at a time. The output can be any of the 183 outputs states. Length of the audio files is not a problem since the input to the DNN/CNN is of same dimension only the number of inputs change with audio length.

e.g 1.wav has 500 features and each feature is 39 dimensional and 2.wav has 300 features,
the trained model will take a 39 dimensional input and output will be 183 dimensional. So depending on length we'll get different number of outputs.

Since all the frames in an utterances are tested against all the 183 possibilities so the output always remains 183 dimensional. There is no need to specify number of phonemes an utterance can have as everything is being done at frame level.

Frame concatenation (9-15 frames) is done to leverage contextual properties of speech data. Phone changes are context dependent. For 15 frame context, we change the input of DNN to [7*39 (left_context) 39 7*39(right_context)], a 585 dimensional vector. So now DNN will take 585 dimensional data as input and will output a 183 dimensional vector.

CNN input

  There exist several different alternatives to organizing these
  MFSC features into maps for the CNN. First, as shown in
  Fig. 1(b), they can be arranged as three 2-D feature maps,
  each of which represents MFSC f
  eatures (static, delta and
  delta-delta) distributed along both frequency (using the fre-
  quency band index) and time (using the frame number within
  each context window). In this case, a two-dimensional con-
  volution is performed (explained below) to normalize both
  frequency and temporal variations simultaneously. Alterna-
  tively, we may only consider normalizing frequency variations.
  In this case, the same MFSC features are organized as a number
  of one-dimensional (1-D) feature maps (along the frequency
  band index), as shown in Fig. 1(c). For example, if the context
  window contains 15 frames and 40
  fi
  lter banks are used for each
  frame, we will construct 45 (i.e., 15 times 3) 1-D feature maps,
  with each map having 40 dimensions, as shown in Fig. 1(c).
  As a result, a one-dimensional convolution will be applied
  along the frequency axis. In this paper, we will only focus on
  this latter arrangement found in Fig. 1(c), a one-dimensional
  convolution along frequency

So the input to the CNN will be an image patch of size 45 * 40 regardless of the length of the audio file, just the number of such inputs will depend on the length of audio file.

Why force alignment?

Now we are doing everything at frame level so for each frame we need the state labels. Usually this timing information is not available.

Transcribed data usually looks like this,
1.wav -> I am a cat

Now I don't know how many frames belong to I or to a.

HMMs are trained on this data and force alignment is done to generate state level labels for each frame.

Better modeling of input-output relation gives improvement over other methods. Filter bank features also contribute to improvements. A DNN trained with filter bank features gives better performance as compared a DNN trained with MFCC features.


Do you think it's normal for data science projects to have some amount of "what should we do" -time? Or does it mitigate by experience? Do you think it's normal for data science projects to have some amount of "what should we do" -time? Or does it mitigate by experience?

By "what should we do" -time I refer to time being spent on reading about and experimenting on "possible ways to do things, when many alternatives exist". This kind of "time" has bothered me, because someone could think it's some kind on inefficiency or failure to concentrate.

On the other hand I've rationalized that it could be a normal part of a data science project, because projects may be different, they may have special considerations or boundaries that are different from other projects. This would lead to time having to be allocated to "particularizing from general and other solution methods". Thus also, to expect that one would instantly understand the "best way" to solve a problem, would be too optimistic and forget that "one may not know, without experimenting, what the best solution is". Since a data science project is not necessarily an "exact mathematical problem".

This is really about differences in the expectations of your employers and you.

In general every job depends on continued learning and making time to expand your knowledge. Data science as a very heavily knowledge and methodological based occupation and therefore life-long learning is even more of a thing.

But how and when to do that learning? In my experience most employers expect you to do that sort of learning on the side or with irregular, slotted and scheduled training.

However I think it is quite normal that a big part of learning is during projects / on-the-job like you describe. Experimenting with new methods, reading up on other solutions, etc.

At the end you have to balance these expectations and just make sure, that your "research time" is goal-oriented and efficient. Don't spend hours reading up on "interesting untested solutions" but also don't worry if you spend time on datascience.SE to look up solutions and guidance.


Which feature to use in feature selection? Objective: Multiclass classification with supervised learning, small dataset (25h)

Context: My dataset is composed of mobile network data collected with a smartphone. The labels correspond to the activity of the user (Stationary, Walk, Subway, Train, Car). My features are calculated based on 3 fields: timestamp, ID, and signal strength (SS). All have different overlapping size windows: 15s, 30s, 602, 90s, 120s. So, I have 3 features based on ID and 16 statistical features based on SS for each window size with a total of 95 features.

My Question: Which feature selection should I use? Am I correct saying the features are not independent?

(I'm using python).

My guess would be that the features are highly correlated. Check this. Regarding feature selection, I suggest starting with Logit and Lasso (L1 regulation). This method (Lasso, l1) can „shrink“ features to zero (so kick them out basically). This happens automatically based on feature importance.

Many methods (e.g. Boosting, Neural Nets) allow you to use L1 regularization. So there may be no need to „manually“ select features.

If you want to do that manually, you may use stepwise feature selection, as e.g. described in Introduction to Statistical Learning, Chapter 6.

Here is some Python code from the chapter: https://github.com/JWarmenhoven/ISLR-python

You may also have a quick look at this post related to Lasso: https://datascience.stackexchange.com/a/55702/71442


Do statistical test with orange between data I'm learning orange and I'm looking to performe some statistical test on my data.
The test I usually do are z-test, t-test, chi-square test and Anova.
Can I use a widget to do these or should I do a python custom script ?

Thank in advance.

The Box Plot gives a T-Test for two-group comparisons and Anova for > 2 group comparisons.

The Sieve Diagram gives a Chi2 test.


Why is patsy used to prepare data for logistic regression? I'm pretty new to both ML & scikit-learn. I've noticed that some example tutorials & codes online use patsy's dmatrices to prepare data for logistic regression. I don't understand why this is done. Example
In the case above for instance, isn't it sufficient to use the data in the dataset directly to train the logistic regression model? What exactly is the point of using patsy?
Thanks!

Mostly convenience. In this particular case, it takes care of one-hot encoding categorical variables e.g. C(occupation).

patsy takes care of other things under the hood as well, like dropping rows with missing values and adding a constant intercept variable.


Hot to use the formula model in t.test I am trying to better understand the formula model for two sample t-tests in R. When I calculate the test in the formula model I get a wrong result.
set.seed(41)
df = data.frame(x1=c(rep(1, 10), rep(0, 10))+ rnorm(20, mean = 0, sd = 0.1),
               x2=c(rep(0, 10), rep(1, 10)))

t.test(x1 ~ x2, data=df)

Output
    Welch Two Sample t-test

data:  x1 by x2
t = 22.365, df = 17.85, p-value = 1.668e-14
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.9247087 1.1165780
sample estimates:
mean in group 0 mean in group 1
      1.0530115       0.0323681

If I use the variable model, I get the expected result.
t.test(x = df$x1, y = df$x2)

Output
    Welch Two Sample t-test

data:  df$x1 and df$x2
t = 0.2581, df = 37.945, p-value = 0.7977
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.2921655  0.3775450
sample estimates:
mean of x mean of y
0.5426898 0.5000000
```

Which result is right or wrong here depends on your objective. You have created two variables (vectors) $x_1, x_2$.
Assuming $x_1, x_2$ are two samples of i.i.d random variables $X_1$ and $X_2$, respectively. Now, with some more assumptions, you want to test the null hypothesis: $\mathbb E(X_1) = \mathbb E(X_2)$.
For this, your second output is the correct one. However, based on the data that you have generated, this is not applicable because each of your samples, $x_1, x_2$ are not coming from the same distribution, as the mean of your first five values is different from the last five.
Ignoring your data, this analysis can be done using the formula approach as well. Join the two vectors $x_1,x_2$ to $x$ and add another column (say, y) which identifies which data point is coming from which sample. Call this new data frame df1. Then an equivalent way of doing the above mentioned test is t.test(x~y, data = df1)
The second approach is helpful when your data is organized in such a format. For example, say, you have data frame with two columns: height ($x$) and gender ($y$). Then running t.test(x~y, data = df1) will test whether the mean height is different between genders.
Your first approach can be considered right only when your $x_2$ is a factor variable which identifies the group or sample of the data point in vector $x_1$.


Explanation behind the calculation of accuracy in deep learning model I am trying to model an image segmentation problem using convolutional neural network. I came across code in Github which I am not able to understand the meaning of following lines of codes for calculation of accuracy -

def new_test(loaders,model,criterion,use_cuda):
    for batch_idx, (data,target) in enumerate(loaders):
        output = model(data)
###Accuracy
    _, predicted = torch.max(output.data, 1)
    total_train += target.nelement()
    correct_train += predicted.eq(target.data).sum().item()

model(data) outputs a tensor of shape B * N * H * W
B = Batch Size
N = Number of segmentated classes
H,W = Height,Width of an image

_, predicted = torch.max(output.data, 1)

On the first line above, it uses torch.max() to find all cases in output.data along the dimension - 1(i.e along each row). In addition, torch.max() returns (values, indices), so only the indices are stored in a variable called predicted while the values are not stored.

total_train += target.nelement()

The second line above just tracks the total size of the train set into a variable called total_train based on the number of elements in target

correct_train += predicted.eq(target.data).sum().item()

The third line above takes the predicted variable, which contains all indices of when the output is greater than 1, checks it against target.data, and sums up all items that are equal. In other words, it is counting how many of the predicted values are equal to the target data.


I am getting different mean_absolute_error when i retrain my model everything same I have set my numpy random seed to 0.
I am training on colab and using keras.
I didn't change anything.
I just re-ran my cell and the val_absolute_error changed.

Code:

np.random.seed(0)
regressor = Sequential()
regressor.add(Dense(10, input_dim=1, activation='tanh'))
regressor.add(Dense(20, input_dim=1, activation='relu'))
regressor.add(Dense(15, input_dim=1, activation='tanh'))
regressor.add(Dense(1))
regressor.compile(optimizer='adam', loss='mean_absolute_error', metrics=['mae'])
model = regressor.fit(X_train, y_train, epochs=450, batch_size=10, validation_data=(X_val, y_val), verbose=1)

print(regressor.evaluate(X_test, y_test)) --> This is the error on the test set

Please note that I made a mistake in the screenshot, it is the error in test set not val. set

I think you need this one too:

from tensorflow import set_random_seed
set_random_seed(123)


Correlations - Get values in the way we want I have :

a matrix X with N lines
a vector Y

I've computed the Euclidean distance with Y for each line of X.

What I get is a vector of distances.

What I want is a vector of scores between 0 and 1, 1 meaning "very" high correlation, 0 meaning "no" correlation.

Here what I did :

I divided the vector of distances by the max distance inside it.
I get vector D.

1 - D is the final result with values between 0 and 1.

The problem is that I get many values (75%) too close to 1.
Do you think what I did is correct ?

How would you get a better result ?
(Between 0 and 1 but not everything too close to 1)

For now, I tried to take the square of the result. (To stay between 0 and 1 but to minimize the values)

Here a picture of the distance values I want to turn in a score

Several kernel functions can serve as similarity functions (=scores). See a list, for example, here. You can try several of them and see which suits you the best.

You need something that drops fast at low distances. You can try
$$ score = 1/(1+distance)^2$$
and adjust coefficient in front of distance so that the score fits between 0 and 1

About your picture: what are axis labels? and what are x-ticks?


change in variable importance I have a multi year dataset. Each time frame  of the data has different predictor importance.

Say for example, I am slicing the data into two partions as follows:

a dataset for the year 2014 (whole year)
a 2015 Jan.

When i look for the predictor importance, the predictor variables are different for both the partions. (1) Hence i am not able to arrive at one  unique decision tree which can explain the model better. (2). I am not able to train a model which can predict the new data correctly.

Is there anything I am going wrong here.

Probably you can use multiple decision trees - each for slice of dataset. Then combine the results from each decision tree. You can also weigh the results from each decision tree. Eg the dataset for 2015 can be given more weight than dataset for 2014. I do not see much harm in spliting dataset and training different trees as it helps to account for the predictor values in a more accurate manner. If you could more details about the data or the "variable" you mentioned, people would be able to help more.


Multi-label classification with nested features I need to perform a multi-label classification. I have three features and they are nested. I am unsure how to combine this or what kind of classification algorithm would be best. Some multi level neural network as shown here seems good, but the nested features don't seem to be taken into account there.
I present the nested features (X) and labels (Y) in the two datasets below: one subject ID can have one or more features and one or more classes. Features and classes can be 'occupied' by one or more subject.
Note: I have about 100k subjects, 1k features (at the third level) and 200 classes.
data_features
       subject_id   feature1   feature2   feature3
               1          a         aa          aaa
               2          a         aa          aab
               3          a         ab          aba
               1          a         ab          abb
               2          b         ba          baa
               3          b         ba          bac
               1          b         ba          bad
               2          b         ba          bad
               3          c         ca          caa
               4          c         ca          caa
               5          c         cb          cba
               6          c         cb          cbb

data_labels
       subject_id   label1   label2   label3   label4
               1        0        1        0        0
               2        0        1        1        1
               3        0        1        1        0
               4        1        1        0        1
               5        1        0        0        0
               6        0        1        1        1
               7        0        0        0        1
               8        1        1        1        1
               9        0        0        1        1
              10        1        0        1        0
              11        0        1        0        1
              12        1        0        0        1

I am quite unsure what algorithm would combine those the best? (I am skilled in R and SAS and decent in Python, but will learn any other language that would be needed)

I think it's always a good idea to start simple, so I'd simply suggest to try with all the features, including the different levels of "nesting" so around 2k apparently. Given that the dataset is large, I don't see any obstacle to trying this way. For the same reason I would start with a very simple model like Decision Trees or SVM, which have the additional advantage that they're fast to train. This could be a first step which provides you with a decent baseline, at least.
If the number of features turns is an issue for a more advanced option, I think this is a good case for using feature extraction (for example PCA): this would reduce the number of features and also merge features which represent the same information.


How can you make use of Json format data? I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an excel format. Thanks for your time.

Here is a tutorial by Corey Schafer on working with JSON data. He explains in detail how to convert JSON data into python objects, but I don't think he explains how to convert them to excel here. But if you are familiar with pandas, you should be able to create dataframes and then convert them to excel files as needed.


How to select features when performing classification with a dataframe of multiple columns? I have a dataframe of 50000 observations and I want to perform a classification task. But I'm struggling with features selection. I have 89 columns, which after getting rid of some redundant features, I get down to 58 columns. Some of them are categorical, some numerical.

Any suggestions would be much appreciated.

You could use logistic regression with regulation (Lasso, Ridge, Elastic Net) to get an idea of what features are relevant. Regulation means, that features which are „not so important“ are shrunken. In R glmnet is a good starting point which is also available for Python.

R: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Python: https://web.stanford.edu/~hastie/glmnet_python/

See Section 6.2 in Introduction to statistical learning (ISL) for some background and examples. You may find a PDF of the book online. http://www-bcf.usc.edu/~gareth/ISL/

Ridge is based on the L2 norm, so that model coefficients are shrunken, but cannot be zero. Lasso in contrast uses the L1 norm, so that coefficients can be shrunken to zero.

Another approach to model selection would be forward / backward stepwise selection (see: ISL, Ch. 6.1).

For forward stepwise selection, the process would be:

Let M0 be the NULL model, which containes no predictors. Now for k=0, ..., p-1:

Consider all p-k models that augment the predictors in M with one additional predictor.
Choose the best among these p-k models (e.g. based on smalles RSS / best R2).
Finally, select a single best model from the M models using cross-validated prediction error, AIC, BIC, or adjusted R2.

Backward stepwise selection works in the same way, but you start with the full model and successively leave out predictors/features.

For Python, sklearn comes with a greedy stepwise (backward) selection module: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html

Also note, that many methods/packages come with a module that allows to assess feature importance, e.g. in catboost (https://catboost.ai/docs/concepts/fstr.html) or lightgbm (https://lightgbm.readthedocs.io/en/latest/Python-API.html). However, a disadvantage here is, that you need to implement a proper model (with some hyperparameter tuning) first, which is time consuming.


Transforming words in sentences into vector form to prepare a model I want to build a simple classifier that classifies if the text is a question or just a simple message. I understand logistic regression and can work to create a simple neural network.

I have the labeled input data in English, Japanese, Korean, Thai. How could I transform this data before I feed it into the classifier?

An approach would be to sort out all the words in your data according to how often they appear, i.e. their "frequency". After that, pick the "X" most frequent words in your dataset to use them for the classification of your dataset.

Assuming that you are working with Python and Keras, you should use the Embedding layer. For more details about how to use that layer, check this.

Shortly, what this layer does is that it maps the input to a high dimensional vector domain. A word is converted to a real-valued vector and word similarity is evaluated by the "closeness" of two word-vectors in the high-dimensional vector space.

Also make sure that your dataset consists of texts of fixed-length, by truncating long sequences or zero-padding short ones.

After all this is done, you can train a recurrent neural network with LSTM neurons as a text classifier. LSTMs have been proven very successful in text processing due to their inherent memory.

A hands-on Python/Keras tutorial that demonstrates all the above can be found here, I am sure it will be of high help :)


PMML XML Model == same as config file or implementation? there is the PMML standard defined as XML Schema for Data Mining Model portability between different DM software. My question is, does this xml file serve as a sort of "config file" with the specifics of the underlying domain-model relation (with the specific model implementation logic for the data mining problem concerned), for the other software that you want to import your model?
Or is the PMML model the implementation itself, I mean is there any logic implemented to execute the model with the PMML xml file itself (e.g if you add some test data and let it run through your xml) or must there exist an implementation of the learning algorithm in the software and the PMML is just the configuration of the specific model?

Essentially, the PMML is just a set of formulas (and the corresponding coefficient values).

The story goes something like this:

You develop a model capable of taking some inputs and outputting certain coefficients that correspond to a certain formula.
Then, these ouputs are turned into a PMML file (an XML-based encoding of your model's outputs).
We can now say, "please evaluate this formula, using these coefficients. As input for the independent variable(s), please use these attributes from my dataset here."

Take a very simple linear regression, for example.

The PMML consumer application parses your PMML file, and finds

model =
coefficient = <5.46>
X = < attribute:  colour of the leaf >

It now goes ahead and scores the formula for the reggression

Note: there’s got a to a file somewhere in your application which contains a column called “colour of the leaf”

And, there you have it.