Null values as useful information for feature engineering I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database because this depends on the chosen sample. Normalization will be done in Python. For my use case, which is a binary classification problem (fraud detection), the presence of null values also correlates with the target variable. Therefore, I would like to preserve this information for the model.

My proposal would be to create an additional binary column called varName_isnull which encodes the presence of a Null in the varName column. In other aggregated features, this binary column would also be used for example to calculate the number of Null values for a particular grouping (i.e. credit card) per unit of time.

Is my proposal reasonable? Are there any alternative representations and if yes what would be their advantages?

It would depend on the type of model you want to train.
If you are doing a classification model ( Yes/No, 0/1 type of class) then I suggest you use dummy variable generation functions in python.

I would also recommend that you convert all your other categorical variables into dummy variables in order for your model to take every bit of information into account.

one popular technique in feature engineering is one-hot-encoding to change the nature of your data.

So generally you would create a new variables ( as you have mentionned) that has 2 values :

1 if the variable is NULL
0 if the variable has a non-Null value
Below are some links with nice examples:

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

https://jamesrledoux.com/code/dummies

Good luck !


Machine Learning: Writing Poems I'm a student of machine learning, and these days I was trying to learn how to use the TensorFlow library. I've gone through various tutorials and trial&errors with tensorflow, and I thought the best way to learn it for real would be to make use of it in a little project of my own.

I've decided that I should attempt to make a program that writes poems. I'm not aiming for top-end quality program; for my first model, I'd be happy with just a string of non-sense words groups together in poem format. The problem is that I'm having problem looking up books or videos about machine learning programs that deal with writing sentence structures.

Can you make any suggestions on what I could look for (even google keywords are fine) to get the sample programs and basic knowledge that I need?

Thank you.

This is just the comment from Emre expanded, but yes you should look into recurrent neural networks for generating text in the style of a given corpus. RNNs and LSTM work really quite well for this.
This writeup is widely cited, and to your question, shows how it's pretty easy to generate something like this, given the text of Shakespeare's plays:

PANDARUS: Alas, I think he shall be come approached and the day When
little srain would be attain'd into being never fed, And who is but a
chain and subjects of his death, I should not sleep.
Second Senator: They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish The earth and
thoughts of many states.

If you follow this you can easily run this on your local GPU too to generate text from whatever input poetry you like. I have had pretty good results with 0.1-0.5 dropout, 2 layers, layers of size 512-1024.


Identifying members who are likely to move to a lower cost alternative product Overview:
I am looking for some technical direction from the ML/data science community about how I could tackle my business problem.

Context:
In a nutshell, I have a group of customers who repeatedly purchase products from a category of products e.g. the frying pan category. In the frying pan category there are 15-20 different manufacturers of frying pans. Each of the different frying pans have different features i.e. large, small, non-stick, stainless steel, dishwasher proof etc. but essentially they're all frying pans. Some people buy the premium product while others buy the budget and some others buy the mid range product.

The most popular brand of product is the most expensive and is a loss making product for us. There is however another product (mid range product) that has nearly the exact same characteristics as the expensive pan but is half the cost and generates the most profit for my company.

Objective:
I would like to identify those customers who currently purchase the most expensive product and reach out to them based on their likelihood of purchasing the mid range product to convince them of the benefits of the mid range pan.

Approaches:
I thought one approach to this would be to find the members who buy the expensive product but are most similar in characteristics/features of the mid range purchasers and offer them a discount to incentivize them to switch products. To achieve this I would use a user - user similarity measurement rather than an item - item similarity measure.

The data:
I have access to a swathe of demographic information, purchase history data, product detail characteristics so building analytical base tables is not a problem.

If you have any suggestions as to how I could approach this problem I would be very grateful for your insights.

We could analyze the customer's purchase history. Here, for the sake of explanation let's assume that customers A and B. First, we'll make the purchase history of customer A,

We'll calculate the frequency of customer A buying the mid-range pan. Also, we'll need the total number of purchases made by the customer. So, for customer A you'll get a ratio which is ( number of purchases of the mid-range pan ) / total purchases.
Similarly, we'll calculate such ratios for the expensive pans both for customers A and B.
Next, we'll compare the ratios for all customers and select the highest N customers who most probably will buy the mid-range or expensive pan.

Why are using the ratios ( or precisely, the probabilities of buying a certain pan )?

We'll need customers who are frequent buyers of a particular pan. These are more chances of switching a product rather than the less frequent customers.
As they are regular customers, they will be more influenced by the discount available on the mid-range pan.

  You may use various metrics to aggregate the customers who can have a product transition. You may also have a look at Bayes Theorem which I think will be appropriate too. It will give the answer- "Which are customers which have the highest probability of buying a mid-range pan given that they have already bought a expensive pan.


Understanding Embeddings input and output sizes I have been trying for a while to understand the dimensionality of embeddings in neural networks and I think that finally things have clicked on my brain, however I would love to check whether or not my understanding is correct.

Embeddings are an effective way to transform words into vectors, or at least to reduce the dimensionality of the data (essentially the Bag of Words approach does not work well as data is sparse)
If I have a text corpus that contains say, 5000 sentences, I could then pad each sentence to a standard size, for example 150 and then use embeddings (possibly the Glove pretrained ones) to get an output with dimensionality of 100, that means that I would have 5000x150x100 elements.

Is my understanding correct? If so this means that I can start training my network using mini-batches of say 16x150x100 elements, the layer after the embedding one could be then a LSTM and so on...

Both are correct.

Too clarify: when training in mini batches, it is more common to do padding after selecting eachmini batch. Say in your mini batch of size 16, if the longest sentence is of length 35, you want to pad this mini batch to 16x35x100.


Why do I need pre-trained weights in transfer learning? I am using a Mask-RCNN. I first chose the resnet50 backbone then downloaded COCO pre-trained weights.

Why do I need pre-trained weights for transfer learning?

The transfer learning approach is to train a base network and then copy its first layers of the target. My base network is resnet50 and I copied the first layers to Mask RCNN. So, why do I need pre-trained weights, for example, coco pre-trained weights?

I understand that I copy layers and pre-trained weights from resnet.

Thanks in advance.

When you are calling a pre-trained model (resnet50 in your case) for performing transfer learning, only the model's architecture is actually called.
This architecture is practically useless to perform any sort of computational operation on the data and predicting any output unless trained. For that, you will need the weights (variables understood by the model on some previous dataset).

Weights are more like mathematical coefficients which take the input and performs the computation on the input to generate a required output. They are more like the coefficient in a linear algebra problem. This is the actual reason why it is required to call the weights for the model.

Even if you are not calling any weights in you model, you can make your model trainable and train the same to generate your own weights that fit your data properly rather than fine-tuning your model. Read more about the same here


how can I solve label shape problem in tensorflow when using one-hot encoding? I used tensorflow to recognize text from natural images by using convolutional neural network; there is no specific number of characters in the text. To make a successful training I should convert the categorical labels into binary using one-hot encoding. So, for each label, I used integer encoding for each character and stored them in one numpy array in order to create TFRecords. For example:

alphabet = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ '
TrainLabel  = ["CNN in Tensorflow"]

# define a mapping of chars to integers
char_to_int = dict((c, i) for i, c in enumerate(alphabet))
integer_encoded = [char_to_int[char] for char in TrainLabel[0]]

if (len(TrainLabel[0])) < 51:
    for j in xrange(51- (len(TrainLabel[0]))):
        integer_encoded.append(52)

# one hot encode
onehot_encoded = []
for value in integer_encoded:
    letter = [0 for _ in range(len(alphabet))]
    letter[value] = 1
    onehot_encoded.append(letter)

label = np.array(onehot_encoded, np.float32)

51 is the maximum number of character in the text, so if the text has less than 51 characters, pad it to 51 characters with spaces.

If we print the label, it will be like this::

array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ...,
       [ 0.,  0.,  0., ...,  0.,  0.,  1.],
       [ 0.,  0.,  0., ...,  0.,  0.,  1.],
       [ 0.,  0.,  0., ...,  0.,  0.,  1.]], dtype=float32)

after creating the batch queue, the label has shape [batch_size, 2703]. 2703 is come from 51*53 which 53 is the number of classes

My problem is in loss function:: the label shape in tf.nn.sparse_softmax_cross_entropy_with_logits() must be [batch_size], but the label that I used here has this shape [batch_size, 53] because I used one-hot encoding?

How can I deal with that??

This is the problem::

  (labels_static_shape.ndims, logits.get_shape().ndims)) ValueError:
  Rank mismatch: Rank of labels (received 2) should equal rank of logits
  minus 1 (received 2).

Since your data are already in a one-hot encoding, you can use tf.nn.softmax_cross_entropy_with_logits(), which expects an input of shape [batch_size, num_classes] for the labels.

(The tf.nn.sparse_softmax_cross_entropy_with_logits() op expects the labels as a batch of integers, where each integer corresponds to the class ID for each example.)


Is there something like class-based object detection? Or class-based selective search? I've been reading a lot about computer vision lately, and while there is a huge amount of info about object classification, and a lot less on object detection, I have not found anything on class-based object detection i.e when I know what I am looking for. For example, looking for cats on a picture. Nowadays you can say if a picture is a cat (object classification), or if the picture has cats among other things.
How would I use this knowledge to improve the performance (in both accuracy and speed)?

Somethings that I wouldn't do (but i'm tempted to do):

Perform a brute-force solution with a binary classifier
Implement a shape-based selective search and then use a binary classifier

What i would really like to do, but haven't found in the literacy:

Implement a trainable selective search to find class-based regions of interest.
Use a binary classifier to check if ROI is in fact the object to be found.

Object detection models (such as SSD, Faster-RCNN, YOLO, R-FCN) are trained to detect specific classes.

If you wish to detect a single class, you could train a custom model on this class.


is this grid search methodology correct? I have some data say 1 million rows, I then put 200k aside (to validate against) and call this remaining 800,000 the training set (X) as you see below, so it is not the entire data and the remaining 200k is the validation set.
My question is, is the below grid search ok? My roc curves were extremely high say 0.96. I suspect it is because of the content below, should I instead use grid search on x_train and y_train instead then further split x_train and y_train again to use on pipeline.fit()??

X=pd.read_csv('data_subset.csv')

X=X.dropna()
Y=X['status'].values
X=X.drop(columns=['status'])

x_train, x_test, y_train, y_test = train_test_split(
    X, Y, stratify=Y, random_state=42, test_size=0.2)

model = lgb.LGBMClassifier(silent=True, subsample=0.8,colsample_bytree=0.2,objective='binary')

parameters={
    'learning_rate_grid_lgbm': [0.015, 0.001, 0.1],
}

i then run grid search on this data:

clf = GridSearchCV(model, parameters, cv=3, scoring='roc_auc')
clf.fit(X, y) ### i.e. instead clf.fit(x_train,y_train)

model = clf.best_estimator_

pipeline = make_pipeline(model)

pipeline.fit(x_train, y_train)
predictions = pipeline.predict_proba(x_test)[:, 1]
roc_auc_score(y_test, predictions)

my confusion is, if I have to use x_train on gridsearch, how should I proceed with the steps after?

Your hyperparameters are chosen based on the whole set of examples, and thus there is a leakage of information of the test set into the model. The validation will, therefore, be too optimistic (as you suspected).

What could be fixed is instead of:

clf.fit(X, y)

just use

clf.fit(x_train, y_train)

As you suggested yourself already. After that line, nothing needs to be changed. You are fitting the model on the training data, and then testing the performance on the (as of yet unseen) test data.

edit: actually, reading the documentation of GridSearchCV it is even a bit worse (depending on how your pipeline.fit method works under the hood): not only are the model hyperparameters chosen on the whole set, by using model = clf.best_estimator_, also your model is now trained using the training + test set, which is a bad thing of course, and will lead to the roc score on the test data to be much higher than on actual unseen data.


Keras Classifier returns similar output for all Predictions I completed training the model with an accuracy of 1.000 and a validation accuracy of 0.9565. Unfortunately whenever i input a image into my model i get the same output regardless. Am i doing something wrong when predicting or during my training. W and A are my class labels.

My folder structure for the image generators are as follows:

images/

a/
   a001.jpg.png..
 w/
   w002.jpg.png..

train_datagen = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2)

test_datagen = ImageDataGenerator(rescale=1./255)

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(150, 150,3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

batch_size = 64

# this is a generator that will read pictures found in
# subfolers of 'data/train', and indefinitely generate
# batches of augmented image data
train_generator = train_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\train',  # this is the target directory
    target_size=(150, 150),  # all images will be resized to 150x150
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')  # since we use binary_crossentropy loss, we need binary labels

# this is a similar generator, for validation data
validation_generator = test_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\val',
    target_size=(150, 150),
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=2000 // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=800 // batch_size)
model.save_weights('first_try.h5')

img = cv2.imread("C:\\Users\\Zahid\\Desktop\\Data\\TrainingData\\images\\a\\img_0201.jpg.png")
resized_image = cv2.resize(image, (150, 150))
x = img_to_array(resized_image)
x = x.reshape((1,) + x.shape)
x = x/255
print(x.shape)
scores_train = model.predict(x)
print(scores_train)

As you are using sigmoid as activation function in last layer. It will output generate output based on if probability above 50% then it belongs to "W" class and if it is less than 50% belongs to "A" class.
If you can print output probability of different images & share it then it will little helpful for us for understand problem


Matlab simulation through FIS and Markov Process I need to simulate for an academical project how the traffic fluxes (input/output with respect to a monitored area, measured in number of cars) of a city area evolves in correspondence of an event (i.e. the opening of a restricted traffic area to decongest the traffic).
I have some simulated sensors that provide the data: I was thinking to use a combination of a fuzzy system (to assign a membership function to each type of data, e.g. PM10 value and CO2 value) and a markov process: I would need to modify the probability to decrement the number of car in the monitored area (simulating that a car is going out the congested area, towards the new opened area) basing on decisions made by means of a fuzzy system.
So my questions are:

It is a good way to interpret the problem or there are better ideas that I have not taken into account yet?
How to implement such a combination of markov chain and fuzzy systems in matlab?

Thanks

I don't really get why would you mix Fuzziness and Probabilities. HMMs already can give you probabilities without the need of adding Fuzzy systems into the mix.

I would just do a random walk with probabilities of transitions defined by the state of the lights.


Why SVM classifier error is much smaller than KNN classifier error? I have a dataset with a binary classification, on which I train a kNN algorithm and a SVM algorithm.

On the tests set I get approximately 75% error when using kNN, but only approximately a 20% error using an SVM.

What does this tell me about the dataset?

Given that we have not applied any preprocessing steps then the first thing that comes to mind is that SVM works better in higher dimensionalities while kNN suffer from the curse of dimensionality and therefore our dataset should have a large dimensionality especially in comparison to the number of instances. Therefore we have a very sparse space where the points can be separated by a hyperplane while the k-nearest neighbors give little information of the actual class.

Note that we should take under consideration that we have no knowledge if the test size follows the rule of thumb (10%-20%) and we have not repeat the experiment with multiple train-test splits to get a range of possible errors. We might have a very unlucky test set..

Finally we should take into account that we have little knowledge if the classes are imbalanced and how much.
A highly imbalanced class could cause most of classifications to belong to the majority class while SVM would be worse because it would try, in vain, to find a hyperplane that would separate the classes and thus have a larger error.
However here we have the opposite scenario. Therefore we should assume the case where the classes are more or less balanced but they are separable only in one (or a few) dimensions while pretty much mixed in the rest of the dimensions. Since knn does not learn any weights for each dimension, it assumes that the ones closest are the correct points while svm has taken a clear cut decision that anything beyond this line should be classified as such.


Is there an upper bound for k in nearest neighbors-based methods? When applying a nearest neighbors-based method to a data of, for instance, 2000 points, what is the largest number of neighbors that can be considered ?
I am using a nearest neighbors method in an unsupervised fashion for anomaly detection.

The predicted value by the nearest neighbor algorithm will actually just take the average value of all points that are close to the new point.

It's up to you to find the number of neighbors that you want to consider. If you have 2000 data points, this are the 2000 'neighbors' of which you will select a number of k neighbors for taking the average and using that average as prediction.

So the maximum number of k is the number of data points in your training set. In your case this is 2000.


Group_by 2 variables and pivot_wider distribution based on 2 others Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all observations w/ POS Direction in both Drew & A/total sum of all amounts for Drew **

Name     Rating   Amount    Price    Rate   Type    Direction
Drew     A        455       99.54    4.5    white   POS
Drew     A        655       88.44    5.3    white   NEG
Drew     B        454       54.43    3.4    blue    NEU
Drew     B        654       33.54    5.4    blue    POS
Drew     C        754       54.43    4.3    green   POS
Jon      A        454       65.23    3.4    blue    NEG
Jon      B        954       86.34    4.3    blue    NEG
Jon      B        545       34.54    4.4    green   NEG
Jon      C        454       65.45    3.4    green   POS
Jon      C        544       65.55    4.4    blue    NEU
Nick     A        675       54.33    3.4    white   POS
Nick     A        565       65.33    3.4    white   POS
Nick     B        343       54.44    6.4    blue    POS
Nick     C        656       65.33    4.3    green   NEG
Nick     C        655       94.44    3.3    green   NEU

To Get the Following Output Calculation for POS/NEG/NEU columns

Name   Rating   sum  count percent wm_price wm_rate mode_type POS NEG NEU
Drew    A                                                    .3735
Drew    B                                                    .3728
Drew    C
Jon     A
Jon     B
Jon     C
Nick    A
Nick    B
Nick    C

Here's what i got so far but im stuck implementing the pivot_wider for calculating/appending the % direction (POS/NEG/NEU) for each rating category of Each Name Any feedback appreciated!

df <- df %>% group_by(Name, Rating) %>%
  summarize(sum_rating = sum(Amount),
                count = n(),
                wm_Price = weighted.mean(Price,Amount),
                wm_Rate = weighted.mean(Rate,Amount),
                mode_Type = mode(Type)) %>%
  mutate(pct_rating = sum_rating/sum(sum_rating)) %>%
  pivot_wider(names_from = Direction, values_from = Amount/sum Amount)

I cannot get pivot_wider to work here either, but I can replicate your described operation without it.

df %>% group_by(Name) %>%                                      # Group only by Name first
  mutate(Total = sum(Amount)) %>%                              # Total Amount by Name
  group_by(Name, Rating) %>%                                   # Now perform your calculations
  summarize(sum_rating = sum(Amount),
        count = n(),
        wm_Price = weighted.mean(Price, Amount),
        wm_Rate = weighted.mean(Rate, Amount),
        mode_Type = mode(Type),
        POS = sum(Amount * (Direction == "POS")) / max(Total), # This is % POS
        NEU = sum(Amount * (Direction == "NEU")) / max(Total), # This is % NEU
        NEG = sum(Amount * (Direction == "NEG")) / max(Total)  # This is % NEG
        ) %>%
   mutate(pct_rating = sum_rating / sum(sum_rating))

Here is the output

# A tibble: 9 x 11
# Groups:   Name [3]
  Name  Rating sum_rating count wm_Price wm_Rate mode_Type   POS   NEU   NEG
  <fct> <fct>       <int> <int>    <dbl>   <dbl> <fct>     <dbl> <dbl> <dbl>
1 Drew  A            1110     2     93.0    4.97 white     0.153 0     0.220
2 Drew  B            1108     2     42.1    4.58 blue      0.220 0.153 0
3 Drew  C             754     1     54.4    4.3  green     0.254 0     0
4 Jon   A             454     1     65.2    3.4  blue      0     0     0.154
5 Jon   B            1499     2     67.5    4.34 blue      0     0     0.508
6 Jon   C             998     2     65.5    3.95 green     0.154 0.184 0
7 Nick  A            1240     2     59.3    3.4  white     0.428 0     0
8 Nick  B             343     1     54.4    6.4  blue      0.119 0     0
9 Nick  C            1311     2     79.9    3.80 green     0     0.226 0.227
# ... with 1 more variable: pct_rating <dbl>

This method works because r can do math on logical values, treating them as TRUE = 1 and FALSE = 0. Thus, if you want to count the number of occurrences of POS in the Direction column, you could sum a logical vector:

sum(Direction == 'POS')

If you want to know the percent of the rows that contain POS, then use the mean:

mean(Direction == 'POS')

You can even weight that mean by another vector:

weighted.mean(Direction == 'POS', Amount)

The results of my calculation do not match your sample output, so I want to check to make sure I understand what you want the column POS to contain for each row. I understand your desired calculation to be as follows: the Amount of POS for Drew and A (455) divided by the total Amount for Drew (455 + 655 + 454 + 654 + 754 = 2972). Thus, 455 / 2972 = 0.153. I cannot replicate the 0.3735 value you have in your sample output.


Function of functions and theirs arguments I would like to build a pandas random dataframe. To fulfill that purpose I need a Python function taking as arguments :

numpy distributions
their arguments.

For example :

distribution 1 : normal | arguments : means = 0 , standard dev = 1 , size = 100

distribution 2 : uniform | arguments : low = 0 , high = 1 , size = 100

etc...

I do not know in advance what will be the different distributions and their arguments.

The main function will then generate random samples of the distributions using each corresponding arguments.

I have tried something like :

import numpy as np

def myfun( **kwargs ) :
    for k , v in kwargs.items() :
        print( k )
        print( v )

When I call that function with these arguments :

myfun( fun_1 = 'np.random.normal' , arg_1 = { 'loc' : 0 , 'scale' : 1 , 'size' : 7 } ,
       fun_2 = 'np.random.uniform' , arg_2 = { 'low' : 0 , 'high' : 1 , 'size' : 7 } )

The output is :

fun_1
np.random.normal
arg_1
{'loc': 0, 'scale': 1, 'size': 7}
fun_2
np.random.uniform
arg_2
{'low': 0, 'high': 1, 'size': 7}

But my purpose is not to print the desired distributions and their associated parameters but to generate a sample for each distributions.

I assume you know which distributions are possible once you are inside your function? I would probably just pass a parameter for the distribution name and then a set of parameters in a Python dictionary.

This method also allows for pretty easy testing, because you know that distributions expects only certain parameters, so you can also add small checks to confirm everything is available.

def my_distribution(dist_name, dist_params):
    if dist_name == "normal":
        assert ("mean" in params.keys()) and ("variance" in params.keys()), "Missing expected parameters for {} distribution".format(dist_name)
    # Perform some checks for other distributions as necessary...

    # perform your own steps...

Now you use it like this:

# assume these are provided by your earlier code
distr = "normal"
params = {"mean": 5.0, "variance": 2.0}

result = my_distribution(dist_name=distr, dist_params=params)

Edit:

Here is an example of a single function that can handle multiple distributions in a single call:

def my_distributions(dist_collection):
    # Perform some checks for other distributions as necessary...
    allowed_dists = ["normal", "uniform", "dirichlet", "rayleigh"]
    assert all(
        dist_name in allowed_dists for dist_name in dist_collection
    ), "Input contains disallowed distribution"

    # Do something for each distribution name with its parameters:
    for dist_name, dist_params in params.items():
        print(dist_name, dist_params)

Now the input needs to be specified a little differently:

all_distributions = {
    "normal": {"loc": 1, "scale": 2, "size": 100},
    "uniform": {"low": 0, "high": 1, "size": 100},
}

A single call can now work over many distributions:

my_distributions(all_distributions)

You could of course make classes that hold all configurations and checks, but I would argue you are starting to make things unnecessarily complicated, as Numpy does so much for you already with it's built-in distribution capabilities.


How does action get selected in a Policy Gradient Method? As I understood, in Reinforcement-Learning a big difference between a Value-based method and a Policy-gradient method is how the next action is selected.

In Q-learning (Value-based method), each possible action gets a score. We then select next action that has a highest score, with a slight possibility to select any other action at random.

How does a policy gradient method decide upon an action? We are no longer allowed to select the highest-scored action, as this will leave us with a non-differentiable policy, that can't be improved.  The actions still have scores - do we select them at random ...using the "cummulative distribution function"? Also, am I using the right term?

In other words, do actions that have a higher score occupy more space and are more likely to be selected?

Edit: there is a really awesome lecture about policy grads. Unfortunatelly the video is deliberately unlisted, so that normal people can't really get to it - but I am for free education, so here it is:  CS294-112 9/6/17

You are on the right track. We no longer select an action that we think maximizes the score. Rather we predict what the best action to take is. This can be very effective in large or continuous state spaces where taking an argmax of the score over all possible actions would be prohibitively expensive.

Our policy network computes the probability of taking an action, and during training we sample actions from the network. This continues until we receive a reward signal, which we multiply by the predicted log probabilities to propagate the error back through the network over past actions leading up to that reward. This will tend to adjust the past actions' predicted probabilities down for more negative rewards, and up for more positive rewards.

Of course not all actions leading up to the reward necessary lead to that good or bad reward, but over many training iterations good actions will tend to lead to more positive reward and negative actions will lead to more negative reward.

For more discussions see Andre Karpathy's blog post Deep Reinforcement Learning: Pong from Pixels and RL Course by David Silver - Lecture 7: Policy Gradient Methods


Dealing with missing data I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both the columns you find the correlation coefficient to be high. Now for the values that aren't there, can you use linear regression to predict or find them out, by using the values you know as training data?

Hi Soumyadeep and welcome to Data Science/Stack Exchange
What you are describing is called regression imputation, and it is a valid method to use on missing data. However, if the data is sparse (lots of missing values), this issue will be more difficult to handle.
In general, missing data can be handled in several ways (row deletion, imputation, substitution, etc). Regression imputation can be used if you have little or no knowledge about the data, but usually it is better to use another method. If you have some domain knowledge about the missing values, like you have an idea what the value should be, usually you can use that knowledge to fill in the missing values. Try some different methods and see which one works best.

A person pointed out that I should check for multicollinearity if the
features are both independent. Does it basically mean that one feature
is falling in the span of the other feature?

Definition of multicollinearity:
There exist one or more exact linear relationships among some of the variables

References:
https://en.wikipedia.org/wiki/Multicollinearity
https://stats.stackexchange.com/questions/234870/is-multicollinearity-the-issue-here


Create recommendation system to recommend products to a customer on any e-commerce website The recommendations should be based on the products consumer has searched on other sites like Google.

This basically means, that recommendations have to be made to the user based on his/her search history. No other information is available for a user.

Has this sort of thing already been implemented? (preferably in Python)

The approach I have thought of is removing stop words and extracting keywords from a search query, based on some criteria. I am falling short at the implementation.

To answer your 1st question,I am not sure if a recommendation system based on search history has been implemented or not.This approach does sound cool.

Secondly, yes there are various algorithms to extract phrases from text. Phrase extraction and Text Summarization are one of the most important aspects in development of chatbots for conversational AI.
To give you an example, there's a good library for this purpose, called PyTextRank.
Go through this link for better understanding of how this works.

Once you have a ranked summary of phrases for the given user's search history, you will be able to generate the list of item/s , that person is interested in.
This is a good start for building the person's unique feature vector.
After this process, you can incorporate normal recommendation algos like Collaborative Filtering , etc.

Hope this helps :)


Learn (common) grammar / pattern from set of sample strings? So I currently have a text pattern detection challenge to solve at work. I am trying to make an outlier detection algorithm for a database, for string columns.
For example let's say I have the following list of strings:
["abc123", "jkj577", "lkj123", "uio324", "123123"]

I want to develop an algorithm that would detect common patterns in the list of strings, and the indicate which strings are not in this format. For example, in the example above, I would like this algorithm to detect the following regular expression:
r"[a-z]{3}\d{3}"

given that the majority of the entries in the list obey this pattern, except the last one, which should be marked as an outlier.
The first idea that come to my mind was to use a genetic algorithm to find the regular expression pattern, where the fitness function is the number of entries on the list that match the pattern. I haven't worked out the details (crossvers function, etc..), and there is already the difficulty in the sense that the pattern ".*" will match everything, hence will always maximize the fitness function.
Anybody already worked on a similar problem? What are my options here? Thank you!

Here are a few ideas:
If the number of strings is not too high, you could consider taking a formal approach and use a finite automata determinization algorithm (I'm very rusty about this stuff but I clearly remember that there is such a thing). The idea is to start from a big automaton made of the union of all the strings, then use the algorithm to find the deterministic automaton, which can then be converted to a regular expression.
A more data-science-y idea is to use character-based similarity/distance measures between all the pairs of strings. Then it should be possible to identify outliers, maybe through clustering based on the distance. Typical character-based measures: Jaro-Winckler, Levenshtein edit distance.
Finally an original (but possibly bad) idea would be to try to train a (character-based) language model on the strings (assuming there are sufficiently many of them). Given an input string, the language model gives you a probability that this string belongs to the "language", so an outlier could be detected by its low probability.

[addition following OP's comment]
Language modeling is normally used for representing the valid sentences in a language, e.g. English, based on the likelihood of sequences of words in this language. It's trained from a large number of correct sentences so that it can estimate the probability of the $n$-grams of words in this language. This is a common NLP task (example) but in your case you would use characters instead of words and strings instead of sentences, so there would be a small adaptation compared to the examples you'll find.


Imputing features with NA values in classification task I currently have a dataset where each observation is a person's traffic ticket history over districts.
For each column, which represents a district:

1 represents that a person has received 1+ traffic violations in a district in 2018
0 if they have been to that district but have no traffic violations.
NA otherwise

GOAL:  to (rank districts to) see which district should have more police presence due to increased traffic violation and also use the features to predict whether or not that person has 1+ traffic accidents in 2019.
PROBLEM: not all people have been to every district. I currently just encode the value to 0 if the person has never been to that district. But this should be a valid NA value.
For example, it seems illogical to rank a district if only one person (in the dataset) has been to that district.
QUESTION(S): How exactly should I handle this? I don't think imputing as 0 is the right call here.
Original Data:
PersonId DistA DistB DistC DistD DistE Accident19
1        0     1     1     0      NA     1
2        NA    0     0     0      1      0
3        0     1     1     0      NA     1
4        1     0     0     0      NA     0

Imputed Data:
PersonId DistA DistB DistC DistD DistE Accident19
1        0     1     1     0      0     1
2        0     0     0     0      1     0
3        0     1     1     0      0     1
4        1     0     0     0      0     0

Many thanks in advance!

Thank you for clarifying the question @Eisen. So the question looks at two main things:

To show which districts need more police presence.
To classify people as to whether they will commit 1+ traffic violations, regardless of the district, given their previous number of violations, by district.

For the first point, I think what would be a good idea is to yes show the break down of people visiting each of the districts and committing 1+ traffic violations. But, I think adding (95%) confidence intervals would be particularly helpful to see what is a reliable estimate of people who commit 1+ traffic violations in a given district.
In terms of the second point, maybe you can use a feedforward neural network which will take as input the traffic violation categories for each district and output whether the person has committed an accident in 2019 (Accident19).
The architecture is pretty much up to you, but the fail layer would need to be a 2-node softmax layer, which will create the probability distribution over the two classes (has had an accident in 2019 or not).
To represent this categorical data, I would suggest making the NA it's own category. Furthermore, best to represent the district traffic violation categories as one-hot encoded vectors. Then for a particular person, you concatenate these one-hot encoded vectors from all districts in your dataset.


Why is it okay to set the bias vector up with zeros, and not the weight matrices? We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process.
But it is safe to set the bias vector up with zeros, and they are updated accordingly.
Why is it safe to do so, and not the opposite?
Why can’t we initialize bias vectors with random numbers and weight matrices with zeros?
My initial thought is that a vector is of rank (n, 1) where $n \in \mathbb{N}$. This is not true for a matrix. And thus symmetry does not really come into play in the case of vectors.
But that does not answer the question that each layer of a deep neural network has its own weight matrix, and there is no need for symmetry across different layers.
So, when we talk about symmetry are we talking about symmetry across different rows of the same matrix?
Column wise symmetry should not matter much as they are for different training examples (for the first hidden layer). Does column-wise symmetry disturb the training process much in the case of hidden layers other than the first one?

As per Efficient Backprop from Lecun (§4.6) weight should be initialized in the linear region of the activation function. If they are too big, activation function will saturate and provide small gradient step to change those weigth. If they are too small they won't really impact the gradient and make the learning too slow.
Yes, if you choose the same weights this will create an artificial symetry that can be problematic. Here 'symmetry' is about neurons of the same layer having the same initial weights, thus being redundant. I think it would be clearer to speak about redundancy than symmetry. This will translate into redundant lines in weight matrices. Of course if all your weights are set to zero, all your weight matrices lines will be the same and you'll have horizontal symmetries in all your weight matrices.
Naturally you want to avoid redundancies in your model, but this is not the main problem. Your main problem is about solving an optimisation problem efficiently, i.e. having a gradient that is sufficiently big relative to your weights to help fast convergence. That's why you set your weights to small but not too small values. The randomness helps to avoid redundancies.
Once you have set your weights to random values, you have some minimal garantee you are in the linear region of the activation function, so you don't really need to add bias.


Can the extent of variability within a dataset be reflected through clustering? As an example: I need to compare the extent of variability amongst houses belonging to 4 different architectural eras - I want to see how different the houses are within each group and then compare such variability between groups (example of hypothetical conclusion: the houses from the baroque era had the most variability when compared to the other 3 groups).
Different descriptive variables on all houses are available (area, number of rooms, number of floors etc).
I wish to use a technique which can take into account all variables.
Would it make sense to perform clustering within each individual group, on the basis that the higher the optimal number of clusters, the larger the extent of variability between houses within each group? Would hierarchical clustering fit?
If not, what would you suggest please?

I think it would make sense to compare the distribution (and means) of standard deviations for each variable between the groups. So the mean of the standard deviation for each era would be something like:
$
\overline{\sigma}_{Era} = \frac{\sum_{i = (features)}  \sigma_{i}}{n}
$
where $n$ is the number of features for each era, and $ \sigma_{i} $ is the standard deviation for one feature of the era.
I think interpreting the results of this would be more straight forward and would be simple to compute. It would also be easy to visualize, as you could just make a box plot for the feature standard deviations for each architectural era and you'd be able to easily see which era has the greatest variance.


How to reduce overfitting in a pre-trained network I have a custom dataset with 10 classes and I am using a pre-trained resnet18 model from torch-vision. I can clearly see it's over-fitting because: the model is trained for 75 epochs with a batch size of 4 and from epoch 30 the validation accuracy stops increasing and the training accuracy keeps increasing.
Things I did to improve generalisation performance and reduce overfitting:

I normalized my data with a calculated mean and std on my training data.
I added random rotation, random horizontal flip and random vertical flip as data augmentations.

The above were the results after all of these, I can't think of other regularization techniques that produce regularization effects without changing the architecture of the model which will completely contradict the purpose of a pre-trained network, and I am using one because I only have around 300 images in my each class.

In transfer learning there are two parameters which influence the basic setup to go for:

Size of new dataset
Similarity of new dataset to dataset of pre-trained model

When your dataset is small the problem is that high capacity pre-trained models can easily overfit if you re-train too many layers. And since you re-trained multiple layers this could be an issue here.
Instead, try the following two options:

Re-train only the last fully connected layer. This is the general approach if your data is similar to the data which the pre-trained model was trained on.
Remove the conv. layers towards the end of the pre-trained model and re-train only the new fully connected layer. For ResNet18 you could try tossing conv4 and 5, for example. This is the general approach in transfer learning for small datasets if your data is not very similar to the original data and you only want to use layers for lower level features since the datasets do not have similar higher level features.

If you are interested in a theoretical reading on the topic, the paper "How transferable are features in deep neural networks?" could be of interest.


Is it appropriate to run correlation on continuous y value with discrete x value? I am dealing with temperature data that are measured in different distances away from the forest edge (0m, 10m,30m,50m,80m) and I would like to find out whether the temperature will drop as the distance away from the edge increases. I tried to use geom_point to plot the temperature with respect to the distance, however, the x axis value are not labelled as 0m, 10m,30m,50m and 80m, may I ask how to change it?

Also I would like to ask is it appropriate to run correlation using temperature and distance from the edge to show that they are negatively correlated in this situation? I saw correlation should be based on two continuous variable, but the distances from the edge are discrete variable, so should I just present the result using boxplot? Or is computing regression appropriate in this case?

Your feature "Distance from the edge" is represented as a factor (), which means it is interpreted as a categorical variable. You need to make sure that it is interpreted as a numerical (either integer or double) and this should label it appropriately in your plot and also correctly display the distance between them (as they are not evenly spaced).
This might help to get the type right.

Check this answer for a discussion on what correlation methods to use for continuous and discrete variables.


Confused about false positive and false negative in confusion matrix? I am working on binary classification for classifying cancer=1 and no-cancer=0, I use confusion matrix from sklearn, this is my confusion matrix on test set:

 # confusion matrix
 [[18  0]
 [ 7 15]]

 # in my reading the order is:
   TN=18
   FP=0
   FN=7
   TP=15

but in some tutorials, I see different ordering for FP and FN, some said same as my reading,See here, but other said reverse of FP and FN, see here.
my question is which one is true in my case? please give me a reference to be sure about the answer.

Think about the order of your test and prediction sets when constructing the confusion matrix. Here is a piece from one of my codes.

cm = confusion_matrix(y_test, y_pred)
print(cm)
Output:
[[TN FP]
[ FN TP]]

However, if I used:

cm = confusion_matrix(y_pred, y_test)
print(cm)
Output:
[[TN FN]
[ FP TP]]

That occurs since the predictions of the model are presented in the rows instead of columns now. Also confusion matrices can be NxN, or the classes may not be labeled as either 0 or 1. You can also change the places of TN and TP, think what should happen if you had classes named as 9 and 10. In other words, decision of Negative/Positive signs are our decision; we say what they are (hopefully in a reasonable way).
Hope I could help, please do not hesitate to ask more.
Regards.


Why doesn't loss go down during Neural Net training? I am working on a Kaggle competition and have tried 2 different code approaches and have the same issue: the loss is large (18247478709991652.0000) and does not go down or is nan.

I'm not sure if there is something wrong with the code or with the data. I tried both scaled and non-scaled data and got the same results. I tried it with the full data set (3,000 examples) and an abbreviated data set.

Here is the abbreviated data.

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

dataframe = pandas.read_csv('data/tmdb/train_processed.csv')
dataframe.drop('id', axis=1, inplace=True)

Y = dataframe['revenue'].values
dataframe.drop(columns=['revenue'], inplace=True)
X = dataframe.values

def baseline_model():
  model = Sequential()
  model.add(Dense(13, input_dim=3, kernel_initializer='normal', activation='relu'))
  model.add(Dense(1, kernel_initializer='normal'))
  model.compile(loss='mean_squared_error', optimizer='adam')
  return model

seed = 7
numpy.random.seed(seed)

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Result: %.2f (%.2f) MSE" % (results.mean(), results.std()))

Your loss does go down, but not significantly. This is because your target values are very large ~10e7 and the default learning rate is scaled for smaller values. The easiest way to fix this is to normalize Y.

If your intention is for your code scale Y, then the problem is that Pipeline does not apply StandardScaler (or any transformations) to Y. You have to use sklearn.compose.TransformedTargetRegressor, or apply the transforms to Y outside of Pipeline.

Pick 1:

Outside the pipeline:
Y = dataframe['revenue'].values
Y = StandardScaler().fit_transform(dataframe['revenue'].values.reshape(-1,1))

Inside the pipeline:
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))

from sklearn.compose import TransformedTargetRegressor
estimators.append(('mlp',TransformedTargetRegressor(
regressor=KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1),transformer=StandardScaler())))


Ordered and unordered categorical features – terminology In the famous book "The Elements of Statistical Learning" by Hastie et al., the authors denoted unordered categorical variables as qualitative variables / nominal variables / factors.

I wonder, do other statisticians strictly follow this or some authors can use these terms (qualitative variables / nominal variables / factors) not only for unordered categorical variables but also for ordered categorical variables?

The statistical programming language R uses the term "ordered factor," so factor isn't completely safe, though I can't find an example of an ordinal variable being called "factor" without the adjective.

I think ordinal variables are fairly often considered either qualitative or quantitative: they have the quantitative feel from the ordering, but lack mathematical operations.  See the end of this post for a few examples.

But "nominal" seems relatively safe as meaning only unordered.  (I found one contradiction to this, the Bonus Note #2 at https://www.mymarketresearchmethods.com/types-of-data-nominal-ordinal-interval-ratio/, but that's immediately contradicted by having "ordinal" in the next section?)

See also https://en.wikipedia.org/wiki/Level_of_measurement, especially the "Debate" section that lists a few other proposals.  (Chrisman's proposal is nice for including "cyclic" features that are sometimes important in ML but don't fit into most standard libraries without some [unfaithful] encoding.)

A few links to show that the lines get blurred:
https://stats.stackexchange.com/q/159902/232706
https://www.mymarketresearchmethods.com/data-types-in-statistics/
https://stats.stackexchange.com/a/158226/232706
https://web.ma.utexas.edu/users/mks/statmistakes/ordinal.html


Perceptron algorithm I am trying to build a simple linear classifier. I have two classes A and B each with two features [x, y] and hence a 2d dataset. Now, I need to find the equation of the line that separates the two classes.

y = $\vec w \vec x - b$

I know exactly what $\vec w$ is. $\vec w$ is vector that is perpendicular to the line that separates the two classes, and b is the bias that shifts the line along $\vec w$. When b is 0, line is exactly at the origin.

I created the algorithm exactly as mentioned in this link.
https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm

But, in some cases it doesn't work properly. I've understood intuitively what the algorithm does. At each step, it takes a misclassified point, and tries to make correct classification by rotating the current line segment, and shifting it so that point is more closer to correct class.

I need an algorithm that helps me find $ \vec w $ and $ b $.

Here's my algorithm in ES6.

const learn = (target, learning_rate = .00001) => {
const { featureVector } = target;
// shift towards object
// const nm = math.norm(lineConfig.normal);
// lineConfig.normal = math.divide(lineConfig.normal, nm);
// lineConfig.bias /= nm;
const observedFeatureVector = featureVector.concat(1);
const wc = lineConfig.normal.concat(-lineConfig.bias);
const boundaryValue = math.dot(wc, observedFeatureVector);
const curOutput = boundaryValue >= 0 ? 1 : -1;
// lineConfig.bias += learning_rate * -target.cls;
// [w -b] [x, 1]
const dv = math.multiply(observedFeatureVector, learning_rate * (target.cls - curOutput));
const w = math.add(wc, dv);
lineConfig.bias = -w.pop();
lineConfig.normal = w;

};

I transform my input 2 dimensional data set into 3d by adding 1 [x 1] is our input vector. Then, all we need to do is to find $ \vec w $ only since bias is automatically adjusted. For every misclassification, I add the training point vector to the $ \vec w $ so that classification moves closer to the goal.

My training set is this:

[200, 200, 1]
[227, 347, 1]
[399, 237, 1]
[358, 344, 1]
[265, 263, 1]
[60, 20, -1]
[26, 100, -1]
[126, 95, -1]
[145, 58, -1]
[51, 141, -1]
[49, 186, -1]

But, in some cases it doesn't work properly.

As you did not provide your data, I can only make a bold assumption.
The perceptron can only classify linear separably data and will not converge for not-linearly separable data. Thus, this could be a reason for the unintended behaviour.

Cheers!


Changing duplicate a line hotkey in Jupyter notebook To duplicate a line in Jupyter-notebook on a Mac, the hotkey is cmd + shift + D. The problem is I use Google Chrome, and this is the same hotkey to bookmark a webpage, which takes priority over the hotkey in Jupyter-notebook. Has anyone struggled with this before, and found a workaround? I would probably prefer to use this hotkey for duplicating a line because it's the same hotkey I use in IDEs and text editors.

You can change the Chrome shortcut like this:
In System Preferences > Keyboard > Shortcuts > App Shortcuts, click the [+] button, and select Google Chrome as the Application. Put Bookmarks->Bookmark All Tabs... in the Menu Title and a different Keyboard Shortcut.


r2 for regression models it is a score or error? some places I have seen it is called as score and some other place as error. Lets suppose r2=0.83 means that score = 83% and Error= 17% or vise versa

R2 of 0.83 means that approx 83% of the variance can be explained by the model, 17% of the variance can't be explained by the model.

This is one of the measures to evaluate a model, but you shouldn't try and maximize any number, just for the sake of maximizing the number. An high R2 could be a sign of over fitting, and a model with low R2 could still be useful.

But in a simplistic way, yes they could be seen as "score" and "error". But the implementations of the number can vary.


RNN vs CNN at a high level I've been thinking about the Recurrent Neural Networks (RNN) and their varieties and Convolutional Neural Networks (CNN) and their varieties.

Would these two points be fair to say:

Use CNNs to break a component (such as an image) into subcomponents (such as an object in an image, such as the outline of the object in the image, etc.)
Use RNNs to create combinations of subcomponents (image captioning, text generation, language translation, etc.)

I would appreciate if anyone wants to point out any inaccuracies in these statements.  My goal here is to get a more clearer foundation on the uses of CNNs and RNNs.

A CNN will learn to recognize patterns across space.  So, as you say, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and then learn to combine these components to recognize larger structures (e.g., faces, objects, etc.).

You could say, in a very general way, that a RNN will similarly learn to recognize patterns across time.  So a RNN that is trained to translate text might learn that "dog" should be translated differently if preceded by the word "hot".

The mechanism by which the two kinds of NNs represent these patterns is different, however.  In the case of a CNN, you are looking for the same patterns on all the different subfields of the image.  In the case of a RNN you are (in the simplest case) feeding the hidden layers from the previous step as an additional input into the next step.  While the RNN builds up memory in this process, it is not looking for the same patterns over different slices of time in the same way that a CNN is looking for the same patterns over different regions of space.

I should also note that when I say "time" and "space" here, it shouldn't be taken too literally.  You could run a RNN on a single image for image captioning, for instance, and the meaning of "time" would simply be the order in which different parts of the image are processed.  So objects initially processed will inform the captioning of later objects processed.


Pandas pivot table, creating ad hoc columns per dimension values I'm new to pivot tables and have the following dataset:

mydict = {'City' : ['Lexington', 'Lexington', 'Louisville', 'Hartford', 'Portland', 'Dallas'],
          'State': ['KY', 'KY', 'KY', 'CT', 'ME', 'TX'],
          'Zip': ['38293', '38293', '40207', '48488', '55849', '44930'],
          'Region': ['South', 'South', 'South', 'Northeast', 'Northeast', 'South'],
          'Sales': [1000, 2000, 3000, 1500, 2000, 2300],
          'Product Type': ['Industrial', 'Consumer', 'Consumer', 'Educational', 'Educational', 'Scientific']}
dftest = pd.DataFrame(mydict)
mypivot = pd.pivot_table(dftest, values=['Sales'], index=['State', 'City'])

This creates a single column for Sales.

                   Sales
State City
CT    Hartford     1500
KY    Lexington    1500
      Louisville   3000
ME    Portland     2000
TX    Dallas       2300

But what I want is e.g. two columns under Sales, corresponding to an ad hoc list of my Product Types for example ('Industrial', 'Consumer').

Like this:

                  Sales
State City        Industrial   Consumer
CT    Hartford     0           0
KY    Lexington    1000        2000
      Louisville   0           3000
ME    Portland     0           0
TX    Dallas       0           0

Is this possible using pivot tables?  Or do I have to manually build up such a dataframe somehow (something that I think would result in excessively complex code)?

EDIT:

I see now that mypivot.columns returns a MultiIndex.  I have heard of these but don't yet know how to manipulate them.  I sense that the solution to the problem lies in how to specify a MultiIndex filter.

The solutions seems to be fairly straight forward. The only thing that is missing in your pivot is, what are the columns you want to put on top to access the pivot. In your case instead of using

mypivot = pd.pivot_table(dftest, values=['Sales'], index=['State', 'City'])

which produces

                  Sales
State City
CT    Hartford     1500
KY    Lexington    1500
      Louisville   3000
ME    Portland     2000
TX    Dallas       2300

you can write

mypivot = pd.pivot_table(dftest, values=['Sales'], index=['State', 'City'], columns=['Product Type'])

which gives you

                    Sales
Product Type     Consumer Educational Industrial Scientific
State City
CT    Hartford        NaN      1500.0        NaN        NaN
KY    Lexington    2000.0         NaN     1000.0        NaN
      Louisville   3000.0         NaN        NaN        NaN
ME    Portland        NaN      2000.0        NaN        NaN
TX    Dallas          NaN         NaN        NaN     2300.0

You can now use .fillna method on chaining to replace non-zero values in your pivot. Hope this helps.

Edit:
If you want to take only a specific set of columns, I can seem to have a not so elegant but workable solution like

mypivot['Sales'][['Consumer', 'Educational']]

Which unfortunately removes the higher column Sales from picture producing something like

Product Type      Consumer  Educational
State City
CT    Hartford         0.0       1500.0
KY    Lexington     2000.0          0.0
      Louisville    3000.0          0.0
ME    Portland         0.0       2000.0
TX    Dallas           0.0          0.0

If I can think of anything, I will add later.


Estimating a metric of a mixture I'm looking for a general approach to solve a problem that is best demonstrated by the following example.

You mix six different batches of orange juice and you know the acidity, the sugar content, the concentration of volatile particles in each batch of the juice, the growing location of the fruits, and other data. You mix those batches, sell the product, and measure how well it was accepted by the market. Tomorrow, you decide to mix four other batches, the next day you will mix two batches, etc: the number of components in your product isn't constant. The goal is to take the information about each component and to predict the outcome metric.

If we could compute the average value of each input parameter, I would start by computing the average value of each parameter in the mix, and trying to use it as the X data in an ML algorithm. However, some of the parameters cannot be added. In our example, such a parameter can be the plantation age, plantation location, cultivar of the fruit, etc

What are the approaches to solve this problem? Does it belong to a certain studies problem class ?

However, some of the parameters cannot be added. In our example, such a parameter can be the plantation age, plantation location, cultivar of the fruit, etc

Can you explain why these can't be added?

There exist many ways to encode categorical information, such as plantation location, such that a ML model can interpret it. In my use-cases, in supply-chain, I deal with thousands of unique location observations, where in I use a geo-encoding API to return latitude/longitude pairs to feed into my models.

In your case, where in you only have 6-factorial observations (that may be wrong), what I would do is to generally categorize your categorical features. So for location, instead of "Florida," I would say "South-East," and then lump all south-eastern locations together. Since you'll have low cardinality in feature size, binary or one-hot encoding may be useful for you.


Very long sequence in neural networks Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like:
X = very long sequence of varying length.
Y = class (assume for simplicity y=0/1).
The class depends on the whole sequence - it is impossible to determine the class without seeing all of it.
How NN handle this? I have read various methods, such as truncating the sequence (which is a problem - there is no class for truncated part of the sequence, so I cannot label it), using encoder/decoder lstm (again, couldn't completely understand how it solves the problem), etc.

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem.
But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do :

Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the Input layer : the 0s will not be taken into account for training the model. Bellow an example using the Keras framework (not complete, not tested) :

from keras.models import Model
from keras.layers import Input, LSTM, Dense
input = Input(shape=[None, nb_of_features_per_timestep])
layer = Masking(mask_value=0)(input)
layer = LSTM(50, return_sequences=False)(layer)
output = Dense(nb_classe)(layer)
model = Model(input,output)

You can also use the bucketing trick : you group your dataset into "buckets" of same size (approximately). You can then create batches of same size. You don't need the Masking layer anymore.
You can also use fully convolutional network. Those networks can also take Input with different size. I let you search if you want to use that.

Concerning encoder/decoder : those model can maybe help you but I think that you will have to use one of the tricks that I mentioned.
If you don't understand how RNNs work, I recommend you to dig a little into the subject before.


Tensorflow take ages for tf.cond and eval() - python code (sorry but i asked on Stackoverflow but none answer me) I got a problem with TensorFlow and need your help.
My need is calculating tensordot between a vector: 1x512 named face in my code and a faces data: N x 512 named input_faces_data. The code will return the max_value_index if the max_value >= 0.1.
I printed out time stamp to timing each step of function i use:
tf.tensordot()
tf.math.argmax()
tf.cond()
tf.Session()
.eval() -> return last value

My questions:

Why tf.tensordot() and tf.math.argmax() take just 1ms or 5ms with any length of faces data array (3.000 or 1.000.0000 - my examples) but time cost a lot with .eval and tf.cond()?
Why the duration of tf.cond() and .eval() is more longer with longer face data array?

I'm using TensorFlow 1.13.1 and my GPU is GTX 2080 (11GiB).
My Python code:
sess = tf.Session()
with tf.device(Config.GPU.GPU_DEVICE):
    start = time.time()
    dot_array = tf.tensordot(input_faces_data, face, axes=1)
    print("Data length {}".format(len(faces_data)))
    print("Compatition time {}".format(time.time()-start))
    start_max_index = time.time()
    max_index = tf.math.argmax(dot_array)
    print("get max_value_index time {}".format(time.time()-start_max_index))
    start_condition_time = time.time()
    new_max_index = tf.cond(dot_array[max_index] > tf.constant(0.1),
                            lambda: max_index,lambda: tf.constant(-1,dtype=tf.int64))
    print("tf.cond time {}".format(time.time()-start_condition_time))
    temp_max_index = -1
    start_seesion = time.time()
    with sess:
        print("Session time {}".format(time.time() - start_seesion))
        start_eval_time = time.time()
        temp_max_index = new_max_index.eval()
        print("Eval time {}".format(time.time() - start_eval_time))
    print("Total time {}, max_index {}".format(time.time()-start,temp_max_index))

And my outputs:

The reason why tf.tensordot(), tf.math.argmax() doesn't change running regardless of the input size is because they don't make the operations, but rather declare it.
It is similar to passing a partial function using functools.partial, the function is not really executed, it just determines it's inputs (or part of it).
from functools import partial
def tensordot(a,b):
    return partial(lambda a,b: a*b,a,b)

While running eval will run the previously mounted model
dot = tensordot(a,b) #dot here is a function
dot() # <- done inside eval

Actually, on tensorflow these "partials" are objects and when you eval one of then, the previously declared objects that it depends of will be evaluated


Scaling of the Streaming Data I have to make prediction on streaming data for which I have trained the model offline.

Is it a common practice to use scaling when we have to train the model offline and have to predict the on streaming data, also how to scale the streaming data?

Scaling of streaming data must be based on your training set parameters. Your model has learnt based on that scaling, therefore, in order to assess its generalization power, you have to repeat the same scaling that your model learned from.


Data Science Take Home Challenge Interpretation of Question I am currently applying for a Data Science position and have to finish a take-home challenge for one of the companies. However, I don't really understand what they want me to do and hope you can help me interpret the question. Unfortunately, I can't reach anyone at the company due to Christmas holidays

I have a data set with transactions of customers and another dataset with information about age, country... of those customers. Now I am asked to define a target metric for engagement and say how I would define unengaged vs. engaged customer. Afterwards, I have to use this logic to build a model to classify engaged vs unengaged customers (they point out overfitting due to feature used in the target metric).

How would you interpret these questions?

Do I first have to say if target_metric > x: engaged and then build a supervised model to classify the users again?

OR

Would you define several metrics that could separate engaged and unengaged customers and then use an unsupervised model to find two clusters that would then be engaged vs unengaged?

I tried to keep the question as general as possible to avoid "cheating". I only want to know how you would interpret them. Thanks!

Second one but dont stop there.

First of all you specifying if target_metric > x: engaged is just wrong and you have to let data tell you this x, not you choosing it.
Second one seems nice I would start with that, do some extensive analysis and only then start finding x empirically. How? just of the bat you can pose it as minimization problem for different x and find the one that has biggest purity/separation between classes.

Why dont stop there? There is no wright and wrong, if you have some plausible approach in theory but it fails in practice I would definitely
include it and elaborate on it, discuss it etc...


Keras model to focus on different metrics? When using the model.compile() attribute, does it matter what metric I place in there? For example, would model.complie(metric='accuracy') focus on increasing accuracy vs model.compile(metrics=[tf.keras.metrics.Precision()]) focusing on Precision? Or do these just state what shows during the epoch training?

Here are the docs for the Model.compile method.
The loss parameter is the objective function whereas the metrics parameter is a "list of metrics to be evaluated by the model during training and testing".
So your second interpretation is correct.


Data store for testing data products? Is there a recommended approach for storing processed data for testing new data products?

Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product to present to users, do the data processing to create it, and then put it in a data store that our application can then access easily.

What I'm not sure about is what kind of data store would be good for this type of "testing" use case.  Since it would need to be flexible enough to handle different types of data products, like aggregates, windowed data, etc.  And ideally it wouldn't require a huge instrumentation process to try out new things.

You might try Azure Table Storage.  Since you can't lock yourself down to a specific schema (since one data product might be aggregates whereas another might be time series or something else), Azure Table storage would give you the flexibility of storing data from multiple sources, each having their own format.

This would also lend itself to making a system highly scalable, as you could use Azure Service Bus in conjunction with Azure Table Storage.

You might check out this tutorial at Pluralsight, Applied Windows Azure, as it shows a number of examples, one using Table Storage and Service Bus, another using Hadoop, and I suspect that some of these might match the extensibility you are looking for.


When to Perform non-linear dimension reduction When do we feel need to go through non-linear transformation like kernel PCA ? Please share an example

The question is too general to have a specific recipe as an answer. Depends on the data and the relation between features but to the best of my knowledge:

When the relation between your features are non-linear i.e. when a non-linear interpretation of correlation between features makes more sense than the linear interpretation.

A detailed explanation of what I mean can be found in this nice answer. Please note that the $X^TX$ in the 4th line of the first paragraph should be $XX^T$, because in data mining, we conventionally use $N\times D$ format for data matrices. So in this example, we are looking for the gram matrix of the rows and not columns.

Hope it helps :)


How can we effectively measure the impact of our data decisions Apologies if this is very broad question, what I would like to know is how effective is A/B testing (or other methods) of effectively measuring the effects of a design decision.

For instance we can analyse user interactions or click results, purchase/ browse decisions and then modify/tailor the results presented to the user.

We could then test the effectiveness of this design change by subjecting 10% of users to the alternative model randomly but then how objective is this?

How do we avoid influencing the user by the model change, for instance we could decided that search queries for 'David Beckham' are probably about football so search results become biased towards this but we could equally say that his lifestyle is just as relevant but this never makes it into the top 10 results that are returned.

I am curious how this is dealt with and how to measure this effectively.

My thoughts are that you could be in danger of pushing a model that you think is correct and the user obliges and this becomes a self-fulfilling prophecy.

I've read an article on this: http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/ and also the book: http://shop.oreilly.com/product/0636920028529.do which discussed this so it piqued my interest.

In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site.  This creates independent samples drawn from the same population.  Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision.

Slight aside: Now you might argue that the A group or B group may differ in some demographic.  That commonly happens by random chance.  To a certain degree this can be taken care of by covariate adjusted randomization.  It can also be taken care of by adding covariates to the model that tests the effect of the design decision.  It should be noted that there is still some discussion about the proper way to do this within the statistics community.  Essentially A/B testing is an application of a Randomized Control Trial to website design.  Some people disagree with adding covariates to the test.  Others, such as Frank Harrel (see Regression Modeling Strategies) argue for the use of covariates in such models.

I would offer the following suggestions:

Design the study in advance so as to take care of as much sources of bias and variation as possible.
Let the data speak for itself.  As you get more data (like about searches for David Beckham), let it dominate your assumptions about how the data should be (as how the posterior dominates the prior in Bayesian analysis when the sample size becomes large).
Make sure your data matches the assumptions of the model.


Time series - is it necessary to retrain the model when new time series data is present Say you're building a sales prediction model to predict tomorrow's sales value, as well as the next 2 weeks of daily sales. The model is being trained using daily data for the previous 1.5 years, and it follows a strong weekly seasonality pattern.

Obviously once you are happy with model performance, would it be necessary to re-train the model everyday to capture data up to and including yesterday in order to get the most accurate prediction for tomorrow's sales value? Essentially, the model would be trained on a rolling 1.5 year data set to capture yesterday's sales value.

Or does it depend on the type of model being used? Whether you need to to completely re-train when new data is present

I can see why it would make sense to re-train your standard time series forecasting models (ARIMA etc.), but I can also understand more sophisticated models (Neural Networks etc.) may generalize well enough to not have to re-train everyday.

I am looking for an explanation of models where you would and wouldn't re-train when new time series data is present.

It is good practice to retrain model once in a while. That "in a while" depends on the specific task, of course. This is due to the fact that countless unmeasurable things change continuously.

Think for example about a model to forecast companies' revenues: the rules of the game change continuously, the market today is not like the market one year ago, not to mention five years ago. This is the first example that I thought about, but it's true for pretty much any possible application of ML.

That is why it was observed that models' performance deteriorates in time. For that reason, it is good practice to keep your models updated with fresh data.


No accuracy in Keras RNN Model with Bitcoin Data I am very new to machine-learning and have made an RNN-LSTM model with no accuracy. My data has been normalized with MinMaxScaler from Sklearn and has a shape of has an input of shape (3, 2)...

My normalization steps:

def get_data(currency):
    url=f'https://coinmarketcap.com/currencies/{currency}/historical-data/?start=20130428&end={time.strftime("%Y%m%d")}'
    data=pd.read_html(url, flavor='html5lib')[0]

    data=data.assign(Date=pd.to_datetime(data['Date']))
    data['Volume']=(pd.to_numeric(data['Volume'], errors='coerce').fillna(0))

    data.columns=[col.lower() for col in data.columns]
    data.columns=[col.strip('*') for col in data.columns]

    return data

df=get_data('bitcoin')
df=df.sort_values(by='date')

def split_data (data, trainsize):
    return np.array(data[:int(trainsize*len(data))]), np.array(data[int(trainsize*len(data)):])

scaler=MinMaxScaler(feature_range=(0,1))

def create_inputs(data, window):
    inputs=[]
    for i in range(len(data)-window):
        inputs.append(data[i:(i + window)].values)

    close,volume=[],[]
    for x in range(len(inputs)):
        close.append(inputs[x][:,0])
        volume.append(inputs[x][:,1])

    close=np.array(close)
    close=scaler.fit_transform(close)

    volume=np.array(volume)
    volume=scaler.fit_transform(volume)

    inputs=[]
    for i in range(len(close)):
        rows=[]
        for x in range(len(close[i])):
            row=[close[i][x], volume[i][x]]
            rows.append(row)
        inputs.append([rows])
    inputs=np.vstack(inputs)
    return inputs

def create_outputs(data, window):
    return scaler.fit_transform(data['close'][window:].values.reshape(-1,1))

# VARIABLES
df=df.filter(['date', 'close', 'volume'], axis=1)
df=df.sort_values(by='date')
df[df.columns] = df[df.columns].apply(pd.to_numeric, errors='coerce')

train,test=split_data(df,0.8)

train=pd.DataFrame(train, columns=df.columns)
test=pd.DataFrame(test, columns=df.columns)
train=train.drop('date',1)
test=test.drop('date',1)

xtrain,ytrain=create_inputs(train, 3), create_outputs(train, 3)
xtest,ytest=create_inputs(test, 3), create_outputs(test, 3)

Here is part of my training data (1607, 3, 2) fetched from CoinMarketCap's Bitcoin History after scaling:

    [[[0.01363717 0.        ]
      [0.01577874 0.        ]
      [0.01463021 0.        ]]

     [[0.01577874 0.        ]
      [0.01463021 0.        ]
      [0.01006721 0.        ]]

     [[0.01463021 0.        ]
      [0.01006721 0.        ]
      [0.00762504 0.        ]]...]

My model has 3 Layers with 1024 LSTM Cells, and Dense layer with 1 neuron:

model = keras.models.Sequential()

model.add(keras.layers.CuDNNLSTM(1024, input_shape=(3,2), return_sequences=True, name='input'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.CuDNNLSTM(1024, return_sequences=True, name='lstm1'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.CuDNNLSTM(1024, name='lstm2'))
model.add(keras.layers.Dropout(0.2))

model.add(keras.layers.Dense(1, activation='tanh', name='output'))

# Compile model
model.compile(
    loss='mse',
    optimizer='adam',
    metrics=['accuracy'],
)

history=model.fit(xtrain, ytrain, batch_size=64, epochs=1000, validation_data=(xtest, ytest), verbose=1)

Remember, accuracy is a classification measure. That is, it's used to evaluate models that attempt to predict membership in one of a few discrete values. No accuracy in this case means that you haven't predicted and values exactly correctly- a pretty common occurance in regression problems- that is, problems that measure the scale of a phenomenon or "how much" of something happens.

A better metric to evaluate your regression model might be MSE, such as used for your function.

Try:

    # Compile model
model.compile(
    loss='mse',
    optimizer='adam',
    metrics=['mse'],
)


Where are WEKA installed packages stored I would like to use the WEKA library in a Java program but I can't seem to find the methods I installed using WEKA's package manager.

Does anyone know where the installed methods are stored?

For clarification, I installed WEKA, installed the extra package using WEKA and can use it with the WEKA GUI. But I can't find the installed package in the .lib file.

I don't know how WEKA stores the extra packages on other installations, but for my Windows 10 with WEKA 3.8 it was here:

WEKA stores the extra packages in a direcory called packages. This package was in my user directory and under the wekafiles folder.

Example: "C:\Users\myUser\wekafiles\packages....."

Hope this helps someone else.


How to forecast labels forward 3 years I have many variables that I am using to predict median gross rent. I want to predict forward 3 years this is what my labels look like:

How do I either preprocess the data in order to get a 3-year estimate or what model can I use to predict forward 3 years?

You dont specify explicitly how far in the future you want to make your predictions, but rather you do that implicitly when you make sure your model is robust and does not over-fit.

What does that mean?

Make sure that distribution of labels with your available independent varaibles has similiar distributions of that what you expect in future. You cant expect your model to learn patterns that were not there in the first place. So variables that show same information for distinct median gross rent values 3 years in the future are what you want to include.


How do I make a labelled line chart? suppose I want to track the ratings of seasons of a TV show. so there are three columns of data-season number, year, and ratings. I want the year and ratings on the axes and season number as labels on the points. how can this be achieved considering visualization tools only take in values for X and Y axes?

Have a look at matplotlib's annotate.

Here are some examples. I believe it works well with seaborn too.


I want to run PCA on a data set that will be aggregated by country. Should I aggregate the data before or after I standardize the data, and why? Basically the title asks my question. I have the results of a survey that was filled out by people from different countries. I have been asked to analyze the data using PCA and see what findings I can get on each country from my analysis. Now of course this means I have to aggregate my data, but should I aggregate the data before or after I standardize the data? From the results that I am getting it seems that aggregating first is the right way to go about it, but I am failing to understand why that would be the case.

Data should be standardized for calculating principal components. If data is standardized before aggregating by country, the resultant data after aggregation will no longer will have mean 0 and standard deviation 1, hence, the results of PCA will be erroneous.
Therefore, data should be aggregated and then standardized in order to obtain correct principal components of the data.


Adding a column to a dataframe in pandas using another Column So I have a column called "plot" in a dataframe and i want to create a new one called "keywords" which only has the important words of plot.
here is the code:

 import pandas as pd
 import numpy as np
 from sklearn.metrics.pairwise import cosine_similarity
 from sklearn.feature_extraction.text import CountVectorizer
 import re
 import nltk
 nltk.download('stopwords')
 from nltk.corpus import stopwords
 df = pd.read_csv('IMDB_Top250Engmovies2_OMDB_Detailed.csv')
 df = df[['Title','Genre','Director','Actors','Plot']]
 df['Keywords'] = ''

 for index,row in df.iterrows():
     plot = row['Plot']
     plot = re.sub('[^a-zA-Z]'," ", plot)
     plot = plot.lower()
     plot = plot.split()
     plot = [i for i in plot if not i in set(stopwords.words('english'))]
     plot = ' '.join(plot)
     row['Key_words'] = str(plot)

And here is the output :(

Link to the csv : https://query.data.world/s/uikepcpffyo2nhig52xxeevdialfl7

Thank you !

Iterrow passes a copy of the row, not the reference. This should fix your problem:

df.loc[index,'Keywords'] = str(plot)

However, I would recommend using apply, imho it is more elegant. And it is alot faster.

That would looks something like this

def string_to_keywords(string):
    plot = re.sub('[^a-zA-Z]'," ", string)
    plot = plot.lower()
    plot = plot.split()
    return " ".join([i for i in plot if not i in set(stopwords.words('english'))])

df["Keywords"] = df["Plot"].apply(string_to_keywords)


When is z-normalization not needed when using DTW? I'm hoping to get some answers to a question I have regarding normalization of DTW datasets, in particular datasets in which two time-series shapes with similar shapes but differences in magnitude are misclassified under z-normalization.

One reason is simply empirical. When I z-normalize my data it doesn't cluster well. When I don't z-normalize it works quite well. But that isn't really enough to ignore the normalization advice.

The second is in reading the links in this thread which does a good job summarizing the other threads related to this, reading the literature, and thinking about my data. I cannot find a good discussion of the case where magnitude matters within a subset of similar clusters and z-normalization negates that important magnitude difference. In cases where the shape of some data can generally be the same (the lengths are the same, but the amplitude differs) z-normalization makes them cluster as identical when they aren't. Does this mean that DTW just isn't the right tool, that I should be normalizing differently, or is my data a special case because it is already normalized the way it is collected?

Briefly, my data is isotopic ratios recorded in fish "ear stones" which change as a fish moves from one location to another and are recorded in the growing layers of the stone (called an otolith). The isotope ratio in each river is very stable, and that ratio is known to the third or fourth decimal place. There is no biological fractionation, the information is recorded as it exists in the river when the fish enters the river. All data is normalized at collection with regard to the global ocean value which is known to the 5th decimal place. What we care about is classifying the data according to the shape of the isotopic curve, which is characteristic of where and when fish moved between rivers, the different decisions fish make in their movements. All the curves are interpolated to the same length prior to analysis.

The problem I see in my data is illustrated by the conceptual plot below.

The warping of DTW helps to better classify fish with mild differences in their movement timing,  like the two black lines in the plot. It also does a good job of classifying patterns with really different shapes (the red line). Without z-normalization, DTW correctly classifies the blue line as a different group. But, using z-normalization, the two black lines classify with the blue line because the magnitude difference is erased.

To take the example from Section 1.2.1 of Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping from the linked question above, the movement of fish in two different habitats is analogous to the action of drawing a gun or pointing a finger. The reason for z-normalization in the gun-point dataset, according to the author, is to account for offsets in scaling of the video. But in this case all of the data is normalized already to the global marine signature to account for machine drift, and the isotopic values are extremely precise and accurate. The error in amplitude of the curves is less than 1%. Is z-normalization required here, or is it just erasing meaningful differences in amplitude that are correctly classified without z-normalization?

I'm curious not just to find an answer regarding my own data, but also because it has been really difficult for me to find a straightforward discussion of this type of amplitude problem in the literature. Most of the advice is a blanket, "In 99% of cases, you must z-normalize." as eamonn stated in the other question...an opinion I respect given his work on this subject. But, I'm curious about what those 1% of examples are. Perhaps I'm just missing it, or perhaps I'm missing the boat completely. Your thoughts, and explanations, would be greatly appreciated.

That eamonn guys sounds smart.

If you do not normalize, small differecnes in the mean and/or STD rapidly swamp any shape similarity.

Here is one way to think about it.

If you do not normalize, and the means or STDs are different, then.......

EuclideanDIST(A,B) is approximately equal to DTWdist(A,B), which is approximately equal to [abs(mean(A)-mean(B)) times a constant]

If you can model the similairy with just the mean, they why do you need to even look at the shape?

BTW, you have a cool problem/dataset


Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance? I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm.

I made sure that the class counts are balanced. i.e., an equal number of occurrences per class.

The four features are Total Cholesterol, Systolic Blood Pressure, Diastolic Blood Pressure, and Cigraeette count. The class variable name is Stroke.

Dataset description is shown below:

       TOTCHOL    SYSBP    DIABP  CIGPDAY   STROKE
count  200.000  200.000  200.000  200.000  200.000
mean   231.040  144.560   81.400    4.480    1.500
std     42.465   23.754   11.931    9.359    0.501
min    112.000  100.000   51.500    0.000    1.000
25%    204.750  126.750   73.750    0.000    1.000
50%    225.500  141.000   80.000    0.000    1.500
75%    256.250  161.000   90.000    4.000    2.000
max    378.000  225.000  113.000   60.000    2.000

SKEW is

TOTCHOL    0.369
SYSBP      0.610
DIABP      0.273
CIGPDAY    2.618
STROKE     0.000

Python + sklearn is used here. The problem is that the classification performance gets very negatively-impacted when I try to normalize the dataset using

 X=preprocessing.StandardScaler().fit(X).transform(X)

or

 X=preprocessing.MinMaxScaler().fit_transform(X)

The classification report (before) normalizing the dataset:

            precision    recall  f1-score   support

      1       0.85      0.79      0.81        28
      2       0.82      0.88      0.85        32
avg / total   0.83      0.83      0.83        60

While the classification report (After) normalizing the dataset:

         precision    recall  f1-score   support

      1       0.47      1.00      0.64        28
      2       1.00      0.03      0.06        32

  avg/total   0.75      0.48      0.33        60

Please note that the class variable includes the outcomes 1(yes) and 2(no) instead of the usual 0 and 1.

at this point, I see the preprocessing have damaged the classification accuracy instead of helping it. Is there any logical explanation for that?
do I "have" to do the normalization step?

Another remark is the probability scores that I get (before) normalizing the dataset are much higher per class as shown below:

 [ 0.65929838  0.34070162]
 [ 0.40999878  0.59000122]
 [ 0.43592976  0.56407024]
 [ 0.40306785  0.59693215]
 [ 0.92748002  0.07251998]
 [ 0.74173761  0.25826239]

compared to the ones (after) normalizing the dataset

  [ 0.51636816  0.48363184]
  [ 0.5183946   0.4816054 ]
  [ 0.51410135  0.48589865]
  [ 0.50739794  0.49260206]
  [ 0.52645649  0.47354351]
  [ 0.5308564   0.4691436 ]

is there any explanation why I get such results?

I have found the answer to my own question, went back to the Python script and in the command that fits the model i.e.

   LR = LogisticRegression (C=0.1, solver = "sag",max_iter=1000).fit (X_train, y_train)

The parameter C was set to 0.001 which is a very small value (meaning lambda is very high as C=1/lambda) (C is the regularization strength and smaller values indicate stronger regularization). more on that matter can be found here and here