Is there a standard data science epistemology? In experimental science there is the 'scientific method', in mathematics there is the 'proof.' What are the standards for knowledge production in data science?

I've gathered that there a few different philosophies one can take; some of them are similar to the scientific method (introduce a falsifiable model and see if the data fits), others seem more hodge-podge (introduce some kind of learning machine and hope that the regularization prevents over fitting, and that cross validation catches it if not).

(There are some loose connections between those two, like how falsifiability fits into VC entropy and learning rates; this is explained in Vapnik's book on the Foundations of Statistical Learning.)

So, supposing that I have some data and I want to extract knowledge from it. What are the community standards for doing so? Where can I learn about the epistemological philosophy behind these standards, and their mathematical expression?

Interesting question.

The way I see it, the justification comes from both mathematics and experimental science.

If you suggest a new algorithm, you should state your assumptions and provide a proof, just like in math. However, you should also demonstrate it on real data, showing that your assumptions are reasonable and that your algorithm can deliver value.

If you suggest a model, you should justify it based on its performance on the observations, just like in experimental sciences. This is true no matter how did you derive the model. However, examining the model logic itself and the way it was derived will help to gain confidence in it and trace where it needs modification.


Detect passive voice in headlines To detect passive voice in sentences, we can use the spacy module to tag each token in the sentence, then build a classifier to classify it as passive or active based on conventional grammatical rules, e.g.

If a clause has all of the following, then it is in the passive voice:

A form of an auxiliary verb (usually be or get)
The past participle of a transitive verb
No direct object
The subject of the verb phrase is the entity undergoing an action or having its state changed

News headlines are written differently. They don't follow conventional grammatical rules. How would one be able to use spacy to detect headlines in passive voice?
For example, if I have this headline, "Church That Defied Coronavirus Restrictions Is Burned to Ground," its part-of-speech tags are ['ROOT', 'dobj', 'compound', 'compound', 'nsubjpass', 'auxpass', 'relcl', 'prep', 'pobj']. We can classify this as passive, based on the fact that it has a nominal subject (passive) and no direct object.
However, if I have this headline, "Rayshard Brooks Fatally Shot By the Atlanta Police," its part-of-speech tags are ['compound', 'ROOT', 'advmod', 'ROOT', 'prep', 'det', 'compound', 'pobj']. We can't use any of the criteria to classify this as passive voice, because the headline has dropped the auxiliary and spacy doesn't detect a nominal subject (passive).

So, your task is to detect the passive voice from sentences. Currently, you have defined some rules to detecting the passive voice and you have noticed that there some exceptions to your defined rules.
Therefore, it would be a good idea to develop a model to predict the probability of a sentence being passive (or active).
You can do this by encoding the sentence as a sequence of words (converted into word embeddings) using a Recurrent Neural Network (RNN) or LSTM. This encode will encode the words into a "hidden representation". This hidden representation can then be decoded using a neural network with a final softmax output layer which then outputs the probability that the sentence is written in the passive voice and the active voice respectively.
As this would be a supervised learning problem, you would need to label examples of passive- and active-voice sentences.


In survival analysis, which is the correct way to introduce a variable which changes the survival rate but occurs at different times? I am making a survival analysis with a cox regression with proportional hazards, we  want to analyze wheter the introduction of a phenomenon influences the time until the death of an individual.

A similar example would be: We have patients which were given a medicine at different stages of the disease (and at different ages too and some were not given the medicine at all), how the treatment variable should be introduced so we can measure the "strength" of the introduced change.

I have a possible solution:

Create a dummy exogenous variable for ranges of the time in which the medicine was introduced. "First 0-6 months", "6 m - 1 year", "1-2 years", "Never", this measured since the birth of the patient.
Create the same exogenous variable but not dummy.
Change the time variable and measure it not from birth but from the introduction of the medicine (the problem would be the definition of this variable for the patients without medicine).
Duplicate the rows of the individuals who had the medicine. In the first repetition, the time would be between birth and the introduction of the medicine as a "censored death"; the second repetition would include the time between the introduction of the medicine and the actual death.

Nowadays, the model includes the introduction of the medicine as a dummy but doesn't have into account the time in which it was introduced.

The problem you'll run into if you are not careful is the "immortal time bias". In short, the problem is that a subject isn't "in" the  "1-2 years" group until they atleast 1 year under observation. This 1 year period is called immortal because patients can't die then. More concretely, if I naively partition my population into "First 0-6 months" vs "1-2 years" and measure survival, the latter group is going to look like they have much better survival in the first year because in order to qualify for the latter group, you need to live longer.

So what do you do? You need to model the time-varying nature of your data. Check out the "long" format for survival data. Below I have a Python example that uses lifelines. There are four individuals, each will be treated in a different treatment period. We use multiple lines (but same id) to denote different time periods (note the mutually exclusive start/stop). A dummy variable of the treatment period is provided.

import pandas as pd
from lifelines import CoxTimeVaryingFitter

df = pd.DataFrame([
    {'id': 1, 'start': 0, 'stop': 12, 'E': 1, 't1': 0, 't2': 0, 't3': 0},  # never received treatment, died at t=12

    {'id': 2, 'start': 0, 'stop': 10, 'E': 1, 't1': 1, 't2': 0, 't3': 0},  # received treatment at very start

    {'id': 3, 'start': 0, 'stop': 3,  'E': 0, 't1': 0, 't2': 0, 't3': 0},   # will received treatment in second "period"
    {'id': 3, 'start': 3, 'stop': 9,  'E': 1, 't1': 0, 't2': 1, 't3': 0},  # received treatment in second "period"

    {'id': 4, 'start': 0, 'stop': 6,  'E': 0, 't1': 0, 't2': 0, 't3': 0},   # will received treatment in third "period"
    {'id': 4, 'start': 6, 'stop': 11, 'E': 1, 't1': 0, 't2': 0, 't3': 1},  # received treatment in third "period"

])

ctv = CoxTimeVaryingFitter().fit(df, id_col='id', start_col='start', stop_col='stop', event_col='E')

Another solution instead of using dummy variables is to create an interaction with the time variable above, but that will be harder to interpret I believe.


Aggregating Decision Trees I have a data set with 3 independent variables and 1 dependent variable. The dependent is play_golf, the independents are Humidity, Pending_Chores, Wind.
   I want to aggregate the probability of playing golf based on those rules of multiple trees. So again, play_golf (this is the dependent value), and there are three independent variables Humidity(High, Medium, Low), Pending_Chores(Taxes, None, Laundry, Car Maintenance) and Wind (High, Low).  A rule would be like (IF humidity = "High" AND pending_chores = "None" AND Wind = "High" THEN play_golf = 77% ).

I was thinking that random forest would weight the rules somehow on the collection of trees and give a probability.  But if that doesn't make sense, then can you just tell me how to get the decision rules with one tree and I will work from that. I believe am I talking about a decision tree vector? I'm not sure.

Have a look on HMM (Hidden Markov Model) also. A concrete example Of HMM is available in wikipedia. Decision tree is better
in generalising and applying learned data in another context and a Markov-model is better to recall the exact
learned machine state.


Should softmax cross entropy with logits always be zero if logits and labels are identical? Apologies for this basic question. I don't have a deep understanding of this function but I was pretty sure it would be zero if logits and label inputs are identical. I found out my MLP model solves my classification problem pretty well but the cost function is never actually 0 (rather it approaches something like -2.14442e+07 but never is actually 0). I have tested what happens if set logits and labels to the same input and still its the same. I use the tensorflow tf.nn.softmax_cross_entropy_with_logits as cost function.
Thank you for your help.

Logits are the pre-transform values in a layer, and are not compared directly to the labels when calculating the cost function. In fact, with softmax layer, setting the matching logit to 1, and the rest to zero isn't even that good, because (using $z_i$ as logit $i$ and $a_i$ as activation $i$):

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Substitute in your 0 and 1 values for $z_i$ and you will see the $a$ values are not that close to 0 and 1 (they would be 0.0853 and 0.2319 respectively for 10 classes). It is these activation values that feed into the loss calculation. So with typical multiclass loss function $l = -\sum_{\forall i} y_i log(a_i)$, you would score a loss of 1.4611 for such an example.

To get a very low cost, instead set the logit matching the true label (or 1) to a higher value, e.g. 10, and the logits matching the false value (or zero) to a low value e.g. -10. This would lead to

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Where

$$z_i = \{10\; if\;1\;\; |\;\;-10\;if\;0\}$$

And varies between 10 and -10 for values between 1 and 0.

That should significantly reduce the calculated loss (1.855e-08 for those values in a 10 class prediction). The logit allows us to 'stretch' the values for correct and incorrect predictions.

Getting an actual 0 loss when using softmax or sigmoid outputs and logloss is not technically possible. You might see it due to rounding in computation. However, neither sigmoid nor softmax equals exactly 0 or 1, except in the limit as input values tend to $\pm \infty$


Why some ML models can't take advantage of text ordering information? On this google tutorial (https://developers.google.com/machine-learning/guides/text-classification/step-4) it is said:

  >
  Build n-gram model [Option A]
  We refer to models that process the tokens independently (not taking into account word order) as n-gram models. Simple multi-layer perceptrons (including logistic regression), gradient boosting machines and support vector machines models all fall under this category; they cannot leverage any information about text ordering.
  >

Then

  >
  Build sequence model [Option B]
  We refer to models that can learn from the adjacency of tokens as sequence models. This includes CNN and RNN classes of models. Data is pre-processed as sequence vectors for these models
  >

Why some ML model (Simple multi-layer perceptrons, gradient boosting machines and support vector machines models) cannot leverage any information about text ordering?

Is it because they can't accept floats on their input? (Not sure if those ML models accept floats on their input. I am guessing they can't.)

This tutorial assumes the text will be represented as a fixed length feature vector. There are several ways to do that. The good old one is using n-grams, indexing and several tricks around that.

First, N-grams can be seen as kind of words (or rather, phrases). The extreme version of it is unigram, or 1-gram, which is just a single word. Bi-grams, tri-grams and so on encode certain word ordering, of course, but the ordering of the n-grams in the text is usually ignored. Let's assume you use unigrams and defined the vocabulary:

Index assigned for every token:
{'the': 0, 'mouse': 1, 'ran': 2, 'up': 3, 'clock': 4 }

By using the count encoding, we come up with the following vector representing the text:

'The mouse ran up the clock' = [2, 1, 1, 1, 1]

But if you simply shuffle the words, you will end up with the same vector.

2-grams, 3-grams and so on capture the word ordering better and it will be more difficult to get similar vectors for different texts. So, this is not completely true that they cannot leverage any information about text ordering. However, just by swapping sentences in a text, you can get very close representations. Especially, if some rare 3-grams are dropped from vocabulary (which is widely used to keep the vocabulary of reasonable size).

Recurrent neural networks and word embeddings are more modern approaches in NLP. In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.


How can I have a sequential model inside Keras' functional model? I'm trying to build a architecture that takes in multiple inputs and has a smaller neural network inside a larger architecture. For me to keep track of of the smaller netowrk's loss, I need to compile the sub-network as a Sequential, but it still needs to be part of the functional model as a whole for the architecture. Is this possible in Keras?

I managed to trial and error through this.

x = Sequential()
# add some layers...
inputs = [Input(...), Input(...), ...]
outputs = ... #some layers stacked together via the functional model
result = Model(inputs=inputs, x(outputs))

If you want the sequential in the middle of the outputs up there for some reason:

output1 = Layer(input)
output2 = x(layer1)
outputs = Layer(output2)


Neural Networks for Unsupervised Learning Why cannot we use neural networks for unsupervised learning problem.
I do know that it can be used using the Kohenon’s Self Organizing Map (KSOM) but is this the only method that we can use or are there any other.

Yes, there are others. The most important dimensionality reduction technique in Deep Learning is Autoencoders. They are neural networks with a "funnel" structure, that shrinks the size of the signal, forcing the Network to learn how to represent the same information with less nodes.

Autoencoders are much more common than SOMs in practical DL. While SOMs are used mostly for data visualization (usually they reduce a dataset in a 2D representation), Autoencoders give you absolute freedom in the number of factors that you can extract. This makes them more useful for any research or production purpose.

A good theoretical introduction can be found here.

Please also take a look at this practical implementation in TensorFlow 2.0.


Recognizing circled numbers on a piece of paper I've built a handful of CNN using tensorflow, keras, pytorch for recognizing text/number/objects in an image. What I'm trying to figure out how to do now is how to recognize numbers on a piece of paper that are circled by a pen or pencil. So on a piece of paper, there are two rows of 6 sets of numbers, and someone will go through and 2-3 sets of numbers, and I'd like to be able to take a picture or scan the paper, and have a model be able to recognize which number sets were circled. What would be the best way to start going about this?

The problem you are describing is known as the Object Localization problem. The main DNN architecture used today includes R-CNN and related algorithms as well as YOLO algorithm. It's a well-studied problem in computer vision. you can found plenty of information online.


Why do we use a Gaussian kernel as a similarity metric? In graph-based clustering, why is it preferred to use the Gaussian kernel rather than the distance between two points as the similarity metric?

Let's be precise. "Distance" has lots of meanings in data science, I think you're talking about Euclidean distance.

The Gaussian kernel is a non-linear function of Euclidean distance.

The kernel function decreases with distance and ranges between zero and one. In euclidean distance, the value increases with distance. Thus, the kernel function is a more useful metrics for weighting observations.
The fact that it's bounded between zero and one is a nice property, whereas the absolute distance (it can be anything) in Euclidean distance can cause instability and difficulty in modelling.
Euclidean distance (without the negative sign) is not a similarity measure, it's a distance function. The gaussian kernel is a similarity measure.
You can think the Gaussian kernel like a normalization function for Euclidean distance.


Find the time between two events by customer id I need to find a customer has bought P1, and after how many days customer will buy P2.
I am unable to find the days between order of P1 and the next order of P2 by the same customer.

I have data as shown below.

Customer ID   Order_Date   Product
       C-87   11/20/2018        P2
       C-87    7/25/2018        P1
       C-87    7/19/2019        P1
       C-87     8/2/2018        P2
       C-87    12/9/2019        P1
        ...          ...       ...
       C-22    9/22/2018        P2
       C-22     9/4/2018        P2
       C-22    1/15/2018        P1
       C-22     9/5/2019        P2
       C-22    3/20/2018        P1

You can split it into two dataframe containing only one of P1 and P2 first.

df1 = df[df.index[df['Product'] == 'P1'].tolist()]
df2 = df[df.index[df['Product'] == 'P2'].tolist()]

And then marge df1 and df2 on Customer ID and Product

df1.merge(df2, 'inner', left_on=['Customer ID', 'Product'], right_on=['Customer ID', 'Product'], copy=False)


What are the relationships/differences between Bias, Variance and Residuals? I've been trying to find an answer to this question for a long time.

What are the relationships/differences between Bias, Variance and
Residuals?

I think I do understand Bias, Variance and Residuals as separate concepts. Correct me if I'm wrong -
Bias is the difference between the average expected results from different runs of the model and the true values from data.
Variance is the variability in the expected results (predictions) of a given data point between different runs of the model.
Residual is the difference between the expected results from a model and the true values from data. y - y^ Residual seems somewhat similar to Bias. But is that my misunderstanding?
Please explain to me in plain words and also simple equations the relationships or differences between these three.

I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.

Bias, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.

Variance, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the "stability" of a given algorithm/parameter set when exposed to different training and testing sets.

(N.B. This can be confusing because there is a specific definition of "variance" used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)

Residuals are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.


Forcing a multi-label multi-class tree-based classifier to make more label predictions per document I'm been experimenting with tree based classifiers for multi-label document classification. All the trees I've created, however, tend to predict only one or two labels per document. Whereas the training set has about 4 labels per document on average. Furthermore, in my particular application, false positives are much less costly to the business than false negatives. So, if anything, I'd like the tree to be making about 6 or 7 predictions per document.

I'm not entirely sure which parameters control this. I've tried experimenting with tree size but without effect. I'd ideally like to just set a threshold for when a prediction is included, and lower this.

I'm using sklearn (and playing with skmultilearn). Here's an example of a forest configuration:

 from sklearn.ensemble import RandomForestClassifier

    clf = RandomForestClassifier(
        n_estimators=20,
        criterion='gini',
        max_features=0.5,
        max_depth=68,
        min_samples_split=4,
        min_samples_leaf = 2,
        n_jobs=3
        )

You seem to experience a class imbalance situation where some classes dominate the others by the number of samples they have, so that your algorithm finds it wise to predict less of the rare classes or non-predict them to decrease the loss at the end. You can manually set the class weights for the Random Forest Classifier, making loss function treat unevenly to different classes, but even in total.

For details you can refer to:
https://stackoverflow.com/questions/20082674/unbalanced-classification-using-randomforestclassifier-in-sklearn

Note: Random Forest is not robust to class imbalance, this is a known situation. You can refer to: https://stats.stackexchange.com/questions/242833/is-random-forest-a-good-option-for-unbalanced-data-classification

Hope if I could help. If not, I will be around for further discussion.


Quick question on basic Basic concept of experience replay Due to my admitted newbie's understanding on the field, I'm about to ask a dummy question.
While sampling batches, for example experience replay buffer which contains number of samples, after getting n (size of a batch) of loss values through forward propagation, what way will the weights updates follow:

Update all weights based on all losses separately, like iterating through that n losses to update
Update all weights based on the mean (or maybe standard deviation) of all losses

The 2nd one is the correct one. We update the weights according to the mean squared error between real value (in case of DQN it is a bootstrapped value) and the current value prediction, obtained for every sample ($s,a,R_t$) in the minibatch. In other words your network is going to get "corrected", for every prediction it has made, by the difference between estimation and ground truth values.


Tool for analyzing a Python matrix and generating a report on the contents (column types, NaN counts, means, etc.) I'm looking for a tool/library that will take a numpy or pandas matrix and generate a list of statistics for the matrix and columns. Specifically, for each column, I'd want info like the following:

Assumed data type (numerical vs string)

Assuming it's determined to be a numerical column:

Mean and Std Deviation
Max and min values
Number of NaN's
% of NaN's

Assuming it's determined to be a string column:

Number of distinct string values

So the tool takes the matrix and outputs some kind of report along these lines. The goal is to do some sanity checking on my input data set as well as checking data transformations along the way. For example, if I'm doing some kind of transformation and values in column A go from 1% NaN to 60% NaN, then I did something really bad.

Does such a tool exist?

You could use pandas.describe.

import pandas as pd
import numpy as np
df  = pd.DataFrame({'first': [1,2,3,4,np.NaN], 'second': [100,200,np.NaN, np.NaN,4]})
df.describe(include='all')

Regarding the percentage of NAs, you could use just a small formula for that:

count_nan = len(df) - df.count()
count_nan / df.shape[1]


Filtering a panda dataframe in one line I had the following data-cleaning question in an interview test that I struggled on (I've changed the details to anonymise it and protect the company's interview process)

  Given the following dataframe df, return a new series with day as the index, and a single column with the set of meals consumed by everyone who ate that day (i.e, both Alice and Bob on days 1 and 3, but only Alice on day 2). Do not use for loops or list comprehensions, only method chaining and a single lambda function that accepts only a single argument.

df = pd.DataFrame({'day':[1, 2, 3, 1, 3]*3,
                   'person':['Alice', 'Alice', 'Alice', 'Bob', 'Bob']*3,
                   'meal':['breakfast', 'breakfast', 'breakfast', 'breakfast', 'breakfast']+
                          ['lunch', 'brunch', 'brunch', 'lunch', 'lunch']+
                          ['dessert', 'dinner', 'snack', 'beer', 'dessert']
                  })

In other words, the goal is to obtain the following dataframe:

goal = pd.DataFrame({'day':[1, 2, 3],
                     'meal':[{'breakfast', 'lunch'},
                             {'breakfast', 'brunch', 'dinner'},
                             {'breakfast'}]
                    }).set_index('day')

Does anyone know how to do this? Thanks!

The first groupby - counts the number of persons per day per meal

The second groupby - counts the number of unique persons in each day

The inner merge between the 2 - matches the number of persons per day+meal to be equal to the number of persons on that day

We've left with a df that contains day-meal couples in each row, with the number of persons that had a meal which matches the total number of persons on that day:

Now we only need to groupby day and create the set of the meals

df.groupby(['day','meal']).count().reset_index()\
  .merge(df.groupby('day')['person'].apply(lambda x: len(set(x))).reset_index(),
         on=['day','person'])\
  .groupby(['day'])['meal'].apply(set)


how to build a time series classifier I have chemical plant data where the product is manufactured in batches.  Each batch takes about 4 hours and I have data for every 5 mins. My objective is to classify the batches as good and bad. How can I achieve this for a time related data like this one?

One method I have read is time unfolding where each time interval in combination with the feature becomes an individual feature (e:g if there is interval T0 and T1, features F0 and F1 then we will have 4 features T0 F0 and T1 F1) - please correct me if I am wrong about unfolding.

So basically we are converting each batch of N rows of data into single row. Is there any other method for this?

I am not clear on how you convert your N rows into a single row. Do you concatenate all of them? This is an option but could become problematic over longer sequence lengths.

A straightforward method could be to simply average the features across time-steps. Although you lose causal information, this has been shown to work surprisingly well for word embeddings.

Another standard approach is to use hidden markov models. Very simply, a hidden markov model assumes a latent state space which generates an output (in this case your feature) at every time-step conditioned on the previous state.
https://en.wikipedia.org/wiki/Hidden_Markov_model

Today the practice is to use sequence neural models such as LSTMs and GRUs which can hold memory over longer time-steps.
https://en.wikipedia.org/wiki/Long_short-term_memory

I suggest your read up on these methods and select the one that suits your purpose :)


Keras error "Failed to find data adapter that can handle input" while trying to train a model I've been following a tutorial on training a model and I've stumbled across an error that I've been struggling to find a solution for.

The code for the model training is bellow:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D
import pickle

X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))

X = X / 255.0

model = Sequential()
model.add(Conv2D(64, (3, 3), input_shape = X.shape[1:]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size = (2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size = (2, 2)))

model.add(Flatten())
model.add(Dense(64))

model.add(Dense(1))
model.add(Activation("sigmoid"))

model.compile(loss = 'binary_crossentropy',
             optimizer = 'adam',
             metrics = ['accuracy'])

model.fit(X, y, batch_size=32, validation_split=0.1)

And the result I get when I run this code is:

  ValueError                                Traceback (most recent call
  last)  in
       28              metrics = ['accuracy'])
       29
  ---> 30 model.fit(X, y, batch_size=32, validation_split=0.1)

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training.py
  in fit(self, x, y, batch_size, epochs, verbose, callbacks,
  validation_split, validation_data, shuffle, class_weight,
  sample_weight, initial_epoch, steps_per_epoch, validation_steps,
  validation_freq, max_queue_size, workers, use_multiprocessing,
  **kwargs)
      731         max_queue_size=max_queue_size,
      732         workers=workers,
  --> 733         use_multiprocessing=use_multiprocessing)
      734
      735   def evaluate(self,

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py
  in fit(self, model, x, y, batch_size, epochs, verbose, callbacks,
  validation_split, validation_data, shuffle, class_weight,
  sample_weight, initial_epoch, steps_per_epoch, validation_steps,
  validation_freq, **kwargs)
      215           validation_data=validation_data,
      216           validation_steps=validation_steps,
  --> 217           distribution_strategy=strategy)
      218
      219       total_samples = _get_total_number_of_samples(training_data_adapter)

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py
  in _process_training_inputs(model, x, y, batch_size, sample_weights,
  class_weights, steps_per_epoch, validation_split, validation_data,
  validation_steps, shuffle, distribution_strategy)
      468                      'at same time.')
      469
  --> 470   adapter_cls = data_adapter.select_data_adapter(x, y)
      471
      472   # Handle validation_split, we want to split the data and get the training

  ~\Miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py
  in select_data_adapter(x, y)
      446         "Failed to find data adapter that can handle "
      447         "input: {}, {}".format(
  --> 448             _type_name(x), _type_name(y)))
      449   elif len(adapter_cls) > 1:
      450     raise RuntimeError(

  ValueError: Failed to find data adapter that can handle input: , ( containing values of types {""})

Thank you in advance if you can help me figure out where the problem is.

There is something wrong with your data. {""} means you have a Python dict that only contains an empty string.


Training an RNN with examples of different lengths in Keras I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for training.

In the keras documentation, it says the input to an RNN layer must have shape (batch_size, timesteps, input_dim). This suggests that all the training examples have a fixed sequence length, namely timesteps.

But this is not especially typical, is it? I might want to have the RNN operate on sentences of varying lengths. When I train it on some corpus, I will feed it batches of sentences, all of different lengths.

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it. But then does that mean I can't make predictions at test time with input length greater than that?

This is a question about Keras's particular implementation, I suppose, but I'm also asking for what people typically do when faced with this kind of a problem in general.

This suggests that all the training examples have a fixed sequence length, namely timesteps.

That is not quite correct, since that dimension can be None, i.e. variable length. Within a single batch, you must have the same number of timesteps (this is typically where you see 0-padding and masking). But between batches there is no such restriction. During inference, you can have any length.

Example code that creates random time-length batches of training data.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed
from keras.utils import to_categorical
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(8, return_sequences=True))
model.add(TimeDistributed(Dense(2, activation='sigmoid')))

print(model.summary(90))

model.compile(loss='categorical_crossentropy',
              optimizer='adam')

def train_generator():
    while True:
        sequence_length = np.random.randint(10, 100)
        x_train = np.random.random((1000, sequence_length, 5))
        # y_train will depend on past 5 timesteps of x
        y_train = x_train[:, :, 0]
        for i in range(1, 5):
            y_train[:, i:] += x_train[:, :-i, i]
        y_train = to_categorical(y_train > 2.5)
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=10, verbose=1)

And this is what it prints. Note the output shapes are (None, None, x) indicating variable batch size and variable timestep size.

__________________________________________________________________________________________
Layer (type)                            Output Shape                        Param #
==========================================================================================
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, None, 8)                     1312
__________________________________________________________________________________________
time_distributed_1 (TimeDistributed)    (None, None, 2)                     18
==========================================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
__________________________________________________________________________________________
Epoch 1/10
30/30 [==============================] - 6s 201ms/step - loss: 0.6913
Epoch 2/10
30/30 [==============================] - 4s 137ms/step - loss: 0.6738
...
Epoch 9/10
30/30 [==============================] - 4s 136ms/step - loss: 0.1643
Epoch 10/10
30/30 [==============================] - 4s 142ms/step - loss: 0.1441


normalize identification values properly I'm building a neural network to analyze a business' sales. I'm normalizing all input values to the range {0,1}.

I'm struggling with the day of the week column. Business days are identified by a number ranging {1-5} (1=Monday). Normalizing these values to the range {0,1} is straightforward, but results in a major bias in the final output.

The reason is that the full range of normalized values for the business day column is explored with every week worth of data, whereas other price-related column explore their full range of normalized values infrequently.

The business day column ends up being the largest contributor to the final output.

How can I normalize it to make its contribution more in tune with the rest of the inputs?

It is possible that the other variables you're feeding into the NN are simply bad at predicting sales.  Sell prediction is a notoriously hard problem.

Specifically the addressing of mapping a multi-state categorical variable to the NN's {0,1} input range:  Another idea is to change that one, 5-state variable into five boolean variables.  Rather than {0,0.25,0.5,0.75,1.0} on your one variable, make each of the five boolean variables represent a single day and make [1,0,0,0,0] equal Monday, [0,1,0,0,0] equal Tuesday, etc.  I've personally had more success both with training good networks and introspecting the network itself when spreading out states of classes like that.

Other hacks you can try:
 * Take out the the 'day' column all together and see if any of the other variables get used.
 * Plot the distribution of spend as a function of day.  Even if nothing else comes of this current model, it sounds like you've found one interesting insight already.
 * Consider also trying different models.


Problem with Logistic regression using categorial variables I have a problem with Logistic regression using categorial variables. This my training datasets :

The issue is that logistic regression cannot make use of features which are strings. To use these features in your model you have to convert them to values first by encoding them. There are several ways of encoding the string values into numerical values, examples of this are using a LabelEncoder, OneHotEncoder, and OrdinalEncoder.


Predefined Neural Networks instead of fine tuning? I usually try to form my ANNs with classic fine-tuning approach but I recently learned that there are different "predefined" networks specially for certain tasks. Is there a good summary about these? Are they really perform better than home-made ones?

It depends. If you are doing a task for which the weights of a very large network, trained on the same type of problem, are present, then it's better to use the weights of that pre-trained network. You can also fine tune the layers later on. One such example is the VGG16 Net. This NN was used in the ImageNet challenge in 2014, hence if you are trying to do image classification tasks where your images are the subset of ImageNet, then you should use the pre-trained VGG16 weights. This is a great tutorial if you want to go in details.

But the above situation might not be true in every case. For example, the above case is not true if you reverse the situation.


Confidence Intervals for Multi-Categorical Votes I have an ngram-based language model that produces a long tag list for a given sentence. For example, the just-previous sentence, broken into bigrams, and run through the model might produce something like:

{I have}=>C1 {have an}=>C2 {an ngram}=>C1 {ngram based}=>C3, etc.

resulting in counts: C1=2, C2=1, C3=1 (for the shown segment above).

Easy enough to pick the winner by sorting either the counts, or after turning them into percentages, which would control for sentence length. But I want a CI on that winner -- that is, I want to know when it's a statistical tie between the top N categories (either by count or percent).

I'm sure that there's an obvious way to do this....

...Pointers appreciated!

I found a concise description of how to do this.

(But note that this doesn't correct for sentence length, so it's not a complete solution bcs longer sentence will have more "votes".)


How to identify text similarity based on training data? I have a set of documents  (1 to 11) for which the labeling is done.
Lets Assume:
Doc No: 1,3,5,7 - Belongs to Type A
Doc No: 2,4,9 - Belongs to Type B
Doc No: 8,10 - Belongs to Type C
Doc No, 6,11 - Belongs to No one

Now, let us say I have new incoming docs - 11,12,13 .. and so on, and I would like to know which Type (A, B, C or none ) they belong to based on the text similarity of existing docs in that Type. Can someone please suggest how I can achieve this?
Should I create my own corpus of data and consider it a supervised problem?

Yes this is a supervised problem. I'd suggest following the example in this article.


How many words should be taken as features in a ML problem? I would like to ask you how many words should be taken as features in a ML program.
For example, if I have 30000 distinct words to make a vocabulary, what would a good number be?
I am currently removing stopwords, words with a few characters, numbers; applying lemmatisation; removing punctuation; analysing word frequency to keep the top 50 words.
However I do not know if out of 30000 distinct words, 50 would be too low.
What would be a good way to determine how many to keep or select?

It depends on many factors: number of instances, number of classes, token/type ratio, etc. A common basic technique in order to avoid having too many words is to discard the ones which appear rarely, for example by discarding all the words which appear in less than 3 documents. Selecting only the top $N$ most frequent words is unusual, even though it's a similar idea. There is a risk to end up mostly with words which appear frequently and don't have a good discriminative power.
The proper way to determine an optimal threshold, for example the minimum frequency $N$, is to run a training/testing experiment for every value of $N$ and select the value corresponding to maximum performance. Note that this experiment should use a validation set different from the final test set in order to avoid data leakage.


Predict the corresponding value in one column using a list of values found in another column Please have a look at this link. This was a question I asked few months back and after some suggestions and exploring I was able to successfully use TFIDF along with MultinomialNB classifier to pretty accurately predict the 'Item' based on the Composition column. I wrote the answer myself to tell how I solve it. But this time the same logic isn't helping really well for a similar dataset.

Old data:(referencing my previous example in the link above)

UID    Item               Composition
1      Water              Hydrogen,Oxygen
2      Sulfuric acid      Hydrogen,Sulfur,Oxygen
3      Alcohol            Spirit
4      Hydrochloric acid  Hydrogen,Chloride
5      Citric Acid        Hydrogen,Carbon, Oxygen

New data example:

UID    Item                 Composition
1      [Sweater]            [Wool, knitting, handmade, knitting needle]
2      [Jeans]              [Denim, cotton, orange thread, stonewash, blue dye]
3      [CottonTrouser]      [Cotton, littlepolyster, weaving, handstitch, vcut]
4      [SilkShirt]          [wormsilk, artificialsilk, weaving, hand looming, color dying, coating]
5      [Carpet]             [Wool, cotton, organic cotton, knitting, sewing]

This time I have a lot of such data. There around 4200 such items in the Item column. I am trying to use TFIDF with bigrams and trigrams and using MultinomialNB to make the algorithm learn all the items in the Composition and predict the Item accurately.

1) I would like TF-IDF to so the n-grams at a word level so it can vectorise the words.
2) I would like the classifier to classify the Items based on the different compositions so it can learn what compositions put together.

For some items in Item column, they have around 10 values that comprise in the Composition column. So I used min_df=8 and ngram_range=(1,8), hoping it can try to build a vocabulary of around 8 words per composition.

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, min_df=8, norm='l2', encoding='latin-1', ngram_range=(1,8),     analyzer='word',lowercase=True,stop_words='english')
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer
features.shape

No matter what combinations of this I try the predictions are going wrong and I notice that the ngrams aren't really look at the word level in the Composition column.

I need expert advice on how to better solve this problem and mistakes in my approach. Consider me a novice in this area.

Not a very clever solution though. But I managed to do some trick to make it work. I am not fully satisfied by the result but the algorithm is able exactly predict the Item based on given Composition column.

from io import StringIO
import json
col = ['Item', 'Composition']
df = dfmin[col]
df['Item'] = df['Item'].apply(lambda x: ''.join(str(x).strip('[]') if      isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: ''.join(str(x).strip('[]') if isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(',',' '))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(' ',''))
df['Composition'] = df['Composition'].apply(lambda x: x.replace("'",''))
df['Composition'] = df['Composition'].apply(lambda x: "".join(x.rstrip()))
df.head()

Doing this removed all spaces in the text in Composition column and it ended up becoming one long string set.

df['Indexer'] = df['Composition'].factorize()[0]
#df['Indexer']
Indexer_dfmin = df[['Item',    'Indexer']].drop_duplicates().sort_values('Indexer')
df_to_Indexer = dict(Indexer_dfmin.values)
Indexer_to_df = dict(Indexer_dfmin[['Indexer', 'Item']].values)
#df['Composition'] = df['Composition'].str.split(",")
df.head()

I factorized the Composition column using pd.factorize().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, norm='l2', encoding='latin-1', ngram_range=(1,8), analyzer='word',lowercase=True)
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer
features.shape

Using sklearn RigdeClassifierCV I was able to model the data and predict the Item name using the string of the Composition column.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import RidgeClassifierCV
X_train, X_test, y_train, y_test = train_test_split(df['Composition'].values.astype('U'),df['Item'].values.astype('U'), random_state=42)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = RidgeClassifierCV(alphas=[1e-3, 1e-2, 1e-1, 1]).fit(X_train_tfidf, y_train)

Issues with this approach:
1) The model is not learning the words in the Composition column and building a vocabulary.
2) So, if I use the words with space from the Composition column the model will not be able to predict properly.
3) The accuracy can go near to 100% percent which means the model is overfit.

That is why I say this is not a clever solution. I would like to get to know how to fix the mistakes and issues in my model.
Appreciate some expert advice on this. Please feel free to comment. Thanks in advance.


activation function for binarized neural networks I am trying to implement a binarized neural network using keras and tensorflow.
Here is my current code:
import tensorflow as tf
from tensorflow import keras
from keras import datasets
from keras.utils.vis_utils import plot_model

from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

def sign(x):
    result = K.sign(x)
    return result

get_custom_objects().update({'custom_activation': Activation(sign)})

digit_data = datasets.mnist
(train_images, train_labels), (test_images, test_labels) = digit_data.load_data()

# import matplotlib.pyplot as plt
# plt.imshow(train_images[0])
# print(train_labels[0])
# print(train_images[0])

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(784, Activation(sign, name='SpecialActivation')),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

# model = keras.Sequential([
#     keras.layers.Flatten(input_shape=(28, 28)),
#     keras.layers.Dense(128, activation=tf.nn.relu),
#     keras.layers.Dense(10, activation=tf.nn.softmax)
# ])

# plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

model.compile(optimizer=tf.keras.optimizers.Adam(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=5)
test_loss, test_acc = model.evaluate(test_images, test_labels)
print(test_loss, test_acc)

The activation function I am currently using is a sign function which returns -1 if x < 0 | 0 if x=0 | 1 if x >0
the binarized neural network needs to have a bitwise operation function which returns +1 if x => 0 and -1 if x< 0 (from what i understand)
I'm not sure how to implement that kind of an activation function.

Create a custom function
As explained in Keras doc for custom Actiation function
Creating custom activations
You can also use a TensorFlow callable as an activation (in this case it should take a tensor and return a tensor of the same shape and type):
model.add(layers.Dense(64, activation=tf.nn.tanh))

def custom_bin(x):
    return tf.where(x >= 0, 1.0 , -1)

import tensorflow as tf
from tensorflow import keras
model = keras.Sequential()

model.add(keras.layers.Dense( 3, activation="linear",input_shape=(5,)))
model.add(keras.layers.Dense( 3, activation="linear"))
model.add(keras.layers.Dense( 4, activation=custom_bin))
model.add(keras.layers.Dense( 2, activation="linear"))

model.summary()

A toy NN to check the Activation output
A simple neural net with all weights = 1 and bias = 0. So, it will keep the output sign same as that of the input i.e.(+Ve, 0, -Ve).Then apply the respective Activation function in the output layer. Then predict using the 3 use-cases
def custom_bin(x):
    return tf.where(x >= 0, 1.0 , -1)

import tensorflow as tf
from tensorflow import keras

#Case I - tf.math.sign
initializer = tf.keras.initializers.Ones()
model = keras.Sequential()
model.add(keras.layers.Dense( 2, activation="linear",input_shape=(2,),kernel_initializer=initializer))
model.add(keras.layers.Dense( 1, activation=tf.math.sign,kernel_initializer=initializer))
print(model.predict([[-5,-5],[0,0],[5, 5]]))

#Case II - custom if-then-else
initializer = tf.keras.initializers.Ones()
model = keras.Sequential()
model.add(keras.layers.Dense( 2, activation="linear",input_shape=(2,),kernel_initializer=initializer))
model.add(keras.layers.Dense( 1, activation=custom_bin, kernel_initializer=initializer))
print(model.predict([[-5,-5],[0,0],[5, 5]]))

Output
[[-1.] [ 0.] [ 1.]]
[[-1.] [ 1.] [ 1.]]


Dealing with biased binary classifier My training data is weighed heavier on the '1' class, with about a 4:6 ratio. This outputs a classifier that is of 82% accuracy with an emphasis on the '1' class, which makes sense.

Confusion Matrix -
[[333 133]
 [ 62 612]]

I have the test proportions as well, in which the data will be tested on, which is 0.3 of '1' and 0.7 of '0' or 1900 0s and 900 1s. My classifier outputs 1400 1s and 1300 0s.

My theory is that I need to build a classifier that favours the '0', If so how can I make the classifier biased to one class over another?

I have tried to used the class weights, this does increase the '0' predictions but only by a very small percentage.

What you have in your data called imbalanced classes

From Datacamp

  Imbalanced data typically refers to classification tasks where the
  classes are not represented equally.

  For example, you may have a binary classification problem with 100
  instances out of which 80 instances are labeled with Class-1, and the
  remaining 20 instances are marked with Class-2.

In this link, you can find a nice article that explains more what it is and how you can handle it -> https://www.datacamp.com/community/tutorials/diving-deep-imbalanced-data

One of the solutions is to use over or under-sampling. You can achieve this with the SMOTE algorithm. Here is an example in Python.

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

sm = SMOTE(random_state=2)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())


How to find the count of consecutive same string values in a pandas dataframe? Assume that we have the following pandas dataframe:

df = pd.DataFrame({'col1':['A>G','C>T','C>T','G>T','C>T', 'A>G','A>G','A>G'],'col2':['TCT','ACA','TCA','TCA','GCT', 'ACT','CTG','ATG'], 'start':[1000,2000,3000,4000,5000,6000,10000,20000]})

input:
 col1 col2  start
0  A>G  TCT   1000
1  C>T  ACA   2000
2  C>T  TCA   3000
3  G>T  TCA   4000
4  C>T  GCT   5000
5  A>G  ACT   6000
6  A>G  CTG  10000
7  A>G  ATG  20000
8  C>A  TCT  10000
9  C>T  ACA   2000
10 C>T  TCA   3000
11 C>T  TCA   4000

What I want to get is the number of consecutive values in col1 and length of these consecutive values and the difference between the last element's start and first element's start:

output:
 type length  diff
0  C>T  2   1000
1  A>G  3   14000
2  C>T  3   2000

Break col1 into sub-groups of consecutive strings. Extract first and last entry per sub-group.

Something like this:

df = pd.DataFrame({'col1':['A>G','C>T','C>T','G>T','C>T', 'A>G','A>G','A>G'],'col2':['TCT','ACA','TCA','TCA','GCT', 'ACT','CTG','ATG'], 'start':[1000,2000,3000,4000,5000,6000,10000,20000]})

df['subgroup'] = (df['col1'] != df['col1'].shift(1)).cumsum()

  col1 col2  start  subgroup
0  A>G  TCT   1000         1
1  C>T  ACA   2000         2
2  C>T  TCA   3000         2
3  G>T  TCA   4000         3
4  C>T  GCT   5000         4
5  A>G  ACT   6000         5
6  A>G  CTG  10000         5
7  A>G  ATG  20000         5

df.groupby('subgroup',as_index=False).apply(lambda x: (x['col1'].head(1),
                                                        x.shape[0],
                                                        x['start'].iloc[-1] - x['start'].iloc[0]))

0        ([A>G], 1, 0)
1     ([C>T], 2, 1000)
2        ([G>T], 1, 0)
3        ([C>T], 1, 0)
4    ([A>G], 3, 14000)

Tweak as needed.


Why normalization kills my accuracy I have a binary car sound classifier. I have a feature set that is extracted from audio with size of 48. I have a model(multi layer neural network) that has around %90 accuracy on test and validation sets. (without normalization or Standardization)

I see that the feature values are mostly around [-10, +10] But there are certain features with a mean of 4000. Seeing unproportional values within features I though some feature scaling might improve things. So using scikit-learn tools I tried:

   - Simply removing the means from features
   - Normalizer
   - Min max scaler
   - Robust Scaler

And all these above ended up dropping my accuracy to ~ %50!  (%100 recall, %50 precision)

So how is this possible? And what is the correct way to normalize my data?

There could a skewed power envelope or non-stationary data. As a result, off-the-shelf feature scaling could attenuate the signal.

There are feature scaling techniques that tend to work better for audio signals, examples include: RMS level (Root Mean Square Level), Cepstral Mean Subtraction (CMS), RelAtive SpecTrAl (RASTA), kernel filtering, short time gaussianization, stochastic matching, and feature warping.

You should make sure you understand your raw data and the assumptions of each feature scaling technique before application. Accuracy-driven machine learning might lead to the wrong conclusions.


Reducing the GPU memory usage when the model is already small enough I trained a model and froze it into a PB (protocol buffer) file and a directory of some variables, and the total size is about 31M. We deployed it using a GPU card and followed this answer and set the per_process_gpu_memory_fraction to a very little number to make the memory to be about 40M. The program performs very well but when we check the GPU usage by nvidia-smi which shows that the memory usage is about 500M.
Then my question is how can I justify this gap? How can we reduce that? Can we do something like quantization to decrease the 500M? We want to deploy it into an edge device so the 500M is too large.

I cannot answer regarding the gap. What I can help you is how to reduce the memory usage.

One thing you can try is you can convert your weights to fp16 and do inference with fp16. Don't worry you would not lose significant numerical precision especially since you are using that for deployment. In fact, this is how people do forward pass on mixed precision training. Theoretically this could increase speed and reduce memory usage. Hope it helps.


Can't understand Output shape of a Dense layer - keras I am following an online tutorial to classify images and started off with dense layers as a starting point to classify cifar10 data.

# Create a model and add layers
model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(32, 32, 3)))
model.add(Dense(10, activation='softmax'))

# Print summary
model.summary()

dense_1 output shape (None, 32, 32, 512). 'None' represents the batch size, but what does '32,32' represent? Why isn't the shape (None, 512)? Same happens with the dense_2 layer.

Can someone explain it here or point me to a resource that explains this?

Keras is applying the dense layer to each position of the image, acting like a 1x1 convolution.

More precisely, you apply each one of the 512 dense neurons to each of the 32x32 positions, using the 3 colour values at each position as input. That's why you have 512*3 (weights) + 512 (biases) = 2048 parameters.

As a consequence, for each neuron in each position you generate an output, and that's why you have 512 outputs for each position, i.e., 32x32x512.

This all happens for each example in the batch: that's the None part.


Using LSTMs for modelling and forecasting several time series generated by the same process My scenario is roughly the following:

Imagine 500 cars, all Toyota Corollas (or whatever). While these cars have many similarities, they are not exactly identical: some of them have 1.5 liter engines while others have 1.8 liter engines, and some have a high mileage while others have barely left their garage. Still, we can imagine that all cars are the product of one process with a bunch of parameters. Some of these parameters are observed, while others are unobserved.

I have data on the condition of each car at a set of times t_i. For our purposes, "condition" is represented by a number between 0.0 and 1.0, where 0.0 means "as good as new" and 1.0 means "basically a wreck". My goal is to build a model which lets me forecast the condition of each car at some future time, based on its condition at the previous time steps as well as its "parameters" like mileage and engine size. I could train a separate model for each of the cars, but I likely don't have enough data for each car for this model to provide accurate forecasts.

I've already made an attempt at this using ordinary machine learning methods by transforming the full history of each car into one data point by encoding the condition at each of the previous time steps as a feature. My results are kind of so-so, and I'd like to try out LSTMs for this purpose.

However, I have a hard time figuring out how to do this with LSTMs. A bit of googling has led me to lots of tutorials on how to use LSTMs to forecast two properties of the same "object", say, condition and mileage for a single car, but none on how to predict the same property for a bunch of "different objects" with additional parameters. I assume just plugging

[conditionCar1, mileageCar1, colorCar1, conditionCar2, mileageCar2, colorCar2, (...)]

observed at a lot of time steps into an LSTM model wouldn't work very well. Is what I'm trying to do viable, and if so does anybody have any pointers to relevant information?

You can frame the problem this way.

Consider each car as a sample of your dataset and use $Car^{i}$ to denote the $i^{th}$ sample. Then for each sample, you have the following features:

Time series of cars' condition $C_0^{i}...C_n^{i}$
Other time-invariant features like colour $A^{i}$ and engine size $E^{i}$

(I am not sure how mileage fits here, do you have a time series on this as well? Intuitively mileage changes over the life of a car so it should be time series as well)

My understanding of your question is how to incorporate those time-invariant features into an LSTM model?

Instead of only passing the $C_{t-1}$ as LSTM cell input you pass it with a vector $[C_{t-1}^i, A^i, E^i]$. Here, using Tensorflow's terminology, you input_size is 3 instead 1. $A^i$ and $E^i$ is always the same at each timestep but different across samples.


PMI between lemma vs surface I was wondering whether it's possible to compute the some sort of pointwise mutual information between lemma and its surface form.

First if we assume,

p('to go') = count('to go') / sum(all lemmas)

p('went') = count('went') / sum(all words)

Breakpoint here, since every word comes with its respective lemma, we have the condition that

sum(all lemmas) == sum(all words)

The joint probability is also a little hard to normalize

# count of "went" being lemmatize to "to go
p('went', 'to go') = count('went'->'to go') / sum(all words)

The catch here is since "went" always lemmatize to "to go", we have special condition of

count('went'->'to go')  == count('went')

The reverse is not true though since the "to go" can be realized as different surface forms.

In that case, it's really awkward, if we put it all together

 p('went', 'to go') = p('went') = count('went') / sum(all words)
 p('to go') = count('to go') / sum(all words)

 PMI('went', 'to go')
 =  p('went', 'to go') / (p('to go') * p('went')
 =  p('went') / (p('to go') * p('went')
 =  1 / p('to go')

But that would assign the same information value for all surface forms given the same lemma.

I think I've made some mistakes in how I'm accounting for the joint or independent probabilities. Could anyone advise how to approach an information score for lemma and its surface?

Or is the information score between lemma and its surface uninformative thus the conclusion of inverse probability of the lemma?

One possible solution is to change the normalization value for the joint probability where:

p('went', 'to go') = count('went') / count('to go')

Then the PMI equation would be:

PMI('went', 'to go')
=  p('went', 'to go') / (p('to go') * p('went')
=  (p('went') / (p('to go')) / (p('to go') * p('went')
=  1 / p('to go')**2

Still the final mutual information disregard the surface probability =(

The result that you are calling awkward is indeed what we expect from the definition of PMI. Here are some identities involving PMI:

$\operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}$

Let $x=$'went' and $y=$'to go' and use the last expression. $p("to go"|"went") = 1$, and voilà, only $p("to go")$ remains in the denominator of the expression.


Foreign exchange market forecasting with neural networks I would like to use ANN to automate trading currencies, preferably USD/EUR or USD/GBP. I know this is hard and may not be straightforward. I have already read some papers and done some experiments but without much luck. I would like to get advice from EXPERTS to make this work.

Here is what I did so far:

I got tick by tick data for the month of july 2013. It has bid/ask/bid volume/ask volume.
Extracted all ticks for the time frame 12PM to 14PM for all days.
From this data, created a data set where each entry consists of n bid values in sequence.
Used that data to train an ANN with n-1 inputs and the output is the forecasted nth bid value.
The ANN had n-1 inputs neurons, (n-1)*2 + 1 hidden and 1 output neuron. Input layer had linear TF, hidden had log TF and output had linear TF.
Trained the network with back propagation with n-125 first and then 10.

For both n, the MSE did not drop below 0.5 and stayed at that value during full training. Assuming that this could be due to the time series being totally random, I used the R package to find partial autocorrelation on the data set (pacf). This gave non zero values for 2 and 3 lags only.

Question 1: What does this mean exactly?

Then I used hurst exponent to evaluate the randomness. In R, hurst(values) showed values above 0.9.

Question 2: It is supposed to be nearly random. Should it have a value closer to 0.5?

I repeated the training of the ANN with n=3. The ANN was trained and was able to obtain a pretty low value for MSE. However, the calculated output from this ANN does not differ much from the (n-1)th bid value. It looks like ANN just takes the last bid as the next bid! I tried different network structures (all multilayer perceptions), different training parameters, etc, but results are same.

Question 3: How can I improve the accuracy? Are there any other training methods than backpropagation?

The results you're seeing aren't a byproduct of your training product, but rather that neural nets are not a great choice for this task. Neural nets are effectively a means to create a high order non-linear function by composing a number of simpler functions. This is often a really good thing, because it allows neural nets to fit very complex patterns.

However, in a stock exchange any complex pattern, when traded upon will quickly decay. Detecting a complicated pattern will generally not generate useful results, because it is typically complex patterns in the short term. Additionally, depending on the metric you choose, there are a number of ways of performing well that actually won't pay off in investing (such as just predicting the last value in your example).

In addition the stock market is startlingly chaotic which can result in a neural net overfitting. This means that the patterns it learns will generalize poorly. Something along the lines of just seeing a stock decrease over a day and uniformly deciding that the stock will always decrease just because it was seen on a relatively short term. Instead techniques like ridge and robust regression, which will identify more general, less complex patterns, do better.

The winner of a similar Kaggle competition used robust regression for this very reason. You are likely to see better results if you switch to a shallow learning model that will find functions of a lower polynomial order, over the deep complex functions of a neural net.


How to divide 7 non-continuous result values into 5 legend categories? I'm making a map which displays score results for river reaches. The score results are integers from 1 to 7.

Now I want to make a legend and color code my results. However, I don't want to use too many color shades as the human eye/brain is not really able to distinguish clearly between more than +/- 5 shades.

So my question is: how do I divide my 7 (non-continuous) score results into 5 categories for my legend?

I thought of the following options:

legend 1:

result = 1 - 2
result = 3
result = 4
result = 5
result = 6 - 7

legend 2:

result <= 1.4
1.4 < result <= 2.8
2.8 < result <= 4.2
4.2 < result <= 5.6
result > 5.6

My issue with the first legend is that I would be visually tricking my audience into thinking that the extreme values are more prevalent than they actually are.
My issue with the second legend is that my result values are not continuous, so in reality the second legend looks like this:

result = 1
result = 2
result = 3 - 4
result = 5
result = 6 - 7

So again, my data will be visually skewed and deceiving my audience.

What are proper ways to handle this type of situation? Or any similar situation for that matter (for instance for 9 or 11 or 13 result values)?

Thanks!

You do not have to use shades of the same colour.  You can keep 7 colours if they have sufficient contrast between them.

The "Gradient" and "Indexed" colour schemes of ColorData may be be used to select the 7 colours.

Some examples include.

In Wolfram Language

BarLegend[{#, {0, 7}}, Range@7,
    LegendLabel -> #,
    LegendLayout -> "Row"] & /@
  {"BrightBands", "DarkBands", "Rainbow", "LightTemperatureMap", "TemperatureMap", "ThermometerColors"} //
 Multicolumn[#, 3] &

BarLegend[{#, {0, 7}}, Range@7,
    LegendLabel -> Last@ColorData[#, "AlternateNames"] <> " (" <> ToString@# <> ")",
    LegendLayout -> "Row"] & /@
  {96, 109, 110, 111} //
 Multicolumn[#, 2] &

The ColorFunctionScaling and ColorFunction options will need to be set to use the scheme.

BarChart[Range@7,
 ColorFunctionScaling -> False,
 ColorFunction -> ColorData[{"TemperatureMap", {0, 7}}]
 ]

For a large number of discrete values a "Gradient" colour scheme can be used without restricting the number of contours.

For example

Plot[x, {x, 0, 41},
    ColorFunctionScaling -> False,
    ColorFunction -> ColorData[{#, {0, 41}}],
    PlotLabel -> #,
    Filling -> Axis,
    PlotLegends -> Automatic,
    ImageSize -> Medium
    ] & /@ {"TemperatureMap", "ThermometerColors"} //
 Column

Hope this helps.


Large action space for deep reinforcement learning I know that in normal Deep Reinforcement Learning(DRL) scenario, we learn a deep neural network to map current states to Q values. The number of the Q values (# of outputs of the neural network) is the same as the number of possible actions so that we can choose actions according to the associated Q values.

However, in this paper "Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads", the author uses states and actions as inputs. The network only outputs a single Q value (see pictures below). The $s_t$ is the state at time $t$ and $a_t^i$ is the $i^{\text{th}}$ action at time $t$ (you can ignore this but each action is a combination of the vector $c_t^1$ to $c_t^3$ ). $Q_t$ is just the $Q$ at time $t$ for action $i$. I am wondering why only a single Q value is learned? If this is the case, how do we determine the action for the next step?

I am wondering why only a single Q value is learned?

It is not a factor here, but it matches the Q-learning theory as written better to model $\hat{q}(s,a,\theta) \approx Q(s,a)$.

It is not numerically superior, and often less efficient than modelling a single network with multiple action outputs. However, it is clearly valid to approximate the action value function directly.

If the policy is separate to Q values, and you are using Q-learning to predict the values of that policy (as opposed to using Q-learning to control and find an optimal policy) then a single action value can be more efficient.

  If this is the case, how do we determine the action for the next step?

If you want to drive a best guess at an optimal policy from action values $Q(s,a)$, then typically you calculate

$$\pi(s) = \text{argmax}_a Q(s,a)$$

This rule is not dependent on a neural network architecture, but the implementation does vary:

If your neural network outputs all action values at once from the state, $f_{nn}(s) \rightarrow [\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2)...]$ then you perform a single forward pass of the network and find the maximising action inside a single output vector.
If your neural network outputs a single action value from a state, action $f_{nn}(s,a) \rightarrow \hat{q}(s,a)$ then you must run it for each possible action (typically in a minibatch) and find the maximising action across all outputs.

Unless you have some other way to generate the policies (e.g. you are using a policy network and policy gradients, the Q value is then for a baseline in something like an Actor-Critic agent), then you must collect all action values from each state in order to find the maximising action. There is no easy way around this, and the paper you link does not have a clever solution to it.

  It seems that people think that make action as input is an effective way to ease the large action space problem. [From comments]

It can be, but not in the way you suppose.

Instead, the nature of the action representation is critical. In your question, you said "you can ignore this but each action is a combination of the vector $c_t^1$ to $c_t^3$" (emphasis mine). However, this is not an ignorable detail, and is an important part of the paper.

By compressing the action representation into a small number of scalar traits, the authors allow the neural network to generalise between similar actions. For example if one action is represented by $c_t^1 = 0.5, c_t^2 = -0.3, c_t^3 = 0.25$ and another action is represented by $c_t^1 = 0.7, c_t^2 = -0.3, c_t^3 = 0.25$, then an
accurate value for the first action may result in at least an approximately accurate value for the second action, even if it has never been taken before in the current state.

Caveat: whether or not this is true or useful depends on the specifics of the problem being solved. It will not always be true, or be useful to transform the action space in this way. Actions with similar vectors need to be similar in some way that affects the outcome.

However, assuming it is true, the network design gives you an advantage in learning rate. Things that the agent learns about action values will be generalised more efficiently than a simpler network that enumerates each possible action. That makes the agent more sample efficient, it will need less experience to reach near optimal behaviour. It is this sample efficiency that the paper you linked is referring as an improvement to original DQN.

None of this changes the need to iterate over all possible actions when you want to evaluate the policy using DQN in this case.

TL;DR: The agent in the paper still has to evaluate all actions in each state in order to choose the best one. However, it does not have to try them all in order to discover which is best in the long term.


Techniques to clean the topics I have set of topics as follows.

"web based", "web-based" -> with surplus symbols
"technology","technologies" -> with singular and plural
"learned", "learnt", "learning" ->suffix stripping

Can you please recommend an accurate tool to perform aforementioned tasks.

For text processing, try using Python and the NLTK package.

For removing surplus symbols, you can use regular expressions. Install package 're' and use the built-in function re.sub to substitute symbols like '-' with empty character.

For suffix stripping, you can either use regular expressions again or use the built-in word stemming functionality in the NLTK package. This tutorial should help.


Absolute-value max pooling in 2D convolutional neural networks i am fairly new to machine learning, so this may be a silly question. if that is the case, I apologise in advance.
i am training a convolutional neural network on oceanographic images, which include both positive and negative anomalies. The implementation i have tested employs a number of 2D-convolutional layers, followed by max-pooling.
Max-pooling naturally has a bias towards positive values. Does it mean that negative anomalies in my images are going to be given less wight in the training? or is the sign difference accounted for in the convolutional filters?
instinctively, i would like to implement some sort of 'absolute-value max pooling', but since i have found no mention of this in the literature, should i assume this is not needed?

If you need to implement such an absolute value max pooling you can convert the output of the convolutional layer to an absolute value and then apply a maxpooling.
Moreover such thought is not necessary as maxpooling works with the output of the convolutional layers and is not influenced by the images directly. By which I mean that the convolutonal layers extract the most important features from the image be it positive or negative that plays a role in predicting a correct output
So for example if a low intensity pixel affects the identification more then it will be given a positive large weight compared to a pixel value that is less influential.


Fitting and transforming text data in training, testing, and validation sets I'm trying to implement a simple text classifier wherein the data is split into training (70%) and testing (30%) sets, but cross validation (k=10) to be performed on the training set.

My main concern here is the basis used for transforming a given set. I've seen tutorials where the whole data set was used to fit a Count/TFIDF vectorizer, but wouldn't this introduce bias when transforming the validation and testing sets since the previously mentioned sets were included in the whole data set? Or is the bias so small that it is acceptable to do so?

Within a fold, would it be better if the training set was used to fit a vectorizer and transform the validation set? And for testing, should the training + validation sets be used to fit a vectorizer to transform the testing set?

And on this note, should the validation set also be treated as "unseen" data similar to the test set?

A lot of the tutorials and notes online show ready made datasets, but there are certain cases with text data where values are dependent on a given set of documents (e.g. like how IDF is computed based on some group of documents) and that there are a lot more nitty gritty details when extracting features.

I think I'm just confused and am seeking a little clarification regarding this manner. Thanks!

It's best to leave the test data unseen. You can create a data transformation pipeline for your test data so that your test data will be transformed into a TFIDF vector after being passed through the pipeline. This way you get to know how well your model performs on unseen data.

And it's OK to transform the entire training data into a TFIDF vector and run the k-fold validation.


Effect of skewness in data I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?

What model are you using?

Ideally, one should always scale / normalize data before feeding them into some ML / statistical model. By using Z-scores you should be able to control for skewness of your variables.


Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays? I want to compare strings and give them score based on how similar the content is in them just like comparing two arrays in scipy cosine similarity.

For example :

string one : 'Pair of women's shoes'

string two : 'women shoes' pair'

Logically I would want a high score between the two strings. Is there any way to do so ? I am comparing array of strings with another array in a single column in my dataframe. I want to find similar rows this way. Can this be achieved ?

Levenshtein distance is computationally expensive and therefore slow for large datasets. For a faster method, you can use sci-kit learn's CountVectorizer or TfidfVectorizer to get frequencies of n-grams for each string. This will produce a frequency matrix, which you can then use as the input for sklearn.metrics.pairwise_distances(), which will give you a pairwise distance matrix. Note that with a distance matrix, values closer to 0 are more similar pairs (while in a cosine similarity matrix, values closer to 0 are less similar pairs).

See this blogpost for a nice tutorial on how to go about this.


Backpropagation: Relevance of the error signal of a neuron During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer:
\begin{eqnarray}
  \delta^l_j \equiv \frac{\partial C}{\partial z^l_j}
\tag{1}\end{eqnarray}

Basically, $\delta^l_j$ measures how much the total error changes when the input sum of the neuron is changed and is used for calculating the weights and biases of the neural network as follows:

\begin{eqnarray}
  \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j
\tag{2}\end{eqnarray}

\begin{eqnarray}  \frac{\partial C}{\partial b^l_j} =
  \delta^l_j.
\tag{3}\end{eqnarray}

Besides being useful for the calculation of the weights and the biases in the sense it makes it possible to reuse its value several times, is there any other reason why this definition is always brought up when addressing back propagation?

Let me explain one thing very clearly "NEURAL NETWORK ARE VERY SIMPLE STRUCTURES AND YET VERY EFFICIENT ALGORITHM WHICH GIVES AMAZINGLY GREAT RESULTS AND IF YOU SEE CLOSELY NEURAL NETWORKS ARE ACTUALLY BIG COMPOSITE FUNCTIONS WHICH REQUIRE YOU TO CALCULATE GRADIENTS SO THAT YOU CAN OBSERVE HOW THE OUTPUT VARIES ACCORDING TO YOUR INPUT WEIGHTS(That is what is backpropagation achieved using gradient descent) THAT IS SIMPLY THE GOAL I.E. TO CALCULATE DERIVATIVES OF THE TWO EQUATIONS YOU HAVE MENTIONED (2) AND (3)".

Now as you stated you can calculate its value and reuse it i.e. eq(1) because eq (1) here gives you the error at a neuron j in layer 'l' with respect to that neuron's weighted sum of inputs it is useful and necessary for calculation of weights when the loss is calculated at the'Lth layer(last layer)' it is then backpropagated to the L-1 th layer where gradient is calculated and then weights are updated it is the eq(1) that allows you to compute the gradients of a neuron j in layer 'l' and then the weight matrix of that neuron j is updated so essentially eq(1) allows you to use eq(2) and eq(3) to optimize your weights in order to minimize your loss function.

One more thing if you calculate its derivative that is of eq(1) its partial derivative it will be: first-rate of change of cost function w.r.t to activation function * rate of change of activation function w.r.t to weighted sum of inputs * rate of change of weighted sum of inputs w.r.t to weights of your neuron so this is essentially a chain in itself which gets longer and longer as the loss is backpropagated through l-1,l-2,l-3.... layers and so on, thus eq(1) is actually calculated using the first two in the chain stated above which is then used to calculate the rate of change of weights w.r.t to your loss function. This is why whenever you read about backprop this equation is bound to be seen as it is kind of connection mechanism that allows you to relate your loss with neurons in the subsequent layers during backpropagation and subsequently update their weights. Hence according to me, eq(1) in itself is the true essence of Backpropagation using gradient descent.


How to determine if a company decision was successful or not? I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not.
I have done some research and saw that A/B testing might be a way to do this but A/B tests can only be carried out having 2 groups (control and experiment), and in this case all I have is past and "after the decision" data.

Can the statistics behind AB testing be used in this case?

The data I have is in the for of sales per day per article only for one company before and after the decision (only one decision).

Example:

sales for articles X and Y from the 2016-01-01 to 2016-02-01.
The decision was to lower discounts on articles A and B on 2016-01-15, so I want to know if sales decreased or not after this decision.

You could use time series approach to model the "what if not" scenario and compare it with your values after the new program introduction. Check the causal_impact packet from Google and you might also find helpful this tutorial for probabilistic programming. http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb


Significance of Object-Oriented Programming (OOP) in Data Science Can someone please explain to me the role of Object-Oriented Programming (OOP) and Object-Oriented Design (OOD) in Data Science? I am from a non-computer science background. Do I need to learn these as well to become a Data Scientist? Also, please tell me if I should be learning Python or R for the same.

It actually depends on the role you get as a Data Scientist. If you have to write production-quality code at a large software company, then you need to be knowing the basics of Object-Oriented Programming (OOP). Object-Oriented Design (OOD), however, is something you need not necessarily know in a Data Science role. Learning OOD in case you plan a switch to software engineering roles in future is one thing you can consider, though.
Regarding the choice of language for doing Data Science, I would suggest you  prefer Python over R as it's more versatile and a much more general purpose language. While R, on the other hand, is limited only to applications in statistical programming.


Data Cleaning without pandas How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the dataset, correcting spelling mistakes, inconsistencies in data formatting, null entries etc.

I work in a restricted environment and I've heard of others that are worse than mine, but no pandas??? Geez, that's rough!

You could do this by reading through the file, line-by-line, column by column and then applying a series of your own functions to test/fix the data in each cell. But you would have to have all of the defined needs well ahead of time and then write the methods to get each data point in a methodical fashion.

So it can be done but you're doing a massive end-run around pandas to get there.


Customize loss function for Music Generation LSTM (?) I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract the data out of every song. The data in every midi song are organized as a series of discrete event messages.

I plan to have my input of the LSTM for each time-step as a vectorized event of three variables:

x_t = [note, velocity, timestamp]

note: categorical variable, since it can take only one integer value in the range [0:127] which corresponds to a pitch.

velocity: I think it is categorical too, as it also takes one integer value in the range [0:127] which corresponds to the force with which the note is played.

timestamp: the continuous variable that corresponds to the number of seconds (mostly milliseconds) or ticks that have passed between the current and previous events.

My question is since I have a combination of categorical and continuous variables to predict:

Is it ok to one-hot-encode note and velocity and keep timestamp as is?
Should I further standardize timestamp?
Can I use a customized loss function (sum of cross-entropy loss for note/velocity plus mean squared error for timestamp), or it would mess up my model?

Please, I feel a bit lost and need your help/suggestions.

Thank you!!!

That's not a good way to represent symbolic monophonic music.
The biggest problem is that time should be modelled as a discrete and not a continuous variable. The reason for that is that music with notes that are off the beat sounds like crap. E.g if the tempo indicates that one beat is 250 ms long and one beat randomly is only 200 ms long it will sound very bad.
Instead, fix the tempo so that, say one quarter note is 125 ms long and have your time variable be an integer factor of that. You could cut the range to [1, 16] so that the shortest note is 125 ms long and the longest 2 000 ms (125 * 16).
Then I would skip the velocity because it is very hard to model. I don't think even state-of-the-art models incorporates it and most music sound fine even without it.
For pitches, I would cut down their range to three or four octaves. Most MIDI songs doesn't use more scales than that and it makes it much easier for your model to generalize.
So with four octaves, you are left with two variables $t \in [1,16]$ for time and $p \in [1,48]$ for pitch. You can either assume these are independent (which they aren't, but the generated music might still sound good!) and model them separately, giving you a sequence of pairs of discreete values:
$$
(t_1, p_1), (t_2, p_2), ...
$$
Or you could multiply $t$ and $p$ so that each input to the network is an integer in the range [1,16*48]:
$$
t_1p_1, t_2p_2, ...
$$
You could also interleave the input so that every other item in the sequence contains timing and pitch information:
$$
t_1, p_1, t_2, p_2, ...
$$
I don't know which way is the best - you'll have to experimetn.
I'm fairly certain it won't matter which loss function you choose. But I would go with crossentropy because it usually works well with discrete variables.


Any efficient way to build non-linear regression model for polynomial features? I am trying to understand how crime frequency affect house price in certain area. To do so, I started with Chicago crime data and zillow real estate data. I want to understand the relation between house price and crime frequency and top 5 crimes in certain areas. Initially, I build up model for this specification, but it wasn't very meaningful to me. Can anyone enlighten me what should I do? any efficient approach to train regression model for potential relation between house price and crime frequency in certain areas? any heuristic idea to move forward?

example data snippet:

here is the merged data that includes annual house price and top crime type in certain areas:

Here is reproducible example data snippet

my attempt

so here is my attempt to fit regression model with above reproducible example data:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

sc_x = StandardScaler()
sc_y = StandardScaler()
X = sc_x.fit_transform(X_feats)
#y= sc_y.fit_transform(y_label)
y = sc_y.fit_transform(y_label .reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_

but to me, above model wasn't that efficient and needs to be done something more. I think I have to use non linear regression model for those polynomial features, and I am not sure to get this done.

Can anyone point me out how to build correct model for house price prediction over type of crimes and frequencies in certain areas? any idea? Thanks

Goal:

I want to build regression model to predict house price based on crime frequencies and types in certain areas. How can I get modeling the relationship between house price and crimes in certain areas? any thoughts?

You are probably finding yourself in one of the most interesting problems in data science, the part which is more art than science.

I will give you some ideas that can give you hints on how to solve this problem:

Prices, Salaries and other variables who have information on "accumulables" many times have a distribution which is skewed to the left (many individuals have a little, a few have a lot), what is reccomended to do is to take logarithm of it. Your new variable should be $Ln(Y)$, with this, you will close the gap between areas with greater avg_price and areas with lower avg_price. When that happens you find a less-skewed-normal-like distribution of your $Y$ variable.
The idea of taking Logarithm also applies to the $X$ variables you have (because crimes also accumulate in certain areas).
The Standard Scalling is not necessary when you run a linear regression, because the relativeness of the variable is not influencial in the regression:

The regression $Y = \alpha_0 + \alpha_1X_1+...+\alpha_nX_n$ (no
    scalled) is mathematically equivalent to $Y = \beta_0 +
\beta_1Z_1+...+\beta_nZ_n$ (scalled)

If you want to use other models, you data seems suitable for it, maybe a regression tree or XGBoost might work well on your problem.

I would bet that getting Logarithm in the avg_price, in some exogenous variables and not scalling would get better results for you.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

X = log(X_feats)
y = log(y_label.reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_


Why is random forest an improvement of decision tree? Let's assume that we have a binary classification problem, and we built a decision tree on our data set.

Assuming that we have 5 features, then the decision tree, in the first step, will choose the best feature of the 5, and on this feature it will choose the best threshold in order to split the data set, and then continue to make the tree deeper etc. The definition of best is the lowest classification error.

My question is: Since the decision tree, on each step, chooses the best feature to split on, and the best threshold to split on, then why random forest (which is many decision trees), is an improvement of decision trees? Shouldn't a decision tree be sufficient?

UPDATE

I more mean it like: If you have a decision tree classifier, and a random forest classifier with the same parameters, when possible, (max_depth, number of children etc), will the decision tree classifier score the same on the training set, with the random forest classifier?

It comes down to overfitting as you scale. Decision trees tend to overfit as they grow deep. After every split there will be fewer and fewer samples for the next split to work with. Fewer samples means that risk of splitting on noise increases.

Random forest avoids the overfitting problem of decision trees by instead scaling by adding more trees instead of building one big tree. Averaging the outputs of the trees in the forest means that it does not matter as much if the individual trees are overfitting.

Regarding your update. No, they will not score the same. Random forest will not have just one decision tree. It has several and divides the features into random subsets for each tree to be trained on. So even if the size of the decision trees in random forest would be the same as a single decision tree, the features they are trained on would not be.

But if you ask what happens if we take a random forest, only use one tree and train it on the same features as a single decision tree of the same size, then yes they would be one and the same.


Prediction on timeseries data using tensorflow I have an input and output of below format:

(X) = [[ 0  1  2]
       [ 1  2  3]]

  y   = [ 3  4 ]

It's timeseries data. The task is to predict the next number. Basically, the input was crafted by the below snippet:

 def split_sequence(arr,timesteps):
     arr_len = len(arr)
     X,y = [],[]
     for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component =  arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

     return np.array(X), np.array(y)

Now, I would like to train the model on the input and predict the next number. For instance, x = [81,82,83] and the predicted output would be y = 84. I learned how to do this in keras. However, I would like to try to do this in tensorflow as well.

Below is the code in tensorflow:

 # Data generator
 def generate_batch(X,y,batch_size):
    m = X.shape[0]
    indexes = range(m)
    n_batches = m // batch_size
    for batch_index in np.array_split(indexes,n_batches):
       yield X[batch_index],y[batch_index]

 # parameters
 n_inputs = 3
 n_epochs = 1000
 batch_size = 40
 learning_rate = 0.01
 n_steps = 3

 # generate the input and output using split_sequence method
 input, output = split_sequence(range(1000),n_steps)

 # Define the input variables
 X = tf.placeholder(tf.int32,shape=(None,n_inputs),name='X')
 y = tf.placeholder(tf.float32,shape=(None),name='y')
 theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

 # predictions and error
 y_predictions = tf.matmul(X,theta,name='predictions')
 error = y_predictions - y
 mse = tf.reduce_mean(tf.square(error),name='mse')

 # train the model
 optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
 training_op = optimizer.minimize(mse)

 init = tf.global_variables_initializer()

 with tf.Session() as session:
   session.run(init)
   for epoch in range(n_epochs):
      for X_batch,y_batch in generate_batch(input,output,batch_size):
         if epoch % 10 == 0:
             print('epoch',epoch,'MSE=',mse.eval())
         session.run(training_op,feed_dict={X:X_batch,y:y_batch})

To be honest, I am completely stuck with the below error:

You must feed a value for placeholder tensor 'X' with dtype float and shape [?,3].

My input is an integer, so that was the reason behind defining:

 X = tf.placeholder(tf.int32,shape=(None,n_inputs),name='X')

Can someone help me fix this? Also, if I wanted to add bias variable will I be able to achieve for the above input?

The error is caused by this line:

print('epoch',epoch,'MSE=',mse.eval())

This happens because the tensor mse also depends on the placeholders X and y. One way to fix this would be to change the training loop to be:

for X_batch,y_batch in generate_batch(input,output,batch_size):
     mse_val, _ = session.run([mse, training_op],feed_dict={X:X_batch,y:y_batch})
     if epoch % 10 == 0:
         print('epoch',epoch,'MSE=',mse_val)

Also you will need to switch X back to tf.float32 since tf.matmul is not compatible with int and float. The data will automatically be casted once you feed it in.

To add a bias variable, you can do it similarly to how you define theta.

b = tf.Variable(0.0, dtype=tf.float32, name='b')
...
y_predictions += b


R problem with fraction numerals Why do I have a non-zero result in the following calculation:

  0.11-0.1-0.01
  [1] -5.20417e-18

Seems to be an easy and akward question, but I will be very greatful for any help!)

You can multiply with (X==)10/100/1000/... to size of precision you want. After performing multiplication to both numbers, Do your operation(-, /, ...) You will get the result.