Similarity between 2 statistical distributions Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in shapes(like one is symmetric and the other is skewed) etc. and gives similarity between [0,1]?
I am using Hedges' index for the same but it does not give a similarity index between 0 and 1. It can be greater than 1 as well, so it is difficult to interpret it.
Also, no pattern of the data is known beforehand, if it helps in any way for the answer.

One method is Kolmogorov-Smirnov test. Kolmogorov-Smirnov test checks whether two samples are drawn from the same continuous distribution where sample sizes can be different. It's p-value is close to 0 when two samples follow the same distribution and close to 1 when they do not follow the same distribution. So you can use 1 - (p-value) as a similarity metric.
import numpy as np
from scipy.stats import ks_2samp

np.random.seed(52)

n1 = 200
n2 = 300

mu_1 = 5
mu_2 = 5.1

sigma_1 = 0.3
sigma_2 = 0.2

sample_1 = np.random.normal(mu_1, sigma_1, n1)
sample_2 = np.random.normal(mu_2, sigma_2, n2)

result = ks_2samp(sample_1, sample_2)

print(result.pvalue)

1.4998994601889137e-08

Note that there are also other methods such as Bhattacharyya distance, Kullback–Leibler divergence. Some implementations for Kullback-Leibner can be found also here.


ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames I want to create a new variable for the dataframe details called lower after iterating through multiple data frames.

list1 is a list of string values of a column named variable_name in the details.
vars_df is another data frame with 2 columns, namely variable_name and direction. Both columns contain with string values. vars_df.shape = (19,2)
Some values of variable_name in vars_df are present in list1 as well as data_set.
data_set.shape = (32,107).df.shape = (96,1)

The following code, which aims to do the above:
    def get_value(df,list1):
        if ((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up'):
            data = data_set[df['variable_name']]
        else:
            data = data_set[df['variable_name']]-20
            if data < 0:
               data = 0
            else:
               data = data
       return data

details['lower'] = details.apply(get_value,list1=li,result_type='expand',axis=1)

produces this error:
ValueError: ('The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().', 'occurred at index 0')
I referred the question.Truth value of a series is ambiguous Still I can not rectify the error

(Assuming the error comes from the 2nd line)
This question on StackOverflow has a great answer to your problem
Essentially, change
 if ((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up'):

to this
if (((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up')).any():

or to this, if all of the values should be True
if (((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up')).all():

Hope this helps :)


MNIST - Vanilla Neural Network - Why Cost Function is Increasing? I've been combing through this code for a week now trying to figure out why my cost function is increasing as in the following image. Reducing the learning rate does help but very little. Can anyone spot why the cost function isn't working as expected?

I realise a CNN would be preferable, but I still want to understand why this simple network is failing.
Please help:)

import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

import matplotlib.pyplot as plt

mnist = input_data.read_data_sets("MNIST_DATA/",one_hot=True)

def createPlaceholders():
    xph = tf.placeholder(tf.float32, (784, None))
    yph = tf.placeholder(tf.float32, (10, None))
    return xph, yph

def init_param(layers_dim):
    weights = {}
    L = len(layers_dim)

    for l in range(1,L):
        weights['W' + str(l)] = tf.get_variable('W' + str(l), shape=(layers_dim[l],layers_dim[l-1]), initializer= tf.contrib.layers.xavier_initializer())
        weights['b' + str(l)] = tf.get_variable('b' + str(l), shape=(layers_dim[l],1), initializer= tf.zeros_initializer())

    return weights

def forward_prop(X,L,weights):
    parameters = {}
    parameters['A0'] = tf.cast(X,tf.float32)

    for l in range(1,L-1):
        parameters['Z' + str(l)] = tf.add(tf.matmul(weights['W' + str(l)], parameters['A' + str(l-1)]), weights['b' + str(l)])
        parameters['A' + str(l)] = tf.nn.relu(parameters['Z' + str(l)])

    parameters['Z' + str(L-1)] = tf.add(tf.matmul(weights['W' + str(L-1)], parameters['A' + str(L-2)]), weights['b' + str(L-1)])
    return parameters['Z' + str(L-1)]

def compute_cost(ZL,Y):
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = tf.cast(Y,tf.float32), logits = ZL))
    return cost

def randomMiniBatches(X,Y,minibatch_size):
    m = X.shape[1]
    shuffle = np.random.permutation(m)
    temp_X = X[:,shuffle]
    temp_Y = Y[:,shuffle]

    num_complete_minibatches = int(np.floor(m/minibatch_size))

    mini_batches = []

    for batch in range(num_complete_minibatches):
        mini_batches.append((temp_X[:,batch*minibatch_size: (batch+1)*minibatch_size], temp_Y[:,batch*minibatch_size: (batch+1)*minibatch_size]))

    mini_batches.append((temp_X[:,num_complete_minibatches*minibatch_size:], temp_Y[:,num_complete_minibatches*minibatch_size:]))

    return mini_batches

def model(X, Y, layers_dim, learning_rate = 0.001, num_epochs = 20, minibatch_size = 64):
    tf.reset_default_graph()
    costs = []

    xph, yph = createPlaceholders()
    weights = init_param(layers_dim)
    ZL = forward_prop(xph, len(layers_dim), weights)
    cost = compute_cost(ZL,yph)
    optimiser = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    init = tf.global_variables_initializer()

    with tf.Session() as sess:
        sess.run(init)

        for epoch in range(num_epochs):
            minibatches = randomMiniBatches(X,Y,minibatch_size)
            epoch_cost = 0

            for b, mini in enumerate(minibatches,1):
                mini_x, mini_y = mini
                _,c = sess.run([optimiser,cost],feed_dict={xph:mini_x,yph:mini_y})
                epoch_cost += c
            print('epoch: ',epoch+1,'/ ',num_epochs)

            epoch_cost /= len(minibatches)
            costs.append(epoch_cost)

    plt.plot(costs)
    print(costs)

X_train = mnist.train.images.T
n_x = X_train.shape[0]
Y_train = mnist.train.labels.T
n_y = Y_train.shape[0]
layers_dim = [n_x,10,n_y]

model(X_train, Y_train, layers_dim)

Tensorflow's softmax function only works if the number of batches are in the rows and the output in the columns. If these are reversed, then you need to transpose the tensors in the cost function.


Does matrix shape for training/testing sets have to be in a particular order? I've noticed that in the Andrew Ng Deep Learning course that for image analysis he always has X_train matrices in the shape of [height, width, 3, num_inputs], or, if flattened, [height X width X 3, num_inputs]. He also has his y_train as [1, num_inputs]. To me, it is more intuitive to flip these so that X_train is [num_inputs, height X width X 3] and y_train is [num_inputs, 1]. Is there any motivating reason or justification that it has to be the way he does it or is it just preference? Is this a standard or does it vary?

In practice

Andrew Ng seems to be using the convention on the Theano framework. If you had 10 colour images, each 100 pixels high and 200 pixels wide, Theano models would expect input of the form:

  (batch size, input channels, input rows, input columns)

and so dimensions: (10, 3, 100, 200). The three is because of the three RGb dimensions in a colour image.

Tensorflow on the other hand reverse this order to instead use:
(num_obs, height, width, channels) - for the same example as above this becomes:

  (batch size, input rows, input columns, input channels)

meaning the dimensions of the input should be (10, 100, 200, 3).

Keras works with Tensorflow and Theano and supports both conventions by simply allowing the user to set the position in which the number of channels are given. This can be set in a config file or using a specific environment variable upon setup. Have a look at the relevant documentation.

(Possible) justifications

There are different points of view on what is more natural. If you come from a computer vision arena (or image processing in general), then libraries like OpenCV use a coordinate system that has (0,0) in the top left of an image, and you specify a single pixel by given the vertical movement, then the horizontal movement from the origin. This means you give a height, then a width. One justification for this might be that many computer vision algorithms (e.g. colour filters) focus on difference across the channels of an image, but it is just convention.

In general graphing and mathematics, it is common to give the X coordinates first, then the Y, which means talking about horizontal movements, then vertical. Linear algebra in general is a good example of this.

All this being said, the best advice is to make sure you use the correct dimensions according to the documentation of the software you use. If you make your own software - you get to choose! (But make sure to document it!)


What is a LB score in machine learning? I was going through an article on kaggle blogs. Repeatedly, the author mentions 'LB score' and 'LB fit') as a metric for effectiveness of machine learning (along with cross validation (CV) score).

With a research for the meaning of 'LB' I spent quite a bit of time, I realised that generally people directly refer it as LB without much background.

So my question is - What is a 'LB'?

In the context of Kaggle, it means LeaderBoard (emphasis mine).


Problem to display decision tree: InvocationException: GraphViz's executables not found in Python scikit-learn? I try to display Decision Tree in Jupyter Notebook in Python.

This is my code:

X = data.drop(["Risk"], axis=1)
y = data["Risk"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
from sklearn.tree import DecisionTreeClassifier
klasyfikator = DecisionTreeClassifier(criterion = "gini", random_state=0, max_depth=4, min_samples_leaf=1)
klasyfikator.fit(X = X, y = y)

dot_data = StringIO()
export_graphviz(klasyfikator, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True,feature_names = X.columns,class_names=['0','1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('diabetes.png')
Image(graph.create_png())

And by using above code I have Error:

InvocationException: GraphViz's executables not found

What can I do to display decision tree ? Where is a mistake ?

You need to add the graphviz/bin folder to PATH. See the tutorial here.


Numerical example of Confusion in understanding learning rate in xgboost I fail to understand as to how learning rate is used in XGBoost?
Can anyone explain using a numerical example?

Each iteration is supposed to provide an improvement to the training loss.
Such improvement is multiplied with the learning rate in order to perform smaller updates.

Smaller updates allow to overfit slower the data, but requires more iterations for training.

For instance, doing 5 iteations at a learning rate of 0.1 approximately would require doing 5000 iterations at a learning rate of 0.001, which might be obnoxious for large datasets.

Typically, we use a learning rate of 0.05 or lower for training, while a learning rate of 0.10 or larger is used for tinkering the hyperparameters...


Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.) There is a business with about 100 computers used by employees, and one high-powered server.  It's called a "distributed system" by the system architect.  It uses Distributed Shared Memory (DSM).  There's also middleware, and the server is hosting Virtual Machines (VMs) which are running the applications that the employees see.

The question is: does the DSM come from physical memory that the server is sharing, creating virtual shared memory, or does the memory come from those 100 computers (or both)?  Aside from the possible speculation here, could portions of the physical memories of the 100 computers even be utilized as DSM in this setup?

Thanks

(Please note that distributed systems and shared memory are course topics in data science programs. That's why I'm posting this here. Data science isn't just data mining - it's a lot more.  The nuts and bolts behind doing it are included.)

Either the host server or the clients can create DSM, or both.  It's likely that a server hosting the applications would manage this.  The host server may create shared memory, or it may direct the clients to do this, and share the memory among themselves, preserving its own memory.

Distributed Shared Memory is an abstraction.  A "server" creates shared memory, and a "client" makes a copy of it in its own memory.  The O/S or middleware keeps them in sync. A "client" for purposes of working of host server, may be a "server" regarding creating shared memory.  This client/server terminology is used in both contexts. When you read about shared memory you will often see the terminology used this way - the computer initiating the creation of DSM (distributed shared memory) is the "server" for that purpose.


Celsius to Fahrenheit conversion simple Neural Network This common example is a simple way to dabble into the most basic neural network (i.e. two neurons, one input and one output).

I have been playing around with this using TensorFlow and keras, and I am wondering, it should be very simple for the NN to learn the weight (1.8) and bias (32) with little to no effort.

However, I find that (albeit, using only 13 examples) after running for 500 epochs I can get a weight of 1.8 but the bias is always off (29.2, for example). Obviously there is no activation function, given this linear problem.

My question is, for such a simple case, will a NN be able to solve this problem exactly?

Hard to say without seeing code, but I can imagine several reasons:

You are using an activation function. You want 'linear' or no activation
Learning rate is much too low
Wrong network (should be one dense layer with one weight)
Poor batch size

This works pretty instantly:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Nadam

model = Sequential([Dense(1, activation=None, input_shape=[1])])
model.compile(loss='mean_squared_error', optimizer=Nadam(lr=0.1), metrics=['mean_absolute_error'])

F = list(range(-200,200))
C = [(f - 32) * 5 / 9 for f in F]

model.fit(F, C, batch_size=16, epochs=50)

print(model.predict(np.array([300])))
print(model.layers[0].get_weights())

[[148.88893]]
[array([[0.5555556]], dtype=float32), array([-17.777737], dtype=float32)]

300 F -> 148.8889 C, good, and it has learned the conversion is C = 5/9 * F - 17.7778, or C = 5/9 * (F - 32)


add labels bar chart There is this dataframe:
print(df)

df.plot.bar(figsize=(10,5),fontsize=14,)
plt.title('counts',fontsize=15)

plt.show()

barh:
df.plot(kind='barh')

How can I add value labels on this bar and barh chart?

Unfortunately, adding text to the plot must be done manually. See the example below.
fig, ax = plt.subplots(figsize=(10,5))
df.plot.bar(ax=ax, fontsize=14)
for i, v in enumerate(df['counts']):
    ax.text(i + .25, v + 3, str(v))

Similarly, you can do for the horizontal barplot.
fig, ax = plt.subplots(figsize=(10,5))
df.plot.barh(ax=ax, fontsize=14)
for i, v in enumerate(df['counts']):
    ax.text(v + 3, i + .25,  str(v))


Distinguising features of linear vs, non-linear machine learning models (algorithms) What are some examples of linear and non-linear machine learning models (algorithms) for purposes of comparison between the two categories? Which are the parameters (or scalars in a linear algebraic sense) and which are the predictors/factors (or vectors in a linear algebraic sense again)?

Actually, there are many linear and non-linear machine learning algorithms. Selecting a right algorithm highly depends on your data-set and the nature of your data. for instance, in machine-learning era, it was customary to estimate functions by assigning a typical model to the problem and reducing the error by predicting the appropriate coefficients using a cost function for regression tasks. In such cases, you should have bias-variance trade-off. That means you should not fit the data nor miss it. you should find a good estimate, a good model that has generalization capability. For doing all of these you have to choose features which help you describe the problem better for making a model. In deep-learning, we usually do not have this trade-off. If you increase the size of your training set, you can almost be sure that you can have better results. In machine-learning, you can always be sure that by making complex non-linear models, you overfit your data while using complex deep-learning models does not necessarily mean that if you employ generalization techniques which avoid overfitting. For choosing a right model, it is customary to use a simple linear model and make it complicated step by step. Problems which their inputs have numerous features, you can not see and visualize the data-set to check whether they are linearly separable in that space or not. Consequently, beginning with a simple model and making it complex step by step is a logical solution. For measuring performance there are different solutions that all depend on your goal. Take a look at here.


Word2Vec: Using pre-trained models I am unable to find details for pretrained word2vec models. If someone is about to use a pre-trained model should it be clear what kind of pre-processing was done before the model was trained? Such as if lower cased was applied or removal of specific stopwords?

Glove: Is not referring to any kind of that information except of recommending stanford tokenizer (splitting words).
Google pre trained model: Doesn't give any info for pre-processing of the pretrained model. Refers in a sentence to use a script for the wikipedia training data, but do not mention how the google news were preprocessed before trained.

And in other cases of pretrained models I cant find this kind of information. Someone could assume that there is a specific way of pre-processing for word2vec but I remember that the instructor of stanford NLP course, Richard Socher, that some people are removing stopwords and some others do not.

I have also noticed that people are using different pre-processing methodology and same pre-trained model for the kaggle competition Bag of Words Meets Bags of Popcorn which doesnt make sense for me.

Any one can give some insights?

I think in a lot of cases, when people are using pre-trained models, they force their words to lower and singular case and then test to see if the lowercase and singular (not plural) version were trained. Sometimes, you have to check yourself like A.M. Bittlingmayer said.

To provide some insight into why sometimes people remove stopwords versus not --- I think it has to do with whether they want to value infrequent words more or less. Because you don't want to value too highly very frequent words, the algorithm 'punishes' extremely frequent words. So, one could argue that because this already happens, you don't need to remove stopwords since they will be 'punished' anyways. So, not everyone removes stopwords.


how to find holiday effect on revenue? I have 2 datasets from 2013-2017 for each day.
a) Revenue generated by Locations and date. b) Holiday name and date

I would like to know how each holiday is impacting the revenue by location. I am confused about what method to use! Should 3-Sigma rule or Bollinger plot be enough or should I try spline regression or something else? FYI I am using python for this analysis.

It might be not the best approach, but without knowing the data its hard to be precise.
If you have no ideas where to start, I can suggest trying this things first.

If you want to get the "impact" of the holiday, try to one-hot encode "holiday" variable, and use some simple model that is interpretable. You need to undestand how each holiday-feature impacts the revenue, maybe linear regressiong is a good way to start and move on to more advanced algo`s.
You can try to play with Facebook`s Prophet - its an package for timeseries predictions, wich shows the daily impacts by default. Here is the link: https://research.fb.com/prophet-forecasting-at-scale/ . And link to python package: https://github.com/facebook/prophet

Additional post about Propet and transactios forecasting: https://www.kaggle.com/armamut/predicting-transactions-fb-prophet-tutorial


Why does Keras need TensorFlow as backend? Why does Keras need the TensorFlow engine? I am not getting correct directions on why we need Keras. We can use TensorFlow to build a neural network model, but why do most people use Keras with TensorFlow as backend?

This makes more sense when understood in its historical context. These were the chronological events:

April 2009 Theano 0.1 is released. It would dominate the deep learning framework scene for many many years.
June 2015 Keras is created by François Chollet. The goal was to create an abstraction layer to make Theano easier to use, enabling fast prototyping.
August 2015 Google hires François Chollet.
November 2015 Tensorflow is released by Google, with much inspiration from Theano and its declarative computational graph paradigm.
December 2015 Keras is refactored to allow for pluggable backend engines, and now it offers backend implementations for Theano and Tensorflow.

Other backends were later supported by Keras (CNTK, MxNet), but they never got much traction.
Time passes by and the overlap between Tensorflow and Keras grows. Tensorflow ends up duplicating many of the functionalities in Keras (apart from the multiple APIs within Tensorflow that also had big overlaps).

September 2017 Theano is discontinued.
November 2017 Keras is bundled with Tensorflow as tf.keras. From this point on there are 2 different Keras: the one bundled with Tensorflow and the one that supports multiple backend engines. Both are maintained by the same people and are kept in sync.

At some point, the roadmap for Tensorflow 2.0 is defined, choosing to pursue an imperative model like PyTorch. The person leading the Tensorflow API refactoring is François Chollet. This refactoring included a reorganization of the functionality to avoid duplications.

November 2018 some crucial functionalities of Tensorflow are to be moved to tf.keras, generating a heated debate
September 2019 Keras 2.3 is announced to be the last release of the multi-backend version of Keras

Now, THE ANSWER to your question: Tensorflow is the most used Keras backend because it is the only one with a relevant user base that is under active development and, furthermore, the only version of Keras that is actively developed and maintained is one with Tensorflow.
So, summing up:

At the beginning of Keras, the overlap with Tensorflow was small. Tensorflow was a bit difficult to use, and Keras simplified it a lot.
Later, Tensorflow incorporated many functionalities similar to Keras'. Keras became less necessary.
Then, apart from the multi-backend version, Keras was bundled with Tensorflow. Their separation line blurred over the years.
The multi-backend Keras version was discontinued. Now the only Keras is the one bundled with Tensorflow.

Update: the relationship between Keras and Tensorflow is best understood with an example:
The dependency between Keras and Tensorflow is internal to Keras, it is not exposed to the programmer working with Keras. For example, in the source code of Keras, there is an implementation of a convolutional layer; this implementation calls package keras.backend to actually run the convolution computation; depending on the Keras configuration file, this backend is set to use the Tensorflow backend implementation in keras.backend.tensorflow_backend.py; this Keras file just invokes Tensorflow to compute the convolution


In what way is recurrent neural network state "hidden"? Recurrent neural networks have hidden state denoted $h$. Why is the state considered "hidden"? It's clear to me what purpose the state itself serves, but I can't figure out why it is "hidden". To me, the state seem perfectly "visible" - not "hidden".

The "hidden" layers and state in a neural network are those that do not have direct input or output with respect to the function being learned.
I don't think you need to read anything deep into the analogy. The term "hidden" was, according to Geoffrey Hinton, coined by him partly because he liked the mysterious sound of it (from hidden markov models). He freely admits that it is partly an awkward name since in the case of deep neural networks, the data in the hidden layers is not inaccessible or unknown.
I heard him state this in one of the lectures in a now closed Coursera course.
Sadly I am not able to find a current reference to this in his own words. I have found a reference to the same story in the book Backpropagation: Theory, Architectures, and Applications.


Can Machine Learning be applied in software developement I'm from programming background. I'm now learning Analytics. I'm learning concepts from basic statistics to model building like linear regression, logistic regression, time-series analysis, etc.,

As my previous experience is completely on programming, I would like to do some analysis on the data which programmer has.

Say, Lets have the details below(I'm using SVN repository)

personname, code check-in date, file checked-in, number of times checkedin, branch, check-in date and time, build version, Number of defects, defect date, file that has defect, build version, defect fix date, defect fix hours, (please feel free to add/remove how many ever variables needed)

I Just need a trigger/ starting point on what can be done with these data. can I bring any insights with this data.

or can you provide any links that has information about similar type of work done.

Definately - Yes.
Good question.
Was thinking about it myself.

(1) Collect the data.
The first problem you have: gather enough data.
All the attributes you mentioned (date, name, check-in title/comment, N of deffects etc) are potentially useful - gather as much as possible.
As soon as you have a big project, a number of developers, many branches, frequent commits and you have started collecting all the data, you a ready to go further.

(2) Ask good questions.
The next question you should ask yourself: what effect are you going to measure, estimate and maybe predict.
Frequency of possible bugs? Tracking inaccurate "committers"? Risky branches? Want to see some groups of users/bugs/commits according to some metrics?

(3) Select the model.
As soon as you have the questions formulated, you should follow the general approach in data science - extract needed features in your data, select appropriate model, train you model and test it, apply it. This is too broad process to discuss it this thread, so please use this site to get right answers.


Scikit-learn decision tree in production I'm working at building a decision tree model that will be used in production.

In documentation here pickle is used to serialize the model however the concerns about this technique make me think there's maybe a better solution to export a model to production.

  pickle (and joblib by extension), has some issues regarding
  maintainability and security. Because of this,

  Never unpickle untrusted data as it could lead to malicious code being executed upon loading.
  While models saved using one version of scikit-learn might load in other versions, this is entirely unsupported and inadvisable. It
  should also be kept in mind that operations performed on such data
  could give different and unexpected results.

So my question is : Using scikit-learn, is there a safe and convenient technique to export the model into production.

PS: Converting the dot data to a python function can be a solution but i'm surprised there's no built-in solution for this.

Persisting the model parameters is the only out of the box solution in sklearn. You should ensure that:

Your model is reproducible from data
The versions of dependencies used (incl sklearn) is locked
Your QA system checks correctness of your model

If you have these things down then the possible changes in model persistence in sklearn is a non-issue. If it happens, you just update sklearn in a controlled manner, retrain and deploy as usual.
And you avoid a whole range of other problems, for instance bugs in sklearn/numpy/whatever catching you by suprise. Or new data needs new models trained and deployed.

You can of course build your own persistence mechanism but you then need to maintain the compatibility. Better spend the effort one the above.


Class imbalance strategies When dealing with the class imbalance problem in a binary classifier, there are three ways I know of to address it: over-sampling, under-sampling and using cost-sensitive methods.

Are there any guidelines, rules of thumb or general strategies to choose among these methods? A possible answer would be: over-sample when the positive class has more than 100 instances (I just made it up).

It's hard to put a general rule with this kind of methods, they depend heavily on the data at hand. You should know pros and cons of each, try all methods and see which performs best on the validation set. Remeber that:

over-sampling the less frequent class can bring a lot of repetition in the data, because you just replicate the observations
under-sampling can be bad because you throw away information in the most frequent class, and so you can lose performance

Hybrid methods tend to outperform both, see SMOTE or ROSE.

For example, SMOTE generates new information from the minority class (synthetic observation which are derived from the true samples), and also under-samples the majority class (you choose the final ratio of the two).

Again you need to test on the data the best solution.


RL Advantage function why A = Q-V instead of A=V-Q? In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s)

Preliminary, from this post:

  First recall that a policy $\pi$ is a mapping from each state, $s$, action $a$, to the probability $\pi(a \mid s)$ of taking action $a$ when in state $s$.

  The state value function, $V^\pi(s)$, is the expected return when starting in state $s$ and following $\pi$ thereafter.

  Similarly, the state-action value function, $Q^\pi(s, a)$, is the expected return of when starting in state $s$, taking action $a$, and following policy $\pi$ thereafter.

In my understanding, $V(s)$ is always larger than $Q(s, a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$. So, why is the advantage function defined as $A = V - Q$ rather than $A = Q - V$ (at minute 1:12:29 in the video)?

Actually, V might not be larger than Q, because $s$ might actually contain a negative reward. In such a case how can we be certain what to subtract from what, such that our Advantage is always positive?

$Q(s, a)$ returns a value of entire total reward that's expected ultimately, after we pick an action $a$. $V(s)$ is the same, just with an extra reward from current state $s$ as well.

I don't see why a value of $Q - V$ would be useful. On the other hand, $V - Q$ would be useful because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$.

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$

This is incorrect. There is not really such a thing as "the reward for current state" in the general case of a MDP. If you mean the $V(S_t)$ should include the value of $R_t$, then this is still wrong, given David Silver's use of the conventions for time step indexing. It is possible to associate immediate reward with either the current time step, leading to sequence $S_0, A_0, R_0, S_1, A_1, R_1$ etc or you can use the convention of immediate reward being on next time step $S_0, A_0, R_1, S_1, A_1, R_2$ etc. David Silver (and Sutton & Barto's book) uses the latter convention.

Under that convention:

$$V(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty}
\gamma^{k}R_{t+k+1}|S_t=s]$$

$$Q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}|S_t=s, A_t=a]$$

You can see that the first term in the expansion of the sum for both Q(s,a) and V(s) is $R_{t+1}$. If you changed the convention, then both would include the equivalent value, but would be labelled $R_{t}$ in any formula.

Q and V do not differ in which time steps they sum reward over. They may differ in the value of $R_{t+1}$ because $V(s)$ assumes following the policy $\pi$ when selecting $A_t$ whilst $Q(s,a)$ uses the value $a$ supplied as a parameter for $A_t$, which can be different.

  how can we be certain what to subtract from what, such that our Advantage is always positive?

Advantage can be negative, that is fine. It means that the action $a$ in $A(s,a)$ is a worse choice than the current policy's.

  I don't see why a value of $(Q−V)$ would be useful. On the other hand, $(V−Q)$ would be useful

Both would be equally useful, it is mainly convention that we work with finding maximum Advantage representing the benefit of choosing a specific action instead of following the current policy, as opposed to finding the minimum "Disadvantage". However, the concept of Advantage in this context is arguably the more natural view.

  because it would tell us the reward we would get on $s_{t+1}$ if we took the action $a$

As explained above, this is wrong. The value in $A(s,a)$ expresses the potential benefit we would get for changing the policy $\pi(s)$. That might include changes to $R_{t+1}$, but is not limited to a single time step.

Some RL approaches do create a predictive function for expected immediate reward $\hat{r}(s,a)$ - typically this is a secondary component, used to help refine parameters for other function approximators.


Clustering : handling categorical data, should we pivot and scale? I'm a SQLServer DBA and in the new version of this tool, there are new features to integrate R scripts and use it easily with the DB objects. That sounds cool.

But to use that, we have to know a little bit about stats and data mining.
And it's quite hard to find friendly and readable documentation for newbies like me. So, I will ask for some best-practice about clustering.

I have a dataset with few hundreds records and with 5 features, both categorical and numerical.

Regarding categorical data, I have a few basic questions :

how can I manage categorical data ? what I understood from articles found is that I can transform categorical data in boolean and "pivoting" it. If a have a column with categorical data (for instance with values 'a' or 'b'), creating to new columns like column_a and column_b with value 0 or 1 ?
should I always center / scale this new column ?

You need to create a dissimilarity matrix first and then apply the clustering technique. Below link has many answer referring your point. Hope below will suffice for this:

https://stats.stackexchange.com/questions/24540/clustering-of-mixed-type-data-with-r


How can i decide on which features to use for clustering? I am clustering on a dataset where each row is a customer and each column is a feature. I have 200 features, this seems like alot for clustering. I plan to experiment with a variety of clustering models e.g. k-means since all my data is numerical. H
How can i reduce/select the features ?
I am only familiar with SelectKbest, etc as these are used for predictive modelling. But here i have NO target variable. Note i plan to use python.

Good points by shepan6. It is also worth to mention about feature extraction techniques to lower the number of  features. You can use common feature extraction techniques such as PCA or eigen projections to transform the data into a lower dimensions. After lowering the number of features like this you can appply  techniques such as clustering


Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory? Consider a very large data set that doesn't fit into memory.  Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?

This code is just hypothetical to illustrate my question.

# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)

# compared to
for x_bat, y_bat in stream_next_batch_from_file():
    x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
    # Append the splits to their respective files.
    append_data(x_train, y_train, "train_set_filename")
    append_data(x_test, y_test, "test_set_filename")
    # etc.

This should be fine. If the data is independent of its position in the list, this should give basically identical results.

If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.


How to get different results running sklearn's MeanShift in a single program? (Python3) I ran into a quirk with sklearn's MeanShift that I don't know how to get around.

MeanShift doesn't predictably give the same results on every run, so I wanted to run it multiple times within one program, but the results are the same on every iteration. They are different across program runs, but the same over every loop.

Minimal code sans convert_non_numerical_data(...).

import numpy as np
import pandas as pd
from sklearn.cluster import MeanShift
from sklearn import preprocessing

# EDIT: added on request
def convert_non_numerical_data_to_numerical(data_frame):
    columns = data_frame.columns.values

    for col in columns:
        text_digit_vals = {}

        if data_frame[col].dtype in [np.int64, np.float64]:
            continue

        x = 0
        for unique in set(data_frame[col].values):
            if unique not in text_digit_vals:
                text_digit_vals[unique] = x
                x += 1

        def convert_to_int(val):
            return text_digit_vals[val]
        data_frame[col] = list(map(convert_to_int, data_frame[col]))

df = pd.read_csv("test_data/titanic.csv")
df.fillna(0, inplace=True)  # bad idea to shove in 0?
convert_non_numerical_data_to_numerical(df)

y = np.array(df["survived"])
X = np.array(df.drop(["survived"], 1))
X = preprocessing.scale(X)

run_data = []
for run_count in range(3):
    clf = MeanShift()
    clf.fit(X)
    run_data.append(clf.labels_.copy())
    clf = None

run_0 = run_data[0]
for other_run in run_data[1:]:
    if np.array_equiv(run_0, other_run):
        print("same")
    else:
        print("different")

Output:

same
same

Any idea what is causing this, or at least how to get around it?

Note:

Data set from pythonprogramming.net

Have you looked at smoothing your dataset with any other methods? What hyper-parameter tuning have you done? Also how did you convert your non_numerical data, just wondering if you can provide pastbin?


Web services to mine the social web? Are there any web services that can be used to analyse data in social networks with respect to a specific research question (e.g. mentioning of certain products in social media discussions)?

Tweepy is one of the best libraries for analyzing and hacking around with the Twitter API.  (Being a contributor for tweepy, I can vouch for it's stability and quality)

For a Python wrapper for the Facebook graph API, you can use the Facebook-Insights library, which is well-maintained and neat documentation.

There are services out there which can mine you information, but they are limited to the complexity of the query.

For example: "How many people tweeted about banana on Monday?" can be answered from existing tools

"How many people happy people tweeted about banana on Monday" can also be done, but would require more efforts and the software or tool should be able to detect emotion.

So, if you are into research, I would advise you to go with the API's and good ol' programming!


looking for public dataset for stock market I want to do some modelling and data visualization on historical stock data, including price, volume, financials, etc. Is there an public dataset available for stock price history? I looked at a few, but either they have a high cost, or not sure they would be reliable. Free would be preferred, also established and reliable. If not, what are some good options for collecting the data myself? Maybe web scraping, or public api's etc.

You can have a look at the kaggle stock dataset.

https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs

This questions are normally done in OpenData stack exchange.

https://opendata.stackexchange.com/


Graphs for representing performance of machine learning classifiers Can any please guide about how many types of the graph other than ROC can be plotted to represent the performance of the machine learning classifier?

The basics are:

Precision-Recall Curve
ROC Curve

Then, you can plot other features of the model to get insights about generalization, learning process, overfitting/underfitting, like accuracy vs epochs, loss versus epochs, to name a few.


word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7 I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), although I'm not using its character level n-grams, so it's basically just word2vec.
I'm testing the results by looking at some of the "most similar" words to key and the model seems to be working very well, except that the most similar words get at most a similarity score (using cosine similarity, gensim's FastText most_similar() function) of 0.7; versus google's or spacy's embeddings synonyms tend to have ~0.95+ similarity. I'm wondering why they don't in my case. This is not necessarily a problem, but it is not clear to me why this would happen; how come even very similar words, with very similar uses only get 0.7 in similarity? Again, the similar words themselves do make a lot of sense, so the training went well, but the score is kind of low.
My best guess is that maybe the embedding size I'm using is too big - 256. Can anyone suggest other explanations?
I trained it using CBOW, 15 epochs, window size=4, a few tens of millions of documents but only ~100000 words in the vocabulary that I'm interested in.

Let us try and understand how Word2Vector actually works before looking at distances:

There are 2 ways of generating vectors for a word :

Continuous bag of words
Skip grams

The following diagram explains the difference between the two approaches.

In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out there.

Now let us look at what the image is trying to tell us.

For CBOW, it looks at the neighboring words to provide a probability of another word co-occurring in that space.

For Skipgram, given a word, it provides probabilities of all the other words occurring in the neighborhood.

Why is this important to know?

Without understanding what these algorithms are doing, its not quite possible to solve an NLP problem as word vectors are the primary inputs to any deep learning / machine learning algorithms.

What are the other capabilities Word Vectors provide us with?

As you have vectorized Words in a n-dimensional space, it gives you the ability to perform fast operations such as computing distances in this n-dimensional space. One such way is cosine distance, that you have outlined to have used.

Understanding how distances are computed can also help. Here is a small explanation of how Cosine similarity works :

When you look at vectors project in n-dimensional space, you find the difference in the angle between these vectors. If the angle is small, lets say 0, then cos(0) = 1, which implies the distance between these vectors is very small, thereby making them similar vectors.

Given this information, what can you infer from your results of distances between words:

Your corpus has diverse distribution of words that could possible co-occur with a large number of words, thereby making distances between similar words not > 0.7
Given the variation of words and possible presence of large amount of stop words in your vocabulary, its hard for a shallow neural network to provide accurate vectors for your words.

Suggestions that can help improve the vector quality:

Using TFIDF, you can remove stop words in case they are not interesting to you.
Pre-process your text to bring them to standard form. For example, lower casing of text, removal of special characters, stripping of spaces.
Experiment with varying embedding sizes. Start with as little as 100, as you are interested in only finite number of words.
Visualize your vectors using tsne / PCA and projecting them to a 2 or 3 dimensional space.
Perform the same for different epochs to see how vectors are converging / diverging.
Its also very important to understand your corpus well, so that you know what these vectors imply and how they co-occur.

Hope this helps.


Explaining feature_importances_ in Scikit Learn RandomForestRegressor For a project, I used the feature_importances_ attributes from the RandomForestRegressor. Everything works well but I don't know how to explain why one feature is more important than another. I mean I know that the higher the score  is the higher the importance, but I don't understand how it is calculated.

For exemple, if a variable as a score of 0.35 what does it mean?

I would appreciate if someone could explain me how it works!

Thanks!

scikit-learn's RandomForestRegressor feature importance is computed in each tree composing the forest. You can find the source code here (starting at line 1053).

What it does is, for each node in the tree where the split is made on the feature, it substracts each child node's (left and right) impurity values from the parent node impurity value. If impurity decreases a lot (meaning the feature performs an efficient split), it basically gives a high score. Of course, all that is weighted depending on how useful the test is for the result: a split between two individuals gives a high impurity decrease, but is trivial, so much easier than between two large populations.

Once the feature importance has been determined for each tree, it is summed up and normalized so that the feature_importances_ vector sums up to 1.

It might introduce some biaises, I guess that it should be the case if variables are not scaled, for instance. However it is quite easy to compute (you just have to read impurity values from the tree), so I guess this is why it is provided by default. But the method is not unequivocal, there are other methods out there that you can implement more or less manually:

shuffling a feature's values in the dataset
reverting the result of each test based on the feature

... and probably a few other approaches. All those will give you other scores that may be of help.


Different representations of dendrograms I have a dendrogram represented in a format I don't understand:
(K_5:1.000030e+00,((K_1:2.000000e-05,(K_2:1.000000e-05,K_3:1.000000e-05):1.000000e-05):1.000000e-05,K_4:3.000000e-05)0.806:1.000000e+00):0.000000e+00;
I am not sure how to interpret the above.
It is an output of hierarchical clustering.
K_1, K_2, K_3, K_4, K_5 are the data points.
I have other dendrograms represented in the following format:
[x_1,x_2,x_3,x_4,x_5] (we start with one big cluster and split a cluster at each step)
[x_1,x_2][x_3,x_4,x_5]
[x_1,x_2][x_3,x_5][x_4]
[x_1][x_2][x_3,x_5][x_4]
[x_1][x_2][x_3][x_5][x_4]
I want a way to convert between these two representations.

This output represents the dendogram as a tree. The innermost parentheses represent the deepest parts of the tree. For instance the top (root) of the tree start with the pair K5 and a subtree, then this subtree is made of another subtree and K4, and so on.
If we ignore the numerical values (distances I assume?) we have this:
(K_5,
  (
    (K_1,
       (
         K_2,K_3
       )

    )
    K_4
  )
)

Which represents this tree:
 --------------------
 |                  |
 |            -------------
K_5           |           |
           -------       K_4
           |     |
          K_1  -----
               |   |
              K_2 K_3

Then it can be converted to the desired format:
[K_1 , K_2 , K_3 , K_4 , K_5]
[K_1 , K_2 , K_3 , K_4] [K_5]
[K_1 , K_2 , K_3] [K_4] [K_5]
[K_1] [K_2 , K_3] [K_4] [K_5]
[K_1] [K_2] [K_3] [K_4] [K_5]


How to deal with a constant value as an output from neural network? I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dense(units=512, activation='relu', input_shape=x_train.shape[1],)))
model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems for me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

[EDIT]
Adding learning curves for trainging and validation sets.

Lowering the initial learning rate from 0.1 to 0.005 helped the problem.

adam = optimizers.Adam(lr=0.005)


How to explain the outcome of k-means clustering? I am currently conducting some analysis using NTSB aviation accident database. There are cause statements for most of the aviation incidents in this dataset that describe the factors lead to such event.

One of my objectives here is to try to group the causes, and clustering seems to be a feasible way to solve this kind of problem. I performed the followings prior to the beginning of k-means clustering:

Stop-word removal, that is, to remove some common functional words in the text
Text stemming, that is, to remove a word's suffix, and if necessary, transform the term into its simplest form
Vectorised the documents into TF-IDF vector to scale up the less-common but more-informative words and scale down highly-common but less-informative words
Applied SVD to reduce the dimensionality of vector

After these steps k-means clustering is applied to the vector. By using the events that occurred from Jan 1985 to Dec 1990 I get the following result with number of clusters k = 3:

(Note: I am using Python and sklearn to work on my analysis)

... some output omitted ...
Clustering sparse data with KMeans(copy_x=True, init='k-means++', max_iter=100, n_clusters=3, n_init=1,
    n_jobs=1, precompute_distances='auto', random_state=None, tol=0.0001,
    verbose=True)
Initialization complete
Iteration  0, inertia 8449.657
Iteration  1, inertia 4640.331
Iteration  2, inertia 4590.204
Iteration  3, inertia 4562.378
Iteration  4, inertia 4554.392
Iteration  5, inertia 4548.837
Iteration  6, inertia 4541.422
Iteration  7, inertia 4538.966
Iteration  8, inertia 4538.545
Iteration  9, inertia 4538.392
Iteration 10, inertia 4538.328
Iteration 11, inertia 4538.310
Iteration 12, inertia 4538.290
Iteration 13, inertia 4538.280
Iteration 14, inertia 4538.275
Iteration 15, inertia 4538.271
Converged at iteration 15

Silhouette Coefficient: 0.037
Top terms per cluster:
**Cluster 0: fuel engin power loss undetermin exhaust reason failur pilot land**
**Cluster 1: pilot failur factor land condit improp accid flight contribute inadequ**
**Cluster 2: control maintain pilot failur direct aircraft airspe stall land adequ**

and I generated a plot graph of the data as follows:

The result doesn't seem like make sense to me. I wonder why all of the clusters contain some common terms like "pilot" and "failure".

One possibility that I can think of (but I am not sure if it is valid in this case) is the documents with these common terms are actually located at the very centre of the the plot graph, therefore they can not be efficiently clustered into a right cluster. I believe this problem cannot be addressed by increasing the number of clusters, as I have just done it and this problem persists.

I just want to know if there is any other factors that could cause the scenario that I am facing? Or more broadly, am I using the right clustering algorithm?

Thanks StackExchange and Data Science Portal.

I made some similar studies on process descriptions on computers. The main difficulty was to discard words that were not meaningful. Which is hard because:

Some words appear in every clusters, but in different proportion. They should not be discarded, like "pilot". It is normal that "pilot" appears often, but depending on the number of occurrences it might signify that the pilot made a decisive action for the accident.
Some words appear very rarely, but they are the key words for the reason why the accident happened. Then it is very hard to find a way to give them more importance without adding direct prior knowledge to your clustering.

What I could suggest is:

First try different k value. Why did you chose 3 ?
Try a different algorithm for 2D visualization. For example t-sne is good for reducing high dimension in 2. Look at the examples here http://scikit-learn.org/stable/auto_examples/manifold/plot_lle_digits.html#example-manifold-plot-lle-digits-py
Try working directly on your word occurrences to check whether your word differ between clusters. For example, you can compute the KL-divergence of each word for each cluster given their global distribution. See here https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
If you can't get good results, think of another way of vectorizing. For example word2vec is good at finding words that occurs often together and could be more suitable to your case.


AB testing : When AA testing doesn't work After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.
AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)
As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12
Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.

Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.
My question : I would like some pointers of people with more experience than me in data science to help me to :
- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here
- Understand if I am forgetting to check anything on my side to explain this discrepancy
The exact process that is used from our server to the final client is :

we send every x hours a list of a few hundred people to Responsys

Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()

then the list is filtered to remove undeliverable people, and each branch is sent the exact same email

Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.

During launch, for each email recipient:
a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()
b. decide the bucket this random number is in
c. send the campaign in that bucket to the recipient

I 'solved' the issue with Oracle's support, I still post an answer here because some of what I encountered can be useful :  According to Oracle's support, this bug in AB testing programs was known for the last 9 month, but corrected only this week-end.

So my problem is supposedly solved (I will re-launch a test and see if results are now consistent), but I still have some pointers if someone encounters the same problem I did :

Make sure this is not a statistical artifact : test your hypothesis on several independant periods. I used a Fischer test because it is exact for a binomial framework, it is what you should use if your population/conversion rate is too small for a normal approximation
Respect hypothesis testing framework : start your test after you formulated your hypothesis
Get a very very low p-value : No one will listen to your complaint about faulty AB testing if you come up with a 3% p-value, and they would be right. A bayesian analysis probably points to a false positive.
Make sure the random numbers that are used to split the populations are actually random (and not, say a 50/50 split)
Imagine all the possibilities that your reporting might be wrong. I thought I had imagined them all, but if you read below the real reason behind the bug, you will find that faulty reporting can be hidden very far away in parts of the environment you don't control or even have a view in.

I have the actual answer from Oracle that describes the bug that happened, in case someone has the exact same problem or wants to know how this kind of bug happens :

For programs triggered by event, their engine does not have the capability to associate the different open and clicks events to the correct email. In the case several emails have been sent through the same program, Responsys assumes the event concerns the earliest email that has been sent through this program.

In my case, the program could have been sent several times during the period.

In the period where I was not splitting my emails through two branches, the name of the campaign I used was already A  (the same than one of my branches in my testing period)

You can see where this is going : Every customer opening any email was marked as having read the "A" email, as long as they had been into the program in the 3 months before I started my AB testing, which caused a faulty reporting.

So actually both branches were sent to the correct customers, and it is just a reporting problem that comes from an approximative algorithm on Responsys side.

The workaround I will try now is simply to create two brand new "campaign" objects before starting a test rather than using a campaign that already existed.


Can word embedding be used for text classification on a mix of English and non-English text? I'm doing text classification on text messages generated by consumers and just realized even though most of the replies provided by consumers are in English, some are in French. I've used Keras word embedding, conv1D and maxpooling to learn the structure in the text and didn't use any other text preprocessing techniques such as stop words removal etc.

In this case, I think it should be fine to use word embedding on both languages since word embedding learns the meaning of individual words regardless of languages...Is this reasonable? Or maybe I do need to separate the languages and build different models for each language?

In this case, I think it should be fine to use word embedding on both languages since word embedding learns the meaning of individual words regardless of languages...Is this reasonable? Or maybe I do need to separate the languages and build different models for each language?

  If I think logically you are correct. Word embedding is merely a
  collection of Tokens, which derived its features on the basis of
  nearby words in a sentence. So if you have sufficient raw data(mix of
  both), I think its good to go, though results will explain you more
  :).

However its good to see how such models will behave in case we have mix of LeftToRight(LTR) and RTL languages.


Overfitting in Siamese Network I am trying to train a Siamese network for an application very similar to this and this. From what I have read about training Siamese networks dissimilar pairs of images outnumber the similar pairs and obviously so. In the papers I have linked to, the authors talk about a 1:20 ratio for similar to dissimilar pairs, i.e., for every similar pair of images, the training set consists of 20 dissimilar pairs. If I have understood it correctly, for a batch size of 64, my batch will consist of 3 similar pairs and the corresponding 20 dissimilar pairs for each of the similar pair.

I implemented the same in Keras, and the network overfits from the very first epoch. I have approximately 6000 similar pairs of images and I am generating the dissimilar pairs randomly. How do I reduce the overfitting? Any tips would help.

NOTE: I am using the last two Inception modules of Inception V3 to train my data.
EDIT: We managed to fix this issue with getting more training data, and using a network pretrained on a very similar task.

The most common way to reduce overfitting in neural networks in your case are:

Dropout - Randomly removing connections during training
Data Augmentation - Creating variations in data, such as rotating or flipping images


Tensorboard with pytorch dont display a graph I am trying to visualize a model I created using Tensorboard with Pytorch but when running tensorboard and going to the graph tab nothing is shown, im adding my code for reference, also im adding a screen-shot of my conda env for all the dependencies

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter()
images, labels = next(iter(trainloader))
grid = torchvision.utils.make_grid(images)

writer.add_image('images', grid)
writer.add_graph(model=model, input_to_model=images, verbose=True)
writer.close()

I had the same problem.
Check out this thread:
https://github.com/pytorch/pytorch/issues/24157

TLDR: Update PyTorch to PyTorch-nightly and the problem should be solved.
https://pytorch.org/get-started/locally/


What is conjugate gradient descent? What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique?

I came across a resource, but was unable to understand the difference between the two methods. It has mentioned in the procedure that:

  the next search direction is determined so that it is conjugate to previous search directions.

What does this mean? Also, what is line search mentioned in the web page?

Can anyone please explain it with the help of a diagram?

What does this sentence mean?

It means that the next vector should be perpendicular to all the previous ones with respect to a matrix. It's like how the natural basis vectors are perpendicular to each other, with the added twist of a matrix:

$\mathrm {x^T A y} = 0$ instead of $\mathrm{x^T y} = 0$

  And what is line search mentioned in the webpage?

Line search is an optimization method that involves guessing how far along a given direction (i.e., along a line) one should move to best reach the local minimum.


Is there an alternative to nltk in golang? Golang is one of my favourite languages and I want to use it for a personal NLP/ML project. Is golang's ecosystem good and rich enough for this?

Is there an alternative package for nltk in golang?

No. Not yet

There is no single package in Golang, which acts as versatile as nltk for NLP. However, there are several packages which aim to do it.

Here is a compiled list of such packages: https://github.com/gopherds/resources/blob/master/tooling/README.md#nlp


Curation of a dataset of audio files in different formats I have a large dataset of audio files (around 80k files of around 45 minutes long each) that are related to my work. Some of them are in format ".m4a" and others in format ".mp3". I am not a computer scientist, but I understand that these are two different formats.

I will use the audios later on for a variety of analyses. My main concern for now is to compile a homogenous dataset. It is bugging me that the audio files are encoded in two different formats. I could convert one into another, but I am afraid I will lose quality on the converted files.

What would be a good strategy for me? Should I just keep the files as they are, and only convert local versions for analysis?

Keep the files the way they are for long term storage.

For experimenting, it might be beneficial to deal with just one format.

Depending on what resources you have available issues may be:

supported formats
disk space/speed
decompression speed
random access

Depending on the requirements of your analyses, you might want to reduce the quality of your tracks to gain speed advantages.

Supported Formats

If the software you use for analysis supports both MP3 and M4A, you're set. If not, you'll need to convert using tools like ffmpeg or sox.

Disk Space/Speed

If you were to convert your files to uncompressed WAV, this will eat up a lot of disk space. If you don't have this space, WAV is not a good format. Instead convert to another lossless format like FLAC or Apple Lossless. You can do so without loosing any quality.

Besides disk space, you need to pay attention to disk speed. For some deep learning tasks you will have to provide your network with lots and lots of different samples, which you need to read from disk. With WAV being roughly 10 times as large on disk, disk I/O may become a bottleneck, so minimizing file size on disk may become desirable.

Decompression Speed

Using a compressed format like FLAC or MP3 comes at a (small) price. You will have to decompress your audio files on-the-fly before analysis.

Random Access

If you need to quickly access audio starting at a certain timestamp, say at 10.455s, formats like WAV have a distinct advantage, as they allow random access of audio very easily. This is usually not possible (as quickly) with compressed formats.

So what format should you choose for your experiments?

It all depends on your requirements.

Do you need quick random access to the audio? Do you really need stereo quality at 44,100 Hz? You can reduce the problem size a lot, by converting to mono 22,050 Hz. If that's still good enough for your purposes, go for it and save yourself a lot of trouble (but keep your original files around!).

Also, to get more technical, if you are going to analyze your data with something like Python and you know you need spectrograms, there's nothing wrong with preprocessing your MP3/M4A files with a library like librosa and simply store numpy arrays, if that's what you ultimately need.


Best way to scale across different datasets I have come across a peculiar situation when preprocessing data.

Let's say I have a dataset A. I split the dataset into A_train and A_test. I fit the A_train using any of the given scalers (sci-kit learn) and transform A_test with that scaler. Now training the neural network with A_train and validating on A_test works well. No overfitting and performance is good.

Let's say I have dataset B with the same features as in A, but with different ranges of values for the features. A simple example of A and B could be Boston and Paris housing datasets respectively. To test the performance of the above trained model on B, we transform B according to scaling attributes of A_train and then validate. This usually degrades performance, as this model is never shown the data from B.

The peculiar thing is if I fit and transform on B directly instead of using scaling attributes of A_train, the performance is a lot better. Usually, this reduces performance if I test this on A_test. In this scenario, it seems to work, although it's not right.

Since I work mostly on climate datasets, training on every dataset is not feasible. Therefore I would like to know the best way to scale such different datasets with the same features to get better performance.

Any ideas, please.

PS: I know training my model with more data can improve performance, but I am more interested in the right way of scaling. I tried removing outliers from datasets and applied QuantileTransformer, it improved performance but could be better.

Your observation stems from two facts

Model performance deteriorates when data distribution changes (shifts). For example, when model is trained on Boston dataset but tested on Paris dataset, or trained on summer dataset but tested on winter dataset.
Model evaluation (wrongly) shows an improvement when information leaks from test set. For example, when test set is scaled with must-not-be-known statistics of the test set like (min, mean, std, max).

If a data set for other cities like Paris is not available at the training time (1), obviously there is nothing we can do. In real world cases, when we see a noticeable difference between validation score and test score, we hypothesize that the test set has a different distribution than the training set, thus, we try to incorporate the new data and re-train (probably not a full re-training in an online learning setting). Otherwise (2), we can

Combine all the cities into one dataset, then split it into train and test sets (later, validation set will be peaked from train set), i.e.

  train = A_train U B_train U ...
  test = A_test U B_test U ...

This way, the information from all cities will be used for scaling. We may under- or over-sample some cities to have a data set with equal representatives from each city.
Or, build a separate (specialized) model for each city (or a group of similar cites) which bypasses the "train on A_train, and  test on B" problem all together.


Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions:

The activation function $g$ is typically chosen to be a function that
is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i).$

Then we see equation 6.3 is defined as (assuming g as ReLU):

We can now specify our complete network as
$f(x; W,c,w,b) = w^T$
max$\{0, W^Tx + c\} + b$

Wondering why the book uses $W^Tx$ in equation 6.3, while I expect it to be $x^TW$. Unlike XOR example in the book where $W$ is a $2\times2$ square matrix, we may have non-square $W$ as well, and in such cases, $x^TW$ is not same as $W^Tx$.
Please help me understand, if I'm missing something here.

Let $\mathbf{y}  = \mathbf{W}^T \mathbf{x}$
Then, $\mathbf{y}^T =(\mathbf{W}^T \mathbf{x})^T =\mathbf{x}^{T}(W^T)^T = \mathbf{x}^{T}W $. Note that $\mathbf{W}$ does not have to be a square matrix.
Let $e^{(i)}_{j} = \delta_{i,j} $.
Then,  $y_{i} = \mathbf{y}^{T}e^{(i)} = (\mathbf{x}^T W) e^{(i)} = \mathbf{x}^{T}(We^{(i)}) = \mathbf{x}^{T}W_{:,i}$
and thus
$h_{i} = g(\mathbf{x}^T W_{:,i}+c_{i}) = g(y_{i}+c_{i})$
On the other hand,
$f(..)  = w^{T} \max\{\mathbf{0},W^{T}\mathbf{x}+\mathbf{c}\}+b = w^{T} \max\{\mathbf{0},\mathbf{y}+\mathbf{c}\}+\mathbf{b}$.
Does that answer your question ?


Proper order of optimizing hyperparameters in ANN I have built a neural network for approximating a certain function and decided on a metric how to evaluate the performance. Now, where do I start with optimizing to get the optimal result? I need to decide on number of layers, hidden neurons, learning rate, learning algorithms and all other hyperparameters. What is the usual  procedure and order in which to optimize all network's parameters? I guess I am looking for some kind of general guidelines to follow to obtain the best possible performance

My initial suggestion is to look in the literature for networks which have been implemented to do solve similar problems and start from there.
Then you should just try a bunch of different values for the parameters and see what works in your case.

See also this Cross Validated question: https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw


So what's the catch with LSTM? I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying different models.

After working with some results and reading more and more about LSTM, it seems like this approach is far superior to anything else I've tried (across multiple datasets). I keep thinking to myself, "why/when would you not use LSTM?". The use of the additional gates, inherent to LSTM, makes perfect sense to me after having some models that suffer from vanishing gradients.

So what's the catch with LSTM? Where do they not do so well? I know there is no such thing as a "one size fits all" algorithm, so there must be a downside to LSTM.

You are right that LSTMs work very well for some problems, but some of the drawbacks are:

LSTMs take longer to train
LSTMs require more memory to train
LSTMs are easy to overfit
Dropout is much harder to implement in LSTMs
LSTMs are sensitive to different random weight initializations

These are in comparison to a simpler model like a 1D conv net, for example.

The first three items are because LSTMs have more parameters.


Looking for a classification (?) algorithm for linearly separable but unlabeled data points I have a dataset that is linearly separable with two lines - something like that:

Now I'am looking for the right kind of algorithm to do what I guess a SVM would do with labeled data - find the margins or decision boundaries for each class (three in this case). I tried spectral clustering and Gaussian mixture, but those don't seem to work.
Origin of the data: edges from tracked bounding boxes for cars on a three lane road. Thanks!

Edit: K-Means is apparently not really working for this kind of distribution:

I also tried out Ismor's suggestion to do some transformation before k-means, which results in:

The output here is very sensitive to the setting of the origin for y0, I couldn't get it right...

I got a partiall solution which migth work for this particular case but it is not easely generalizable: Since all your data converges to the same point: (1400, 500), you can set the center of coordinates there and then cluster in the space of angles. i.e. use the transformation $$\alpha = \arctan\left(\dfrac{y - y_0}{x - x_0}\right)$$ and cluster $\alpha$ (be carefull! you might divide by 0)

I have prepare this synthetic example in R and it works!,  but again: this is not easely generalizable

library(data.table)
library(ggplot2)

x <- c(1,2,3,4,5)
y1 <- x + 1
y2 <- -x + 11
y3 <- -3*x + 21

# creates a data.table with 3 lines converging at point (5,6)
dt <- data.table( x = rep(x, 3)
                , y = c(y1,y2,y3)
                )

The data.table looks like this:

Now is time to transform it and make the clusters

dt_transform <- dt[, .(x, y, a = atan((y - 6) / (x - 6)) )] # avoid x - 5 == 0

km <- kmeans(dt_transform$a, 3)

dt_cluster <- cbind(dt_transform, cluster = as.factor(km$cluster))

The clustered data.table looks like this:

If you go this way please, share the results with me. I'm interesting to know if it works with real data


How to improve accuracy of ethnicity prediction using name? I've got a Excel file having columns Name, Surname and Community, having around 31 thousand entries. For my college project, the task I have is to write a program to predict "Community" of a person, given their name and surname. For example, the first entry in the file is the name: Yogita Singh. If someone enters a name like Yogta Singha, then approximately it should predict that that person belongs to the Punjabi communtiy.

I did manage to write a program, which does the prediction using the Levenshtein distance. I gave more weightage to the surname (10 times more), than the first name, because surname and community has a greater correlation than first-name and community (naturally, because some first-names are very common and used by people from many different communities).

Any ideas how to improve the accuracy of my model? Has there been any relevant theoretical study regarding name - ethnicity mapping/correlation models? I think Levenshtein  distance is a good start but I'm not sure the exact weights I should give to the surname and first name. Also, perhaps I need to give more weight to the characters towards the beginning of the surname, rather than characters towards the end.

Any suggestions regarding how to improve the accuracy of my model will be greatly appreciated. Thanks.

Feature engineering

The question you want to ask yourself is "When you look at a name how do you as a human deduce the community?" I would claim that this is possible by looking at the way the letters are grouped together. For example "aoui" and "wi" is very common in India and Arabic, whereas "Zh" would be very common in Chinese. So how can we extract these types of features?

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

This would capture the distribution of letter combinations which are present for each community.


Install using Anaconda not working I tried installing PyFerret on my Macbook through the terminal.
I got into my user directory with the terminal and gave it the following commands from this website https://anaconda.org/conda-forge/pyferret:

conda install -c conda-forge pyferret
conda install -c conda-forge/label/broken pyferret
conda install -c conda-forge/label/cf201901 pyferret

However, I noted that I only needed to download just one of them. I tried removing the files using

conda remove
conda uninstall

but they did not work. Later I tried following the instructions from this website
https://github.com/NOAA-PMEL/PyFerret/blob/master/README.md
which instructs to type the following command

conda create -n FERRET -c conda-forge pyferret ferret_datasets --yes

then it says you can use PyFerret by calling it in the terminal as

conda activate FERRET

but the response I got was

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

I tried using

conda init -bash

and it sort of worked after that, I could use the command to activate PyFerret but it would only say PyFerrer on the name of my command line and I did not really know what was going on.

Could you please instruct me in how to uninstall everything?
 and if possible I would like to be instructed on how to get PyFerret to work as well.

It seems that Conda can only install one version of a package at a time,thus whatever installing command you use it will replace whatever other build you might have installed. So I reinstalled it using conda create -n FERRET -c conda-forge pyferret ferret_datasets --yes. After that I was able to activate the environment with conda activate FERRET If the terminal complains and asks you to initialize a shell, conda init bash should suffice to allow you to activate the environment.


Can I classify set of documents using classifying method using limited number of concepts ? I have  set of documents and I want classify them to true and false

My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these documents or I can take only some words that I interested in then I compare it with the documents. Which one is more efficient in classify document and can work with SVM.

Both methods work. However, if you retain all words in documents you would essentially be working with high dimensional vectors (each term representing one dimension). Consequently, a classifier, e.g. SVM, would take more time to converge.

It is thus a standard practice to reduce the term-space dimensionality by pre-processing steps such as stop-word removal, stemming, Principal Component Analysis (PCA) etc.

One approach could be to analyze the document corpora by a topic modelling technique such as LDA and then retaining only those words which are representative of the topics, i.e. those which have high membership values in a single topic class.

Another approach (inspired by information retrieval) could be to retain the top K tf-idf terms from each document.


Does using user-specific accumulative variables causes data leakage? Let's say I have a scenario in which my observational unit is a bill that was issued after a certain service was given and my goal is to predict if this bill is going to be paid or not. I have users in the system so I include user-variables like number of unpaid bills the user has, user's time in the service system (seniority) etc. I train on month 1 and test on month 2 (bills that were created in those months, respectively).

In the testing month I will have user-variable's count increased, so e.g. if during training user_1 had 100 days time in my system, of course that if there is a bill associated with him in the test month - his count of days will be higher.

Is this accumulative nature of such variables is considered a data leakage between train and test sets (because part of the information that was used in training is being used, in a sense, in testing)?

If the data is available at prediction time, then it is not data leakeage.
In your specific example, historical user data should be used at prediction time to build a better model.
You do have to decide on how to do the train/test split. You can split by user, by time, or combination.


Extending DTW 1-NN Classification to On-line Scenario I am familiar with Dynamic Time Warping classification using a 1-nearest neighbour approach. However, in most benchmark datasets and applications, it used ex-post, i.e. classifying a time series after you have observed it.

Is there a way to use the dynamic time warping similarity distance in an on-line setting before I have fully observed the series I am comparing?

Furthermore, if there is a way to probabilistically categorize as the series comes in that is even better. E.g. After observing 1/10 of the time series, the model predicts this sub-series has a 20% of being in group A, 10% of being in group B, etc

Hmm
Yes and no.
This is called "early classification". There as about two dozen papers that do this, but at least 20 are flawed...

See https://www.youtube.com/watch?v=d_qLzMMuVQg
http://www.cs.ucr.edu/~eamonn/UCRsuite.html


3d contour plot in R I have 2 vectors $1000 \times 1$, lets call them $y_{1}$ and $y_{2}$. Each vector represents a normal distribution with certain mean and variance.

I plot the contour plot using the following R code:

x <- y1
y <- y2
s <- subplot(
  plot_ly(x = x, type = "histogram", showlegend=FALSE),
  plotly_empty(),
  plot_ly(x = x, y = y, type = "histogram2dcontour", showlegend=FALSE),
  plot_ly(y = y, type = "histogram", showlegend=FALSE),
  nrows = 2, heights = c(0.2, 0.8), widths = c(0.8, 0.2),
  shareX = TRUE, shareY = TRUE, titleX = FALSE, titleY = FALSE
)
layout(s)

and i get the following plot

What I would like to do now, is to plot a 3D contour plot (so that I can actually see the "mountain" that is created after plotting the histogram of $y_{1}$ against the histogram of $y_{2}$). So in the z-axis I would like to have the frequencies of the values. Any suggestions ?

This plot not use frequencies but kernel density:

freqz <- with(data.frame(x,y), MASS::kde2d(x, y, n = 50))
with(freqz, plot_ly(x = x, y = y, z = z, type = "surface"))


Tensorflow.js error: must be either an HTMLVideoElement but was HTMLVideoElement I am getting this error in the console using tensorflow.js

Uncaught (in promise) Error: pixels passed to tf.browser.fromPixels()
must be either an HTMLVideoElement, HTMLImageElement,
HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData
in webworker or {data: Uint32Array, width: number, height: number},
but was HTMLVideoElement

I am just passing a video html element after it's loaded so what does it mean?
My js code code:
...
const input_tensor = tf.browser.fromPixels(document.getElementById("video"));

My html tag:
<video id="video" autoplay muted></video>

I finally managed to get it to work by opening a issue in the tfjs repo
I copy here the solution provided by the github user tafsiri

After digging around a bit, it looks like your project is running in
separate context mode and there are two JS contexts in which code is
run. Your code is being run in the browser context and the tfjs
library code is being run in the node context. In the node context
video instanceof HTMLVideoElement returns false. The nw.js docs
suggest that browser api's aren't usable in the node context so there
may be incompatibility there.
Their docs suggest you can switch to mixed context mode, I tried that
by adding the following entry to the package.json and then it worked.
"chromium-args": "--mixed-context", If you want to stay in separate
context mode (and I don't know enough about nw.js to know the
advantages and disadvantages of each) you would need to not use any of
the browser specific apis of tfjs.


model.predict() accuracy extremely low on training dataset This question is similar to this.
I'm new to ML, and I am trying to classify breast cancer histology images using EfficientNets with Transfer Learning. The dataset is small (400 images in total - there are 4 classes and all classes are equally balanced) and I am using ImageNet weights, and fine-tuning the model by freezing the first two blocks.
I've implemented a model with Keras that reaches a training accuracy of ~90% after 30 epochs.
When trying to use model.predict on the training dataset (to understand the results of the predict), I expect the results to be good since the prediction is being done on data that the model has already seen but the results I get are extremely low. The prediction accuracy in the report created by sklearn.metrics.classification_report is 27.81%.
After trying various methods to change this, I still have not been able to improve or understand the result.
The labels I use:
{0: 'Benign', 1: 'InSitu', 2: 'Invasive', 3: 'Normal'}
After playing around with code, I found that the expected labels and the results were as follows:
expected:
[0, 3, 1, 0, 0, 3, 2, 0, 2, 2, 1, 0, 3, 1, 2, 3, 2, 1, 3, 2, 3, 3, 1, 0, 3, 1, 3, 2, 2, 1, 1, 3, 0, 1, 3, 0, 1, 2, 0, 3, 2, 3, 2, 3, 1, 0, 1, 0, 1, 1, 2, 1, 2, 2, 2, 1, 1, 0, 3, 3, 0, 3, 1, 1, 3, 1, 1, 0, 2, 1, 0, 0, 1, 1, 1, 2, 2, 1, 3, 1, 1, 3, 3, 3, 0, 1, 1, 0, 2, 3, 0, 3, 1, 2, 2, 3, 0, 3, 3, 0, 0, 0, 2, 0, 1, 0, 2, 1, 3, 1, 0, 2, 2, 0, 1, 3, 1, 1, 3, 2, 3, 0, 2, 1, 2, 3, 3, 0, 0, 1, 3, 0, 0, 3, 2, 3, 1, 2, 0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 3, 0, 0, 3, 2, 1, 3, 2, 1, 1, 2, 2, 3, 3, 1, 2, 1, 0, 3, 3, 0, 1, 2, 2, 1, 3, 1, 3, 0, 1, 3, 2, 3, 3, 0, 2, 1, 3, 2, 0, 2, 0, 3, 3, 2, 3, 0, 2, 2, 0, 3, 2, 2, 1, 1, 3, 2, 3, 0, 2, 0, 3, 1, 0, 1, 2, 0, 2, 0, 0, 2, 2, 2, 1, 1, 0, 3, 0, 1, 1, 2, 2, 0, 0, 3, 0, 1, 0, 1, 1, 1, 3, 2, 2, 3, 2, 1, 2, 0, 1, 2, 1, 1, 1, 3, 1, 2, 3, 1, 2, 0, 2, 3, 1, 2, 1, 2, 1, 2, 2, 2, 0, 3, 3, 1, 3, 3, 3, 0, 1, 1, 2, 2, 3, 3, 2, 0, 0, 2, 1, 0, 1, 3, 0, 2, 0, 0, 3, 2, 1, 2, 2, 1, 0, 0, 0, 2, 3, 3, 0, 2, 3, 1, 0, 0, 0, 0, 3, 0, 3, 2, 2]
result:
[3 0 3 0 3 3 3 1 0 2 3 3 2 1 3 2 1 1 3 1 2 3 2 1 0 3 3 0 2 1 2 2 2 1 3 3 0 2 0 2 1 1 2 0 0 0 0 0 2 3 0 1 3 0 1 1 2 0 0 2 3 3 0 2 2 1 2 3 2 2 1 2 1 3 3 3 1 0 1 3 0 1 2 3 2 0 1 3 0 3 3 0 0 3 3 3 1 3 0 2 3 3 3 0 3 0 0 1 3 1 1 3 0 0 0 3 1 1 0 1 3 2 0 3 2 3 0 1 0 2 1 1 2 3 2 2 1 2 1 2 1 1 0 1 1 1 3 2 2 1 3 1 1 0 2 1 1 1 0 1 0 0 0 0 2 2 0 0 3 2 3 3 2 3 3 3 2 1 2 3 2 3 2 2 1 1 0 1 0 2 3 2 1 3 0 1 0 1 2 0 3 1 2 0 1 1 2 0 3 1 3 2 1 2 2 1 3 0 1 2 3 3 1 3 2 1 2 3 0 3 0 2 2 3 0 1 3 1 0 2 2 1 0 2 3 1 0 1 3 3 2 2 0 0 2 1 0 0 3 2 2 3 1 0 2 1 0 0 0 2 0 2 1 3 1 0 3 1 1 1 1 1 2 0 3 3 0 3 0 0 2 0 3 1 1 3 1 0 2 2 3 2 2 2 2 2 1 0 3 3 2 0 0 0 1 2 3 2 1 2]
The code I use to predict:
training_data.reset()
predicts = model.predict(training_data, steps=20, verbose=1)
predict_class = np.argmax(predicts, axis = 1) # could this be the problem?
errors = np.where(predict_class != truth)[0]
print("No of errors = {}/{}".format(len(errors),training_data.samples))

Where training_data is an image generator defined as follows:
training_gen = ImageDataGenerator(
    rescale=1./255,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.15,
    horizontal_flip=True,
    vertical_flip=True,
    preprocessing_function = preprocess_img
    )

training_data = training_gen.flow_from_dataframe(
    dataframe=df_train,
    directory = train_dir,
    x_col="ID",
    y_col="Truth",
    target_size=(height,width),
    batch_size=batch_size,
    # shuffle=True,
    class_mode="categorical",
    seed=seed_augment
)

Am I misunderstanding when I assume that the predict function will give me good accuracy on training data? Does this imply that my model is actually really bad?
Any direction would be appreciated.

The problem with the way the prediction is done in the code found in the question is that the generator is shuffling the data (even though the shuffle=True is commented out, True is the default). Additionally, the batch_size in the code is set to 16. Setting batch_size = 1 produced the correct output for me, and the resulting accuracy increased to what I was expecting.
Oh, and don't use the same generator to test the predict function!
Hope this helps someone! :)


How to analyse player and enemy position for data analysis I am trying to analyse a Serious Game for students learning.

In one of the game levels,
There are multiple positions like Player Position (x, y, z), enemy position (x, y, z), Player shot at position (x, y, z).
Other features like items acquired,

How do I use these for analysis, for e.g. to categorize some students as "more-exploring" than others.

I am sorry if the question is not clear. But I am just confused with the project.

Either do unsupervised learning with something like k-means clustering or DBScan where you attempt to segment students into groups and see if you can discern any insights based on the cluster generated or pick a threshold for certain categories, create a class column and label each student, and do a classification model.


Zero Mean and Unit Variance I'm studying Data Scaling, and in particular the Standardization method.
I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit variance.

Can you explain me ?

The questions of whether and why it's important, depends on the context.

For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it.
For linear predictors, for example, scaling can improve interpretability of the results. If you'd like to think of the magnitude of the coefficients as some indication of how much a feature is affecting outcome, then the features has to be scaled somehow to the same area.
For some predictors, in particular NNs, scaling, and in particular scaling to a particular range, can be important for technical reasons. Some of the layers use functions that effectively change only within some area (similar to the hyperbolic-family of functions), and if the features are too much out of the range, saturation can occur. If this happens, numerical derivatives will work badly, and the algorithm might not be able to converge to a good point.


Cleaning data automatically Do you use automatic cleaning tools for data?

I mean something similar to h2o.ai's auto ml function but applied to preprocessing data.
Or do you always clean data 'by hand'.

Cleaning data largely varies from data to data. Considering you are talking mostly about unstructured data, it can be of two types, namely Image data and text data. The process of cleaning both these data depends on the type of goal required to be achieve using these. A little brief process could be explained as follows:

Image data: While performing convolutional neural network on the data, the process of feature extraction from the data is automated. Specific pre-processing need to be carried out aiming it to the target. This includes:

Keeping the data set close to the actual data on which the model is to be tested.
Removing any unnecessary images that do not co-ordinate with the corresponding label.
Resizing the images as per the model requirement.

Text data: Text data has a robust procedure for of pre-processing consisting of Removing stop words, tokenizing, lemmetizing the tokens etc. At times when we are using the word to vec approach, we need to convert the text tokens to their corresponding word-vectors.

Generally functions designed for these processes are fed into the pipeline and so individually every entry is not required to be processed in a similar way.

There are some papers and blogs I would suggest you to read: Towards automated patient data cleaning , Turn Messy Data into Tidy Data


keras predicts nan values I implemented a Keras model for my all-integer dataset with values greater than or equal to 0. The train data has dimensions of (393, 108) and prediction data has (1821, 108). Code is as follows.
import keras
from keras.models import Sequential
from keras.layers import Dense

X = data.iloc[:, :-1]
y = data.iloc[:, -1]

model = Sequential()
model.add(Dense(X.shape[1]-1, input_dim=X.shape[1], activation='tanh'))

for i in range(X.shape[1] - 2, 2, -100):
    model.add(Dense(i, activation='tanh'))

model.add(Dense(1, activation='tanh'))

opt = keras.optimizers.Adam(learning_rate=100)
model.compile(loss='categorical_crossentropy', optimizer=opt)
model.fit(X, y)
model.predict(X0)

I am getting all nan values as results.
array([[nan],
       [nan],
       [nan],
       ...,
       [nan],
       [nan],
       [nan]], dtype=float32)

Your input is not standardized
The learning rate is way too high, start with the Default i.e. 0.001

Other suggested changes -

Use "relu" in the hidden/input layer
OHE the target
If the target is multi-class, the output layer should have same number of Neurons with softmax as activation
Data points are very less, Neural Net might not be the best option


Signal correlation - matching specific points Question: What are some recommended techniques for matching specific patterns in data sets?
Background
I have several thousand sites for which I have collected time series data. In the example image below we have increasing time in y-direction with data streams from 2 different instruments shown from two sites. I have added hand-picked, color-coded correlation markers for four significant events.  Normally we would pick 10-15 of these markers at each measurement location.
Typically, these markers are correlated by hand, however, with 1000s of sampling sites this is not feasible in a reasonable amount of time.  I have performed hand correlations on ~150 of the data streams, but would prefer to use an automated or semi-automated process to do the correlation.
We have the following beliefs about the data:

By picking a sparse data set (e.g., the first 150 hand picked correlations) we can make a strong guess about the location of the marker in adjacent sites.
The time-separation between markers is not constant, but has only minor variation
Marker A will always come before Marker B, and Marker B before Marker C and so on.
Not all markers will be present at all measurement sites
Not all measurement sites were able to evaluate a full time series (in example below, imagine if one had been cut off at 8000ms instead of going past 10,000ms
The data can easily be normalized to have a similar range of values.

Prior Approach
I have used both DTW and Fast DTW (dynamic time warping) to perform the task, but it only works well when there is a full and complete data stream and when all markers are present.  The downside to DTW is that it is an O(N2) approach, and with data streams of 20,000+ samples and about 5000 measurement sites, it is simply too computationally slow.
Restatement of Question

What are some recommended techniques for matching specific "markers"
in time series data?
Are there techniques that just evaluate certain
portions of the data stream vs. something like dynamic time warping
that evaluates the whole data stream?
What data pre-processing should
I consider to make this computationally easier?

Example Data

I'm no expert with signal processing but my first attempt to solve the complexity issue would be to downsample the time series. For instance if you resample a 10k points series to say 100 points, methods such as DTW can be applied to get a first approximation of the "global" similarity between 2 signals. This would take much less time than comparing the full time series, and then based on the result (for instance using a threshold) only the pairs of signals with high similarity would be compared fully. In other words, the first step acts as a filter so that only a subset of the pairs need a detailed comparison.