How to sample a statistically uniform dataset I have a data set of with a distribution that looks something like this:

I need to take random sample data from the set so that distribution will be more even. Something like this (take the data in the green area):

I know how to do this by taking the data, putting into separate "buckets"
 (distribute the data into X buckets, taking up to Y samples from each bucket), but I was wondering if there is there an easier way.

P.S. the result doesn't have to be 100% accurate - a good approximation is enough.

There are a variety of sampling options:

Sample directly from the data which perfectly models the empirical distribution.
Fit a kernel density estimation (kde). Sample from the estimated kde function.
Create a histogram of the data, aka bin the data. Then treat each histogram as a probability mass function (pmf). Sample from bins proportional to their frequency.

You can create variations of the data or distributions:

Apply a transformation to the data. For example, a log transformation will transform skewed distribution to be more normal.
The histogram values could be changed to any shape.

Then sample from the changed data or distributions.

The most extreme option would define a uniform distribution across the data domain and sample from that distribution. The green box is a uniform distribution.


Spatial Transformer Networks vs Deformable Convolutions As I understand STN as described by the the deepmind paper https://arxiv.org/abs/1506.02025

allow a neural network to learn how to perform spatial transformations on the input image in order to enhance the geometric invariance of the model.

whereas Deformable Convolutions https://arxiv.org/abs/1703.06211

add 2D offsets to the regular grid sampling locations in the standard convolution.

From the paper:

...deformable convolution does
not adopt a global parametric transformation and feature
warping. Instead, it samples the feature map in a local
and dense manner. To generate new feature maps, it has
a weighted summation step, which is absent in STN.

To me STNs augment the input image whereas DC augment the kernel shape. Results should be similar/same in terms of field of view of the network? Please correct me if mistaken.

The main difference, as they mentioned in the paper, is that STN has a global parameter to transform the features. That is, it computes one set of parameters to transform the input. DC computes a 2D offset map for each location in the input, so for each location in the input features there is a separate transformation.


Predicting New Data with Naive Bayes Say I had the following training set for a Naive Bayes algorithm.

   Outlook    Person    Play Golf?
   -------    ------    ----------
   Sunny      Joe       Yes
   Sunny      Mary      Yes
   Raining    Joe       Yes
   Raining    Mary      No
   Raining    Harry     Yes

If try to predict whether Harry will play golf on a sunny day (which I have no data for).
Would it be correct to exclude the person attribute and use the remaining outlook attribute to calculate the probability of this happening? Or could that potentially cause problems with a larger data set that I'm unaware of?

Among Naive Bayes assumptions the main one is that features are conditionally independent. For our problem we would have:

$$P(Play|Outlook,Person) \propto P(Play)P(Outlook|Play)P(Name|Play)$$

To address question is Harry going to play on a sunny day?, you have to compute the following:

$$P(Yes|Sunny,Harry) = P(Yes)P(Sunny|Yes)P(Harry|Yes)$$
$$P(No|Sunny,Harry) = P(No)P(Sunny|No)P(Harry|No)$$

and choose the probability with bigger value.

That is what theory says. To address your question I will rephrase the main assumption of Naive Bayes. The assumptions that features are independent given the output means basically that the information given by joint distribution can be obtained by product of marginals. In plain English: assume you can find if Harry plays on sunny days if you only know how much Harry plays in general and how much anybody plays on sunny days. As you can see, you simply you would not use the fact that Harry plays on sunny days even if you would have had that record in your data. Simply because Naive Bayes assumes there is no useful information in the interaction between the features, and this is the precise meaning of conditional independence, which Naive Bayes relies upon.

That said if you would want to use the interaction of features than you would have either to use a different model, or simply add a new combined feature like a concatenation of factors of names and outlook.

As a conclusion when you do not include names in your input features you will have a general wisdom classifier like everybody plays no matter outlook, since most of the instances have play=yes. If you include the name in your input variables you allow to alter that general wisdom with something specific to player. So your classifier wisdom would look like players prefer in general to play, no matter outlook, but Marry like less to play less on Rainy.

There is however a potential problem with Naive Bayes on your data set. This problem is related with the potential big number of levels for variable Name. In order to approximate the probability there is a general thing that happens: more data, better estimates. This probably would happen with variable Outlook, since there are two levels and adding more data would probably not increase number of levels. So the estimates for Outlook would be probably better with more data. However for name you will not have the same situation. Adding more instances would be possible perhaps only by adding more names. Which means that on average the number of instances for each name would be relatively stable. And if you would have a single instance, like it is the case for Harry, you do not have enough data to estimate $P(Harry|No)$.

As it happens this problem can be alleviated using smoothing. Perhaps Laplace smoothing (or a more general for like Lindstone) is very helpful. The reason is that estimates based on maximum likelihood have big problems with cases like that.

I hope it answers at least partially your question.


Why is IoU said to be non-differentiable? I have been trying to find an answer online but I couldn't really find one. If anyone could help me I would appreciate it

Intersection over Union (IoU) can be differentiable. The paper "UnitBox: An Advanced Object Detection Network" by Yu et al. describes it detail.


When is there enough data for generalization? Are there any general rules that one can use to infer what can be learned/generalized from a particular data set?  Suppose the dataset was taken from a sample of people.  Can these rules be stated as functions of the sample or total population?

I understand the above may be vague, so a case scenario: Users participate in a search task, where the data are their queries, clicked results, and the HTML content (text only) of those results.  Each of these are tagged with their user and timestamp.  A user may generate a few pages - for a simple fact-finding task - or hundreds of pages - for a longer-term search task, like for class report.

Edit:  In addition to generalizing about a population, given a sample, I'm interested in generalizing about an individual's overall search behavior, given a time slice.  Theory and paper references are a plus!

It is my understanding that random sampling is a mandatory condition for making any generalization statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, clarifying the @ffriend's comment, I believe that you have to calculate needed sample size, based on desired values of confidence interval, effect size, statistical power and number of predictors (this is based on Cohen's work - see References section at the following link). For multiple regression, you can use the following calculator: http://www.danielsoper.com/statcalc3/calc.aspx?id=1.

More information on how to select, calculate and interpret effect sizes can be found in the following nice and comprehensive paper, which is freely available: http://jpepsy.oxfordjournals.org/content/34/9/917.full.

If you're using R (and even, if you don't), you may find the following Web page on confidence intervals and R interesting and useful: http://osc.centerforopenscience.org/static/CIs_in_r.html.

Finally, the following comprehensive guide to survey sampling can be helpful, even if you're not using survey research designs. In my opinion, it contains a wealth of useful information on sampling methods, sampling size determination (including calculator) and much more: http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm.


What does it exactly mean when we say that PCA and LDA are linear methods of learning data representation? I have been reading on representation learning and I have come across this idea that PCA and LDA are linear methods of data representation, however, auto-encoders provide a non-linear way. Does this mean that the embedding learned by PCA can be transformed only linearly to reproduce data points ?

LDA is linear classifier since the classification boundary in LDA has the following form :

As you can see, the above equation is linear in $x$. In above equation, ${\Pi_i}$ is the prior probability of class $i$, which is estimated from the training data and ${\mu_i}$ is the mean of class $i$ also estimated from the training data. ${\Sigma}$ is the common covariance matrix of all classes (it is assumed to be same in all classes in LDA, and that's why we get linear boundary, if you don't assume the same covariance matrix, the classification boundary no longer remains the linear in $x$).
PCA on the other hand is not a regression/classification algorithm. It is rather a feature extraction/ dimensionality reduction method, which helps you to represent your data in lower dimensions. It generally extracts the most significant $k$ features of your data. The value of $k$ is decided by you based on how many features you want to keep in your data. Or in what dimension you want to represent your data. PCA is the best representation of your data in to a lower dimensional space. This is linear transformation because of following equation :
$$ y \space = \space  W^Tx $$
As you can see, this equation is also linear in x. In this equation, the matrix W is a matrix obtained from the covariance matrix of your data. The first row of matrix $W$ corresponds to the eigen vector of covariance matrix of $x$ corresponding to highest eigen value. This is because that eigen vector gives the maximum variation in your data. (This is proved in the optimization of PCA). The second row corresponds to second max eigen value because it represents direction with second highest variation in your data. You can refer to the notes of Prof. Ali Ghodsi or Prof. Andrew NG for the proof of why we select the eigen vectors in this order in PCA.
Now coming back to your question about reproducing the data only linearly. Yes, you can only reproduce back your data by linear transformation in PCA. Why do you want want to reproduce your data non linearly when your transformation was linear? Even if you want to reconstruct your data non linearly from transformed data $y$, do you have any enough information to do so? You just have a matrix $W$ using which you linearly transformed your data $x$ into a lower dimensional space. A non linear reconstruction might look like this :
$$ x \space = \space y^TW_1^Ty \space+\space W^{-T}y $$
This is a non linear reconstruction. But do know what is $W_1$ here? Even if you try to estimate $W_1$, you'll endup getting it as a zero matrix since $y = W^Tx$ and so $W^{-T}y = x$. So, any other factor in the estimation will give you nothing but the zeros as a coeffocients of the other non linear factors (factors like $y^Ty$).


How to get the weights of a linear model by solving normal equation? In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168).

Linear Model:

$f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$

MSE Loss:

$J(\mathbf{w},b)= \frac{1}{4} \sum(f^*(\mathbf{x})-f(\mathbf{x;\mathbf{w},b})) $ , where  $f^*(\mathbf{x})$ is the XOR function.

Normal equation:

According to the same book on page 107, the weights can be obtained by solving the gradient of the loss function, which will result in a normal equation (5.12).

$\mathbf{w}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$

My Attempts:

Since it is a XOR function, we know that if the input is $\mathbf{X}=\begin{bmatrix}
    0 & 0 \\
    0 & 1 \\
    1 & 0 \\
    1 & 1 \\
\end{bmatrix}$, then the corresponding output will be $\mathbf{y}=\begin{bmatrix}
    0 \\
    1 \\
    1 \\
    0 \\
\end{bmatrix}$.
So I just plug everything into the normal equation as shown above.

However, the solution I get is
$\mathbf{w}=
\begin{bmatrix}
    \frac{1}{3} \\
    \frac{1}{3} \\
\end{bmatrix}$.

What am I doing wrong here? And also how to find the bias $b$?

Hey I know this is a year old but I wanted to provide the answer in case you haven't got it. So page 124 says how we derive the normal equations, however the following few paragraphs explains how to deal with a bias term.
You've got our dataset right, however we need to add an extra 1 to take the place of our bias in the weight vector. I'll append the ones at the end.
$\mathbf{X} = \begin{bmatrix}
0 & 0 & 1 \\
0 & 1 & 1 \\
1 & 0 & 1 \\
1 & 1 & 1 \\
\end{bmatrix} $
You've got our $y$ vector correct though, i'll write it down below for completeness sakes
$\mathbf{y} = \begin{bmatrix}
0 \\
1 \\
1 \\
0 \\
\end{bmatrix} $
Now we compute solution to the normal equations, $w = (X^{T}X)^{-1}X^{t}y$, below
$\mathbf{w} = \begin{bmatrix}
0 \\
0 \\
\frac{1}{2} \\
\end{bmatrix}$
Now the element in the 3rd position of our vector is the bias term and hence $b = \frac{1}{2}$ and the first two element of $\mathbf{w}$ is our weight vector equalling zero.
Hopefully you find this useful!


convert single index pandas data frame to multi-index I have a data frame with following structure:

df.columns
Index(['first_post_date', 'followers_count', 'friends_count',
       'last_post_date','min_retweet', 'retweet_count', 'screen_name',
       'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],
        dtype='object')

Inside the tweets series, each cell is another data frame containing all the tweets of an user.

df.tweets[0].columns
Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')

I want to convert this data frame to a multi-index frame, essentially by breaking the cell containing tweets. One index will be the uid, and another will be the id inside tweet.

How can I do that?

link to sample data

One way to pull the embedded dataframe up into the main dataframe and build a multi index is like:

Code:

def expand_tweets(tweets_df):
    tweets = []
    for uid, sub_df in tweets_df.set_index('uid').tweets.iteritems():
        sub_df['uid'] = uid
        tweets.append(sub_df)
    return pd.concat(tweets).merge(
        tweets_df.drop('tweets', axis=1).reset_index(),
        how='outer', on='uid').set_index(['uid', 'id'])

How:

Pull all of the tweet dataframes out of main dataframe using uid as an index, and concat() them together with their uid.
Then merge the main dataframe into the concatenated tweets dataframe.
Set the desired index.

Test Code:

import json
import pandas as pd
with open('test.json') as f:
    df = pd.DataFrame(json.load(f))
df['tweets'] = df.tweets.apply(lambda x: pd.DataFrame(x))

print(expand_tweets(df).text.head())

Results:

uid         id
1153859336  655060275025047552    Article on my new Haunted Stevenage book Paran...
            653912439940120576    Big thank you to @bobfmuk for interviewing me ...
            643709869908996096    Another interesting non-toadstool tweet today,...
            547107275681579008    @sisax67 Thanks, Simon. All the best to you &a...
            546693940024733696    Paul Adams @SkySportsDarts The Wanderer from W...
Name: text, dtype: object


categorizing articles using deep learning Is it possible to write a program using deep learning that categorizes an article (news, entertainment,..., etc. ) into a positive category or negative category based on a specific theme?

Yes, this is classic sentiment analysis. You'll need training data in order to write this program.

Typically, people will build a language modeling using the text data in the article, tweets, posts, etc. Most folks specify the language model using a Recurrent Neural Network with a Long Short Term Memory. Here's a blog post that has code linked to their github.

Before diving into a more advanced deep neural network like an LSTM, it may be easier to start with a multilayer perceptron using n-grams.

It seems like you may be asking for a couple of different things. So, let me be very explicit. The models I listed above take as input the articles and a label that evaluated the articles as either positive or negative sentiment, the model then returns the likelihood of a given article being positive or negative sentiment. It in no way returns a topic.

To return the "topic" your best bet would be to run Latent Dirichlet Allocation, which would then return the likelihood of a given article being generated from a given topic that was previously learned. It's important to note that these will only be from topics learned on previous training data and not on new topics.

There are standard algorithms available to visualize topic models (e.g., LDAVis) and the embeddings of language models (e.g., word2vec visualizations).

It's worth noting that there is research being done in learning these items jointly (i.e., the latent topic and the classification of good/bad). I'd recommend reviewing this paper.

It may be worth reviewing these items more thoroughly before actually diving into applying them. It sounds like you may not have a complete picture of what your goal is and how deep learning can help you.


Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?) I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given second, then that second's data point would contain a positive value in the pertinent feature. If there are more seller initiated trades, then there would be a negative value. And if either there is an equal amount of buy vs seller initiated trades OR if there are simply not any trades in a given second, there will be a 0 for the feature in that data point. Along with this feature, there are several other features that are based on what occurred in the preceding seconds (eg if the value discussed above was 12 for the data point immediately preceding the current point, then the second feature for the current data point would be 12 - please let me know if this is not clear) After much troubleshooting, I have concluded that if there are too many data points with too many 0's for features, the classifier simply wont work. When I print out the probabilities of evaluation data points falling into different classes, I simply get

0:NaN,1:NaN

for all model evaluation points I try to classify. (I am using logistic regression from apache-mahout. In total have 183 features, but over 40million data points. There are three categories to which the data point can be classified)

I have found that if I set the default value to 1, then I no longer encounter this error, e.g. if there are no trades, the value will be a 1, if there is one seller initiated trade, the value will be 0.

So with all this in mind, I have two related questions:

1) Has anyone else encountered this issue? e.g. if you have a vector with x features, and for a majority of the data points, a majority of the features contain 0's, is this know to give issues?

2) Is shifting all values up by a constant (such as 1) a valid fix to this issue? I assume that if this constant is applied to all values, then it shouldn't skew the data, but I figure it won't hurt to check with the experts.

Also, I'm new to this, so if you believe that my question could use more info please let me know, and if you could give me ideas of what information to include, it would be greatly appreciated.

thanks in advance

From your question I am not sure how you are using logistic regression. The "vanilla" version of logistic regression gives a binary classifier, but your target has three values: a positive value, a negative value, and zero. The Apache site, on the page for logistic regression refers to the thesis of Paul Komarek, in which "logistic regression" refers to the binary classifier. So your problem might be that there your target variable has 3 values.

Assuming that that is the case, you have many options, for example:
 - a) use "multi-nomial logistic",
 - b) use ordinary logistic regression 3 times to fit 3 models, one per level, to distinguish it from the other two combined


Perform classification on market basket analysis I have the following problem that I don't know how to solve:

I have the data for different market baskets with a corresponding class. So for example I know:

Student - {beer, milk, water}
Professional -  {nuts, pizza, bananas}
Student - {oranges, tomatoes, beer}
...

Is there a method to create a classification model so that I can use the content of the market basket in order to determine the corresponding class (Student, Professional, ...)? Thank you!

In case that the number of items is quite small, turning the problem into a classification problem will be the most convenient solution. Use each item as a feature, the class as a concept. Now you can apply the methods that @OvisAmmon recommended on or other classification methods.

However, usually in market basket analysis there are many items and each user buy few of them.  That leads to a very sparse matrix and most classification methods have problems dealing with them.

A common method to analyze market basket data is using
Association rules which are implemented by first locating frequent item sets.

You can use association rules on order to learn on market basket in the following way:

Add item for every class
Add the owner class to its basket (e.g, transform {milk, beer} basket of a student in to the basket {mlik, beer, student}
Use a frequent item set algorithm to find them
Remove all frequent item sets that don't have a class member in them. Such sets can't help deduce the class.
From the frequent items set find rules that imply the classes (with good enough support and confidence
Use the these rules that match a basket in order to predict class

You should note some points:

Some association rules with different implication might apply to the same basket. You might have the rules milk -> X and beer -Y and the basket {milk, beer}. If the set {milk, beer} is frequent enough you might have a rule specifically to it. Otherwise you'll need a met rule in order to base by the rules. Such rules might be taking the rule with the highest confidence or the most likely class.

Association rules are based only on the existence of item, not on the lack of them. If students never buy diapers, you will not find such association rule. You can add "no diapers" as an item to your data set but that will turn it from a sparse one to a full one and the storage might be unfeasible.

The complexity of association rules is exponential. The is simply since the output it self is exponential , as can be noted when you have a Clique in the data.
Usually people don't try to find all association rules due to that problem, which is the practical thing to do.
However, it means that there might be association rules in your data that you won't find.


How to measure the performance of a domain adaptation /Transfer learning technique? Given that the performance you achieve depends on how far the target from the source domain is, how can you judge the performance of an algorithm?

You can measure the divergence between the source and target domain using KL-Divergence (there are some ways to estimate k-l divergence e.g. depdended on k-nn algorithm). Then you can check if there is a correlation between the divergence and the accuracy of the models considering a few cases of source-target pairs of datasets. You can compare several algorithms of Transfer Learning/Domain Adaptation using the same source-target datasets.


Why is convnet transfer learning taking so long? I am using transfer learning to train a binary image classification model using keras' pretrained VGG16 model. The code can be found below :

training_dir = '/Users/rishabh/Desktop/CyberBoxer/data/train'
validation_dir = '/Users/rishabh/Desktop/CyberBoxer/data/validation'
image_files = glob(training_dir + '/*/*.jpg')
valid_image_files = glob(validation_dir + '/*/*.jpg')
# importing the libraries
from keras.models import Model
from keras.layers import Flatten, Dense
from keras.applications import VGG16
#from keras.preprocessing import image

IMAGE_SIZE = [64, 64]  # we will keep the image size as (64,64). You
can increase the size for better results.

# loading the weights of VGG16 without the top layer. These weights are
trained on Imagenet dataset.
vgg = VGG16(input_shape = IMAGE_SIZE + [3], weights = 'imagenet',
include_top = False)  # input_shape = (64,64,3) as required by VGG

# this will exclude the initial layers from training phase as there are
already been trained.
for layer in vgg.layers:
    layer.trainable = False

x = Flatten()(vgg.output)
#x = Dense(128, activation = 'relu')(x)   # we can add a new fully
connected layer but it will increase the execution time.
x = Dense(num_classes, activation = 'softmax')(x)  # adding the output
layer with softmax function as this is a multi label classification
problem.

model = Model(inputs = vgg.input, outputs = x)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
from keras.preprocessing.image import ImageDataGenerator
from keras.applications.vgg16 import preprocess_input

training_datagen = ImageDataGenerator(
                                rescale=1./255,   # all pixel values will be between 0 an 1
                                shear_range=0.2,
                                zoom_range=0.2,
                                horizontal_flip=True,
                                preprocessing_function=preprocess_input)

validation_datagen = ImageDataGenerator(rescale = 1./255, preprocessing_function=preprocess_input)

training_generator = training_datagen.flow_from_directory(training_dir,
target_size = IMAGE_SIZE, batch_size = 200, class_mode = 'categorical')
validation_generator =
validation_datagen.flow_from_directory(validation_dir, target_size =
IMAGE_SIZE, batch_size = 200, class_mode = 'categorical')

training_images = 3717
validation_images = 885

history = model.fit_generator(training_generator,
               steps_per_epoch = 3717,  # this should be equal to total number of images in training set. But to speed up the execution, I am only using 10000 images. Change this for better results.
               epochs = 1,  # change this for better results
               validation_data = validation_generator,
               validation_steps = 885)  # this should be equal to total number of images in validation set.

I am training it on just 3700 images but still a single epoch is taking around 10-12 hours. Is this supposed to happen ? Am I doing anything wrong ? I had to downgrade my keras to 2.1.4 for the code to run so is it something affecteing the learning ?

It might have to do with ImageDataGenerator. Data augmentation can be computational expensive. Remove that code and see if the model trains faster.


How to create an ensemble that gives precedence to a specific classifier Suppose that in a binary classification task, I have separate classifiers A, B, and C. If I use A alone, I will get a high precision, but low recall. In other words, the number of true positives are very high, but it also incorrectly tags the rest of the labels as False. B, and C have much lower precision, but when used separately, they may (or may not) result in better recall. How can I define an ensemble classifier that gives precedence to classifier A only in cases where it labels the data as True and give more weight to the predictions of other classifiers when A predicts the label as False.

The idea is, A is already outperforming others in catching true positives and I only want to improve the recall without hurting precision.

Feature-Weighted Linear Stacking might be what you are looking for.

  FWLS combines model predictions linearly using coefficients that are
  themselves linear functions of meta-features.

In your example you can use the meta-feature "Does A label the example as True?"


Handling hierarchical category independent variables I have data with huge categorical attributes.

For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes.
If if take dummy variable on these columns the column count is increased to 1000.

How to handle this kind of hierarchical attributes for a classification problem ?

Thanks !!

I'd suggest the following:

3 features, one for each level main_column, sub_column1, sub_column2
2 additional features representing the hierarchical relation:

main_column/sub_column1
main_column/sub_column1/sub_column2

This way the training can select the most informative level of information between main_column, main_column/sub_column1, main_column/sub_column1/sub_column2. Depending on the data and algorithm used it might also make sense to discard rare cases for a subcategory and use a kind of "misc" category instead.


Improve the speed of t-sne implementation in python for huge data I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec).
I am using TSNE implementation from sklearn.manifold module for it and the major problem is time complexity. Even with method = barnes_hut, the speed of computation is still low. Some time even it runs out of Memory.

I am running it on a 48 core processor with 130G RAM. Is there a method to run it parallely or make use of the plentiful resource to speed up the process.

You must look at this Multicore implementation of t-SNE.

I actually tried it and can vouch for its superior performance.


What are the inputs to a logistic regression? Probability or trial result? It is a very basic question, but cannot find a satisfactory answer to. When we do logistic regression, what are the inputs? Suppose we have a dataset of students giving number of hours each student spent studying, and the end result of whether they passed or failed. The wikipedia article on LR goes to do a fitting with the logistic curve based on probability as a function of number of hours. But how do I get the probabilities if all I have is whether they passed or failed?

From Wiki: The graph shows the probability of passing the exam versus the number of hours studying, with the logistic regression curve fitted to the data.

The probability curve is the output of the LR, not the input. Typically during training, the output class (or target class) will be discrete class labels with 1 or 0. During inferencing, the output will be a continuous value between 0 and 1. To generate the probability curve, just feed in different values of "hours studying" into the trained model.


Is there a difference between "classification" and "labeling"? Until recently, I thought that "labeling" and "classification" are synonyms. But when I started another question about terminology in computer vision I thought about it: Is there a difference between "labeling" and "classification"?

I thought that the "class" is the concept you want to detect and "label" is what you assign to data. So the "class" is a concept which leads to the data and "label" is only the name. Hence "labeling" would be the same as "classification" as both want to make a statement about the underlying class which lead to the data.

Articles

A quick search via Google Scholar revealed that some articles use both terms in the title:

Markus Eich, Malgorzata Dabrowska, and Frank Kirchner: "Semantic Labeling: Classification of 3D Entities Based on Spatial Feature Descriptors"
Chunlin Li, Dmitry B. Goldgof, and Lawrence 0. Hall: "Knowledge-based classification and tissue labeling of MR images of human brain"
Ray Blanchard: "The classification and labeling of nonhomosexual gender dysphorias" - another research area but probably it is the same difference between the two words?

So I guess there is a difference between "labeling" and "classification". What is the difference?

Google N-Gram

classification seems to be a much boarder term.

Totally disagree with @Derek Janni. Be careful about notation however you should not get lost in terminology. Those papers you mentioned used the term "labeling" literally but in Machine Learning/Data Mining community, labeling is the process of preparing data for supervised learning (classification)! It has nothing to do with the ML task!

Those papers used the term to show that after supervised learning they can recognize different labels of different objects so they used the term labeling but you probably can not find in any literature that they use these two terms as synonyms.


what's the difference between autoencoder and autoassociative neural networks? Seems like Autoassociative neural networks and Autoencoders do the same thing.
Are they the same, or is there some subtlety I'm not picking up on?

The "autoassociative" paper is from 1992. The field had not settled on the term "autoencoder" for this concept.


Simple Time Series Prediction I have a data set like this. Here the first column is date, the second column is Temperature, third one is humidity, fourth and fifth column are two other boolean data. I have data of 6 years like this.

2010-01-01,25.6,59,0,1
2010-01-02,25.6,60,0,1
2010-01-03,24.2,45,1,1
2010-01-04,26.3,20,0,1
2010-01-05,26.2,17,0,1
2010-01-06,24.3,65,0,0
2010-01-07,23.1,50,0,1
2010-01-08,26.3,25,1,0
2010-01-09,26.6,23,0,1
2010-01-10,24.3,60,0,1

And the label (The variable I want to predict) for this data set is: (boolean)

0,0,0,1,1,0,0,0,1,0

Now I want to implement Machine Learning to predict from this data set for a future time frame. I have almost no knowledge on machine learning. I want to use python to do it. Which library or methodology will be best and easiest for this? And can I have a simple sample code?

Your problem looks to me more like a classification problem than a time series problem. My suggestion: Split the date into several sub-variables (year, month, day, week-of-day (maybe). Then just use this and the other values as input for a classification algorithm. Ideally you try several. I can recommend sklearn for this (http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html)

One advice: Depending on the classification algorithm you need to first normalize your data. I find the following blog useful for entry-level problems (with code examples--  if you want to approach this from a time-series perspective he also talks about this) and documentation http://machinelearningmastery.com/blog/.

Hope that helps.


How to use binary relevance for multi-label text classification? I'm trying to use binary relevance for multi-label text classification.
Here is the data I have:

a training set with 6000 short texts (around 500-800 words each) and
some labels attached to them (around 4-6 for each text). There are
almost 500 different labels in the entire set.
a test set with 6000 shorter texts (around 100-200 words each).

The difference of size between my two sets exists because of the source is different.

So, I want to use binary relevance to find the labels of the texts in the test set. To do it, I created a dictionary with all the different words in the entire training set and removed stop words, words who appear only once and words who appear in more that 10% of the texts. I got 14714 different words in my dictionary.
My idea was to create a matrix where each row represents a document and each column a words and each value was the number of occurrence of a word in a document. But with 14714 words and 6000 documents, I will get a matrix of 88 millions of integers! I tried, just to see, to create it and my laptop didn't support it. :)
I even didn't have the time to create my Y matrix and generate a model (I wanted to use a logistic regression) for only one label...

So, my questions are:

Was it a good way to make multi-label classification or is there a better method?
Is it a problem to have a training from one source and to use it to make a model to predict data from another source? Is the different size of the documents a problem?
Did you use logistic regression for this kind of problem?

Thank you!

Edit: I also want to add the most frequent words in my dictionary (after the cleaning part) are common words and totally useless in the field of my research (biology): used, much, two, use, possible, example, ... How can I pass through it?

Just some views/suggestions. After removing the stop words did you stem/lemmatize the text you got?  That would probably reduce the number of unique words in your corpus and brings some forms of words to the same level. But caution while using stemming as it sometimes create noise.
Try postagging and see what can be the important tags you want to keep and eliminate the ones that you feel are giving less relevance to the text.
Did you try to find the some important terms from each document by using tf-idf or chi-square method. They might be helpful to see the relevance of terms for each class/document.
See how far can you reduce the dimensions of the matrix after using the above, if not you have already used them and then apply logistic regression or whatever classifier you wanted to.

On your second question.

I don't think having data from different sources should be a problem since you are trying to create model based on keywords, Hopefully it should be able to see. I am not confident on this part.

I had worked on text classification before with bigger documents and larger corpus with not so fruitful results with various models.

Hope this helps.


Flattening output before calculating metrics I use scikit-learn to calculate precision, recall and f1 scores which only accept 1D arrays, but my model's outputs are 2D (binary segmentation maps). My question is, is it ok to simply flatten the outputs, or is there some other function I should use to calculate the metrics in my case?

You need to evaluate the percentage of pixels in the output map which were correctly classified, and it doesn't matter whether it is represented as 2D or 1D. The demand is just that mask's and the ground truth pixels have to match each other.

You can find an example about Pixel Accuracy here.

So, if you would like, you can convert (stretch 2D) into a 1D vector. However, you can also implement computation of precision and recall by yourself. If you use python, it's easy to leverage construction with np.sum(mask), np.sum(target), or np.sum(mask*target) to calculate it for 2D:

TP = np.sum(mask*target)
FP = np.sum(mask*np.where(target == 0, 1, 0))
FN = np.sum(np.where(mask == 0, 1, 0)*target)
Precision = TP / ( TP + FP )
Recall = TP / ( TP + FN )
F1_measure = 2 * Precision * Recall /(Precision+Recall)


ML regression poor performance I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.

There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.

The data that I am processing look like this below all recorded in 15 minute intervals.

            Date_Time    kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00

The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)

EDIT rolling average plot

Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...

In python if I do a df.describe:

Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...

Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?

Any tips to try or avenues to march down is greatly appreciated :)

EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data

It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.

For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.

So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.


does random forest require more trees to be built if # rows increase in train data I am building a randoForest model on some data. To be able to tune the model faster, I was working with 10% sample. I have tuned it now and now I want to build the model on the whole (100% data).

I am wondering, if I will have to increase the number of trees now that I am building the model on more training data points? Please let me know.

The best way to look at this problem is to consider the extreme case: let's say you duplicate every row in your dataset.

If you train a decision tree directly, there are certain hyperparameters you will want to change. For instance, in sklearn, you will want to double hyperparameters like min_samples_split or min_samples_leaf because in each node you'll have double the observations.

With regards to a random forest, usually, a random forest is learn by bootstrapping with resampling=True in the entire dataset. By doubling every observation, you make each observation just as likely to be picked. Even if you use a random forest which subsamples, doubling the dataset makes no difference.

Now, you are not doubling your data. You are introducing new data which might change the distribution of your data. To the extent that it does, then it will mean that whatever hyperparameters you have previously used might no longer be optimal. So you should just try. But, to answer your question, no.


Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations? I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.

As I would say my knowledge about more complex analysis is quite limited I've done mostly all basic methods to gather a overall view of the data. This includes simple graphs which already help to identify similar patterns. I am also using SAS Enterprise 9.4.

However I am looking for a more number based approach.

Data Set

The (self) collected data set I am using contain around ~54.000 fares.
All fares were collected within a 60 day time window, on a daily basis (every night at 00:00).

Hence, every fare within that time window occurs $n$ times subject to the availability of the fare as well as the departure date of the flight, when it is passed by the collection date of the fare.
(You can't collect a fare for a flight when the departure date of the flight is in the past)

The unformatted that looks basically like this: (fake data)

+--------------------+-----------+--------------------+--------------------------+---------------+
| requestDate        | price| tripStartDeparture | tripDestinationDeparture | flightCarrier |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 725.32    | 16APR2015:10:50:02 | 23APR2015:21:55:04       | XA            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 966.32    | 16APR2015:13:20:02 | 23APR2015:19:00:04       | XY            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 915.32    | 16APR2015:13:20:02 | 23APR2015:21:55:04       | XH            |
+--------------------+-----------+--------------------+--------------------------+---------------+

"DaysBeforeDeparture" is calculated via $I=s-c$ where

I & interval (days before departure)
s & date of the fare (flight departure)
c & date of which the fare was collected

Here is a example of grouped data set by I (DaysBeforeDep.) (fake data!):

+-----------------+------------------+------------------+------------------+------------------+
| DaysBefDeparture | AVG_of_sale | MIN_of_sale | MAX_of_sale | operatingCarrier |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 880.68           | 477.99           | 2,245.23         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 904.89           | 477.99           | 2,534.55         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 1,044.39         | 920.99           | 2,119.09         | LH               |
+-----------------+------------------+------------------+------------------+------------------+

What I came up with so far

Looking at the line graphs I can already estimate that several lines will have a high correlation factor. Hence, I tried to use correlation analysis first on the grouped data. But is that the correct way? Basically I try now to make correlations on the averages rather then on the individual prices?
Is there an other way?

I am unsure which regression model fits here, as the prices do not move in any linear form and appear non-linear. Would I need to fit a model to each of price developments of an airline

PS: This is a long text-wall. If I need to clarify anything let me know. I am new to this sub.

Anyone a clue? :-)

Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.

Airline fares have two components to them: all the actual fares (complete with fare rules and what have you) that an airline has available for a certain route, most of which are published the Airline Tariff Publishing Company (a few special-use ones are not, but those are the exception rather than the rule) and the actual inventory management performed by the airline on a day-to-day basis.

Fares can be submitted to ATPCO four times a day, at set intervals, and when airlines do so, it will usually consist of a mixture of additions, deletions, and modifications of existing fares. When an airline initiates a pricing action (assuming their competitors aren't trying to make their own moves here), they usually have to wait until the next update to see if their competitors follow/respond. The converse goes when a competitor initiates a pricing action, as the airline has to wait until the next update before they can respond.

Now, this is all well and good with respect to fares, but the problem is that, because this is all getting published in ATPCO, fares are the next best thing to public information... all your competitors get to see what you've got in your arsenal, so attempts to obfuscate are not unheard of, such as publishing fares that will never actually be assigned any inventory, listing all the fares as day-of-departure, etc.

In many ways, the secret sauce comes down to the actual inventory allocation, i.e. how many seats on each flight will you be willing to sell for a given fare, and this information is not publicly available. You can get some glimpses by scraping web info, but the potential combinations of departure time/date and fare rules are quite numerous and may quickly escalate beyond your ability to easily keep track of.

Typically an airline will only be willing to sell a handful of seats for a very low fare and the people who snag those have to book quite far in advance lest the fare rules lock them out, or other travelers simply beat them to the punch. The airline will be willing to sell a few more seats for a higher fare, and so on and so forth. They will be quite happy to sell all of the seats for the highest fare they've got published, but this is not usually feasible.

What you're seeing with fares getting higher the closer you get to the day of departure is simply the natural process of having the cheap seats get booked farther out, while the remaining inventory gradually gets more expensive. Of course, there are some caveats here. The RM process is actively managed and human intervention is quite common as the RM team generally strives to meet its revenue goals and maximize revenue on each flight. As such, flights that fill up quickly may be "tightened up" by closing out low fares. Flights that are booking slowly may be "loosened up" by allocating more seats to lower fares.

There is a constant interplay and competition between airlines in this area, but you are not very likely to capture the actual dynamics just from scraping fares. Don't get me wrong, we had such tools at our disposal, and, despite their limitations, they were quite valuable, but they were just one data source that fed into the decision-making process. You'd need access to the hundreds, if not thousands of operational decisions made by RM teams on a daily basis, as well as state-of-the-world information as they see it at the time. If you cannot find an airline partner to work with in order to get this data, you might need to consider alternate data sources.

I'd recommend looking into getting access to O&D fare data from the Official Airline Guide (or one of their competitors) and try to use that for your analysis. It's sample-based (about 10% of all tickets sold) and aggregated at a higher level than would be ideal so careful route selection is imperative (I'd recommend something with plenty of airlines, flying non-stop multiple times a day, with large aircraft), but you may be able to get a better picture of what was actually sold (average fare) and how much of it was sold (load factor), vs. merely what is available for sale at a given point in time. Using that information you might be in better position to at least explore the outcomes of the airlines' pricing strategy, and make your inferences from there.


What does the color coding and normalized values in confusion matrix actually specify? I am unable to infer anything about the model from the following confusion matrix. What is the color coding actually specifying?

For example, when predicted label is 1 and true label is 1, the value in the matrix at that point is 0.20. Does that mean its accuracy? Does it mean that the model is only able to predict 1 only 20% of times when the label is actually 1?

PS: SO URL for a more clear image

Each element_ab shows the probability of predicting label a (horizontal axis) when the true label is b (vertical axis)
For example, when the true label is 0, it will be predicted as label 2 with the probability of 0.14

The color intensity indicates the probability of each element in a row


Which algorithm to use to classify plots based on graphical features? I have plots which I need to classify based on some features. For example, I need to differentiate between the following plots having smooth features or 'valleys' at certain x values. Which machine learning algorithm would be most appropriate to do so? I was thinking a combination of anomaly detection, clustering and classification. Any help is appreciated. Thank you!

Your question is unclear. Please provide more information about the dataset, the dataset size, the target, the motivation.
Until then I can only assume and try to help nontheless.

Assumptions

Since you speak of classification I assume that you have labels.
I further assume that you have access to the underlying data used to produce the plot.
In that exact case, the following will be useful:

Solution

Instead of classifying the plots, you could classify the underlying two-dimensional datasets treating them as time-series.

For your specific case, Dynamic Time Warping distance could be useful. You could therefore use this code:

import numpy as np
from scipy.spatial import distance
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report

#toy dataset
X = np.random.random((100,10))
y = np.random.randint(0,2, (100))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

#custom metric
def DTW(a, b):
    an = a.size
    bn = b.size
    pointwise_distance = distance.cdist(a.reshape(-1,1),b.reshape(-1,1))
    cumdist = np.matrix(np.ones((an+1,bn+1)) * np.inf)
    cumdist[0,0] = 0

    for ai in range(an):
        for bi in range(bn):
            minimum_cost = np.min([cumdist[ai, bi+1],
                                   cumdist[ai+1, bi],
                                   cumdist[ai, bi]])
            cumdist[ai+1, bi+1] = pointwise_distance[ai,bi] + minimum_cost

    return cumdist[an, bn]

#train
parameters = {'n_neighbors':[2, 4, 8]}
clf = GridSearchCV(KNeighborsClassifier(metric =DTW), parameters, cv=5)
clf.fit(X_train, y_train)

#evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))


Could you help me to resolve this exercise of K-NN? I am just a young learner in Data Science. Could you help me to resolve this exercise of K-NN?

Ex.1

The table below provides a training data set containing six observations, three predictors, and one qualitative response variable:

x1  x2  x3  y
0   3   0  Red
2   0   0  Red
0   1   3  Red
0   1   2  Green
−1  0   1  Green
1   1   1  Red

Suppose we wish to use this dataset to make a prediction for Y when x1 = x2 = x3 = 0 using
K-nearest neighbors.

1) With the euclidean distance, what is the prediction with K = 1 and with K = 3 for the test
point (0, 0, 0)?

2) If the Bayes decision boundary in this problem is highly non-linear, then would we expect the
best value for K to be large or small? Why?

An algorithm implementing KNN for classification tasks goes as follows:

Compute the distance (in this case Euclidean) from the test point in question to all points in the training data.
Using the computations from 1), sort the training points in ascending order, according to their distance from the test point in question (smallest distance first).
Take the first K training points from the list in part 2), and collect the classes assigned to each in one list.
Assign to the test point in question the class which appears most often in the list from part 3).

In this case, since we are interested in the Euclidean distance, you can use the formula

$$ d(p, p') = \sqrt{(x_{1}- x_{1}')^{2} + (x_{2}- x_{2}')^{2}  + (x_{3}- x_{3}')^{2}} $$

to calculate the distance between a training point $p=(x_{1}, x_{2},x_{3})$ and a test point in question $p' = (x_{1}', x_{2}', x_{3}')$.

  With the euclidean distance, what is the prediction with K = 1 and with K = 3 for the test point (0, 0, 0)?

For the K=1 case, if my algebra serves me correctly, the training point with the smallest distance to the test point $p' = (0,0,0)$ is the point $p=(-1,0,1)$ (with distance $\sqrt{2}$). Assuming my calculation is correct, this means we should associate the class of the point p, namely the class "Green", to the test point $p'$. The K=3 case can be approached by the same method, but this time we will need to calculate all distances from train points to p', keep the train points with the 3 smallest distances, and assign the class which occurs most often among those 3 train points. I'll leave it to you.

  If the Bayes decision boundary in this problem is highly non-linear, then would we expect the best value for K to be large or small? Why?

Some intuition can be gained by thinking about the K=1 case. In this case, each training point has a neighborhood around it such that any new test points in that region will be classified in the same way as the training point. By definition, this neighborhood is given by those points which are closer to the training point in question than they are to any other training points. These neighborhoods are known as Voronoi cells, and together they form a Voronoi tesselation (see wikipedia for some pictures). The decision boundaries for KNN with K=1 are comprised of collections of edges of these Voronoi cells, and the key observation is that traversing arbitrary edges in these diagrams can allow one to approximate highly nonlinear curves (try making your own dataset and drawing it's voronoi cells to try this out). To answer the question, one can investigate how this ability to approximate nonlinear curves changes as K gets made larger, and try to relate the findings to the existence of a highly nonlinear decision boundary from a Naive Bayes classifier.


Best clustering algorithm to identify clusters and determine the closet cluster each individual response is near? I have a survey where each question is related to a different 'shopper' type (there are 5 types so 5 questions). Each question is either binary (True/False) or scale based.

IE:

1. Do you like to shop at our physical location store ? (True/False)

2. Do our discounts entice you to shop more? a. no b. maybe c. yes

For each response I convert the answer choice to a numerical value. So True becomes 1, answer choice 2C becomes 3 etc.

At this point, I am clueless as too what clustering algorithm to use so I can create clusters for each of the 'shopper' types and measures each individual survey response submitted to determine a single cluster closet to the responses given and label the response as that cluster.

IE. This individual that submitted the survey response is 'location conscience shopper type'

Open to any new method of analysis not just clustering

Since they are categorical variables, I would cluster them using the k-medoids clustering method. Before applying this method, one-hot encode all the predictors.

See a tutorial here:
https://towardsdatascience.com/k-medoids-clustering-on-iris-data-set-1931bf781e05

Sklearn has an implementation:
https://scikit-learn-extra.readthedocs.io/en/latest/generated/sklearn_extra.cluster.KMedoids.html


Database on time to churn in telecomunication I would like to research about time to churn1 in the telecommunication market. Does anyone have a link to such a database? The only ones I found did not include the time of churn, but only if a customer is labeled as churn or non-churn, what I would need is time to event data.

The papers I researched all seemed to use private databases.

1churn is defined here as the moment in time, where a customer quits the service that he/she book from the service provider.

instead of the database, search dataset

https://www.google.com/search?q=telecomunication+churn+dataset&ie=utf-8&oe=utf-8&client=firefox-b-ab


Python: forecast unevenly spaced time-series? My data has timestamps corresponding to the failure occurrences of a specific component in machinery. The timestamps are not uniformly distributed.
My question is:
1) what methods can I use to (almost) accurately to forecast future occurrences (timestamps) of Failure?
2)What other features can I derive?

What I've tried so far:

Since the timestamp sequence is unevenly spaced I've derived a feature datediff= difference between sequential fault occurrences.
Since it is now a univariate time-series I have tried classical time-series forecasting methods like ARIMA and SARIMA (hasn't worked out well)

I am posting the seasonal decompositions of the time-series
freq=7(weekly)

freq=30

acf/pacf

I reccomend you to use Prophet:

  Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

In the documentation you can see that is really easy to implement in python.

Also you could convert the problem to a supervised learning one. You can read this blog where they make an introduction of how to face the problem.


Pandas loads a csv file incorrectly, but without throwing an error So I tried loading some data through pandas to practice manipulating it, but I ran into a slight problem. Basically, the pandas loads the data improperly. And it does so consistently. Let me show you what I mean.
This is a snapshot of the dataset (a csv file)

I think you can more or less see how it looks here. However, when I load it in Pandas, I get this:

As you can see there's a clear distortion. For one thing, the Date column has been pushed from the first column in the spreadsheet to the second in the csv. Its values have also been replaced with what looks to me like a copy of the "Travels" column. Moving on, you can also see that the column I'm interested in ("Total Direct Remittances") is totally incorrect. In the spreadsheet it starts with 342, but when loaded it starts with 23423, and the rest of its values are completely off.
Here's the code:
import pandas as pd

data = pd.read_csv("/home/user/Downloads/IntPayments26102020.csv")

You can get a copy of the data here. Any ideas guys?

You need to read the file with the index_col=False argument:
# wrong way:
df = pd.read_csv("IntPayments26102020.csv")
print(df.head())
# result :
                  Date     Travels  ...  Debt Services And Payments  Total
10/12/2003   234234.00     23424.0  ...                3.574570e+05    NaN
1/12/2005        12.00         2.0  ...                1.160000e+02    NaN
4/28/2006   7119901.48   9950000.0  ...                3.887479e+08    NaN
5/2/2006        346.00    345435.0  ...                4.632060e+05    NaN
5/12/2006    152204.22  11500000.0  ...                4.743224e+08    NaN

[5 rows x 11 columns]

# correct way:
df = pd.read_csv("IntPayments26102020.csv", index_col=False)
print(df.head())
# result :

         Date     Travels  ...  Debt Services And Payments         Total
0  10/12/2003   234234.00  ...                    23423.00  3.574570e+05
1   1/12/2005       12.00  ...                       21.00  1.160000e+02
2   4/28/2006  7119901.48  ...                 45602245.83  3.887479e+08
3    5/2/2006      346.00  ...                    34534.00  4.632060e+05
4   5/12/2006   152204.22  ...                 71560790.20  4.743224e+08

[5 rows x 11 columns]

From the docs:

index_col : int, str, sequence of int / str, or False, default None
Column(s) to use as the row labels of the DataFrame, either given as string name or column index. If a sequence of int / str is given, a
MultiIndex is used.
Note: index_col=False can be used to force pandas to not use the first column as the index, e.g. when you have a malformed file with
delimiters at the end of each line.

From where you can arguably guess why the default setting index_col=None will not work with your data - your rows are also separated by comma ,:
"Date","Travels","Inward Money Transfers","Cash Sales to BDC And Banks","Letters of Credit","Total Direct Remittances","Remittances to Travelex","Remittances to Amex","WDAS","Debt Services And Payments","Total"
"10/12/2003","234234","23424","23423","2342","342","23423","3423","23423","23423","357457",
"1/12/2005","12","2","12","12","21","12","12","12","21","116",


Should we apply normalization to test data as well? I am doing a project on author identification problem. I had applied the tf-idf normalization to train data and then trained a svm on that data.

Now when using the classifier should I normalize test data as well. I feel that the basic aim of normalization is to make the learning algo give more weight to more important features while learning. So once it has trained it already knows which features are important, which are not. So is there any need of applying normalization to test data as well?

I am new to this field. So please ignore if the question appears silly?

Yes you need to apply normalisation to test data, if your algorithm works with or needs normalised training data*.

That is because your model works on the representation given by its input vectors. The scale of those numbers is part of the representation. This is a bit like converting between feet and metres . . . a model or formula would work with just one type of unit normally.

Not only do you need normalisation, but you should apply the exact same scaling as for your training data. That means storing the scale and offset used with your training data, and using that again. A common beginner mistake is to separately normalise your train and test data.

In Python and SKLearn, you might normalise your input/X values using the Standard Scaler like this:

scaler = StandardScaler()
train_X = scaler.fit_transform( train_X )
test_X = scaler.transform( test_X )

Note how the conversion of train_X using a function which fits (figures out the params) then normalises. Whilst the test_X conversion just transforms, using the same params that it learned from the train data.

The tf-idf normalisation you are applying should work similarly, as it learns some parameters from the data set as a whole (frequency of words in all documents), as well as using ratios found in each document.

* Some algorithms (such as those based on decision trees) do not need normalised inputs, and can cope with features that have different inherent scales.


Combining different features as input to Neural Network I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 dimensional vector) and its Valence, Arousal, and Dominance Lexicon (3 dimensional vector). I concatenate them and obtain 1027 dimensional vector for each word.

My questions are:

Should I normalize/preprocess those vector before concatenating them? My lexicon values are between [0,1] and my embeddings are the ELMO embeddings produced depending on a context. Therefore I am not sure about the range of the embeddings.
If yes, should I preprocess one of them before merging, what should I do ?
Is there any good resource that might have an answer for my question?

You should have all of your features in a comparable range. You can leave lexicon values in the [0, 1] range but normalize embedding to have approximately the same range (with using, for example, MinMaxScaler).


Determine what data, does certain column have in common I have an Excel sheet with n columns, these columns contain info about the students. For admissions we have the score of test scores in multiple subject areas, scores from an interview, and scores of a written test and comprehension test. There is a column which contains student's academic level (High, M.High, Middle, M.Low, Low). I want to compare the last column with the others variables and see whether there are common features that passing students have in common.

Is there software for this? If this can be done with excel, how can I do it? Does SPSS provide this kind of analysis?

A good place to start would be to look at the Spearman's Rank Correlation between your dependent variable (academic level) and your individual independent variables (other columns). This should give you a basic indication of whether any of your columns are correlated with academic level.  This should be straightforward to implement in both Excel and SPSS.

If you wanted to go a step further you could set the problem up as one of Multinomial Logistic Regression. This would allow you to build a model to directly attempt to predict academic level from your other variables. I'm certain there's probably some way to do this in Excel, but SPSS can definitely handle it.


Data Interpretation with a double line graph The atmosphere is made up of 4 distinct layers: the troposphere, stratosphere, mesosphere, and thermosphere. Different types of clouds form in the different layers depending on the pressure in the atmosphere and the ambi- ent temperature. The cloud types include nimbus, stratus, cumulus, and cirrus. Figure 1 shows the location of the
barriers of the atmosphere when the temperature and pressure are at an ideal condition for cloud formation. It also shows the different types of clouds formed at the different levels.

->How do you interpret this kind of graph? I sort of get it that as we go higher, pressure increases, while temperature decreases. But the graph looks weird to me and I can't seem to understand the right side of the graph, which sort of "shows that temperature also increases". Can anyone help me explain the concept behind this type of graphical representation? Thanks in advance.

That looks a double axis or dual axis graph but isn't well done or I don't understand clouds well enough. It is almost like a Sankey diagram. Problems:

What does the X axis convey, if anything?
It appears that the cloud types are relatively stratified but I'm not sure what causes the waviness over the X axis. Is it related to the two lines for temperature and pressure?

Compare that to a phase diagram like below. I'm not a chemist, but this is a lot easier for me to interpret. The x axis is the temperature, the y axis is pressure, and the lines clearly show where the material switches phases, similar to cloud types. There isn't ambiguity at any point about which phase would be in place. Now try figuring out what type of cloud you would have in your diagram at 30C and 8psi.


How to measure correlation between several categorical features and a numerical label in Python? I have for a few weeks measured the time it takes for a product to be released through a automated release pipeline. I have several different categorical features such as "Product Category", "Product Owner". Then I also have some numerical categories such as "Hour of day when started the job", "Number of sub products" and more. In total I have around 16 different categories, where each category can take around 10 different values.

I will now try and train a regression model to see if I can predict the lead time(time it takes for the product to go through the pipeline) based on these features.

I would want to see if there are any of these features which are more correlated with the lead time than others. As I have understod it I have to seperate the numerical and categorical features and perform tests seperately on them. For numerical variables I have read about pearsonr and for correlating categorical and numerical variables I have read about ANOVA but I can't seem to find any way of implementing ANOVA in Python. Has anyone any experience with this? And if not, any other tips on how to do this?

Example data frame

        product_category     product_owner    #sub_products     hour     lead_Time_min
              A                  Bill             5             13         123
              B                  Lisa             14            19          40
              B                  Lisa             2             16          20
              D                  Eric             1             11          10
              C                  Ben              7             11          4
              C                  Lisa             14            10          25
              B                  Lisa             2             19          252

If you want to do an ANOVA test, you can do it with scipy and stats package. Link to documentation

You can do it like that

def anova(data):
    if len(data.groupby(level=1)) <= 2:
        raise Exception('ANOVA requires a secondary index with three or more values')

    return pd.DataFrame(
        [f_oneway(*[v for k, v in data[col].groupby(level=1)]) for col in data.columns],
        columns=['f', 'p'],
        index=data.columns)

Source

or just choose the two columns you want to test

statistic, pvalue = stats.f_oneway(data['col1'], data['col2'])

However, I would advise you to take a different path. You need to test how important a feature is in your dataset to predict the lead_time. You can train a simple Decision Tree with the whole dataset and get the feature importance for each of the features. Then you remove those that are below a threshold.

from sklearn.tree import DecisionTreeRegressor

tree = DecisionTreeRegressor().fit(X, y)
tree.feature_importances_


Checking trained CNN on the images I trained my CNN (model) classifier and want to check it on some new images. I have image x, so this syntax works for me for one image:
torch.argmax(model(x))

What if I want to classify 2 more images (different classes),  let's say images y and z? Should I for every image write a new line or is it possible to put in the code above 3 together?

torch.argmax has an extra argument dim which you can specify such that the maximum value is taken over a specific dimension. If you specify the dimension which represents the number of images it will return an array of indices where each value is for one image. For example:
import torch

# 3 images with 5 classes
t = torch.randn(3, 5)

# tensor([[-1.2917,  1.3740,  0.6967, -0.0575,  0.3702],
#        [ 0.5428,  1.0863,  0.3951,  1.8535,  1.0926],
#        [ 0.5865,  0.8522, -0.6858,  0.5297, -0.1320]])

# get the argmax over the first dimension, which specifies the number of images
torch.argmax(t, dim=1)

# tensor([1, 3, 1])


Suggest text classifier training datasets Which freely available datasets can I use to train a text classifier?

We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified our content based on a predefined bag of words we can recommend to him engaging content by getting his feedback on random number of posts already classified before.

We can use this info to recommend for him pulses labeled with those classes. But we found If we used a predefined bag of words not related to our content the feature vector will be full of zeros, also categories may be not relevant to our content. so for those reasons we tried another solution that will be clustering our content not classifying it.

Thanks :)

Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.


How to split natural language script into segments? I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the following categories:
MainConceptDescription-> Explanation of the Main concept(s)
SubConceptDescription-> Explanation of Subconcept related to the main concept
Methodology / Technique-> To achieve something, what should one do
Summary-> Summary of the discussed material or of the whole course
Application-> Practical advise for the concept
Example-> Concept example

Now, for the first 2 i think I should try to apply Latent Dirichlet Allocation to extract the topics. Another idea was to look in the resource name and search for these words in the text. Another idea was to read some of the resources and manually fix some sort of a dictionary for every category and then create regex patterns and search for them in the text.

But the latter seems too lame. So now I am not sure what can I do. I've seen similar works for research papers, however research papers have their own specific expressions etc that are more or less constant and seen in most of the papers, but that is not the case with my video scripts where it's 100% spoken natural language I need to work on. Do you have any ideas how can I approach this? I do have a list of keywords that kind of signifies whether an example follows or a concept is explained, but I am doing this manually which is surely not what I want to do for 563 files that may as well become many more.

Furthermore, I'd like to connect topics found to ontologies in order to enrich the metadata about each file. I have no idea how can I approach this either. Any advice would be VERY appreciated.

Forgive me if my explanations make no sense. I am not too familiar with the terminology. So if you also explain some of the terminology you use, i'd appreciate that too. And please advise on algorithms I can try with.

I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your broader categories; description, methodology, classifier, application, and example. That is, I would manually label some transcripts at the paragraph level. If you don't have paragraph-segmented text, smooth the classification probability over the sentences so they share the same label. Or use a paragraph segmentation model as in Automatic Paragraph Segmentation with Lexical and Prosodic Features.

The classifier could be a CRF or an RNN. The modern way to induce the ontology would be through word embeddings; cf. e.g, Learning Semantic Hierarchies via Word Embeddings. Formerly I would have recommended a hierarchical topic model, such as hLDA; cf. e.g, Non-Parametric Estimation of Topic Hierarchies from Texts with Hierarchical Dirichlet Processes or Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling

Welcome to the site and good luck!


What is the number of neurons for the input layer of the BERT? I think it is the vocab size. However I am not sure and I appreciate your help.

It is not very clear what you are referring to with "number of input neurons".
The input layer in BERT is an embedding layer, which is a table of vectors. Each of those vectors has dimensionality 768, and each vector is associated to one of the tokens in the vocabulary (so the number of vectors in the embedding table is the vocabulary size).


Different available packages in TensorFlow virtualenv? I have installed TensorFlow on Linux (Anaconda) by following the documentation which states that one should create and activate a virtual environment tensorflow. So far, so good (albeit it is not entirely clear why is this virtual environment is necessary when I want to incorporate TF to my existing env).

But when I activate the tensorflow environment I observe that several packages are unavailable in the new environment while available outside the environment at the same time:

$source activate tensorflow
$python

>>>import h5py

... No module named h5py

$source deactivate tensorflow
$python

>>>import h5py
>>>

No problem in this case.

I guessed that I should install the missing package in the tensorflow environment as well, but when I try to, I get informed that the package in question has already been installed and nothing happens:

$source activate tensorflow
$pip install h5py

Requirement already statisfied ....

The same inconsistency occurs with several other packages. What is the problem here?

I can't offer any suggestions as to why your virtual environment can't find the packages (I'm not familiar with anaconda). The only thing that comes to mind is a PATH problem.

Perhaps try installing the package with the following command:

conda install -n yourenvname [package]

or, in your case:

conda install -n tensorflow h5py

  To install additional packages only to your virtual environment, enter
  the following command where yourenvname is the name of your
  environemnt, and [package] is the name of the package you wish to
  install. Failure to specify “-n yourenvname” will install the package
  to the root Python installation.

source: https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/


Why does frequency encoding work? Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't understand why it works.

Does it work in very specific cases where frequencies are correlated with the target or is it more general? What's the rationale behind it?

Check this post.

  In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data.

Check also this thread.

  What's the rationale behind it?

High cardinality may result in dimensionality curse and actually decrease the quality of your model.


Drastic drop in Somers' D ? Why? I came across to find the correlation between the ratings assigned by two coaches to a same group of 40 players.

I have tabulated the results as below:

The Somers' D is 50%.

However, for the case below,

The Somers' D is 94.7%.

My question is, why both scenarios are having 2 deviations but the first scenario has so much lower Somers' D compared to the second scenario?

Say you have a set of pairs $(x_i,y_i)$, Somers D is linked to Kendall's tau, which is defined by :
$$\tau=\frac{N_C-N_D}{n(n-1)/2}$$
Where $N_C$ is the number of concordant pairs (ranks of both elements agree, $x_i>x_j$ and $y_i>y_j$ or $x_i<x_j$ and $y_i<y_j$) and $N_D$ the umber of pairs that are not. First it is possible that there might be a problem with the special case where $x_i=x_j$ or $y_i=y_j$, of which you have a lot of examples. This case might be handled differently depending on your implementation. Overall it quantifies the number of difference between the number of concordant and discordant pairs. By construction Kendall's tau is symetric in X and Y. (And this is probably what you wanted to use.)
Somers D, treat the variables differently. Somers D of $Y$ with respect to $X$ is defined as : $$D(Y,X)=\tau(X,Y)/\tau(X,X)$$
That is it quantifies the difference between the number of concordant and discordant pairs, divided by the number of pairs with $X$ values in the pair being unequal. You can see that it is not symmetric. It explain that there is a difference between your cases.
As to why there is such a difference : the main explaination would be a bout what is your "reference" in one case, if you set B as a reference, it is about 2 weak people (among 4) that A believe are strong, if you set A as the reference, the disagreement seems about 2 (among 38) strong people that B believe are weak.(2/4 is 50%, 2/38 is near 95%, not sure where the discrepency come from exactly)


Predicting service date If I had an automated system that pays my bills, but the website where I pay them won't tell me when the next bill will be available. What is a good approach of predicting the date of the next bill? I don't see a regression model here as a good approach. I believe it's better to get a simple statistical mean or median between bill dates in days, and then guess the next bill date as: nextBill = lastBillDate + mean. Is that the right approach?

I don't know your data, so I can't say if your approach is good or not. But your approach does not cover weekdays. So I would apply time series analysis methods predict the service date.


Why would Pytorch be non-deterministic for batch only? I am training a LSTM network on CPU and can achieve deterministic results when not using a dataloader. But when I used Pytorchs dataloader I achieve non-deterministic training error results, despite the actual batches being loaded from the dataloader being deterministic.

I have set pretty much every seed I could think of here

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled   = False

The two pieces of code are:

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    for batch_idx, (X_train, y_train) in enumerate(train_loader):

        # Zero out gradients
        optimiser.zero_grad()

        # Forward pass
        y_pred = model_test(X_train)

        # Loss Function
        loss = loss_fn(y_pred, y_train)

        # Backward pass
        loss.backward()

        # Update parameters
        optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

and

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    # Zero out gradients
    optimiser.zero_grad()

    # Forward pass
    y_pred = model_test(X_train)

    # Loss Function
    loss = loss_fn(y_pred, y_train)

    # Backward pass
    loss.backward()

    # Update parameters
    optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

I have seen some posts on Github talking about there being issues with determinism on GPU, but this is just on CPU.

The cudnn implementaton of the LSTM has determinism issues that appear to be fixed in the 7.6.1 release. Check your cudnn version.

https://github.com/pytorch/pytorch/issues/18110


Changing multiple models into 1 model I am working for a recruitment company on developing machine learning algorithms to automatically classify job applicants as either to be interviewed or not be interveiwed. The data is highly imbalanced (i.e most applicants will not be interviewed.) Currently, to do this we use a machine learning model per job. We feed it examples of applicants with their status. However, this has generally low performance due to the small size of the training dataset.

A potential improvement on this approach could be creating a single model which takes in as input information on the applicant and information on requirements of the job and then predicting whether or not to interview. Is this approach likely to work and a technique that is used/has been researched in any context?

Note: the actual problem I am working on is to do with classifying publications for inclusion in medical systematic reviews. The 'different jobs' represent different review topics, the 'job requirements' are the criteria for inclusion and each 'applicant' is actually a medical publication. I used this example as it is comparable and requires less domain expertise to understand.

This questions is best posed in its original domain, ie medicine, because it will require significant domain knowledge about the problem structure and the nature of the data to reason about how well a single model might do. I doubt the analogy to job applications translates perfectly here.

Right now you have one model per review topic which is able to generalize to different publications with different criteria for inclusion. You are asking whether a single model will also be able to generalize to different review topics.

Of course you could build the model and evaluate its accuracy. If that will require significant effort, then it is fair try and reason about how well you might do first, but that will only get you so far.

Again you will have to apply your own domain knowledge here, but I can at least try and give you a framework for thinking about this problem. In order to answer the main question of how well one model will generalize, you need to investigate:

Do different review topics interact with publications / inclusion criteria in similar ways?
Do any of my existing features capture the interaction between review topics and publications / inclusion criteria?
Are there any additional features I can design to capture these interactions in a general way?

To give a very simple example, if having the topic keyword x number of times in the paper was sufficient for inclusion for paper X, topic Y, publication Z with review criteria A,B,C,..., and you think this data helps you predict inclusion for a completely different paper, topic, etc. example, then you might reason that one model will generalize here.


Architecture help for multivariate input and output LSTM models I am working on a sequence prediction problem using Keras.  My end-goal is to have my input be several features, one being a categorical variable, that I have used 1 hot encoding for, and another which is a numeric variable, for which I would like to predict the next time step(s) for BOTH variables (any other variables just as inputs).

My first attempt was to eliminate the numeric and just use the sequence of "words" (not really words, but essentially the same concept here) to learn model that could predict the rest of the sequence, given partially complete "sentences".  I have built a working model that attempts this, with decent results.

Model thus far:

def LSTMModel(timestep_len,hidden_size,vocab_size):
    model = Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(timestep_len,num_of_features)))
    model.add(LSTM(hidden_size, return_sequences=True))
    model.add(TimeDistributed(Dense(vocab_size)))
    model.add(Activation('softmax'))
    return model

input size to masking: (None, 27, 1)

output size from softmax: (None, 27, 6997)

I pad the sequences to all be of the same length, which is why there is a Masking layer.

I an stuck on how I can incorporate a numeric variable in with this.  How do I need to adjust my architecture to accommodate for this?  I realize this current one will not work for what I'm trying to do.  This was just a simple first pass approach.  Do I need a custom loss function for each variable and then add them together some how?  My input is size (batch_size, max_seq_len, 1), where the 3rd dimension is a unique integer encoding for the "word".  My input Y is of size (batch_size,max_seq_len,vocab_size) where vocab _size is the one-hot vector representation of the t+1 timestep. How do I extend this properly to add in another variable?  How can I use a Softmax on part of the output of a dense layer?  Is that even the path I should be thinking?

I am having not much luck finding good resources for this type of sequence prediction.  Everything seems to be more simple and restricted.

Thanks for any help!

Basically every answer I was looking for was exampled and explained in this tutorial. Absolutely great resource for trying to understand how to model multi-output networks. This one goes through a lengthy walkthrough of a multi-output CNN architecture using Keras functional API. It only took me about three weeks to stumble upon, however.

https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/


What is layout of weights I'm working on neural networks. I have one very important question. For instance, there are two layers with 2 neurons in each. They are all fully connected. Which neurons do the weights belong to? Do they belong to neurons in next layer or the layer preceding it? Moreover, when weights are saved in Keras, are they saved for each neuron completely or they are saved as linknumber(first weight of every neuron)? I hope I'm able to clarify my question

The neurons themselves only hold their activation values (or input values when we consider the input layer of neurons). The weights you mention actually indicate the strength of the connections between neurons in subsequent layers (and possibly the bias per layer). Using your terminology the weights would belong to the neurons in both the current and next layer.

For every neuron the strength of the connection to the other neurons is saved. In other words, all weights (and possibly biases) are saved.

For more information about the basics of neural networks, and more specifically multi-layer perceptrons, I can remmond this website.


Data mining: Clique based clustering to make comparison in social network analysis I am a very beginner in data mining. I want to work on Clique based clustering method. I want to make a comparison between various datasets for social network analysis or community detection of social network analysis. Now I need more than 3 datasets and source code (Python code) to make the comparison in terms of social network analysis. The data set can be older or new that would not be any problem. But I want to work on at least 3 datasets from different domains and sizes to make a better comparison.

Please may help me to give enough instruction.

Thanks a lot.

There are over 300 social networks captured in the Index of Complex Networks.  Thats usually the best place to start.
While your quantitative method is not very clear, it sounds like you could start with networkx and use something like find_cliques.  There is also a community detection method built for all types of subgraphs, not just cliques Benson et al. (2018).


How to migrate application which has backend running using DLL methods and front end in JAVA to Apache spark? I have one Java Client which Reads 4 to 5 GBs of data and uses DLL file in order to process it and give O/P back to the JAVA client.
Can anyone tell me if this can be implemented using Apache Spark or not ?
And if yes can you back you Answer by some examples or documentation Links ?

EDIT

I have knowledge about JNI and how to do this in windows environement.
What I'm actually looking for is How do you implement such thing in Apache Spark.

As all methods and processing is done via DLL files and JAVA Client is just having represntational logic so any documentation is helpful.

I'm currently using windows development env. but gonna use Ubuntu 14.04 for Apache Spark implemetation.

EDIT after Hold

This question is really intending to ask where to find such documentation or help with which we can migrate such application smoothly.

You will need to use the Java Native Interface (JNI) for this. Refer to a Stack Overflow question for how to call DLL methods via Java over JNI.


How to interpret Low Pearson correlation coefficient between stable signals and high Pearson correlation coefficient between unstable signals? I calculated the Pearson correlation coefficient between two signals, that described the state of the unit. During normal operation of the unit, both signals were fairly stable and fluctuated very little.
At some point in time, the unit began to form a defect, in connection with which the oscillations of the signals increased, and also a trend of growth in their absolute values began to be observed.

Signals describing the normal operation of the unit.
Signals describing emergency operation of the unit.
Signals that describe both normal and emergency operation of the unit (combination 1) and 2)).

In each of these three cases, I calculated the correlation coefficients between the signals.
In the first case, it was minimal at 0.5, in the second case, at 0.9, and in the third case, at a level close to 1.
I have the following questions:

Why is there such a low correlation between the signals during normal operation of the unit, when the signals behaved "by eye" in the same way? In fact, both signals at this time had minimal variation and behaved in the same way.

High correlation during the occurrence of a unit failure. Both signals showed an upward trend, both again behaved the same "by eye". But why did the correlation become high in this case? During normal operation, both signals also behaved in the same way, stably kept at the same level.

Maximum correlation when I combined the normal and emergency signals of the unit. Why, by combining data samples with low (normal operation of the unit) and high (emergency operation of the unit) correlation, I get the samples, the correlation between which turned out to be the highest of all three cases? What could have caused this effect?

"How can we determine the strength of association based on the Pearson correlation ?
Briefly speaking, Pearson  (linear) correlation (r ) is not to be interpreted the way you interpret . It simply tells the linear (constant) relationship  between two variables. It does not tell anything  about the causal relation between two variables. It is association measure based on Covariance. Another measure- is called Pearson (sample) correlation coefficient measure which is based on distance. This measure reflects a cause and effect relationship.