Data Type required in Weka I want to run several Association rule mining techniques such as Apriori, Eclat and FP Growth. I want to know the format of the data to run these algorithms as they are disabled (marked in grey)to me.

Also I don't see the algorithm Eclat in the 'Association' tab of Weka. Please recommend me a suitable tool/approach to perform this.

I want to run several Association rule mining techniques such as
  Apriori, Eclat and FP Growth. [...] Please recommend me a suitable
  tool/approach to perform this

Christian Borgelt's website offers several tools for frequent pattern mining:

FPgrowth,
Eclat,
Apriori, and some others.

Check out the full descriptions to see how to use each one.

Here are some requirements for Weka's FPgrowth algorithm.


Are there any good out-of-the-box language models for python? I'm prototyping an application and I need a language model to compute perplexity on some generated sentences.

Is there any trained language model in python I can readily use? Something simple like

model = LanguageModel('en')
p1 = model.perplexity('This is a well constructed sentence')
p2 = model.perplexity('Bunny lamp robert junior pancake')
assert p1 < p2

I've looked at some frameworks but couldn't find what I want. I know I can use something like:

from nltk.model.ngram import NgramModel
lm = NgramModel(3, brown.words(categories='news'))

This uses a good turing probability distribution on Brown Corpus, but I was looking for some well-crafted model on some big dataset, like the 1b words dataset. Something that I can actually trust the results for a general domain (not only news)

I also think that the first answer is incorrect for the reasons that @noob333 explained.
But also Bert cannot be used out of the box as a language model. Bert gives you the p(word|context(both left and right) ) and what you want is to compute p(word|previous tokens(only left contex)). The author explains here why you cannot use it as a lm.
However you can adapt Bert and use it as a language model, as explained here.
But you can use the open ai gpt or gpt-2 pre-tained models from the same repo
Here is how you can compute the perplexity using the gpt model.
import math
from pytorch_pretrained_bert import OpenAIGPTTokenizer, OpenAIGPTModel, OpenAIGPTLMHeadModel
# Load pre-trained model (weights)
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.eval()
# Load pre-trained model tokenizer (vocabulary)
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)

a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
21.31652459381952, 61.45907380241148, 26.24923942649312


location of the resampled data from SMOTE I am using SMOTE in Python to perform oversampling of the minor class in an unbalanced dataset. I would like to know the way SMOTE formats its output, that is, whether SMOTE concatenates the newly generated samples to the end of the input data and returns that as the output or whether the new synthetic data points are positioned randomly among the input data points. I'd appreciate your help.

SMOTE is an algorithm used to generate " synthesize" new samples from the real samples.
It selects randomly one of the k-nearest neighbors, find the distance between these two pints , synthesize new point by modifying the sample considering the distance and a random number between 0 and 1.
SMOTE algorithm does not use samples from majority class only samples from minority.
It synthesizes new samples
It expects a high density minor class with small variation within the class
Anything else is related to the implementation, many implementation return only the synthesized samples


NameError 'np' is not defined after importing np_utils I am running a MNIST example in a Jupyter notebook running in an Anaconda virtual environment. I have tried to run the code below (not yet finished, I was testing it) when it comes up with an error (can be seen below the code).
(X_train, y_train), (X_test, y_test) = mnist.load_data()

#print("X_train shape", X_train.shape)
#print ("y_train shape", y_train.shape)
#print("X_test shape", X_test.shape)
#print ("y_test shape", y_test.shape)

from keras.models import Sequential #imports the sequential model
from keras.layers import Dense, Dropout, Conv2D, MaxPool2D
from keras.utils import np_utils

#flattens the images from the 28x28 pixels to 1D 787 pixels
X_train = X_train.reshape (60000, 784) #flattening the image into 28x28 pixels, so into an array of 784.
X_test = X_test.reshape (10000, 784)
X_train = X_train.astype('float32') #using a 32-bit precision when training the neural network. This is most commonly used.
X_test = X_test.astype('float32')

X_train /=255 #Used for normalisation. Each image has a 'degree' of darkness within the range of 0-255 so you need to reduce that range to 0-1 for your Neural Network
X_test /=255

#one-hot encoding using keras' numpy-related utilities
n_classes=10
print ("Shape before one-hot encoding: ", y_train.shape)
Y_train = np.utils.to_categorical(y_train, n_classes)
Y_test = np.utils.to_categorical(y_test, n_classes)
print ("Shape after one-hot encoding:", Y_train.shape)

Error:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-0063f366d5bd> in <module>
     22 n_classes=10
     23 print ("Shape before one-hot encoding: ", y_train.shape)
---> 24 Y_train = np.utils.to_categorical(y_train, n_classes)
     25 Y_test = np.utils.to_categorical(y_test, n_classes)
     26 print ("Shape after one-hot encoding:", Y_train.shape)

NameError: name 'np' is not defined

I guess I need to define np somewhere, but the practise code on the website I am using doesn't actually define np anywhere. Any suggestions?
Many thanks

Y_train = np_utils.to_categorical(y_train, n_classes)
Y_test = np_utils.to_categorical(y_test, n_classes)

please update code line as np_utils not np.utils


Proper/Possible methods for extracting unstructured data from websites I'm working in Python, using Scrapy, and NLTK to try to understand how I can extract data from college websites.

My scraper can navigate through the university websites and find their tuition fees pages perfectly , but when trying to extract specific fees like :

Resident
Non Resident
Per Credit Hour
Per Semester

I'm running into trouble due to the data being so unstructured from site to site.

I've tried using NLTK to parse data based on parts of speech tags and regex chunking to try to extract sentences such as "tuition cost for resident: $12,500" but colleges can display this data in a number of ways.

Here is my question:

Are there any better ideas/methodologies that I should be looking into that can help me with extracting this type of data?

You need to build a couple of classifiers. First, you need a classifier to give you thumbs up that you want to parse the page at all. Call this the "is_relevant" model. Once you've determined that a page is relevant, you should pass it through a separate classifier for each data element you're hoping to capture (or a multiclass classifier capable of recognizing each of those elements and distinguishing them from content you're not interested in).


Can a GAN-like architecture be used for maximizing the value of a regression predictor? I can't seem to convince myself why a GAN model similar to regGAN  couldn't be modified to maximize a regression predictor (see the image below).  By changing the loss function to the difference between the current predicted value and the maximum predicted value generated so far, wouldn't gradient decent converge such that the generator builds the inputs that will maximize the prediction in the Discriminator CNN?

In math terms, the loss calculation would look like:

  yhat = current prediction
  ymax = best prediction achieved yet
  Loss = ymax - yhat
  if Loss < 0 then Loss = 0; ymax = yhat
  Back-propagate the loss using SGD

If the current predicted value is higher than the maximum predicted so far, then the loss is 0 and the loss function is updated.  Essentially, we are changing the objective from generating inputs that look real to generating inputs that optimize the complex function encoded in the CNN.

Note that GANs are generally considered either as a way to perform implicit density estimation or defined by the game-theoretic minimax problem they usually imply. The situation you're describing doesn't really seem to fit in either of those, especially the latter, because your generator $G$ and $D$ are not really working against each other; rather, they are working together to maximize the function you are optimizing. In other words, the situation you're describing seems to simply be doing generation that maximizes some objective (in a somewhat complicated manner it appears), rather than realism. Since the objective is readily directly computable (as far as I can tell), we can just do standard optimization maximizing it. The reason why the alternating GAN optimization is needed is because the specific objective of realism is not directly computable nor optimizable without first training the critic.

Nevertheless, combining GANs and regression is quite sensible, so maybe I'm misunderstanding your question and notation.
Let $X$ be the data space, and let $f:X\rightarrow\mathbb{R}$ be the true value of the function you are trying to maximize (after regressing). Let $R:X\rightarrow\mathbb{R}$ denote a learned regression function approximating $f$. Denote $G:\mathcal{U}\rightarrow X$ as the generator and $D:X\rightarrow[0,1]$ as the critic (discriminator), aiming for realism.
Let me consider two possibly relevant scenarios, assuming a dataset $(x_i,f_i)$:

Simply directly optimize a GAN with an additional loss term maximizing $f$:
$$ \mathcal{L}(G) = (1-\eta)\mathbb{E}_{x\sim G}[\log D(x)] + \eta \mathbb{E}_{x\sim G}[R(x)] $$
for the generator. We can optimize $D$ and $R$ in the standard way together. In other words, we are telling $G$ to generate $x\in X$ that balance realism and high $f$, with the balance controlled by $\eta\in[0,1]$. When $\eta=0$, it is a regular GAN (not caring about $f$); when $\eta=1$, the critic is not used at all and there is no regard for realism - $G$ will learn to generate random samples that maximize $f$ without caring about how realistic they are (i.e., without matching the true distribution $p(x)$).
Separately, train the regressor $R$ and the GAN generator $G$. Then we can generate a high $f$ data point by gradient descent in the GAN latent space, i.e. doing:
$$ u_t \leftarrow u_{t-1} +\xi \nabla_u R(G(u_{t-1})) $$
so that our output $x=G(u_T)$ has high $f(x)$. Since $u$ is in the GAN latent space, we expect $x$ to be realistic as well.

In both of these simple scenarios, we are using the GAN to maintain realism, while maximizing $f$. Notice that if we don't care about realism then the GAN is not needed at all. Simply generating samples does not make a model a GAN: there are many generative models, such as Deep Boltzmann machines, VAEs, etc... that can do this. What sets GANs apart is adversarial learning, where two networks work against each other, and it's not clear that's happening in the case you described.

I think this question is somewhat relevant to yours as well.


What can I infer from similar accuracy using two different DNN architectures? I am working on a 'two-class classification of multivariate time-series' problem. I used two different approaches:

1) Manual time-series feature engineering (such as slope, intercept, variance, etc. using tsfresh python library) followed by a multi-layer perceptron.

2) Conv1D followed by LSTM followed by Dense layers (keras/Python) without any feature engineering.

They both yield very similar results (~40% precision, ~95% recall).

Can I infer anything from the similarity of results from two different architectures?

I believe the only way to make a science-backed inference, would be if you fine-tuned both of your proposed solutions, to ensure that both are performing at their maximum potential. Only then you can say that method A is better than method B, otherwise it might just be a matter of improper selection of parameters that leads to those results.

However, if both methods still yield similar results after you have fine-tuned them, then you can follow Occam's razor and select the simplest one as the best one. By simplest, you can consider the one with the least amount of parameters and least time needed for training/inference.


Can colors be detected using Neural Nets? How do I represent a color as an activation value within a neuron? ( might be off-topic)

I want to detect colors Using Neural Nets

In my Knowledge any activation function which is generally used will push the values in between 0-1 which isn't of any use to me...

Just wondering how Windows 10 decides on the theme color change when we switch themes.. (led me to think about this or its just simple averaging on pixel values of the channels?)

So seems like an impossible task for Neural Nets then?

Normally color spaces are not considered to be one dimensional.

Given three types of human cone cells, the most natural approach is probably not to use a single but three neurons. If color input is encoded as RGB values, then one neuron would be for red, one for green, and one for the blue channel.


How to create model for selecting a set of categories with a set of attributes? I have a couple of hundred categories where each of these categories has a specific set of attributes having different values (historical).

The problem I need to solve is to select the best set of categories out of a smaller group which meet some constraints.

I'm new to datascience and was wondering how do I go about solving this problem?

One option I thought was to use multiple regression for the different attributes to assign a weight to each category and then use these to generate a random forest on the historical groups of categories to train and test them.

Does this make sense?

If I understand your question correctly, the problem you are trying to solve is a "multiclass classification" problem, so I'd suggest doing some research on that. Possible solutions range from Support Vector Classification (simple) to deep neural networks (harder). In most machine learning problems, it's best to start with a simple approach. This means that:

You learn about simple patterns in your data
You get a benchmark
If the result from your simple is good enough, you can stop there :)

I recommend checking out scipy's docs on classification models. Also if you're a beginner you might find this cheatsheet handy for similar questions in future.


Why are results without Transfer Learning better than with Transfer Learning? I developed a neural network for license plate recognition and used the EfficientNet architecture (https://keras.io/api/applications/efficientnet/#efficientnetb0-function) with and without pretrained weights on ImageNet and with and without data augmentation. I only had 10.000 training images and 3.000 validation images. That was the reason I applied Transfer learning and image augmentation (AdditiveGaussianNoise).
I created this model:
efnB0_model = efn.EfficientNetB0(include_top=False, weights="imagenet", input_shape=(224, 224, 3))
efnB0_model.trainable = False

def create_model(input_shape = (224, 224, 3)):
    input_img = Input(shape=input_shape)
    model = efnB0_model (input_img)
    model = GlobalAveragePooling2D(name='avg_pool')(model)
    model = Dropout(0.2)(model)
    backbone = model

    branches = []
    for i in range(7):
            branches.append(backbone)
            branches[i] = Dense(360, name="branch_"+str(i)+"_Dense_360")(branches[i])
            branches[i] = BatchNormalization()(branches[i])
            branches[i] = Activation("relu") (branches[i])
            branches[i] = Dropout(0.2)(branches[i])
            branches[i] = Dense(35, activation = "softmax", name="branch_"+str(i)+"_output")(branches[i])

    output = Concatenate(axis=1)(branches)
    output = Reshape((7, 35))(output)
    model = Model(input_img, output)

    return model

I compiled the model:
opt = keras.optimizers.Adam(learning_rate=0.0001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=["accuracy"])

And used this code to fit it:
hist = model.fit(
    x=training_generator, epochs=10, verbose=1, callbacks=None,
    validation_data=validation_generator, steps_per_epoch=num_train_samples // 16,
    validation_steps=num_val_samples // 16,
    max_queue_size=10, workers=6, use_multiprocessing=True)

My hypotheses were:
H1: The EfficientNet architecture is applicable to license plate recognition.
H2: Transfer learning will improve accuracy in license plate recognition (compared to the situation without Transfer Learning).
H3: Image augmentation will improve accuracy in license plate recognition (compared to the situation without it).
H4: Transfer Learning combined with Image augmentation will bring the best results.
I now got this results:

So, H1 seems to be correct. But H2, H3 and H4 seem to wrong.
I was thinking about it and got an explanation for H3 and H4, which seem to be logical for me. That is, that image augmentation is too heavy and deteriorates the quality of images by a degree which makes it very hard for the network to recognize the characters.
1. Is this a suitable explanation and are there other ones additionally?
It seems to be the case, that image augmentation was too strong. So, first question is solved.
Regarding H2 I am little confued to be honest. The network seems to overfit but stagnates completely regarding validation accuracy. So, the conclusion that the Imagenet weights are not applicable seems not logical to me because the network learnt something for the training data. I also excluded the possibility that the data volume is to small since we had that good recognition rates without using Transfer learning or image augmentation...
2. Is there any logical explanation for this?

As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a "very different" dataset. Typically if the pre-trained network extract information that is not relevant for your problem.
Moreover, in the paper License Plate Recognition System Based on Transfer Learning (that you shared with me), they have tried to freeze some layers of a pretrained Xception (based on ImageNet weights) to see the impact on the training. They conclude that ImageNet data and license plate data are too different to freeze layers. So your results are confirmed.
Now changing efnB0_model.trainable = False to True would allow the pre-trained network to update and to be more relevant to your problem. Generally, if you don't have time issues, it seems to be always better (see this post). Will it give better results than initialize the weights randomly ? I think one can presume but cannot know.


Supervised learning with tagged images I am new to ML and looking to learn with some project. I have a medical imaging dataset where an image (image is a time series of an object so multiple images) has been looked at by radiologist and they have graded it on a scale of 1-5 for some pathology.

Now, I would like to basically use this to predict the pathology on new images. I am guessing there are multiple approaches one could take to do so. Could someone point me to some methods I could try (simple to more advanced) as I would like to also learn about them.

Another issue is that different images are of different sizes. Is this usually a problem for these approaches? I could try something where I can register them so that they are of the same size.

I have worked on similar projects (using medical images such as PET to predict outcomes).  A method being used more and more for predicting cancer treatment outcomes is texture analysis: https://www.ncbi.nlm.nih.gov/pubmed/21321270

Another method of texture analysis uses wavelet transforms: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3505569/

After deriving the texture features, you could then use those in a ML model for making predictions.

A benefit to texture analysis is that it is mostly independent of the size of the target.  It is however, more dependent on resolution. If your images are produced on different scanners, you may need to resample in order to normalize the images.

There are fewer studies about deep learning for medical imaging, but it definitely has some exciting potential.  The question is whether or not a CNN can pick up subtle differences.  From what I've seen (I'll look for a reference), CNN hasn't outperformed any other method yet.

Maybe add a few more details so we can get a better idea of what your objective is?


Recurrent neural network (LSTM) dimensions error I have data in a dataframe named ddf as follows:

labels          X
L1          [1,2,3,7,8,9...]
L1          [4,2,6,9,8,7...]
...
L2          [5,6,8,9,6,3...]
L2          [7,8,5,6,9,0...]
...

There are 250 rows, 7 labels and 2000 elements in every list under X. These 2000 elements are values of a signal over a period of about 60 seconds.

I am trying to build a recurrent neural network for above data. Following is my code:

Xall = ddf['X'].values
Xall = np.array(Xall)

ydf = pd.get_dummies(ddf.drop('X', axis=1))
Yall = np.array(ydf.values)

# Split the data
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(Xall, Yall,  test_size=0.1, random_state=0)

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
model_lstm = Sequential()
model_lstm.add(Embedding(2000, 128))
model_lstm.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(Dense(Yall.shape[1], activation='softmax'))
model_lstm.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model_lstm.fit(X_train, Y_train, epochs=50, verbose=True, validation_data=(X_test, Y_test))

However, I am getting error at second LSTM layer:

ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

I think this has something to do with LSTM arguments. Also are arguments of Embedding layer OK? How are both these adjusted? Where is the error coming from and how can it be solved? Thanks for your help.

EDIT after comments

  Also are arguments of Embedding layer OK?

Yes. But You need to pass return_sequences = True in the first LSTM layer so that it will pass sequences to the next LSTM layer.

From the docs

return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

  How are both these adjusted?

From the docs.

input_dim:
Size of the vocabulary, i.e. maximum integer index + 1. This determines the largest integer in the input data. The largest integer in the input should be no larger than the vocabulary size. This should be the

output_dim:  Dimension of the dense embedding. int >= 0

input_length: Length of input sequences, when it is constant. This argument is required if you are going to connect Flatten then Dense layers upstream (without it, the shape of the dense outputs cannot be computed).

I am posting code below for dummy data. The size of vocabulary has been taken 100.

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
input_array = np.random.randint(100, size=(250, 2000))
input_y = np.random.randint(7, size = (250))

Y_dumy = pd.get_dummies(input_y)
X_train, X_test, Y_train, Y_test = train_test_split(input_array, Y_dumy,
test_size=0.1, random_state=0)

model = Sequential()
model.add(Embedding(input_dim = 100, output_dim = 64, input_length=2000))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences
=True))
model.output_shape

#Output shape should be:
#model.output_shape = (None, 2000, 64)
#3D tensor with shape: (batch_size, sequence_length, output_dim)

model.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(Y_dumy.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=
['accuracy'])
model.fit(X_train, Y_train, epochs=50, verbose=True, validation_data=
(X_test, Y_test))`

  Where is the error coming from and how can it be solved?

I believe the error is coming because of absence of input_length. For similar errors please have a look at this post.

After the comments
The error is coming because of return_sequences =False in the first LSTM layer.


How to use a dataset where attribute names are changed? I am trying to use UCI credit approval dataset to build a credit approval system of a bank.(Undergraduate project). But dataset description says attribute names are changed.

My goal is to use dataset to decide whether new credit requesters request is approved or not, base on the data provided by them.

I see this dataset is widely use, but I do not know use It without knowing the attribute names?(I am aware that some attribute values can be changed to protect confidentiality)

What piece of knowledge am I missing here to use this dataset?

What piece of knowledge am I missing here to use this dataset?

By anonymising the data and altering the attribute names, the providers of this dataset have made it into an abstract machine learning exercise, where data has similar qualities to real-world credit approval data, so that approaches to train models that work well on it are likely to work well in a real-world scenario. However, those models could not be used in a production system. You are not in a position to train a model based on it and then input real world data you have collected elsewhere to make a prediction.

You can still train a model on this data to predict A16 (the class attribute). You can measure the accuracy - or any other metric - by holding out a test set. You can experiment with feature engineering, feature selection etc, in an abstract way without being able to apply much domain knowledge. You can try out different model classes, different hyper-parameters, different approaches to imputing missing values or cross-validation etc.

Your project can draw conclusions about the approach you have taken, based on test results. What you cannot do is explore real-world scenarios with simulated customer data. This might make it less compelling as a demonstration - for instance users could not explore exactly what factors might lead to an application being approved or not, instead you would just have to show some graphs that demonstrate you have selected a model with good performance.


Every one knows data-driven modeling, but what is model-driven (or non data-driven) modeling? There are hundreds of data-driven machine learning models. It is easy to name a few: neural networks, linear regression, SVM, etc etc... but what is model-driven (or non data-driven) modelling and what are famous and useable examples for e.g. regression tasks?

If the model is not derived from the data then it must be built manually, so non data driven means rule based.
This was the big trend in AI in the 80s before Machine Learning, these pre-ML automatic prediction systems were called expert systems and they were quite successful at the time in industry (here are some examples of applications).
The way one would build a system for a regression task is essentially this: do the whole regression analysis manually, find the parameters and hard-code them in the prediction system.
As far as I know Machine Learning has pretty much made this kind of rule-based systems obsolete, due to their total lack of flexibility and the very high cost in manual labor to build one.


What are some of the best practices for sharing data and models with colleagues? As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central server or location where all of us can access (which means unix permissions etc). For models I also tend to send a weights file over to my colleague and share my github. Both I've found pretty cumbersome. What have some of you done?

You can try using dvc, which stands for data version control.
https://dvc.org/


Is it acceptable for a forecasting model to predict moving averaged version of the data? I'm working on a project in which I'm developing a precipitation forecasting model.

When I try to predict the original data, the model (LSTM neural network) is not able to predict the peaks.

This is the predicted values for the original data:

 So, I decided to perform

1) A differencing method in which I simply subtract values of the past from current values.

2) A moving average method on the resultant data from step 1.

Then I try to predict this processed version of the data.

Below is the predicted values for the smoothed data:

1) Is this kind of forecasting models acceptable among academicians?

2) should I try just to predict the original data?

3) What probable usage can these types of predictive models (which forecasts smoothed version of the data) have?

4) Any suggestion on how to deal with this situation?

In estimation theory, the simplest way to estimate a random variable is to predict the average of its realizations. Very easy, very simple and very naive to be useful.

Let me explain more: in your problem which is the weather, the correlation is very high between the data from the last several days and the next day, which makes your estimation a good one but in many problems, the corelation between the data from previous days and the next day is small ( the stock market). Another thing you need to keep in your mind which is the cost of doing a mistake, meaning, predicting the temperature to be the average of the last few days (75), if the actual temperature is 78, then the difference is 3 may be acceptable, in stocks, if the average from the last few days is 10 and the actual value is 10.5. This difference 0.5 may not be acceptable.

What does mathematically this mean?
If the acceptable error in you estimation is << than the variation within the data having an average based estimator is not acceptable.

Regarding the issue of transforming the data from one domain to another, it is okay if you a deterministic mapping function that can take you back to the original domain. We prefer to have one to one mapping


In case this can help anyone else. Here is a solution that is more computationally efficient.

TL;DR version

If each row already has a unique index, then do this:

>>> df.loc[df.groupby('A')['C'].idxmin()]

If you've already indexed by 'A', then convert 'A' back into a column first.

>>> df2 = df.reset_index()
>>> df2.loc[df2.groupby('A')['C'].idxmin()]

Step by Step explanation:

Step 1.

First, make sure each row in your dataframe is uniquely indexed. This is the default when importing csv data. e.g.

    >>> df = pd.read_csv('questionData.csv'); df
        A       B       C
    0   196512  1325    12.901051
    1   196512  114569  12.926770
    2   196512  118910  12.898335
    3   196512  100688  12.950509
    4   196795  28978   12.780517
    5   196795  34591   12.899411
    6   196795  13078   12.913575
    7   196795  24173   12.876965
    8   196341  118910  12.898335
    9   196341  100688  12.950509
    10  196641  28972   12.780517
    11  196641  34591   12.899411
    12  196346  118910  12.898335
    13  196346  100688  12.950509
    14  196646  28980   12.780517
    15  196646  34591   12.899411

Aside: If you already converted column 'A' into an index, then you can turn the index back into a column (https://stackoverflow.com/questions/20461165/how-to-convert-pandas-index-in-a-dataframe-to-a-column) by doing: df.reset_index()

Step 2.

Use the pandas.DataFrame.idxmin function to retrieve the indices of the minimum of each group.

The semantics of the example below is this: "group by 'A', then just look at the 'C' column of each group, and finally return the index corresponding to the minimum 'C' in each group.

>>> indices = df.groupby('A')['C'].idxmin; indices
A
196341     8
196346    12
196512     2
196641    10
196646    14
196795     4
Name: C, dtype: int64

Step 3.

Finally, use the retrieved indices in the original dataframe using pandas.DataFrame.loc to get the rows of the original dataframe correponding to the minimum values of 'C' in each group that was grouped by 'A'.

>>> df.loc[indices]
    A       B       C
8   196341  118910  12.898335
12  196346  118910  12.898335
2   196512  118910  12.898335
10  196641  28972   12.780517
14  196646  28980   12.780517
4   196795  28978   12.780517

Note: The groupby('A') operation returns groups sorted by A. Thus 'indices' is sorted by A. If we want the original order, we just have to do

>>> df.loc[indices].sort_index()
    A       B       C
2   196512  118910  12.898335
4   196795  28978   12.780517
8   196341  118910  12.898335
10  196641  28972   12.780517
12  196346  118910  12.898335
14  196646  28980   12.780517


The notation of $splits(label)$ under Random Forest On the "Fair Forests: Regularized Tree Induction to Minimize Model Bias", it is written that

We propose a simple regularization approach to constructing a fair decision tree induction algorithm. This is done by altering the way we measure the information gain $G(T, a)$, where $T$ is a set of training examples, and $a$ is the attribute to split on. We will denote the set of points in each of the $k$ branchs of the tree as $T_{i \ldots k}$. This normally is combined with an impurity measure $I(T)$, to give us
$$
G(T, a)=I(T)-\sum_{\forall T_{i} \in \text { splits }(a)} \frac{\left|T_{i}\right|}{|T|} \cdot I\left(T_{i}\right)
$$
The information gain scores the quality of a splitting attribute $a$ by how much it reduces impurity compared to the current impurity. The larger the gain, the more pure the class labels have become, and thus, should improve classification performance. In the CART algorithm, the Gini impurity is normally used for categorical targets.
$$
I_{\mathrm{Gini}}(T)=1-\sum_{\forall T_{i} \in \text { splits }(\text { label })}\left(\frac{\left|T_{i}\right|}{|T|}\right)^{2}
$$

Especially, those two things make me confused very much.

The definiton of $\mathrm{splits(a)}$ is not written in the paper.
It is written that $T_i$ is the branch, however, for me, it seems a node.

I have already learned about Random Forest on the ESL, so I know the normal definition of impurity as:
$$
I_G(p) = 1 - \sum_{i=1}^{k} p_i^2
$$
However, I can't figure out each definition in this paper...

You correctly mentioned the definition of impurity which is
$$I_G(P) = 1 - \sum _{i=1}^k p_i^2$$
This can be written as
$$I_G(P) = \sum _{i=1}^k p_i*(1 - p_i)$$
At any split, for any branch $T_i$, you calculate the probability using the classical definition i.e.,
$$p_i = \frac{|T_i|}{|T|}$$
Using this, you can derive the impurity definition $I(T_i)$ given in the paper.
Once you are able to calculate the impurity of a branch you can calculate the impurity of a node using the weighted sum of each branch, where weights are the probability of that branch. Hence,
$$I_{node} = \sum_{\forall T_i  \in splits(a)} \frac{|T_i|}{|T|} I(T_i)$$
Using the above impurity, you can calculate the change in impurity or the information gain as given in the paper.
I hope this solves your doubt.


Efficient database model for storing data indexed by n-grams I'm working on an application which requires creating a very large database of n-grams that exist in a large text corpus.

I need three efficient operation types: Lookup and insertion indexed by the n-gram itself, and querying for all n-grams that contain a sub-n-gram.

This sounds to me like the database should be a gigantic document tree, and document databases, e.g. Mongo, should be able to do the job well, but I've never used those at scale.

Knowing the Stack Exchange question format, I'd like to clarify that I'm not asking for suggestions on specific technologies, but rather a type of database that I should be looking for to implement something like this at scale.

See Lucene NGramTokenizer

Are you sure you can't just use lucene or similar indexing techniques?

Inverted indexes will store the n-gram only once, then just the document ids that contain the ngram; they don't store this as highly redundant raw text.

As for finding ngrams that contain your query sub-n-gram, I would build an index on the observed ngrams, e.g. using a second lucene index, or any other substring index such as a trie or suffix tree. If your data is dynamic, probably lucene is a reasonable choice, using phrase queries to find your n-grams.


Where can I get Insurance claim data for practicing NLP(Natural Language) processing? I am looking for specifically Insurance dataset for practicing Machine Learning & NLP, but unable to find much in kaggle, udemy or other websites. Is there a way to get that dataset or any website that stores this.

I am mainly looking for NLP practice for automatic claim generation or other insurance activities, please suggest the approach

I am working in the same industry for a few years now and I can tell you that there are no publicly available datasets because of the nature of the documents. They are quite private and contains sensitive information, that are bound by rules and regulations.


numer.ai: how does their leaderboard system work? There is a data mining competition website called numer.ai.

Presumably, behind the website is a hedge fund which makes use of the predictions that people send. People within the 100th top places continuously make money, until the next dataset is revealed, and the competition resets.

What I don't understand is that websites like Kaggle avoid overfitting by having a public and a private leaderboard. The private leaderboard is only revealed at the end of the competition and only then are prizes handed out.

Numerai says under rules that it uses the same approach. Quoting:

  If models overfit the public leaderboard and perform poorly on the
  private leaderboard, those users will suffer penalties potentially
  eradicating all gains. This discourages overfitting.

What do they mean by "penalties" being applied? I know someone who is making money on that website by continuously playing the public leaderboard. Does this mean that if he does not withdraw his bitcoin he is in danger of losing what he has earned in case he is overfitting?

I found the answer in the comments section of their blog:

  Earnings listed on the public-score leaderboard are potential winnings. Actual winnings are determined by the private-score leaderboard. At the time of withdrawal and when new datasets are released, your actual winnings will be revealed to you.


How to resize MNIST images to fit AlexNet model I am using the keras API to load in the MNIST dataset. My problem is I need to use AlexNet as my algorithm. Understanding the AlexNet model, I require to start with 277x277 images but the MINST dataset has 28x28. How can I reshape the numpy array so that each image is 227x277 to then use the full AlexNet model?
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

This is how I load my data in. Could someone show me the solution to change the initial images to match the AlexNet model?

You can use tf.image.resize, as follows:
(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

print(x_trian.shape) # (60000, 28, 28)

# train set / data
x_train = np.expand_dims(x_train, axis=-1)
x_train = tf.image.resize(x_train, [32,32]) # if we want to resize

print(x_train.shape) # (60000, 32, 32, 1)


Best way to predict ranges of money? I am currently working on a project in which I have to develop a model to predict how much money other companies will make by using the services provided by my company. The money made is a type of tax return. The model should predict a range (e.g 100k$-200k$).
Basically, if I give the model the data of the target company, it will give me how much they can potentially make with us.
The data I have is the financial statements of the companies we worked with. This includes their general and financial data such as capital, number of employees, city, type of industry, and of course the amount of money they made with us.
After researching online, I found that most solutions are forecasting and time series based. While the data I have has the year in which we worked with the clients, I do not think this is the right approach for me. I experimented with decision trees and a regression model but I do not seem to get a good result.
Any tips on where to look to solve this problem?

If you have many available input parameters/features (financial data like balance, depreciations, tax for several years for several companies, the business areas where those companies are working in (e.g. telco, banks, media, etc) and you already have "labelled" data (I mean, you already have the "tax" saving for you previous customers).... well, a good start could be a standard process of feature selection and designing a neural network.


Unexpected Error Message in RStudio; while using 'twitterR' Package I'd install an R package twitterR in my RStudio for accessing Twitter data for a particular handle. But whenever I do try to access the Twitter data it's showing me an unexpected message:

In doRppAPICall(“search/tweets”, n, params = params, retryOnRateLimit = retryOnRateLimit, :
100 tweets were requested but the API can only return 0

Any thoughts on that?

I tried different parameters, handles and functionalities but still getting the same issue.

Here's my code for the particular functionality--

my_tweets <- searchTwitter('@jabhij', geocode='20.593684, 78.96288, 2000km', n = 100)
my_tweets_df <- do.call("rbind", laaply(my_tweets, as.data.frame))
View(my_tweets_df)

Any thoughts on that will be very helpful.
Cheers! :)

It's because of the whitespace in the parameter for geocode.

It should be:

my_tweets <- searchTwitter('@jabhij', geocode="20.593684,78.96288,2000km")


What does a negative coefficient of determination mean for evaluating ridge regression? Judging by the negative result being displayed from my ridge.score() I am guessing that I am doing something wrong. Maybe someone could point me in the right direction?

# Create a practice data set for exploring Ridge Regression

data_2 = np.array([[1, 2, 0], [3, 4, 1], [5, 6, 0], [1, 3, 1],
           [3, 5, 1], [1, 7, 0], [1, 8, 1]], dtype=np.float64)

# Separate X and Y

x_2 = data_2[:, [0, 1]]
y_2 = data_2[:, 2]

# Train Test Split
x_2_train, x_2_test, y_2_train, y_2_test = train_test_split(x_2, y_2, random_state=0)

# Scale the training data
scaler_2 = StandardScaler()
scaler_2.fit(x_2_train)
x_2_transformed = scaler_2.transform(x_2_train)

# Ridge Regression
ridge_2 = Ridge().fit(x_2_transformed, y_2_train)
x_2_test_scaled = scaler_2.transform(x_2_test)
ridge_2.score(x_2_test_scaled, y_2_test)

Output is: -4.47

EDIT: From reading the scikit learn docs this value is the R$^2$ value. I guess the question is though, how do we interpret this?

A negative value means you're getting a terrible fit - which makes sense if you create a test set that doesn't have the same distribution as the training set.

From the sklearn documentation:

  The coefficient $R^2$ is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum(). The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a $R^2$ score of 0.0.


Why does balancing the test dataset improve precision-recall curve? I have a fairly imbalanced dataset for default-risk credit scoring (2:98). Both costs are fairly important i.e False negative means loss from default and false positive is a lost-revenue opportunity.

I tried training some models on the imbalanced set and tried over/under-sampling approaches as well as weight-balancing based on class. All of these approaches led to a decent ROCAUC. However, the precision-recall curves have looked horrible overall.

But if I were to balance the test set, the precision-recall curve looks much better. Why does this happen? And is there some adjustment I should be applying to improve precision-recall under the imbalanced data distribution?

Trying to answer

When dealing with class imbalance ROC is not a good criteria. There is a post on Kaggle about Credit Card Fraud Detection with experiment over the class imbalance, possible ways of mitigating the imbalance, better metrics and code in python for every experiment.

Since it is a really long post (actually it is a notebook on class imbalance and roc), but here I quote author's conclusion while Comparing PR curves to ROC:

  For a PR curve, a good classifer aims for the upper right corner of
  the chart but upper left for the ROC curve.

  While PR and ROC curves use the same data, i.e. the real class labels
  and predicted probability for the class lables, you can see that the
  two charts tell very different stories, with some weights seem to
  perform better in ROC than in the PR curve.

  While the blue, w=1, line performed poorly in both charts, the black,
  w=10000, line performed "well" in the ROC but poorly in the PR curve.
  This is due to the high class imbalance in our data. ROC curve is not
  a good visual illustration for highly imbalanced data, because the
  False Positive Rate ( False Positives / Total Real Negatives ) does
  not drop drastically when the Total Real Negatives is huge.

  Whereas Precision ( True Positives / (True Positives + False
  Positives) ) is highly sensitive to False Positives and is not
  impacted by a large total real negative denominator.

  The biggest difference among the models are at around 0.8 recall rate.
  Seems like a lower weight, i.e. 5 and 10, out performs other weights
  significantly at 0.8 recall. This means that with those specific
  weights, our model can detect frauds fairly well (catching 80% of
  fraud) while not annoying a bunch of customers with false positives
  with an equally high precision of 80%.

  Without further tuning our model, and of course we should do cross
  validation for any real model tuning/validation, it seems like a
  vanilla Logistic Regression is stuck at around 0.8 Precision and
  Recall.

  So how do we know if we should sacrifice our precision for more
  recall, i.e. catching fraud? That is where data science meets your
  core business parameters. If the cost of missing a fraud highly
  outweighs the cost of canceling a bunch of legit customer
  transactions, i.e. false positives, then perhaps we can choose a
  weight that gives us a higher recall rate. Or maybe catching 80% of
  fraud is good enough for your business if you can minimize also
  minimize the "user friction" or credit card disruptions by keeping our
  precision high.

My conclusion:

Even though your PR curve looks bad, given the class imbalance, that is the best metric and you should try to improve so your model is best fitted for your application

Observation:

You have a dataset with 100 samples and only 2 are from $C_1$, gathered right so far?

Supose now that from these 2, one is a complete outlier to its one class $C_1$, he provides more harm than good in your model fitting, and since he accounts for 50% of your $C_1$ samples he is going to make your model perform porly.


Methods to analyze time series data when only small amount of it is labeled I have time series data out of which only 10% is labeled into 10 classes. What should be the methods or models that I should look for to analyze this problem? I know the question is a bit vague but I am not looking for exact answers. I would appreciate any pointers or resources on how to approach these types of problems.

As mentioned in the comments semi-supervised methods are worth having a read. If the time-series are labelled per time-step you can make use of dynamical systems and Gaussian processes. Some methods that could be useful using NN's are:

Pretraining a neural network using an auto encoder. You can use an RNN to encode the temporal aspect of the series (https://papers.nips.cc/paper/5271-pre-training-of-recurrent-neural-networks-via-linear-autoencoders.pdf).
GAN's (Generative adversarial networks) currently very popular way of making use of your copious amounts of unlabeled data by jointly training two networks (supervised vs unsupervised) to play a zero sum game vs each other. https://arxiv.org/pdf/1611.09904.pdf (some work on RNN-GAN's again if you want to account on the temporal structure of your data). Note these are very hard to train.
Ladder Network https://arxiv.org/pdf/1507.02672.pdf . A very interesting semi-supervised method for regular fully connected networks. Which yielded very interesting results on the MNIST data set (only 300 labelled samples were used to reach state of the art results).
CNN's are known to be effective with time series too. Since they implement adaptive FIR filters on the data. It can be worth trialing in conjunction to RNN's.

All the methods above still require some reasonable amount of labelled data.


Subsetting and Ordering in R I have a random created dataframe named a:

  group marks  upd
1     a     2   up
2     b     3   up
3     a     4 down
4     b     5   up
5     a     6 down
6     c     7   up
7     a     8 down

I wanted to subset it with only group='a' and marks in decreasing order of a. I used this query but the marks are not arranged in decreasing order. What is the error in my code?

Here is my code and output:

> a[a$group=="a" & a$marks[order(a$marks,decreasing=T)],]
  group marks  upd
1     a     2   up
3     a     4 down
5     a     6 down
7     a     8 down

Warning message:
In a$group == "a" & order(a$marks, decreasing = TRUE)[1:3] :
  longer object length is not a multiple of shorter object length

a$group=="a" gives

# [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE

and a$marks[order(a$marks,decreasing=T)] gives

# [1] 8 7 6 5 4 3 2

Combining those to using a logical and (&) does not make much sense here. TRUE & 8 is TRUE - the subsetting may accidentally work. But you want order to look at your subset only, not at all rows in a.

Try for instance

newa <- a[a$group=="a", ]
newa[order(newa$marks, decreasing = TRUE), ]
#   group marks  upd
# 7     a     8 down
# 5     a     6 down
# 3     a     4 down
# 1     a     2   up

or, more convenient and readable:

library(dplyr)
a %>% filter(group=="a") %>% arrange(desc(marks))


Naive Bayes for Categorical Features (Non Binary) How do i use Naive Bayes Classifier (Using sklearn) for a Dataset
considering that my feature set is categorical, ie more than 2 categories per feature are present.

I've looked everywhere, some people have used GaussianNB even though
the data is categorical (1,2...8)

https://www.google.com/url?q=https://blog.sicara.com/naive-bayes-classifier-sklearn-python-example-tips-42d100429e44&sa=D&source=hangouts&ust=1567686518476000&usg=AFQjCNGORHakHbk2ZntzNZ1wsTYG6QUhig

Some people recommend converting it to One-Hot encoding and then using BernoulliNB which doesn't make sense to me because then the newly created features would have a high correlation which is against one of the core assumptions of Naive Bayes. (For example color feature has 3 values - Blue, Green, Red and we create 3 features out of it; then if Blue is 1 then its obvious that Red and Green will be 0. Hence the dependency)
Some people recommend using MultinomialNB which according to me doesn't make sense because it considers feature values to be frequency counts.

Can someone point me in the right direction?

In a recent scikit-learn release (v0.22.1), the developers have added Categorical Naive Bayes to their list of Naive Bayes implementations:

https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.CategoricalNB.html

You could also use my implementation of categorical and/or Gaussian Naive Bayes:

https://github.com/remykarem/mixed-naive-bayes


Modeling uncertainty from Logistic Regression Logistic regression is a part in a simulation pipeline that I use for some scenario analysis. The dataset that this is based on is not small but relatively noisy, and only one explanatory variable/feature. Of course I can say something about this uncertainty using frequentist or Bayesian methods but I would like to use this in the sequential simulation step as well, to get a fairer final estimate. What I'm planning on doing should work but is somewhat computationally expensive and I would like to know if someone has a better idea.

Approximate distribution of the coefficients of logistic regression with MCMC

Now simulate this n times:

Sample one of the two coefficients from this distribution
Fix this coefficient and use MCMC again to get approximated conditional distribution over second coefficient (could I get this in a better way? This seems highly correlated with the first coefficient)
Do the rest of the simulation run using this sampled model

I feel like there should be a better way but I'm uncertain.

I see many facets of your question and in what follows will present my top 2 :-)

Facet 1 -- assessing the uncertainty in estimated coefficients

In logistic regression, assessing the uncertainty in the estimated coefficients is virtually the same as for least-squares regression .

In both logistic regression and least-squares regression, the regression coefficient table will include a column for the regression coefficients followed by a column of standard errors, then by a column of test statistics, and finally a column of p-values. The table below shows the example coefficient table output for the some regression problem (where the   probability of buying some magazine for kids is estimated).

Note that the test statistics are labeled “z value” and the p-values are labeled “P(>|t|)” in the table above.
The standard errors can be used to construct confidence intervals for the regression coefficients. Roughly speaking, going plus or minus 2 times the standard error from the regression coefficient gives approximately a 95% confidence interval for the coefficient.

For example, for Residence Length, the regression coefficient is 0.024680. The next column gives the standard error of the regression coefficient which is 0.013800. Thus an approximate 95% confidence interval for the Residence Length regression coefficient is:

$$
0.024680 \pm 2 \times 0.013800 = 0.024680 \pm 0.0276\
$$

This means that the regression coefficient for Residence Length could be anywhere from -0.00292 to 0.05228 (with 95% confidence).

As is well known, we often use the odds-ratio, which is the exponential of the regression coefficient (i.e., $\exp(\beta)$ ), to help to interpret the meaning of the regression coefficient. The odds-ratio for the Residence Length coefficient, as shown in the coefficient table, is 1.0250. This means that there is a 2.5% increase in the odds of buying the magazine associated with each additional year of residence.

We can also compute the odds-ratios corresponding to the ends of the confidence interval. These odds-ratios will give us an equivalent confidence intervals for the odds. So continuing the example using Residence Length, the odds ratios corresponding to the ends of the confidence interval are

$$
 \exp(-0.00292)=0.99708
$$
 and
$$
 \exp(0.05228)=1.05367.
$$
Thus, the interval [0.99708,1.05367] is an approximate 95% confidence interval for the odds ratio. This means that there could be anywhere from a 0.292% decrease to a 5.367% increase in the odds of buying the magazine associated with each additional year of residence.

Facet 2 -- modeling data with measurements uncertainty

Here again many options are known. Take a look at this to get started: https://pdfs.semanticscholar.org/0a7a/a5e3d407b24e2f5c24287bfdda20e573bd05.pdf

I will just cite the source:

  in some datasets, the outcome of interest is measured with imperfect
  sensitivity and specificity. It is well known that the
  misclassification induced by such an imperfect diagnostic test will
  lead to biased estimates of the odds ratios and their variances.

  In this paper, the authors show that when the sensitivity and specificity
  of a diagnostic test are known, it is straightforward to incorporate
  this information into the fitting of logistic regression models.

  An EM algorithm that produces unbiased estimates of the odds ratios and
  their variances is described. The resulting odds ratio estimates tend
  to be farther from the null but have greater variance than estimates
  found by ignoring the imperfections of the test. The method can be
  extended to the situation where the sensitivity and specificity differ
  for different study subjects, i.e., nondifferential misclassification.
  The method is useful even when the sensitivity and specificity are not
  known, as a way to see the degree to which various assumptions about
  sensitivity and specificity affect one's estimates.

Hope this helps!


Structure of LSTM gates It is my impression that a single layer LSTM architecture consists of $t$ LSTM cells that are identical duplicates, where $t$ is the number of time steps. Then there are gates within the LSTM cell. I have struggled to find a rigorous explanation of what each “gate” actually consists of. Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?

Thanks

Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?

Close. Each gate is an activation function (typically sigmoid for actual gates, but tanh is used for other functions within the cell) over a weighted sum of all inputs to the cell's layer. Although spome reading of the diagrams may imply LSTM cells are processing a few single inputs in isolation, recombined later, in fact the inputs to each cell are the whole previous layer and the whole previous timestep cell states for the whole layer. This is similar to a view of a single neuron in a fully-connected feed-forward network.

Each gate in each cell has its own set of weights - one weight for each input from previous layer plus one weight for each cell state in the same layer, plus a single bias. When you combine these cells into a layer of multiple cells (e.g. when you choose to have an LSTM layer with 64 cells), this results in a separate matrix plus bias for each gate. In descriptions of LSTMs, these different matrices are often named for the type of gate they parametrise, so e.g. there will be a matrix plus bias for the layer's "forget" gate, which might be noted as $\mathbf{W}_f$.

If you have $N_i$ inputs, and $N_c$ LSTM cells in a single LSTM layer, then $\mathbf{W}_f$ will be a $N_c \times (N_i + N_c)$ matrix of weights, and so will all the other parameter matrices describing other gates and value calculations for the combined cells.

In practice the calculations don't need to be handled per cell, but calculations over a layer of cells can be vectorised, and it is more like having a few parallel fully connected layers that combine in various ways to generate output plus next cell states.

There would be nothing stopping you extending this architecture and making any single gate or value calculation deeper by giving its own hidden layers. I suspect this has been tried by researchers, but cannot find any references. But without this customisation, the gates in standard LSTM are more like logistic regression over concatenated input and cell state.


Issues with training SSD on own dataset I'm new to ML and trying to train a SSD300, with some Keras-Code github.com/pierluigiferrari/ssd_keras I found on github.
For training I'm using an own (very small) dataset of objects that are not in any of the bigger known datasets.
My dataset has the following characteristics:

objects have very different sizes in images (from around 20x40 to 250x200)
there is only one class labeld in the images
images are in RGB
all images are sized to fit in 300x300
dataset contains 319 images for training and validation

Now my problem is, that the loss-function for validation doesn't converge, but training loss does. See this image showing the loss functions over the epochs. I trained 120 epochs with 1000 steps each:

When I try to use the trained weights, coming out of this training, I get zero detections in image. It seems like the model didn't learn anything.
I'm using pretrained weights for the underlaying VGG-16 network provided in the github-repository. It is trained on imagenet dataset.
My parameters are as follows:

img_height = 300 # Height of the model input images
img_width = 300 # Width of the model input images
img_channels = 3 # Number of color channels of the model input images
mean_color = [123, 117, 104] # The per-channel mean of the images in the dataset. Do not change this value if you're using any of the pre-trained weights.
swap_channels = [2, 1, 0] # The color channel order in the original SSD is BGR, so we'll have the model reverse the color channel order of the input images.
n_classes = 1 # Number of positive classes, e.g. 20 for Pascal VOC, 80 for MS COCO
scales_pascal = [0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05] # The anchor box scaling factors used in the original SSD300 for the Pascal VOC datasets
scales_coco = [0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05] # The anchor box scaling factors used in the original SSD300 for the MS COCO datasets
scales = scales_pascal
aspect_ratios = [[1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5]] # The anchor box aspect ratios used in the original SSD300; the order matters
two_boxes_for_ar1 = True
steps = [8, 16, 32, 64, 100, 300] # The space between two adjacent anchor box center points for each predictor layer.
offsets = [0.5, 0.5, 0.5, 0.5, 0.5, 0.5] # The offsets of the first anchor box center points from the top and left borders of the image as a fraction of the step size for each predictor layer.
clip_boxes = False # Whether or not to clip the anchor boxes to lie entirely within the image boundaries
variances = [0.1, 0.1, 0.2, 0.2] # The variances by which the encoded target coordinates are divided as in the original implementation
normalize_coords = True

Can someone please help me by pointing out these questions:

How to interpret the loss function? Is it because of the small dataset or because of wrong parameters or something else?
Do I have to train my own classifier (VGG-16) or can I use the pretrained one even when my objects don't appear in the pretrained dataset?
Do I have to train for a longer time? Means for more epochs?

As additional information:
I already trained a faster R-CNN model with the exact same dataset. It worked quiet good and gives me good results.

I would appreciate any help you can provide!

What you are experiencing is called overfitting and it happens because of your very small dataset.

All the model cares about is performance on the training dataset, so given the opportunity, it will simply attempt to memorize it. This is what happens in you case, you feed a model which contains over 130 Million parameters less than 319 images.

So regarding your questions:

The loss function shows a clear case of overfitting.
On general, it is okay to use a trained model, especially when you only have a small dataset, but in your case, the dataset is too small for any deep-learning model. When I say small dataset, I mean 10k images, not several hundreds.
You should not train for longer time, once the validation loss stops improving, it is a clear sign to stop. There is even a training technique named "early stopping" which is designed to stop training once the validation loss stops to drop.

You have to understand that currently, your dataset of 300 images, is irrelevant to the world of deep-learning. So if you still want to use it for object detection, you need to revert to more classic computer-vision techniques like using HOG or SIFT features, or even manually engineering the features for your special case.


Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function) A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time series analysis for every column, what should be the correct approach?

To replace by column means, an easy approach would be to use the base R function colMeans. Let's say you have a data frame df.

1) If you want to replace the NAs per column one by one, you could try this:

df <- sapply(df, function(x)ifelse(is.na(x), mean(x, na.rm=TRUE), x))

2) If you want to replace all NAs in one go, you could try this:

df <- ifelse(is.na(df), rep(colMeans(df, na.rm=TRUE), rep(nrow(df), ncol(df))), unlist(df))


What (probabilistic models) can only output decisions when they are certain? I'm basically looking for approaches, models, algorithms for the following situation (a fault diagnosis problem):

I have an input set $\{x_i\}_{i \in \{1..m\}}$ with $n$ binary features of cases (think of "faults" or "alarms" that fired) and $k$ classes. Each case $x_i$ can belong to at least one class and at most $k$ (so I'm dealing with multi-label classification).

Now some relations in the data set are utterly boring/uninformative (say, feature $a$ says "Mechanical Error occurred" and label $b$ means "Mechanical Error fixed").
But even more generally, whenever $x_a = 1$, I see all kinds of labels, i.e., $a$ is not predictive. Put differently, the relation is not "functional".

Other input features $c$ might have a much more "functional" nature, such that whenever $x_c = 1$, I can easily deduce $y_d = 1$.

For instance my training set could look something like this:

$ [0, 1,0] \mapsto \{4, 1\}$

$ [0, 1,0] \mapsto \{2\}$

$ [1, 0, 0] \mapsto \{1\}$

$ [1, 0, 0] \mapsto \{1\}$

So, knowing $[0,1,0]$ is not really informative whereas $[1,0,0]$ tells me (with high confidence) that the label 1 is active.

I'm looking for the latter pairs, a classifier that only extracts meaningful pairs and ignores uninformative inputs.

Could you point me to relevant techniques / keywords? Thanks a lot!

You can get probabilities for all labels. You can set a threshold for confidence if proba is greater than some value classify otherwise remain silent.


How Ipsative Standardization actually works for time series? In this link: http://optimalprediction.com/files/pdf/V2A17.pdf
Ipsative Standardization is explained intuitively and also the why it is useful.

Is there any python code or math formulas online someone could use to calculate the z1 score?

The pdf describes the following way:

Let us call the variable as var
where

m = mean(var)

sd  = sd(var)

x = (var-m)/sd

This will create a list of values which are Ipsative standardized, which is called Zi score


sk-learn - ValueError: array is too big. I have a large dataset with characters and 90000 intances and I have the error ValueError: array is too big when I have the following code before the plot_kmeans_digits.py code:

data2=list(csv.DictReader(open('C:\diabeticdata.csv', 'rU')))
vec = DictVectorizer()
data = vec.fit_transform(data2).toarray()

Do you know how I can solve this error?

Thanks in advance.

The problem is that you're converting a large sparse dataset into a dense array via toarray(). Don't do that :) Why do you need it in dense array format? Try posting more of your code to see if we can figure out better ways to solve the problem. With any large dataset, you want to stay sparse as long as possible and only work with smaller sets of dense data (if dense is needed at all).


Convert Atypical Date Format for Time Series in Python I have an atypical time format that I need to convert into a datetime index for time series analysis.  I'm working in Python / Pandas.

The column is 'BC_DT', and the format is "27-MAR-18".  Example is below.

BC_DT
27-MAR-18
28-MAR-18
29-MAR-18

I tried this method, but I'm getting an error: ValueError: time data '27-MAR-18' does not match format '%d-%b-%Y'

df['Converted_Date'] = df['BC_DT'].apply(lambda x: dt.datetime.strptime(x, '%d-%b-%Y'))

Let pandas determine what datetime format you are using automatically.

import pandas as pd
raw_data = pd.DataFrame(data={'BC_DT':['27-MAR-18','28-MAR-18','29-MAR-18']})
raw_data['BC_DT'] =  pd.to_datetime(raw_data['BC_DT'])
print(raw_data)

  BC_DT 0:   2018-03-27 1:  2018-03-28 2:  2018-03-29


What is the purpose of a 1x1 convolutional layer? SqueezeNet uses 1x1 convolutions. I try to understand it in this simple example: if the input is one MNIST digit, i.e. of shape 1x28x28x1 (I use Batch x Height x Width x Channel).

Then applying a Conv2D(16, kernel_size=(1,1)) produces an output of size 1x28x28x16 in which I think each channel 1x28x28xi (i in 1..16) is just the multiplication of the input layer by a constant number. Is that right?

More specifically: Output[channel i][x,y] = InputLayer[x,y] * alpha_i for x,y in 1..28, where alpha_i is a constant for each channel.

Is this correct?

It's like going from 1 channel to 16 identical channels (except that they are multiplied by one global constant per channel).

What is its purpose?

Note: I have already read How are 1x1 convolutions the same as a fully connected layer? and 1x1 Convolution. How does the math work? but here it's slightly different.

First of all, Conv2D(16, kernel_size=(1,1)) applied to 28x28x1 will produce 28x28x16 as it doesn't change the number of dimensions. Secondly, I took a look at the paper you refer to, and haven't found that they apply 1x1 filters to an input with just one channel as in the example you provided. In case of just one channel in the input your reasoning is correct, but it cannot be transferred to input with several channels.

Conv2D by definition, assign weights between all channels of a selected area (in case of 1x1 kernel to a whole column of nodes) and corresponding column of nodes in the output grid. Then it applies these weights subsequently to all valid areas according to stride and padding parameters. Thus applying 1x1 kernel on 1-channel input doesn't make sense as all output channels will be proportional to each other and to the input channel as well. In case of multiple channels, as per definition Conv2D each node of the output will be a weighted (non-trivial) sum of a corresponding input column and won't be proportional to either channel of the input in most cases.


Model for balancing 2 sport teams with multiple players? I need some guidance on what to start researching for this. I have 10 players and I need to auto-generate 2 teams. Every time a match is finished, I need to re-balance the teams. For that, I have two features:

Win/loss ratio.
Players that have better performance playing together.

Any ideas on which model can I use to get this automatic balance running?

Thanks!

There is some lack of information on your post.

Does it take long time to try a match? (can I do 10.000 simulations or matches in minutes?)
What's your goal? Is to create teams that are as equal as possible?
Do the players really have better performance when they are together or you are assuming it?

But, I will try to give you an answer:
If it takes long what you can do is shuffle the players and measure the win/loss ratio, that resumes all you need, statistically speaking you will be conducting what is known as a experiment. If the players really "interact" you will have to measure the win/loss ratio by pairs.


Neural Network written in Kotlin working for simple math problems but not for MNIST classification I'm building a neural network in Kotlin while reading the book "Neural Networks and Deep Learning" from Michael Nielsen.
At the moment the network uses: sigmoid neurons, backpropagation, stochastic gradient descent, $1 \over \sqrt {n_{in}}$ standard deviation for weight initialization, L2 regularization, and cross-entropy. I also created a plotter to monitor the learning process.
Here is how the network behaves with a simple problem, finding the bigger of two numbers:

Pretty good, we have a peak of 99.99%.
I can't understand why the cost is increasing instead of decreasing, but still the accuracy is good.
Things get way worse with MNIST classification; the net is basically giving random answers:

I wonder what can change the behaviour of the network this much.
I checked the MNIST data, and it seems all right.
What problem can cause this kind of learning absence? The algorithm is broken? The hyper-parameters are wrong?
All help is appreciated.

I managed to solve the problem on my own. Turns out that the problem was indeed in the hyper-parameters, as I suspected.
I was using the same values as Michael Nielsen so it seemed strange to me that they could be wrong - by a mathematical point of view, my network is exactly the same as his one! - anyway, lowering the $\eta$ and raising the $\lambda$ (in practice, lowering the weights/biases changes on a single batch) lets the network to better understand which direction it should take to minimize the cost function.
I can now get accuracies above 85%, and I'm sure that with a bit of fine-tuning I can reach at least 95%.
I also solved the cost plot problem:

I can't understand why the cost is increasing instead of decreasing [...]

The problem was that instead of doing something like
$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j)] + {λ \over 2n} ∑_w w^2$
I was doing
$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j) + {λ \over 2n} ∑_w w^2]$
which doesn't make sense, since the last term equals to
$∑_{xj} {λ \over 2n} ∑_w w^2 = n {λ \over 2n} ∑_w w^2 = {λ \over 2} ∑_w w^2$
and so we get a wrong regularization factor for the weights.


What is seed in Keras? I have seen 'seed' parameter in many codes of Keras.
I have a code example here:

validation_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset = 'validation')

Can you explain what is it? And when to use it?

As most of statistical models (regression, neural networks etc.) use probabilistic frameworks in their optimazation process (E.g. softmaxfrom a random distribution). You need a seedto reproduce your results. A seed is a fixed "value set" drawn from a random distribution.

If you would run your model several times you will get slightly different estimates as the starting values from your optimazation will slightly differ every time.

EDIT

Some additional information on the optimazation process when dealing with Neural Networks.

What you are optimizing in neural networks are the weights (connections) between the Neurons. Connections that have strong predictable character get somewhat stronger (more weight) relative to other connections with less predictive power. This process is done when you "train" your data. However to train your network you need some "inital weights" which are normally assigned by random (there are different approaches, but this is the most common one).

Like previously said - The inital weights will have impact on your optimazation and the results. That other users can check your model / reproduce your results you need to save the inital weights. - Therefore you use a seed


Why my training and testing set are about 99% but my single prediction does wrong prediction? I have performed fruits classification using CNN but i am paused at a point where all things are going right confusion matrix accuracy score all are correct it seems there is no overfitting but it always classifies wrong fruit. Why would this happen. Link to source code is provided below. Thank You!
Github source code link

It looks like the new data has a different distribution from the training data. It looks like the training data is just a single fruit, with white background, and the new image you've passed is a picture of bananas with blue background. The model has probably learned something like: if blue image, then blueberries, and for this reason it classifies the blue bananas picture as blueberries.
Whenever the distribution of new data is different from the data you've trained on, don't expect the model to work very well, as ML models just interpolate.


Learning AI, Machine Learning, Deep Learning I am a recent college graduate with a bachelors in computer engineering. I am very interested in deep learning however someone had mentioned to me that I should learn machine learning before I learn deep learning. Is that true ? And if so which online course would you recommend I start at ?

Bascially this is true. Deep Learning is kind of like stacking your machine learning algorithms on top of each other. Have a look here and the associated picture.


Anomaly detection k-means in Time Series I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset:

8   2018-09-06  -2760.14
9   2018-10-04  -1149.73
10  2018-11-07  -1551.41
11  2018-12-06  -1208.17
12  2019-01-07   -244.02
13  2019-02-06    111.00
14  2019-03-06    139.24
15  2019-04-04 -27315.10
16  2019-05-07 -30326.58
17  2019-06-06  -1633.44

I used 2 as number of clusters and I get the following centroids:
[[  -237.045]
 [-28820.84 ]]
In my case the anomalies are rows 15 and 16 but now they form their own cluster so the distance to their centroid will be insignificant and thus not an anomaly
My question is, how can I detect the anomalies if they form their own cluster?

I don't suggest to use k-Means clustering for outlier detection. You can check the outlierness of observations by taking the standardized distance of each observation from the series' trend.

Alternatively, you can use DBSCAN algorithms: they are clustering models specifically designed to isolate outliers. I don't know if it makes sens on time series data. I'd go for the first method.


Predicting contract churn/cancellation: Great model results does not work in the real world I'm busy with a supervised machine learning problem where I am predicting contract cancellation. Although a lengthy question, I do hope someone will take the time as I'm convinced it will help others out there (I've just been unable to find ANY solutions that have helped me)

I have the following two datasets:

1) "Modelling Dataset"

Contains about 400k contracts (rows) with 300 features and a single label (0 = "Not Cancelled", 1 = "Cancelled").

Each row represents a single contract, and each contract is only represented once in the data. There are 350k "Not Cancelled" and 50k "Cancelled" cases.

Features are all extracted as at a specific date for each contract. This date is referred to as the "Effective Date". For "Cancelled" contracts, the "Effective Date" is the date of cancellation. For "Not Cancelled" contracts, the "Effective Date" is a date say 6 months ago. This will be explained in a moment.

2) "Live Dataset"

Contains 300k contracts (rows) with the same list of 300 features. All these contracts are "Not Cancelled" of course, as we want to predict which of them will cancel. These contracts were followed for a period of 2 months, and I then added a Label to this data to indicate whether it actually ended up cancelling in those two months: 0 = "Not Cancelled", 1 = "Cancelled"

The problem:

I get amazing results on the "Modelling Dataset" (random train/test split) (eg Precision 95%, AUC 0.98), but as soon as that model is applied to the "Live Dataset", it performs poorly (cannot predict well which contracts ends up cancelling) (eg Precision 50%, AUC 0.7).

On the Modelling Dataset, the results are great, almost irrespective of model or data preparation. I test a number of models (E.g. SkLearn random forest, Keras neural network, Microsoft GbmLight, SkLearn Recursive feature elimination). Even with default settings, the models generally perform well. I've standardized features. I've binned features to attempt improving how well it will generalize. Nothing has help it generalize to the "Live Dataset"

My suspicion:

In my mind, this is not an over-training issue because I've got a test set within the "Modelling Dataset" and those results are great on the test set. It is not a modelling or even a hyper-parameter optimization issue, as the results are already great.

I've also investigated whether there are significant differences in the profile of the features between the two datasets by looking at histograms, feature-by-feature. Nothing is worryingly different.

I suspect the issue lies therein that the same contracts that are marked as "Not Cancelled" in the "Modelling Dataset", which the model trains to recognize "Not Cancelled" of course, is basically the exact same contracts in the "Live Dataset", except that 6 months have now passed.

I suspect that the features for the "Not Cancelled" cases has not changed enough to now make the model recognize some of them as about to be "Cancelled". In other words, the contracts have not moved enough in the feature space.

My questions:

Firstly, does my suspicion sound correct?

Secondly, if I've stated the problem to be solved incorrectly, how would I then set up the problem-statement if the purpose is to predict cancellation of something like contracts (when the data on which you train will almost certainly contain the data one which you want to predict)?

For the record, the problem-statement I've used here is similar to the way others have done this. And they reported great results. But I'm not sure that the models were ever tested in real live. In other cases, the problem to be solved was lightly different, e.g. hotel booking cancellations, which is different because there a stream of new incoming bookings and booking duration is relatively short, so no bookings in common between the modelling and live dataset. Contracts on the other hand have long duration and can cancel at any time, and sometimes never.

If your model makes a prediction 6 months into the future, then it doesn't make sense to judge its performance before 6 months. If only 2 months have passed, then possibly 2/3 of the true positives have yet to reveal their true nature and you are arriving a premature conclusion.

To test this theory, I would train a new model to predict 2 months out and use that to get an approximation of live accuracy while your wait 4 more months for the first model. Of course, there could be other problems, but this is what I would try first.


Organizing a csv file of multiple datasets into a list of Pandas dataframes I have a csv file, containing results from a Computational Fluid Dynamics (CFD) simulation (a sample of my csv file is attached as a google drive link; file size: 1,392KB). In particular, the csv file has information about multiple streamlines (number of streamlines may reach 1000 depending on the case). All the data for all the streamlines are saved back to back in the csv file (so there is no empty row or something to tell us the end of one streamline and start of the next one). The only way I can distinguish streamlines from each other is that when the value in column "IntegrationTime" is zero, it indicates the start of a new streamline, until we hit another zero in the"IntegrationTime" column which is the start of the next streamline.
I need to read this csv file, and organize its data into a list of Pandas datafreames, like:
streamlineList = [df_for_streamline_1, df_for_streamline_2, ...., df_for_streamline_N]

Note (extras question here): This is not crucial but would be nice to have: if you look at the end of my csv file, you see multiple rows where IntegrationTime is zero (100 rows to be exact). Preferably, I don't need these lines to be included in my final list of data frames.
Can somebody help me a way to do this?
https://drive.google.com/file/d/1lJhOJadGrno1C-KZOUqxV-KNkaOHIJNk/view?usp=sharing

It is possible to solve this problem procedurally by thinking line-by-line or streamline-by-streamline.
For each streamline that can be matched with IntegrationTime == 0.0, extract the slice from the data frame, and append the slide to an output list (if it has more than 1 data point). A code like the following should address this problem:
import pandas as pd

# read the dataset
df = pd.read_csv("vel.csv")

# get row indexes where integrationtime is zero (start of each streamline)
start_index_list = df.loc[df['IntegrationTime']==0.0].index.values
stream_line_list = [] # the output list

for i in range(len(start_index_list)):
    # for each stream line, obtain the slice of the original dataframe that corresponds to it
    start_index = start_index_list[i]
    end_index = None
    if i+1 < len(start_index_list):
        end_index = start_index_list[i+1]-1
    stream_line_df = df.loc[start_index:end_index]
    # only append streamline with more than 1 data point
    if len(stream_line_df) > 1:
        stream_line_list.append(stream_line_df)

print(f"number of complete streamlines found: {len(stream_line_list)}")
```


Calculation and visualization of islands of influence I recently saw the graph of correlated technologies from the SO Developer Survey, which was pretty; reproduced below:

I have a weighted graph that represents the amount of communication between pairs of people. I'd like to create a similar visualization as above. I want to be able to tease out "islands" of people with high influence among each other, but little influence elsewhere.

To make this concrete, let's say I have four animals: dog, wolf, cat, lion. The canines talk a lot amongst themselves; the felines do too, but members of one group only rarely talk to members of the other group. So my influence graph (which is symmetric, hence the upper triangular matrix) looks like:

+------+-----+------+-----+------+
|      | Dog | Wolf | Cat | Lion |
+------+-----+------+-----+------+
| Dog  | ∞   | 100  | 1   | 1    |
| Wolf | -   | ∞    | 3   | 2    |
| Cat  | -   | -    | ∞   | 130  |
| Lion | -   | -    | -   | ∞    |
+------+-----+------+-----+------+

I know how to create graphs whose edges are visually represented by the weight between two nodes, but I am not sure how to account or calculate islands of influence. Are there general visualization techniques to apply to a data set like this, and also for figuring out islands of influence?

Thanks in advance!

As you've explained it, you want to detect communities, which is a common problem in social network analysis. If your network had been directed and temporal, I would have suggested you look into influence analysis; e.g., as in Detecting Communities of Authority and Analyzing Their Influence in Dynamic Social Networks.


Multiple classification algorithms are predicting always exactly with the same scores. Is that normal? If not, what should I suspect? I have been working on a multilabel classification problem. I am using Python machine learning libraries to implement the classification algorithms. For the cross-validation, I am using repeated K-Fold cross-validation. I have experimented with SVM, Logistic Regression, Random Forest, Decision tree, K-Neighbour, and Naive Bayes and using Binary Relevance, Classifier Chain, and Label Powerset transformation methods for all of them.

I noticed, for Classifier Chain, SVM, Logistic Regression, Random Forest, and K-Neighbors are always achieving same subset accuracies and hamming losses. For Label Powerset, SVM, Logistic Regression, and Random Fores are achieving the same scores. However, for the binary relevance, all scores are different. No matter, what random seed I use or how many repetitions of cross-validations I run, they always end up scoring the same. I was wondering if this is a normal occurrence or they are suffering from class imbalance or overfitting or anything of that sort problems?

P.S. I am a beginner. Sorry for bothering with a naive question.

Same accuracies and losses for multilabel classification usually refer to the fact that model just chooses majority class. Please check the confusion matrix. Definitely, it is not a normal occurrence and you need to deal with it.


Differences between a Statistician and a Data Analyst in industry What is the Difference in the job of a Statistician and a Data Analyst In Industry? My take is that although both analyse data, a Statistician deals with the more theoretical aspects of data such as using mathematics to analyse data and to create mathematical models of data. So their work is more towards the mathematics side of things. Whereas the job of a Data Analyst deals more with the programming and side of things and their job is more practical in nature and they focus more on the implementation side.

Im more of a theoretical person and im more mathematically orientated and i wanted to do a job that involves lots of math, so is a statistician job more suited for me rather than a Data Analyst?

A major difference is the job market: you'll find a lot of job ads for data analysts/scientists, very few for theoretical statisticians. In most sectors (there are some exceptions, in banking for example), companies are interested in applying existing models to their data because this is what can increase their profits.

Devising new theoretical models is more on the research side of innovation. Most "pure research" job opportunities are in academia, although some big companies have research departments as well.


Constituency vs Dependency Parsing: What is more effective for Sentiment Analysis? Parsing is often used to understand the sentiment of complex sentences filled with double negations or very articulated.
There are two main ways of parsing a sentence: Constituency and Dependency Parsing. What is the most successful application for Sentiment Analysis?

Both approaches have been used for sentiment analysis in the literature. From a quick search, we can find these results:

Constituency parsing:

Deep Recursive Neural Networks for Compositionality in Language [NeurIPS'14]
Less Grammar, More Features[ACL'14]
A Statistical Parsing Framework for Sentiment Classification [Computational Linguistics journal'15]

Dependency parsing:

What's great and what's not: learning to classify the scope of negation for improved sentiment analysis [NeSp-NLP'10]
UBham: Lexical Resources and Dependency Parsing for Aspect-Based Sentiment Analysis [SemEval'14]

It seems to me that constituency parsing has been used more than dependency parsing. My sample, however, is very small and focused on the English language. I think this may change in languages with more complex (and more non-projective) syntax.
That been said, I think currently neither of them are considered to be state of the art in sentiment analysis. If we take a look at the leaderboards of sentiment analysis tasks from paperswithcode, all of the leading approaches (BERT, RoBERTa, M5) handle text as a mere sequence of tokens. While these results are from only a


Time series data combined with multiple feature. Predicting difference from the mean I'm trying to predict the % attendance of people to gym classes that have previously been booked. It is heavily dependent on the time of day and also a load of other features (is it raining, fraction of class that booked yesterday compared to a week ago, etc.). Random forest alone performs very poorly.
Instead, I tried to predict the difference from the mean for the hour of the day using random forest.  Then I just add that on to the mean to recover the desired estimate.
This again performs worse than just the mean itself.

Is this approach (predicting the difference from the mean) a bad idea?
I can't find people using similar methods which makes me think it isn't a good idea.

Is there a better algorithm suited to this task?

With random forest there is no need to modify your output variable (i.e difference from the mean) when performing a regression.

It may be that the features you have are not relevant, or that you don't have enough data. Also try different model parameterisations (num trees, num layers etc).

Try to do some feature engineering to create intelligent features that might help the classifier.


Performing 1D Depthwise conv using Keras 2D Depthwise conv I would like to perform a 1D Depthwise convolution (ie the first step of the depthwise-separable convolution) for a machine learning model I am working on. This means that for an input activation space of h x w, there are h kernels, each 1xk, where k is the length of the kernel (see attached figure).
This is not currently offered by Keras, but one can use the available Depthwise 2D convolution (which operates on 3D inputs), and simply add an extra dimension to the 2D inputs, and permute such that the original 2D height will span the channels of the new 3D activation space (see figure). The model report, showing these dimensions and permutations is below (the format for the Depthwise layer is "channels first").
However, I do not seem to be able to mimic this behaviour in a matlab golden model which uses the same input activation space and kernels. Does this method of using a Depthwise Conv2D to operate like a Depthwise Conv 1D make fundamental sense, or practical sense with keras, or am I missing something with the channel permutations perhaps?

Indeed I was able to prove that this method works, confirmed by my Matlab golden model. The problem was that in order to read the output activations of an intermediate layer in Keras, you have to create a separate model with the specified output.
This separate model in my case did not have the same weights as the original model, leading to incorrect outputs. But now that I read the output activations correct, I can confirm this method works.


Simple question about prediction classes of item in question vs not item in question Let's say I wanted to use transfer learning to train a model to detect object A vs everything else. In this case, do I provide 2 types of input, images of object A and images of everything else, and then have the final layer of the model output either object A or not-object A?

What about in the case where I want object A vs object B vs everything else. Would it make sense in this case to provide images of A and B and then have only two output classes, but based on the confidence of the output, interpret it as 3 classes? Say that it's object A if the confidence in that is > 50%, object B if the confidence in that is > 50%, and anything else if neither of those two conditions are met?

It depends if the percentages are required to sum to 100%. Typically when training on only 2 classes, the model will make predicts that sum to 100% for 2 classes. There will be no chance for out-of-class predictions.


How to approach mapping families of vectors on a lattice and forecast resulting value I describe here a model to describe how neighbours influence a node.

I wish to implement it to attempt forecasting to values associate nodes;
I post here asking for suggestions on mathematical model and machine learning techniques that could have already considered a similar approach, but I am not aware of, and hints for their implementation (python).

Suppose you a have a squared 2D lattice (a grid of 9 squares for simplicity), and:

for each time t
from each cell in the grid

you have vectors pointing to or from the central square.

Like this example, at time t1 :

I want to "map" the vectors to a parameter, associated to the central square.

I think vectors could be modelled as "a matrices X " ( or a family of vectors), and a resulting value y, but I am actually struggling to correctly represent it in a formal way.

Like in this second image:

I then want to forecast the possible y value in the future.
I think I could explore linear or non linear models, like regressions or forecasting applied to times series:

but I want to train on the time-series of vectors, not on the time-series of ys (Hope it s clear).

Could you help to suggest models to look at and possible implementations (python), which may have already approached this dynamics?

Could you also suggest approach to generalise to manifolds structures (3d) ?

As examples for generalisation, I am thinking at:

a lattice of 3D ( neighbors cubes influences a central cube);
Ys as vectors, not scalars ( see final image to explain what I mean ) where a resulting vector is interpreted as the resulting vector at the next step in the future t + 1

It really seems like a problem of formulating your probkem. On your 2D exemple you can split your $X_i$ along your axes and juxtapose them to build a data set. Each $X_i$ has two components : along axis 1 ($X_{i1}$) and 2 ($X_{i2}$).

Basically you would just build your data set by juxtaposing the 18 columns :

$X_{11}$ , $X_{12}$ , ..., $X_{i1}$ , $X_{i2}$ ,..., $X_{91}$ ,$X_{92}$

To try to predict your y.

You can augment the dimension of you problem by similar juxstaposition of your features.
It would get a bit more complex if you augment the dimension of y, as you would need multiple models.

Trying to predict $y_{t+1}$ would lead to other considerations, like considering $y_t$ as a feature of your model.