Random Forest Techniques/Models Can anyone tell about different Techniques/algorithms of Random forest?
I know, Random Forest is itself an algorithm/model, but I'm looking for another version of it as we have in decision trees.
List of Algorithms based on Random Forest?
Thanks

Random Forest is a very specific technique, is itself a variation of other techniques.

In this link you will find a list of techniques related to random forest:

Forest of randomized trees


How to reach time points clustering? What I met a problem is I do time-series clustering, and I found the clustering result isn't ideal.

I can't use elbow method to know what clustering result is good, that means I have no ways to watch the clustering results and tune parameters.

What I want to do is cluster the same trends data into the same group.

But I met a problem like below:

>> arr # It's a time-series array with 10 time intervals.
array([[  0,   0,   100, 0,   0,   0,   0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0, 100,   0],
       [100,   0,   0,   0,   0,   0,   0,   0,   0,   0]])

>> model = KM(n_clusters=2).fit(arr) ; model.labels_
array([1, 0, 0], dtype=int32)

It clusters arr[1] and arr[2] the same group!

But!With our eye watching, we all know arr[0] and arr[2] should be the same group because they have 100 in the near time intervals.

How to do time-points clustering? And specify random_state=N is useless becuase there is always one corner case to let me fail.

UPDATE:

There's one algorithm to solve this and its name is KShape. It can be found in tslearn github.

It clusters time-series data based on the shape of each data, so it match my needs.

In k-means there is no similarity or order of axes. They are all assumed to be independent. Hence, these three series are all equally dissimilar. You got what you asked for - but the question you asked (minimize the squared errors in each component) was not what you wanted.

You'll need to choose a different algorithm. One that knows about time, and allows for some tolerance in time, or one where you can use a distance function such as DTW to measure similarity.

Don't treat clustering as a black box. That will not work, you'll get results that you don't like. Instead, understand the algorithms first, then guide them carefully to solve your problem.


Why this model does not converge in keras? This case has an underlying story but I have essentially boiled it down to the simplest possible re-producible example I could.

Essentially let us think that I have up to 1000 nodes and each node represented by a small (this case is a 3-cell vector) vector and I concatenate and represent these nodes as a padded 3*1000 input vector and need to find out which one is more suitable. So the model is trying to predict 1000 float values, one for each node.

Let's imagine the function to score nodes is this arbitrary code:

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

And essentially my model is supposed to learn this function plus an argmax to find the node that has the highest score. This looks to me like a pretty simple problem compared to the problems I have solved so far (but it is different too).

So my question is why doesn't this model converge? I am thinking it could be due to its differentiability but really kinda lost and started to doubt everything I know about NN (which is not a lot).

Here is the repro code:

import numpy as np
import math

from keras import Sequential, Input
from keras.layers import Flatten, Activation, Dense
from keras.optimizers import Adam

class Defs:
  VECTOR_SIZE=3
  NODE_COUNT=1000
  MAX_REAL_NODE_COUNT=400
  MIN_REAL_NODE_COUNT=20
  INVALID_SCORE=0

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

def build_vector():
  a = np.random.randint(1, 100)
  c = np.random.randint(1, 50) if np.random.choice([False, True, True]) else 0
  b = 0 if c == 0 else np.random.randint(c, c*3)
  return [float(a), float(b), float(c)]

def build_vectorset_score():
  n = np.random.randint(Defs.MIN_REAL_NODE_COUNT, Defs.MAX_REAL_NODE_COUNT)
  vectorset = []
  for i in range(0, n):
    vectorset += build_vector()

  # pad it
  vectorset += [0. for i in range((Defs.NODE_COUNT-n) * Defs.VECTOR_SIZE)]
  scores = [score_vector(vectorset[i*Defs.VECTOR_SIZE:(i+1)*Defs.VECTOR_SIZE]) for i in range(0, Defs.NODE_COUNT)]
  index = np.argmax(scores)
  scores = [1. if index == i else 0. for i in range(0, len(scores))]
  return vectorset, scores

def build_model():
  model = Sequential()
  model.add(Dense(Defs.VECTOR_SIZE * Defs.NODE_COUNT, input_dim=Defs.VECTOR_SIZE * Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT))
  model.add(Activation('softmax'))
  print(model.summary())
  model.compile(loss="categorical_crossentropy",
                optimizer=Adam(lr=0.001), metrics=['categorical_accuracy'])
  return model

if __name__ == '__main__':
  SAMPLE_SIZE = 1 * 1000
  X = []
  Y = []
  for i in range(0, SAMPLE_SIZE):
    x, y = build_vectorset_score()
    X.append(np.array(x))
    Y.append(np.array(y))
  model = build_model()
  model.fit(np.array(X),
                  np.array(Y), batch_size=100, epochs=200, verbose=1)

I'm not totally sure exactly what you're doing with your scoring equation, but the first thing you need to look at is your loss function. Categorical Crossentropy is for multilabel classification, and you're trying to predict a float value.

So, you should have your network output be a single value (and don't squash it through a sigmoid unless the range of your function is (0,1)).  You should be using a regression loss function - I'd definitely start with mean squared error. Check out the regression example here (under Regression Predictions) for some sample code.

Edit: Further to discussion below, the network cannot predict the index of the highest-valued node because each node's score has no dependency on the node's position among the 1000 nodes being scored. As the probability of the best node being in each position is the same, there is no optimum prediction to be learned.


Improve NER label results on Non-English text I am working on some Medieval Latin text and was using various methods of NER such as CLTK (Latin Model), Spacy (Multilingual, Italian, Spanish Model) and StanfordNER (Spanish Model). When I used the non-Latin models I used the original Latin text as the translated one was not making any sense.

Fortunately Spacy Multilingual model managed to extract all Persons and Places of the sample documents but with additional words that I am not considering them as Entities. Moreover, the labels are incorrect.

Here is an example output:

{'LOC': ['Artali', 'Artalis', 'Bruges', 'Unde'],
'MISC': ['Marianum lu Tignusu'],
 'PER': ['Simone de Mazara',
  'Artalem de Alagona',
  'Apoca',
  'Coram',
  'Pero de Naso',
  'Pero Caruana',
  'Bartholomeo Xacara',
  'Testamur',
  'Artalis de Alagona',
  'Melite',
  'Simonis de Mazara',
  'Simonem',
  'Simone',
  'Mariano',
  'Artalis',
  'Artalem',
  'Simoni',
  'Panormi',
  'Renunciando']}

where the LOCATIONS should be: Panormi, Bruges, Melite and PERSONAL names should be all others except Unde, Apoca, Coram, Testamur, Renunciando which are neither locations nor personal names.

I was thinking of ignoring the labels and do some classification ML algorithm. The problem is that I do not have any training data available and the only possible usable corpus that I think it might be useful is Proiel treebank which labels proper nouns as NE. How would you go with such a problem?

One approach you can take is Multi-Task Learning. This approach is a little more complicated but tackles your problem in hand.

The idea is that you train a neural network to perform different NLP tasks. For example:

Translation
Part-of-speech tagging
Named entity recognition
Chunking

or any other task.

However, you might have data for translation (e.g. Spanish to Latin, or English to Latin), you have data for NER in Spanish and English but not for NER in Latin.

The theory and studies behind Multi-Task learning suggest that if you learn a task with that language, translation for example, you can get better at other tasks as well, NER for example.

So you can learn NER in Latin by learning NER in other languages and learning translation, chunking and POS tagging.

This should take care of your lack of NER data in Latin.


What is the best way to visualize data that is over 100% capacity I’m trying to visualize some data that is usually under 100% (for example, workload, or activity) but on some occasions it will rise over 100, meaning that the resource has been used too much.
I was thinking on something like the rings Apple use in their activity on Apple Watch, which spins several times before reaching the final value, indicating that you have overdo. The problem is that that type of visualization requires to be dynamic and I need something static (to print on a report)
I can also use a bar chart with a defined boundary if the value exceed 100% but I’m afraid it won’t clearly represent values in small and big numbers.
So is there any type of chart that I’m clearly missing?
Any help would be appreciated.

If you want to see small numbers and large numbers on the same chart, you can use a log() scale for the y axis. This will work for bar charts, or other types of charts. Log scaling will show more details than linear over a larger range. Just need to point this out to the user as they may be used to seeing linear scales. For something that changes over time, it can be helpful to show a time based plot, with any data over the limit in red and the data under in green or blue. Also you can add a horizontal line showing the cutoff value.
Example:
Linear vs Log

Flag with red points including a horizontal line to show the cutoff


Using pre-trained word2vec vs re-training a word2vec model I am relatively new to using word2vec. I am interested in solving the topic-word intrusion introduced here by using the vector spaces of words generated by word2vec and SVC.
I have a corpus with a vocabulary of 8000 words, the vocabulary is perfectly contained in Google's word2vec trained model. I was wondering which model would provide a better representation of the words, the pre-trained model on 3M words or a model trained only on the 8000 words appearing in my corpus?

As always, "is A better than B" always depends on what you consider better? Is it accuracy, is it speed etc.

The dumb but correct answer to your question is: "try both and see which one is better".

Performance of techniques is always to some extent dependent on the data. What you need to remember with word vectors, is that they are learned based on a certain context. If the context that was used by Google's model is similar to yours, you might be better off using their model. But if it's different, you might run into some problems.

Just imagine the following case. You have 4 words: King, Man, Queen, Woman. Which pairs of two words would you create? Depending on the context, you could make a case for several

King/Man and Queen/Woman because of gender
Man/Woman and King/Queen because of the use of the word etc.


LSTM fot text classification always returns the same results Hello fellow Data Scientists,
I'm trying to make a classifier that was to classify sequences of text into some predefined classes, but i always get the same output, can anyone help me understand why?
The training of the model:

   # The maximum number of words to be used. (most frequent)
MAX_NB_WORDS = 100
#2155
# Max number of words in each complaint.
MAX_SEQUENCE_LENGTH = 100
# This is fixed.
EMBEDDING_DIM = 20

cf.go_offline()
cf.set_config_file(offline=False, world_readable=True)

def treina(model_name):

    df = pd.read_csv("divididos.csv",sep='§',header=0)
    df.info()

    max_len = 0
    for value in df.Perguntas:
        if(len(value)>max_len):
            max_len = len(value)
    max_words = 0
    for value in df.Perguntas:
        word_count = len(value.split(" "))
        if(word_count>max_words):
            max_words = word_count

    tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
    tokenizer.fit_on_texts(df['Perguntas'].values)
    word_index = tokenizer.word_index
    X = tokenizer.texts_to_sequences(df['Perguntas'].values)
    X = pad_sequences(X, maxlen=MAX_SEQUENCE_LENGTH)
    Y = pd.get_dummies(df['Class']).values

    X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.05, random_state = 42)
    print(X_train.shape,Y_train.shape)
    print(X_test.shape,Y_test.shape)

    #Balance data
    sm = SMOTE(random_state=12)
    X_train, Y_train = sm.fit_sample(X_train, Y_train)
    print(X_train.shape,Y_train.shape)

    #LSTM net
    model = Sequential()
    model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))
    model.add(LSTM(20, dropout=0.2, recurrent_dropout=0.2,activation="relu",return_sequences=True))
    model.add(LSTM(10, dropout=0.2, recurrent_dropout=0.2,activation="relu"))
    model.add(Dropout(0.2))
    model.add(Dense(11, activation='softmax'))
    opt = adam(lr=0.3)
    model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

    epochs = 100
    batch_size = 20

    history = model.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size,validation_split=0.1)
    accr = model.evaluate(X_test,Y_test)
    print('Test set\n  Loss: {:0.3f}\n  Accuracy: {:0.3f}'.format(accr[0],accr[1]))
    model.save(model_name)
    return model

and the testing:

def corre(modelo):
    labels = ["a","b","c","d","e","f","g","h","i","j","k"]
    model = load_model(modelo)
    a = 0
    tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
    while (a==0):
        new_complaint = input()
        new_complaint = [new_complaint]
        seq = tokenizer.texts_to_sequences(new_complaint)
        padded = pad_sequences(seq, maxlen=MAX_SEQUENCE_LENGTH)
        pred = model.predict(padded)
        print(pred, labels[np.argmax(pred)])

Thank you for your time

You are not using the same Tokenizer in testing which you used in training, so texts_to_sequences is not outputting the required result.


Is shuffling training data beneficial for machine learning? I was curious to know if shuffling ML training data is beneficial to better results?

Sorry not a lot of wisdom here, but I have been reading a post from pythonprogramming.net for this topic.

I copied this function from the post and modified to just save my shuffled data to csv file.

def Randomizing():
    df2 = df.reindex(np.random.permutation(df.index))
    df2.to_csv('C:\\Users\\Machine-Learning-Electric-Data\\randomized.csv')

Randomizing()

What appears to happen is only the index gets shuffled and all other data stays the same. I have many columns in my pd dataframe where I would need to keep all rows the same. (randomly shuffle all rows, its time series data) If this is beneficial can someone give me a tip on how to randomly shuffle my data more than just the index?

Shuffling the training data is generally good practice during initial preprocessing steps.

When you do a normal train_test_split,  where you'll have a 75% / 25% split, your split may overlook class ordering in the original data set. For example, class labels that might resemble a data set similar to the iris data set would include target variables that resemble the following:

For example: [0, 0, 0, 1, 2, 2, 2, 3, 3, 3, 3, 3]

You could see from this example above, that splitting your data without shuffling might lead to very poor performance in your test set evaluation. Said another way, you may only capture the classes 0, 1, and 2 in your training data and only 3 will be represented in your test data. Specifically for classification tasks, but also for other ML tasks it may be useful to shuffle your data. However, each situation is different so the best idea would be to try it both ways to see whether you see a significant improvement or not.

Hope this answers your question. Drop a comment if you would like any further clarification.


Classification model challenge I want to implement ML to monitor log file, classify them as normal and abnormal. The model must learn via this process of classification and after time be able to classify the log files itself. This classification of the log files is based on time of login and place of login. Any hint / advice of ready made model for the scenario would be of great help.

If you want to use only time of login and place of login, there are several models that would do the work.
Take a look at Python's Sklearn. It offer lots of different options.
An example one is random forest: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Normally you don't event need to understand the model under the hood to be able to use it, so no need to feel intimidated by the names


Which observation to use when doing k-fold validation or boostrap? I have to perform predictive model over the dataset $D$ (with 1000 obs).
From $D$, I extract 700 obs for training $(T)$ and 300 obs for validation $( V )$.

I need to perform bootstrap or 10-fold cross validation sampling.

The question is which of these sets should I use?

Divide $D$ in 10 subsets and alternate training and validation between them ?
Divide $T$ (the training subset) in 10 subsets and perform training/validation on those subsets? $V$ is used only for final validation.

I recommend using the second option you presented.  I would use $T$ with 10-fold CV to select my modeling technique and optimal tuning parameters.  Take a look at what performed the best ("best" being the model that gives us the best error, but also doesn't have the error fluctuate too much from fold to fold).  After selecting a model, you can use the model on $V$ to get a realistic error rate.

The reason I don't recommend the first option is: There are varying degrees of over fitting that can occur when going through model selection and model tuning, then using that same data to get an error rate.  CV is a great way to limit this overfitting and it gives us a sense of performance variance which is great, but a classic hold-out validation set is the gold standard for model performance.  In your case the first option might not be wrong (depends a lot on data/techniques), but if a hold-out validation set is available I would go for that.


Distance of a people in an image What is the best approach to find the distance of a person/people from camera, using an image?

Updated: Measuring distance from camera to object

Perhaps you could search and find tons of materials. This is a well-established task, for example this tutorial, of more advanced methods combining object detection with distance measurements, see Object-Detection-and-Distance-Measurement, and references at the bottom. In a nutshell, when you know details about your camera focal length and the details about the object in the image, it is a simple formula to estimate the distance base on those details. I am guessing that you may need to do some calibration, but I haven't done this one myself, so I wouldn't know for sure.

Earlier I have had: Measuring distance between objects

One idea that might be worth exploring is to segment your image, for example use the recent package by Facebook Detectron2, that seems to be capable of segmenting people out-of-the-box. Once segmented (filtered only for humans), for each segmentation calculate the center of segment (Centroid) or something similar assigned to each person. And given that would be (x,y) coordinate, you can easily calculate distance from a person to all neighboring people.

For the center of segment , OpenCV is quite handy. See this post Find the Center of a Blob (Centroid)! Or this great tutorial to estimate distance between objects in an image with OpenCV!

In a nutshell: Image Segmentation -> Centroid -> Distance


Best method for similarity searching on 10,000 data points with 8,000 features each in Python? As mentioned in the title I am attempting to search through 10,000 vectors with 8000 features each, all in Python. Currently I have the vectors saved in their own directories as pickled numpy arrays. The features were pulled from this deep neural network.

I am new to this space, but I have heard about M-Trees, R-Trees, inverted tables, and hashing. Is any one of them better for such a large number of features?

This implementation needs to be done quite rapidly and is just a prototype, so simplicity is valuable.

Thank you for your help.

There are two main paths:

Load all vectors into memory. If you are able to load vectors into memory, then you might be able to search the space with "clever" brute force. One such method is found in this paper.
Keep vectors on disk. If you follow this path, then you have to index the vectors. You are basically building a search engine. Common open source search engines are: Apache Solr and Elasticsearch

It also depends on what kind of what kind of search is needed. A working definition of "close" vector is needed. The most common definitions of "close" vectors can be found here.

Fais is a new library for efficient similarity search of vectors. It is designed for many vectors (> 1,000,000) each being relatively small (10s to 100s  of dimensions). It may or may not scale to your problem. If it does not scale because of high dimensionality, you can reduce dimensionality with Principal component analysis (PCA) or t-SNE.


Forecasting energy consumption with no historical data when there is a trend I want to forecast new customers' energy consumption. Let's say I can construct a set of attributes to describe new and existing customers (e.g. size of business, type of business etc.) and I have the time series data on energy consumption of existing customers. However, I have no historical data on new customers' consumption.

If existing customers' consumption has seasonality but no trend, I could use the set of attributes to create a regression model to predict new customers' consumption based on existing customers' consumption. But what if existing customers' consumption has a trend? How do I know at which point of the timeseries I should start the forecasting?

Let's take the following example time series of an existing customer:

If I want to forecast new customers' consumption for one year, how do I know at which point of this time series to start the forecasting?

I do not have a dataset yet, so I cannot provide data examples. The answers could include an idea on what types of additional information I would need (if any) to solve this problem. Enrgy consumption is just an example, my question is theoretical and could be generalized to any other field. Also, names of python libraries to deal with this problem would be welcomed!

Thank you for the help.

Most time series models assume stationarity. Thus, you should account for trend and seasonality in your data and subtract the corresponding components from your original data. After that, you can model your normalized data also with arbitrary machine learning models; and forecast with new data the following months.

Carefully note that you may have features that explain the trend or seasonality component. If these components are also changing or expecting to change over time you could combine (e.g. add for additive time series models) predictions for all components. I.e. the sum of the trend, the seasonal and the residual prediction.

Importance of Normalization
E.g. https://stats.stackexchange.com/q/31387/214243

Detrending
Subtract the %yoy change for each month.

Seasonal decomposition
E.g.  http://www.statsmodels.org/stable/release/version0.6.html?highlight=seasonal#seasonal-decomposition

Forecasting
There is an abundance of statistical time series models (AR, ARIMA, GARCH,...). But when data is normalized common machine learning models like decision tree or random forecast may also be applicable (see sklearn library for this)


How to do feature analyzing : pandas groupby(). mean I'm analyzing Titanic data from Kaggle1 with a kind of guiding book. In the book, feature analyzing about the relationship between Pclass data and Survived data is done like below.

train_set = pd.read_csv('train_csv)
fig = plt.figure(figsize=(12,4))
ax1 = fig.add_subplot(121)
PclassPlot = train_set['Survived].groupby(train_set['Pclass]).mean()
ax.bar(x=PClassPlot.index, height=PClassPlot.values)

Why do you need to use mean data?
I thought I didn't need to culcalate mean to see the relationship between Pclass and Survived data.

Please give me tips for analyzing data.
Thank you for your help.

Means are calculated just to have an idea of the relationship. For more definite analysis, one can use simple statistics to know relationship between two variables. The test to be applied depends on type of variables:

If both are numeric: Correlation (Pearson or Spearman)

If both are grouping (nominal) variables: Chi-square test
   Fisher exact test can also be used if only 2 groups in each variable.

If one is grouping and other numeric:
   if only 2 groups: Student t-test or Mann-Whitney U test
   if more than 2 groups: ANOVA or Kruskal–Wallis test

Means of different groups will give you only partial information about relationship. However, it may be sufficient to know if the feature is likely to be important for prediction. As a method of feature selection, if groups' mean do not differ significantly, this feature may be dropped from analysis.


Can the output of convolution on image be higher than 255? I have probably a very simple question. When I convolve an (grayscale) image using some kernel I get some output. The original pixel can be only between 0 and 255. Is possible that the output of a convolution can be higher? Because we are creating feature map, which I understand is another image. Is the output only up to 255, do we scale it down or it does not matter?

Thanks.

It is possible.  You may have values less than 0 or greater than 255.  It will depend on the values in the kernel.  If you want to display the convolution output properly you will need to scale it first.  This process is referred to as 'normalization".


Application of Deep Reinforcement Learning I'm new to deep learning, and especially to reinforcement learning. I would like to know if it's possible to predict which combination of hashtags (from a subset of chosen hashtags) would produce the most likes for a certain image.

Is it possible to have a convolutional neural network with each hashtag as a label, and take something like reward = likes / followers as a reward in a reinforcement learning like scenario?

In what other way could I face this problem? My goal isn't to predict the amount of likes, but to maximize the probability to get the most likes.

I chose this title because I think the answer could actually be question agnostic: I could use the same knowledge to define which combination of stocks would maximize my investment.

Thanks in advance.

This honestly sounds more like a supervised learning problem.
For reinforcement learning to work, you would need a model that can constantly return values for a given input.

With social media, that would mean

a) posting the same image with all kinds of different hashtags and expecting people to give the most appropiate hashtags likes in a short amount of time. This will not happen.
b) searching for all occurrences of an image with different hashtags. This is basically supervised learning.

I recommend finding a social media dataset first. Try to group all occurrences of an image and find the median of likes for each hashtag. Don't forget to compensate for follower count, as more popular posters will get more likes on average regardless of hashtag. Store the best hashtag as label for the given image.

You now have Y different hashtags for X different images.

From now, you can treat it as classification problem. X is your sample count, Y is the amount of possible outputs. If you don't want to write your own loss function and only want to predict the single best hashtag, use cross entropy as loss function.

Of course you still have to choose your social media dataset, appropriate images for training and NN structure, but I hope this helps as a general approach.


Sigmoid vs Relu function in Convnets The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.

The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives.

Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between 0 and 1. The maximum of the $\sigma'$ derivative function is equal to $\frac{1}{4}$. Therefore when we have multiple stacked sigmoid layers, by the backprop derivative rules we get multiple multiplications of $\sigma'$. And as we stack more and more layers the maximum gradient decreases exponentially. This is commonly known as the vanishing gradient problem. The opposite problem is when the gradient is greater than 1, in which case the gradients explode toward infinity (exploding gradient problem).

Now let's check out the ReLU activation function which is defined as: $$R(x) = max(0,x)$$ The graph of which looks like

If you look at the derivatives of the function (slopes on the graph), the gradient is either 1 or 0. In this case we do not have the vanishing gradient problem or the exploding problem. And since the general trend in neural networks has been deeper and deeper architectures ReLU became the choice of activation.

Hope this helps.


Multiple Regression, Classification and Boundary Poins I have two gangs which are doing crimes. And i want to classify them.
Lets say I'm looking for a regression function:
 M(x1, x2) = w1x1 + w2x2 + w3

Now I have found all three parameters w1, w2, w3.
Now I want to do classification. I get some boundary points which look like a line and they separate two classes from each other. Should I do another regression over that boundary points so that i have a exact line for my separation?
Because lets say i want Point(5,3). I want to know if its more likely that the crime is done from Gang A or B. But I have just some boundary points to separate. Should I use them for a regression?

I think you want a clustering algorithm rather than regression. You will have a decision boundary between clusters of data points which will determine whether that particular point e.g. 5,3 belongs to group (cluster) A or group (cluster) B
Fit your clustering model in x1 vs x2 feature space. Take the image below, we have x1 and x2 as the x&y axis. And then the black lines are the decision boundaries, essentially the model.

You can of course just cluster between 2 groups also, as in your case.
You can checkout different clustering algorithms here


Which algorithm to apply for choosing the right point Picture below shows 7 points around the origin. One of them has been selected by a human based on rules and experience and is colored red (the one in the lower left quadrant).

Now we have over 1000 of these sets of points and for each set a human has selected a single point. These conditions apply to all sets:

Each set has about 3 - 10 points
There are no outliers
Points can have positive and negative values
No mistakes were made when selecting a point

My question is: Does there exists a machine learning algorithm to learn from these sets and human made selections so it can automatically decide which point to select when a new set of points is given? This new set satisfies the first 3 conditions from above of course.

2 final remarks:

The example I gave is just a randomly constructed example by me to support the idea about points in a plane around the origin together with a selected one. In real life there might be more structure but for now I am curious and would like to know what is possible for this case.
Would variations be possible? Say it is about 2 selected points or you have circles with a given radius instead of points.

This is a fascinating problem!
Two things make it especially challenging:

How should we compare two point sets? Classical problems in Machine Learning have a fixed number of attributes, and these attributes are not interchangeable: For example, I might have data on different persons with attributes age and height (in centimeters). Every sample has one entry for each, and of course (age, height) = (22, 180) is not the same as (age, height) = (180, 22).
Neither is true in your problem. A point set has between 3 and 10 points, and the order in which we enter the points should not make a difference when comparing two point sets.
How do we make a prediction? Say we have found a way to pick point sets from our training set that are similar to your point set above. We face the problem that our prediction must be one of the 7 points in your picture; but none of these points might be contained in the similar point sets.

Let me outline an algorithm that deals with both challenges. The prediction accuracy is not very good; but maybe you see a way how it can be improved. And at least it predicts something, right?

1. Simulating samples

To be able to test the algorithm, I wrote functions that generate samples and labels.

Generating samples:
Each sample contains between 3 and 10 points. The number of points is random, drawn from a uniform distribution. Each point is of the form (x_coordinate, y_coordinate). The coordinates are again random, drawn from a normal distribution.

import numpy as np
from random import randint

def create_samples(number_samples, min_points, max_points):

    def create_single_sample(min_points, max_points):
        n = randint(min_points, max_points)
        return np.array([np.random.normal(size=2) for _ in range(n)])

    return np.array([create_single_sample(min_points, max_points) for _ in range(number_samples)])

Generating labels: As a toy example, let us assume that the rule for choosing a point is: Always pick the point that is closest to (0, 0), where 'closest' should be understood in terms of the Euclidean norm.

def decision_function_minnorm(sample):
    norms = np.apply_along_axis(np.linalg.norm, axis=1, arr=sample)
    return sample[norms.argmin()]

def create_labels(samples, decision_function):
    return np.array([decision_function(sample) for sample in samples])

We can now create our train and test sets:

n_train, n_test = 1000, 100
dec_fun = decision_function_minnorm

X_train = create_samples(number_samples=n_train, min_points=3, max_points=10)
X_test = create_samples(number_samples=n_test, min_points=3, max_points=10)
y_train = create_labels(X_train, dec_fun)
y_test = create_labels(X_test, dec_fun)

2. Comparing point sets via Hausdorff distance

Let us tackle the first problem: How should we compare different point sets?
The number of points in the point sets is different.
Also remember that the order in which we write down the points should not matter: Comparing to the point set [(0,0), (1,1), (2,2)] should yield the same result as comparing to the point set [(2,2), (0,0), (1,1)].
My approach is to compare point sets via their Hausdorff distance:

def hausdorff(A, B):

    def dist_point_to_set(x, A):
        return min(np.linalg.norm(x - a) for a in A)

    def dist_set_to_set(A, B):
        return max(dist_point_set(a, B) for a in A)

    return max(dist_set_to_set(A, B), dist_set_to_set(B, A))

3. Predicting via k-nearest neighbors and averaging

We now have a notion of distance between point sets.
This makes it possible to use k-nearest neighbors classification:
Given a test point set, we find the k point sets in our training sample that have the smallest Hausdorff distance relative to the test point set, and obtain their labels.
Now comes the second problem: How do we turn these k labels into a prediction for the test point set? I took the simplest approach: average the labels and predict the point in the test point set that is closest to the average.

def predict(x, num_neighbors):
    # Find num_neighbors closest points in X_train.
    distances_to_train = np.array([hausdorff(x, x_train) for x_train in X_train])
    neighbors_idx = np.argpartition(distances_to_train, -num_neighbors)[-num_neighbors:]

    # Get labels of the neighbors and calculate the average.
    targets_neighbors = y_train[neighbors_idx]
    targets_mean = sum(targets_neighbors) / num_neighbors

    # Find point in x that is closest to targets_mean and use it as prediction.
    distances_to_mean = np.array([np.linalg.norm(p - targets_mean) for p in x])
    closest_point = x[distances_to_mean.argmin()]

    return closest_point

4. Testing

Everything is in place to test the performance of our algorithm.

num_neighbors = 70
successes = 0
for i, x in enumerate(X_test):
    print('%d/%d' % (i+1, n_test))
    prediction = predict(x, num_neighbors)
    successes += np.array_equal(prediction, y_test[i])

For the given decision function and num_neighbors = 70, we get a prediction accuracy of 84%.
This is not terribly good, and it is of course specific to our decision function, which seems fairly easy to predict.

To see this, define a different decision function:

decision_function_maxaverage(sample):
    avgs = (sample[:, 0] + sample[:, 1]) / 2
    return sample[norms.argmin()]

Using this function via dec_fun = decision_function_maxaverage brings down prediction accuracy to 45%.
This shows how important it is to think about the decision rules that generate your labels. If you have an idea why people choose certain points, this will help you find the best algorithm.

Some ways to improve this algorithm: (1) Use a different distance function instead of Hausdorff distance, (2) use something more sophisticated than k-nearest neighbors, (3) improve how the selected training labels are turned into a prediction.


How to optimize cohort sizes to reduce pair-wise comparisons? I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. So this is a total of 57,000 choose 2 pairwise combinations. (This is a tiny example I know, but I have other databases with the same problem that are much larger.)

Based on other analysis, I concluded that I do not want to consider people with birthdates of more than three years apart. So I can subset the database into overlapping cohorts, and then only do all pair-wise comparisons within each cohort.

It is easy to show examples of where this cohort approach will reduce the number of pairs I need to compare. Here is one example with my data just based on the quintiles of the year of birth (and those with missing birthdays go into all cohorts).

(Min,1968]  : 13,453
[1962,1980] : 17,335
[1974,1988] : 21,188
[1982,1993] : 21,993
[1987,Max)  : 17,449

Which saves me around 0.7 billion comparisons.

So this brings up two questions:

are the choosing the bins based on quantiles a good strategy, or is there anther strategy that works better?
how many bins should I make?

There is a better way if I understand your question correctly. Here is the algorithm I propose:

Initialize a 'window' list and a 'pairs' list
Sort your data on birthday from old to young (or the other way around)
Loop over your rows and keep track of all the rows that are still in the last three years since your current row. When you get to a new row, throw out rows that are now more than three years apart, add the current row together with the rows in your 'window' to your pairs set and add current row to your 'window'.

This means you only iterate over your main list once, and if you implement your 'window' list properly (like a linked list for example) you don't need to do a lot of looping in that regard either. You also get all the pairs only once as a by product. Plus you actually get all the pairs, while with your binning approach you get missing pairs around the bin edges (if you don't overlap).

If you use the overlapping binning approach I think you should have bins of 6 years width and the borders shift 3 years each time, that way all true pairs are in at least 1 bin together, and there is the least amount of unwanted pairs.


Visualizing N-way frequency table as a Decision Tree in R I have a N-way frequency table generated from a regression model fit. This is a reproducible example of such table:

data("CO2")
lm.fit = lm(uptake ~  Type + Treatment, data = CO2)
lm.fit$coefficients

test = count(CO2, c('Type','Treatment'))
test$res = predict(lm.fit, newdata = test)
test$freq = NULL

I am trying to visualize test as a decision tree with nodes as Type and Treatment and res as leaves. I would interpret it as the path the regression model takes, leading to the final value for a particular segment.

I am not able to generate a tree with test. I am also open to other novel ways of visualizing these results. My original problem has many categorical variables, so I am looking for a customizable visualization, something from party::ctree or rattle::fancyRpartPlot.

You could try

library(data.tree)
test$pathString <- with(test, paste("lm", Type, Treatment, round(res, 2), sep="/"))
(tree <- as.Node(test))
#             levelName
# 1  lm
# 2   ¦--Quebec
# 3   ¦   ¦--nonchilled
# 4   ¦   ¦   °--36.97
# 5   ¦   °--chilled
# 6   ¦       °--30.11
# 7   °--Mississippi
# 8       ¦--nonchilled
# 9       ¦   °--24.31
# 10      °--chilled
# 11          °--17.45
plot(tree)

The plot uses the DiagrammeR package.


Create Custom Python Model in Azure ML Studio? I came across this article on how to create a Custom R Model and use it in Azure ML studio

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/create-r-model

Can we create a Python model instead ? While I have seen articles of python scripts for evaluation or data processing , I haven't not found enough documents on creating / using custom model built using Python ?

All I have to do is , expose my custom model as web / rest service

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/python-language-modules This article describes the modules provided in Azure Machine Learning Studio that support running custom Python code in a machine learning experiment.


Loading and querying a Spark machine learning model outside of Spark I'm currently working on a project where we're building a data pipeline. We have spark setup and have generated models. Sadly, loading the model in Spark and querying it isn't fast enough for us. What's the most straightforward way of exporting a model and loading it into memory on a local server? I've researched PMML and some libraries and that appears to be one path.

If PMML supports the model you're trying to express, it's probably the best option. Spark has some partial support for exporting as PMML. Look into JPMML + OpenScoring to score the PMML in an embedded instance.

There are loads of tools in this space that implement something like this, but they're typically entwined with their own formats or model management system.


Word2vec using gensim I am using gensim library to find most similar words to some words that I have. Using 10000 data samples (short text mainly 1-2 sentences) to train, I get really bad results! Why this is so? Also by repeating train and test it get different results. Should not it be deterministic?

model = gensim.models.Word2Vec(senforword2vec)
print(model.most_similar(positive=['bad'], topn=1))

  [('amount', 0.9987443089485168)]

In another train and

  [('yet', 0.9984885454177856)]

It seems that you are training on a very small dataset. A dataset of size 10000 samples will bring out less than 5k words after preprocessing, which I think isn't enough for your model to learn the relations. Try increasing your dataset and then train. Also, you can tune your parameters while training. Try changing window size (default is 5), embedding size and min count. Generally, 100 is a good embedding size, but you can tweak it and see what gives you better results.

You can try using already trained word2vec model released by Google. It's 1.5GB and includes word vectors for a vocabulary of 3 million words and phrases that they trained on roughly 100 billion words from a Google News dataset.


Identity covariance matrix, decorrelated data? Why would you want to decorrelated data?

As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is?

Is it to remove interaction between variables, thus allow simpler models to express interactions without having to compute x1*x2?

Having only taken a few courses on this stuff, I'm going to offer my understanding on this.

The main reason seems to be simplifying, as you assume. It manifests in different areas though. One is mathematical - if your co-variance matrix is identity matrix then the math becomes much easier. The one directly following from this is computational - inverting a matrix is very costly, but identity matrix is its own inverse, so that's perfect.

Another part of the simplicity is definitely the ease of interpretation. Independent variables are easy to explain to a non-technical person, but as soon as you get into covariances, that goes out the window.


How to find the datasets for skill test (like java, python, c c++, etc.) I want to extract the skills from a resume using Natural languge processing.

To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume?

a) You may try to find an anonymized resume dataset in the openbase of Kaggle datasets

b) You may do some web scraping on professional social networks like LinkedIn (take the description of a profile as resume, and the LinkedIn skills as supervised training set) to build your own sample database (do not forget to anonymize it).

Hope this helps!


Multi class Imbalanced datasets under-sampling imblearn I have an imbalanced dataset. I am looking to under-sample. Even though, the oversampling process takes less time, the model training takes a lot of time. I have taken a look at imbalanced-learn website. There are several under sampling methods. I am looking at method that tries to undersample the classes with much as possible information intact. I tried .ClusterCentroids() methods and found it takes way too long to balance the classes.

I have tried other methods that have been mentioned in the website. However, even with sampling_strategy to equal values eg: sampling_strategy={0: 2000, 1: 2000, 2: 2000} The resulting dataset is not balanced. Such as in .CondensedNearestNeighbour() and .AllKNN() methods.
Would anyone be able to help me create a class balanced dataset using these methods.

Thanks

Michael

If you're looking for a fast workaround to solve this you have to increase n_neighbors parameter in AllKNN, but I wouldn't recommend using this type of undersampling algorithm for what you want to do!

Explanation:

AllKN is an under-sampling technique based on Edited Nearest Neighbors. These techniques try to under-sample your majority classes by removing samples that are close to the minority class, in order to make your classes more separable. The way they work is that they remove samples from the majority class that have at least 1 nearest neighbor in the minority class. The thing is that if the classes are separable enough and the majority samples have no minority nearest neighbors, they can't be removed!

If you want a technique that undersamples your data in order to get exactly the same number of samples from the minority and the majority class, I'd recommend using a different technique (e.g. ClusterCentroids, which you've used is such). ENN-based undersampling techniqes aren't built for that. You can also read this tutorial which compares different resampling algorithms imblearn.

As a final remark, if possible, I'd recommend oversampling...


Should we identify outliers from population prior to taking sample? I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population.

While ensuring that sample is a decent representative of the population, I am left with a question.

Should we care about identification and rectification of outliers prior to taking sample from population ?

Here is my work from where i came up with this question

It depends on how you want to treat outliers. Outliers can either be generated by chance from the data's distribution or can be a result of human error (error in measurement, error in data entry, etc.).

If you consider the outlier to be from the first category, in my opinion you shouldn't discard it, in order to take your samples from the true distribution of the data.
Outliers in this category, might indicate a heavily-skewed distribution, or might simply appear due to the nature of the problem (e.g. King's effect).
If you consider it to be an error (e.g. a human height of 3.75m), you should discard it as it corrupts the distribution of the data.

In order to determine which category the outliers belongs to, either requires domain knowledge (e.g. knowing that human height can't exceed 3m) or requires you to make assumptions on the data.


Regularization in simple math explained I read a lot of articles online about how regularization works and most of them just show the equations with regularization terms but did not use example numbers to explain how the coefficient values change as lambda increase.

For example:
L1 Regularization

theory states that When Lambda is large, coefficients tend to 0.

X  =

How is the coefficients(beta i assume) value changing when lambda changes?
Is it because X a fixed value?

Example, does beta have to decrease when lambda increase to ensure that X is the same?

The function you have described is a loss function. It is the function which we want to minimize in order to train our model. The loss function is called ordinary least squares. We can also see that the model that we are trying to fit is a linear function with model parameters $\beta$. This is an optimization function and regularizarion is often used in optimization problems to attain a solution which is less likely to be a result of overfitting.

No regularization

Without any regularization the solution to a linear regression is

$\beta=(X^TX)^{-1}X^TY$

L2 regularization

To understand regression it is much easier to first start with the more widely used L2 regularization, ridge regression. This is defined as

$\min_\beta (Y-X\beta)^T(Y-X\beta) + \lambda ||\beta||^2$

We can solve this to get a closed-form solution using a derivative

$0 = -2X^TY + 2X^TX\beta+2\lambda\beta$

$X^TY = (X^TX + \lambda I)\beta$

$\beta = (X^TX + \lambda I)^{-1}X^TY$

Let's analyze this result in contrast to the solution without regularization and see what it means. We can see that the only difference is the added regularization term $\lambda$ within the inverse. As we should know, the inverse of a larger value, will cause its result to get smaller. For example the inverse of 1/5, is larger than 1/2. So by adding the regularization term we actually reduced the associated weight $\beta$.

This is why regularizers are often called a penalty term. They further constrain the optimization function. By adding an additional cost associated to each weight, we can reduce the impact that each weight will have on our model. Thus reducing the potential for overfitting.

L1 regularization

L1 regularization is the penalty used for LASSO regression, it's cost function is defined as

$\min_\beta(Y-X\beta)^T(Y-X\beta) + \lambda||\beta||$

From what we learned above we can already tell that this additional cost will cause the resulting weights to be penalized. Unfortunately L1 regularization does not have a closed form solution because it is not differentiable when a weight $\beta$ falls to 0. Thus this requires some more work to solve. LASSO is an algorithm for finding the solution.

First let's remember that our constraint here is $||\beta||\leq c$ where c is a real number which is inversely related to $\lambda$. The plan is to find an appropriate $\lambda$ such that the minimum of the function falls within or on the contour of the constraint. This constraint has very sharp edges which lie on each dimensional axis at a distance $c$ from the origin. You can imagine it like a diamond in 2D space, an octohedron in 3D space, etc.

In high dimensional space these spikes have a very high likelyhood to being hit by the function you wish to optimize thus this will cause many of the features to have an associated weight of 0. For example if we compare the regularization of this line in 2D space using both L2 and L1 regularization.

In this picture the distance $r$ is equivalent to the factor $c$ I used above which is inversely related to the regularization term $\lambda$. The function being optimized (the cost function) is the described using the ellipses which represent their levels (like an altitude map). The shape at the center of the axis is the regularizer which is used to constrain the weights associated to each feature.

We can first notice that the original function is minimized exactly at its minimum. However, choosing this solution has a high probability of being one which overfits the training data.

We see that L2 regularization did add a penalty to the weights, we ended up with a constrained weight set. However both weights are still represented in your final solution. The function being optimized touches the surface of the regularizer in the first quadrant.

However, in the L1 case we can see that $\theta_2$ is the only feature that is required for this regression problem and $\theta_1$ will have a weight set to 0. Thus we can ignore it. It just happens that smooth functions encountering a function with pronounced edges tends to hit these edges with higher probability than hit the vertices. I do not know the proof for this, but I have seen it in a course as supplemental material and it was complicated. This is why L1 regularization is often used for feature selection.

Combining both together

What is often done is first using L1 regularization to find out what features have lasso weights which tend to 0, these are then removed from the original feature set. Then with this new feature set we can apply L2 regularization in order to still control these new features to control for overfitting.


Create a measure that converts minutes into hours My searching is weak so I didn't find anything that satisfied my need.

I have a dataset of work items. I have a lead time column present on each record. It is in minutes, and I want to create a measure that displays lead time in hours. I want that measure to be (in pseudo code):

lead time (hrs) = lead time / 60

I'm a Tableau user and am finding this difficult to do in Power BI. What is your advice?

Ok, I got an answer from the powerbi community. Here it is:

lead time (hrs) = SELECTEDVALUE(Dataset[lead time])/ 60

I hope this helps someone.


One-class classification in Keras using Autoencoders? I am trying to develop a model based on one-class classification approach. For example, the model should to identify if a given picture contains a cat or not. Keep in mind that my training dataset only contains pictures of cats and nothing else. Sort of like a anomaly detection problem

I have seen some suggestions on using autoencoders for unary classification but I couldn't find a concrete example on the net? Can someone point to an implementation of this approach using Keras. Is there any other way to about solving this problem?

If you have images of cats only, you could create boundary boxes (BB) of your images. Some BB will have cats an others won't. You will label those BB with cats inside as 1 as the others as 0.
This way you can set up a dataset with a binary class. It will be much easier if you already have boundary boxes for the cats in each image since this way it will be possible to generate the dataset automatically (generate random BB and label them as 1 if the intersection with the cat is big enough and 0 otherwise).
There is some software that helps to create boundary boxes.


Doc2Vec network architecture I have been familiarizing myself with Word2Vec and Doc2Vec. After reading multiple papers including the the ones by T Mikolov (the creator of Doc2Vec), I am not clear on how does the neural network for Doc2Vec looks like.

I get Word2Vec: a neural network with 3 layers (1 hidden). The input layer has all words in the vocab. The hidden layer is the same size as that of embedding, while the output layer, again the same size as vocab. This is, of course, the CBOW framework.

How does Doc2Vec change things?

Nothing just the starting randomly initialized paragraph vector is concatnated/ averaged with the randomly initialized word vector. Everything else pretty much works the same.


Problem with two layers: how to predict? Let me get straigt into it:

Think of an scheduled operation. There is a surgeon and, say, three other medical personel that carry out various tasks. For instance, a nurse, an anesthesiologist, and a backup.

Each one reports to duty at a certain time. However, with some probability p (to be predicted) one or many member(s) of the team won't show up for work. If an individual does not show up for work, but his particular skill set is still covered due to the backup, then the operation is done. Otherwise, it is cancelled. For instance, if the backup is a certified anesthesiologist, absence of the scheduled anesthesiologist does not prevent the surgery from taking place since a minimum required skill mix is given. If, however, the backup is a nurse, and the anesthesiologist does not show up, then the surgery has to be cancelled.

This problem involves two layers:

Does an individual team member show up for work?
Is an individual team member's skill critical for the execution of the surgery.

My goal is to predict cancellation of the surgery, which involves step 1 and step 2 simulataneously (alternatively, you could think of step 2 being conditional on step 1).

I have a good understanding of how to use standard machine learning algorithms to predict 1. But how do I generate final predictions taking both layers into account? The problem set up seems reasonably general so that I would assume there exists a "standard" method?

This problem is best addressed with discrete-event, stochastic simulation modeling.  The prediction you get from such a model is a probability distribution on the surgery being canceled.

It's easy to code up such a model given your problem statement.  You need:

A boolean function for each kind of surgery, which, given the cast of characters and skill sets as input, determines True/False if the surgery can proceed.
A probability distribution for the appearance of each kind of character, when scheduled.
An object which is initialized with the schedule.

The idea is to do this check (surgery/surgery canceled) many times, thereby growing a set of outcomes for each policy you wish to explore.

If you don't want to roll your own, there are packages for many languages such as SimPy for Python and Simmer for R.

The nice thing about discrete event simulation is that you don't need to make any simplifying assumptions about probabilities, and the business logic can be arbitrarily complex.  For example, if the surgeon and one of the nurses are having an affair, the chance that one cancels on the same day as the other can be increased.


Transform single-label data set into multi-label data set I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in.

When the dataset was put together, it was assembled under the assumption that each string of text can only belong to one group. In actuality, the text can belong to more than one group simultaneously.

Instead of going back to the drawing board and manually labeling the data again, I want to try and convert this single-label data set into a multi-label data set.

I've tried one method with questionable results. I built a linear regression that predicts each category individually, and appended those predictions to the original data. While this gave me data in the structure I needed, it yielded lackluster results. Most strings of text still only belong to one category (many should belong to multiple), and a good portion weren't assigned any label at all.

It seems that even if I can "Frankenstein" this data together, it may not serve as quality training data. I'm curious, is there's any great way of transforming this single-label data into multi-label data?

You should consider using a neural network for this. By using binary crossentropy across multiple categories, you can get a probability "rating" for each category and how it applies to the text. From there, you can develop a script that establishes a threshold (say 0.8) and then creates a new entry of labeled data for that one particular piece of text across multiple categories. There are many examples out there of people taking IMDB data and the movie descriptions and assigning multiple genres to a single movie (like a "horror" movie can also be a "suspense" movie or a "comedy" movie can also be an "animated" movie). Those types of examples should fit what you need here.


Accuracy decrease in production after adding additional input datas I am trying to predict TimeSeriesA by using a CNN. I create snapshot images of the timeseries and these are then labelled.

With a very simple snapshot I get reasonable training and test accuracy. When I apply the model to real world in production I also get reasonable accuracy.

Inorder to improve the accuracy, I added other timeseries to the snapshots that may or may not add value.

Both my training and testing accuracy increased (Training much more so). However, my production accuracy has gone down greatly.

Why might this happen? The original data is still in the snapshot in exactly the same format. Can a CNN be confused (wrong word!) by the additional data?

Here is a look at the stationarity of TimeSeriesA:

Year: 2000 Ave: -0.0003 Std: 0.0076 Skew: 0.2166
Year: 2001 Ave: -0.0002 Std: 0.0072 Skew: 0.0158
Year: 2002 Ave: 0.0006 Std: 0.0056 Skew: -0.2445
Year: 2003 Ave: 0.0007 Std: 0.0065 Skew: -0.0402
Year: 2004 Ave: 0.0003 Std: 0.0067 Skew: -0.2640
Year: 2005 Ave: -0.0005 Std: 0.0056 Skew: 0.2420
Year: 2006 Ave: 0.0004 Std: 0.0047 Skew: 0.2711
Year: 2007 Ave: 0.0004 Std: 0.0039 Skew: -0.3177
Year: 2008 Ave: -0.0001 Std: 0.0087 Skew: 0.3768
Year: 2009 Ave: 0.0001 Std: 0.0076 Skew: 0.2327
Year: 2010 Ave: -0.0002 Std: 0.0074 Skew: 0.0112
Year: 2011 Ave: -0.0001 Std: 0.0074 Skew: -0.2599
Year: 2012 Ave: 0.0001 Std: 0.0051 Skew: 0.2541
Year: 2013 Ave: 0.0002 Std: 0.0046 Skew: 0.0818
Year: 2014 Ave: -0.0005 Std: 0.0039 Skew: -0.1489
Year: 2015 Ave: -0.0004 Std: 0.0076 Skew: 0.2973
Year: 2016 Ave: -0.0001 Std: 0.0051 Skew: 0.0076
Year: 2017 Ave: 0.0005 Std: 0.0045 Skew: 0.3101
Year: 2018 Ave: -0.0002 Std: 0.0045 Skew: -0.1658
Year: 2019 Ave: -0.0002 Std: 0.0033 Skew: -0.1124

I train the model with data up to end of 2010. I get a training accuracy and validation accuracy of around 65%.

When I then appy the model to data from 2011 to 2019, I get a drop-off in accuracy. The green years are those included in the training.

Obviously, without knowing the exact data, everything I will say is mere speculations. Here are some of my guesses why this might happen:

Your production data has a different distribution from your training data

This can happen, data analysis would be required to double check how your production data might deviate. If this is the case, I would investigate what might cause this, is it something you can adapt to? Or is it due to an external factor you can measure?

Your network is too large and overfitted artifacts in training

If your network is large, you might have overfitted on certain artifacts that are present in your train and test set and therefore, it performs badly on your production data. I would train smaller networks or on many smaller snapshots to double check that your model can learn appropriately.

Your evaluation method is not appropriate

When training time series, make sure to always evaluate on future data and not past (that's cheating!)

Bugs

Obviously, bugs may always be the cause of some issues. Maybe preprocessing is doing something funky.


Is it better to have binary features rather than class ones I have a dataset containing several features, which have class values (e.g. DBF4, JUL23, ...). In a classification problem and when using decision trees, is it better to convert these values as new binary features: so DBF4 will become a feature, which value is either 0 or 1, or is it better to keep them as they are. Knowing that there are a lot of values (more than the number or rows actually). If this is the case, is there a reason why is it better ?

In general, it’s better NOT to binarize them if the decision tree algorithm that you are using supports categorical features, as variables with fewer levels are less likely to be used in splits. Note however that not all implementations in all software are able to use categorical features without one-hot encoding – for example, most R implementations of decision trees and tree ensembles support categorical features natively, whereas scikit-learn’s and spark’s do not.

Here’s a blog post with a comparison of categorical as-is vs. one-hot encoded for random forests:
https://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/


How to do scatterplot Visualization of Text Classification I am working on my first ever ds problem on classification. I have got end to end process but not much visualization except bar charts and pie charts.

I want to see how well classes are distinguished in sample dataset itself to find anomaly in sample data or to see how many outliers sample have.

I see many nice scatterplot clustering done in many articles, pappers and books which are mostly based on numerical features. Something like following taken from scikit learn : http://scikit-learn.org/stable/auto_examples/datasets/plot_random_dataset.html

However, most of those are for numeric features. How can I draw similar plot for textual features?

One way I can do it is converting all features into vectors; plot all the indexes(token indexes) on x-axis and their respective metric (TF or TF-IDF) on y-axis? But then there are so many features!?

Also, how can I analyze raw sample data in such clustering format without transforming them into vectors. I can do tokenization and some basic normalization.

For example  my sample data is
text, category

"chiense restarant near me", chinesefoodlover
"japanese food near me", japanesefoodlover

I can break it down to

[chiense, restarant, near, me], chinesefoodlover
[japanese, food, near, me], japanesefoodlover

But then how to draw clusters of classes from there.

You should look at t-SNE for visualising high dimensional datasets such as words : http://lvdmaaten.github.io/tsne/


Running out of memory while processing csv file data I'm getting an error while processing 0.2 million of text data. I'm using CNN text classification in tensorflow.
Output raw data shape is 204177x22000. while passing to numpy.array(out_raw), here it is consuming 100% memory(Using 8GB RAM).
Tried with data in batch but didn't work.
If i need to increase my RAM size then kindly help me out with formula.

What are the methods to take care of this problem statement?

Like @Sagar said, you could convert your pipeline to pyspark (So Spark with python API), and you can set your memory usage to not go above 1G of RAM for example and this will be faster because of the parallelization.

You can try to use generator with Tensorflow, that will fit back and forth your data so it never explode your RAM.

If the issue is before, like getting the data into a variable, you could use Dask. Which is simply a distributed Pandas that helps you fit your data in RAM by processing them by batch


Why are Machine Learning models called black boxes? I was reading this blog post titled: The Financial World Wants to Open AI’s Black Boxes, where the author repeatedly refer to ML models as "black boxes".

A similar terminology has been used at several places when referring to ML models. Why is it so?

It is not like the ML engineers don't know what goes on inside a neural net. Every layer is selected by the ML engineer knowing what activation function to use, what that type of layer does, how the error is back propagated, etc.

The black box thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the explainability of the function modelled by the machine learning algorithm.

In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly catalogued (e.g. because the value of certain component of the input vector was too low).

The same applies to decision trees: you can follow the logic applied by the tree and understand why a certain element was assigned to one class or the other.

However, deep neural networks are the paradigmatic example of black box algorithms. No one, not even the most expert person in the world grasp the function that is actually modeled by training a neural network. An insight about this can be provided by adversarial examples: some slight (and unnoticeable by a human) change in a training sample can lead the network to think that it belongs to a totally different label. There are some techniques to create adversarial examples, and some techniques to improve robustness against them. But given that no one actually knows all the relevant properties of the function being modeled by the network, it is always possible to find a novel way to create them.

Humans are also black boxes and we are also sensible to adversarial examples.


Understanding output stepAIC I am using the stepAIC function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function means.

The output is:

          Df     Sum of Sq    RSS       AIC
<none>                        350.71   -5406.0
- aaa      1     0.283        350.99   -5405.9
- bbb      1     0.339        351.05   -5405.4
- ccc      1     0.982        351.69   -5400.5
- ddd      1     0.989        351.70   -5400.5

Question Are the values listed under Df, Sum of Sq, RSS, and AIC the values for a model where only one variable would be considered as the independent variable (i.e. y ~
aaa, y ~ bbb, etc.)?

After consulting with someone I found out that the  corresponds to a model that would include all the variables, in other words none of the variables were removed. So consider the line in the output for the variable aaa. The listed RSS and AIC are the values for a model that would include all variables but aaa and we see an increase in the RSS and AIC. The other listed results can be considered in the same fashion. The best model is then the one where none are removed since this has the smallest AIC.


Do you need a virtual machine for data science? I am brand new to the field of data science, want to break into it, and there are so many tools out there.  These VMs have a lot of software on them, but I haven't been able to find any side-by-side comparison.

Here's a start from my research, but if someone could tell me that one is objectively more rich-featured, with a larger community of support, and useful to get started then that would help greatly:

datasciencetoolKIT.org -> vm is on vagrant cloud (4 GB) and seems to be more "hip" with R, iPython notebook, and other useful command-line tools (html->txt, json->xml, etc). There is a book being released in August with detail.

datasciencetoolBOX.org -> vm is a vagrant box (24 GB) downloadable from their website. There seems to be more features here, and more literature.

Do you need a VM?
You need to keep in mind that a virtual machine is a software emulation of your own or another machine hardware configuration that can run an operating systems. In most basic terms, it acts as a layer interfacing between the virtual OS, and your own OS which then communicates with the lower level hardware to provide support to the virtual OS. What this means for you is:
Cons
Hardware Support
A drawback of virtual machine technology is that it supports only the hardware that both the virtual machine hypervisor and the guest operating system support. Even if the guest operating system supports the physical hardware, it sees only the virtual hardware presented by the virtual machine.
The second aspect of virtual machine hardware support is the hardware presented to the guest operating system. No matter the hardware in the host, the hardware presented to the guest environment is usually the same (with the exception of the CPU, which shows through). For example, VMware GSX Server presents an AMD PCnet32 Fast Ethernet card or an optimized VMware-proprietary network card, depending on which you choose. The network card in the host machine does not matter. VMware GSX Server performs the translation between the guest environment's network card and the host environment's network card. This is great for standardization, but it also means that host hardware that VMware does not understand will not be present in the guest environment.
Performance Penalty
Virtual machine technology imposes a performance penalty from running an additional layer above the physical hardware but beneath the guest operating system. The performance penalty varies based on the virtualization software used and the guest software being run. This is significant.
Pros
Isolation

One of the key reasons to employ virtualization is to isolate applications from each other. Running everything on one machine would be great if it all worked, but many times it results in undesirable interactions or even outright conflicts. The cause often is software problems or business requirements, such as the need for isolated security. Virtual machines allow you to isolate each application (or group of applications) in its own sandbox environment. The virtual machines can run on the same physical machine (simplifying IT hardware management), yet appear as independent machines to the software you are running. For all intents and purposes—except performance, the virtual machines are independent machines. If one virtual machine goes down due to application or operating system error, the others continue running, providing services your business needs to function smoothly.

Standardization

Another key benefit virtual machines provide is standardization. The hardware that is presented to the guest operating system is uniform for the most part, usually with the CPU being the only component that is "pass-through" in the sense that the guest sees what is on the host. A standardized hardware platform reduces support costs and increases the share of IT resources that you can devote to accomplishing goals that give your business a competitive advantage. The host machines can be different (as indeed they often are when hardware is acquired at different times), but the virtual machines will appear to be the same across all of them.

Ease of Testing

Virtual machines let you test scenarios easily. Most virtual machine software today provides snapshot and rollback capabilities. This means you can stop a virtual machine, create a snapshot, perform more operations in the virtual machine, and then roll back again and again until you have finished your testing. This is very handy for software development, but it is also useful for system administration. Admins can snapshot a system and install some software or make some configuration changes that they suspect may destabilize the system. If the software installs or changes work, then the admin can commit the updates. If the updates damage or destroy the system, the admin can roll them back.
Virtual machines also facilitate scenario testing by enabling virtual networks. In VMware Workstation, for example, you can set up multiple virtual machines on a virtual network with configurable parameters, such as packet loss from congestion and latency. You can thus test timing-sensitive or load-sensitive applications to see how they perform under the stress of a simulated heavy workload.

Mobility

Virtual machines are easy to move between physical machines. Most of the virtual machine software on the market today stores a whole disk in the guest environment as a single file in the host environment. Snapshot and rollback capabilities are implemented by storing the change in state in a separate file in the host information. Having a single file represent an entire guest environment disk promotes the mobility of virtual machines. Transferring the virtual machine to another physical machine is as easy as moving the virtual disk file and some configuration files to the other physical machine. Deploying another copy of a virtual machine is the same as transferring a virtual machine, except that instead of moving the files, you copy them.

Which VM should I use if I am starting out?
The Data Science Box or the Data Science Toolbox are your best bets if you just getting into data science. They have the basic software that you will need, with the primary difference being the virtual environment in which each of these can run. The DSB can run on AWS while the DST can run on Virtual Box (which is the most common tool used for VMs).
Sources

http://www.devx.com/vmspecialreport/Article/30383
http://jeroenjanssens.com/2013/12/07/lean-mean-data-science-machine.html


How to handle "year" variable for Machine Learning models I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?

Thanks in advance,

First of all I would suggest you  to analyze the data. Do you really need the year? Have you studied the correlation with the other variables? I mean, study the domain of the model you intend to create before create any model.

You can treat year as categorical variable and use some of the techniques such as One Hot Ecoding or Dummy Variables for better performance.
You could also perform a normalization of the years, treating them as numerical variables, which are between 0 and 1.

The most important thing is to know what we are going to do to make good use of the information or data.


Recurrent Neural Networks Over Multiple Documents Over Time So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on how to actually implement it let me know. Or if I am over-thinking a solution.

I am trying to classify accounts into one of two groups, good and bad. I have multiple text documents per account. What I want to do is take the text documents for a single account and order them chronologically. Then use a recurrent neural network to essentially learn an embedding for each document (traditional text recurrent architecture should work fine), and then plug those embeddings into a document-level recurrent neural net, to predict when an account switches from good to bad. That way, as new documents come in per account I get updated "badness" scores for an account, and potentially flag an account above some threshold. I would like some sort of end-to-end architecture too. Like I said, not sure how to get what is my head into the computer.

Here's my question, is anyone aware of an implementation like that?

A simple implementation of such a network would be:

Input -->
Embedding layer -->
LSTM layer -->
Softmax layer

This would allow the embedding layer to learn your document representation, and the lstm layer to learn states based on the observed sequences of those embedding instances, and finally the softmax layer to interpret these states at the account level as "good" or "bad".


How to perform bootstrap validation? I am working on a binary classification problem.

I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below

logreg=LogisticRegression(random_state=41)
logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')
logreg_cv.fit(X_train_std,y_train)
y_pred = logreg_cv.predict(X_test_std)

But I came across the term bootstrap validation on how it can be used to avoid test data poor performance due to split imbalances (in test data).

Can someone guide me on what is bootstrap validation and how to implement this?

I do not agree that Bootstrapping is generally superior to using a separate test data set for model assessment.

First of all, it is important here to differentiate between model selection and assessment. In "The Elements of Statistical Learning" (1) the authors put it as following:

  Model selection: estimating the performance of different models in
  order to choose the best one.

  Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

They continue to state:

  If we are in a data-rich situation, the best approach for both
  problems is to randomly divide the dataset into three parts: a
  training set, a validation set, and a test set. The training set is
  used to fit the models; the validation set is used to estimate
  prediction error for model selection; the test set is used for
  assessment of the generalization error of the final chosen model.
  Ideally, the test set should be kept in a “vault,” and be brought out
  only at the end of the data analysis. Suppose instead that we use the
  test-set repeatedly, choosing the model with smallest test-set error.
  Then the test set error of the final chosen model will underestimate
  the true test error, sometimes substantially.

In line with this the author of "Machine Learning - A Probabilistic Perspective" (2) writes:

  Often we use about 80% of the data for the training set, and 20% for the validation set. But if the number of training cases is small, this technique runs into problems, because the mode won’t have enough data to train on, and we won’t have enough data to make a reliable estimate of the future performance. A simple but popular solution to this is to use cross validation (CV)

Since in model selection you are comparing different models (different hyperparameters for a given technique or even different techniques) the absolute level of the prediction error estimate is not that important. As long as the relative error, i.e. how does model A compare to B, is sufficiently well estimated. In contrast, when assessing your final model you would like to obtain an estimate of the absolute prediction error. Also note that in the text quoted by Carlos Mougan the authors write

  [...] it can be evaluated on these instances, without the need for a
  separate validation set.[...]"

i.e. we are talking validation (model selection) and not testing (models assessment) here. Both, Cross-Validation and Bootstrapping are more common for model selection (i.e. estimate the validation error) but less common for model assessment (i.e. estimate the test error).

To summarize:

Cross-Validation and Bootstrapping are usually applied when data limited
Cross-Validation and Bootstrapping are usually applied for model selection/estimation of validation error. They are less common for test error estimates.

Nevertheless Bootstrapping can be used to estimate the validation and even the test error. There are at least 3 ways to apply it (I will now refer to "prediction error" which could either mean "validation error" or "test error" depending on the way you use Bootstrapping):

1. Simple Bootstrap

You produce $B$ Bootstrap samples for your data by randomly selecting $N$ (sample size) data points with repetition. You then train your data on each Bootstrap sample and calculate the prediction error on the whole data set. This is then averaged across all Bootstrap samples $B$. Since the prediction error is based on the whole data set, incl. the examples in the Bootstrap samples, it means that training data is also used to estimate the prediction error. Which can lead to overfitting.

2. Leave-one-out Bootstrap

Here, the approach is similar to Cross-Validation: for a predictor $\hat{f^b}$ trained on Bootstrap sample $b$ the prediction error is estimated based on all data except the examples in the Bootstrap sample $b$. (However, this is not the same as Cross-Validation as Bootstrap samples are produced by drawing with repetition.) This way the problem of overfitting is solved as training and prediction error estimation are based on different sub-data sets for each bootstrap sample.

3. .632 Bootstrap

This approach uses a linear combination of the Leave-one-out Bootstrap prediction error $error_{leave{\text -}one{\text -}out}$ and the simple training error $error_{training}$:
$$error_{.632} = 0.368 \cdot error_{training} + 0.632 \cdot error_{leave{\text -}one{\text -}out} $$

Note that $0.632$ is the probability that when drawing a Bootstrap sample a data point $(x_n,y_n)$ from the overall population is included in that Bootstrap sample. A very casual interpretation of this equation is that you use an estimate that is somewhere between the test error and the leave-one-out Bootstrap error with more weight being put on the latter.

For more details I suggest to read section 7.11 in "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" (1).

(1) "The Elements of Statistical Learning - Data Mining, Inference, and Prediction"; Hastie et al; 2nd Ed

(2) "Machine Learning-A Probabilistic Perspective"; Murphy; 1st Ed


What's the easiest way to quickly plot several features simultaneously for new data? I'm starting a very beginner tutorial for DS/ML. One of the first things to do is to poke at the data they gave me. After doing the typical head() and describe() in pandas, I want to look more carefully at the features that are numerical, to see if there's any correlation between them and the labels (for example, a simple linear relationship).

I've used matplotlib and stuff before, but I want to do this the "correct" way. The tutorial I'm looking at suggests using seaborn, but to be honest it seems difficult to apply to what I'm suggesting (they're looking at more complicated relationships/etc).

I could use matplotlib, but I've never found a smooth, seamless way of making a grid of several plots simultaneously. Doing the "subplot" thing was really clunky. Is there a better way?

What I'd like to do is, let's say I have 4 numerical features, and I want to plot each of them (or their average, etc) vs the label, and display it, so I can quickly see patterns. What's the best way to do this?

I believe seaborn's pairplot is what you are looking for.

Seaborn is a plotting library built on top of matplotlib for exactly the reason you want. To make quick aesthetically pleasing graphs for data analysis. Seaborn also work very well with pandas DataFrames.


Encoding and cross-validation Recently I've been thinking about the proper use of encoding within cross-validation scheme. The customarily advised way of encoding features is:

Split the data into train and test (hold-out) set
Fit the encoder (either LabelEncoder or OneHotEncoder) on the train set
Transform both the train and test set using fitted encoder.

This way is claimed to prevent from any data-leakage. However, this seems to often be omitted during cross-validation. Let's suppose I am performing cross validation on the aforementioned train set. If I encode train set and then perform cross-validation it doesn't really mimic the steps above. Shouldn't the encoding be performed "within" cross-validation then? For example, assuming that we perform 5-fold cross-validation, shouldn't we fit the encoder on 4 folds and transform on 5th fold in each cross-validation step? I believe it is what's usually done in target encoding, but not really with label or one-hot encoding.
My questions therefore are:

Am I right about the necessity to fit the encoder on 4 folds and not on the 5th validation fold in each cross-validation step if we really want to prevent overfitting?
If not, why is it really necessary to perform all 3 steps mentioned before while dealing with train and test (hold-out) set?

You're right, the encoding step itself can be a source of data leakage and normally it should be done inside the CV loop using only the current training set, as you describe.
The reason is indeed the one you mention in the comment: if there is a class label or a feature category which doesn't appear by chance in a particular training set during CV, the model is not supposed to know that this class/category even exists.
In general I would think that this issue can only decrease the performance on the test set, so it's probably not as serious as other kinds of data leakage. Still, it's definitely a cleaner experimental design to encode using only the training set.
A closely related issue in NLP is when the system is not designed to deal with out-of-vocabulary (OOV) words: if all the words in both the training and test set are encoded (same mistake), then it wrongly looks as if any text can be fully encoded, potentially causing to bad surprises later.
That being said, it's usually a good idea to discard rare features or label values, and if this is done then the result should be the same using either the proper method or the sloppy one.


Missing population values in census data I have population data from Census.gov:

Total US population by age by year from 1940 through 2010

Depending on the range of decades, the data is missing discrete population values for ages greater than a certain age.  Instead an aggregate amount is provided that represents all ages greater than the cutoff.

Specifically it follows this pattern:

1940 to 1979:  Discrete data from 0 to 84 and aggregate for ages 85
and greater
1980 to 1999:  Discrete data from 0 to 99 and aggregate for ages 100 and greater
2000 to 2010:  Discrete data from 0 to 84 and aggregate for ages 85 and greater

The desired outcome is to have discrete data points for each age and year from 0-99 and then an aggregated lump sum figure for ages 100 and greater.

Therefore I want to input the missing discrete population values for ages 85 to 100 for years 1940 through 1979 and years 2000 through 2010.

And I want to use the actual discrete population values for ages 85 to 100 for years 1980 through 1989 to achieve that outcome.

Some Observations:

The pattern of missing values is MNAR (Missing Not At Random) - these were systematically omitted but the aggregate value representing the missing detail is provided
Population data for this time frame is deterministic:  Population levels rise linearly each year;  The duty cycle of a human body is finite and constraints and limits are well known.

Looking at the data we can see that each of the three subsets of years have very similar patterns.  More variation in younger ages and variation flattens out for ages greater than 60

Then if we focus on the years 1980 through 1989 we can fit a nice curve for ages 0 through 100 with a Multiple-R-Squared of .979.

Then if we narrow the focus to ages 60 to 100 and even narrower to ages 85 to 100 the Multiple-R-Squared increases to .9996.

Now if we flip our focus and look at the increasing levels of population we can observe that these relationships are linear.  Population rises at a steady rate year over year.

Total population 1940 through 2010:

Ages 85 through 100 for years 1980 through 1999.  Each age is linear.  Each age has a slightly lower rate of increase (lesser slope).

My question

This is where I could use some guidance to move forward:

When imputing discrete missing population values by age and year, how do I combine the fitted curve that models changes in population when age increases with the linear regression that models changes in population year over year?

Does one or more documented methods naturally apply to the problem as I have described it?
For example:  KNN, PCA, BPCA, Mean, MICE, other?

If there is recommended method can you point me to available R or Python packages and documentation that describes the mechanics of applying a given approach?

I think you need to be wary of using curves to extrapolate beyond the age thresholds - specifically I think you should consider:

mortality increases with age; I would imagine it increases at an increasing rate with age (especially at higher ages). Would you be able to capture this effectively?
there is obviously some sort of overlap between populations in subsequent years (e.g. someone who is included in 1985 data could also be in 1986 data). What effect would this have on the data imputation?

I would suggest a different approach. Actuaries have traditionally produced "life tables" which capture mortality in population cohorts. If you can find a set of tables which is applicable to the period in question, you could use these to calculate population numbers.


Scripting code for class imbalance in Biolabs Orange I'm trying to manipulate some data in Biolabs Orange, using the built in Python Script widget and information at Biolabs Orange tutorial on scripting.
However, I'm struggling with taking the results of SMOTE and putting them into a format for Orange:
This is the code I'm using in the Python Script widget:
# Get libraries
import Orange
import numpy as np
from Orange.data import Domain, Table
from imblearn.over_sampling import SMOTE

#in_data = Orange.data.Table('WORKING_temp.csv')

df = in_data.copy()

# set variables for SMOTE
sm = SMOTE(random_state=42)

# get table of data (X) and class variables (y)
X, y = df.X, df.Y

# resample data and classes
X_res, y_res = sm.fit_sample(X, y)

df.X = X_res
df.Y = y_res

temp = Orange.data.Table(df.X, df.Y)
temp.domain = df.domain

out_data = Orange.data.Table(temp)

The result is a ValueError, which I think is related to changing the length of the class variables and data table, while leaving the original index length?
"ValueError: could not broadcast input array from shape (3724,10) into shape (3724)"

With some help from the Orange team, I was able to solve the problem!
Link to thread on github

The following code can be pasted into a Python Script widget, and will balance your classes before training a classifier in Orange:

# Get libraries
import Orange
import numpy as np
from Orange.data import Domain, Table
from imblearn.over_sampling import SMOTE

# How Orange passes data to widget
df = in_data.copy()

# set variables for SMOTE
sm = SMOTE(random_state=42)

# get table of data (X) and class variables (y)
X, y = df.X, df.Y

# resample data and classes
X_res, y_res = sm.fit_sample(X, y)

# Get the target and feature variables
d = Domain(df.domain.attributes, df.domain.class_vars)

# Create a new Orange Table object with the appropriate headers
# This is how Orange passes the data on to the next widget
out_data = Orange.data.Table(d, X_res, y_res)


Advice on imputing temperature data with StatsModels MICE This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for five neighboring stations (LOC1:LOC5) over many years and I'd like to impute the missing data for any given site. Following the examples I have:

imp = mice.MICEData(dfLocal)
fml = 'LOC1 ~ LOC2 + LOC3 + LOC4 + LOC5'
mice = mice.MICE(fml, sm.OLS, imp)
results = mice.fit(10, 10)
print(results.summary())

dfLocal.dropna(axis=0, how='all', inplace=True)
imp.data = imp.data.set_index(dfLocal.index)

# In this case I only want to fill one specific set of missing data
# hence gap_start and gap_end
dfLocal.loc[gapStart:gapEnd, 'LOC1'] = imp.data[fillSite]

My understanding of MICE is broadly that missing values are imputed multiple times and then combined to find the best value from the many. The only way I've found to actually get any numbers out of the above code is with imp.data but I'm afraid that might just be one of the individual imputations before they're combined? All I can seem to get from fitting the model (results), though, is the summary?

I'm far from a statistician (and not much of a programmer either) so I've been reading through the code for mice.MICE and other resources on general MICE applications, but I'd appreciate any guidance on this as I can't find much about using statsmodels' MICE online. Normally I'd post some data on Gist but the full set is a bit large. That said, I'll upload it if ya'll think it would help.

Thanks!

MICE does generate several datasets, but it does not then combine these datasets.  Rather, it fits your model on each of those datasets and combines those models.  If you really need an imputed dataset, you could just choose one or combine them in whatever way makes sense for your problem (or you might be better off with another method):

https://stats.stackexchange.com/questions/188288/how-to-know-which-imputation-is-best-for-impute-my-dataset-from-multiple-imputat
https://stackoverflow.com/questions/28814384/how-to-extract-aggregated-imputed-data-from-r-package-mice
https://www.theanalysisfactor.com/multiple-imputation-in-a-nutshell/

Now, for the statsmodels implementation, imp.data only keeps track of the latest imputed set [1]; you can loop through updates rather than using fit to get all of the datasets as in an example in [2].