Simple example of genetic alg minimization I have been looking for a while for examples of how I could find the points at which a function achieves its minimum using a genetic algorithm approach in Python. I looked at  DEAP documentation, but the examples there were pretty hard for me to follow. For example:

def  function(x,y):
     return x*y+3*x-x**2

I  am looking  for some references on how  I can make a genetic algorithm in which I can feed some initial random values for both x and y (not coming from the same dimensions). Can someone with experience creating and using genetic algorithms give me some guidance on this?

Here is a trivial example, which captures the essence of genetic algorithms more meaningfully than the polynomial you provided.  The polynomial you provided is solvable via stochastic gradient descent, which is a simpler minimimization technique.  For this reason, I am instead suggesting this excellent article and example by Will Larson.

Quoted from the original article:

  Defining a Problem to Optimize Now we're going to put together a simple example of using a genetic algorithm in Python. We're going to
  optimize a very simple problem: trying to create a list of N numbers
  that equal X when summed together.

  If we set N = 5 and X = 200, then these would all be appropriate
  solutions.

lst = [40,40,40,40,40]
lst = [50,50,50,25,25]
lst = [200,0,0,0,0]

Take a look at the entire article, but here is the complete code:

# Example usage
from genetic import *
target = 371
p_count = 100
i_length = 6
i_min = 0
i_max = 100
p = population(p_count, i_length, i_min, i_max)
fitness_history = [grade(p, target),]
for i in xrange(100):
    p = evolve(p, target)
    fitness_history.append(grade(p, target))

for datum in fitness_history:
   print datum
"""
from random import randint, random
from operator import add

def individual(length, min, max):
    'Create a member of the population.'
    return [ randint(min,max) for x in xrange(length) ]

def population(count, length, min, max):
    """
    Create a number of individuals (i.e. a population).

    count: the number of individuals in the population
    length: the number of values per individual
    min: the minimum possible value in an individual's list of values
    max: the maximum possible value in an individual's list of values

    """
    return [ individual(length, min, max) for x in xrange(count) ]

def fitness(individual, target):
    """
    Determine the fitness of an individual. Higher is better.

    individual: the individual to evaluate
    target: the target number individuals are aiming for
    """
    sum = reduce(add, individual, 0)
    return abs(target-sum)

def grade(pop, target):
    'Find average fitness for a population.'
    summed = reduce(add, (fitness(x, target) for x in pop))
    return summed / (len(pop) * 1.0)

def evolve(pop, target, retain=0.2, random_select=0.05, mutate=0.01):
    graded = [ (fitness(x, target), x) for x in pop]
    graded = [ x[1] for x in sorted(graded)]
    retain_length = int(len(graded)*retain)
    parents = graded[:retain_length]
    # randomly add other individuals to
    # promote genetic diversity
    for individual in graded[retain_length:]:
        if random_select > random():
            parents.append(individual)
    # mutate some individuals
    for individual in parents:
        if mutate > random():
            pos_to_mutate = randint(0, len(individual)-1)
            # this mutation is not ideal, because it
            # restricts the range of possible values,
            # but the function is unaware of the min/max
            # values used to create the individuals,
            individual[pos_to_mutate] = randint(
                min(individual), max(individual))
    # crossover parents to create children
    parents_length = len(parents)
    desired_length = len(pop) - parents_length
    children = []
    while len(children) < desired_length:
        male = randint(0, parents_length-1)
        female = randint(0, parents_length-1)
        if male != female:
            male = parents[male]
            female = parents[female]
            half = len(male) / 2
            child = male[:half] + female[half:]
            children.append(child)
    parents.extend(children)
    return parents

I think it could be quite pedagogically useful to also solve your original problem using this algorithm and then also construct a solution using stochastic grid search or stochastic gradient descent and you will gain a deep understanding of the juxtaposition of those three algorithms.

Hope this helps!


LSA, LDA or NMF in Topic Modeling? I'm trying to implement Topic Modeling via Python & NLP but can't figure out what algorithm should I use. I have studied Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA) and Non-negative matrix factorization (NMF) but how to decide which algorithm fits best for certain task? If I just try all of them in a row then how to measure the result?

As far as I know LDA is the state of the art approach for topic modeling, but I'm not following the field very closely. So I would say that it's pretty safe to use LDA, and my guess is that the different approaches are likely to give similar results overall.
In case you want to try different methods, the question of evaluating topic models is quite complex. This article might help.
Side note: there is a non-parametric variant (no need to choose the number of topics) of LDA called Hierarchical Dirichlet Processes.


NLP - paraphrase extraction in python I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like you have high levels of cholesterol, you need to have a low-salt diet that emphasizes fruits, vegetables and whole grains; limit the amount of animal fats and use good fats in moderation(this will be the suggestion; and it is an example suggestion from doctor).

So, now when i was researching on how this could be accomplished i stumbled on this research paper and i learned about something called paraphrase extraction from it. and also that i need to build a parallel corpus with tag and suggestion.

I want to do this in python. I couldn't find much information from internet on how paraphrase extraction can be done in python, but there are many articles talking about paraphrase detection and things. So, do we have any libraries in python for this purpose? or any kind of help is appreciated. How do i build a corpus for this same pupose?

Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best solution" to this problem.

In order to build a corpus you might want to look at how shared tasks/competitions have done it before. I know at least of SemEval which often proposes tasks related to paraphrases (there might be others). I haven't checked but usually the overview paper of the task (e.g. here) explains how the corpus was built and presents the main approaches submitted by participants.


ImageDataGenerator for multi task output in Keras using flow_from_directory I am creating a multitask CNN model and I have two different classification properties (one with 10 classes, 2nd with 5 classes) and my directory structure looks like this:

    -Train
       - image1.jpg
          ...
       - imageN.jpg

    -Test
       - image1.jpg
             ...
       - imageN.jpg

    -Vald
       - image1.jpg
          ...
       - imageN.jpg

And labels are in a csv file as propA, propB. So, a single image will have two classes, one from property A and one from property B.
The model uses VGG16 :
baseModel = VGG16(weights="imagenet", include_top=False,input_tensor=Input(shape=(img_size, img_size, 3)))
flatLayer = baseModel.output
sharedLayer = Flatten(name="flatten")(flatLayer)
sharedLayer = Dense(1024,name="Shared")(sharedLayer)
sharedLayer = Dropout(0.5)(sharedLayer)
task1 = Dense(512, activation="relu")(sharedLayer)
task1 = Dense(10, activation="softmax",name='PFR')(task1)
task2 = Dense(512, activation="relu")(sharedLayer)
task2 = Dense(5, activation="softmax",name='FT')(task2)
model3 = Model(inputs=baseModel.input, outputs=[task1,task2])

The number of images is large, therefore I cannot load them in memory and need to use flow_from_directory like functionality. But, in my train directory, there are no class sub directories because it is not possible to generate class directories as there are total 15 classes and I am not sure on which property to generate class subdirs for. (and flow_from_directory doesnt work if there are no class subdirs)
The labels are available in  array, propALab and propBLab.
So far, I havenot found anything helpful, can someone help?

Have you tried creating a custom Data Generator for your use case? The structure for it can be found below. You will need the __data_generation method such that you return y as an array with 2 elements, with the first element being your labels for task 1 and the second element for task 2.

import numpy as np
import keras

class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1,
                 n_classes=10, shuffle=True):
        'Initialization'
        self.dim = dim
        self.batch_size = batch_size
        self.labels = labels
        self.list_IDs = list_IDs
        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.list_IDs) / self.batch_size))

    def __getitem__(self, index):
        'Generate one batch of data'
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        list_IDs_temp = [self.list_IDs[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(list_IDs_temp)

        return X, y

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        self.indexes = np.arange(len(self.list_IDs))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, list_IDs_temp):
        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
        # Initialization
        X = np.empty((self.batch_size, *self.dim, self.n_channels))
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, ID in enumerate(list_IDs_temp):
            # Store sample
            X[i,] = np.load('data/' + ID + '.npy')

            # Store class
            y[i] = self.labels[ID]

        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

You can then use your custom generator with the Model object you have created as shown below:
training_generator = DataGenerator(partition['train'], labels, **params)
validation_generator = DataGenerator(partition['validation'], labels, **params)

# Design model
model = Sequential()
[...] # Architecture
model.compile()

# Train model on dataset
model.fit_generator(generator=training_generator,
                    validation_data=validation_generator,
                    use_multiprocessing=True,
                    workers=6)

The codes have been taken from https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly


Weighting of words in lexicon based sentiment analysis I have a a question regarding my current project, i am trying to do a lexicon based sentiment analysis on my data, where i calculate the sentiment score as following:

$$ Score = \frac{\sum_{i}{word_i}}{\mid words \mid} $$

So according to the score the word will be classified in either negative or positive. But i have also calculated for every word in the article its salience and frequency and would like to know if its possible to use them in my sentiment analysis formula above.

 words| salience| frequence
 sad    0.8       3
 happy  0.5       2

Yes, you can. Not quite sure what else to add. Your formula can then look like:

$$ Score = \sum_{i}{f(salience_i, frequency_i, sentiment_i)}$$

Where $f$ is a function that weighs your sentiment score with the salience and frequency. Up to you to define how.

What if you don't know which $f$ to use?

Now, bear with me, this isn't something I've tried per se, but this could be an interesting approach. You could use a recurrent neural network and your input could be the salience, frequency, and sentiment score for each word. Not only will your RNN "create" (ideally) the best $f$ for your particular problem, but it will also use the sequential information of the words, which may even improve your results.


Information retrieval / slot filling / NLP Excuse if this has been answered before.

I need to extract features and parse from a piece of text and run some analysis. For e.g. "Plot the past 5-year sales of Apple" should give me the following

Information:

Company: Apple
Item: Sales
Period: past 5 years
Action: Plot

What deep learning techniques / algorithms should I be looking to use?

Any pointers are highly appreciated.

This is usually handled as a two step process:

intent detection
slot filling

Intent detection is basically just a kind of classification. So if your program has multiple kinds of questions it can be asked you build a list of examples for each and train a classifier.

Slot filling is typically modelled as a sequence labelling problem, so you can look into seq2seq.

You can find more detailed information in any tutorial on how to make a chatbot.


How to perform link prediction in text based relationship data I need to establish if there is a link between 2 columns from two different datasets with one matching column, where;

Dataset1: bipartite:  (M, DS)
M    G
m23  ds3
m23  ds67
m54  ds325
...  ...

Dataset2: tripartite: (M, G, DG)
M    G    DG
m23  g6   dg32
m23  g8   dg1
m54  g32  dg65
...  ...   ...

These 2 datasets have one column in common(i.e., M), and the relationship among the elements is shown below:

M  ----affects----> G
M  ----causes-----> DS
DG ----affects----> M

Primary Goal: To calculate the probability of a possible link/edge that might exist between indirectly related columns(eg. DG and DS) via the common column(M).

So, for a given list of DS entries, how to find the probability of the existence of a link/edge between
selected DS, and all the other DGs

DS <---- ----> DG

If DS; (ds3, ds67) were selected, the output should be like this:

element1 - element2 - probability/statistical value to signify the existence of direct relationship OR link.

ds3 - dg32  - 100% (common M value)
ds3 - dg1  - 100%  (common M value)
ds3 - dg65 - 43.66%
---
ds67 - dg32 - 100% (common M value)
ds67 - dg1  - 100% (common M value)
ds67 - dg65 - 55.12%

I am trying to code this in Java, but Python based solutions can work too.

I am sorry I am not too familiar with graph theory, a little descriptive solutions would be really appreciated.
Thanks.

You are describing building a Probabilistic Graphical Model (PGM).

The most commonly used Python library to build a PGM is pgmpy.


What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem I am trying to write a model that will extract certain details from financial documents.

It must be able to extract the; contract start date, contract duration, contract value and all named entities. Preferably it would also be able to categorise the named entities.

I have roughly 600 tagged documents that I can use for training and testing purposes.

What model should I use?

I have had a look at different named entity recognition models, such as the Skip-Gram model. However these models do not utilise the fact that each one of these documents will contain all of this information. To illustrate this, the Skip-Gram model is often implemented in such a way that only the words in the same sentence contribute to its semantic vector.

Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming "unfashionable" as of late due to the prevalence of deep learning, I would suggest that you try using Conditional Random Fields (CRF).

CRFs offer very competative performance in this space and are often used for named entity recognition, part of speech tagging and variants thereof.

Another great thing about CRFs is that they takes into account the position of the current 'field' in a document and also the preceeding and succeeding fields in order to calculate what the current class should be so you don't have to spend too much time engineering structural features into your model.

I have had a great deal of success with the library  crfsuite in particular which is written in C but had a python wrapper.


How to use sklean pipeline to deal with data that read in line by line The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass operations that I wrote in the future. If I read data line by line, is there any way I can still use sklearn pipeline API?

Sklearn has some strategies on its website. Mainly, some of their estimators allow incremental learning, meaning that you can feed the data in streaming fashion during training.

To use it with a pipeline, I would refer you to this StackOverflow post. You essentially have to break down the pipeline during training, and then during inference, you can use it as is.


How best to embed large and noisy documents I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect.
Right now I creating a doc for each page, processing it with TFIDF and then creating a dense feature vector using UMAP.
My final goal is to really pick out the differences in the articles, for similarity analysis, clustering and classification - however at this stage my goal is to generate the best possible embeddings.
For this type of data what is the best method to create document embeddings?
Also is it possible (is yes how) to embed various parts of the page; title, description, tags separately and them maybe combine this into a final vector?

Doc2vec and similar algorithms are useful methods to create document embeddings.
You should try to include as much data and metadata as possible.
The size of the documents does not matter much because they will be project into a fixed dimensional embedding space.
As far as noise, the effect will be task specific. If you are doing similarity analysis and clustering, you can use approximate methods which are robust to noise.


Why is there a difference in performance across the feature descriptors for the same imaging modality? I've been using GIST, HOG and SURF descriptors for extracting features from different collections of Chest-X-rays and measuring performance using accuracy and area under the curve. These collections are obtained using different machinery, from different medical institutions, and, with different pixel resolutions. I could repeatedly see that one descriptor performs better than the other and the performance is not the same across these collections though all are frontal chest X-rays. What attributes to these differences in performance across the collections though they are from the same imaging modality?

Feature-extraction mechanisms like GIST, HOG, etc are built and optimized to improve performance on given datasets.  Because of this, they don't perform as well across datasets.  It's kind of like putting specialized fuel in a vehicle that isn't built to utilize it - it might even do harm.

Hand-engineered features are, as a rule, brittle.  I once heard it said that the dirty secret of machine learning is just knowing how to transform your domain-specific information into meaningful features - after that, you can use an extremely simple classifier and it may do surprisingly well.  The drawback is that the rules you built are very specific to your domain.

Deep neural networks, and convolutional neural networks in particular, are an advancement in that they learn what features are useful about raw data - for CNNs, these are the raw pixel or time-series values.  Instead of hand-building feature extraction mechanisms, these architectures automatically build them.

One benefit of this is that if you use a CNN to identify images in general, you can re-use the top few feature extraction layers of the CNN on a different image recognition dataset, and re-train the bottom few layers to make the network specific to recognizing e.g. dog breeds.  You can transfer what you learned about the statistical structure of natural images in general to other, more specific questions (general -> specific).  In your case, the 'top few layers' are analogous to your GIST/HOG methods - and they wouldn't be expected to perform well when the task changes, because they were constructed for a specific task (specific -> other specific).


ConvNet with concatenated data I have a basic question regarding convolutional neural network. Assume I have a set of 1000 RGB images and I train a CNN from this set. I can obviously split each of my RGB images into 3 different greyscale images, each representing red, green, and blue array - thus creating 3000 greyscale images.

My question is: if I train a CNN from those 3000 greyscale images, would I get the same parameters from the first CNN? More precisely, how does CNN behave when we feed it with RGB images? Does it extract the combined features from the RGB images or does it merely learn the features in each R,G,B channel separately?

It behaves like a fully connected network regarding the channels dimension. For each pixel location, the CNN takes the weighted sum of all pixels in that location on the channels dimension. The weights are of course trainable.

This is only for the first layer, though, because afterwards layers have an arbitrary number of channels that has nothing to do with the colors in the original image.

You could also read this answer where I explain how the operation works in a bit more detail.


Comparing Parameter Importance Across Models I have a dataset with 20 features(columns that is). I create a few models pairs with a subset of these parameters.

For example: If I have 6 columns (named A, B, C, D, E, F) with 10k lines of data, one of model in the pair will have (A, B, C) with all 10k lines of data and the other (A, B, C) with maybe say 6k lines of data. Yet another pair will have the parameters (B, C, E), however, the number of lines is always fixed at 10k and 6k.

I am limited to using a sci-kit learn's decision tree and this cannot be changed. I use the following function to calculate my features:

model = clf_gini.best_estimator_
print(model.feature_importances_)

My question is: **Is there any formal, programmatic way to compare feature importance across model pairs using sci-kit, other than intuition? Also, does it make sense to compare to models trained on some similar some different parameters? **

Please let me know if the question is ambiguous, I will clarify.

This may not be the answer you are hoping for (I can't leave a comment because my reputation is too low) but I believe feature importance in Sci-Kit Learn is derived from the average depth at which each feature appears. I don't know of any built-in functions in Sci-Kit Learn but perhaps you could scale the feature importance using the depth of the tree and the number of features?

This article may help with developing a comparative measure of feature importance: https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3.


Ridge regression - varying alpha and observing the residual I am trying to reproduce this figure from Bishop:

Residual vs. Alpha (lambda in figure)

The code is pasted below:

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

x_train = np.linspace(0, 1, 10)
noise_10 = np.random.normal(0, 0.3, 10)
y_train = np.sin(2*np.pi*x_train) + noise_10
x_train = x_train[:, np.newaxis]

x_test = np.linspace(0, 1, 100)
noise_100 = np.random.normal(0, 0.3, 100)
y_test = np.sin(2*np.pi*x_test) + noise_100
x_test = x_test[:, np.newaxis]

n_alphas = 200
alphas = np.logspace(-40, -18, n_alphas)

errors = []
for a in alphas:
    ridge = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=a))
    ridge.fit(x_train, y_train)
    mse = mean_squared_error(y_test, ridge.predict(x_test))
    errors.append(np.sqrt(mse))

print(errors)

However, the errors array has the same value for all values of alpha. It's taking the first value of alpha = np.exp(-40) and all the other values seem to be the same for all future iterations of the for loop. How can I correct this error?

I still haven't figured out what the previously posted code is doing wrong. However, manually populating the alphas array gives me results that are close to the original figure.

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
np.random.seed(12344)
x_1000 = np.linspace(0, 1, 1000)
x_train = np.linspace(0, 1, 10)
noise_10 = np.random.normal(0, 0.3, 10)
y_train = np.sin(2*np.pi*x_train) + noise_10
x_train = x_train[:, np.newaxis]
x_test = np.linspace(0, 1, 100)
noise_100 = np.random.normal(0, 0.3, 100)
y_test = np.sin(2*np.pi*x_test) + noise_100
x_test = x_test[:, np.newaxis]
low_alpha = make_pipeline(PolynomialFeatures(degree = 9), Ridge(alpha=np.exp(-18)))
low_alpha.fit(x_test, y_test)
plt.figure(1)
plt.plot(x_train, low_alpha.predict(x_train), label = 'alpha = ln(-18)')
plt.plot(x_1000, np.sin(2*np.pi*x_1000))
plt.legend()
plt.show
high_alpha = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=np.exp(0)))
high_alpha.fit(x_test, y_test)
plt.figure(2)
plt.plot(x_train, high_alpha.predict(x_train), label = 'alpha = 1')
plt.plot(x_1000, np.sin(2*np.pi*x_1000))
plt.legend()
plt.show
alphas = np.array([np.exp(-30), np.exp(-29), np.exp(-28), np.exp(-27), np.exp(-26), np.exp(-25), np.exp(-24), np.exp(-23), np.exp(-22), np.exp(-21), np.exp(-20), np.exp(-19), np.exp(-18), np.exp(-17), np.exp(-16), np.exp(-15), np.exp(-14), np.exp(-13), np.exp(-12), np.exp(-11), np.exp(-10), np.exp(-9), np.exp(-8), np.exp(-7), np.exp(-6), np.exp(-5), np.exp(-4), np.exp(-3), np.exp(-2), np.exp(-1), np.exp(-1), np.exp(0), np.exp(1)])

test_errors = []
train_errors = []

for a in np.nditer(alphas):
    ridge = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=a))
    ridge.fit(x_train, y_train)
    mse_train = mean_squared_error(y_train, ridge.predict(x_train))
    mse_test = mean_squared_error(y_test, ridge.predict(x_test))
    train_errors.append(np.sqrt(mse_train))
    test_errors.append(np.sqrt(mse_test))
plt.figure(3)
plt.plot(alphas, test_errors, 'g^', label = 'Test Error')
plt.plot(alphas, train_errors, 'bs', label = 'Train Error')
plt.xscale('log')
plt.xlabel('Regression coefficient Lambda')
plt.ylabel('Residuals')
plt.legend()
plt.show()

The output (only for the third figure) is shown below:


python linear search code, what’s a better example if a linear search Algorithm? Having issues with this python code

def linerar_search(list, target):
  """ Returns the index position of the target if found, else returns None"""

for i in range(0, len(list)):
    if list[i] == target:
      return i
    return none

    def verify(index):
        if index is not None:
            print("Target found at index: ", index)
        else:
                print("Target not found in list")

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

result = linear_search(numbers, 12)
verify(result)

showing error in line 6

A few issues with the code above:

Misspelling of variable name in line 1 - linerar_search instead of linear_search
None is the reserved keyword in python, not none. Trying to return none would throw a NameError: name 'none' is not defined
The indent for function verify is off but it shouldn't cause errors.

The code below runs for me:

def linear_search(list, target):
    """ Returns the index position of the target if found, else returns None"""
    for i in range(0, len(list)):
        if list[i] == target:
            return i
        return None

def verify(index):
    if index is not None:
        print("Target found at index: ", index)
    else:
        print("Target not found in list")

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

result = linear_search(numbers, 12)
verify(result)

Hope this helps! :)

P.S: This seems more so like a python problem then data science. Perhaps you could have posted this question in Stackoverflow instead.


Plotting three lines on the same plot (with four hr frequency) I want to plot the graph with datetime . I saw so many questions similiar to my questions. But that answers didn't work for me. Here is the code that I used:

 data = pd.read_csv('p1.csv')
 df1 = pd.DataFrame(df1, columns=['date', 'time', 'x1', 'x2','x3'])
 data['date_time'] = pd.to_datetime(data['date'] + ' ' + data['time'])
data['date_time'] = pd.to_datetime(data['date_time'], format='%m/%d/%Y %H:%M:%S')
data.set_index('date_time',inplace = True)
plt.plot(data.index, data.x2)

plot graph :

Can anyone suggest me a solution for this?

csv file after reading it:

i want to plot like this but withdate

First, in order to plot three different lines with a 4 hour interval you will need to resample your data. (This is how you get your frequency conversion). You can read about that more here.

pd.resample()

Then as long as all three of your "lines" are contained in the same data frame, you should have no problem plotting them by just calling

three_lines = your data frame

plt.plot(three_lines)

Although one note - just looking at your data, you might not get the desired result given the scale difference in your data. Said a different way, the values < 1 might be minimally visible when plotted on the same figure as values ranging around 100.

Hope this helps. Please drop a comment if you need any further clarification.


Implicit feature selection I have heard that Random Forest and other tree based machines apply some kind of implicit feature selection.

My Question is: Does this also apply for machines like the SVM? As far as I understand is the choice of the support vectors also some kind of feature selection.

I assume that you are referring to the fact that training a decision tree (e.g. with the C4.5 algorithm or a variant) involves the selection of the features used as conditions in the nodes. Typically at each iteration the algorithm selects the most discriminative feature by ranking all the features using for instance their information gain.

This is indeed very close to a feature selection process:

ranking features by their informative power w.r.t the label is a very common way to do feature selection
the decision tree can end up selecting only a subset of the available features, exactly as if feature selection had been applied.

SVM works very differently and (in general) doesn't have a similar process of selecting particular features. But it's true that in a loose sense any classifier does some kind of feature selection: it measures how relevant each feature is in order to predict the correct class.

However it's important to understand that a real feature selection process consists in removing features before training, i.e. the learning algorithm doesn't get the option to use the features which have been removed. This differs a lot from the above cases because:

if the feature selection removes too many features, the model might not be able to perform as well as with all the original features
if the feature selection correctly removes redundant and/or useless information, the model is likely to perform better than if all all the features were provided. This is because the algorithm can get confused by the abundance of information, either because it's harder to find what is relevant among many options or because it doesn't have enough instances to properly assess the relevance of all the features.


How to interpret Variance Inflation Factor (VIF) results? From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question.

As we can see in the below output, the rad feature has the highest VIF and the norm is that VIF till 10 is okay.

How does VIF calculate collinearity when we are passing an entire linear fit to the function? And how to interpret the results given by VIF? Which variables are collinear with which variables?

lm.fit2 = lm(medv~.+log(lstat)-age-indus-lstat, data=Boston)
> summary(lm.fit2)

Call:
lm(formula = medv ~ . + log(lstat) - age - indus - lstat, data = Boston)

Residuals:
     Min       1Q   Median       3Q      Max
-15.3764  -2.5604  -0.3867   1.8456  25.2255

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  53.942455   4.823309  11.184  < 2e-16 ***
crim         -0.126273   0.029185  -4.327 1.83e-05 ***
zn            0.021993   0.012238   1.797 0.072934 .
chas          2.270669   0.768911   2.953 0.003296 **
nox         -13.959428   3.187365  -4.380 1.45e-05 ***
rm            2.619831   0.378737   6.917 1.43e-11 ***
dis          -1.374045   0.166350  -8.260 1.35e-15 ***
rad           0.286993   0.057004   5.035 6.72e-07 ***
tax          -0.010756   0.003033  -3.546 0.000428 ***
ptratio      -0.840540   0.116431  -7.219 1.99e-12 ***
black         0.008015   0.002402   3.336 0.000913 ***
log(lstat)   -8.672865   0.530188 -16.358  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.258 on 494 degrees of freedom
Multiple R-squared:  0.7904,    Adjusted R-squared:  0.7857
F-statistic: 169.3 on 11 and 494 DF,  p-value: < 2.2e-16

> vif(lm.fit2)
      crim         zn       chas        nox         rm        dis
  1.755719   2.269767   1.062622   3.800515   1.972845   3.418391
       rad        tax    ptratio      black log(lstat)
  6.863674   7.279426   1.770146   1.340023   2.827687

The Variance Inflation Factor (VIF) looks at how well a single $x_i$ is determined by all the other $x_i$ (jointly) in your model.
How does the VIF work?

For each $x_i$ in your model, you run a (auxiliary) linear regression:
$$ x_{1,i} = \beta_1 + \beta_2 x_{2,i} + ... + \beta_n x_{n,i} + u .$$
You retrieve the $R^2$ for each of these models and calculate the $VIF$:
$$ VIF_1 = 1 / (1-R^2_1). $$

Example in R:
Calculate VIF:
library(car)
library(ISLR)
reg = lm(mpg~disp+wt+qsec+hp, data=mtcars)
vif(reg)

Result:
    disp       wt     qsec       hp
7.985439 6.916942 3.133119 5.166758

Do this manually (for disp)
rsq = summary(lm(disp~wt+qsec+hp, data=mtcars))$r.squared
1/(1-rsq)

Result:
7.985439

What about the $VIF=10$ rule of thumb?
$VIF = 10$ is equal to having an $R^2=0.9$ in the auxiliary regression in step 1 above (because $1/(1-0.9)=10$). This means that your other $x_i$ (in the model) explain the $x_i$ under consideration to a large extent (90% if you want to say so). This of course is just a rule of thumb.
In essence, the $VIF$ boils down to the question: "How well is one of my $x_i$ explained by all other $x$ jointly".
In your example tax has the highes $VIF$ (tax=7.279426). This means that the auxiliary regression (step 1) for tax has an $R^2=0.862627$. This means that tax is well explained by all the other $x$ so that there may be a problem with multicollinearity.


How should we use a sparse matrix in classification? I have a sparse matrix beside my data. I think these data could be beneficial but because it is sparse in usual usage is useless. Is there any method to transform it? for example, PCA, do you have any experience?

additional detail:

I working on titanic competition from Kaggle.com, I add nationality (base Name feature) to data. my nationality is something like: "Nordic,Scandinavian,Norway" so I have converted it to dummy variable, but then I have a sparse matrix.

I think usually sparse matrix could not be helpful for classification and we should transform it. (??? Is it true?)

I think in some way I should merge my features to reduce sparsity.

Your whole question makes sense, except for:

  because it is sparse in usual usage is useless

Therefore, I will try to answer this from different angles.

Implementation of sparse matrix

A sparse matrix is not useless because many packages contain algorithms that accept a sparse matrix as input.

You can look at Sklearn's SVMs as an example.

You need a special implementation to deal with sparse matrix because they are stored differently from a normal matrix. However, as mentioned in the comments by @Majid Mortazavi you can follow this answer to convert your sparse matrix to a normal one and use any other implementations out there. You just need to be careful with your RAM usage.

Theoretical application of sparse matrix

A sparse matrix is not useless because many fields in machine learning make use of sparse features.

You can look at NLP as an example. A common feature used in text categorization is TF-IDF which is a sparse representation of a document.

Since the data is sparse, it is often enough to use linear models for classification, since most of the features for a data point will be $0$. This can be considered a good thing because they are easier and faster to compute and no polynomial kernels are needed.

Solving your challenge

You also mentioned PCA for dimensionality reduction but I don't understand why you would need it at this points. Seems like you are creating a sparse matrix to encode a categorical feature.

Maybe, instead of reducing your sparse matrix, you can have a look at better ways to deal with categorical features.


Where in practice can we see following information propagation model I am not certain, if this is the right place to ask the following question. I am looking for some practical scenarios in social networks where the following information propagation model can arise:

Basically, I have a source node and some information propagating radially from it and each recipient receives the information from a single sender.

Assuming the central node is at level 0 (root), this graph becomes a tree. Now it is easy reason about the different types of trees that can be formed and different types of social networks that can be modelled.

In order for the graph to be a tree, communication between nodes has to be restricted, such that the tree structure is not violated (e.g. two nodes on the same level cannot communicate). I can't think of any social network where communication is naturally restricted in this way, but such a social network can be created artificially.

Examples of such artificial social networks would commonly follow a chain-of-command structure. For example in a military setting, the official information flows would have a tree structure.


When to use deep learning for java as opposed to python I have been asked to explore options to build deep learning based applications using java, so i happend to browse a website called dl4j (https://deeplearning4j.org) which has got implemantations of different neural networks starting from MLP to RNN/LSTM.

But I couldn't understand the rationale of using dl4j over python based implemenation. So, could someone please clarify on following items please.

ETL
Data pre-processing
Making use of pre-trained models / transfer learning
Distributed computing
Processing large voulme of data (images,time series data,sensor data..etc)
Production implementation (batch or real time / online)
Post production training (bacth or real time / online)
Mobile app development
IoT edge device computing

When it comes to picking a language for a certain application, many factors may play a role:

Who will be building/maintaining the application and are they familiar with that language?
What other systems will the application need to communicate with and is one language easier to do that with than the other?
Is this project experimental or is it supposed to be a cornerstone for the next few years?
Are there existing implementations/frameworks of what needs to be done in that language? Does it fit the need or does it need to be altered?
Etc.

If we look at your list, here is my take:

ETL

ETL is quite a language agnostic, both Java and Python have good bindings to tools necessary for ETL.

Data pre-processing

Similar to ETL, although Python may be easier to do simple things.

Making use of pre-trained models / transfer learning

Python is much more convenient as most existing frameworks and pre-trained models are compatible with Python, and likely not with Java.

Distributed computing

The Hadoop stack is very Java friendly. Good to note that Python is increasingly become easier to use with Hadoop products (but Java is still running underneath)

Processing large volume of data (images,time series data,sensor data..etc)

Although Python has a reputation of being slow, if you know what you're doing, most Python libraries you'll use will ultimately run C/C++ code anyway, so speed shouldn't be a huge concern.

Production implementation (batch or real time / online)

Java is statically typed which offers a lot of advantages for production, it's becoming steadily easier to do this with Python as well (i.e. pydantic, mypy, etc.)

Post production training (bacth or real time / online)

This is independent of the language, this has more to do with your MLOps and your pipelines in place.

Mobile app development

Java definitely has the edge, although in a world of microservices, Python can prove very helpful.

IoT edge device computing

When it comes to device computing, Java probably has the edge, although, C/C++ are likely your best allies.


How to use Chi-square test in dataset with negative values I could not fully explain the title. In order to use the Chi-square test in my dataset, I am finding the smallest value and add each cell with that value. (for example, the range of data here is [-8,11] so I added +8 to each cell and the range turned to [0,19]).

dataValues variable is DataFrame type that holds my all data and contains ~2000 features, ~1000 rows, dataTargetEncoded variable is Array type that contains results as 0 and 1.

for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] += 8

#fit chi2 and get results
bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances = pd.Series(fit.scores_, index=dataValues.columns)

#print top 10 feature
print(feat_importances.nlargest(10).index.values)

# back to normal
for i in range(len(dataValues.index)):
    for j in range(len(dataValues.columns)):
        dataValues.iat[i, j] -= 8

But this causes performance problems. Another solution I'm thinking of is to normalize it. I wrote a function that looks like this:

def normalization(df):
    from sklearn import preprocessing

    x = df.values  # returns a numpy array
    min_max_scaler = preprocessing.MinMaxScaler()
    x_scaled = min_max_scaler.fit_transform(x)
    df = pd.DataFrame(x_scaled, columns=df.columns)

return df

My program has accelerated lots, but this time my accuracy has decreased. The feature selection process I have done with the first method produces 0.85 accuracy results, this time I am producing 0.70 accuracy.

I want to get rid of this primitive method, but I also want accuracy to remain constant. How do I proceed? Thank you in advance

First and foremost, it does not matter to the chi-square test whether your data is positive, negative, string or any other type, as long as it is discrete (or nicely binned). This is due to the fact that the chi-square test calculations are based on a contingency table and not your raw data. The documentation of sklearn.feature_selection.chi2 and the related usage example are not clear on that at all. Not only that, but the two are not in concord regarding the type of input data (documentation says booleans or frequencies, whereas the example uses the raw iris dataset, which has quantities in centimeters), so this causes even more confusion. The reason why sklearn's chi-squared expects only non-negative features is most likely the implementation: the authors are relying on a row-by-row sum, which means that allowing negative values will produce the wrong result. Some hard-to-understand optimization is happening internally as well, so for the purposes of simple feature selection I would personally go with scipy's implementation.

Since your data is not discrete, you will have to bin every feature into some number of nominal categories in order to perform the chi-squared test. Be aware that information loss takes place during this step regardless of your technique; your aim is to minimize it by finding an approach that best suits your data. You must also understand that the results cannot be taken as the absolute truth since the test is not designed for data of continuous nature. Another massive problem that will definitely mess with your feature selection process in general is that the number of features is larger than the number of observations. I would definitely recommend taking a look at sklearn's decomposition methods such as PCA to reduce the number of features, and if your features come in groups, you can try Multiple Factor Analysis (Python implementation available via prince).

Now that that's out of the way, let's go through an example of simple feature selection using the iris dataset. We will add a useless normally distributed variable to the constructed dataframe for comparison.

import numpy as np
import scipy as sp
import pandas as pd

from sklearn import datasets, preprocessing as prep

iris = datasets.load_iris()

X, y = iris['data'], iris['target']
df = pd.DataFrame(X, columns= iris['feature_names'])
df['useless_feature'] = np.random.normal(0, 5, len(df))

Now we have to bin the data. For value-based and quantile-based binning, you can use pd.cut and pd.qcut, respectively (this great answer explains the difference between the two), but sklearn's KBinsDiscretizer provides even more options. Here I'm using it for one-dimensional k-means clustering to create the bins (separate calculation for each feature):

def bin_by_kmeans(pd_series, n_bins):
    binner = prep.KBinsDiscretizer(n_bins= n_bins, encode= 'ordinal', strategy= 'kmeans')
    binner.fit(pd_series.values.reshape(-1, 1))
    bin_edges = [
        '({:.2f} .. {:.2f})'.format(left_edge, right_edge)
        for left_edge, right_edge in zip(
            binner.bin_edges_[0][:-1],
            binner.bin_edges_[0][1:]
        )
    ]
    return list(map(lambda b: bin_edges[int(b)], binner.transform(pd_series.values.reshape(-1, 1))))

df_binned = df.copy()
for f in df.columns:
    df_binned[f] = bin_by_kmeans(df_binned[f], 5)

A good way to investigate how well your individual features are binned is by counting the number of data points in each bin (df_binned['feature_name_here'].value_counts()) and by printing out a pd.crosstab (contingency table) of the given feature and label columns.

  An often quoted guideline for the validity of this calculation is that the test should be used only if the observed and expected frequencies in each cell are at least 5.

So the more zeroes you see in the contingency table, the less accurate the chi-squared results will be. This will require a bit of manual tuning.

Next comes the function that performs the chi-squared test for independence on two variables (this tutorial has very useful explanations, highly recommended read, code is pulled from there):

def get_chi_squared_results(series_A, series_B):
    contingency_table = pd.crosstab(series_A, series_B)
    chi2_stat, p_value, dof, expected_table = sp.stats.chi2_contingency(contingency_table)
    threshold = sp.stats.chi2.ppf(0.95, dof)
    return chi2_stat, threshold, p_value

The values to focus on are the statistic itself, the threshold, and its p-value. The threshold is obtained from a quantile function. You can use these three to make the final assessment of individual feature-label tests:

print('{:<20} {:>12} {:>12}\t{:<10} {:<3}'.format('Feature', 'Chi2', 'Threshold', 'P-value', 'Is dependent?'))
for f in df.columns:
    chi2_stat, threshold, p_value = get_chi_squared_results(df[f], y)
    is_over_threshold = chi2_stat >= threshold
    is_result_significant = p_value <= 0.05
    print('{:<20} {:>12.2f} {:>12.2f}\t{:<10.2f} {}'.format(
        f, chi2_stat, threshold, p_value, (is_over_threshold and is_result_significant)
    ))

In my case, the output looks like this:

Feature                      Chi2    Threshold  P-value    Is dependent?
sepal length (cm)          156.27        88.25  0.00       True
sepal width (cm)            89.55        60.48  0.00       True
petal length (cm)          271.80       106.39  0.00       True
petal width (cm)           271.75        58.12  0.00       True
useless_feature            300.00       339.26  0.46       False

In order to claim dependence between the two variables, the resulting statistic should be larger than the threshold value and the p-value should be lower than 0.05. You can choose smaller p-values for higher confidence (you'd have to calculate the threshold from sp.stats.chi2.ppf accordingly), but 0.05 is the "largest" value needed for your results to be considered significant. As far as ordering of useful features goes, consider looking at the relative magnitude of the difference between the calculated statistic and the threshold for each feature.


New to data Science. Which techniques best to use Large data set in insurance company? Dataset Features

Insurance underwriting dataset for 8 years.

age
location
amount insured
some other features...(medical evidence)

Not all feature will be available to all applicants.

Target Variable

Decision on whether the applicant can be insured

Question

What techniques can be used and which ones would work best?
Outline a high-level overview - I do not think I have to go into too much detail as I do not have any data.

Things I have considered

I am thinking to first slice the data and analyse it in parts and see can I find  a pattern.

Regression analysis could be carried out. Possible Logistic regression?

I could take a sample and perform hypothesis tests?

I know that this is an ideal machine learning situation. I don't have any experience in this field and I think I am better to stick with methods I have some knowledge of.

I know this is very ambiguous, but a little nod in the right direction and I would be very appreciative.

I think you need to do couple of tests to see what all variables are important with respect to your Target Variable(client can be insured: Yes/No) - this kind of test is called Predictor Importance test.

As you you have mentioned this is sector is new for you, I would suggest you to take all the variable you think are useful. Convert categorical variables to factors using as.factor() numeric variable to numeric as as.numeric(). The reason for explicit transformation is sometimes algorithms cannot understand like:

test1 <- c(1,2,2,4,4,1,2)
summary(test1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.000   1.500   2.000   2.286   3.000   4.000
test1 <- as.factor(test1)
summary(test1)
  1 2 4
  2 3 2

Now once the data is ready, you can give them as an input to Boruta, by using this you get the predictor importance graph.
For better understanding you can go through this Link
or else if you want to learn why, how and when you can go through this Link for different tests for different variables.

Boruta does all the above by itself and the outcome is set of important features, With Respect to that you can feed those respective features to your model for getting better results/accuracy.

As your problem is Binary Classification, you can use the following Algorithms:

Decision Trees
Random Forest
Bayesian Network
Support Vector Machines(SVM)
Neural Network

For all the above algorithms I've attached a link for your reference in which you can find at-least one example for each.

Hope my answer is helpful, mark as answer if you got what you needed.


Is there any way to analyze the format of text strings? I have a lot of data which basically consists of alphanumeric text on individual lines which can very in length and contain delimiters.
Since there are many thousands of lines of text, I'm looking to see whether there is an automated way to determine the different formats of text.
A sample of which is:
90665013-163
90731046-103
90840069-009
90847069-009
90880046-103
90889046-103
90897-051
9089744-103
9089844-103
90901-46909
90901-lep
9091046-103
9091046-909
90764046-1037
can10043E
can90065-op016
9094344-103
90669j4-4438718
90666ie79
90664046-103
90710-077
004-919
4A1900935
can90064-op016
can90066-E016
9094544-103
9094646-103
4A1900597
4A1900588
4A9443198
4A94431

So, from this sample, we can see that there are several lines that are the format of 8 numbers, a dash, and then three numbers.
Another format is 3 numbers, a dash, and then another 3 numbers.
I'm looking for a way to determine all of the unique formats available...
Not sure whether this is possible either by some code, an online web service, or perhaps a feature of some software but I'm asking on the off chance this actually possible.

A machine learning approach to this would be create character-level encoding for the data. Then, you can run K Means (use silhouette score for judging the fit) or Louvain community detection (doesn't need the number of clusters as input). You can finally look at cluster heads for the formats.


Is there a deep learning method for 3D labels? As the question says, I want to feed labels into a neural net that are three dimensional. Let's say that I have 3 possible labels and each one of my data points corresponds to a percentage of those labels. e.g, my first datapoint contains 20% of label A, 30% of label B, and 50% of label C.
Is there any architecture able to deal with this shape of label data?

Since the probability are summing up to zero, so you can simply treat it as Multi-class problem and use a network with Softmax at the end.
Last layer and compile -

model.add(keras.layers.Dense( 3, activation="softmax"))
model.compile( optimizer='adam, loss="categorical_crossentropy", metrics='accuracy')

Metrics - Accuracy is not appropriate. Define a custom metrics based on the interpretation of 3 probabilities
The labels will be as per the probability-
e.g. This is for MNIST 10 digits -

Digit 1 - [0.05, 0.55, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]

Prediction - [0.064, 0.356, 0.059, 0.069, 0.068, 0.050, 0.044, 0.122, 0.064, 0.101]

Code for MNIST - Colab link


Intuition in Backpropagation (gradient descent) I am reading about backpropagation, one of many processes involved in gradient descent, and I couldn't quite grasp the intuition in one of the equations.

http://neuralnetworksanddeeplearning.com/chap2.html#MathJax-Element-185-Frame

  An equation for the error $\delta^l$ in terms of the error in the next layer, $\delta^{l+1}$: In particular

  $$
\delta^l=((w^{l+1})^T\delta^{l+1})\circ\sigma'(z^l),
$$

  where $(w^{l+1})^T$ is the transpose of the weight matrix $w^{l+1}$ for the $(l+1)^{th}$ layer.

  ...

  When we apply the transpose weight matrix, $(w^{l+1})^T$, we can think intuitively of this as moving the error backward through the network, giving us some sort of measure of the error at the output of the $l^{th}$ layer.

I know how each terms are derived algebraically (from further down the text), but I can't grasp the intuitive part. In particular, I don't understand how applying the transpose of the weight matrix is like moving the error backwards.

The mathematics is quite straightforward. The equation is the simplification of the error in layer $l$ expressed in terms of error in layer $l+1$. But, the whole intuition bit doesn't come so naturally.

Could someone please explain, in what sense applying the transpose is like "moving the error backwards"?

in what sense applying the transpose is like "moving the error backwards"?

It isn't. Or at least you shouldn't think too hard about the analogy.

What you are actually doing is calculating the gradient - or partial derivative - of the error/cost term with respect to the activations and weights in the network. This is done by repeatedly applying the chain rule to terms that depend on each other in the network, until you have the gradient for all the variables that you control in the network (usually you are interested in changing the weights between neurons, but you can alter other things if they are under your control - this is exactly how deep dreaming works too, instead of treating the weights as a variable, you treat the input image as one).

The analogy is useful in that it explains the progression and goal of the repeated calculations across layers. It is also correct in that values of the gradient in one layer depend critically on values in "higher" layers. This can lead to intuitions about managing problems with gradient values in deep networks.

There is no relation between taking the matrix transpose and "moving the error backwards". Actually you are taking sums over terms connected by the weights in the NN model, and that happens to be the same operation as the matrix multiplication given. The matrix multiplication is a concise way of writing the relationship (and is useful when looking at optimisations), but is not fundamental to understanding neural networks.

The statement "moving the error backwards" is in my opinion a common cause of problems when developers who have not studied the theory carefully try to implement it. I have seen quite a few attempts at implementing back propagation that cargo cult in error terms and multiplications without comprehension. It does not help here that some combinations of cost functions and activation functions in the output layer have been deliberately chosen to have very simple derivatives, so that it looks like the neural network is copying error values around with almost magical multiplications by activation value etc. Actually deriving those simple terms takes a reasonably competent knowledge of basic calculus.


Is is possible to build a recommender system with this dataset? I have an extremely sparse user items ratings matrix with 0.018 % non NA values. Correct me if I am wrong but I think we need a lot of products compared to number of users to build a recommender system. I have around 20,000 users and 50,000 products.

Is is possible to build a recommender system with this dataset? If so, what would be the most appropriate approach?

It is definitely possible to build a recommender system with the dataset you have.

Although there are a few questions.
1. What is your objective from the recommender system?
2. What approach do you want to take with each recommendation?
3. What is the final goal of the recommendation?

Novice Approach:
A very simple approach could be to recommend similar products, for e.g. if a person is looking at a phone A (on website) you suggest an alternative as phone B purely based on the similarity of the product and nothing else.

There are three steps to Recommender System:
1. Query Identification - What to Recommend?
2. Candidate Generation - How many to Recommend?
3. Ranking - In what order to Recommend?

If you answer these questions you will have a fairly good recommender system.


What is the interpretation of the expectation notation in the GAN formulation? I'm confused about the expectation notation in the context of GAN loss functions.

The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not.

real = $D(x)$ (ie: give the discriminator a real image)
fake = $D(G(z))$  (ie: generate a fake image and ask discriminator what it is)

Then the binary crossentropy is:
$$log(p) - log (1-p)$$

When used as a GAN loss we replace p with either "real class" or "fake class".

$$log(real) - log (1-fake)=\\
log(D(x)) - log (1-D(G(z)))$$

So far, this is ok (i think haha).

But the actual formulation adds an expectation sign... which I don't understand why it's there.

$$E_{x~data}log(D(x)) - E_z log (1-D(G(z)))$$

The functions $$ L_a(x) = \log(D(x))\;\;\;\;\&\;\;\;\; L_b(z) = \log(1-D(G(z))) $$
are defined for one value (either $x$ or $z$).

We don't want to optimize $L_a$ or $L_b$ for one input, we want to minimize it for all possible inputs $x$ and $z$. Suppose $x\in X\equiv\mathbb{R}^n$ (e.g., by "unfolding" it as an image) and $z\in Z\equiv \mathbb{R}^m$. Then we can write minimizing over all possible inputs as minimizing $\int_{\mathbb{R}^n} L_a(x)\,dx $ and $\int_{\mathbb{R}^m} L_b(z)\, dz$. But there's something wrong with this! In fact, we don't care about, say, pure red images or electron microscope micrographs (if $x$ were images), we only care about "real" photographs, i.e., the "natural images". So instead of optimizing over all $X$ and $Z$, we prefer to optimize over the ones that "matter". We do this by placing a probability distribution on $X$ and $Z$, say $P_x$ and $P_z$, such that sampling from these distributions gives us the $x$ and $z$ values that we care about. Low probability is given to unnatural images $x$; high probability is given to e.g. images of dogs and cats.
For GANs, we often choose $P_z$ in advance, as a uniform or normal distribution.
(Some people prefer to say that the natural images form a manifold in high-dimensional space; this $P_x$ can be thought of as inducing that structure.)

But how to optimize over these "likely" images. The obvious thing to do is optimize the expected value over these densities. In other words, we want to optimize the values of $L_a$ and $L_b$ that we expect to get if we compute them on samples drawn from $P_x$ and $P_z$. This is a weighted average over all of data space, where the weight is given by the probability of seeing that data point:
$$
\mathbb{E}_{x\sim P_x}[L_a(x)] = \int_X L_a(x) p_x(x)\, dx
\;\;\;\;\&\;\;\;\;
\mathbb{E}_{z\sim P_z}[L_b(z)] = \int_Z L_b(z) p_z(z)\, dz
$$
where this integral (expectation) is approximated by a sum or average over a minibatch in practice.

In summary, we add the expectations because we care about how we do across all "important" $x$ and $z$ values, not just one.

The explanation in this post might also be of use.


CNN accuracy and loss doesn't change over epochs for sentiment analysis I am performing text classification as Good [1] or Bad [0]. The texts are preprocessed and converted to Vectors using Google Word2Vec. Further CNN architecture is used for training. I have roughly 13000 texts as Bad[0] and 5450 texts as Good[1] for training (making it a roughly 70:30%)
The issue starts when I realize I don't have enough compute power (2GB GPU). Hence I compromise and use 100 dimensions of word embeddings from Word2Vec (instead of the 300). After certain hyperparameter tuning in the CNN architecture, I am able to obtain a 30-35% precision, which I am happy with.
After months, I have an 8GB GPU in the server and I implemented the 300-dimensional word embeddings in Word2Vec and kept for training. Ideally, I should have obtained better results; instead, the loss and accuracy don't change with time for every epoch. Thus it predicts all texts as Bad[0].
Can you please help me identify the problem and If I am missing out anything here!

EDIT:

I would like to add some clarifications:
In the Linux server with GPU 1070Ti 8GB, I tried with three experiments in this order: a) 300-dimensional word embeddings b) 100-dimensional word-embeddings and c) 105-dimensional word embeddings
I have obtained no change in accuracy and loss for a) and c). However, for b), it is exactly the same results as I obtained with my local GPU(750Ti Nvidia 2GB). In short, its working fine for 100-dimensional in the server.

Now, since I can't assign 300d word vectors in my local GPU, I make up the same experiment as c) with 105d vectors in the local GPU to just check if there's any fault in code, and surprisingly it's giving around 30% precision much similar to earlier results.

I am having a hard time figuring out the issue in the server GPU as its working fine for 100d word vectors but fail to give proper predictions for other dimensional word-embeddings.

I am attaching some results which might make it a lot clear:

1.) Trained with 100d vectors in local(2GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.31**      0.17      0.22       850

avg / total       0.58      0.63      0.60      2823

2.)  Trained with 100d vectors in server (8GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.77      0.73      1973
    1.0       **0.30**      0.23      0.26       850

avg / total       0.58      0.61      0.59      2823

3.) This is the same result for both 105d and 300d vectors trained in server GPU
             precision    recall  f1-score   support

    0.0       0.70      1.00      0.82      1973
    1.0       **0.00**      0.00      0.00       850

avg / total       0.49      0.70      0.58      2823

4.) Trained with 105d vectors in local(2GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.30**      0.18      0.23       850

avg / total       0.57      0.64      0.61      2823

You have imbalanced Data, that's why your network always predicts the most frequent answer (he is in a local minima).

There are solutions to avoid this kind of problem:

Balance your data, use as much "good" as "bad" in your training set.
Change your cost function,  e.g. the cost of error on "good" should be higher than "bad".

(don't use both at the same times or your model will be imbalanced in the other class)

If you look for imbalanced data on google you will find lot of articles on this problem, since it is common.


Credit card fraud detection - anomaly detection based on amount of money to be withdrawn? I am trying to figure out how the amount of money that a customer would want to withdraw on an ATM tell us if the transaction is fraudulent or not.There are other attributes, of course, but now I would want to hear your views on the amount of money that the customer wants to withdraw.

Data may be of this form:

Let us assume that a customer, for ten consecutive transactions, withdrew the following amounts:

100.33, 384 , 458, 77.90, 456, 213.55, 500 , 500, 300, 304.

Questions:

How can we use this data to tell if the next transaction done on this account is fraudulent of not?
Are there specific algorithms that can be used for this classification?

What I was thinking:

I was thinking to calculate the average amount of money, say for the last ten transactions, and check how far is the next transaction amount from the average. Too much deviation would signal an anomaly. But this does not sound much, does it?

What's the underlying model of how much someone requests from an ATM? It doesn't seem like it's a simple distribution like a Gaussian, where comparing new amounts to the mean is sensible. Consider a person who always pulls out either \$40 or \$400. Ideally we want to build a distribution of what normal transactions from a user look like, and notice if new datapoints don't look like they're sampled from that distribution.

idclark's suggestion, to look at the nearest n datapoints from that user and compute the distance from just them, is a good and fast implementation of that sort of test.

One other possibility is to try to find similar users, and then aggregate data across users. If I only have 10 withdrawals from each user, I'm not going to be able to reject any new withdrawals with confidence, but if I have seven clusters of users, with a thousand withdrawals per cluster, I can notice when a user who was in a particular cluster deviates from the overall cluster distribution. (This also helps you make use of knowledge about which previous transactions were fraudulent.)


In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data? Given a classifier using softmax, is it possible that say, for data point a which our model has correctly classified, has higher loss compared to data point b which is misclassified by our model?
If yes, please bring an example. If not, please prove.
================================================================
Update: I forgot to mention consider cross-entropy loss.

Assuming cross-entropy loss (as you specified in the comment), it can happens.
The reason is that the loss in cross-entropy is "activated" only for the probability of the ground truth (t_i is 1 for the ground truth and 0 otherwise).
Imagine a classification task with five classes, with the two following probability situations:
(a) 0.4    0.5    0.04  0.03   0.03

(b) 0.3    0.2    0.2     0.15  0.15
Assume that the true label is the first class.
In situation (a) the predicted class is the second one, but the probability of the first class (which is the only one which matters) is 0.4.
In situation (b) the predicted class is the first one (so the right one), but the probability of the first class is less than the situation (a).
Since cross-entropy cares only of the probability of the true label, in the first case you will obtain a lower loss because the probability is higher, even if the predicted class is the wrong one.
This phenomenon is more probable as the number of classes grows. For few classes, this is highly improbable or even impossible (think to the binary classification case, where this can't happen).
Cheers


ship class Object detection with custom model weights I have made a model to classify different categories of ships(yacht,catamaran,rubber boat) in Python and i hit a 70%  accuracy in training so now i have my weights.hdf5 file.Now i need to object detection the images i used with boundaries boxes and rectangles to find the different ships in each image, like a Yolo pretrained model but with my own weights that i trained for classes,so that the detector is going to find the ship in the image and display is's label with is's probability,
etc yacht(70.0% )rubber boat(65.0%)...
Can anyone with some more experience drive me to achieving this task?

You should be able to use keras.models.load_model to load the hdf5 model file. See also the tensorflow documentation.
# saving the model
model = ...  # Get model (Sequential, Functional Model, or Model subclass)
model.save('weights.hdf5')

# loading the model
from tensorflow import keras
model = keras.models.load_model('weights.hdf5')


Data Science Career: From Researcher To Data Scientist I’m currently working as a Research Assistant in Computer Science, specializing in both Human-Computer Interaction (HCI) and Health Informatics (HI) fields. As part of my role, I collect data from several clinical professionals (clinicians, physicians, and doctors) and analyze it (e.g., t-test, ANOVA and so on). Transforming the analyzed data and organizing it into several datasets, we report and use it by creating scientific papers that will fundament our analysis.

Recently, I started to have an interest in the Data Science field, and I’m considering to develop in that direction and become a Data Scientist. As I mentioned, I’m already applying statistical analysis as part of my job. That rises few questions:

How do the HCI and Data Science areas relate to each other?
How can I shift from being a Research Assistant in the HCI field into Data Science career?
What kind of studying/courses should I take?

Great question! I can appreciate that you have a base of statistics to work from. Most data scientists - especially the ones that go to these data "camps" or whatever - out there do not have this and it should serve you well in your career. Here are some answers to your questions, in order:

Your background will help you because you are used to things like study design and execution and working with data. But I don't seen anything specific to HCI that would help you. When you are a good data scientist, the data itself takes on a muted importance. Right now, when I work on a project, I care that the data was collected in a sustainable way, but what the data is doesn't really matter. Eventually you will reach this too - it's all just numbers and math :-)
You have the statistics side. So now you need the (1) programming skills and the (2) the modeling skills. You should choose to learn a language like Python or R and develop your skills there. From there, you should start looking at the relevant libraries like Tensorflow and Keras to help you build your modeling skills. You should also pick up a textbook or some other resource that talks about what models actually do and how you can tune them, agnostic of the language you choose. Finally, you'll be ready for some research on neural networks and how those work. The measure of success here is being able to write some "traditional" models in Tensorflow (like regression, decision trees, etc) and also being able to write a neural network in Keras.

I think that your base of already working around experiments and (presumably) applying the scientific method will serve you well and already puts you ahead of most data scientists out there. Now, it's just a matter of being able to do all of that with code and, if you follow the steps above, I think you will see those connections and how you can enter the world of data science.


How to visualize multi-instance multiclass classification? Let's say I have 3 classes and 1 score for each data point

score z
class 1
class 2
class 3

E.g. the input looks like:

0.529 5 7 4
0.310 3 4 2
0.774 10 7 6
0.774 10 8 5
0.172 3 0 2

In code:

>>> import pandas as pd
>>> df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0.31, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [0.172, 3.0, 0.0, 2.0]])
>>> df
       0     1    2    3
0  0.529   5.0  7.0  4.0
1  0.310   3.0  4.0  2.0
2  0.774  10.0  7.0  6.0
3  0.774  10.0  8.0  5.0
4  0.172   3.0  0.0  2.0

Each row represents a data point and column 0 is the score and column 1,2,3 are some sort of coefficients for the classes.

The score is some sort of computed entity independent of the coefficients and the goal is to visualize the interaction between the coefficients and the score.

How to visualize the interaction between the 3 classes (column 1,2,3) and the score (column 0)?

You could use a 3d-scatter plot, where each class would correspond to one axis and the color-intensity of the point would indicate the score value(e.g. for a grayscale colormap, the whiter the closer to 1 the value of the score).

Using the above format:

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [1, 3.0, 0.0, 2.0]])
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(df[1].values, df[2].values, df[3].values, c=df[0].values, cmap='gray', s=50, vmin=0.,vmax=1)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()

You can change the scatter plot attributes for marker-size, colormap etc. according to the documentation to match your tastes.


Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation? I'm working on neural machine translator that translates English sentences to American sign language sentences(e.g below). I've a quite small dataset - around 1000 sentence pairs. I'm wondering if it is possible to fine-tune BERT, ELMO or XLnet for Seq2seq encoder/decoder machine translation.

English: He sells food.

American sign language: Food he sells

You can view models like ELMo or BERT to be encoder-only. They can be easily used for classification or sequence tagging, but the tag sequence is typically monotonically aligned with the source sequence. Even though the Transformer layers in BERT or XLNet are in theory capable of arbitrary reordering (which is used in non-autoregressive machine translation models), this is not what BERT or XLNet were trained for and therefore it will be hard to finetune for that.

If at least the vocabulary is the same on both the source and target side, I would recommend pre-trained sequence-to-sequence models: MASS or BART.

If the both the grammar and vocabulary and grammar of the sign language are quite different, maybe using BERT as an encoder and training your own lightweight autoregressive decoder might be the correct way.


Is a large number of epochs good or bad idea in CNN In my CNN model, by using large number of epochs like 400 or above, the validations accuracy and some times test accuracy gets better, but I think this large number of epochs is not good idea? I am right or not? why?

Assuming you track the performance with a validation set, as long as validation error is decreasing, more epochs are beneficial, model is improving on seen (training) and unseen (validation) data.

As soon as validation error starts to increase, it signals that model is over-fitting on training data, thus the learning process should be stopped. This method is independent of the details of a model.

In practice, save the model corresponding to the best validation error found so far, and stop the process only when validation error is consistently increasing. This can be done manually or with an ad-hoc threshold based on the distance of current validation error from the minimum error so far. This way, learning will not be stopped prematurely due to fluctuations in validation error.


Kmodes for Mixed Data Kmodes is for categorical Data. But sometimes I read it somewhere that it is applicable on Mixed Data i.e., Numeric and categorical.

So whats right?

As far as I remember Kmodes is used for Categorical data, even in the documentation I couldn't find anything related to Mixed Data Type, if you have some reference do share.
I've used Kproto for Mixed Data types as you have mentioned and the package is ClustMixType, it combination of Kmeans and Kmodes.
Do have a look and let me know if have any questions.


how to use (read) google pre-trained word2vec model file? I am trying to apply open() function in keras to use Google news-vectors-negative300.bin which is a pre-trained file via word2vec such as GloVe, but after downloading GloVe it contains 4 files with txt prefix vs the Google news-vectors-negative300.bin folder contains a file with binary prefix namely 'data' which is 3.4 GB.
I write the commands on ubuntu 17.10 via keras with tensorflow backend on spyder with python 3.5, and after implementing the command it gave me this error:

File "/home/mary/anaconda3/envs/virenv/lib/python3.5/codecs.py", line 321, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 19: invalid start byte.

the written code is as follow: f = open('data').
I have already implemented the same code successfully when I applied
f = open('glove.6B.100d.txt').
What is the main problem?

I have searched about it and fixed the error through these steps:
you should load the "GoogleNews-vectors-negative300.bin.gz" file at first then extract it by this command in Ubuntu: gunzip -k GoogleNews-vectors-negative300.bin.gz. [ manually extracting is never recommended]. Secondly, you should apply these commands in python 3:

import gensim
model = gensim.models.Word2Vec.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)

I hope it will be useful.


Building a static local website using Rmarkdown: step by step procedure I am trying to understand the procedure of building a static local website using R and Rmarkdown. I am aware of a Rmarkdown website where the procedure is outlined, but unfortunately I do not understand the steps.

Does anybody here have some experience in building a static local website and would be so kind as to describe the procedure in more detail?

In most things, related to R, there are many approaches to solve a problem, sometimes too many, I would say. The task of building a static website, using RMarkdown, is not an exception.

One of the best, albeit somewhat brief, sets of workflows on the topic include the following one by Daniel Wollschlaeger, which includes this workflow, based on R, nanoc and Jekyll, as well as this workflow, based on R and WordPress. Another good workflow is this one by Jason Bryer, which is focused on R(Markdown), Jekyll and GitHub Pages.

Not everyone likes GitHub Pages, Jekyll, Octopress and Ruby, so some people came up with alternative solutions. For example, this workflow by Edward Borasky is based on R and, for a static website generator, on Python-based Nicola (instead of Ruby-based Jekyll or nanoc). Speaking about static website generators, there are tons of them, in various programming languages, so, if you want to experiment, check this amazing website, listing almost all of them. Almost, because some are missing - for example, Samantha and Ghost, listed here.

Some other interesting workflows include this one by Joshua Lande, which is based on Jekyll and GitHub Pages, but includes some nice examples of customization for integrating a website with Disqus, Google Analytics and Twitter as well as getting custom URL for the site and more.

Those who want a pure R-based static site solution, now have some options, including rsmith (https://github.com/hadley/rsmith), a static site generator by Hadley Wickham, and Poirot (https://github.com/ramnathv/poirot), a static site generator by Ramnath Vaidyanathan.

Finally, I would like to mention an interesting project (from an open science perspective) that I recently ran across - an open source software by Mark Madsen for a lab notebook static site, which is based on GitHub Pages and Jekyll, but also supports pandoc, R, RMarkdown and knitr.


Neural Network Sensordata as Input I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values.

Each recording of a movement has about 1000 lines contained in a csv file.

My problem: I don't know how to present those recordings to a neural network (TensorFlow) so that it can be trained on the movement classes and even predict what was done in recording by getting those 19000 values.
I don't even know which Neural Network Model I should use and therefore need your help.

Correct me if I'm wrong: As far as I understood, you have files of 1000 entries with 19 numerical sensor values and a type of 22 movements in each line.
You would like to predict the type of movement based on the sensor values.

Are the entries being recorded in a fixed interval?
If so, you could use an RNN.

Either way, it seems to be a classification problem. You can find a tutorial for Tensorflow here.

You can read data from csv files with pandas.read_csv.

As for the encoding of your data: If the types of movement cannot be sorted and are not related to each other, you should use one-hot encoding to present the type of movement to the NN. You can keep numerical sensor data as is, just make sure to normalize it.


Should bias updates be porportional to overfitting? According to questions on the internet, the bias is a learnable parameter,
 and there are different solutions to updating it, but I failed to find a concise methodology of
 correctly updating biases during training.

When I tried to overfit a small network, it failed when the bias was introduced into the training:

When I tried to scale down bias updates it produeced similar, but delayed results:

Next I tried to make bias updates porportional to the train set error, again only delaying the trend:

Next I tried to make bias updated inversely porportional to the error in the training set, but I suppose this would not have any shown benefits without a validation set. Alas the effect was the same:

According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.

Based on this I would suppose the more
overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?

I think you are mixing up the bias of a model as in here, with the bias terms of a neural network which are just the constant term of the linear model of each layer. Updating the biases for training will not reduce overfitting since each bias is an additional parameter of the model.
Remember that the weights (and the bias is also a weight) are updated proportionally to the negative gradient of the loss function. Therefore there must be an error in your implementation since the training error gets larger which is highly unlikely for gradient descent (unless your learning rate is far too high).


Multi-Feature One-Hot-Encoder with varying amount of feature instances Let's assume we have data instances like this:
[
    [15, 20, ("banana","apple","cucumber"), ...],
    [91, 12, ("orange","banana"), ...],
    ...
]

I am wondering how I can encode the third element of these datapoints. For multiple features values we could use sklearn's OneHotEncoder, but as far as I could find out, it cannot handle inputs of different length.
Here is what I've tried out:
X = [[15, 20, ("banana","apple","cucumber")], [91, 12, ("orange","banana")]]

ct = ColumnTransformer(
    [
        ("genre_encoder", OneHotEncoder(), [2])
    ],
    remainder='passthrough'
)
print(ct.fit_transform(X))

This will only output
[[1.0 0.0 15 20]
 [0.0 1.0 91 12]]

as expected, because the tuples are handled as the possible values this feature can be represented with.
We can't embed our features directly (like [15, 12, "banana", "apple", "cucumber"]), because

we don't know how many instances of this feature we will have (two? three?)
each position would be interpreted as an own feature and thus if we had banana in the first nominal slot in one datapoint and in the second one in our second nominal slot, they would not count to the same "pool of values" a feature can embody

Example:
X = [["banana","apple","cucumber"], ["orange","banana", "cucumber"]]
enc = OneHotEncoder()
print(enc.fit_transform(X).toarray())

[[1. 0. 1. 0. 1.]
 [0. 1. 0. 1. 1.]]

This representation contains 5 slots instead of 4, because the first slot is interpreted as using banana or orange, the second one as apple or banana and the last one only has the option cucumber.
(This would also not solve the problem of having different amounts of feature values per datapoint. And replacing empty ones with None does not solve the problem either, because then None faces this positional ambiguity.)
Any idea how to encode those "Multi-Muliti-"features, that can take multiple values and consist of a varying amount of elements? Thank you in advance!

I think you can transform this into a text preprocessing problem and then use CountVectorizer. You basically build "documents" by putting together all the words in your raw data and then use CountVectorizer on those documents.
from sklearn.feature_extraction.text import CountVectorizer

X = [["banana","apple","cucumber"], ["orange","banana", "cucumber"]]
# Create documents
X_ = [' '.join(x) for x in X]
enc = CountVectorizer()
print(enc.fit_transform(X_).toarray())

Returns
[[1 1 1 0]
 [0 1 1 1]]

which has 4 different values as you expected.


best NN architecture for point prediction I'm training to predict a single value y (continuos in [0,1]) based on a number of variables x1, ..., x45. These are variables related to various measurements of the same value using different techniques -  some of them may be irrelevant.

1) Would it make sense to use a CNN? I read that it works best for things like image recognition/classification, and this is clearly not the case

2) Which NN would work best? The benchmarks would be the naive linear regression and the average x

Since your outcome $y$ is restricted, you should check „beta regression“ in which the outcome is „squeezed“ into the interval $ \hat{y} \in [0, 1]$. See an example in R here: http://r-statistics.co/Beta-Regression-With-R.html

To my best knowledge, there is no "off the shelf" NN equivalent of beta regression.

You could of course use a "normal" NN for regression with dense layers with a sigmoid output layer. I.e. use a normal regression setup and change that last layer to have a sigmoid output (which is between 0,1).

What could be interesting as well is a boosted beta regression: https://cran.r-project.org/web/packages/betaboost/betaboost.pdf

Beta regression is also available for GAM (generalised additive models), which are very flexible in handling non-linearities (https://stat.ethz.ch/R-manual/R-patched/library/mgcv/html/Beta.html).

Ultimately, you need to check which approach works for you given your objectives and second order restrictions. It could (probably) also be okay to use a normal boosting approach (e.g. LightGBM or Catboost) - so without a beta link function. In this case some of the $\hat{y}$ could be $>1$ or $<0$. If you do not care too much about few "overshooting" predictions, you could also achieve good results with "normal" regression.


Interactive dashboard for time series data I'm searching for some tools that allow me to build a dashboard in order to visualize information about time series. This dashboard needs to be interactive and allows to be integrated into a web site (like a web application).

I made some projects with D3.js, but for this one I would prefer something faster in term of implementation (python+matplotlib like).

What kind of options do I have?

D3 (data Driven Documents) is a great and powerful tool, but does require a certain afinity for JavaScript. There are a few tools that piggy-back off it though and are more friendly to the aspiring data scientist and plotly is behind a few others. All are easier to use compared to D3 in my opinion.

Check out a few of the following:

Bokeh (python)
Dash (python - based on plotly, looks very promising!)
Plotly (many languages)
Shiny (R)

Here is a comparison of Bokeh and Dash.

Here is a related question on StackOverflow.


Changing the default scale of a plot to a "custom scale" I am trying to change the scale of the x_axis for a plot.

The default is generating divisions of 20 units (0-20-40-60-80-100-120).

I tried changing to log, but then I get 0-10-100, which helps understand the data, but I would like to try to see divisions with 10 units because my data is not logarithmic (its linear, but there are many values in the 0-10 range, which results in a plot that is very hard to read).

How can I achieve that effect, or another one that will help me visually the data?

The code is below:

import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
sns.set()
plot = data3.plot(kind="scatter", x="A", y="B") # data3 is the name of my data frame

plt.xscale('log') # this was my change (plt is correct)

plot

I also tried the following (instead of the 'log' line above), but it gives the default (0-20-40-etc):

plt.xticks = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]

You are almost there when you tried to set the xticks values.

Try

ticks = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
plt.xticks(ticks, ticks)

The two parameters are the tick locations and the tick labels and they could be different, if you wanted.

Here's a full sample code

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import random

point_min = 0 # low value for random data
point_max = 101 # high value for random data
point_count = 100 # number of data points

# generate a DataFrame
data3 = pd.DataFrame({'A':random.sample(range(point_min, point_max), point_count)
                    , 'B':random.sample(range(point_min, point_max), point_count)}
                    )

tick_freq = 10 # define how far apart you want your ticks

data3.plot(x='A', y='B', kind='scatter'
         , c=np.repeat(np.atleast_2d([.5, .6, .7]), data3.shape[0], axis=0) # marker color
         , figsize=(15, 5)) # wide plot to show all tick labels
plt.xticks(ticks, ticks)
plot;

The resulting graph looks as below


Sampling from a multivariate von Mises-Fisher distribution in Python I am looking for a simple way to sample from a multivariate von Mises-Fisher distribution in Python. I have looked in the stats module in scipy and the numpy module but only found the univariate von Mises distribution. Is there any code available? I have not found yet.

-- edit. Apparently, Wood (1994) has designed an algorithm for sampling from the vMF distribution according to this link, but I can't find the paper.

It looks like you can sample the von Mises-Fisher distribution with that R package. Have you thought about calling R from within Python using the rpy2 package? I haven't tried this for myself, but could you try the following?

from numpy import *
import scipy as sp
from pandas import *
from rpy2.robjects.packages import importr
import rpy2.robjects as ro
import pandas.rpy.common as com
from rpy2.robjects.packages import importr

# import the movMF R package
movMF = importr('movMF')
# call the rmovMF sampling function from the R package
print(movMF.rmovMF(10, 3 * c(1, -1) / sqrt(2)))


Neural Network Self-organizing maps I'm studying the training algorithm for the self-organizing maps.
The slides i'm studying says that : after i've found a winner neuron ( in reference to a specific example in the example set), i have to apply this formula to its, and its neighbours.

where $d_{t}$ is the specific example, $m_{ij}$ the neuron ij.
What i've not understood is the meaning of the dist function in the radius . And what this radius means and what is its purpose. Can you help me ?

I gave some explanations here that may help.
The radius $r_{ij}$ is the radius of the neighborhood of the winner. Only those neurons that are closer than $r_{ij}$ to the winner are allowed to update.
The formulas that you show here say that this radius is not constant but decreases with each iteration. Initially you take a large radius, and then make it smaller and smaller with each iteration.
Since you probably initialize your weights randomly, you might want to change many of them in the beginning of training but you might want to disturb less and less weights as they become more trained. That's why you decrease the radius.
This formula is just a particular policy of change of the radius with each iteration. In this case it is exponential. You can choose another policy or keep the radius constant or decrease it slightly (not exponentially) which may be reasonable if you use some domain knowledge when initializing the weights. Also, you can keep the learning rate constant if you like. The radius of the neighborhood and the learning rate are hyper-parameters. It is up to you how to choose them.


Is there a way to cluster words based on how similarly they sound? I have a list of words for a fictional world I've made (don't judge lol).

My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build a model that will classify words into their region by similarity to the other words in that region.

My first attempt used Levenshtein distance, and this performed... well, not very good.

I then tried using the Jaro-Winkler distance plus the Jaro-Winkler distance for the inverted string divided by two, to get the average distance (to refresh, Winkler considers the beginning of the word to be more important, so I did this to get the end as important too). This was better, with 80% train/25% test accuracy, but obviously not good enough at all.

As you can see, for the multiclass models I'm trying to use, these are poor predictors of the regional class I've assigned to each word. I have only found this:

https://www.oreilly.com/library/view/python-cookbook/0596001673/ch17s09.html

As any sort of mechanism that takes sound into account, but it seems like words have to be similar length, and mine aren't necessarily within one or two character lengths of each other.

I mean, might be able to apply it, but I wanted to see if there was a better solution. My research hasn't turned up anything useful.

I haven't tried it myself, but you could try the IPA (International Phonetic Alphabet) version of your words and then calculate the Levenshtein distance.

There is a Python library called panphon. I have no affiliate with it. Just found it with Google Search.

import panphon.distance
dst = panphon.distance.Distance()
dst.dogol_prime_distance(u'pops', u'bobz')
>> 0
dst.dogol_prime_distance(u'father', u'mother')
>> 1
dst.dogol_prime_distance(u'school', u'fool')
>> 3