How to do multivariate survival analysis on dataset having only categorical variables I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having values in range 0-5(Its a survey dataset where user has to mark the experience between 0 to 5 values, where 5 means the highest). Now, the survey has 10 questions which are to filled from time to time(5 times) in order to assess the change in the user experience. So I have the dataset containing the values for same 10 questions(different values) repeated 5 times to make it 50 columns.

My questions are:

I want to do survival analysis on this dataset. Should I keep the dataset as it is(as the variables have different values from time to time) or should I process the dataset as the questions are repeated 5 times. If yes, can someone please advice any method to do so.
I have tried cox regression and survival trees to learn a model. But the performance of these models is not so good. Also, I am performing feature selection but there is not much difference as almost all the feature have same characteristics.

Below is the image of few columns of the dataset. For privacy reasons, could not share the complete dataset. Kindly understand that almost all the variables of the dataset have such data type except for age and gender.

Any help would be appreciated. Thanks in advance!

It seems a problem that can be solved with a correct approach for survival analysis with time-dependent covariates.

https://cran.r-project.org/web/packages/survival/vignettes/timedep.pdf

This vignette is useful also for those not interested in R code.

Have you tried a random forest survival analysis approach?


Oscillating loss in CNN So I designed my own CNN with 10 layers of convolutions and no maxpoolings or any other connections. When I ran it on a dataset I got the following loss curve (blue) the other one is accuracy vs number of epochs. What do you think might be the problem here? And how to rectify it?

NOTE: My data-set of size 20 due to lack of computational resources, so the oscillation is probably looking more prominent or happening in the first place.

Oscillating loss can be attributed to either of the following:

Learning rate: Reduce the learning rate so that the gradient descent doesn't overshoot the minima.
Optimizer: Choose ADAM optimizer over the others like SGD. It works well.


Is standardization needed before using scikit-learn SVM? I am using the SVM function provided by scikit-learn. I would like to know whether I need to perform standardization before fitting the model. As I know, LibSVM tends to require pre-processing the data. I am not sure whether scikit-learn automatically normalizes the data instead of expecting us to handle it ourselves.

scikit learn does not standardize data, but it does offer utilities for you to standardize your input data yourself:
http://scikit-learn.org/stable/modules/preprocessing.html

the rule of thumb is to standardize if your data aren't related.  That is, if channel X is not a function of channel Y, you should standardize

Qualitatively, think about it this way, SVM 'creates a hyperplane' to separate data into categories; if the data are skewed too far in one axis, that will make it harder to draw a plane to separate them


Using argmax with model.predict is the way to go.
# testdata is the dataframe of Generator
paths = testdata.filenames # Your files path

y_pred = model.predict(testdata).argmax(axis=1) # Predict prob and get Class Indices
classes = testdata.class_indices  # Map of Indices to Class name

from keras.preprocessing import image
a_img_rand = np.random.randint(0,len(paths))   # A rand to pick a rand image
img = image.load_img(paths[a_img_rand])
img = image.img_to_array(img)
from google.colab.patches import cv2_imshow
cv2_imshow(img)
print(f'Class Predicted ---- {list(classes)[y_pred[a_img_rand]]}')


Can I use scipy.optimize module with PyTorch? I need to use scipy.optimize module after encoding some data with pytorch. However, scipy.optimize does not take torch.tensor as its input. How can I use scipy.optimize module with data with gradient path attached?

Here's a workaround I'm currently using:
# model definition
class MLP(Module):
    # define model elements
    def __init__(self):
        super(MLP, self).__init__()
        self.hidden1 = Linear(2, 200)
        self.hidden2 = Linear(200, 100)
        self.hidden3 = Linear(100, 1)
        self.activation1 = LeakyReLU()
        self.activation2 = LeakyReLU()

    # forward propagate input
    def forward(self, X):
        optimize_flag = False
        if not torch.is_tensor(X):
            optimize_flag = True
            X = Variable(torch.from_numpy(X)).float()
        X = self.hidden1(X)
        X = self.activation1(X)
        X = self.hidden2(X)
        X = self.activation2(X)
        X = self.hidden3(X)
        if optimize_flag:
            return X.detach()
        return X

Just make sure when you are using scipy.optimize.minimize or any other function that uses the x0 flag to input to pass in a detached tensor (i.e. minimize(surrogate, x0=x.detach().numpy()))


Treatment and Control selection in A/B Testing I'm hoping to get a better understanding of A/B Testing design. In particular, I'm interested in understanding how treatment and control units are selected. I read that these 2 groups are selected randomly (for example, here), but then there are also approaches where after picking the treatment (either randomly or not) the control is selected based on "similarity" to the treatment group. Are both approaches valid and what's the rationale for picking one or the other?

For example, Alteryx has specific
Treatment and Control Tools for this purspose, and they are not random (they use nearest neighbor methods).

If I understand your question clearly(?), you are describing two different things, 1)AB testing and 2)case-controlled studies.

Think back to Stats class, you did not learn about AB testing BUT what you did learn WAS Hypothesis testing or Null hypothesis testing, Testing the null hypothesis, Hypothesis-alternative testing (whatever) maybe even sometimes called a one-tailed or two-tailed test. Give or take, these are all fairly synonymous concepts. However, like almost all ideas when people leave academia the names change.

One typical experiment is to take 300 stores and sell either A or B. After a day, week, month, you measure the amounts sold then ask, 'Did the two items sell statistically different volumes?'

The second idea I read in your question considers a different way to set up your experiments.

Another type of experiment is called a 'case-controlled study.'

So I want to test people this time. I can't in good conscience NOT give a person with a bad heart his medicine. I can't separate two groups of old folk and see who dies or not. So the tests are retrospective. I go to the hospital and check for all the patients that A)did or B)did not take the medication over the past x years.

For example, I find a 64 yr old male with heart problems who takes blood thinners. Then I look for a male approx. 64 yrs old with heart problems that did not take blood thinners then look at liver functions for both and test difference, let's say.  I look for similarity among my patients in my study. Alternatively, If I choose a large enough sample size A)took blood thinners or B)no thinners. I can assume that most factors will cancel each other out. I go case by case.


How data representation affects neural networks? Suppose A's possible values are ON or OFF.

Suppose I represent it as: if A ON then feature f=1 else f=0

Or, suppose I represent it with 2 features, where:

-if A is ON then f1=1 and f2=0

-if A is OFF then f1=0 and f2=1

How this kind of representation affects neural networks?

It will have very little effect

The answer most will give is that it will have no effect, but adding one more feature will decrease the ratio of records to features so will slightly increase the bias and will hence make your model slightly less accurate.  Unless, of course, you have overfit your model , in which case it will make your model slightly more accurate (a good data scientist would never do this because they understand the importance of cross-validation :-).

If you normalize your data and then attempt some sort of dimensionality reduction, your algorithm will immediately eliminate the feature that you added since it is perfectly negatively (linearly) correlated with the first feature.  In this case it will have no effect.

Please also consider the following:

I always see big red flags when someone asks a very fundamental data science question with the words neural network included.  Neural networks are very powerful and receive a great deal of attention in the media and on Kaggle, but they take more data to train, are difficult to configure, and require much more computing power.  If you are just starting out, I suggest getting a foundation in linear regression, logistic regression, clustering, SVMs, decision trees, random forests, and naive Bayes before delving into artificial neural networks.  Just some food for thought.

Hope this helps!


data analysis EDA issues, indent or type I am a newbie in DS world, right now i am working on some EDA practice, and run into an issue here.

Here is my code:

Convert some columns to numeric values

for column in cols:
    auto[column] = pd.to_numeric(auto[column]
    return auto

auto = read_auto_data()

error 1:
File "", line 4
return auto
^
SyntaxError: 'return' outside function

I tried to change return's indent:

Convert some columns to numeric values

for column in cols:
    auto[column] = pd.to_numeric(auto[column])
return auto

auto = read_auto_data()

error 2:
TypeError                                 Traceback (most recent call last)
 in ()
  1 ## Convert some columns to numeric values
  2 for column in cols:
----> 3     auto[column] = pd.to_numeric(auto[column])
  4 return auto
  5

TypeError: 'function' object is not subscriptable

Can anyone help me understand what went wrong here?

Thanks

Try something like this (the function also needs to know what cols is - perhaps pass that in as another parameter?):

def read_auto_data(auto):
    for column in cols:
        auto[column] = pd.to_numeric(auto[column])
    return auto

auto = read_auto_data()


Using a White noise image to minimise the loss in a (convolutional) neural network I'm reading someone else's code, which uses vgg19 to transfer the style of an artist to a 'content' image. I'm wondering why they have put

INI_NOISE_RATIO * noise_img + (1. - INI_NOISE_RATIO) * content_img)

instead of just noise_img (INI_NOISE_RATIO=0.7 here).

Why use a mix of the noise image and content image as input for the white noise bit in the content loss? The paper https://arxiv.org/pdf/1508.06576.pdf by Gatys et.al doesn't mention the mixing?

The loss functions are not related to the noise image.

The network is run twice to determine parameters to the loss functions. Once for the content image, and once for the style image. The code you are asking about occurs afterwards.

First the network is run against the content image (note no noise mixing here):

sess.run([net['input'].assign(content_img)])

Aside: Uses an assignment function - not a feed_dict - because in order for the style transfer to work, the input has become a tf variable, not a placeholder.

Then the first loss function is defined against some of the network parameters from this run (this extracts the activations of interest from the networks as fixed params to the loss function - because it uses output from sess.run, not references to tensor variables):

cost_content = sum(map(lambda l,: l[1]*
  build_content_loss(sess.run(net[l[0]]),net[l[0]]) , CONTENT_LAYERS))

The same thing is done for the style image:

sess.run([net['input'].assign(style_img)])
cost_style = sum(map(lambda l: l[1]*
      build_style_loss(sess.run(net[l[0]]), net[l[0]]), STYLE_LAYERS))

And a final cost function defined:

cost_total = cost_content + STYLE_STRENGTH * cost_style

Now, after the cost function is defined, the code initialises a new input (which is also our style transfer output):

sess.run(net['input'].assign( INI_NOISE_RATIO* noise_img +
                   (1.-INI_NOISE_RATIO) * content_img))

This line has nothing to do with defining loss. It just initialises the network ready to generate gradients against a starting image and the loss functions that were already defined beforehand. You can start with pure noise or a copy of the content image to start the process, and each will give a slightly different result.

The original paper started from a random image. Quoting from the paper (on page 10):

  Thus we can change the initially random image $\tilde{x}$ until it generates the same response in a certain layer of the CNN [. . .]

. . . however, starting from the content image was tried and used very soon afterwards by many in practice, and it works well.

This mix of both approaches is what the author has chosen, presumably because it gets pleasing results. As there are no lower layers in the network than the input, in fact you can start from an arbitrary image (e.g. all zeros), and you should still get a result, although it may not look as nice.

To (sort of) answer your question as written:

  Why use a mix of the noise image and content image as input for the white noise bit in the content loss?

The code is not doing that. The mixing is occurring as you say. But the line you show is not part of the content loss. The mixed image is the initial solution, and iterations will turn it into the stylised image.


solving quadratic equations using machine learning? I am attempting to solve quadratic equations using machine learning. I tried to write code using mxnet but fails because it is using linear regression while my problem is non-linear (as far as I know).

It is possible to predict the solution for arbitrary equations provided the at least one real root exists?

It is possible to predict the solution for arbitrary equations provided the at least one real root exists?

Yes.

You‘ll just need a sufficiently large training dataset and network that is flexible enough (start with one hidden layer).

You’re right your problem is not linear, thus you’ll need non-linear activation function.


When to use Linear Discriminant Analysis or Logistic Regression The Wikipedia article on Logistic Regression says:

  Logistic regression is an alternative to Fisher's 1936 method, linear
  discriminant analysis.  If the assumptions of linear discriminant
  analysis hold, application of Bayes' rule to reverse the conditioning
  results in the logistic model, so if linear discriminant assumptions
  are true, logistic regression assumptions must hold.  The converse is
  not true, so the logistic model has fewer assumptions than
  discriminant analysis and makes no assumption on the distribution of
  the independent variables.

Could someone help me to understand what the assumptions of linear discriminant are, an example of where they hold, and how application of Bayes' rule results in the Logistic Model.

Would it be correct to say that Logistic Regression is always the preferred choice?  Are there conditions where Linear Discriminant Analysis is to be preferred?

This question was asked and answered at the Cross-Validate SE. The answer is a few years old but it is still useful. If you want you can click here to find their answer.


Python list formatting I have a list, which looks like this:

params
['h\x00i\x00', '\x00t\x00h\x00e\x00r\x00e\x00']

Now, all I want is to merge these two elements into the string "hi there", but when I use

print ' '.join(params)

I get

h i  t h e r e

How do I strip out those hex characters to make the string "hi there"?

You have the null equivalent hexadecimal '\x00' in between each character. To remove those hexadecimal characters, first join the list with a space and then split it at '\x00' and then join again. Print statement simply looks like follows.

>>> print(' '.join( ' '.join(params).split('\x00')))


Each time the result of XGBoost is different I'm running xgboost and I fix the seed number
but the result is different for each time when I rerun the xgboost with same dataset

xgb = XGBClassifier(
     bagging_fraction= 0.8,
     boosting= 'gbdt',
    colsample_bytree= 0.7,
    feature_fraction= 0.9,
    learning_rate= 0.05,
    max_bin= 32,
    max_depth= 10,
    min_child_weight= 11,
    missing= -999,
    n_estimators= 400,
    nthread= 4,
    num_leaves= 100,
    predictor= 'gpu_predictor',
    seed= 1000,
    silent= 1,
    subsample= 0.8,
    tree_method= 'gpu_hist',
    verbose= True
)

This is my parameter and

# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split

# This is explicitly used for undersampling.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
kf = StratifiedKFold(n_splits=5, shuffle=True)
# partially based on https://www.kaggle.com/c0conuts/xgb-k-folds-fastai-pca
predicts = []
for train_index, test_index in kf.split(X_train, y_train):
    print("###")
    X_train, X_val = X.iloc[train_index], X.iloc[test_index]
    y_train, y_val = y.iloc[train_index], y.iloc[test_index]
    xgb.fit(X_train, y_train, eval_set=[(X_val, y_val)],
            early_stopping_rounds=30)
    predicts.append(xgb.predict(X_test))

You are missing some seeds there:

Numpy
Pandas
StratifiedKFold

I will say that StratifiedKFold(shuffle=True) without a seed can make your results change.


Different types of analysis which can be done with Webdata of users I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are:

Product Usage details
Geographic details
Session data: Capture what all they did during a session onsite, unique to timestamp
User details
Device used to login
Product looked at/ downloaded
Campaign data for promotion of a product
Search queries made on the on-site search bar
Results of those search queries

and so on.

I wanted to know what are the different types of analysis I can perform using this data, more inclined towards the Data science aspect of it. I have already thought of the following:

Churn Analysis
Campaign effectiveness
Find out which product is being sold the most and which at which geography
User behavior analysis
and such.

Could the community help me come up with more ideas which i can use to perform analysis on this data?

Thanks

Recommendation Engine is one such application of ML that can be used to extract the useful information from the user available data and offer them the product that is the most relevant to him. You can map this to classic classification problem where  expected output of ML algorithm is to come up with the product that customer will buy.


Guidelines for vocabulary sizes for BoW I am currently trying to get a vocabulary for BoW-vector generation out of a set of 200k scientific abstracts.

I do some basic filtering of tokens already like lowercasing, stop-word-removal, stemming, not taking tokens with size < 2, leaving tokens out that can be converted to a number and so on. But still I count more than 121k distinct tokens, which seems like a lot to me.

As I am quite new to all this I am wondering if there exist guidelines for how big such a vocabulary should be in average, maybe even depending on the originating field.

I don't think there's any definitive answer for this and it will depend on your particular domain. Here's how I go about it:

The English language contains about 20,000 words (or at least the most common) so I use that as a baseline
I expand this number to account for some common misspellings
Does my data contain special things like emojis? Emojis can still convey meaning, so I expand my vocabulary to include de-emoji'd text
Does my data contain specialized text like scientific and/or academic terms? I expand my baseline number based on this.

Finally, you can always check  your token index to find how  many words you have out-of-vocabulary. If that number seems reasonable enough to you to proceed then you move forward, otherwise you expand your baseline number a little more.


Finding the inverse of a matrix with neural networks I have a generic matrix A which is symmetric, positive definite and sparsely populated (it's also quite big, say composed of tens to hundreds thousands rows). I would like to have a neural network learn how to find a matrix B that resembles the inverse of A (ie: that multiplied by A will yeld a minimal spread of the spectral radius) as accurately and as fast as possible.

I was wondering if there are some AI branches that are focused on this kind of problems or if not how could I approach it.

I know the classical algorithms used to do this, but I would just like to have a look at the possibilities offered by machine learning on the subject.

In general this problem falls under the umbrella of "structured prediction" since you are trying to estimate a number of things that are related by virtue of being embedded in a PSD matrix.

Instead of estimating the inverse in one swoop, I'd pick an appropriate algorithm, say inversion by eigendecomposition, estimate the components (eigenvalues and eigenvectors), then piece them together. Here's one paper that shows how: Neural networks based approach for computing eigenvectors and eigenvalues of symmetric matrix.

Alternatively, you can investigate approximation algorithms such as the one elaborated in Approximating the inverse of a symmetric positive definite matrix

Welcome to DataScience.SE


What algorithms should I use to perform job classification based on resume data? Note that I am doing everything in R.

The problem goes as follow:

Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: based on the text on their CVs, I want to classify them into different job sectors. I am particular in those cases, in which the candidates do not have any experience / is a student, and I want to make a prediction to classify which job sectors this candidate will most likely belongs to after graduation .

Question 1: I know machine learning algorithms. However, I have never done NLP before. I came across Latent Dirichlet allocation on the internet. However, I am not sure if this is the best approach to tackle my problem.

My original idea:  make this a supervised learning problem.
Suppose we already have large amount of labelled data, meaning that we have correctly labelled the job sectors for a list of candidates. We train the model up using ML algorithms (i.e. nearest neighbor... )and feed in those unlabelled data, which are candidates that have no work experience / are students, and try to predict which job sector they will belong to.

Update
Question 2: Would it be a good idea to create an text file by extracting everything in a resume and print these data out in the text file, so that each resume is associated with a text file,which contains unstructured strings, and then we applied text mining techniques to the text files and make the data become structured or even to create a  frequency matrix of terms used out of the text files ? For example, the text file may look something like this:

I deployed ML algorithm in this project and... Skills: Java, Python, c++ ...

This is what I meant by 'unstructured', i.e. collapsing everything into a single line string.

Is this approach wrong ? Please correct me if you think my approach is wrong.

Question 3: The tricky part is: how to identify and extract the keywords ? Using the tm package in R ? what algorithm is the tm   package based on ?  Should I use NLP algorithms ? If yes, what algorithms should I look at ? Please point me to some good resources to look at as well.

Any ideas would be great.

Check out this link.

Here, they will take you through loading unstructured text to creating a wordcloud.  You can adapt this strategy and instead of creating a wordcloud, you can create a frequency matrix of terms used.  The idea is to take the unstructured text and structure it somehow.  You change everything to lowercase (or uppercase), remove stop words, and find frequent terms for each job function, via Document Term Matrices.  You also have the option of stemming the words.  If you stem words you will be able to detect different forms of words as the same word.  For example, 'programmed' and 'programming' could be stemmed to 'program'.  You can possibly add the occurrence of these frequent terms as a weighted feature in your ML model training.

You can also adapt this to frequent phrases, finding common groups of 2-3 words for each job function.

Example:

1) Load libraries and build the example data

library(tm)
library(SnowballC)

doc1 = "I am highly skilled in Java Programming.  I have spent 5 years developing bug-tracking systems and creating data managing system applications in C."
job1 = "Software Engineer"
doc2 = "Tested new software releases for major program enhancements.  Designed and executed test procedures and worked with relational databases.  I helped organize and lead meetings and work independently and in a group setting."
job2 = "Quality Assurance"
doc3 = "Developed large and complex web applications for client service center. Lead projects for upcoming releases and interact with consumers.  Perform database design and debugging of current releases."
job3 = "Software Engineer"
jobInfo = data.frame("text" = c(doc1,doc2,doc3),
                     "job" = c(job1,job2,job3))

2) Now we do some text structuring. I am positive there are quicker/shorter ways to do the following.

# Convert to lowercase
jobInfo$text = sapply(jobInfo$text,tolower)

# Remove Punctuation
jobInfo$text = sapply(jobInfo$text,function(x) gsub("[[:punct:]]"," ",x))

# Remove extra white space
jobInfo$text = sapply(jobInfo$text,function(x) gsub("[ ]+"," ",x))

# Remove stop words
jobInfo$text = sapply(jobInfo$text, function(x){
  paste(setdiff(strsplit(x," ")[[1]],stopwords()),collapse=" ")
})

# Stem words (Also try without stemming?)
jobInfo$text = sapply(jobInfo$text, function(x)  {
  paste(setdiff(wordStem(strsplit(x," ")[[1]]),""),collapse=" ")
})

3) Make a corpus source and document term matrix.

# Create Corpus Source
jobCorpus = Corpus(VectorSource(jobInfo$text))

# Create Document Term Matrix
jobDTM = DocumentTermMatrix(jobCorpus)

# Create Term Frequency Matrix
jobFreq = as.matrix(jobDTM)

Now we have the frequency matrix, jobFreq, that is a (3 by x) matrix, 3 entries and X number of words.

Where you go from here is up to you.  You can keep only specific (more common) words and use them as features in your model.  Another way is to keep it simple and have a percentage of words used in each job description, say "java" would have 80% occurrence in 'software engineer' and only 50% occurrence in 'quality assurance'.

Now it's time to go look up why 'assurance' has 1 'r' and 'occurrence' has 2 'r's.


Network Analysis using R I've the following dataset:

**Strenght  Movie1  Movie2**
  23            2        3
  80            1        2
  10            4        3

And I want to create a graph with the relationships between movies having the first column as the strenght of the relationship. How can I do this using R?

Many thanks!

Try this R code:

library(igraph)
dfr <- data.frame(idMovie1=c(2,1,4), idMovie2=c(3, 2, 3), strength=c(23,80,10))

igr <- igraph::graph.data.frame(dfr)
plot(x = igr,
     edge.curved=FALSE, edge.width=log(edge_attr(igr)$strength), edge.label=edge_attr(igr)$strength,
     main="Graph of Movie Strengths")

Works only for small daasets. visualizations get ugly quickly.


Remove rows that are too much alike not to be duplicates I have a dataset of real estate advertisements. Several of the lines are about the same real estate property so it's full of duplicates that aren't exactly the same. What would be the best methods to remove rows that are too much alike not to be duplicates?

It looks like this :

    ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE
0   22c05930-0eb5-11e7-b53d-bbead8ba43fe    http://www.avendrealouer.fr/location/levallois...   A_VENDRE_A_LOUER    APARTMENT   False   Au rez de chaussÃ©e d'un bel immeuble rÃ©cent,...   ["https://cf-medias.avendrealouer.fr/image/_87...   72.0    NaN NaN ... Lamirand Et Associes    AGENCY  54178039    Levallois-Perret    92300.0 92  2017-03-22T04:07:56.095 NaN 2017-04-21T18:52:35.733 NaN
1   8d092fa0-bb99-11e8-a7c9-852783b5a69d    https://www.bienici.com/annonce/ag440414-16547...   BIEN_ICI    APARTMENT   False   Je vous propose un appartement dans la rue Col...   ["http://photos.ubiflow.net/440414/165474561/p...   48.0    NaN NaN ... Proprietes Privees  MANDATARY   54178039    Levallois-Perret    92300.0 92  2018-09-18T11:04:44.461 NaN 2019-06-06T10:08:10.89  2018-09-25

So far I tried to compare the description :

df['is_duplicated'] = df.duplicated(['DESCRIPTION'])

And to compare the array of photos :

def image_similarity(imageAurls,imageBurls):
    imageAurls = ast.literal_eval(imageAurls)
    imageBurls = ast.literal_eval(imageBurls)
    for urlA in imageAurls:
        responseA = requests.get(urlA)
        imgA = Image.open(BytesIO(responseA.content))
        print(imgA)
        for urlB in imageBurls:
            responseB = requests.get(urlB)
            imgB = Image.open(BytesIO(responseB.content))
            hash0 = imagehash.average_hash(imgA)
            hash1 = imagehash.average_hash(imgB)
            cutoff = 5

            if hash0 - hash1 < cutoff:
                print(urlA)
                print(urlB)
                return('similar')
        return('not similar')

df['NextImage'] = df['IMAGES'][df['IMAGES'].index - 1]
df['IsSimilar'] = df.apply(lambda x: image_similarity(x['IMAGES'], x['NextImage']), axis=1)

Take a look into record linkage methods. They typically are used to find the same entity between two datasets but can also be used to link a dataset to itself and find duplicate records (it’s called deduplication in the literature)


How to label images for CNN use as classifier I have theorical question that I couldnt decide how to approach. I have tons of grayscaled shape pictures and my goal is seperate these images to good printed and bad printed. For this, I look at roughness of images because some of those is corrupted while printing (e.g I expected triangle shape but I saw some corruption in its edges. So its bad printed). Should I train my model to give each of them spesific label names (like good_printed_triangle, bad_printed_triangle, good_printed_square etc.) or just good and bad is ok? I was thinking about second option but how can CNN find correlation between different shapes, it makes me confuse. I am open for all ideas and thank you in advance.

In general in this situation you should be training to just the specific problem you are trying to solve - which from the sounds of it is just whether edges of any kind on the shape are blurry or cleanly printed.

Unless you specifically need the shape labelled as a square triangle etc you should drop it as a label as will just add unnecessary complexity and overhead to the model.


Clustering categorical variable values based on continuous target values Let's say I have $n$ data points with just one categorical feature $x$ and a continuous target variable $y$. I want to divide the possible values of $x$ into subsets such that the value of $y$ doesn't vary much within a subset.

As an example, suppose there are only $5$ possible values of $x$: $x_1,\ldots,x_5$. We observe that the values of $y$ don't vary much across $x_1$ and $x_2$. $y$ doesn't vary much across $x_3$ and $x_5$, but is different from the typical $y$ value for $x_1$ and $x_2$. For $x_4$, $y$ values are quite different from the above groups. So we can say that $(x_1,x_2), (x_3,x_5)$ and $(x_4)$ can be considered as "clusters".

Now what's a concrete way of saying that $y$ doesn't "vary much across $x_1$ and $x_2$"? One natural way to define that is for $y$ to have the same distribution for $x_1$ and $x_2$, i.e. the same mean and standard deviation of $y$ values for both $x_1$ and $x_2$.

Is there a better way to characterize the fact that $y$ doesn't "vary much across $x_1$ and $x_2$? Maybe the way I defined it above is too idealistic and I need a criterion more suited to a real-life dataset?
Are there any popular existing methods or library functions to solve such kind of a problem (i.e. clustering categorical feature values on the basis of continuous target values)?

Since you are looking for a degree of similarity regarding $y$ and the values of $x_1,...,x_5$ do not matter you can view this as a clustering problem regarding $y$:

Let $y_1,...y_5$ be the target values with $f(x_i)=y_i$ for $i\in\{1,...5\}$ then you need to define a distance measure $d(y_i,y_j)$ which, since your variables $y$ are continuous, could be the euclidian distance: $d(y_i,y_j) = (y_i-y_j)^2$. But you could also choose the absolute difference. (note that I am assuming your $y_i$ to be one-dimensional here, i.e. $y_i\in \mathbb R$). That gives you a formula to measure "does not vary that much".

It also provides an answer to your second question: you can choose from a range of unsupervised ML algorithms to do clustering. The most popular one probably being $K$-means. The idea here is very straight forward:

1. select a number of clusters K
2. initialize the position of the K clusters randomly
3. Assign each y_i to the closest cluster
4. For each cluster k calculate the new cluster position as the mean of all y_i belonging to that cluster
5. repeat 3 and 4 until the assignments of y_i does not change anymore

Mathematically this gives you a mapping $c(y_i)=k$.

And eventually when you are done with your clustering you just pick a cluster $k$ and for each $y_i$ in that cluster you look up the corresponding $x_i$. Which will return what you asked for.


Training on data with inherently non-applicable data cells I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense.

Chemical Dataset

Contains thousands of rows of chemical mixtures with many columns of properties. Example properties: bromine content, density.

Inherently non-applicable data

The chemicals can be gas, liquid or solid but some properties are only applicable to samples of a certain state. An example could be viscosity in liquids, bond type (ionic, molecular, covalent) in solids or density in gas.

So far...

...all research has pointed towards methods of fixing "missing values" via column means, data imputation or something similar. There doesn't seem to be any sense in imputing the freezing point of a gas. A gas mixture does not have a freezing point. I am still in the process of data preparation and unsure how to proceed.

I am working in python and missing data is stored as NaN values. Perhaps there are some models that can deal with such NaN-values.

Side-Note:

The majority of the dataset is comprised of distillation curve datapoints (sequential data describing what percentage of a chemical sample evaporates as temperature is increased). This data is present for all samples.

Follow-up 1: Is there a model that will give me NaN values for the freezing point when I give it something that resembles a gas?

Follow-up 2: Can this be compared to image object detection where the object is partially obscured? or part of the image is corrupt?

Welcome to the site!

The usual approach to missing values is to handle them manually. There are a few algorithms which can do this automatically, such as LightGBM and XGBoost, but in most cases it's better for model performance to decide on how you should indicate that a value is missing in your data.

For example in with a Pandas dataframe in Python, I may decide to replace all NA/NaN values in a particular column which should contain positive integers with -99:

dataframe[column] = dataframe[column].fillna(-99)


In machine learning how to find feature interdepencies? Given a data set of N features, wherein some the features in this set were derived from other features from the same set, I am trying to discover inter dependencies between features (something like this Input feature(s) -> output feature(s)).

Note that,there can be multiple dependencies in the same feature set. Can someone suggest some technique to approach this problem.

For me it seems you are looking for Association Rule Learning. A quite simple example for an algorithm you could use is the Apriori Algorithm. You could try that one first and see if it does what you want to achieve. Then you can take a look into more sophisticated algorithms.

Generally these algorithms try to find associations between your features. E.g. they are able to find the implication, that many people who buy Coca Cola & Beer, will buy a pack of chips - if supported by the data:

$(Coca Cola, Beer)\implies(Chips)$.


Mechanical engineer having troubles with neural network. Am I going in the right direction? Is my data set insufficient? I’m doing a project in which the suitability of Neural Networks (NN) shall be assessed and I am fairly new to the subject. More precisely the NN should replace an analytical model. This model calculates forces and torques applied on a structure based on displacement data. The task of the neural network is to approximate forces/moments via regression.
Some Authors state that a neural network consisting of input, one hidden layer and output layer might is sufficient for this purpose. Therefore I constructed a NN with these characteristics. The NN has 12 inputs, 8 hidden and 6 output neurons. The 12 inputs represent the deformation in two directions measured on 6 different points. The 6 Outputs represent the applied forces and torques (Fx,Fy,Fz,Mx,My,Mz).
For training and validation of the model 52 measurements, each one consisting of 12 deformations and 6 forces/torques have been used. Afterwards the data was extended to 174 measurements. In both cases the NN could not generalize well and the training “loss” was huge. I tried changing various parameters of the neural network (such as number of hidden neurons, activation function, optimizer, scaling of the data…). None of this led to any significant change. Afterwards I tried to reduce the complexity of the problem by reducing the number of output variables. Still with little success.
My main concern is that the amount of data available for training is way too small. Do any of you have any tips or suggestions? Could anyone verify my concern?

If the training loss is “huge” it suggests that the is a bug in the model or the model is not sophisticated enough.
Do not confuse training loss with generalisation. Generalisation is measured by accuracy on the test set.
With regards to training data have you tried data augmentation? Can you apply transformations to the training data to derive new samples. E.g if I train a model to drive a car based on video input and control output the video and control inputs can be flipped to turn left turns into right turns?


Does my prediction improve when I use more, but worse classifiers? I have a logical problem when programming my tumor identification algorithm.
In my data sample, I have tested multiple antibodies on tumors - to identify whether those tumors are good or bad. This is essentially a binary classification. Each antibody (respectively "classifier") showed to have a different explanatory / prediction power when used on its own.
My question in regards to the guideline I now need to program is:
Do I get a higher overall prediction power if I use all antibodies on a new tumor? Do classifiers with a worse explanatory still improve my overall prediction?
Or would only using the antibody with the "best" individual prediction power lead to a better result?
The resulting algorithm would then, step by step, test each antibody against each new tumor - each test would be done independently.
And I somehow can't wrap my head around what's better and whether I should simply use only the "best" antibody (i.e. with the best predictive power) or use various to improve my prediction result.

I would approach the problem as follow:
Start by splitting your dataset into a test dataset and a training dataset. You would then proceed by building a classifier for each antibody (feature) using only the training dataset and then measure the success of your model on some test dataset (using the measure that you care about e.g. accuracy).
Usually you cannot repeat the process I described above for all combination of antibodies (features), due to time constraints. Their is a variety of methods you can consider which can be found if you search for feature selection techniques e.g. wrapper, embedded, filter etc.
Since it sounds like you already have a good idea of which features (antibodies) are important, I would recommend trying forward features selection. The idea is that you start building multiple models using only 1 feature (antibody) and determine which one is the best. Let's refer to these as selected features from now on.
For each unselected feature

Build a model using the selected features and one of the unselected features (in this case we will build a model using your best antibody and one other antibody)
Measure the performance of the model on the test set and store it

Once you are done you check if by adding only one feature did it improve the performance in comparison to using only one antibody. If it did you add the best antibody to your original best antibody, so now your model has two selected features.
You can repeat this process, until the performance starts to drop. The method does not search the whole space, but it still gives some descent results.


Accuracy reduces drastically when using TruncatedSVD with hashingvector I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given product description. Since the dataset was large I was not able to able TF-IDF on that data it throws MemoryError.

I found that Hashingvector was desirable when dealing with large data. But when Hashingvector was applied I found it produced data with 1048576 features. It took around 1 hour to train and SGD model and produced 78% accuracy.

Code:

import pandas as pd
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import SGDClassifier
from sklearn.calibration import CalibratedClassifierCV

data_file = "Product_description.txt"
#Reading the input/ dataset
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

train_data, test_data, train_label,  test_label = train_test_split(data.Product_description, data.Category,
                                                                   test_size=0.3, random_state=100, stratify=data.Category)

sgd_model = SGDClassifier(loss='hinge', n_iter=20, class_weight="balanced", n_jobs=-1,
                          random_state=42, alpha=1e-06, verbose=1)
vectorizer = HashingVectorizer(ngram_range=(1,3))
data_features = vectorizer.fit_transform(train_data.Product_description)
sgd_model.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data.Product_Combined_Cleansed)
Output_predict = sgd_model.predict(test_data_feature)
print(accuracy_score(test_label, Output_predict))

Output:

Accuracy 77.01%

Since the dimension was high I thought reducing the dimension would increase the accuracy and reduce the training time. I used TrancatedSVD to reduce the dimension but this drastically reduced the prediction accuracy but reduced the training time to 10 minutes.

Code2:

from sklearn.decomposition import TruncatedSVD
clf = TruncatedSVD(100)
clf.fit(data_features)

Output:

Accuracy 14%

Edit:

When I tried TruncatedSVD with 1000 as limit it throws memory error so only I opted to use 100 as the limit.

It is stated that reducing n_features on HashingVector will cause collisions there can be collisions: distinct tokens can be mapped to the same feature index. However, in practice, this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems) in scikit site.

I got optimum accuracy when I used ngram between 1 to 3 so only used that.

Some possible directions:

when you're unsure how your BoW model works check different n-gram ranges (did you even check how it works with only unigrams?)
TruncatedSVD by default only iterates 5 times (n_iter), so you could increase that
It's not at all surprising to get such drastic change of classification quality for such low dimensionality. Some questions you could ask are why 100 dimensions? Does TSVD work 1000 dimensions or something like that? What is the reconstruction error (how does TSVD's explained_variance_ratio_ look like)?
HashingVectorizer also has parameter that can control number of features, namely n_features.


When to use cosine simlarity over Euclidean similarity In NLP, people tend to use cosine similarity to measure document/text distances. I want to hear what do people think of the following two scenarios, which to pick, cosine similarity or Euclidean?
Overview of the task set: The task is to compute context similarities of multi-word expressions. For example, suppose we were given an MWE of put up, context refers to the words on the left side of put up and as well as the words on the right side of it in one text. Mathematically speaking, similarity in this task is about calculating
sim(context_of_using_"put_up", context_of_using_"in_short")

Note that context is the feature that built on top of word embeddings, let's assume each word has an embedding dimension of 200:
Two scenarios of representing context_of_an_expression.

concatenate the left and right context words, producing an embedding vector of dimension 200*4=800 if picking two words on each side. In other words, a feature vector of [lc1, lc2, rc1, rc2] is build for context, where lc=left_context and rc=right_context.

get the mean of the sum of left and right context words, producing a vector of 200 dimensions. In other words, a feature vector of [mean(lc1+lc2+rc1+rc2)] is built for context.

[Edited] For both scenarios, I think Euclidean distance is a better fit. Cosine similarity is known for handling scale/length effects because of normalization. But I don't think there's much to be normalized.

When to use cosine similarity over Euclidean similarity

Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points.

Let's say you are in an e-commerce setting and you want to compare users for product recommendations:

User 1 bought 1x eggs, 1x flour and 1x sugar.
User 2 bought 100x eggs, 100x flour and 100x sugar
User 3 bought 1x eggs, 1x Vodka and 1x Red Bull

By cosine similarity, user 1 and user 2 are more similar. By euclidean similarity, user 3 is more similar to user 1.

Questions in the text

I don't understand the first part.

  Cosine similarity is specialized in handling scale/length effects. For case 1, context length is fixed -- 4 words, there's no scale effects. In terms of case 2, the term frequency matters, a word appears once is different from a word appears twice, we cannot apply cosine.

This goes in the right direction, but is not completely true. For example:

$$
\cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}2\\1\end{pmatrix} \right) = \cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}4\\2\end{pmatrix} \right) \neq \cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}5\\2\end{pmatrix} \right)
$$

With cosine similarity, the following is true:

$$
\cos \left (\begin{pmatrix}a\\b\end{pmatrix}, \cdot \begin{pmatrix}c\\d\end{pmatrix} \right) = \cos \left (\begin{pmatrix}a\\b\end{pmatrix}, n \cdot \begin{pmatrix}c\\d\end{pmatrix} \right) \text{ with } n \in \mathbb{N}
$$

So frequencies are only ignored, if all features are multiplied with the same constant.

Curse of Dimensionality

When you look at the table of my blog post, you can see:

The more dimensions I have, the closer the average distance and the maximum distance between randomly placed points become.
Similarly, the average angle between uniformly randomly placed points becomes 90°.

So both measures suffer from high dimensionality. More about this: Curse of dimensionality - does cosine similarity work better and if so, why?. A key point:

Cosine is essentially the same as Euclidean on normalized data.

Alternatives

You might be interested in metric learning. The principle is described/used in FaceNet: A Unified Embedding for Face Recognition and Clustering (my summary). Instead of taking one of the well-defined and simple metrics. You can learn a metric for the problem domain.


Understanding Word Embeddings Can someone direct me to research papers that have tried to understand word embeddings?

The original papers written by Thomas Mikolov are word2vec and doc2vec.

Lecture notes given by Richard Socher is simple to understand.

word2vec explained gives an intuition how the negative sampling in word embedding works. Not the whole algorithm though.

Nevertheless, there are implementations in various languages and libraries explaining how it works in their docs.

Python - gensim, Tensorflow

java - deeplearning4j


"Rare words" on vocabulary I am trying to create a sentiment analysis model and I have a question.
After I preprocessed my tweets and created my vocabulary I've noticed that I have words that appear less than 5 times in my dataset (Also there are many of them that appear 1 time). Many of them are real words and not gibberish. My thinking is that if I keep those words then they will get wrong "sentimental" weights and gonna make my model worse.
Is my thinking right or am I missing something?
My vocab size is around 40000 words and those that are "rare" are around 10k.Should I "sacrifice" them?

Instead of dropping rare words or incorporating them risking their scarcity in the training data leads to poor predictions, you can opt for a third alternative: using a subword vocabulary.
You can use approaches like byte-pair encoding (BPE) to extract a subword vocabulary, that removes the out-of-vocabulary word problem and reduces data sparsity in general. There is the canonical python implementation as well as the popular implementation by Google called sentencepiece.


Error using decision tree regressor I'm new to data science , while i'm implementing decision tree. I'm facing the following error. Where i went wrong;

Sample data in csv is:

x=dataset.iloc[:,:-1].values
y=dataset.iloc[:,:2].values
from sklearn.preprocessing import LabelEncoder,OneHotEncoder
from sklearn.compose import ColumnTransformer
labelencoder_x=LabelEncoder()
x[:,0]=labelencoder_x.fit_transform(x[:,0])
onehotencoder=OneHotEncoder()
columntransformer=ColumnTransformer([('dummy cols',onehotencoder,[0])],remainder='passthrough')
x=columntransformer.fit_transform(x)

from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.2,random_state=1)

from sklearn.tree import DecisionTreeRegressor
regressor=DecisionTreeRegressor(random_state=0)
regressor.fit(x,y)  #Error occurance

Error:
'''
ValueError: could not convert string to float: 'Business Analyst'
'''

The problem is with the y value.

x=dataset.iloc[:,:-1].values -> selecting all columns but last #this one is fine

y=dataset.iloc[:,:2].values -> selecting all columns till 2nd #this one is wrong

Change this to :

y=dataset.iloc[:,2].values #selecting only 2nd column.


What is a "shot" in machine learning? I keep on hearing this term "shot" used in machine learning.
Is a "shot" well-defined?
From what I can tell, "shot" is a synonym for "example". Most machine learning systems seem to be "multi-shot" meaning you have a huge dataset that has many different examples of different categories. However, for a system to have "one-shot" capabilities means that it is able to predict the category of something given exactly one example. Similarly, "few-shot" applications seem to only need a few examples in order to perform some function with the input. And "zero-shot" learning seems to be making predictions without any examples during training.
Is a shot just an example?
Given the evidence above, it seems like this is the case, but it also seems like it's a little bit more nuanced, something like a shot is a post-training example when 0 examples were given during training. But I'm not sure if this is right, thus the question.

A shot is a single example available for machine learning. So "one-shot" means you're given just the one example for each class.


Unable to figure out the linear embedding layer in the convolutional neural network? I have the network architecture from the paper "learning fine-grained image similarity with deep ranking" and I am unable to figure out how the output from the three parallel network is merged using the linear embedding layer.
The only information given on this layer, in the paper is

  Finally, we normalize the embeddings from the three parts, and combine them with a linear embedding layer. The dimension of the embedding is 4096.

Can anyone help me in figuring out what exactly does the author mean when he is talking about this layer?

Linear embedding layer must be just a fancy name for a dense layer with no activation.
'Linear' means there is no activation (activation is identity). And the embedding is rather a concept for a vector representation of the input data (e.g. word embeddings). I believe the elements from the second vector are simply added to the first one element-wise.


approach for multi label text classification I want to make a classifier that will label each text in a corpus with the correct label(s).
I can go straight to ML using sklearn multi-label text classification, or even to DL using LSTM.
But is it not better to start simple, and first use a rule-based system. That will help me understand the problem, and also set a benchmark accuracy score. Then I can make my algorithm more sophisticated (ML, DL) gradually in ways that only help the precision and recall.

So, you are asking about how to develop this system / model, which can classify text.
Yes, it is a great idea to instantiate a "baseline" or dummy model, which can be rule-based or randomly assigns a label to a certain piece of text.
From this dummy model, yes you can then use RNN/LSTMs that does multiple-inputs (e.g. words in text) to single output probability over classes as a more sophisticated model and yes you would then compare the validation and test accuracy, F1-score, etc. to see if that improvement to the model is warranted by the change in the model's functionality to classify the texts.


Why do we convert skewed data into a normal distribution I was going through a solution of the Housing prices competition on Kaggle (Human Analog's Kernel on House Prices: Advance Regression Techniques) and came across this part:

# Transform the skewed numeric features by taking log(feature + 1).
# This will make the features more normal.
from scipy.stats import skew

skewed = train_df_munged[numeric_features].apply(lambda x: skew(x.dropna().astype(float)))
skewed = skewed[skewed > 0.75]
skewed = skewed.index

train_df_munged[skewed] = np.log1p(train_df_munged[skewed])
test_df_munged[skewed] = np.log1p(test_df_munged[skewed])

I am not sure of what is the need for converting a skewed distribution into a normal distribution. Please, can someone explain in detail:

Why is this being done here? or How is this helpful?
How is this different from feature-scaling?
Is this a necessary step for feature-engineering? What is likely to happen if I skip this step?

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$".

For your coefficients to be interpretable, linear regression assumes a bunch of things.

One of these things is no multicollinearity. That is, your $X$ variables should not be correlated against each other.

Another is Homoscedasticity. The errors your model commits should have the same variance, i.e. you should ensure the linear regression does not make small errors for low values of $X$ and big errors for higher values of $X$. In other words, the difference between what you predict $\hat Y$ and the true values $Y$ should be constant. You can ensure that by making sure that $Y$ follows a Gaussian distribution. (The proof is highly mathematical.)

Depending on your data, you may be able to make it Gaussian. Typical transformations are taking the inverse, the logarithm or square roots. Many others exist of course, it all depends on your data. You have to look at your data, and then do a histogram or run a normality test, such as the Shapiro-Wilk test.

These are all techniques to build an unbiased estimator. I don't think it has anything to do with convergence as others have said (sometimes you may also want to normalize your data, but that is a different topic).

Following the linear regression assumptions is important if you want to either interpret the coefficients or if you want to use statistical tests in your model. Otherwise, forget about it.

Applying the logarithm or normalizing your data, is also important because linear regression optimization algorithms typically minimize $\|\hat y - y\|^2$, so if you have some big $y$ outliers, your estimator is going to be VERY concerned about minimizing those, since it is concerned about the squared error, not absolute error. Normalizing your data is important in those case and this is why scikit-learn has a normalize option in the LinearRegression constructor.


Is train/test-Split in unsupervised learning of neural network necessary? I am using autoencoder for anomaly detection in warranty data. It is unsupervised. I calculate the reconstruction error by the model and the records with high reconstruction error value is considered as an anomaly. I would like to know, if it is necessary to train/test-split the data.

Any help is much appreciated!

Yes it is still necessary, you are fitting your model on that data and learning it to find a good representation for that sample. Validating whether or not this was actually an anomaly is a lot more difficult then.


How can we evaluate DBSCAN parameters? yes, DBSCAN parameters, and in particular the parameter eps (size of the epsilon neighborhood).
In the documentation we have a "Look for the knee in the plot".

Fine, but it requires a visual analysis. And it doesn't really work if we want to make things automatic.
So, I was wondering if it was possible to find a good eps in a few lines of code.

Let's imagine something like :

evaluate kNN distance
sort these values
scale them (so that
the values are always between 0 and 1)
evaluate the derivative
find the first point where derivative is higher than a certain
value, let's try with 1

In R, it would look like (using iris dataset as in the DBSCAN documentation) :

# evaluate kNN distance
dist <- dbscan::kNNdist(iris, 4)

# order result
dist <- dist[order(dist)]

# scale
dist <- dist / max(dist)

# derivative
ddist <- diff(dist) / ( 1 / length(dist))

# get first point where derivative is higher than 1
knee <- dist[length(ddist)- length(ddist[ddist > 1])]

and the result is 0.536 which looks quite good.

Is this approach relevant or totally nonsense ?

OPTICS gets rid of $\varepsilon$, you might want to have a look at it. Especially the reachability plot is a way to visualize what good choices of $\varepsilon$ in DBSCAN might be.

Wikipedia (article) illustrates it pretty well. The image on the top left shows the data points, the image on the bottom left is the reachability plot:

The $y$-axis are different values for $\varepsilon$, the valleys are the clusters. Each "bar" is for a single point, where the height of the bar is the minimal distance to the already printed points.


Theano vs Tensorflow for building Neural Networks for NLP tasks I am trying to learn Theano and TensorFlow for building neural networks for NLP based tasks. Any suggestions as to when one should choose one over the other or what works better and when or is it just a personal choice of what one is comfortable with?

At the end this comes down to one's opinion and experience but I ll attempt to give you a decent answer.

Theano has been around longer and you will be able to find more resources on it. Both to learn and people's published work on it.

On the other hand Tensorflow has been getting better and better since it came out. It has a free online course available and Google even released an open model for natural language understanding.

You may also have to check what hardware you have available. Tensorflow does not run on as many hardware as Theano currently. (Although they are actively working on Windows and OpenCl support.)

A good comparison on various Deep Learning Frameworks can be found there.


What is the objective that is optimized with Random Search? I have recently learned about Random Search (or sklearn.model_selection.RandomizedSearchCV in Python) and was thinking about the theory behind the optimization process. In particular my question is, given that one performs Random Search on a certain algorithm (let's say random forest), what are the best hyperparameter based on? More specifically in what sense are they the "best" hyperparameters for the model? Do they maximize accuracy of the model? If not what is the (performance-)criterion that is optimized? Or is it entropy/gini?

According to the documentation, the function RandomizedSearchCV accepts a scoring string that can take any value from this table and you can even implement your own custom scorer depending on what your goal is.

The default parameter is None in which case it uses the models score function that is defined to:

  Return the mean accuracy on the given test data and labels.


Aggregating target-encoded array-like categorical features? I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables for machine learning classification algorithms.

One-hot encoding leads to very high dimensionality. The approach I've landed on is target-encoding/mean-encoding. I understand how to use this when the categorical feature is a single choice (eg current zip code). But, when the feature can take on multiple values from a large list (eg favorite hobbies, illness symptoms, university coursework), I am not sure how to combine the values.

My intuition says that the wrong approach would be to take each unique combination as its own factor and encode that, as it would lead to overfitting. Other things that come to mind would be simple aggregations like sum/avg/product/variance.

How should target encoded values be combined?

There are several options:

Domain knowledge - Given what you know about the domain, combine the categories that make the most sense.

Empirical - Treat combing categories as a hyperparameter. Search through the space of options and pick the best combinations based on cross-validation score.


Difficulties of getting raw data I am trying to obtain raw data for (violent) crime rates of a US/Canadian city (any city would do), but I need the data to be granular and raw. All I could find is either interpretations, summary data or useless editorials. I'm trying to do analysis, and I need day-by-day (detailed-level, granular) data that shows the number of crimes recorded per day. Does anyone have any good sources/suggestions for finding this? Thank you immensely!

I found some sites providing open data.

You can find crime data by year, by city in here.
Data contains incident number, crime type, date, time, location.
This site also provides crime data.

I hope this help.


What are the key differences between a MLP with lagged features and a RNN I've been working with MLP's for a while. Whenever I assumed that the past values of a feature might be useful for predicting the future values of Y, I would just create a new column in my data frame with Feature(t-1). This process would be repeated for further lags t-2,t-3...t-n.

Besides the obvious problem of the curse of dimensionality, I am worried that the MLP doesn't know how to weight those time lagged features that are now in a new separate column.

So in a nutshell:

Is the above approach wrong?
How does an RNN solve this?

Let's assume you have 3 features + considering 5 timestamps for each feature. That means you have 15 features as X values.

Machine learning techniques other than RNN can utilize the features as individual columns i.e. 15 columns.

RNN can use these data as 5 arrays containing 3 feature in them. We can feed this 5 array in RNN cell at each time step for a single record. The way RNN cell works when we feed second-time stamp data or subsequent time stamp data it gets processed important information from last time stamps.


Annotation tool for classification experiments What tools are available that provide an interface to present text classification results? I need to keep and compare the results of different classification algorithms for unsupervised data. The tool should allow users to give input (like the good match or not) for the presented examples of the test data.

Try prodigy. An annotation tool powered by active learning.


Can we use feature selection and dimensionality reduction together? I have a dataset having about 10,000s of features. The features have a hierarchy inherent to them. I found an algorithm performing feature engineering, taking the hierarchy of the features into consideration. After the procedure the feature space will be changed and the original features may not exist. This algorithm will reduce the number of features to about 2000 features. As the next step I am planning to use autoencoders(to perform dimensionality reduction) and obtain a latent representation to perform the classification task. The reason I didn't use the original dataset for the autoencoders is because I want to use the information on the hierarchy of the features for my model. Is this a meaningful model? Is it pointless to compress the feature space twice?
Thank you!

Every time you compress the feature space you are losing some information. The original feature engineering stage you outlined sounds like a meaningful compression & might make sense in the context of your problem. The second compression on the other hand might only serve to lose some information.
I would only perform the second compression if the classifier you are training is struggling resource-wise to train your model after the initial compression. In this case you might want to sacrifice some information to train the model quicker. But this step should only be investigated after you try training your model with only the original compression & are not satisfied with the speed of training.
Good luck!


PCA, SMOTE and cross validation- how to combine them together? I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a split between those who perform SMOTE before or after PCA. Yes, i am very confused, but even authors of published papers seem to be confused as well about this (Me being a newbie). Even here (on CrossValidate) there are different answers for the same Q about SMOTE and PCA, and the gist is, see what suits your data. Anyway, I know how to perform each by it's own, but don't know how to do PCA in each fold, for example in Kfold, or any other cross validation method.

1- Is it possible for someone to share with me how to do PCA inside Kfold? Through a loop or pipeline maybe?

2- I personally have a bipolar opinion on the SMOTE/PCA thing: a- I think that PCA should be performed before SMOTE so that the dimentionality reduction is done on the main (real) training data before we sample it (In my case upsampling). b- My other thought on it, is that if you upsample and then perform SMOTE then your dimentionality reduction is based on "non-biased" group sample (where both groups now have same number). My Q here is, if i CV (cross validated) PCA, can i include SMOTE in the mix?

3- Last Q, I saw a video of a prof. who said that if you test/split your data and then CV PCA on the training set, this is wrong. Since cross validate e.g. Kfold already does the train/test split for you. Is this the case? If so, then how would i eventually select my X_test X_train y_test y_train for further analysis?

To avoid data leakage, it's important to do any transformations after you split your data - so yes, you want to fit PCA to your training set/fold, but apply that transformation to both training and test data (when you want to consider generalisation).

Scikit-learn makes this process straightforward with sklearn.pipeline.Pipeline and there's a great example where 5-fold CV is done on a chain of PCA to logistic regression which should give you something build on.

To address your second question, there's nothing stopping you from including SMOTE in the mix after doing your train-test or CV split, but without knowing more about the data and situation I won't comment further.

Finally, it is not necessary to nest a train-test split within k-fold CV, since as you say, k-fold CV incorporates a test set. However, for final model selection, a more rigorous approach would be to split off a final validation set before you do anything else. Carry out your k-fold CV experiments on that training data to choose whatever model and parameters, and use the validation results as a final estimate of generalisation.

To implement this, the scikit-learn example above shows how you can evaluate the effect of different parameters on the classifier. What it does not show is final model selection and evaluation, but all that needs to be done is split off a reasonable amount of data prior to everything in the example, and then take the best model(s) and evaluate their performance on that validation set.


How to use GAN for unsupervised feature extraction from images? I have understood how GAN works while two networks (generative and discriminative) compete with each other. I have built a DCGAN (GAN with convolutional discriminator and de-convolutional generator) which now successfully generates handwritten digits similar to those in MNIST dataset.

I have read a lot about GAN's applications for extracting features from images. How can use my trained GAN model (on MNIST dataset) to extract feature from MNIST handwritten digist images?

Typically to extract features, you can use the top layer of the network before the output. The intuition is that these features are linearly separable because the top layer is just a logistic regression.

For GANs, you can use the features from the discriminator. These features are supposed to give a probability if the input came from the training dataset, "real images". In Radford's DCGAN paper, they use all the convolutional layers of the discriminator and run a max pooling layer extract features for CIFAR-10.

  To evaluate the quality of the representations learned by DCGANs for supervised tasks,
  we train on Imagenet-1k and then use the discriminator’s convolutional features from all layers,
  maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then
  flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM
  classifier is trained on top of them.


What is dilated pooling and how it works mathematically? While I understand the concept of dilated convolution as there are lot of papers explaining about it, I have heard less about dilated pooling.

Can someone explain what it is?
What is the internal implementation of it? Preferably with an example.

From Tensorflow Github:

Dilated max-pooling is simply regular max-pooling but the pixels/voxels you use in each "application" of the max-pooling operation are exactly the same pixels/voxels you would select with dilated convolution.
Dilated convolution/pooling are useful for connectomics and 3D shape datasets (3D deep learning).

So, similarly as in convolution every nth pixel is selected for the operation.
Edit
Selecting points like this:

making pooling for them like this:

Picture sources:
https://www.quora.com/Is-a-pooling-layer-necessary-in-CNN-Can-it-be-replaced-by-convolution
https://www.quora.com/What-is-max-pooling-in-convolutional-neural-networks
Edit2:
2 x 2 max pooling with 3 x 3 dilation:
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (5) 0  0 [6] 0  0 (3) 0  0 [6] 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (3) 0  0 (4) 0  0 (2) 0  0 (4) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (1  0  0 (2) 0  0 (2) 0  0 (1) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (3) 0  0 [4] 0  0 [6] 0  0 (3) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0

becomes
6  6
4  6


Sklearn PCA with zero components example I'm simply trying to repeat a benchmark from the sklearn's docs.

The unclear part is: n_components = np.arange(0, n_features, 5).
They are applying a PCA transform with 0 components!

Can somebody, please, explain, what's the mathematical meaning of this transform?

Think of it this way: a PCA "transform" with $k$ components essentially approximates your $n$-dimensional data points by projecting them onto a $k$-dimensional linear subspace, trying not to loose too much data variance along the way.

More precisely, what you are doing is representing your original points $y \in R^n$ as:

$$y \approx \mu + Vx$$

where $x \in R^k$ are the lower-dimensional "new coordinates", $\mu \in R^n$ is the mean of your data and $V\in R^{n \times k}$ is the matrix of principal component vectors.

The "new coordinates" $x$ tell you how many steps you need to make along the $k$ principal components in order to reach the best possible linear approximation to $y$ by starting your trip from $\mu$.

Now, if $k=0$ the model becomes:

$$ x \approx \mu. $$

In other words, you are modeling all of your data as a single, fixed center point. Of course, you do not need to store any "new coordinates" here (because you do not need to move away from the mean), hence it does not make much sense as a "transform", but it is, none the less, a proper probabilistic model (a maximum-likelihood fit for a Gaussian distribution of errors, to be precise).

In particular, you can speak about the log-likelihood of data under this model (which is, up to an affine transform, equal to the sum of squared errors here, but is not as trivial as you might think in the general case) and we can compare various models, choosing the one with the best likelihood. This is exactly what is done in the example from the docs you mention in the question.


Different Initial Q-Values in Q-Learning When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?

In the long-run, tabular Q-learning converges toward the optimal regardless of initialization.
However, the speed of convergence may be affected, similarly to an n-armed bandit setting : http://incompleteideas.net/book/first/ebook/node21.html
For more on initialization in Q learning, I recommend "Potential-based shaping and Q-value initialization are equivalent" by Eric Wiewiora.


Which book is the best for introduction to analysis social network using python3 I am a beginner studying social network analysis.
I installed python 3 just 2 weeks ago.
There are a lot of books for python and social network analysis.

I couldn't choose one of them.
I found one book named "Mining the Social Web (Analyzing Data from Facebook, Twitter, Linkedln, and Other Social Media Sites)" written by Matthhew A. Russell.
This book looks very interesting and fits in my purpose, but it is based on python 2.

Is there any good books with python 3? I usually use Twitter, Facebook, or Blog data.

In addition, could you recommend any good book for nodeXL and UCINET?

Russell's book is fine. You might also like Social Network Analysis for Startups. All the examples are in python. You can do all your analysis in that using packages like networkx. NodeXL is for the Excel crowd. Definitely not the ideal tool for the job; I would shy away from it.

The obvious book for NodeXL is Analyzing Social Media Networks with NodeXL, which is written by the authors of NodeXL.


Choose threshold to get 90% precision classifier - ML Binary Classification problem I have chosen threshold value with below code to get 90% precision classifier
from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train, cv=3)

z_scores = cross_val_predict(sgd_clf, X_train, y_train, method='decision_function')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores)

threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

y_train_pred_90percent_precision = (z_scores >= threshold_90_precision)
print(precision_score(y_train, y_train_pred_90percent_precision))

I'm expecting precision_score to be 90% but it returned 95%. Is this expected? Anything incorrect with my code? If it's expected, can you please explain the reason?

threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

Above snippet is not doing what you are expecting it to do.
 Try these changes
precisions[precisions < 0.9] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

Also, I am not sure whether you are calculating the accuracy properly since z_scores is decision function, not Class.
This is a working example using  method='predict_proba' for 40%, you may change to 90%
model.fit(x_train, y_train)

from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(model, x_train, y_train, cv=3)

z_scores = cross_val_predict(model, x_train, y_train, method='predict_proba')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores[:,0])

import numpy as np
precisions[precisions < 0.4] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

y_train_pred_90percent_precision = z_scores[:,0] >= threshold_90_precision
from sklearn.metrics import precision_score
print(precision_score(y_train, y_train_pred_90percent_precision))


Why Self-Organizing Map (SOM) units are called neurons? In the self-organizing map method, each of the output units is called neurons. But after reading the details about the algorithm, I could not find any logistic regression unit which is the neuron in a neural network architecture. I find it closer to the clustering algorithms like K-NN.

Is there any explanation why SOM units are called neurons? And why is it linked to neural networks (while ANN does a completely different job i.e., classification)

P.S.: In this paper (https://ieeexplore.ieee.org/document/6007170/), the SOM is called as an ANN. Here is a sentence from the abstract:

  the SOM is a type of artificial neural network that is trained using unsupervised data to produce a low-dimensional, discretized representation of an input space of training samples in a cell map.

Neurons do not have to be only logistic regression units. Even neurons in a simple feed-forward network may not all be logistic. For example, neurons in the output layer of a network for a regression problem are usually linear.

An artificial neuron is any computing unit that takes inputs and performs some computation. It can be any computation, linear, non-linear, logistic, weird, whatever. The goal of connecting multiple neurons into a network is achieving the ability to perform a complicated computation, even though each neuron can do only a very simple task.

A neuron in a self-organizing map does perform computations. When an input vector is presented to the network, every neuron in a SOM computes the distance between its weight vector and the input vector. This is a computation. Then it compares this distance with the computations of other neurons. This is also a computation which is possible only because the neurons are connected in the network.

The subsequent behavior of the neuron depends on the result of these computations. If it is the winner, it does another computation - modifies its weights, and one more computation - pulls the neighboring units (also a networking thing).

If it is not the winner, it performs a very important action: it does nothing, which is often the best thing that one can do.

The decision for a neuron to update its weights or not to update them is a non-linear operation: if an input vector changes slightly, the response of the neuron may change sharply or not to change at all.

The result is that the network performs a complicated computation: it builds a discrete, topology-preserving approximation of the distribution of training samples. This is useful in many applications.

When a SOM is trained, then every neuron also does a non-linear computation. It either fires or not depending on whether the input vector is closer to that neuron than to any other neuron.


When should we consider a dataset as imbalanced? I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.

My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.

Examples:

If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples?
The same question for 1,000 positive example and 100,000 negative.
The same question for 10,000 positive and 1,000 negative.
etc...

I think subsampling (downsampling) is a popular method to control class imbalance at the base level, meaning it fixes the root of the problem.  So for all of your examples, randomly selecting 1,000 of the majority of the class each time would work.  You could even play around with making 10 models (10 folds of 1,000 majority vs the 1,000 minority) so you will use your whole data set.  You can use this method, but again you're kind of throwing away 9,000 samples unless you try some ensemble methods.  Easy fix, but tough to get an optimal model based on your data.

The degree to which you need to control for the class imbalance is based largely on your goal.  If you care about pure classification, then imbalance would affect the 50% probability cut off for most techniques, so I would consider downsampling.  If you only care about the order of the classifications (want positives generally more higher than negatives) and use a measure such as AUC, the class imbalance will only bias your probabilities, but the relative order should be decently stable for most techniques.

Logistic regression is nice for class imbalance because as long as you have  >500 of the minority class, the estimates of the parameters will be accurate enough and the only impact will be on the intercept, which can be corrected for if that is something you might want.  Logistic regression models the probabilities rather than just classes, so you can do more manual adjustments to suit your needs.

A lot of classification techniques also have a class weight argument that will help you focus on the minority class more.  It will penalize a miss classification of a true minority class, so your overall accucracy will suffer a little bit but you will start seeing more minority classes that are correctly classified.


Find 10 most similar data points from 100? I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the top 10 most similar cells from 100.
Is there any way i can solve this?

That is often called nearest-neighbor search.
The most common methods require a distance metric. Given the features of battery pack, how close to each other are they?


Is there a common strategy to measure if a difference-significance of two areas under two ROC curves I conduct sound detection experiments with mice.
I have a stimulus sound and a "noise" sound that shoukd be ignored.

I want to measure how well the mouse ignors the noise (with respect to, say, ignoring 100% of the noise stimuli).

I have sessions with, say 200 trials of stimulus sound that sould be detected and 40 trials of noise sounds that should be ignored.

I created the confusion matrix and ROC curve (I use Matlab).
Now I want to know how well my mouse is performing compared with "ignoring all noise stims".

Is there a way, or a common used formula, to get a evaluation or a confidence interval of "how good is my ROC AUC comparing to AUC(ROC) = 1"?

Thanks!

The AUC is a summary statistic of your dataset. As such, if you did the same experiment again you would get a slightly different value - and if repeated several times, you'd get a distribution of values.  You probably want to show that your distribution rarely or never includes 0.5 (indicating random classification). When you have an empirical distribution like this, or a histogram of repeated experiments, you can just count how many of your estimates are $\approx 0.5$.  If none, you're golden.

Since you don't want to do your experiment multiple times, you could 'simulate' that scenario by using samples of your data, but otherwise performing the same process. This is called 'bootstrapping'.  You can use this method to construct confidence intervals around your AUC metric, or equivalently, do a statistical test to see if the metric you get is different from 0.5.

Some packages have bootstrapped confidence intervals built in to the AUC calculation, so you could just use that.


Finding a range of values for each feature that contribute to positive classification Consider a classification problem(lets say 2 classes, 'good' and 'bad), where all features are continuous.

what I need is a range of values for each feature that contributes to 'good' classification.
What I thought was simply partitioning the feature values based on good or bad label, problem is all values don't equally contribute for good/bad classification.

So what methods can be applied to find such range for each feature ?

It's impossible in general, simply because a particular value or range for feature A might correspond to class 'good' if feature B has a certain value/range but correspond to class 'bad' otherwise. In other words, the features are inter-dependent so there's no way to be sure that a certain range for a particular feature is always associated with a particular class.

That being said, it's possible to simplify the problem and assume that the features are independent: that's exactly what Naive Bayes classification does. So if you train a NB classifier and look at the estimated probabilities for every feature, you should obtain more or less the information you're looking for.

Another option which takes into account the dependency between variables is to train a simple decision tree model: by looking at the conditions in the tree you should see which combinations of features/ranges lead to which class.


BPTT vs Vanishing Gradient Problem I know that BPTT is the method to apply Back Propagation on RNN.

Which is works fine with RNN as it stops at certain point as changes approach to zero

but isn't it the exact Vanishing Gradient Problem?

if it is the same then Why does it have 2 names, one is a problem and one is a method.

if not then what am I missing here, what is the difference between them?

BPTT is the process of backpropagating the gradients calculations (chain rule) of a loss function at each time step in a recurrent network. The parameters $U$ (input weights) and $W$ (recurrent connection weights) will have a contributions to the gradient at each time step so the total gradient will be the sum of each contribution.

The vanishing gradient is a generic issue in neural networks of having decaying gradients values over the parameters of the network, meaning that updates for some parameters will be very small and hard to propagate. This gradient-based learning problem may occur with deep networks, vanilla recurrent networks (because of the large product of contributions to the gradient over timesteps) and often raised by the inherent nature of some activation functions.


How does one go about feature extraction for training labelled tweets for sentiment analysis? I am currently working on a sentiment analysis project where the end goal is to try and predict who wins the Nigerian 2019 presidential election. My goal is to create a model that analyses tweets and detects the tone of a tweet (whether positive, negative or neutral). This model can in turn be used to predict who wins. Here's what I have done so far.

I have a program running on a server streaming tweets daily
I have labelled about 5000 tweets manually into different categories
(positive, negative and neutral tweets).
I previously used text blob to determine sentiment on each tweet. But it
was not accurate enough.
I was then advised to train a model on the way Nigerians speak on twitter,
which can sometimes be a little different from normal English

From my research, I found out I have to use feature extraction before I can train the model with my labelled data. My problem now is that I don't understand how to go about these feature extraction. I have read materials about what people use (bag of words, bigrams, unigrams, part of speech tags etc). I don't understand the explanation I got so far, how it all comes together or which to use. Please could anyone explain these in layman's terms or point me to resources that explains this in layman's terms. Thank you

Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like

I do not like the views of @Candidate1 on #Topic1. Too conservative!! I can't stand it!

then we can't just feed this into a learning algorithm. We need to convert it into a proper format, so we perform pre-processing on our data.

To start, we might want to try to tokenize the tweet. To tokenize is something like segmentation. If you're familiar with Python, we can use ready made libraries (like NLTK) to aid us. Depending on how your tokenizer is made, you could transform the previous tweet into something like

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@', 'Candidate1', 'on', '#', 'Topic1', '.', 'Too', 'conservative', '!', '!', 'I', 'can', "'", 't', 'stand', 'it', '!']

I manually segmented the tweet with the rule that words separated by a space should be tokens and punctuation should be tokenized separately. The way you build a tokenizer (or adjust the settings of a ready made one) will determine the output from a tweet. Notice how '@' and 'Candidate1' are separated? A tokenizer for regular text might not be able to identify that this is a social media entity -- a user mention. If you can adjust your tokenizer to account for social media identities and contractions (like "can't"), you could produce a list of tokens as such

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@Candidate1', 'on', '#Topic1', '.', 'Too', 'conservative', '!', '!', 'I', "can't", 'stand', 'it', '!']

Now, you mentioned bigrams and unigrams. An n-grams (e.g. 1-gram == unigram) is just a sequence of tokens. So what we produced awhile ago was just a unigram list of tokens. If you want a bigram, then you'd want to take 2 tokens at a time. An example output of a bigram list of tokens would be

['I do', 'do not', 'not like', 'like the', 'the views', 'views of', 'of @Candaite1', '@Candidate1 on', 'on #Topic1', '#Topic1 .', '. Too', 'Too conservative', 'conservative !', '! !', '! I', "I can't", "can't stand", 'stand it', 'it !',]

Notice how words repeat? Imagine what a 3-gram or 5-gram would look like.

Before anything, why would we use bigrams over unigrams or those of higher n values? Well, the higher the n, the more about of order you're able to capture. Sometimes order is an important factor in learning. Playing around with how you'll represent your data might show you important features.

Now that we have our text tokenized, we can start extracting features! We can turning our example test, and other text samples, into a Bag-Of-Words (BOW) model. Think of a BOW as a table with column headers as the words/terms and rows as your text samples/tweets. A cell could then contain the number of words/terms for a given sample of text. You could start with counting each term in a sample, so based on the tweet, you'd come up with something like

tweet1: {
   'I': 2,
   'do': 1,
   'not': 1,
   'like': 1,
   'the': 1,
   ...
   '!': 3,
   ...
}

I don't want to manually write it all, but I hope you get the picture here. See how 'I' is 2 because it was mentioned twice in the sample. '!' was mentioned trice, so its value was 2. You'll find that there will be a lot of 1 values, specially in tweets, because there isn't room for much to be written.

You'd do this for each of your tweets and you'll come up with something like

       | 'I' | '!' | '#Candiate1' | ...
tweet1 |  2  |  3  |       1
tweet2 |  0  |  0  |       0
...
tweetn |  1  |  5  |       0

This is a BOW! Adjust your n-grams and you'll get different features with different values. This representation is actually a simple starting point for you because this is now understandable by a computer. You mentioned you labeled your tweets, so imagine that table, but with your label appended to it.

       | 'I' | '!' | '#Candiate1' | label
tweet1 |  2  |  3  |       1      | negative
tweet2 |  0  |  0  |       0      | negative
...
tweetn |  1  |  5  |       0      | positive

And with that, you can build a simple model with the appropriate steps after feature extraction.

There is a lot more to feature extraction! So many different combinations of methods to try out, but stick to things simple for now, if you don't feel so adventurous. Here are some things to consider:

Instead of counting terms, you could go with frequency (i.e. number of occurrence of a term divided by total number of terms)
Instead of extracting the words themselves, you can use part-of-speech (POS) tags (POS tags are language dependent, so you'll have to look for a POS reference for the specific language you're dealing with)
Consider extracting topic models or maybe meta information (avg length of words, avg length of tweets)
Is there an emotion lexicon for the language you're dealing with? Consider counting the number of emotion related words
Consider reducing the number of unique words or applying weighted schemes (like TFIDF)

There are so many ways to approach this, but it ultimately boils down to a design you'll finalize. I guess the best advice I can give you is to refer to related literature. Look at other papers/articles that process text written by Nigerians. What are the methods they use to process text? Is it language dependent or independent? Did they reduce the number of features? Can I create the language resources? Once you identify a good source, try modifying and experimenting from there.


Shape classification algorithm I have a dataset of shape edges, that I am trying to make a model for with sklearn. I'm new to the machine learning world, so I am struggling to create a good model. Using SVM, I was able to get a supposed 81% precision, but when I feed it an image outside the training or test set, it consistently returns the wrong prediction, almost every time.

Question: Is there is a better way of doing this than using SVM? Or are these shapes too similar? I have 90 images in the training set.

Here is a link to my ML code.

That sounds like you're suffering from overfitting, probable duo the "curse of dimensionality", which means 2 things.

When trying to measure distance between two points in high dimensional space and trying to interpret this data, the difference between the longest and the shortest distance get's less meaningful.
Trying to train an algorithm with way less samples then features, like in your case, will cause that the algorithm will memorize your samples.

I see 3 options. Getting more data, reduce your feature set or try greater values for C. C will regularize the decision boundary.

More to read:

Curse of dimensionality
n >> p