Prediction using words which were not in training in a CNN with pre-trained word embeddings In sentence classification using pre-trained embeddings(fasttext) in a CNN,  how does the CNN predict the category of a sentence when the words were not in the training set?
I think the trained model contains weights, these weights are not updated in the prediction stage, are they?. Then, what happens when the words in the sentence (for which the cnn will predict a category) were not seen in the training? I think they do not have a word vector, only the words that were found in the training.

If you keep the FastText embeddings unchanged a do not finetune them during training, it does not really matter that the words were not in the training set as long as they are in the FastText embeddings. After all, this is the biggest advantage of using pre-trained word embeddings.
The important property of the embeddings is that similar words get similar embeddings. The CNN might not have seen the exact same embedding, but similar words probably were in the training data.
Words that are not covered by the pre-trained embeddings, got a common representation for an unknown (out-of-vocabulary, OOV) word. These are usually proper names. It is usually good if you make sure that CNN learns to deal with them already at the training time (you can randomly replace some infrequent words by some random strings) because if the unknown token embedding (that is typically dissimilar to all other embeddings) appears at the inference time and it was never seen at training, it can lead to unexpected behavior.


Decision Tree Regressor: domain of the y variable just wondering about a thing. suppose you fit a Decision Tree Regressor and your training y variable has got a domain that spans from 0 to 1 included, is it possible that the predicted y variable of the fitted model on the test set to exceed these boundaries?

Decision Tree Regressor won't predict values outside the range of values they are given in the train set. If your extreme values are -4 and +10 the predicted values will be between -4 and +10. The reason for that is that a decision tree splits the training data in groups. The prediction associated with a node is then the mean value of the training data of that node (or if you choose something else than MSE, whatever value that minimize the cost function for this group).
An example of how a regression tree works : https://www.saedsayad.com/decision_tree_reg.htm The output for Windy and Sunny is 26.5 as the training data has 2 data points that fit these criteria (23 and 30)


What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way? What is the difference between gradient descent and gradient boosting?  Are they  interdependent on each other in any way ?

They're two different algorithms, but there is some connection between them:

Gradient descent is an algorithm for finding a set of parameters that optimizes a loss function.  Given a loss function $f(x, \phi)$, where $x$ is an n-dimensional vector and $\phi$ is a set of parameters, gradient descent operates by computing the gradient of $f$ with respect to $\phi$.  It then "descends" the gradient by nudging the parameters in the opposite direction of the gradient.  This process is repeated for different points in the space of inputs (i.e. different $x$s) until a minimum of $f$ is found.

Gradient boosting is a technique for building an ensemble of weak models such that the predictions of the ensemble minimize a loss function.  I think the Wikipedia article on gradient boosting explains the connection to gradient descent really well:

  . . . boosting algorithms [are] iterative functional gradient descent algorithms. That is, algorithms that optimize a cost function over function space by iteratively choosing a function (weak hypothesis) that points in the negative gradient direction.

So the connection is this:  Both algorithms descend the gradient of a differentiable loss function.  Gradient descent "descends" the gradient by introducing changes to parameters,  whereas gradient boosting descends the gradient by introducing new models.


Training a CNN on a large dataset I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like:
model = Sequential()
model.add(Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', input_shape = training_data.image_shape))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.3))

model.add(Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu'))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.2))

model.add(Conv2D(filters = 126, kernel_size = (3, 3), activation = 'relu'))
model.add(MaxPool2D(pool_size = (2, 2)))
model.add(Dropout(rate = 0.15))

model.add(Flatten())

model.add(Dense(units = 32, activation = 'relu'))
model.add(Dropout(rate = 0.15))

model.add(Dense(units = 64, activation = 'relu'))
model.add(Dropout(rate = 0.1))

model.add(Dense(units = 42, activation = 'softmax'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

However, the training time takes super long and each epoch takes around 35 minutes to run. The accuracy is also very low and increases very slowly.
My jupyter lab will sometimes stop and have to refresh everything again. So is there a way to train in smaller batches? Or a way to improve the training speed? Any help is appreciated. It is a very huge dataset.
Epoch 1/15
2307/2307 [==============================] - 3999s 2s/step - loss: 3.5377 - accuracy: 0.0687 - val_loss: 3.3247 - val_accuracy: 0.1223
Epoch 2/15
2307/2307 [==============================] - 3764s 2s/step - loss: 3.2884 - accuracy: 0.1239 - val_loss: 3.1065 - val_accuracy: 0.1739
Epoch 3/15
2307/2307 [==============================] - 2204s 955ms/step - loss: 3.1435 - accuracy: 0.1562 - val_loss: 2.9825 - val_accuracy: 0.2069
Epoch 4/15
2307/2307 [==============================] - 2193s 951ms/step - loss: 3.0526 - accuracy: 0.1778 - val_loss: 2.9059 - val_accuracy: 0.2171

Changing the batch size will not change the overall training time too much. Since with any batch size you are passing almost 80K images.
One(and the best) approach will be to use transfer learning.
If you have a compelling reason to do full training, you will need a GPU powered bigger hardware. Google Colab can be an option. There are many other options available
Before that you may try to gauge the model with a sample of ~5k images


Using KNN to categorise inventory (physical stock items) - is it the best way? I'm working on a machine learning problem involving inventory (i.e. physical retail stock), however through the cleaning (outlier removal) process some of the items (via their corresponding transactions) will be removed. Therefore, I thought of using KNN to group similar items into respective categories.
There are 1245 items
The info for each item is

Average Weighted Price
Total Quantity Sold
Total Revenue Achieved
Min Sold per Transaction
Max Sold per Transaction
Min Sell Price
Max Sell Price
Number of Unique Transactions

Am I right in thinking that KNN is a good option - and if so, how do I decide on the number of clusters?

So your question is on the effectiveness of KNN to categories items based on features you have listed above.
As you might already know, KNN is a unsupervised clustering algorithm which creates K clusters with a minimal intra-cluster variation. This is method can be particularly use for when you know what the number of groups K you need. Also, it is particularly handy if you do not have any labels for categories for all examples.
At the same time, this method isn’t deterministic, which means that groupings do vary after each execution.
From this information, you might get a better idea for yourself as to whether KNN would be useful for this task.


How to retrive the results saved in model of gensim? I have a file that contains words. I have to calculate cosine similarity of each word with every other words. For that I used the following formula in gensim:

model.most_similar([word])
model.save('text.txt')

I want to save the result of cosine-similarity of each words with respective to other words in such a way that I can used them further in classification and clustering. But don't know how to retrieve the results which are saved using gensim model ?

The question is not really clear.

If the question is how can you save and retrieve the cosine-similarity of each word with every other word, then you can save them in an array and use pickle for saving and loading them again.

Or, If the question is how you can retrieve the gensim model, then the function is gensim.models.word2vec.Word2Vec.load()


How to find vertical clusters in 1-D data I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters and get the indices of the points in each cluster so that I can go back and look at the actual data and get the mean values of various parameters of a cluster.

I tried 1d Gaussian ,Kmeans etc but they all seem to cluster the data horizontally.

I want each spike to be a separate cluster and get the indices of the values in the cluster. Can any one suggest a technique to overcome this issue.
Thanks.

You might find some more useful methods if you look for event detection or outlier detection, rather than clustering. Given the shape of the spikiness in the data, you could try using the mean value of the time series as a threshold, then contiguous chunks of values that are greater than the global mean will be your abnormal states. Now this method might be a little bit fragile in the long run, so you might want to make it more sophisticated. You could use the mean value over a large rolling window, rather than the global mean, if you are more interested in local changes.

If you wanted to keep things in a clustering framing, then it might work better if you did the clustering in 2D, i.e. your points will be (time, value) tuples/vectors, rather than doing the clustering on the 1D data. Another thing that would probably work would be to use K-means clustering on the 1D data, but only have 2 clusters (normal and abnormal) you would then have to use time information to separate individual events/anomalies.


How to calculate perplexity in PyTorch? I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]

if(train_on_gpu):
    net.cuda()

counter = 0
n_chars = len(net.chars)
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1

        # One-hot encode our data and make them Torch tensors
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output, targets.view(batch_size*seq_length))

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
...
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
    ...
    loss = criterion(output.view(-1, ntokens), targets)
    loss.backward()
    total_loss += loss.item()
    log_interval = 200
    if batch % log_interval == 0 and batch > 0:
        cur_loss = total_loss / log_interval
        ...
        print('ppl {:8.2f}'.format(math.exp(cur_loss)))
        ...

As @SpiderRico reminded, I got it from this link


How to interpret skimage orientation to straighten images? I have a bunch of images that I am trying to straighten so the images are horizontal (major axis is horizontal) but I don't understand the orientation output from regionprops method in skimage. How to convert it into degrees ? What is the axis reference for the output angle ?
Here is the skimage doc:

orientation : float. Angle between the 0th axis (rows) and the major axis
of the ellipse that has the same second moments as the region, ranging
from -pi/2 to pi/2 counter-clockwise.

My question:
Given the orientation, how do I calculate the angle in degrees to rotate the image so that the major axis is horizontal with skimage?
Code sample
Basically, the main major and minor axis of this object belong to index 1 of the pandas dataframe. The orientation of the object is -1.184075 and should belong to the major axis.
from skimage.measure import label, regionprops_table

# connected pixels of same label get assigned a value
label_img = label(binary_image_here)
props = regionprops_table(label_img, properties=('centroid',
                                             'bbox',
                                             'orientation',
                                             'major_axis_length',
                                             'minor_axis_length'))

df = pd.DataFrame(props)

Based on the doc you provide, orientation is in radians, ranging from -pi/2 to pi/2 counter-clockwise:

orientation : float. Angle between the 0th axis (rows) and the major
axis of the ellipse that has the same second moments as the region,
ranging from -pi/2 to pi/2 counter-clockwise.

Moreover, as it is said in the regionpropos doc, since 0.16.0, they use "rc" corrdinates everywhere. They say for the majority of computation you could find the old results with a simple transformation, but for others, it is more complicated, such as the orientation:

For example, the new orientation is π/2 plus the old orientation.

Which means that the right formula to get the angle you want is this one:
angle_in_degrees = orientation * (180/np.pi) + 90

And the orientation refers to this angle on the image:

Now: If you want your major axis and the 0th axis align, then rotate your image by -angle_in_degrees:
from skimage.transform import rotate
rotate(image, -angle_in_degrees, resize=True)


When is a Model Underfitted? Logic often states that by underfitting a model, it's capacity to generalize is increased. That said, clearly at some point underfitting a model cause models to become worse regardless of the complexity of data.

How do you know when your model has struck the right balance and is not underfitting the data it seeks to model?

Note: This is a followup to my question, "Why Is Overfitting Bad?"

A model underfits when it is too simple with regards to the data it is trying to model.

One way to detect such situation is to use the bias–variance approach, which can represented like this:

Your model is underfitted when you have a high bias.

To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training and test errors:

High bias: This learning curve shows high error on both the training and test sets, so the algorithm is suffering from high bias:

High variance: This learning curve shows a large gap between training and test set errors, so the algorithm is suffering from high variance.

If an algorithm is suffering from high variance:

more data will probably help
otherwise reduce the model complexity

If an algorithm is suffering from high bias:

increase the model complexity

I would advise to watch Coursera' Machine Learning course, section "10: Advice for applying Machine Learning", from which I took the above graphs.


Best practical algorithm for sentence similarity I have two sentences, S1 and S2, both which have a word count (usually) below 15.

What are the most practically useful and successful (machine learning) algorithms, which are possibly easy to implement (neural network is ok, unless the architecture is as complicated as Google Inception etc.).

I am looking for an algorithm that will work fine without putting too much time into it. Are there any algorithms you've found successful and easy to use?

This can, but does not have to fall into the category of clustering. My background is from machine learning, so any suggestions are welcome :)

Cosine Similarity for Vector Space could be you answer.
Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity?
"This is a tree",
"This is not a tree"
If you want to check the semantic meaning of the sentence you will need a wordvector dataset. With the wordvector dataset you will able to check the relationship between words. Example: (King - Man + woman = Queen)
Siraj Raval has a good python notebook for creating wordvector datasets.


How to get an intuitive value for regression module evaluation? For regression module evaluation, I think only the MAE (Mean absolute error) value is not objective or practical.
 Consider following situations:

A
MAE=1 while target value follows the uniform distribution on the interval [1,100]
B
MAE=1 while target value follows the uniform distribution on the interval [1,10]

Obviously A model is better.

So how to get an intuitive value for regression module evaluation, regardless the data set's target value's scope?

To get an intuitive contrast, I came up with:

Refer to
$$R^2=1-\frac{\sum ( y_i - \hat y_i)^2} {\sum (y_i - \bar y)^2}$$

My method is:
$$My Value=\frac {\sum |y_i - \bar y|} {\sum | y_i - \hat y_i|} $$
Indicating how many times the model better than other model, always predicts mean value.


Training images with multiple channels So I have a set of images with 16 layers each.
Is there a good reason to split the channels in a format like
example :   [images_length,16,image_width , image_height]
instead of creating something more simple like
example 2 : [images_length,1,image_width * 16 , image_height]

In the second example I use a single channel and lay each channel next to the other in order to create an image that has 16 times the length of the original image, but same height etc.

So here are the actual questions:

Are there any negative trade offs on using the second method ?
Which method is more memory efficient ( on the graphics card side of things ) ?

I tagged both keras and tensorflow as I use tf as backend for keras.

Thank you for your time!

I don't know what kind of image you have, 16 channels?! oh boy :)
Anyway, if they are images, the first one is better. The reason is that in the second approach you are somehow unrolling the input signal. By doing so, you are removing the information about the locality. You are removing the information of near adjacent inputs. Convolutional neural nets attempt to find these kinds of features. As an example, consider the MNIST dataset. You can learn using CNN and MLP but the former is used due to the fact that CNNs care about patterns which are somehow replicated in different parts of inputs.
If they are not images and you are aware that adjacent pixels or inputs are related, again you should exploit CNNs. Consider that the convolutional layers are CNNs are for extracting appropriate features. The classification task is done using dense layers in CNNs.

About efficiency, consider two points. Graphic cards are called SIMD computers. It stands for single instruction multiple data. Matrix operations are done using GPUs very efficiently as the name of Graphic cards implies. Consequently, Dense layers are very fast in GPUs compared to CPUs. The other point is about parallel programming. Each filter in convolutional layers are independent; consequently, they can be applied using paralyzed instructions. Again appropriate GPUs are very good at this too. I know I've said two things but there are actually three things to be kept in mind. Forget all of the above mentioned points! The most important thing to consider is the memory and the bus. There are situations that you don't have a graphic card with more than 6 Gig of memory. In those situations, I really prefer the last generations of CPUs instead of GPUS. The reason is that you have to deal with the limitations of memory.

In your case consider that if you use dense layers exactly after inputs, the number of parameters will be science fictional although You are supposed to use CNNs.


Dropping features after final evaluation on test data Would you please let me know if I am committing a statistical or machine learning mal-practice in this procedure?
I want to estimate meteorological variable y1 from ${x_1, ..., x_{10}}$ variables. I use data from different weather stations. I keep some weather stations as test sites/data.
I do feature selection and hyper parameter tuning with cross-validation on training data. My model is Random Forest (RF) and two other tree based models.
Before I evaluate my models on test sites I was skeptic about keeping one of the features: Elevation of the weather station, $x_{10}$. This is a static feature that would be present/same in all rows of data related to a station. Knowing a tiny bit about RF made me worried that the model will use this as a kind of "site_id" and possibly overfit to this feature. It wouldn't make me worried if I was using linear/nonlinear regression models.
So I train my models once with and once without $x_{10}$ as a feature.
Then I evaluate my models on the test sites and turns out that the models without $x_{10}$ do significantly better on test sites.
Even before testing this hypothesis about the static feature, I wanted to do similar tests with dropping other features as well, say $x_{9}$.
Now my question is: now that I know that $x_{10}$ hurts my model, I like to retrain my models without $x_{10}$ and test model performance with and without $x_{9}$ in the filtered feature set.
To me it seems like Im using my test data to kind of filter out my feature so it is not right.
But then, I have this information and if $x_{10}$ is hurting my models in the end, why should I go on testing hypothesis and preparing my models with $x_{10}$ being in them?

What you're doing is manual feature selection based on the test set. You're right that it's not correct to proceed this way: in theory, feature selection should be done using only the training set and a validation set, not the final test set. The risk is data leakage: you're modifying the model using information from the test set. Maybe the performance is better without these features on the test set because they happen to be bad for this particular current test set by chance. As a result the model could be overfit, and you wouldn't be able to detect this problem on this test set since it's the source of the overfitting.
So in principle it's always better to separate the data first, keep the test set aside until the final evaluation and use a validation set for intermediate evaluation until the final model (including the set of features) is determined.
In practice, it sometimes happens that we realize that we should have done something differently after applying the model on the final test set. It's a mistake but it's not the end of the world, usually the risk of bias is low. As you said, obviously we don't have to ignore an information which is important for the performance of the model. However if you know that you're going to repeat this procedure with several features, you should defintely do it using a separate validation set (taken from the training data), not the test set: the more you use the test set like this, the higher the risk of data leakage and bias.


Antonym search for expanding search terms I some code that identifies ngrams of interest in a small bit of text, then searches for the ngram in a larger bit of text to provide a snippet of the term used in context.  One ngram might be unordered numerical set.  Right now, if I can't find unordered numerical set, I start trimming the string from the left to see if numerical set is available in the larger text.

What I would like to do before I trim from the left, is to see if the any of the words have an antonym, like ordered for unordered.  This is because my ngram might be defined in one way, but used in the opposite.

Is there a known list or other way of finding out if there is a highly correlated antonym I might try for a set of words?

For instance, I think I could write some regexps that look for prefixes, like "un" or "dis" and see if the word that results from removing these prefixes is a valid english word.  This seems likely to be already solved, so before I try to create anything, I want to find out what might already exist.

I am currently using Python using Gensim, NLTK, and Word2Vec for the rest of my processing if it matters.

That's an interesting problem.

This according to me is the most comprehensive way(if speed is not a problem. Or you can just pull all these words and create a dictionary/database of your own).

You can this ,
https://wordsapiv1.p.mashape.com/words/love/antonyms (More info about this API at this link)

However, you can restrict results to antonyms with this api.

You can use requests to make API calls.

import requests
import simplejson
response = requests.get(url)
result = simplejson.loads(response)

Then search for the antonyms from the result. If you are getting a big list of antonyms, use only the top n results for your searching.

Although, W2V gives most commonly used words in the context of the keyword, it's hard to guess which is the antonym of the keyword.


Policy Gradients vs Value function, when implemented via DQN After studying Q-learning, Sarsa & DQN I've now discovered a term "Policy Gradients".

It's a bit unclear to me how it differs to the above approaches. Here is my understanding, please correct it:

From the moment I first encountered DQN, I always imagined DQN's input vector as only consisting of current state's features. On the output layer of DQN we have a vector of actions.  We then take the index of the highest-scored action $a$ and execute it.  It puts us into the next state $s'$.
To compute the error (how incorrectly we estimated the score of $a$) we supply $s'$ into our DQN, then discover its highest action similar to 1.  Once again, it will be on the output layer of DQN
Compute the error by bootstrapping $a$ to $a'$. The "error" vector will have zeros everywhere except at the index of that chosen action $a$

Was this in fact a "Policy Gradient", and not a Value-function approach as I thought initialy?

In that case, would the value-approach be a DQN with:

[state_features; action_one_hot_encoded_vector]  concatenation as input, and a single value on the output?

I got this impression after reading this Link

Is the basic idea of DQN Policy Gradient that simple, or am I getting things wrong?

Edit: there is a really awesome lecture about policy grads. Unfortunatelly the video is deliberately unlisted, so that normal people can't really get to it - but I am for free education, so here it is:  CS294-112 9/6/17

Is the basic idea of DQN Policy Gradient that simple, or am I getting things wrong?

This is not correct. DQN is a value-function approach, as you thought initially.

Your confusion seems to stem from the two options for action representation possible in DQN. You can either estimate a single $Q(s,a)$ function by having $a$ as an input to a neural network, or can estimate all possible $Q(s, *)$ with multiple outputs, one for each $a$. However, that is an implementation detail in how you set up the neural network. It might change the efficiency of the resulting system, but it doesn't change the nature of the RL algorithm at all.

Policy gradient methods are based around modifying a parametric policy function $\pi(a|,s,\theta)$ and learning $\theta$. The most basic policy gradient algorithm is REINFORCE, which requires an episodic problem, and updates on-policy after each episode.

Importantly, you cannot use the relationship between $Q(s,a)$ and resulting policy in Q-learning to create a policy gradient approach. The main blocker to doing this is that the effective policy $\text{argmax}_a \hat{q}(s,a,\theta)$ is not differentiable with respect to $\theta$, so it is not possible to calculate a gradient and make an update to improve the policy.

In general, if an algorithm learns a state value $V(s)$ or action value $Q(s,a)$ and then uses that to decide the policy, then it is value-based. If it learns the policy directly then it may be a policy gradient method (in RL terms it is very likely to be policy gradient, but you can also do policy search using e.g. genetic algorithms).

Policy gradient methods also include Actor-Critic approaches which learn both the policy and an associated value function (usually state value $V(s)$). This is a more advanced algorithm than REINFORCE, in that it can be applied to continuous (non-episodic) problems, and updates estimates on every step. One popular Actor-Critic approach is A3C


How to refit a model? I have built a classification model that I have to refit.

New instances have been added to the former data set on which the model was build.

I was thinking to proceed that way :

split my new data set in 3 sub data sets : ( train , test_01 ) data sets to refit the model , ( test_02 ) test data set to compare the original model and the refitted one ( on the metric formerly used ).

Of course the ( train , test_01 ) data sets have to contain the original ones ( those used to fit the former model before being refitted ).

Is this the correct way to proceed?

To make it a fair test, you need to ensure that neither model has seen any of the data in test_02. The closer you can get each subset of the data to match the distribution of the overall combined data set, the better. If you steer by that, the approach you suggested is fine.

A more direct way of answering the question of whether you need a new model is to work out the average accuracy of a new model trained on the updated data set (with the new samples) by using k-fold cross-validation, and then compare that to the accuracy of the average model trained on the old data and tested on the combined data.


Training the document page layout and classifying good/bad layouts I have a use case where I am supposed to get the coordinates of each block element in a page (whether its paragraph, image, table) where I train a model to understand how they are placed in a given page where some documents are with good layout and other with bad ones and I want to train this and throw in some coordinates of a new document and try to understand whether it has a good layout or a bad layout, I want to understand how I can achieve this using some deep learning techniques ?

can someone suggest me an approach for solving this?

Was trying to workout with RNN but not sure if that's the correct approach.

When you're talking about layout, I am assuming you are talking about the way elements are arranged in the page. In that case, dividing your page into grids and having each element represented into grid numbers should solve your problem. Let's assume your page is divided into 9 x 9 grids, then you can have this data frame as an input. Grids can be replaced with your co-ordinates. You can add in more features like font-size, font-style, colour, etc.,

number header1 header2 header3 paragraph1 font-size-h1 font-size-h2 target
1      (2,3)      4     (4,2)     3           12           8           0

The tuple represents the co-ordinates and the individual number represents the grid number. The font-size can be added for each element which will increase  the dimensions of the dataset. May be try running a simpler algorithm like Random Forest first and then move onto deep learning techniques. Sometimes, simpler algorithms work better than complex algorithms.

Aesthetic Value of a Webpage, Layout Classification


What do graphs of signal vs background neural network outputs represent? For example, this image shows the output of a neural network I assume but I am not sure how the output is not symmetric. So if the neural network gives a .6 for signal it should give a .4 for background.

This is my neural network output that is symmetric. However, I have not seen anyone else get a symmetric output like this. Does this mean that I'm doing something wrong or does it mean that the first graph is not the output of the neural network.

Okay, your output makes sense, given what you wrote in the comments. But if you don't know where the other output comes from, all else is speculation. It could be that it was generated by a neural network with two sigmoid outputs, rather then softmax, and the classification was done based on which output was larger. In that case you are not guaranteed symmetric graphs. So the asymmetric graph does not necessarily imply that it's not generated by a neural network, only that the output does not come from a two-dimensional softmax or a pair of sigmoid/1-sigmoid.


Generally, without knowing the source of the data, we can't tell you much about the columns. But I assume the first two columns correspond to $x$ and $y$. The third is probably some meta-data? Maybe a cluster number?
For an illustration I found this figure coming from this publication. Maybe that helps you imagine the data's shape.
https://www.researchgate.net/publication/338167806_A_Fast_Method_for_Estimating_the_Number_of_Clusters_Based_on_Score_and_the_Minimum_Distance_of_the_Center_Point


Is it possible to run clustering methods by only knowing the distance between pair of points? By knowing each data point's coordinate, it is easy to apply them with clustering methods as k-means etc. By if the case is we only know the distances between each pair of data points without knowing the definite location coordinate of every data point, is it possible to apply any clustering methods in this case?

K-Medoids

It would be possible with an adapted semi-supervised K-Means, also known as K-Medoids.

The tricky part with K-Means is that you do not know the centroids. However, you could hot start by assuming that some of your data points are centroids. Then, when figuring the new centroid at each iteration, instead of figuring out the "imaginary" central position, pick the point in the cluster that minimizes the sum to all other points in the cluster.

Hierarchical Clustering

You could also try a hierarchical clustering method. One example is the AgglomerativeClustering from Scikit-Learn. The idea is that you start merging points that minimize their linkage distance. Then, there is a certain criteria for determining points that are "too far away" and a new cluster should be created.

This is the fit() method documentation. Notice how you can either pass the instance features or the distance matrix between instances.


Framing Sentences based on keywords Given a few specific words, which techniques of Natural Language Processing can I use to achieve creating a meaningful sentence from those words?

eg.

Words:
jackets, highest sale, sweaters, lowest sale

Sentence:
Jackets exhibited the highest sales while sweaters sold the least.

If my question is too broad, let me know so I can ask more specific questions.

I resolved the issue by making use of the Natural Language Generation, Python Library , nlglib

The solution I came up with is depicted through the Python Code below.

import nlglib
realise_en = Realiser(host='nlg.kutlak.info', port=40000)

p = Clause(NP('jackets'), VP('achieve','highest sales'))
p['TENSE'] = 'PAST'
q = Clause(NP('sweaters'), VP('exhibit','lowest sales'))
q['TENSE'] = 'PAST'
print(realise_en(p))
print(realise_en(q))
r=Clause(realise_en(p)[:-1],'while',realise_en(q))
print(r)

This prints the statements

Jackets achieved highest sales.
Sweaters exhibited lowest sales.
Jackets achieved highest sales while Sweaters exhibited lowest sales.

This is the best solution I found so far.


Multi Label Classification on Data Columns in Tables I am seeking guidance on a machine learning problem involving the tagging of data columns. Currently, I have a system where users can add multiple tags to a columns in a table. However, I want to automate the tagging of new columns by using Multi Label Classification. I have extracted 21 features from each column by doing a column analysis on the column values. The features obtained would include statistical values such standard deviation, max,min, kurtosis and etc. Am I on the right path in using these features as inputs for a Multi Label Classification model ? Right now I am focusing on numeric values in columns

Example:

As an example, the above table on the left represent some arbitrary table which consist of 3 columns. As a user I would tag the column with the appropriate tags. So the Rainfall column would have rainfall and precipitation and Temperature column would have temperature. The table on the right just represents the tags being assigned to a column in a table format.

Example sample data set in the above image

In order for me to do multi-label classification to automate the tagging of columns automatically, when tables with similar columns are ingested into the system, I would need to extract some features or properties that describes the already tagged columns to use as input for the multi label model. So I did some column analysis and placed just several example features in the table above. This includes standard deviation, maximum,minimum, median and kurtosis. I have about 21 features in total. The output labels are also represented for each column in the above image where 1 signifies the label is present and 0 is not present.

In the end the model will decide which tags are assigned to a newly discovered column based on its features.

@DPascal Here is something you could definitely try doing:

Using the features you generated, you could add a label to each of the column for these features.
You could generate these feature value for different time slices.
Once that is done, you can then run something as simple as Random Forest Classifier on this data.


Unsupervised learning if existing image captions match the images I need to train a system on a large set of images and associated captions to determine which (image, caption) pairs are correct and which are not. I don't have any labeled pairs, but I can assume that more pairs are correct than are not. Furthermore, the images and captions have small subsets that are reasonably similar, but not exactly.

Current idea

My current idea is to build a sort of GAN architecture where I have a generator and a discriminator, except that the generator is simply sampling from the existing corpus of images and captions to generate false pairs and the discriminator is in charge of determining which pairs are correct or not.

images, captions = [], []
bpg = bad_pair_generator(images, captions)

clfr = classifier()

for epoch in range(epochs):
    good_pairs, bad_pairs = zip(images, captions), bpg.generate(N)
    X, y = good_pairs + bad_pairs, [1]*len(good_pairs) + [0]*len(bad_pairs)

    clfr.train(X, y)

    trickiness = [yi == yi_pred for yi, yi_pred in zip(y, clfr.predict(X))]
    bpg.fit(bad_pairs, trickiness[-len(bad_pairs):])

After this, I would hope that the classifier was stressed sufficiently to learn a good model for whether the pairs are good or not.

Questions

Are there better researched approaches?
If not, does this approach seem sane and reasonably safe?
What kind of model could I use to compared the image to the text? Clearly a CNN/RNN would be good for the image/text, but how do you combine them?

A thought:

Maybe an approach like Show and Tell: A neural image caption generator would be useful to look at. There they use exactly as you say a combination of CNN (for image) features and LSTM (for text) + search to generate text given the features from the image.

It is possible that then a discriminator can be used to learn whether 'generated text' is close enough to 'given label' in order to give the final result.

It is also possible that you do not need the discriminator, and instead of using the output of the show and tell network directly (like removing the last linear layer) and feed one of the earlier steps (which represents a probability distribution of possible texts to generate) into the discriminator so that it has more possible combinations of text outputs to work with...

This is probably more supervised than you want, but maybe starting with pre-trained weights of this model can get you somewhere faster.


What to do if a specific label of a category appears only a few times? Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its color, model, etc. The model is the name of a car(e.g: Sportage, Mazda3, etc.) and some of the more famous models such as Sportage appear many times whereas some of the less popular ones might appear only once or twice. In that case, what would be the ideal way to deal with this?

More info:

In my case, I have about 3000 different car models and the first two or three make up about 20% of my data but the rest just appear once or twice in the entire dataset. I have tried one-hot encoding and that did increase my score immensely but it's still not good enough (I know as a matter of fact it could be better).

P.S: I have already looked at the posts regarding a high cardinality and although I do think it's related to my problem, it's still a different issue.

Thank you so much!

Since a few cars models make your 20% data, you can create a similarity matrix of all the car classes with these 2-3 car models. So in this matrix, each car model has 2-3 values depicting its similarity with those 2-3 car models. You can now add these new features to your existing features to get some improvement over the classification of the underrepresented car models.

You may play with the number of cars models to compare each car model to. Here, these 2-3 car models cover 20% of your data. It might be that 30 car models would cover say 40% of the data which is still useful as there are 3000 car models.


What can i do after a PCA with the results? After performing a PCA and studying the proceeding i ask myself what the result is good for in the next step. From the PCA i learned how to visualize the dataset by lowering the dimension, i got handy new vectors to describe the members of the population in a more efficient way and i learned which original predictors correlate and contribute more than others.

I ask myself questions like, is there more to learn from PCA, is it a good idea to use the PCA model for a learning algorithms to perform better and is it possible to respect nomial or ordinal predictors in the PCA somehow?

Well PCA, as suggested above by @CarltonBanks, does help you remove features with the least correlation and use mash the features together such that they have the highest correlation.

To answer your question, how to visualize higher dimensions using PCA

Transform the feature matrix with the number of components of your data set to 2 or 3
This ensures you can represent your dataset in 2 or 3 dimensions. To simply see your answer just plot this transformed matrix into a 2d or 3d plot respectively.
This helps you visualize a higher dimensionality data as a 2d or 3d entity so while using regression or some predictive modeling technique you can assess the trend of data.

Should we use PCA in machine learning algorithms more often?

Well, that strictly depends, using PCA reduces the accuracy of your data set so unless you need to save up some space caused due to a lot of features with bad correlation and the overall accuracy doesn't matter. If your machine learning model scenario is similar to this then it is ok to proceed.

However, most use of PCA is as you asked before for visualizing higher dimensionality data to determine the data trend and to check which model fits best.


AWS machine learning prediction schema problems I've trained an AWS Machine Learning model with the training data from here : https://www.kaggle.com/c/titanic/data

I'm now trying to run a batch prediction with the test data from the same source but I get the following error when I try to load the data : "  The schema in this data file must match the datasource used to create the ML model ml-xxxxxxxxx. Ensure that the data file you are using matches the schema structure."

The schema, as far as I can see, is identical. I have tried it with and without the 'survived' column which is the value I'm trying to predict. I even tried it with the same training set which obviously has an identical schema and got the same error.

What am I doing wrong?

I ran into the same problem today, tried googling for people having the same issue and found your question.

I solved my problem by creating the data source first and then running the prediction from there. So, instead of selecting the following option,

Batch Predictions > Create new batch prediction > ML model for batch prediction > My data is in S3, and I need to create a datasource

which fails, I first did:

Datasources > Create a new datasource...

Next, I ran the batch prediction from an existing datasource successfully.


Mapping a set of corrupted strings to the correct ones I am fairly new into Data Science but encoutered it before. The following problem troubles me and i hope you guys can point me in the right direction.

The input are some strings where some carry the same information others not. An unknow number of these strings are crooked* to a warrying degree. From only one letter off to complete garbage. On the output side are the corrected strings from the input. The catch is that there are only certain, already known, combinations of valid strings possible.

In a naive approach i chained some fuzzy searches and already got some promising results. Now i don't know where to start or if there are similar problems already solved.

* (are we still allowed to say this?)

That problem is called approximate string matching or fuzzy string searching. It has been well studied in computer science.
If there is a limited, known collection of valid strings (aka, a dictionary), then the problem can be framed as spell correction.


Publish without validation score? My mentor wants me to write and submit an academic paper reporting a predictive model, but without any validation score.

Everything I have read in textbooks or the Internet says that this is wrong, but is there any case where only reporting a train score makes sense?

Background

The model was fit "by hand" by someone in our team, using a visual inspection of features extracted from our entire dataset. It is a linear model based on hand-crafted features extracted from some very nonlinear and high-dimensional data. The linear model is based on less than fifty features, but those features were extracted from thousands. We do not have any data left to use as validation.

You are right: decent journals/conferences are unlikely to accept a paper without a proper evaluation. Moreover the model is handcrafted, which probably means that it's hardly reproducible, right? And also that you can't do cross-validation I guess?

I think one would need pretty strong arguments to justify a contribution which has no scientific validation and cannot be reproduced. The only ways I can think of would be:

if the method solves a problem never solved before (or has specific characteristics which make it likely to solve such problem in the future)
if some sort of qualitative analysis demonstrates that the method is much better than state of the art approaches


Group datetime64 values per week in dataframe I want to plot a bar of my data grouped per week. I made the following dataframe, with nonsense data, but it should do the trick.

What I want to do is plot per week a bar chart (%)

Does anyone know how I can do this? I'm using datetime64 so can this be done easily?

I want to make one graph which contains 52x3 bars.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import random

date_expected = np.arange('2005-06', '2006-07', dtype= 'datetime64[D]')
cat = ['True','False', 'Maybe']

value = [random.choice(cat) for i in range(len(date_expected))]

data = {'Date_expected': date_expected,  'Value': value }
df = pd.DataFrame(data)
df = pd.DataFrame(data)

print(df.head())

Is this what you want to do? I am not sure why you need 52 x 4 columns, since you have 3 categories in your Value field.

What I did is to create a new column with the number of Week in that Year, using the dt.week.

import pandas as pd
import numpy as np
import random

date_expected = np.arange('2006-01', '2006-06', dtype= 'datetime64[D]')
cat = ['True','False', 'Maybe']

value = [random.choice(cat) for i in range(len(date_expected))]

data = {'Date_expected': date_expected,  'Value': value }
df = pd.DataFrame(data)
df = pd.DataFrame(data)

df['week'] = df['Date_expected'].dt.week

df.groupby('Value').week.value_counts().unstack(0).plot.bar(figsize=(10,30))


How can I figure out section headings in a document? I have a large number of different XML documents with associated style sheets (SEC financial filings). Are there any natural language processing (or other) tools that I can use to try to automatically extract section headings?

It seems you are lucky and the section headings are XML-tagged. So you can use XSLT transformations or Python lxml.etree to get rid of them or to extract them.


Training and Test set I was asked by my supervisor to replicate a result from a former graduate student. My supervisor believes the result of that paper is not accurate and he asked me to find out why! The paper was about conducting a random forest classifier to classify some sort of diseases. I was reading through the simulation process, and I realize that the training set and the test test were generated separately (the same set of parameters were used in the simulations). In other word, the training was simulated first, then another simulation was carried on to generate the test set. My understanding for training and test set is to simulate one data set then use the same data for training and test set.

My question, is it correct to simulate the training and test set separately? Does it affect the accuracy of the classifier? any reference will be truly helpful.

In general, generating independently training and test sets is a legitimate option. The crucial aspect is that both the generating processes are equal. You can check this looking at this example from the author of the caret R package and the Applied Predictive Modeling book.
However, it is something that can be easily proven with simulations. In what follows, both generating independently training and testing data or splitting the same data in training and testing subsets give the same results. The glm has a median accuracy of 92%.

# simulations with training and test data generating at the same time
n <- 100
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n)           # generate X
  z <- 1 + 4*x + rnorm(n) # linear combination with error
  pr <- 1/(1+exp(-z))     # inv-logit function
  y <- pr > 0.5           # 1 (True) if probability > 0.5
  df <-  data.frame(y = y, x = x)
  train <- sample(x = 1:n, size=n%/%2, replace = F) # sampling training data units
  glm.fit <- glm(y ~ x, data = df[train,]) # fit on the training data
  predicted <- predict.glm(glm.fit, newdata = df[-train,]) # predict on the other data units
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df[-train,]$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy

# simulations with training and test data generating independently
n <- 100 # dataset size
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n%/%2)           # generate X
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.train <-  data.frame(y = y, x = x)
  glm.fit <- glm(y ~ x, data = df.train) # fit on the training data
  # generating independent test data
  x <- rnorm(n%/%2)
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.test <-  data.frame(y = y, x = x)
  predicted <- predict.glm(glm.fit, newdata = df.test) # predict on the test data
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df.test$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy


How to detect the begin word and end word in a sentence with machine learning I have some English text that has been tokenized. For example, the length of text token is about 20000 and each word (tokenize) has an index. Also, each index has a label, as the beginning word in a sentence labeled as 'b', as an end word (include symbol) in the sentence labeled as 'e', with other words labeled as 'o'. There is labeled training data and unlabeled test data.

My question: how do we use a machine learning method or deep learning model to solve the issue that predicts the label of words in the test data?
I mean how do we make the label and data for training just a word for training? I am confused.

For this kind of problem, you can study some of the latest research areas of reading comprehension in which people are trying to predict the starting index of the answer & with the help that index they are predicting ending index of the answer.

Following screenshot took from BiDaF research paper
Refer to BiDaF for more details


Manually creating plants images dataset for machine learning plant type classification and leaf segmentation A group I work with wants to create its own plants data set that will be used for multiple projects like plant type classification and leaf segmentation for starters.
They are willing to provide all the means necessary for the creation of this data set. A partner agriculture company is willing to assist with planting what ever is needed and have someone monitor and capture images every X duration as per requirements. Different growth stages are desired as well.

I was asked to write the requirements for how the images should be acquired, however I'm not quite sure of that myself. What are some key points one should know for such a task and are there any known guidelines or tutorials available for data sets creation?

Note that currently for this task only 1 plant will be captured in each image as the non-learning methods that will be used to label and segment the images support only 1 plant per image.

I was thinking that based on the time it takes a specific plant to fully grow (which is known, approximately), N images should be captured every X days covering 360° of the plant. Also, images are to be captured from different angles and distances.

Some questions I have in mind that I cant seem to answer yet,

Number of images, the more the merrier as long as there are differences?
What about the location of the (single) plant in the image? Is it crucial or could it be augmented after the data set has been acquired?
What about camera resolution? is any decent camera sufficient for the task or should it be high end?

I'm still researching what other considerations there are and would appreciate any information in that matter

Well, it all depends on what you plan to accomplish with this dataset:

If you are planning on using drones to identify crops in the wild then you should use a drone to take the images and get images in the wild. (By in the wild I mean a free environment that differs from a lab or something like it)
If you are planning on monitoring plants on a constrained environment using fixed cameras so you should gather images on this conditions.

As Rajat pointed, the camera you choose needs to be able to grab all the feature needed to the identification, maybe machine learning approaches can identify plants with way less features than human eye does but maybe it can not. Either way you should consult with someone with expertise on plants to decide what method you gonna use.

Numbers can't hurt:

Well, they can give you some extra work but the more the merrier. One thing you need to keep in mind is that datasets must be representative enough to enable learning methods to work on the real world scenario. For example, if plan on training a algorithm to classify images token from above there is no reason to train it with images from under the plants.

If you plan on using deep learning, (i.e. you have the time and resources to train it) you should gather a really large number of images or find a way to augment them with meaningful transformations. One way is to use simulated images, from a computer rendering (that is used in UnityEyes for example) and maybe augment it to be more realistic using GANs.

In short terms, my advice would be to formalize what you want this dataset for, consider what restrictions you can guarantee on the test environment and this should give you a list of what you need to create your training dataset.


As with anything, it depends. :) Your question will very much depend on the model.

This question is reminiscent of the broader debate about discriminative versus generative models. Usually, we work with discriminative models. That is, you model $\text{P}(Y=1\,|\,X_1, X_2, \dots)$, where $Y$ are your classes and $X_i$ your observations.

There is another class of models: generative models, like Restricted Boltzmann Machines, which model joint distributions, $\text{P}(Y,X_1, X_2, \dots)$. They can be used for classification because you can easily turn a joint probability into a conditional probability. These models usually are broader in scope, and usually do not perform as well, which is why they aren't used as much. But they do exist.

Assuming that you are talking about discriminative models, like decision trees or the logistic regression, then what you want is how to invert this conditional probability, so that you have a distribution along $X_i$; that is, $\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y)$.

From Bayes's theorem, $\text{P}(Y=1\,|\,X_1=x_1, X_2=x_2, \dots)=\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y=1)\frac{\text{P}(Y=1)}{\text{P}(X_1=x_1, X_2=x_2, \dots)}.$

So, yes, you can. Where is the catch?

$X$ and $Y$ have to be discrete. It is very easy to create a probability density function from categorical variables by just getting conditional probabilities out of your model by just trying all combinations of $X$. And, once you have a probability density function, then you can just use a sampling technique to generate values.

It is less clear to me how you could make this work for continuous values, unless you are willing to use an approximation by discretizing your observations. I think an exact solution will depend very much on the discriminative model at hand. You cannot do this alone by getting conditional probabilities out of the model.

But if sampling is important for you, then you should just use a generative model.


Extract seperated value from columns I am working on this CSV file which is a collection of movie details from IMDB.
In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|)
What I need is to extract the first two genres from the genres column and store them in two new columns: genre_1 and genre_2.
And for the columns where there is only 1 genre, extract the single genre into both the columns, i.e. for such movies the genre_2 will be the same as genre_1.

I am sharing the screen shots of the code and results that I have got.

Now, I can create a new data frame with the genres created and can then remove the unwanted columns and can concatenate the remaining the with original data frame. But that looks pretty clumsy.
How can I crate split the column in my original data frame only and remove the unwanted expanded columns.
Any help is appreciated.

Try:
# Create an example dataframe
df = pd.DataFrame({"genres":["Fantasy|Sci-Fi",
                "Action|Adventure|Fantasy",
                "Thriller",
                "Action|Adventure|Thriller|bbv","Action","Action|Adventure|thriller"]})

# Get a dataframe with as many columns as there are genres
df = df.genres.str.get_dummies(sep = "|")

# Get the genres as values
df = df.multiply(df.columns)

# Rename the columns to have the genre id
df.columns = ["genre_" + str(x) for x in range(len(df.columns))]

Input:

Output:

EDIT:
you can simply use pandas assign method:
df.assign(genre1= df.genres.str.split("|", expand = True).iloc[:,:1],
          genre2 = df.genres.str.split("|", expand = True).iloc[:,1:2])

*Output:


How many channels do ultrasound images have? I have some ultrasound images. When I read them using cv.imread and check their shape or check them with ndim, it shows 3 channels:
im = cv.imread(data_path+'a1.jpg')
print(im.shape)
im = np.array(im)
print(im.ndim)

the output is:
(154, 139, 3)
3

But when I open them in photoshop and check their mode in Image > Mode, it shows grayscale with 8 bit. Also in windows properties > detail, tab shows 8 bit:

Dose anyone have any idea about that?

What is happening has nothing to do with the type of image (ultrasound in this case). It has to do with the type of file (JPEG in this case). By default, cv.imread assumes it is a 3-channel color image. You can specify grayscale when calling cv.imread if appropriate. The last dimension (the color dimension) in the im object could be empty.

There is also depth info (8 bits per channel). The default is 8 bit, you could specify 16 bit or 32 bit if apporiate.

Do not rely on function or software defaults. It is better to work from first principles of the raw data.


Fuzzy and FuzzyWuzzy: what are the differences in text comparison? I have found a lot of information about fuzzy logic, but less information about fuzzywuzzy. I would like to know more about this, the function which determines the logic, if possible, and understand what partial_ratio in Python does.
Any information will be well welcome.

In the source code you can find a simple explanation of what does partial ratio in fuzzywuzzy does:

Return the ratio of the most similar substring
as a number between 0 and 100

In this code snippet you can find the differences
from fuzzywuzzy import fuzz

fuzz.ratio("this is a test", "this is a test!")
Out:    97

fuzz.partial_ratio("this is a test", "this is a test!")
Out:    100

[1] https://github.com/seatgeek/fuzzywuzzy
[2] https://github.com/seatgeek/fuzzywuzzy/blob/master/fuzzywuzzy/fuzz.py


Why does PCA assume Gaussian Distribution? From Jon Shlens's A Tutorial on Principal Component Analysis - version 1, page 7, section 4.5, II:

  The formalism of sufficient statistics captures the notion that the
  mean and the variance entirely describe a probability distribution.
  The only zero-mean probability distribution that is fully described by
  the variance is the Gaussian distribution. In order for this
  assumption to hold, the probability distribution of $x_i$ must be
  Gaussian.

($x_i$ denotes a random variable - the value of the $i^\text{th}$ original feature.i.e. the quote seems to claim that for the assumption to hold, each of the original features must be normally distributed.)

Why the Gaussian assumption and why might PCA fail if the data are not Gaussian distributed?

Edit: to give more info, in the tutorial, page 12, the author gave an example of non-Gaussian distributed data that causes PCA to fail.

TL;DR

PCA does assume normal distribution of features  See p.55 SAS book1 or Rummel, 19702 or Mardia, 19793.
If you expect the PCs to be independent, then PCA might fail to live to your expectations.
Assuming that the dataset is Gaussian distributed would guarantee that the PCs are independent.

Long Answer
PCA doesn't assume the dataset to be Gaussian distributed
Most of the sources I have found (e.g. wikipedia) don't list Gaussian distribution as a requirement of PCA.
Moreover, it seems that Shlens himself doesn't believe that anymore:
I found 2 more versions of Shlens' tutorial: version 2 and version 3.02. The latter seems to be the current version (as Shlens' web page links to it), so I will refer only to version 3.02 in my answer.
In version 3.02, the paragraph you quoted was removed from the "Summary of Assumptions" section, so that currently, the section lists only the following assumptions:

Linearity
Large variances have important structure.
The principal components are orthogonal.

When might PCA fail to live to our (false) expectations?
In page 10 Shlens gives an example for when one might see the result of PCA as a failure, and then explains why PCA didn't really fail:

The solution to this paradox lies in the goal we selected for the
analysis. The goal of the analysis is to decorrelate the data, or
said in other terms, the goal is to remove second-order dependencies
in the data. In the data sets of Figure 6, higher order
dependencies exist between the variables. Therefore, removing
second-order dependencies is insufficient at revealing all
structure in the data.

i.e. PCs are guaranteed to be uncorrelated, so that's exactly what we should expect them to be. However, if we expected the PCs to be independent, then we would consider PCA to fail when the PCs aren't independent (e.g. the examples in figure 6).
(See this answer for another explanation of this paragraph.)
How can the assumption of Gaussian distribution help with our (false) expectations?
If we assume that the original dataset is Gaussian distributed (i.e. the features are jointly normally distributed), then by definition every linear combination of the original features is normally distributed.
Each of the PCs given by PCA is a linear combination of the original features. Thus, also every linear combination of the PCs is a linear combination of the original features, and so every linear combination of the PCs is normally distributed.
So, by definition the PCs are jointly normally distributed. PCA guarantees that the PCs are uncorrelated, and therefore they are also independent.
(Note that in the original paragraph quoted in the question, Shlens seems to claim that each of the original features should be normally distributed. However, I believe that was a mistake, and he actually meant that the original features should be jointly normally distributed (I deduced that's what he meant mainly from footnote 7 in page 10 in version 3.02). This answer explains why these conditions are not equivalent in the 2D case. Similarly, they aren't equivalent for any dimension $>1$.)
Thus, under the assumption that the original dataset is Gaussian distributed, PCA guarantees that the PCs are independent.


Why do we need for Shortcut Connections to build Residual Networks? Why do we need for Shortcut Connections to build Residual Networks, and how it help to train neural networks for classification and detection?

The short answer is that when a net is very deep it becomes very difficult for gradients to propagate backwards all the way. Skip connections offer "short cuts" for gradients to propagate further and allow for efficient training of very deep nets.


Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are input into a BLSTM model.  I am using an AzureML server GPU-N6 for training.
machine specs : STANDARD_NV6 (6 Cores, 56 GB RAM, 380 GB Disk) GPU - 1 x NVIDIA Tesla M60
It exits with code 137, which is an out of memory error. I see the data is downloaded into the local node and NumPy array is created. the OOM happens when the batch processing starts.
I have tried to run the model with only one input (eGeMaps) but run into memory issues.
My question is, is there a way to batch load the NumPy file, similar to an image dataset? I have tried to split the input into individual data files and use tensor dataset or slices, but it doesn't work when loading a NumPy file. Ideally, I would like an equivalent of ImageDataGenerator for NumPy files.
EGEMAPS={'FEATURES':100,'TIMESTEPS':14050}
MFCC={'FEATURES':100,'TIMESTEPS':14050}
DENSENET201={'FEATURES':1920,'TIMESTEPS':1415}
The training dataset consists of 136 participants each with an array of the sizes mentioned above. I created a single large NumPy array, each row representing a participant and array of the features.
This is the model I am training.
#Create Model :
inputs_eGeMAPS = Input(EGEMAPS_INPUT_SHAPE)
inputs_MFCC = Input(MFCC_INPUT_SHAPE)
inputs_Densenet = Input(DENSENET201_INPUT_SHAPE)

blstm1_eGeMAPS = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm2_eGeMAPS= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm1_MFCC = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm2_MFCC= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm1_Densenet = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

blstm2_Densenet= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

audio_lstm_output = concatenate([blstm2_eGeMAPS,blstm2_MFCC,blstm2_Densenet])

dense1= Dense(500,activation='relu')(audio_lstm_output)

print(f"dense:{dense1}")

dense2= Dense(100,activation='relu')(dense1)
print(f"dense:{dense2}")

dense3= Dense(60,activation='relu')(dense2)
print(f"dense:{dense3}")

output = Dense(1)(dense3)
print(f"output:{output}")

model = Model(inputs=[inputs_eGeMAPS,inputs_MFCC,inputs_Densenet], outputs=output)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])

print(model.summary())

epochs = 1
hist = model.fit([BoAW_eGeMAPS_Dev_tf,BoAW_MFCC_Dev_tf,Densenet_Dev_tf],Y_train_tf,epochs=epochs, batch_size=batch_size,validation_split=0.2,shuffle=False)

Looking at your code, it looks like you're using Keras w/ a Tensorflow backend -- or Tensorflow functions using its Keras submodule. In that case, you can train your model using its fit_generator function.
As the name implies, if you convert your training data into a generator, it will substantially reduce the memory cost needed to train your model. Here's an example of training a Keras model using fit_generator and how you can preprocess your data to utilize it.


How do we predict what is in an image using unsupervised deep neural networks? From my understanding of unsupervised DNNs for image classification:

The input layer is a 4,096 dimension vector (for 64 x 64 images)
The hidden layers represent much lower "features" as identified by the back propagation
As the model is generative, the output layer is also a 64 x 64 image

Therefore, how do can we make a prediction that a new unseen image contains a specific image class (e.g. cat) if we lack labelled data?

After a lot of reading, I think I now understand. We really need to build 2 models.

Model 1

Unsupervised
Lots of unlabelled images
Used to 'learn features' (i.e. better that we have done manually through years of research e.g. edge detection, colour features etc).

Model 2

Supervised
Few labelled images
Use model 1 as the 'feature extractor'. i.e. pass a training image through model 1 and use the output layer as the feature vector.
Use the same approach to test images e.g. model 1 to extract features, then use the second model to output label predictions


How important is advanced SQL for data science? Is advanced level SQL required to be competitive as a data scientist? Is it more important for a data analyst to be good in SQL? Is it enough to be able to extract data using simple SQL queries?

I know it is faster to manipulate data in SQL than to copy data into R or Python, but are there any other advantages or disadvantages?

What do you mean exactly with competitive as a data scientist? Unfortunately, many employers will have different expectations of someone they hire to be a Data Scientist, so there isn't a single answer!

In any case, I think it is a good idea to know three components to be effective with databases:

Managing a connection: how to create and connect to a database using a simply library in Python or R. This sort of workflow should look familiar to you:

import mysql
connection = mysql.connector.connect(
                 host="127.0.0.1",
                 user="mj_whales",
                 passwd="somethingsecret",
             )

query = "CREATE DATABASE my_table"
cursor = connection.cursor()
cursor.execute(query)

Getting data: understand basic syntax of a standard SQL-style query, for example:

FROM my_table SELECT my_feature WHERE my_other_feature > 0

Writing data: understand how to insert data efficiently. This will depend on the database type e.g. document database, SQL-style, InfluxDB for time-series data.

Here is a really good introduction to databases with Python that might help you understand your current strengths and weaknesses on the subject.


Binarized neural network I am currently looking at a paper by Hubara et al  on binarized neural network.

I am stuck in understanding algorithm 2 of the paper. The algorithm uses shift-based (bit-shifting) AdaMax, where AdaMax is an extension of the Adam optimizer. In particular, they are using

$$m_{t} = \beta_{1}.m_{t-1} + (1 - \beta_{1}).g_{t} $$

$$v_{t} = \max(\beta_{2}.v_{t-1}, |g_{t}|) $$

$$\theta_{t} = \theta_{t−1} − (\alpha\oslash(1−β_{1}^t)).(m_{t} \oslash v_{t}^{-1} ) $$

where $g_{t}$ is the gradient, $\theta_{t-1}$ is the previous parameter, $\alpha$, $\beta_{1}$, $\beta_{2}$ are learning rate, and the betas of Adam optimizer. They stated that $\oslash$ stands for both left and right  bit shift. I know the left shift and the right shift on its own, but I am not sure how do we have both at the same time? Help will be much appreciated. Thank you.

I take a look at their github.

Here are the relevant parts:

local stepSize = lr/biasCorrection1 --math.sqrt(biasCorrection2)/biasCorrection1

stepSize=math.pow(2,torch.round(math.log(stepSize)/(math.log(2))))

and also

state.v:copy(torch.pow(2,torch.round(torch.log(state.v):div(math.log(2)))))
state.v:add(epsilon)
tmp:addcdiv(1, state.m, state.v)

It seems that for $\alpha \oslash (1-\beta_1^t)$, what is being done is $$2^{\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor}$$

which is a left shift if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor > 0$ and a right shift  if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor < 0$.

On the other hand, for $m_t \oslash v_t^{-1}$, what is being done is

$$\frac{m_t}{2^{\lfloor \log_2 v_t\rfloor}}$$

which is a right shift if $\lfloor \log_2 v_t\rfloor> 0$ and a left shift if $\lfloor \log_2 v_t\rfloor < 0$.


How to train the predicting boxes in a YOLO network? I have just finished this tutorial that explains how YOLO networks work. Instead of training the network's weights with a training set, the author loads pre-trained weights and uses them to test the network. I'm interested in how you would train this network.

I want to run the network on an image that is size 1000 x 1000 and can hold objects with 10 different classes. The network's forward function returns a tensor that has dimensions

[1, num_bounding_boxes * final_layer_grid_width * final_layer_grid_height, 15].

For each cell and bounding box in the final layer, the network predicts 15 values:

x_center, y_center, bbox_width, bbox_height, object_confidence_score, class_1_confidence_score, ..., class_10_confidence_score

I have a list of labels corresponding to the images. Each element in the list (corresponding to an image) looks like this:

[[313,567,47, 23, 4],
[398, 122, 57, 32, 6],
 ...,
[499, 993, 47, 19, 8]]

The first element indicates that the point (313, 567) of the original H x W image is the center of an object, that the corresponding bounding box is of width 47, of height 23, and that the class is 4.

My plan was to check to which grid cell in the final layer each object should be assigned based on its coordinates and the stride of last layer. Suppose that the last layer is 100 x 100 and the input image is 1000 x 1000 then the first element in the example label should be predicted by grid cell [31,56], because the stride is 10. I think that I can create a target Tensor based on this method and I think that I can train the
object confidence score and the 10 different class scores this way.

What I'm not quite sure of is how to train the x_center, y_center, bbox_width, bbox_height. What values should these parameters hold if the grid cell in the final layer should not predict a bounding box?

Question: How do I train the x_center, y_center, bbox_width, bbox_height parameters?

This is a good question and recaps the thinking of how YOLO works.
So first of all you say your forward path gives you a tensor of
[1, num_bounding_boxes * final_layer_grid_width * final_layer_grid_height, 15].

For simplicity let us reshape this to [grid_width, grid_height, num_boxes, x_center, y_center, bbox_width, bbox_height, object_confidence_score, class1, ..., class10], this means for each Gridcell and each Boundingboxprior you have one value for x_center, y_center, bbox_width, bbox_height.
x_center and y_center predicts a value which is sigmoid activated the offset from gridcell border to the actual midpoint of the bounding box. So b_x = sigmoid(x_center)+ c_x, where c_x is x coordinate of grid Cell, for b_y = sigmoid(y_center)+ c_y, accordingly.
bbox_width and bbox_height are values which are shifts of the BBoxprior in log space so BB_width = BB_prior*e^(bbox_width), for BB_height respectively.
You can think about it this way: If an object is in a grid cell, the object_confidence_score will tell you. So you just have to adjust the Boundingbox. First step is to calculate the bounding box with highest IOU, next step is to adjust it's center with x_center, y_center and the height/widht with bbox_width, bbox_height.
Check the Yolo-V3 Paper for the used formulas and especially figure 2 depicts the parameters you asked for.


Lime Explainer: ValueError: training data did not have the following fields I'm attempting to gather ID level drivers from my XGBoost classification model using LIME and I'm running into some odd errors. I'm using this link as a reference.

Here is the overall code that I'm using:

explainer = lime.lime_tabular.LimeTabularExplainer(Xs_train.values, class_names = [1.0, 0.0], kernel_width = 3)

predict_fn_xgb = lambda x: trained_model.predict_proba(x).astype(float)
data_point = Xs_val.values[5]

exp = explainer.explain_instance(data_point, predict_fn_xgb, num_features = 10)
exp.show_in_notebook(show_all = False)

Key:

trained_model: trained xgboost classification model
class names: This is a binary classification model
Xs_train: This is a (73548, 84) dimension training set. This was used to build the training_model
Xs_val: This is a (4910, 84) dimension training set. The columns are the same with the training and validation set.
data_point: one specific validation point

Now, when I run this code, I get the following error:

ValueError: expected res_time, email_views...training data did not have the following fields: f6, f49, f34, f21,...

I don't know where the f# column names are coming from. Seems really bizarre and I believe I'm following the example correctly.

Any help would be much appreciated. Let me know if any additional information is required.

It looks like a dataframe/nparray mismatch.  Probably your model was trained on a dataframe; doing the training instead on an nparray may fix the problem.

I've seen such errors in working with xgboost; see e.g.
https://github.com/dmlc/xgboost/issues/2334 and
https://stackoverflow.com/a/52578211/10495893


Can PCA be applied to reduce dimensionality of only a subset of features? Lets say I have a feature set of f0 to f1000. I am thinking of applying PCA on f500 to f1000 reducing their dimensionality. Can I combine this reduced set with the features f0 to f499 as the feature space for training a learning algorithm?

Yes, absolutely. Simply split your data into two sets feature-wise, apply PCA to one of them, and then stick them back together again. How to actually perform this will vary depending on your programming language/frameworks, but it is trivially easy in python + pandas, for example.


Feature extraction from sequence of images with Siamese Neural Network I am trying to train a neural network to recognize certain actions in short movies.
Each such movie consists of a fixed number of frames, each frame - the image is of course the same size, after preliminary preprocessing.
And now I'd like to do some feature extraction of each of these images using the Siamese Neural Network (SNN). I found articles somewhere that SNN might be great for this, but without implementation details.
These articles show that they take a pair of two vectors together (in my case it will be a picture, but it does not matter if it is 1d or 2d) and then they use this SNN model which return to them how much the two vectors are similar to each other.
To quote the article:

We employ the contrastive loss function [29] while training our
models. We choose this formulation over a standard classification loss
function like cross entropy since our objective is to differentiate
between two audio frames. Let X = (X1, X2) be the pair of inputs X1
and X2, W be the set of parameters to be learnt and Y be the target
binary label (Y = 0 if they match and 1 if otherwise).

The article itself here: https://www.eecs.qmul.ac.uk/~simond/pub/2020/AgrawalDixon-EUSIPCO2020.pdf
Well, in general, I do not really understand how to feed this SNN with data, since I have these 100 movies, each has 35 frames, and each of these 100 movies is assigned y categories ranging from 0-10 (10 frames per action). And the SNN itself operates on a single frame, how do I train this SNN? I have to choose randomly a pair of two frames, regardless of what action they come from and whether they are from the same or not calculate this vector as close if from the same and far if from another?
Will such a model be useful at all? The first thing that came to my mind was not to compare single frames with each other in this SNN, but whole sequences and on this basis calculate the similarity of the entire sequence to another.
How do you think which approach is better and what exactly should be done
in this approach with comparing single frames to each other?
Thank you in advance
EDIT:
To be precise, I have 2 questions. First, which solution could work better, SNN operating on the entire sequence of 35 frames or maybe SNN operating on a single frame.
Second, for a single-frame approach, how exactly is that model supposed to work. Should these single two frames be chosen randomly? For example, frame 5 of 35 in Action X and frame 25 out of 35 in Action Y, will this even work? Or maybe selecting the i-th frame but from random actions, e.g. 5 frame from action X and 5 frame from action Y.

Which is better single frame similarly check or sequence check?

Sequence Check. Because in the Single frame Similarity check (Consider SNN), You need to input pairwise images. So that you need so many pairwise images for your SNN for your Batch Generation.
For the Batch Generation idea is to make usuable batches for training the network. We need to create parallel inputs for the A and B images where the output is the distance. Here we make the naive assumption that if images are in the same group the similarity is 1 otherwise it is 0. If we randomly selected all of the images we would likely end up with most images in different groups.
So no problem if you chose randomly or not.
For the Sequence Check, I picked this paper : Deepfake Video Detection Using Recurrent Neural Networks

Here propose a two-stage analysis composed of a CNN to extract features at the frame level followed by a temporally-aware RNN network to capture temporal inconsistencies between frames. And in this paper, they actually detect deepfake from the videos. They used 600 videos to evaluate the proposed method and get 94% more accuracy.


What is the difference between parameters & cooficients in Machine learning? are both terms interchangeable?
I'm kinda new to machine learning field and very confused about these terms in machine learning perspective.

When talking about coefficients, we usually refer to some fixed numbers or constants, e.g. learning rate coefficient is 0.001 and it will not be changed during training. Parameters are usually changing, e.g. model is training to adjust right parameters. However, there are no strict borders between these two notions.


Computing variance of an SGD iteration It is known that SGD iteration has huge variance.
Given the iteration update:
$$
w^{k+1} := w^k - \underbrace{\alpha \ g_i(w^k)}_{p^k},
$$
where $w$ are model weights and $g_i(w^k)$ is gradient of loss function evaluated for sample $i$. How do I compute variance of each update $p^k$?
I would like to plot it for each iteration and study its behavior during minimization process.

You could plot a graph of update versus iteration and analyze the variation of each update as the number of iteration increases. Like in here, where they are comparing the variance of the standard gradient descent algorithm versus its stochastic version.


Unable to figure out nInputPlane in SpatialConvolution in torch? Documentaion for Spatial Convolution define it as

module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH])

nInputPlane: The number of expected input planes in the image given into forward().

nOutputPlane: The number of output planes the convolution layer will produce.

I don't have any experience with torch but i guess i have used a similar function in keras

Convolution2D(64, 3, 3, border_mode='same', input_shape=(3, 256, 256))

which takes as input the shape of the image that is 256*256 in rgb.

I have read usage of Spatial Convolution in torch as below but unable to figure out what does the nInputPlane and nOutputPlane paramter correspond to?

local convLayer = nn.SpatialConvolutionMM(384, 384, 1, 1, 1, 1, 0, 0)

In the code above what does these 384,384 represent ?

The nInputPlane is the depth or the number of the layers of the input image. In case of RGB images, this should be 3 which corresponds to the first number in the input_shape=(3, 256, 256).

The nOutputPlane is the the number of the layers of the volume that the convolution step will produce which is also the number of filters/kernels applied to the input. By convention, there is an output layer for each filter. This corresponds to the first argument of the Convolution2D function.


Difference between Jordan, Elman and normal RNN As far as I know for history, the Jordan network was proposed first in 1986 as a form of RNN with this diagram:

Actually, this is the solution that makes sense when thinking about sequence data that the current output is an input in the next time (with some weight and activation as shown in figure).
However, after this in 1990, Elman network was proposed with feeding back the hidden state not the output like this?

What is the reason or benefit of this modification?
And what is the difference between these two types of network and the ordinary RNN (before LSTM and GRU) we know as shown in this figure?

The figure of RNN seems to be very similar to both of them (Elman especially) because we take the hidden state and make it an input again.
What is the difference between RNN and Elman and Jordan networks? and what is the usage difference in both of them?
Please note that I am taking about RNN before LSTM and GRU. They are out of comparison.

The architectures in your second and third pictures are identical because the link $I$ from the hidden units to the context units in the Elman's network is not trainable. It is always equal to 1.
In his paper [1], Elman first provides the Jordan's architecture, then introduces his own network but does not discuss why his one is better.
Anyway, this is what I think about why it may be preferred.
The neurons in the hidden layer learn features or internal representations of the inputs. Backward connections allow hidden units to learn features which encode sequential properties of the inputs.
In a deep network, the neurons in the layer that is further from the input learn more abstract features than the neurons in the layer closer to the input. Feeding the output from a layer back to the same layer allows learning sequential properties at the same level of abstraction. Feeding the output from a layer to the input of a different layer mixes the levels of abstractions.
In the Jordan's network, the output of the whole network directly influences the hidden layer's input, so the learned representation is split between the hidden layer and the output layer. Elman's network allows more separation of features by levels of abstraction.
Elman spends most of his paper trying to understand the features his neurons have learned even for those tasks where his network does not perform well. I am not completely sure but I guess that backward connections between different layers would make such analysis more difficult.

Elman, J.L. (1990), Finding Structure in Time. Cognitive Science, 14: 179-211. doi:10.1207/s15516709cog1402_1


Explain to a layperson why text-to-speech (TTS) has made little progress? I am a layperson who has searched for years in vain for a human-sounding TTS program, but it seems that over the years, hardly any progress has been made (a purely unscientific evaluation using just my ear and also the ears of many others). It still sounds like the same robotic word-by-word reading-out that I believe was already accomplished decades ago.

This is puzzling (for a layperson like me who doesn't know anything about data science, linguistics, machine learning, etc.). Especially when contrasted to say speech-to-text or driverless cars, which have noticeably improved over the years.

Could you explain to a layperson (like myself) why, in early 2017, do most text-to-speech (TTS) programs still sound so robotic and non-human-like? And why so little progress has been made (especially in comparison to some other achievements)?

Might it just be that there simply isn't much demand for TTS (as compared to speech-to-text or driverless cars) and so few resources have been devoted to this?

(If this isn't the right StackExchange for the question, I apologize in advance. Please migrate this to a more suitable StackExchange site instead, thanks!)

Text-to-speech suffers from problems similar to the drivers of "uncanny valley" effects when viewing faces of humanoid robots and computer renderings of faces. Our ability to distinguish depth of meaning, emotional content and other subtle cues makes humans very sensitive to small details in audio containing spoken language. Note this is not quite the same as claiming text-to-speech actually has an "uncanny valley"; there doesn't appear to be much analysis of that claim either way around.

Existing text-to-speech systems have two basic generative processes: Concatenative models are essentially databases of phoneme samples that the system strings together. These suffer from lack of flexibility. Parametric models attempt to represent sound generation at a lower level to improve on this, but it is quite a challenge to create a model that includes all the fine detail that we notice. For instance, we notice many non-verbal cues including imperfections such as mouth noise, breathing etc.

There has been recent progress in this area by researchers running a very detailed generative model - one that literally generates sound, sample by sample, after consuming a large amount of training data.

This report on WaveNet by DeepMind team explains the different synthesis techniques used to date and shows off the capabilities of the new approach.

The process creates state-of-the-art results, and it is easy to hear the improvement from examples on the site. However, it is too computationally intensive to be used in real-time systems yet. Give it a few years to be refined, and it could be the basis of far better text-to-speech systems.

The above written in January 2017. In the intervening time (it is now October 2017) the DeepMind team have been working on the efficiency of their model, and it is now much faster and sounds even better. This is close to becoming a solved problem, albeit with some proprietary ownership. Give it a little while longer though and this breakthrough will allow real-time and natural-sounding parametric voice models in many applications.


nltk measure the accuracy of the new features I have been playing around the NLTK algorithm for some data prediction.

Starting from this gib, I started my understanding process. However, there are some bits that don't make sense.

If I have a set of 100 features, all classified, what's the sense to split them, take 10% and build the training set on that alone?
I thought the training set should have included all the list, and accuracy is measured against the new keywords being tested against?

Any hint would be helpful.

If you don’t have a classified testing set - which allows to measure a performance score, then it is useful to use part of your training data as a validation set, meaning that you test the performance of your model on it (because you have the true labels/values).

The percentage of the split is arbitrary and depends on the degrees of freedom you have (number of individuals vs number of variables) - 0.1 to 0.25 is a common choice.


Deciding on the number of components in PCA I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).

If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.

Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45

If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

So what should i do?

Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it?
Can someone help please?

Here is my code

# Search for the best combination of PCA truncation
# and class reg (LogReg).

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])

# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'logreg__penalty':['l2'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
                    'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'logreg__penalty':['l1'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['liblinear', 'saga'],
                    'logreg__multi_class': ['ovr', 'auto'],
                    }]

clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
                      return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)

# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

This is not unexpected. Choosing the right number of components requires balancing the extra information given by the additional dimensions and the useless noise and redundancy present therein. Even though PCA is a linear transform, and even if you used a linear classifier, the dependence of the classifier performance on the number of components is generally very non-linear.

Two options (or alterations thereof) are common:

Pick a predefined level of variance to keep (usually 90% or 95%, at least for me) and just stick with it. Think of it as regularization, rather than as a hyper-parameter.
Treat it as a hyper-parameter and optimize over a validation set for the number of components to keep $k$. This is essentially what you are doing already. It's perfectly reasonable to simply choose the $k$ that gave the best performance (on a held-out set).