Are there real world applications where deep fully connected networks are better suited than ConvNets I would like to give some brief background for my question to avoid answers that explain the difference between fully connected nets and ConvNets.

I completed the first 3 courses in the deep learning specialization from deeplearning.ai (Coursera).  Simultaneously I've worked through the first two home work assignments from Stanford's CS321n course for the Spring of 2017, as well as Michael Nielson's book on deep learning.

I believe I have a reasonable understanding of the architecture of both types of networks, and how to write them in python/tensoflow. I also implemented my own very simple fully connected feedfoward net in C where I was able to get 98% on MNIST.

I've also done some reading about this question in terms of which type of data is better suited for a ConvNet, and the answer I've come up with is similar to:

  If the order of the input features matters to their meaning, then a
  ConvNet will work best, but if you replace the position of one column
  with another and the meaning is still intact, then a ConvNet will not
  work.

This means that spreadsheet data (if I've understood correctly) is not suited for a ConvNet.

Unfortunately I don't know any deep learning practitioners to ask this question: are there fully connected deep neural networks in the wild that are currently in use?  The best example I can think of is financial data.  If that's true, what are some others?

Yes, there are plenty use case where a fully connected network is more appropriate.

ConvNet works because the weights are shared in its kernels. It makes a lot intuitive sense to share the weights in image recognition task, because a flower at the top left of a picture has the same local pattern as the flower at the right bottom of the picture. What we want is to learn of pattern that is local.

However, this logic no longer applies when you consider program such as the famous iris  classification problem. The idea of "local" simply doesn't exist for such data set. each of the features (i.e, petal width, petal length etc) are all "standalone" concept. In those case, it simply doesn't make sense to use ConvNet.

BTW, you can use ConvNet on financial data as well. As financial problem usually involves a time dimension, where the concepts of "local" does exist.


How to user Keras's Embedding Layer properly? I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the following:

I understand that Embedding layers turn word values in a sentence into fixed-dimension-long representation. But I observe two distinct usage of Embedding layers: one on one hand (like this tutorial on Keras Blog) utilizes external pre-trained word2vec vectors via the weights parameter:

from keras.layers import Embedding

embedding_layer = Embedding(len(word_index) + 1,
  EMBEDDING_DIM,
  weights=[embedding_matrix],
  input_length=MAX_SEQUENCE_LENGTH,
  trainable=False)`

while in other cases there is no such an external output but users just leave to the Embedding layer to decide the representation vectors. I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?

Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer? Am I correct guessing that this set to True let the Embedding layer fine-tune the imported word2vec weights to take the actual training examples into consideration?

Further, how to instruct Embedding layer to properly encode "metacharacters"? Setting the mask_zero parameter True it can incorporate padding zeroes but what about UNK (unknown), EOS (End of Sentence)? (By the way, I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input...)

And finally: how could a model predict the translation of a word which is not represented in the training set? Is it tries to approximate it with the "closest" one in the vocabulary?

You've brought up some very good points. Let's walk through all of this:

A word embedding is a mathematical representation of a word.
This is needed since we cant work with text as plain input.

In order to get these word embeddings, there a different ways, methods and settings on how to calculate them. cbow, skip-gram and so on.

There are different pretrained word embeddings out there e.g.

Fasttext 2017, trained on Wikipedia; cudos to Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas
Word2Vec 2013, trained Google News dataset; cudos to Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
Glove 2013, Wikipedia 2014 + Gigaword 5; cudos to Jeffrey Pennington, Richard Socher, Christopher D. Manning

This is just an excerpt of the most well-known ones. As you can see, they used different data sets - thus different word vocabulary and word embeddings respectively.

  I don't understand what is the real difference between these
  approaches regarding the desired outcome? Maybe the internal-only
  solution is not a semantic representation? What is the point of
  applying embedding layer to an external matrix of which the rows
  already have fix length?

Keras is an awesome toolbox and the embedding layer is a very good possibility to get things up and running pretty fast. Convert the text into one-hot/count matrix, use it as the input into the word embedding layer and you are set.

On the other hand if you use pre-trained word vectors then you convert each word into a vector and use that as the input for your neural network. This approach would give you more flexibility when it comes to feature engineering.

As mentioned above, pre-trained word vectors were given mostly general text data sets. You might run into the point where you have some special kind of data (e.g. Tweets) where people write or behave differently. So you might look into training your own embeddings, on your own dataset - at the end of the day it depends on your task/problem and the metrics that you are tuning towards.

  Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer?

As you said correctly, it is to retrain the weights of the embeddings with the data set you use.

  I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input

One of the most important things in NLP is feature engineering. It is the same as you sitting in school and learning a language, what needs to be considered, vocabulary, it's grammar and rules. Things that makes it easier for you as a human being to understand the language. The same is applied here. You can see it as one part of feature engineering, it all sums up to the bigger picture.

  And finally: how could a model predict the translation of a word which is not represented in the training set?

Either you convert the word that couldn't be found to an <UNK> token (unknown word) which then represents its group. However it requires having the <UNK> token in the trained word embeddings. Or you could use fasttext binary file, which calculates word vectors on the fly for unknown words.


Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams? I'm dealing with outlier detection in data streams. I'm looking for a way to summarize my data and obtain important statistics such as means and variance, etc.  I want to know if the cluster features or microclusters are suitable or not.

No.

Because assignment to microclusters is distance-based, and distances do not work in high-dimensional data anymore. Most likely one mucrocluster will become most central by chance and collect all the samples.


Classifying objects based of a varying number of the same type of feature vector for each object For a congressional session, I have created a doc2vec model of speeches made. Using the vectors from this model, I have a dataset of each congressperson, their political affiliation, and a list of the vector representations of each speech they made. Each of these document vector representations is a 300 element vector.

I am now trying to classify each congressperson by party using these document vectors representing their speeches. So far I have tried using the mean vector for each speaker, but I was looking for ways to use the whole set of vectors.

If I did not get you right please comment me. I would not go for doc2vec as you do not want to discriminate docs but persons. So better to concatenate speeches of each person to a single document and then feed it to word2vec/doc2vec (I assume you would like to use ANNs otherwise there are other options e.g. TF-IDF, etc.) In this manner each person will have a 300-d feature vector including a everything. (I would still try TF_IDF and CountVectorizer as well!)

If you insist on your current option, you may discard person info (you want to get a speech and say what kind of political-party the speaker is coming from, right?!) and set up your data as a 300-d feature vector for each speech whose tag is a political view (instead of taking mean of all speeches of 1 person). Then your feature vectors are all the same size (in this approach, each person might have several entry in the data which may bias the data so you may add a column including the speaker e.g. "speaker a", "speaker b", etc. Here you end up with a 301-d data).

Hope it helped. Good luck!


How to approach TF-IDf based analysis? Problem statement :

We have documents with list of words in them.
Overall these documents are classified into 2 group (say, good quality vs bad)

docs -

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]

doc group -

good_grp = [doc2, doc1]
bad_grp = [doc3, doc4]

Now we have to find out which words actually are important to make the document good vs bad ?

Idea 1:
Merge all words from documents that belong to document group 1 into single document say (good quality doc) and other one being (bad quality doc) and calculate tf-idf score per doc; but in this case we lose information of document level words and now just see document group level word importance.

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]

good_grp = [w1,w2,w3,w4,w4,w3,w3,w4]
bad_grp = [w2,w4,w8,w1,w5,w4,w0,w9]

Can someone help me to direct to a better approach tf-idf or any other technique to solve this problem?

I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it.

Best classification I am anticipating for this problem would be naive bayes..


Which train test split performs better: 50:50 or 60:40? I have 10,000 customer data of a supermarket. And I want to split the data into training set and testing set. So, which train test split gives me a better accuracy: 50:50 or 60:40?

It sounds like you have a lot of data, so probably a simple train-test split is enough. No need for cross validation.

I would just use something like 75-25. That is in fact the default value in sklearn.

I would use less data in the training only if your algorithm is too slow and cannot cope with the extra data. In that case, instead of throwing away the data, you might as well use it for validation or for testing hyperparameters.

All this being said, more important than all is how you split the data. You should make sure that customers in the testing data are not in the training data to make sure your algorithm is generalizing, and not merely memorizing customers. This is standard procedure in medical data mining, and it is very important. Make sure you do not have customer overlap.

You might also want to make sure that the distribution of the variable you want to predict is similar in the training and the testing data.


Deep Learning for Recommender System I read about Recursive Neural Networks that they can convert Documents to distributed word representation.

In the context of new article recommendation, I am thinking to use this model to convert all news articles to vectors and then recommend to a  particular user, news articles similar to ones he browsed.

In vector space this will boil down to finding 'similar' vectors to a given vector(user's news read).

How likely is that this model will work well in practice? Any comments and/or suggestions?

There are many ways to calculate similarity between articles. I have not seen anybody doing a vector conversion and running comparisons. However, there is a text mining strategy called "Term-Frequency / Inverse-Document-Frequency" which is a clever way to find unique words and phrases in documents. You can run this on multiple documents, and compare the extracted keywords to match them for recommendations.

Check out my ebook for more details: https://lizrush.gitbooks.io/algorithms-for-webdevs-ebook/content/chapters/tf-idf.html

If you want to leverage this technique on web documents (like a blog), there is a free service to do so: https://algorithmia.com/recommends


CV (Curriculum Vitae) Recommender System using Machine Learning, Python, Apache Solr(Back-end), AngularJS (Front-end) I am very new to Machine Learning. This is my college project. I want to develop web application of CV RECOMMENDER SYSTEM in Python. I have lots of CVs in the format of .txt.

My questions are following:

What type of learning I can apply? (Supervised/Unsupervised)
I have different file in one folder. So Is there need to make a corpus? If there is a need to make a corpus then can I create a corpus?

I am very confused with the flow of the application.

Can please anyone suggest me the steps how can I start to develop web application?

Your help will be appreciated.

I guess, there are many ways, to recommend CVs, but here is what I would do:

I would use TF-IDF or LSI, so an unsupervised learning approach and I would use a corpus.

I would apply word stemming, create a dictionary containing unique word stems of the CVs, create a corpus containing word stem id from dictionary and word stem count for each word stem in each CV, so the document frequencies (DF) for all word stems, then create TF-IDF (term frequency inverse document frequency) model or an LSI (latent semantic indexing) model from dictionary and corpus. Then you have TF-IDF or LSI vectors for all CVs.

For matching and recommending, you apply word stemming to text input you want to match CVs with, calculate the LSI or TF-IDF vector for this input and match it with the most similar CV by using cosine similarity calculation.

Here and here you can find short Python code examples for the approach I described using LSI or TF-IDF.


What are the merges and vocab files used for in BERT-based models? The title says it all. I see plenty online about how to initialize RoBERTa with a merges and vocab file, but what is the point of these files? What exactly are they used for?

The vocab file contains a mapping from vocabulary strings and indices used for embedding lookup in the model.
The merges say how to split the input string into subword units. The algorithm is as follows: At the beginning of merging, a word split into characters and then you greedily search for neighboring symbols that can be merged (i.e., are in a list of allowed merges), you stop when there is nothing to merge and this is the subword segmentation.


CASE WHEN SQL Type Statement in pandas I'm trying to create a new column called Class which is a calculated column.
Basically when the Math class is NA then the class will be the CS class, otherwise it will be vice versa.

How can I do this in python? Table name is Classes

MathClass |  CSClass |  Class
------------------------------
Trig      |  NA      |  Trig
NA        |  Java    |  Java
NA        |  Python  |  Python

Replace the missing value with empty string and add these columns up

Classes.fillna(value='').sum(axis=1)


sales price prediction I have to find make a classifier for price prediction of a item. The question I have is which columns I should choose for price prediction.

Also which machine learning classifier would be good to perform this, at present I choose random forest.

Do I need to use time series concept in here?, I think No

We should not use time series concept here. You have to do a pair plot analysis to find your best predictor. I do not think index column is much useful here. I feel length, width or breadth are features if you classify the vehicles as sedan, suv or small car. Not sure if it be of much help in predicting the price. If you add more breadth to the dataset, make sure you have enough depth or training examples

Do a heat map and find the co-relation between the feature and target variable as well. You can remove those features which does not have good co-relation with the target variable.

Try to start with Linear regression model, since you have to predict the price. then move on to more complex models. Random forest regressor is also a good choice for smaller datasets


I am trying to implement Isolation forest for anomaly detection but I am not able to understand and visualise the decision function I am trying to implement Isolation forest for anomaly detection but I am not able to understand the scores. Also, I want to plot and visualise the decision function that my Isolation forest is creating.

Below is the code that I have written :

def IForest(df):
    clf = IsolationForest(n_estimators = 300, contamination = 0.01, behaviour = "new")
    clf.fit(df)
    anomaly_score = clf.decision_function(df)
    anomalies = clf.predict(df)
    print("\n Decision function for Isolation forest")
    print(anomaly_score)
    print("\n The label -1 suggests that the point is anomolous and 1 suggests that the point is normal")
    print(anomalies)
    anomolus_dp = data[np.where(outlier == -1, True, False)]
    print("\nThe data has {} anomolus points and the data points are the following for  : ".format(len(anomolus_dp)))
    print(anomolus_dp)

The output that I am getting is  the following :

On what basis is  this score -0.01010126 is classified ? It is not the probablity of a data point to be considered an anomaly and why is -1(suggesting anomaly) assigned to it, then how can I get the probablity of a a data point to be considered anomlous ?
Also, How do I understand which variable(s) is contributing for this data point to be classified as anomaly ?

According to this answer the range of output from scikit-learn IsolationForest decision_function is between -0.5 and 0.5, where smaller values mean more anomalous.
The predict function then applies a threshold to this function to get either -1 (anomaly) or 1 (not anomaly). The decision threshold is stored as model.threshold_ and defaults to 0.0.
So if you want to mark the decision threshold in a decision boundary plot, then you could use that value unless you have chosen another one. Note that it is quite common in Anomaly Detection to select your own threshold. Ideally it is selected by using a labeled validation set, or otherwise based on the histogram of anomaly scores.
As for interpreting the values that contribute to the anomaly scores, I would recommend using Shap for IsolationForest.


CNN model contains several images that are null I'm using a deep CNN with the ReLU activation function. When visualizing the layers (each with 32 filters), several of the filtered images are zeros.
I am trying to reason why this may be happening? What would cause the entire image to be zero? I can understand having a sparse image (given the ReLU), but what would cause all zeros?
I hope this is clear and has sufficient detail to answer appropriately.

In principle, it is to be expected that several feature maps result in zero output all over the image. What the convolutional layers are doing, essentially, is to move a stencil across the image and give a high value where this stencil shape is found in the image, and zero everywhere else. Each of the feature maps, i.e. each of the channels in the convolutional layers, corresponds to a stencil, the lower layers encode simple shapes, the higher layers more complex ones.

If a channel outputs zero all over, it simply means that this feature was not detected in the image.

As an example, consider an MNIST classifier. One feature map may have learned to look for straight horizontal lines. When the model is presented with a hand-written "7", the straight horizontal lines will be detected in the top part of the image and maybe also in the middle. So the feature map will give high values in the top part and maybe in the middle as well, while it will be zero everywhere else.

If the model is presented with a hand-written "0", there is usually no horizontal lines and therefore the feature map will be zero everywhere.

So if a feature map is zero everywhere, it usually means that the particular feature is not found in the current input, but it might be present in other images.

If, however, a feature map is all zeros for all input images, i.e. the channel in the convolutional layer is zero everywhere for all 60k MNIST examples, then there might something wrong with the network and you might want to apply batch normalization or dropout.


Azure automl time series forecasting error I'm using Microsoft Azure automl to try and generate models for time series forecasting but I keep getting an error:
Error: Could not determine the data set time frequency. All series in the data set have one row and no freq parameter was provided. Please provide the freq (forecast frequency) parameter or review the time_series_id_column_names setting to decrease the number of time series.

My data set looks like:
          Date,       Temp,
1,2019-05-07 13:51:00,25.19,
2,2019-05-07 13:51:58,25.14,
3,2019-05-07 13:53:00,25.14,
4,2019-05-07 13:54:00,25.14,
5,2019-05-07 13:55:00,25.1,
6,2019-05-07 13:55:58,25.1,
7,2019-05-07 13:57:00,25.1,
8,2019-05-07 13:57:58,25.11,
9,2019-05-07 13:58:58,25.11,
10,2019-05-07 14:00:00,25.07,
11,2019-05-07 14:01:00,25.07,
12,2019-05-07 14:02:00,23.11,

I followed the steps on the ml.azure.com gui, I don't see a place for freq or time_series_id_column_names
Seems to be an issue with the data set, any ideas on how I can prevent/fix this error would be great, thanks!

The data either has duplicates or you may have some rows with empty values. I removed my rows at the bottom that had empty values separated by comma which resolved the issue


Keras difference beetween val_loss and loss during training What is the difference between val_loss and loss during training in Keras?

E.g.

Epoch 1/20
1000/1000 [==============================] - 1s - loss: 0.1760, val_loss: 0.2032

On some sites I read that on validation, dropout was not working.

val_loss is the value of cost function for your cross-validation data and loss is the value of cost function for your training data. On validation data, neurons using drop out do not drop random neurons. The reason is that during training we use drop out in order to add some noise for avoiding over-fitting. During calculating cross-validation, we are in the recall phase and not in the training phase. We use all the capabilities of the network.

Thanks to one of our dear friends, I quote and explain the contents from here which are useful.

  validation_split: Float between 0 and 1. The fraction of the training
  data to be used as validation data. The model will set apart this
  fraction of the training data, will not train on it, and will evaluate
  the loss and any model metrics on this data at the end of each epoch.
  The validation data is selected from the last samples in the x and
  y data provided, before shuffling.

  validation_data: tuple (x_val, y_val) or tuple (x_val, y_val,
  val_sample_weights) on which to evaluate the loss and any model
  metrics at the end of each epoch. The model will not be trained on
  this data. This will override  validation_split.

As you can see

fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0, steps_per_epoch=None, validation_steps=None)

fit method used in Keras has a parameter named validation_split, which specifies the percentage of data used for evaluating the model which is created after each epoch. After evaluating the model using this amount of data, that will be reported by val_loss if you've set verbose to 1; moreover, as the documentation clearly specifies, you can use either validation_data or validation_split. Cross-validation data is used to investigate whether your model over-fits the data or does not. This is what we can understand whether our model has generalization capability or not.


Logistic Regression outperforms MLPClassifier I am new in ML and I am trying to train classifier. I have a tiny dataset, just 90 examples, I divided it 70/30 train/test set and started to train.

As I know MLP must outperform Logistic Regression, but I have an opposite situation in my case, firstly I thought that I give bad hyperparameters, so I tried to use GridSearchCV for finding best hyperparameters, but after it, again Logistic Regression was better, I am wondering maybe is it because of the small number of dataset or I missed something?

In ML, all algorithms are useful depending on the dataset. Its naive to generalize an algorithm to be always better than the other.

In your case since you have only 90 examples, mlpclassifier couldn’t have trained proper as compared to logreg.

A general suggestion: if you post like rows of your dataset along with accuracy of the two algorithms, it would have been a bit easier to discuss why there were differences.


How to train on extended data set correctly I have trained my classifier on pictures with a mixture of several classes
on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes
on the images.

Now I got more data with pictures showing class G. To minimize my work, I only labeled
class G on the images and left the rest out (Invalid).

Two questions for my training arise:

If there are no examples of class G in my first dataset (because it could have been a subclass
of A-F), how do I train it correctly?
Suggested Solution: I can add my new data to the old one, but I have to tell the CNN when there is an image with class G, because I have to change the error measurement of my CNN somehow. But how?
If G was included in my first dataset (assume G is 'police car' and B is 'car', but
some images showed police car), how do I train in this case correctly?

Transfer learning is what you want to need.

Checkout this note from CS231n, it provides some general advices on transfer learning and model fine-tuning.

In the meanwhile, this blog from keras shows how to use pre-trained VGG16 network to classify dogs&cats.


imbalanced target dataset(multi class) I have a multi-class prediction problem
but the 300classes is imbalanced
should I make it balance all 300 class will predict the better result?
is there an easier method to do this job?
if I'm using the random-forest imbalance dataset is matter?

Nothing better if you could get more data and make classes (at least close to) balanced!

Choice of algorithm (I believe you using only RFC) entirely depends upon problem statement and as we all know, there is no free lunch in statistics, so you might've to try other algorithms (or just create a pipeline trying few more) as well.

Try over/under sampling and penalize your model by applying some custom matrix for miss classification, if required. Another point to keep note of is performance metric (avoid Accuracy paradox). Apart from deeper dive with F1, Recall & Precision; also try to look into [Kappa]1 or [ROC curves]2.

Based on the limited information (better idea would be to add a graph showing class imbalance) you've provided, this is the best I recommend. Hope it helps!


Random forest vs majority voting I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is split equally among base classifiers for majority voting. Nemenyi test shows, that majority voting is significantly better (for 11 benchmarking datasets from keel).

From what I understand, the difference between those two methods is that data used to train random forest (base classifiers) might not sum up to the whole dataset. Is my understanding correct? If so, what might be the reason for the observed difference?

Also, could you point me to any articles comparing those two methods?

Edit: If someone was interested in this topic, here's an article comparing bagging with horizontal partitioning in favor of the latter.

Random forest, predicts the class with highest probability estimate. The predicted class probabilities of an input sample is computed as the mean predicted class probabilities of the trees in the forest. The class probability of a single tree is the fraction of samples of the same class in a leaf.

Majority voting, which is also called Hard Voting, every individual classifier votes for a class, and the majority wins. In statistical terms, the predicted target label of the ensemble is the mode of the distribution of individually predicted labels.

Majority voting may works better in cases where there are some outliers. Consider these votes: $\{0.51, 0.51, 0.51, 0.01\}$ and $\{1,1,1,0\}$.


Using GPUs of other machines in the network with Keras My situation is as follows:

I have a rather cheap laptop with Ubuntu 18.04 running on it that unfortunately is not powerful enough (old, cheap GPU) to train deep learning models with. I am located in city A.
My father (located in city B) has two computers (with powerful GPUs) with Windows 10 Professional running on both of them that I could theoretically use at nighttime. I can connect to the network via VPN and have SSH root access to both machines.

I am now looking for a Keras setup that allows me to train neural networks from my local machine making use of the remote GPUs. Ideally, I would have a Jupyter notebook with all the logic running on my conputer and could somehow connect to a thin client (probably exposing some interface to CNTK) running on the remote machines. I'd prefer to keep the Windows machines as dumb as possible and would ideally only make use of their GPUs (with some local caching of course). Also, I would prefer to stick with Windows on the remote machines not having to set up some dual-boot with a Linux distro. Enabling the Linux subsystem would be fine though.

Any thoughts on how to tackle my problem?

I'm not sure how exactly you're imagining it but the way you're describing it is not possible. You can't perform some computation locally and some remotely and even if you could set it up it would be inefficient due to latency.

A lot of people (including myself) have similar situations and the way we deal with it is executing everything remotely. I.e. you have your data, your code and your notebooks all in the remote machine, from where you run them. However, you access these from your own PC. E.g. you might have a notebook running on your own browser, that actually executes its commands on the remote machine (think of it like google colab or kaggle notebooks).

How to set this up

First of all, creating a dual-boot linux isn't necessary. You should do it only if you feel that you're a lot more familiar Linux than Windows. What you need to do is:

Install all necessary NVIDIA tools so that you can run keras on GPU. These include installing (a) the proper graphics card drivers, (b) CUDA and (c) cuDNN
Setting up your python ecosystem, which needs to have all necessary libraries (jupyter, tenrosflow, keras, etc.). I'd suggest a virtual environment. At this point you should make sure keras runs properly on the GPU on your remote machine.
Setup an ssh connection that forwards everything sent to port 8888 of your remote machine to a port of your choosing (e.g. 9999) in your own machine:
ssh -N -f -L localhost:8888:localhost:9999 username@remote_hostname.
Connect via ssh to your remote machine and run jupyter notebook (I prefer running it on the background or in a screen so that I don't have to keep the terminal always open). Take note of the token that it outputs to the screen as it won't log you in automatically.
Open your browser and type localhost:9999. You need to add the token here for authentication. Now you're using a notebook that's actually running on your remote machine!

Additionally, you can set up other tools to make your life easier. For instance, I have a FTP client for transferring data easier and have set up PyCharm to use a remote interpreter (i.e. execute code I write on the remote machine).


Define difference between feature selection and feature reduction What is the difference between feature selection and feature reduction?

When do we use feature selection and what happens when we don't use it? How is this different than feature reduction?

Feature selection refers to deciding on what features to include in model training.

Feature reduction refers to assigning weights to features regarding how important they are for training. In linear regression, this is known as Ridge Regression (shrinking features based on L2 norm), Elastic Net, and Lasso (shrinking based on L1 norm). Lasso can shrink features to zero, so that they are effectively excluded from training (this would be similar to selection iff features are shrunken to zero).

Ridge and Lasso are very well described in the book „Introduction to Statistical Learning“. If you want to know more about Ridge/Lasso, have a look there.


How does deep learning helps in detecting multiple objects in single image? Let's say there are two cars in an image. How can it detect these cars, given that it can detect single car in an image?

Although many solutions in production systems still use a sliding window as described below in this answer, the field of computer vision is moving quickly. Recent advances in this field include R-CNN and YOLO.

Detecting object matches in an image, when you already have an object classifier trained, is usually a matter of brute-force scanning through image patches.

Start with the largest expected patch size. E.g. if your image is 1024 x 768, but always a distance shot of a road maybe you do not expect any car to take up more than 80 x 80 pixels in the image. So you take an 80x80 block of pixels from one corner of the image, and ask your classifier what chance there is a car in that corner. Then take the next patch - perhaps move by 20 pixels.

Repeat for all possible positions, and decide which patches are most likely to contain cars.

Next, take a block size down (maybe 60 x 60, moving 15 pixels at a times) and repeat the same exercise again. Repeat this until you have hit the expected smallest block size for your goal.

Eventually you will have a list of areas within the image, with the probability that each contains a car.

Overlapped blocks both with high probability are most likely the same car, so the logic needs to have thresholds for merging blocks - usually taking the overlapped area with the highest probability score - and declaring there is only one car in that area.

As usual with ML approaches, you will need to experiment with correct meta-params - in this case block sizes, step sizes and rules for merging/splitting areas - in order to get the most accurate results.


TLDR Bot - Sentence Tagging w/ BERT Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I can improve the results using LSTM.

I'm now batching 5 sentences together, calculating their BERT encodings, and then using 2 LSTM Layers, one backwards one forwards, to predict if the sentence is important.

Unfortunately I'm now calculating 5 times the number of embeddings, and if it doesn't work, I can't seem to figure out how to feed a variable number of things into BERT using Tensorflow, to see if I can tweak some results.

Are there other methods to add surrounding sentences to this context?

You're trying to do Extractive Text Summarization.

You're first approach is the right one, using BERT + CLassifier.

From NLP Progress, the best method so far is Presumm.


Should I set higher dropout prob if there are plenty of data? I have some excessive amount of data for the size of NN I am able to teach in a reasonable time.

If I feed all the data into the network it stops learning at some point and a resulting model shows all signs of being overfit.

Intuitively if I increase dropout prob the model should learn less aggressively from data and gain from more data being fed into it.

Is my logic sound?

You are right, increasing the dropout proportion will help.

However, this looks like a setting where early stopping will be a very good choice. Although dropout, weight decay and batch-norm can work propperly, the fact that you easily overfit your training set would make it an appropiate scenario to try early stopping.

In addition, as the neural network takes very short to be trained you can train many of them (on some subset of the training set, making them weak learners) and create an ensemble to make the final predictions.


Training error is oscillating, cross validation error is continuously decreasing I created a two layered fully connected neural network as a part of a recommendation engine (after I use embedding layers for products and users). I have been trying to tune the hyper-parameters for the past couple of weeks now. Additionally I am using 5 fold cross validation and 'mse' as the error metric. Below is the curve for training and cross-validation error that I obtain post training on a GPU for 10 epochs.

I am not sure what really to make of this, fundamentally. The model doesn't overfit or underfit the data. What is even more perplexing is that the validation error is across five folds. Is this an inherent bias in the data and alludes to the fact that the data is structured such that a weak model does well on unseen data ?

Any pointers or comments will be greatly helpful. Thanks !

By definition, the training error should always decrease with a proper learning rate until it reaches his plateau, then starts to oscillate because it's crossing forward and backward the minimum of the loss function.

Instead, what you expect from the validation error is that it decrease with the training error, and at some point it starts to increase, sign of over fitting.

And it is not unusual that at the begin of the training you found the validation error lower than the training error.

From your plot I would say:

Train and validation errors are simply swapped: the red line is actually the training error, and the blue one is the validation error;
Stop training just after the crossing point. This because the validation set is just a set where you tune your parameters, and not the set of the "Revealed Truth", so it is also prone to over fitting.

If you make a third dataset ("test" dataset), and you try to check your metric (accuracy?) on it, I bet that the model trained after two epochs performs better than the model trained with ten epochs.


Why PyTorch is faster than sklearn models? Recently, I get to know about the hummingbird library for Python. I trained a RandomForest on a 10M-sized dataset with 2 labels. With sklearn it was taking 450 ms for inference. But after converting the same model to PyTorch, now it takes 128ms on CPU inference.
If both are running on the CPU, then why hummingbird's Pytorch model is faster than sklean model?
I am not getting what hummingbird does to my sklearn model to increases speed.

It is difficult to answer your question without the access to your code. The best way to understand the difference is to profile the code and see where the bottlenecks are for your specific problem.
For this, you can use different profiling modules in python:

cProfile
python line profiler


Why is my validation loss going up while my validation accuracy also goes up? Scenario:
I've been training a CNN for the cifar10 dataset.
I'm using tensorflow, and a CNN with 12 conv layers and 1 dense layer before a softmax dense layer.
I'm using data augmentation as well with batch normalization.

After a few hundred epochs I archieved a maximum of 92.73 percent accuracy on the validation set.

My problem:

Validation loss goes up slightly as I train more.
While validation loss goes up, validation accuracy also goes up.

Example:

One epoch gave me a loss of 0.295, with a validation accuracy of
90.5%. My best epoch for validation accuracy gave me 92.73% with a validation loss of 0.33.

Question:

Why is my validation accuracy increasing while my validation loss is
going up?
Should I use a loss metric diferent to cross_entropy?

It is possible that accuracy and cross entropy increase at the same. For example, for a positive sample your predicted probability could go from 0.4 to 0.1 (still wrong but worse increasing the entropy loss) and for another positive sample your predicted probability could change from 0.49 to 0.51 (changes from wrong to right improving accuracy). The first case would increase your entropy loss while the second would improve accuracy without significantly changing the entropy loss.

It's a little difficult to say if cross entropy is not a good metric for your case without knowing any details. But most likely you would want to stick with it for a couple of reasons. For example, the cross entropy gives better probability estimates and has nice properties for training with gradient descent (smooth gradients, training doesn't stall for large values because the log off-sets the exp in the sigmoid activation etc.).


Feature extraction - wavelet transformation + autoregression I am working on a feature extraction problem (ECG signal). Within my literature review I stumbled across the following text:

"The wavelet transform is used to extract the coefficients of the transform as
the features of each ECG segment. Simultaneously, autoregressive
modelling(AR) is also applied to obtain the temporal structures
of ECG waveforms." here

My question now is, how do the wavelet transformation and the AR differ, i.e. how does the AR 'obtain the temporal structure'?

The SVM is used after the wavelet transform. The wavelet transform is a way to detect correlations between the parts/segments of the signal. Once you have these 'features' you can train more or less any classifier on that data.


What does it mean the term variation for an image dataset? I am working with convolutional neural networks, and I have seen that often we need to pre process the images before feeding them to the network. In particular, I have seen that often we have to do image augmentation using an image generator. Now, when looking for a clarification on why we need to do this, I came across an article which says:

  Image Augmentations techniques are methods of artificially increasing the variations of images in our data-set by using horizontal/vertical flips, rotations, variations in brightness of images, horizontal/vertical shifts etc.

What I don't understand is what is the variation of a dataset.

Can somebody help me? Thanks in advance.

Assume, you have two different datasets of cat images:

Dataset 1

Images were taken under full daylight.
The cat covers always almost the whole image.
The cat is always in the center of the image.

Dataset 2

The images were taken under various lighting conditions: some under full daylight, some in rooms, some during the night.
The cats were differently far away from the camera, e.g. sometimes the cat covers only 20% of the image.
The cats are in different positions in the images, e.g. sometimes, the cat is in the center but other times, it is in the upper right corner.

In this case, we would say that Dataset 2 has a higher variation than Dataset 1.

Generally, we use data augmentation to simulate missing information in datasets. For example, you could decrease the brightness of some images in Dataset 1 to simulate how a cat looks at night. Or you could zoom out to simulate how a cat looks from farther away.
Note that a perfect dataset that contained all relevant information would not require any data augmentation at all. However, gathering data is expensive and data augmentation is cheap. Therefore, we almost always have to augment.


Optimization function returns the same optimal parameters for two labels I've recently enrolled in the Coursera machine learning, and am working my way through making my own classifier for the Iris dataset problem using matlab. I'm training a classifier for each species (the one-vs-all method). The code runs smoothly without any errors, but the results are not quite what I was expecting. Here's my matlab code:

clear, clc

% loading the data
data = csvread('data.csv');

% extracting the features
X = data(:, (1:end-3));
X = [ones(size(X, 1), 1) X];

% computing the number of training examples (m) and features (n)
[m, n] = size(X);

% extracting the labels
y_setosa = data(:, end-2);
y_versicolor = data(:, end-1);
y_virginica = data(:, end);

% initializing the options for optimization algorithm
options = optimset('GradObj', 'on', 'MaxIter', 10);

% creating the initial theta vector
iniTheta = zeros(n, 1);

% tarining a classifier for each label
[theta_setosa, min_setosa] = fminunc(@(theta) costFunction(theta, m, X, y_setosa), iniTheta, options);
[theta_versicolor, min_versicolor] = fminunc(@(theta) costFunction(theta, m, X, y_versicolor), iniTheta, options);
[theta_virginica, min_virginica] = fminunc(@(theta) costFunction(theta, m, X, y_virginica), iniTheta, options);

Here's the code for my costFunction:

function [jValue, gradient] = costFunction(theta, m, X, Y)
    V = X*theta;
    jValue = (-1/m)*sum(Y.*log(sigmoid(V)) + (1-Y).*log(1-sigmoid(V)));
    gradient = (-1/m)*X'*(sigmoid(V)-Y);
end

And finally my sigmoid function:

function sigX = sigmoid(X)
    sigX = arrayfun(@(n) 1/(1+exp(-n)), X);
end

The resulting theta_setosa and theta_versicolor vectors are all zeros, and the costFunction has the same minumum for the both of them. the virginica classifier seems to be working fine however. Although for each fminunc I get the following message displayed

Local minimum possible.
fminunc stopped because it cannot decrease the objective function
along the current search direction.
<stopping criteria details>

The error stemmed apparently from the formula for the gradient: there should be no minus sign (-) at the start of it. The correct code for this would be:

function [jValue, gradient] = costFunction(theta, m, X, Y)
    V = X*theta;
    jValue = (-1/m)*sum(Y.*log(sigmoid(V)) + (1-Y).*log(1-sigmoid(V)));
    gradient = (1/m)*X'*(sigmoid(V)-Y);
end


Retraining of object detection models I came across the concept of retraining of models a few days back.
So tensorflow recommends to retrain an already trained model instead of making a new one from scratch so that the whole process takes less time. But I can't understand how this is going to work.
Like if I take a pretrained model (trained to classify cats and dogs), how can I use its parameters to train a whole new model that I want to make (to classify different flowers).

It will not
It's about using a model which was trained on thousands of Classes on Millions of images of ImageNet.
 Chances are very high that most of the classes you have in your dataset is already there.
In general, if you trained a model on a super-class (e.g. vehicle), then you may reuse it to classify the Car variant(Utilizing its initial layers).
Point is to reuse the characteristics already learned by another model. It can reduce training time drastically.


grid search result max_features = 'sqrt' in random forest - how to understand I did a grid search at random forest params. the result of
print(randomforestreg.best_params_)

The result is =
{'max_depth': 28, 'n_estimators': 500 ',max_features': 'sqrt', 'min_samples_split': 2,  'min_samples_leaf': 1'}
The Random Forest documentation:
If “auto”, then max_features=sqrt(n_features).
If “sqrt”, then max_features=sqrt(n_features) (same as “auto”).
So 'sqrt' is like max_features=sqrt(n_features) --> same as 'auto'.
--> the number of features with which tree is build. I dont understand this. When I have 19 columns(features) and 100.000 rows, what is now the answer of the question; "what is the best value for max_features?". Is it the squared value of n features = 19^2 = 361?

The way to understand Max features is "Number of features allowed to make the best split while building the tree". The reason to use this hyperparameter is, if you allow all the features for each split you are going to end up exactly the same trees in the entire random forest which might not be useful. To overcome this we let the model select a fixed number of features randomly, in this case, the no of features allowed = Square root of total no of features in your dataset.
Hope this clears up !!


How to normalise estimate weighted on correlations (some will obviously be negative) I would like to work out an estimate for some variable Xi using values of a bunch of other variables Xj, j in Y (I guess this should include Xi too) and how correlated they are to Xi.

So if I do weight(j)=correlation(i,j)/sum over Y of other correlations, then this sum can be zero but I feel that might not be the only problem. I could use the min-max remapping but then the most negatively correlated variables will have no or little effect, whereas in fact they should have a strong negative effect.

One obvious fix would be to ABS() the sum but I could not prove to myself that this is the correct thing to do. It may be possible that each correlation needs to ABSed in the sum (then it would resemble the way stdev is calculated - then do I need to square my correlations?).

I simplified the problem I am trying to solve for this question but I am not sure if the details would affect the solution. There is a time dimension to it, and I am trying to use changes over each variable over time and their correlations to work out the estimate for the next change or a chain of changes I then multiply with the current value. (I don't know if this step is necessary or if proper normalisation of correlations would correct for this even if I used absolute values and not relative changes).

The appropriate method, if you have the data to use it, is linear regression.  Simply using correlations fails to take into account that the other variables are also correlated with each other; for example, any averaging method like the one you describe will overweight any duplicated variables.


How to Classify Documents whether they are similar to previous documents or not? I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those 100 documents or not.Which is the best model for this problem?

Only way to check which is best model is to try different in practise and compare. Do you have any documents to train with that are not similar? Do you have labels or topics assigned to these 100 ?

Assuming you have some irrelevant documents you can use a "spam" approach to this type of problem.

I would begin with Bag of Words representing of your documents for a Naive Bayes model generation (common, good performing, baseline here) - this link http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification is a good explanation of this for documents .

I would also try using a library like Python gensim that will allow you to try its various methods, and you could try that library's Doc2Vec method. Then train a linear discriminant classifier like Log. Regression with the document vectors it gives you, and try see if its resulting accuracy is better than the Naive Bayes method.


Is there a person class in ImageNet? Are there any classes related to humans? If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew as a daddy longlegs, a kind of spider :-). How is that possible? I would have at least expected a person class, and even something more specific such as man, woman, toddler, etc. Nothing of the sort. Why? Did Fei-Fei Li and her team make a conscious choice not to have people images in the database? Am I looking at the wrong file? For the sake of the question, we can consider the ImageNet versions from 2014 onwards.

You can also take a look at here for the labels in the imagenet. I guess you are right, there is no label for human in the data-set but there is something to notice. There are labels in imagenet like cowboy or some specific hats and other related things to human like shirt and t-shirt. You can take a look at here and also here. In the latter link Yosinski et al, has tried to show that the popular AlexNet has learned to recognize human faces although there is no label as human face in the imagenet data-set. In their paper, they have investigated that Convolutional neural networks may try to learn things that are distributed among layers or maybe not and they may not have special label in the training data. As an example, the face of cats and humans can be referred to. Moreover, as you can see here maybe the aim was attribute learning in large-scale datasets, as quoted in the last line of the page, as the reference.


Information extraction with reinforcement learning, feasible? I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the Google DeepMind & AlphaGo's stuff) to parse and extract information from text.

For example, could it be a competitive approach to structured prediction such as

Named Entity Recognition (NER), i.e. the task of labelling New York by "city", and New York Times by "organization"
Part-of-speech tagging (POS), i.e. classifying words as determinant, noun, etc.
information extraction, i.e. finding and labelling some target information in texts, for instance 12/03 is date given the context meaning 3 December and has the label "expiry date"
What would be a relevant modelling to do these tasks?

Rather naively I would think of a pointer that read the text from start to end and annotate each 'letter' by a label. Maybe it would learn that neighbouring letters in a 'word' share the same label, etc. Would it be able to learn long-term dependencies with this approach?

I am interested by any ideas or references related to this subject.

You ideally want to use Reinforcement Learning in situations where there is delayed feedback and stochastic transitions in the environment. Although you could potentially apply RL, in your case, you might be better off with a Sequence to Sequence learning framework (https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf) since you have access to the entire sentence and there is no stochasticity involved.

On the topic of RL with Information Extraction, this might be of interest:
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning (http://arxiv.org/abs/1603.07954)


Correct sequence of data prep steps? Say you have a binary classification problem, and a dataset with 20,000 observations and 20 columns. The target variable is very imbalanced, there are are missing values, skewed distributions, outliers, etc.

My question is, in a general sense, what order should these data preprocessing steps be performed?

Fill in missing values,
Normalize/standardize data,
Deal with skewness,
Deal with outliers,
Balance target variable classes

You're asking a complex question that is dependant on what you aim to find.

If there are missing values, are they in columns relevant to the result you're looking for? If not then they needn't be filled at all, so the order does not matter. That being said, if a column with missing values is pertinent to your desired result, then it's often prudent to fill the missing dataset before analysing it further.

Again, outliers need to be addressed in relation to what you're trying to ascertain. In many cases, presence of the outliers can highlight a result in itself. Which may not answer what you're asking, but instead make you question the accuracy of the data itself, or make you determine to include them or exclude them.

You need to be more specific in what it is you're trying to ascertain. Then approach each of the preprocessing steps in the order that is relevant to you. This changes in many different situations, so it's difficult to give a definitive answer.

The tutorial at kaggle to resolve the Titanic Data Science Solutions lays out a good methodology for approaching such problem, and reinforces the concept of sometimes reorganising workflow tasks.

  Workflow stages

  The competition solution workflow goes through seven stages described in the Data Science Solutions book.

  Question or problem definition.
  Acquire training and testing data.
  Wrangle, prepare, cleanse the data.
  Analyze, identify patterns, and explore the data.
  Model, predict and solve the problem.
  Visualize, report, and present the problem solving steps and final solution.
  Supply or submit the results.

  The workflow indicates general sequence of how each stage may follow the other. However there are use cases with exceptions.

  We may combine mulitple workflow stages. We may analyze by visualizing data.
  Perform a stage earlier than indicated. We may analyze data before and after wrangling.
  Perform a stage multiple times in our workflow. Visualize stage may be used multiple times.
  Drop a stage altogether. We may not need supply stage to productize or service enable our dataset for a competition.

  Kaggle - Titanic Data Science Solutions


How to approach clustering of time-series with a single variable Let me preface this by saying that I'm a complete beginner to R and data science in general, so my apologies if this is a rather trivial question. I do have a rough idea of what I would like to achieve though, and how the end result should look like. I just don't know which algorithm is best suited for dealing with this kind of data and how to go about implementing it.

I would like to run a clustering algorithm on a data frame with 13369 distinct units, each having just a single variable that is tracked over 15 years. So my data looks something like this:

id  y1  y2  y3  y4  y5  y6  y7  y8  y9  y10 y11 y12 y13 y14 y15

01  0   6   0   3   0   0   0   0   0   0   0   1   1   0   0
02  1   4   1   6   4   3   7   6   3   8   11  10  9   10  10
03  0   0   2   0   5   0   1   0   3   0   0   0   0   0   0
...
...
...
13369 4  9  0   12  5   1   1   0   6   1   2   7   0   0   3

To establish some context for this task, this is the data for authors and the number of publications they have published in each year, spanning over 15 years.

I would like to end up with a dendrogram that depicts various clusters that authors fall into, (I'm guessing) depending on how prolific they were in a certain time period (beginning vs. later years, for example).

I hope I explained everything clearly.
Thank you for taking your time and assisting a student in need.

Welcome to the community Blitva!

There are things to clear at the beginning:

You may cluster authors based on their time-series but it does not necessarily answer the question you asked. How prolific they were in a certain time-period might need another approach to the problem. In other words, you may cluster the authors but you don't necessarily know what was the reason behind that unless you explicitly define Similarity based on which clusters are formed.
Feature Extraction is sometimes needed when you want a specific question to be answered. See my example bellow to understand it more.
What does being prolific in a certain time period mean? Imagine a person who published 1 2 1 papers in 3 consecutive years and another one who published 10 20 10 papers in those days. Are they similar? If being prolific is calculated between them they are certainly not similar. But if it is calculated according to each person, then they have completely similar patterns of being prolific! Which one are you looking for?

Now back to the question. My answer is as follows:

If you are looking for similar patterns i.e. rise and fall in the number of publications, go for DTW for calculating the similarities. I am sure there are R implementations of DTW but I am a Python guy ;) After calculating pairwise DTW similarities, you may apply your Hierarchical Clustering to it (or maybe some other algorithms like Spectral Clustering).

Good Luck!


SMOTE-NC does not help to oversample my mixed continuous/categorical dataset When I use SMOTE-NC to oversample three classes of a 4-class classification problem, the Prec, Recall, and F1 metrics for minority classes are still VERY low (~3%). I have 32 categorical and 30 continuous variables in my dataset. All the categorical variables have been converted to binary columns using one-hot encoding. Also, before going for the over-sampling process, I am imputing all missing values using Iterativeimputer.

Regarding the classifiers, I am using logistic regression, random forest and XGboost. May I have your thoughts on this? Any suggestions to over-sample a multiclass and highly imbalanced dataset?

Before going through the process of oversampling, always see if the implementation of your algorithm supports assigning different weights to individual classes. The sklearn RandomForestClassifier has for example a class_weights parameter with which you can do that. I found this method to work better than over- or undersampling.

Also, I have to add an obligatory part: if you minority classes have only very few samples so that the charachteristics of the respective classes are not well captured, there is little you can do except collecting more data.


How to use prediction model after onehot encoding? I have created a prediction model for this dataset
>>df.head()

    Service    Tasks Difficulty     Hours
0   ABC         24     1           0.833333
1   CDE         77     1           1.750000
2   SDE         90     3           3.166667
3   QWE         47     1           1.083333
4   ASD         26     3           1.000000

>>df.shape
(998,4)

>>X = df.iloc[:,:-1]
>>y = df.iloc[:,-1].values
>>from sklearn.compose import ColumnTransformer
>>ct = ColumnTransformer([("cat", OneHotEncoder(),[0])], remainder="passthrough")
>>X = ct.fit_transform(X)
>>x = X.toarray()
>>x = x[:,1:]

>>x.shape
(998,339)

>>from sklearn.ensemble import RandomForestRegressor
>>rf_model = RandomForestRegressor(random_state = 1)
>>rf_model.fit(x,y)

How can I use this model to predict Hours for user input in this format [["SDE", 90, 3]]
I tried
>>test_input = [["SDE", 90, 3]]
>>test_input = ct.fit_transform(test_input)
>>test_input = test_input[[:,1:]

>>test_input[0]
array([24, 1], dtype=object)

>>predict_hours = rf_model.predict(test_input)
ValueError

Since my dataset has many categorical values its not possible enter the encoded value of "SDE" as input, I need to convert "SDE" to onehot encoded format after receiving the input [["SDE", 90, 3]]
I don't know how to do it can anyone help?

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestRegressor

df.head()

    Service    Tasks Difficulty     Hours
0   ABC         24     1           0.833333
1   CDE         77     1           1.750000
2   SDE         90     3           3.166667
3   QWE         47     1           1.083333
4   ASD         26     3           1.000000

df.shape
(998,4)

X = df.drop(["Hours"],axis = 1)
y = df.Hours

ct = ColumnTransformer([("cat", OneHotEncoder(handle_unknown = "ignore"),[0])], remainder="passthrough")

rf_model = RandomForestRegressor(random_state = 1)

model = Pipeline([("preprocessing",ct),("model",rf_model)]).fit(X,y)

x_test = pd.DataFrame({"Service":"SDE", "Tasks":90, "Difficulty":3}, index = [0])

# Ideally you split your data into train and test,in this case you need to pass x_test that is a pandas dataframe with the values you want to predict
model.predict(x_test)


Why the my Tensorflow code just use one GPU when I assign more than one I am trying to run my code on a supercomputer with 8 gpus. Though, I assign 8 gpus but one of them is just occupied. I read some notes in websites and it seems that Tensorflow automatically use gpu if it is applicable but still I don't know how I can use all the gpus. The code is just a deep network to be trained using model.fit() and then predict the test data using model.predict()

If a TensorFlow operation has both CPU and GPU implementations, TensorFlow will automatically place the operation to run on a GPU device first. If you have more than one GPU, the GPU with the lowest ID will be selected by default.
TensorFlow 2.0 now has the tf.distribute module to distribute training across multiple GPUs, multiple machines or TPUs. It builds on the concept of "distribution strategies". You can use tf.distribute.MirroredStrategy() as a scope, like
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    #DO WHATEVER YOU WANT TO DO HERE

Please check this Distributed training with TensorFlow guide for implementation details and other strategies.


Training AI to do repetive retouching job I'm new to AI, so bear with me... What would be the easiest way to do this using AI? Where do I start? What technologies are already available?

Sample data:
50.000 before images ex. there's a red ball that needs to be removed
50.000 after images the red ball is removed out of the image (by a human in photoshop)

Feed the data
AI learning magic

Feed the AI Input:
1 other before image featuring the red ball
Output expected:
1 image where the red ball is removed from the image.

This is a very (very) broad question.

What you are trying to do is utilize some neural model to perform image processing. So your output needs to be the same size as your input, meaning that you need to use a fully-convolutional model for this.

I suggest you read the following article:

Fast Image Processing with Fully-Convolutional Networks

They are doing exactly what you need, only for different image-processing methods (and more than one).

However! I think that before going on to implementing a neural model, you should read a bit about the general subject of machine learning and deep learning. How Do I Learn Neural Networks?


Training with data of different shapes. Is padding an alternative? I have a dataset of about 1k samples and I want to apply some unsuspervised techniques in order to clustering and visualization of these data.

The data can be interpreted as a table of a spreadsheet and unfortunately it doensn't have a very defined pattern of structure. The number of table lines varies, but not the columns.

The data is structured like this:

sample 1:

{
  "table1": {
    "column1": [
      "-",
      "-",
      "-"
    ],
    "column2": [
      "2017-04-16 10:00",
      "2017-04-16 10:00",
      "2017-04-16 10:00"
    ],
    "column3": [
      "-",
      "-",
      "-"
    ],
    "column4": [
      "name X",
      "name Y",
      "name Z"
    ],
    "column5": [
      "0",
      "0",
      "0"
    ],
  }
}

sample 2:

{
  "table1": {
    "column1": [
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-"
    ],
    "column2": [
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00",
      "2017-04-10 22:00"
    ],
    "column3": [
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-",
      "-"
    ],
    "column4": [
      "name A",
      "name Z",
      "name B",
      "name X",
      "name C",
      "name D",
      "name E",
      "name F"
    ],
    "coumn5": [
      "",
      "",
      "3",
      "1",
      "0",
      "3",
      "0",
      "0"
    ]
  }
}

These samples comes from alarms generated by a systems that collects informations from a lot of nodes (these nodes are named as "name A", "name B"...). My objective is to transform these data into a matrix (n_samples x n_features) to apply clustering and visualization algorithms.

How can I work with these data for unsupervised training? Is padding a way forward for this problem? If so, how can I apply the padding on this case?

Whether or not padding is ppropriate really depends on the entire structure of your dataset, how relevant the different variables/columns are and also the type of model you want to run at the end.

Padding would be used, whereby you would have to fix the length of each sample (either to the length of the longest sample, or to a fixed length - longer samples would be trimmed or filtered somehow to fit into that length). Variables that are strings can be padded with  empty strings, variables with number can be padded with zeros. There are however many other ways to pad, e.g. using the average of a numerical variable, or even model-based padding, padding with values that "make most sense" for filling gaps in that specific sample. Getting deep into it like that might more generally be called imputation, instead of padding - it is common in time series data, where gaps aren't always at one end of a sample.

Below I outline one approach to padding or standardizing the length of each sample. It it not specifically padding.

As you did not mention a programming language, I will give and code snippet in Python, but the same is easily achievable in other languages such as R and Julia.

The Approach

Based on the two examples you provide, it seems each example would be a calendar day, on which there are a variable number of observations.

There are also columns that are strings, and others are strings of numbers (e.g. column 5 in sample 2).

In time-series analysis in general, it is desirable to have a continuous frequency of data. That means have one day give one input. So my approach would be to make your data into a form that resembles a single input for each of the variables (i.e. columns) of each sample.

This is general approach, and you will have to try things out or do more research on how this would look in reality for your specific data at large.

Timestamps

I would use these as a kind of index, like the index in a Pandas DataFrame. One row = one timestamp. Multiple variables are then different columns.

Dealing with strings

I will assume that your dataset has a finite number of possible strings in each column. For example, that column 4 (holding names), will always hold a name from a given set. One could perform set(table2['column 4']) to see which values there are (removing duplicates). Or even:

# Gather a single list containing all strings in column 4 of all samples
all_names = []
[]    # list comprehension to loop

# Check how many strings there are
from collections import Counter
counter = Counter(table2['column4'])
print(len(counter))                 # see how many unique values exist
print(counter)                      # see how many times each string appears
print(counter.most_common(5))       # see the most common 5 strings

Now assuming this shows there is a finite number (more than likely the case), You could look into using a sparse representation of each sample (that means for each day). For example, if all the words in the entire dataset were: ['hi', 'hello', 'wasup', 'yo', 'bonjour'] (duplicates removed), then for one single sample with column 4 holding e.g. ['hi', 'hello', 'yo', 'hi'], your sparse representation for this sample would be: [2, 1, 0, 1, 0], because the sample has two 'hi', one 'hello', zero 'wasup' and so on. This sparse representation would then be your single input for column 4 for the timestamp (that single sample).

It might be worth looking into something like the DictVectorizer and CountVectorizer from Scikit-Learn.

Dealing with number columns

As I mentioned right at the beginning, you could pad these to a chosen length, perhaps matching the length of the string based representation above (or not!), depending on your final model.
You can then pad the inputs with a value that makes sense for your model (see the kind of options I mentioned at the beginning of my answer).
This should land you with, once again, a single vector for the given day, containing the information in the numerical column.


"Object" Detection in Textual Data I have a task where the input is a parsed document (i.e., full text in 1 string or tokens) and I need to classify parts of the text into say 5 classes (i.e., 5 tokens from the entire text are labeled into 5 different classes).
Example:
Document #1: "... cat ..." (the token "cat" belongs to class "0" which is animals)
Document #2: "... fish ..." (the token "fish" belongs to class "0" which is animals)
It is important to note that at inference time, I have the entire document (in text), and so most of the tokens from it do not belong to any of the classes.
What would be a good approach to this task? I thought about a simple classification problem where I take the labeled tokens from each document and input it into an RNN classifier, but that ignores the rest of the document and at test time irrelevant tokens can have larger probabilities than the labeled tokens.
I also had an idea inspired by YOLO, and maybe apply a 1D CNN object detector (with the respective number of classes) on the entire text. Is this reasonable?
Thanks.

This looks quite similar to Named Entity Recognition (NER), which is traditionally done with a sequence labeling model such as Conditional Random Fields. Normally NER is used when:

The list of possible entities is not predefined: the training data might contain "Mr James Smith" but the test data could contain "Mr John Doe". In other words, the classes are open.
It is assumed that the context of the text can help the model predict an entity. For example in a sentence like "Today X said that ...", the word "said" after X should help the model predict that X is either a person or an organization, but it cannot be a location.


Maximize one data point I am completely new to data science and looking to narrow down the search and reduce the learning curve required to solve problems like the one given below

I have a data set with 7 columns ,
Column A(all positive decimal) is the data point I want to maximize.
Column B and C are boolean values
remaining columns are a combination of positive and negative decimal numbers.
I want to find some relation and insights from all colums such that I can maximize the sum of column A.

In R you can run a linear regression. Consider this "academic" minimal example:

df = data.frame(c(3,5,2,7,5,3), c(1,0,1,0,1,0), c(0,1,1,0,1,0))
colnames(df) = c("A", "B", "C")
df

Take this data as an example:

  A B C
1 3 1 0
2 5 0 1
3 2 1 1
4 7 0 0
5 5 1 1
6 3 0 0

Now we can see how B and C describe A in the best way.

reg = lm(A~B+C, data=df)
summary(reg)

Output:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    4.917      1.322   3.719   0.0338 *
factor(B)1    -1.750      1.774  -0.987   0.3966
factor(C)1     0.250      1.774   0.141   0.8968
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.048 on 3 degrees of freedom
Multiple R-squared:  0.2525,    Adjusted R-squared:  -0.2459
F-statistic: 0.5066 on 2 and 3 DF,  p-value: 0.6463

This tells us that when B, C is 0, A=4.1917 if B=1 we would have A=4.917-1.750 and if C=1 we would have A=4.917+0.25.

So, we can also make predictions:

predict(reg, newdata=df)

Which would be in this case:

       1        2        3        4        5        6
3.166667 5.166667 3.416667 4.916667 3.416667 4.916667

This is a simple form of ML (linear regression), where the sum of squared residuals is minimized in order to find the coefficients for the intercept as well as B and C which best describe A.

You would write this model like: $A = \beta_0 + \beta_1  B + \beta_2  C + u$, where $u$ is the statistical error term. You would solve this model by minimizing $\sum u^2$ (the sum of squared residuals).

In matrix algebra you could write $y=\beta X + u$, and you would solve this by $(X'X)^{-1}X'y = \hat{\beta}$.

So we do not "maximise" but minimize the statistical error $u$ in order to find the best "fit" for columns B, C given column A.

Have a look at the great book "Introduction to Statistical Learning" to get the main concepts sorted.


How can I annotate labels to images automatically? I have 80 classes, for each class, there are 100 images. I want to label all images for object detection.
For this task I have downloaded the LabelIMG tool but it's taking a long time to do it. How can I automate labeling?

Automatic image annotation has its limitations. It's basically another already trained machine learning model that's doing the annotation for you.

Keep in mind that automatic annotation also requires another step in between, which is selecting elements with low prediction confidence and review them by hand. These are the most problematic elements when you use an automatic tool to label images and almost always require manual review.

Image source and suggested reading: https://medium.com/anolytics/what-are-the-limitations-of-automatic-image-annotation-vs-manual-a8ff7edb3152


Bellman operator and contraction property Currently, I am learning about Bellman Operator in Dynamic Programming and Reinforcement Learning. I would like to know why is Bellman operator contraction with respect to infinity norm? Why not another norm e.g. Euclidian norm?

The proofs I have seen for contraction of the bellman operator (this page has a really nice run through, explicitly utilize the fact that an infinity norm is being used. This does not necessarily mean that contraction doesn't occur under other norms, but it does suggest one of two possibilities:

The contraction proof is only valid under the infinity norm.
The infinity norm is just the easiest metric to prove the contraction property.

When showing that the Bellman Operator converges to a fixed point it is satisfactory to simply show that it is a contraction, it doesn't matter what sort of contraction it is, so we would typically prove the contraction that is easiest to show.
That being said, intuitively, I would imagine that the question of whether or not this contraction holds under other norms would impact questions surrounding the rate of convergence to the fixed point (ie., how many times we must apply the bellman operator for convergence). After a cursory search I was not able to find any theory explicitly proving or disproving the contraction under other norms. This might be an interesting question to raise in the mathematics stack exchange.


The output of Model from Decision Tree and Random Forests are different? I have been made a model using both Decision Tree and Random Forest. But, when I tried to test the model on the same DataFrame the output is different. How is this possible?

The data file from my repo:

#This is the function to help me preparing the dataframe
def process_df_for_ml(df):
    """
    Process a dataframe for model training/prediction use.

    Returns X/y tensors.
    """

    df = df.copy()
    # Map salary to 0,1,2
    df.salary = df.salary.map({"low": 0, "medium": 1, "high": 2})
    # dropping left and sales X for the df, y for the left
    X = df.drop(["left", "sales"], axis=1)
    y = df["left"]
    return (X, y)

I used the decision tree:

from sklearn import tree
from sklearn.model_selection import train_test_split
# Train a decision tree.
X, y = process_df_for_ml(df)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)
clftree = tree.DecisionTreeClassifier(max_depth=3)
clftree.fit(X_train, y_train)

With test_score: 0.96. Afterwards, I test this DecisionTree to the same df and the output I got is [424 rows x 11 columns]

and then I tried to use Random Forest algorithm

X, y = process_df_for_ml(df)

from sklearn.model_selection import train_test_split
# implementing train-test-split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0, stratify=y)

from sklearn.ensemble import RandomForestClassifier
# random forest model creation
rfc = RandomForestClassifier()
rfc.fit(X_train,y_train)
# predictions
rfc_predict = rfc.predict(X_test)

With test_score: 0.99. Afterwards, I test this RandomForest to the same df and the output I got is [11 rows x 11 columns].

How is this possible?
Here is the link to my works:
DecisionTree and RandomForest

Your df row count is 14,999

Your test data is 33% ~ 4950

So, your y_predict should be (4950,1) i.e. a binary prediction for all test rows.

This is what I am getting when running your code which you have posted here.

import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/bhaskoro-muthohar/DataScienceLearning/master/HR_comma_sep.csv')

print(df.size)
# Map salary to 0,1,2
df.salary = df.salary.map({"low": 0, "medium": 1, "high": 2})
# dropping left and sales X for the df, y for the left
X = df.drop(["left", "sales"], axis=1)
y = df["left"]

#splitting the train and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0, stratify=y)
# Train a decision tree.
from sklearn.tree import DecisionTreeClassifier
clftree = DecisionTreeClassifier(max_depth=3)
clftree.fit(X_train, y_train)
y_pred = clftree.predict(X_test)
print(y_pred.size)
from sklearn.ensemble import RandomForestClassifier
# random forest model creation
rfc = RandomForestClassifier()
rfc.fit(X_train,y_train)
# predictions
rfc_predict = rfc.predict(X_test)
print(rfc_predict.size)


Checkers playing Neural Network evolved with Genetic Algorithm becomes too sensitive to input data changes I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:

Neural Network Structure:

Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.

Genetic Algorithm

30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:

$m' = m * e^{r*0.9} $
$w' = w * m' * r$

Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.

Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.

First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?

That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.


Significance: As far as I know, sklearn does not come with a module to get p-values. You could go for statsmodels:

https://stackoverflow.com/a/55655645/9524424

Interpretation: Remember that Logit uses a logistic link function, so there is no easy on-the-go interpretation of the coefficients. You need to calculate marginal effects, so the change in $y$ when you alter some $x$, all other things equal. Stata has such module(s). However, I think sklearn has no such implementation. Python statsmodels seems to come with an option to derive marginal effects as well, but I never tried it. Look for .get_margeff.

https://www.statsmodels.org/stable/generated/statsmodels.discrete.discrete_model.DiscreteResults.get_margeff.html


Has anybody created interesting ConvNets on 3D movie data? In the last video of his course on Convolutional Neural Networks, Andrew Ng was discussing using ConvNets on 3D input data. He mostly discussed cat scans, but also mentioned you could treat a movie as 3D data and run a ConvNet on it (where the z axis is time, i.e. the # frame of the movie).

I find this pretty fascinating. I'm just curious if anybody is aware of anybody doing something like this in practice, and what they accomplished.

The 2015 Thumos challenge was about action recognition in video collected from youtube - totalling 13k videos (430 hrs or 45 million frames) and 101 'action classes'. The video is of activities like people brushing their teeth, playing basketball, and surfing, and some video clips included multiple activities.

The video is untrimmed, so it's closer to a real-world setting than video edited down to exclusively a single action of interest. The winning and second place entries for action classification included CNNs in various guises (after a quick scan I couldn't find any entries that didn't).

Another use you might be interested in is video captioning (not classification): this paper incorporates CNNs to describe in words what is taking place in each video (examples from p9 of the article, I love the descriptions at bottom right):


Ideal strategy for multi variable regression attempting to maximize the target I am trying to implement machine learning for the following data

Data Input

What I am trying to achieve is to keep the ad bid & cost per sale as low as possible while increasing sales.

This is a multi variable, multi response regression and I am looking for ideas and machine learning models to implement this.

I don't think reinforcement or multi-output regression will work because I want to reduce ad bid + cost per sale and at the same time increase sales (not target)

Technology/SDK is not a problem. I am looking for architecture/model ideas.

Interesting question. As you know, indeed, it can be seen as a pure optimization problem. For that you need to come up with mathematical modeling of dependent variable based on independent ones. I might have an easier ML-based suggestion:

Create a new target out of all 3 variabales in a way you want them to behave. For instance, you said bid and cost low but sales high, so you might want to turn them simply to:

$y_{new} = \frac{bid+cost}{sales + \epsilon}$

$\epsilon$ is simply a factor to avoid dividing by zero. Please note that there seem to be a strong $TotalCost = CostPerSale\times Sales$ relation. Have it in mind in case you observe bad/strange results.

As seen in $y_{new}$, smaller it is, more interesting it is to you. You apply a regression to the new data (it will be a uni-response regression now as we combined all targets) and if prediction is less than a threshold (defined by you through experiments) you consider it as good. Or you can just report the predictions and the speak for themself; smaller the better.

PS1: That is the simplest formulation i came up with. Indeed, there is an optimization problem to be solved to find a proper formula. Here we assumed the summ of bid and cost is the minimum. One may say their product should be in the enumerator for example. You need to design that formula.
PS2 Strongly recommend to inspect your variables carefully, have an insight about probable correlations between them, remove non-informative features if any, etc.

Good Luck!


Handling categorical missing values ML I have gone through this regarding handling missing values in categorical data.
Dataset has about 6 categorical columns with missing values. This would be for a binary classification problem
I see different approaches where one is to just leave the missing values in category column as such, other to impute using from sklearn.preprocessing import Imputer, but unsure which is better option.
In case if imputing is better option, which libraries could I use before applying the model like LR,Decision Tree, RandomForest.
Thanks!

First of all I would look how many missing values there are in the column. If there are too much (~20%, generally its difficult to say how much is too much), I would drop the column because imputing 20 % of your data (without prior expert knowledge) or even more probably does not give you meaningful information anymore.

Secondly I would look at correlations between missing values and other features. Maybe you are lucky and there is some correlation between missing values in column x and a categorical value in column y. Simply look at conditional distributions.

If you choose to impute, check the distribution of the categorical values for non-missing entries. If the distribution is heavily skewed, say there is 95% value 0 and only 5% value 1, you can use the median to impute. Again, the question is how informative this is in the end. Otherwise create an additional categorical value which simply represents a missing value.


What is the input of LSTM network? Hello I am trying to understand LSTMs but have a few problems:

What is the input? Since LSTM is seq2seq I would think it is a sequence of words, but in a Codecademy lesson is mentioned that each sentence is represented as a matrix with a bunch of vectors containing 1 or 0 for the timestep -> sentence "I like Bobo" like = [0, 1, 0], so what is now the input? The matrix or the sequence of words?

What is passed to the next LSTM cell after a prediction before was false? Since the false prediction is noted in the hidden state, how does the network know whether previous predictions were false? Or does it even know when predicting the next step?

I am excited for the answers,
love Phiona.

The input of an LSTM is a sequence of vectors. In your case, each of these vectors represents a word encoded as a one-hot vector. One-hot encoding is a way to express a discrete element (e.g. a word) numerically. Each one-hot vector is a vector of length $d$, where $d$ is the total number of words we can represent, and where all positions in the vector are 0 except the position associated with the represented word, which contains a 1.
The hidden state passed to the next LSTM cell is not the final binary prediction, but the dense numerical vectors we obtain before computing the binary prediction.


Getting very low/ wrong accuracy from RandomizedSearchCV I am currently using RandomizedSearchCV to optimize my hyper-parameters. However the reported scores of each iteration is very low. When I then evaluate the highest scoring candidate I get very high accuracy (0.97), while the RandomizedSearchCV reports something much lower (0.32).
search = clf_rand_search.fit(x_traintest, y_traintest)
print(search.score(x_validation,y_validation))
0.32

print(accuracy_score(y_validation.flatten(), search.predict(x_validation).flatten()))
0.9798260869565217

My input and output are both 2-D matrix with (100,9) and (100,230) shape for the train/test data. With lower samples for the validation data.
Should I format my data differently for the RandomizedSearchCV?
Input features first two are normalised and the last one-hot encoded.

Output classification 0 or 1 for 230 nodes.

clf = MLPClassifier(solver = 'adam',
                max_iter=9999,
                alpha=1e-5
                )

hidden_layers = 8
neurons = list(range(10,210,5))

m = [0]*(hidden_layers*len(neurons))

for i in range(1,hidden_layers+1):
    for idx,i2 in enumerate(neurons):
        m[((i-1)*len(neurons)) + (idx)] = [neurons[idx]]*i

param_space = {
    'hidden_layer_sizes':   m,
    'activation':           ['identity', 'logistic', 'tanh', 'relu'],
    'learning_rate':        ['constant','invscaling','adaptive'],
    'learning_rate_init':   np.arange(1e-4,0.1+1e-4,1e-4)
    }

clf_rand_search = RandomizedSearchCV(clf, param_space, n_iter=10,
                                        scoring="accuracy", verbose=True, cv=2,
                                        n_jobs=-1)

UPDATE AFTER EXCHANGING COMMENTS
You might be facing issues with the computation of the accuracy

I think the MLP with the log-loss can work well with your output data. Your output data is a vector $(N, 230)$ with $N$ the number of samples, with only 1s or 0s. This data is indeed a one-hot encoded vectors with multiple 1s.

You are computing the accuracy by flattening the predictions and comparing them elemt-wise ($N*230$ elemetns),

For computing the accuracy, I guess if the classifier is not comparing the one-hot vectors (ground truths and predictions) element wise , but comparing if they re the same for each one of the $N$ samples.  Think that if only one element of the 230 onte-hot encoded is missclassified in one sample, this accuracy will drop by $1/$N$*100$ %. If accuracy is computed element-wise, the drop in the accuracy will be $1/(230*N)*100$ %...

Try to update your accuracy computation assingning a 1 only if each predicted (230, 1) vector of each sample is equal to its (230,1) ground truth vector.

PREVIOUS ANSWER BEFORE COMMENTS
I guess you are experimeting problems with your data...
Your data does not seem to be a binary classification problem, for binary classification your data should have the following dimensions:

Input: $(N, K)$
Output: $(N, 1)$ or $(N, 2)$ if is one-hot encoded.

With $N$ the number of sample of the split, $K$ the dimension of the feature input space
If your output data is of size
$(N, D)$,
then it could be a regression problem that maps a feature space of $K$ to $D$. Yout MLP should try to mimmic a function.
$f'(X): K \to D$
For regression problems you must use other metrics, like MAE or MSE
You can treat the problem from various perspectives (just some ideas).

as a masking/segmentation problem in 1D (making an analogy with mask segmentation for 2D images...)
A Bayesian approach could work well also. You can try to estimate the posterior probability of parameter of the Binomial distribution which models the nodes being 1 or 0 (posterior) from your data (likelihood)... and then update your posterior distribution with new data so that you can have updated and more accurate credible intervals. Obviouslly you may thing about if a Binomial distribution models fine your data (i.e. independancy, etc...) Maybe other probability distribution works better

Thge following example works well. I know it does not answer the question (I'm just trying to help), but at least, we can see with an example that RandomizedSearchCV should give the same score as the accuracy_score with your code for a binary classification problem with a MLP
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import accuracy_score
from scipy.stats import uniform
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_classification(n_samples=1000, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf = MLPClassifier(solver = 'adam',
                max_iter=9999,
                alpha=1e-5
                )

hidden_layers = 2
neurons = list(range(10,210,5))

m = [0]*(hidden_layers*len(neurons))

for i in range(1,hidden_layers+1):
    for idx,i2 in enumerate(neurons):
        m[((i-1)*len(neurons)) + (idx)] = [neurons[idx]]*i

param_space = {
    'hidden_layer_sizes':   m,
    'activation':           ['identity', 'logistic', 'tanh', 'relu'],
    'learning_rate':        ['constant','invscaling','adaptive'],
    'learning_rate_init':   np.arange(1e-4,0.1+1e-4,1e-4)
    }

clf_rand_search = RandomizedSearchCV(clf, param_space, n_iter=2,
                                        scoring="accuracy", verbose=True, cv=2,
                                        n_jobs=-1)

search = clf_rand_search.fit(X_train, y_train)
print(search.score(X_test,y_test))

print(accuracy_score(y_test.flatten(), search.predict(X_test).flatten()))


Algorithms for Sentiment Analysis on Entity I want to make sentiment analysis for an entity which was found, like Google NLP.

Entity should have magnitude and score.

Please share with me the possible research papers.

p/s please not propose to make sentiment for sentence where the entity is located and them assign to entity from such sentence.

A roll-your-own solution using Mathematica is described at https://pdfs.semanticscholar.org/0245/c38fd1f3c7859c5e22382426178090faa395.pdf

I've not found such capability built into Mathematica's core, although that research paper is from 2013.

IBM Cloud's Text Analysis API also includes this capability: https://console.bluemix.net/apidocs/natural-language-understanding#entities

See also the responses to sentiment analysis for multiple entry in one text

There may be others...