Why are ordered lists considered one dimensional? At Uni I am currently taking data visualization as a course.
While reading thru the script I stumbled upon two examples for one dimensional data.
Example one:
Bus stops with their name and order (current -> next bus stop).
Example two:
A list of values (not a set!)(the type of value does not matter for the sake of simplicity).
Now I am wondering how an ordered list can be two dimensional. To me it looks like two dimensional data.
The first dimension being the index, the second one being the value. For the first example the two dimensions could be the station name and the next stations name.
Since v(t) is considered as two dimensional data (according to my Prof) I am wondering why value(index) us not two dimensional data.
Representation wise I can understand why the data can be represented one dimensional, since they can be represented in a straight line.
But feature wise I don't understand how value(index) and v(t) are different.
Please enlighten my mind since the prof only told me to stick to the script...

The specific ordering is only metadata as it is not explicitly in the data itself, and the index doesn't always represent order.
If you have an additional row with the order in then it would be two dimensional. But, for example, you wouldn't be able to distinguish between ascending and descending order by looking at the index itself.
v(t) is velocity by time
data[index] is data point by location in a data object (rather than known order).
Perhaps you could argue that the index is a second dimension, but consider this, how would you represent 1D data in a computer otherwise?


Plotting clustered sentences in Python I have the following three sentences, extracted from a dataframe. I would like to check the similarity and create clusters based on their level of similarity.

Authors       Sentences
John Smith   Some people do not completely understand the risk of UV rays.
Jane Lower   People do not understand the risk of UV rays, wrote the journalist in the Herald.
Craig Avatan In Berlin, people do not know how dangerous can be for their health a long exposure to UV rays.

I would need to cluster them based on words and their sequences (like plagiarism).
I have tried to use k-means, but I have not completely understood how to create clusters to plot.
Something like this:

I have tried to use k-mean as follows:

def sent_tokenization (line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words_list]
    return words_lemmed

tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

kmeans = KMeans(n_clusters=2).fit(tfidf)

However I am not able to plot the results. What I am looking for is something that can be easily visualised. Specifically, I would need to plot in a scatter plot as in the example, which can show the name of authors based on their sentence similarity, like in plagiarism. I am trying to see which authors have written similar texts.

In my example, I should have the first two authors closers than the third one, as their sentences are very similar (in terms of words and structure).

Could you please give me advice on how to plot/cluster the above information?
If you need more information, feel free to ask.

It can be made in many ways. My starting approach would be (1) to apply the tf-idf as in your code snippet, then (2) reducing the output matrix to a lower dimensionality (say 2D for seeing it in a scatterplot) with a decomposition method (or with topic modelling approach), and finally (3) apply a clustering algorithm to visualize the samples/documents of each cluster.

However, this may have many flaws, because I don't know if the decomposition results would make any sense and I'm not aware of how many clusters are good enough, but anyway it would be useful as an exploratory analysis.

The logic would be something like that: (didn't tested)

# 1
tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

# 2
decomp_method = NMF(n_components=2)
tf_2d = decomp_method.fit_transform(tfidf) # output shape: [N, 2]

# 3
kmeans = KMeans(n_clusters=2)
tf_kmeans_labels = kmeans.fit(tf_2d).labels_ # array

# visualization
data = pd.DataFrame(tf_2d, columns=['c1', 'c2'])
data['kmeans_labels'] = tf_kmeans_labels
scatterplot(x='c1', y='c2', hue='kmeans_labels',data=dat)
```


random forest classifier - impact of small n_estimator and repeated training trying to have a better understanding of random forest algorithm here. With the same training and holdout datasets, I tried two things here:

Set a small n_estimator (10), train on my training dataset and apply to my holdout dataset. If I repeat this several times, the result (e.g. correctly predicted target class) varies somewhat from run to run. My understanding is that since the # of trees is small in my model, there are variations in my model after training thus leading to different results.
Set a high n_estimator (300) and do the same. Then the results don't vary. My take is that impact of high n_estimator reduces variation in the model and thus i get the same prediction every time.

So if I run my scenario 1 a bunch of times and consolidate the results (i.e. run 1 predicts A B in class 1, run 2 predicts A C in class 1, run 3 predicts D in class 1), my final results would be A B C D are in class 1. My question is:
1. Is this essentially the same as running it once with a large n_estimator?
2. Is this approach problematic because I am relying more on "guessing" (e.g. small n_estimator leads to larger variation in outcomes)?

Thanks!

there is a parameter in estimators that is called random_state that fixes a seed for your algorithm run and reruns exactly the same so you can expect the same results at every rerun of the exact same code. It makes your code deterministic. But yes, in random forests in particular, due to their nature of averaging across all trees created when the forest is grown, variance in your result becomes less evident. So your reasoning is correct.

If you want to make your low_estimators RF produce the same results at every run, just add random_state and give it a random number


Orange Changes the data type I have a column as month in my excel data set, when i load the data in the file, orange changes it to some numerical values.. Help! I want the Month as it is. Tried changing the value type as short or long date in excel too.

The orange loads Dec-16 as 42055.00 something.

I have the same problem, I was told to avoid Excel as it did not conform to the ISO 8601 standard of 2016-12-27.

Then I transferred the data to Google Sheets as illustrated in Orange tutorial YouTube videos.

The dates were altered to match the required format however the results remain exactly the same.


Classification algorithm with multiple output for a set of features I want to build a classification algorithm that will predict multiple values for a set of features. For instance, lets say I have a customer demographic data like Income, age, sex, city and I want to predict 5 top products they will buy: like if a customer is male, has income less than 30k, lives in Tier1 city and has age 20-35 years--then the output is: 1. Mobile 2. Car 3. Furniture i.e., there are three outputs which means they would most probably buy Mobile, Car and Furniture with decreasing value of probability.
Which algorithm will be best suited for this problem?

Your problem can be considered as multiclass classification problem. So, you have a dataset of features X and the predictor y. Where X contain Income, age, sex,etc. and y is an item that one customer will buy with higher probability.

To achieve your goal and predict the probability of a customer you can use any classifier from scikit-learn Library (if you use python) and call the method predict_proba after you fit the classifier, as shown in the following example:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

# Make a random multiclass classification problem
X, y = make_classification(n_samples=1000,
                           n_features=10,
                           n_informative=10,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=5,
                           random_state=0,
                           shuffle=False)

# Fit any classifier
rfc = RandomForestClassifier(n_jobs=-1,max_features= 'sqrt' ,n_estimators=50, oob_score = True)
rfc.fit(X,y)

# Prediction for one example
one_example = X[10,:].reshape(1,-1)
outcome     = rfc.predict_proba(one_example)

The outcome of this problem with 5 classes is

array([[0.76, 0.12, 0.02, 0.06, 0.04]])

Which represent the probability of each class for this example (or customer in your case)


Trying to make compelling plot for classification results with python I've been working on a classification problem and have some good results, but now I struggle with trying to put together a good plot to illustrate the probabilities for each prediction.  Here is my current data:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df_voting_output.head(n=5)
    prob    actual  pred    correct
0   0.460200    0   0   1
1   0.548478    1   1   1
2   0.270609    0   0   1
3   0.686557    0   1   0
4   0.527935    0   1   0
5   0.098687    0   0   1

I've been able to create a bar chart with probabilites using this code:
plt.bar(np.arange(len(voting_predictions[:,1])), voting_predictions[:,1])
plt.xlabel("record number")
plt.ylabel("probability")
plt.title("Classification Probabilities")
plt.show()

However, I'm thinking there has to be a better way to include more information and a key.  I'd like the "correct" labels to be clear to see, so from a visual standpoint you can see how the probabilities relate to a correct classification.

It's really rare that you'd show a plot of the probabilities for each example in your set. Are you sure you want to do this? A better presentation might be a confusion matrix. Here's how it works:

1) The columns are the true class labels

2) The rows are the predicted classes

3) Along the right hand side of the plot you can show the probability of correctly assigning to a class (or the classification error, if you prefer).

For example, say I have three classes in my dataset.
I have 10 examples of each class, so 30 examples total.

My classification results from my model are:

8 of the 10 examples in class 1 were correctly labelled; 1 was misclassified to class 2 and 1 was misclassified to class 3.

7 of the 10 examples in class 2 were correctly labelled; 3 were misclassified to class 3.

9 of the 10 examples in class 3 were correctly labelled; 1 was misclassified to class 2.

Then my confusion matrix looks like this:

$\begin{bmatrix} 0.8 & 0.1 & 0.1 \\ 0.0 & 0.7 & 0.3 \\ 0.0 & 0.1 & 0.9 \end{bmatrix}$

And along the right hand side of my confusion matrix I can include the classification error: 0.2 for class 1; 0.3 for class 2; 0.1 for class 3.

Notice the rows must add to 1.


Classifying objects in video without machine learning Recently, Nick Bourdakos posted a series of videos demonstrating bottle detection in a video stream using Tensorflow.js. Specifically, he is using SSD-mobilenet.

The problem could be summarised as follows:

Three different drinks bottles appear together or individually in a video stream. Let's assume Coke, Mountain Dew and Pepsi as in the original video
Each bottle must be classified and labelled with its name
A rectangular bounding box should be placed around each bottle

I am interested to know if there are there any established techniques that can achieve similar results without machine learning?

So far I have tried:

Identification based on colour thresholding. This works surprisingly well but is not very robust. For example the red lettering inside the Mountain Dew label can be falsely detected as "Coke". Also, bounding boxes can only be drawn around coloured regions rather than the entire bottle
Template based matching. This didn't work at all. I assume this is intended for static images where an exact match is required.
KCF Tracking. This seems to work well, but I have to manually define a region of interest to track first. That is, objects are tracked but not classified
I also tried using edge detection to classify "bottle" or "not bottle" based on the aspect ratio of edged objects, however because the bottles are "hand held" separating the bottle edge from the user's hand is problematic

Is there some computer vision technique or steps that I'm missing that would come closer to solving the problem?

You could try SIFT (Scale-invariant feature transform) to recognize the labels on the bottles. Althoug you need some preprocessing, just for comparing the labels on the bottle and to distinguish the different brands it should work.
Also it could get kind of complex (if you show the bottles from behind and the brandlogo is not visible). So you maybe have to extract features from many perspectives of the bottle.

Here is one example: https://towardsdatascience.com/bibirra-beer-label-recognition-8546c233d6f4


How do I determine which variables contribute to the 1st PC in PCA? Given the coefficients of PC1 as follows for each variable (0.30, 0.31, 0.42, 0.37, 0.13, -0.43, 0.29, -0.42, -0.11) which variables contributes most to this PC? Does the sign(+/-) matters or considering the absolute value is enough?

Welcome to the site. PCA is an unsupervised dimensionality reduction algorithm. It works by transforming the original feature-set into eigen-vectors that are difficult to map with the original feature set. As such, the first Principal Component (PC) contains the features with maximum variance. The subsequent PCs contain features with decreased variance to the first PC.
With this background, I invite you to read this Q on SO. It has the solution to programmatically determine the features deemed most important by PCA.
[edited]
Regarding the sign of the components, eve if you change them you do not change the variance that is contained in the first component. Moreover, when you change the signs, the weights (prcomp( ... )$rotation) also change the sign, so the interpretation stays exactly the same:
set.seed( 2020 )
df <- data.frame(1:10,rnorm(10))
pca1 <- prcomp( df )
pca2 <- princomp( df )
pca1$rotation

gives
                PC1        PC2
X1.10     0.9876877  0.1564384
rnorm.10. 0.1564384 -0.9876877

and pca2$loadigs gives,
               Comp.1 Comp.2
SS loadings       1.0    1.0
Proportion Var    0.5    0.5
Cumulative Var    0.5    1.0

Then the question arises that why the interpretation remains the same
You do the PCA regression of y on component 1. In the first version (prcomp), say the coefficient is positive: the larger the component 1, the larger the y. What does it mean when it comes to the original variables? Since the weight of the variable 1 (1:10 in df) is positive, that shows that the larger the variable 1, the larger the y.
Now use the second version (princomp). Since the component has the sign changed, the larger the y, the smaller the component 1 -- the coefficient of y< over PC1 is now negative. But so is the loading of the variable 1; that means, the larger variable 1, the smaller the component 1, the larger y -- the interpretation is the same.
The conclusion is that for each PCA component, the sign of its scores and of its loadings is arbitrary and meaningless. It can be flipped, but only if the sign of both scores and loadings is reversed at the same time.
Furthermore, the directions that the principal components act correspond to the eigenvectors of the system. If you are getting a positive or negative PC it just means that you are projecting on an eigenvector that is pointing in one direction or 180∘ away in the other direction. Regardless, the interpretation remains the same! It should also be added that the lengths of your principal components are simply the eigenvalues.


Jaccard similarity between two items Calculating similarity between two users is rather straightforward.

Consider following example:

User A = {7,3,2,4,1}
User B = {4,1,9,7,5}

Products in common = {1,4,7}
Union of products = {1,2,3,4,5,7,9}

Hence the Jaccard similarity: 3/7 = 0.429

However it is not clear to me how to calculate similarity between two products.
Let's say I want to calculate similarity between products 7 and 1 from previous example, how can one achieve that?

In any commerce setting, the concept of item similarity is very not straightforward. Two users usually buying same kinds of products can be considered as similar, but we cannot say the same about two items bought by same user.

There are two different concepts of item similarity for recommendation purposes. One is, if the two items are physically similar, for example : Blue Reebok Shoes and Red Reebok Shoes, and other is if they have functional dependency on each other, for example: Reebok Shoes and Reebok Socks.
For finding physically similar items, one can create a dictionary of attributes defining the product and do Jaccard similarity on those attributes.
For example:

Item A = {color: Blue, size: 10, material: Cotton, brand: Reebok}

Item B = {color: Red, size: 10, material: Cotton, brand: Reebok}

Thus, the intersection of sets would be number of attributes that match up, i.e.

Intersection(A,B) = {size, material, brand}

Union(A,B) = {color, size, material, brand}

Jaccard Index = 3/4 = 0.75

For finding behaviorally dependent items, one proxy that is generally used is more the two items are bought together in same session, more dependent they are for each other's functioning, thus more valuable recommendation. For this setting, one can create a matrix of products a user buys in a single session. For m users and n products it would be sparse m X n matrix. If we read the same matrix column wise, that would be set of users who bought the item in a particular session.

Thus,

Item A = {Ua, Ub, Uc}

Item B = {Ub, Ud}

Jaccard Index = 1/4 = 0.25


Preprocessing of 3D CAD files for Keras Conv3D input I'd like to apply some machine learning on 3D CAD data. File format should ideally be mesh-based like STL. Keras offers 3D convolutional layers (https://keras.io/layers/convolutional/), so it can handle 3D (5D) input data. The problem I have is the preprocessing of the input data. There seem to exist integrated methods in Keras for image preprocessing (https://keras.io/preprocessing/image/). However that does only seem to apply to 2D images.

I plan to voxelize the STL files somehow, and would appreciate input on the process of voxelization and preprocessing the data to be suitable for Keras.

PS: I know there are CNNs which directly take mesh data. However I'd like to approach the voxelation way to compare performance and results.

I found a way that works now.

Keras Conv3D can be fed with a numpy array of the voxelization and the corresponding class of the file.

So first the files have to be voxelized with a tool like binvox.

Afterwards those files can be read into a numpy array with binvox-rw-py:

np.int32(binvox_rw.read_as_3d_array(f).data)

In combination with the classification, the files can be used as input for a Conv3D layer in Keras.

A complete example can be found here.


emission probability using hmmlearn package in python I am learning hmm and try to implement it in  Python hmmlearn package(http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples).

However I am not quite understand what the documentation says:

  Classes in this module include MultinomialHMM, GaussianHMM, and
  GMMHMM. They implement HMM with emission probabilities determined by
  multimomial distributions, Gaussian distributions and mixtures of
  Gaussian distributions.

Does this means when we try to estimate the model given an observation sequence, the  emission probability can only be these three kind? what if I want to type in the emission probability by meself?

According to that same documentation you provided, the way to use custom emission probabilities is:

"1.1.3. Implementing HMMs with custom emission probabilities
If you want to implement other emission probability (e.g. Poisson), you have to implement a new HMM class by inheriting the _BaseHMM and overriding the methods init, _compute_log_likelihood, _set and _get for additional parameters, _initialize_sufficient_statistics, _accumulate_sufficient_statistics and _do_mstep."

NOTE: The documentation pages are for hmm.learn version 0.1.* which is no longer the most recent one. That is, it seems that the documentation page available at any given time might not correspond to the most recent version of hmmlearn. For example, sample code for hmm  0.1.* does not run with hmm learn 0.20.


Are CNNs best approaches to find and count objects in images? I'm starting my first real data-science project, I made a reserach and want to ask if my approach is correct:

I HAVE:
600 photos of electronic components, hundreds of components in a single photograph, components shape and distance between them may vary between photographs but there is only 1 shape in a single photo. components shapes are squares, rectangles, ovals, "E"-shape and similar

I NEED: generic counter for components (no matter what is the type of elements on the photo)

PLANNED APPROACH: analyze each picture with openCV to gain predictions and then train CNN with Tensorflow and GPU (no previous experience).

is it the correct way of thinking?

For your situation, I would recommend using an object detection network to both locate and classify your components.  Although you do not need the location or classification data, you can still count the number of objects the network detects.  Moreover, this approach does not require any data science background because there are many tutorials on this subject and tons of out-of-the-box tools, such as tensorflow's object detection API.

You will need to annotate your data, but there are tons of programs to help you with that too.  Here's one I found quickly.

I suggest using faster-RCNN with the NAS feature extractor because, although it is slower than other CNN architectures, it is the most accurate.  Also, the network would probably perform better if you did not combine all the components into a single class.

The steps necessary to accomplish this task are essentially the following:

Install and setup tensorflow
Download faster RCNN NAS from the tensorflow detection model zoo
Annotate your data by drawing bounding boxes around the components and labeling each bounding box.
Train your model
After saving the trained model, you just need to setup a pipeline that takes an input image, feeds it to your network, and returns the total number of detections ignoring the bounding box and classification data.  Here's a great tutorial for deploying a trained model.


Path prediction using the historic path taken by person Suppose I have a dataset which has the columns date, time, latitude, longitude, place name visited by a user. I am wondering whether there is a way to write any algorithm to predict the path, the user would take at a particular time.

For example, I have the last two months date, time, latitude and longitude, can we predict the path the user would take at a particular date next month. I am not sure whether this is possible. But if any one can give me a idea of algorithm or any packages in R or python, it would be helpful. I can research on it.

I think we've covered this question before but I can't find my response... A lot of work has been done in this area. I don't know of any packages, but the basic idea is to discretize the search space. One way is to just to use the cellular grid, which makes sense if you have that information. Another is to first extract the "locations" from the location trace; i.e., places the users revisits and spends significant time in. Then you can apply well-known techniques such as HMMs to estimate the next discrete location from the recent past. Most of the papers I read on this subject are about a decade old, so they came out before the deep learning flurry. If they were revisiting the subject I'm sure I'd see papers using recurrent neural networks.

Here are some papers for inspiration:

Learning and Inferring Transportation Routines
A personal route prediction system based on trajectory data mining
A Comparison of First- and Second-Order HMMs in the Task of Predicting the Next Locations of Mobile Individuals
Using GPS to Learn Significant Locations and Predict Movement Across Multiple Users


What do you think of Data Science certifications? I've now seen two data science certification programs - the John Hopkins one available at Coursera and the Cloudera one.

I'm sure there are others out there.

The John Hopkins set of classes is focused on R as a toolset, but covers a range of topics:

R Programming
cleaning and obtaining data
Data Analysis
Reproducible Research
Statistical Inference
Regression Models
Machine Learning
Developing Data Products
And what looks to be a Project based completion task similar to Cloudera's Data Science Challenge

The Cloudera program looks thin on the surface, but looks to answer the two important questions - "Do you know the tools", "Can you apply the tools in the real world".  Their program consists of:

Introduction to Data Science
Data Science Essentials Exam
Data Science Challenge (a real world data science project scenario)

I am not looking for a recommendation on a program or a quality comparison.

I am curious about other certifications out there, the topics they cover, and how seriously DS certifications are viewed at this point by the community.

EDIT: These are all great answers.  I'm choosing the correct answer by votes.

As a former analytics manager and a current lead data scientist, I am very leery of the need for data science certificates.  The term data scientist is pretty vague and the field of data science is in it's infancy.  A certificates implies some sort of uniform standard which is just lacking in data science, it is still very much the wild west.

While a certificate is probably not going to hurt you, I think your time would be better spent developing the experience to know when to use a certain approach, and depth of understanding to be able to explain that approach to a non-technical audience.


Evaluation metric for Information retrieval system I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to
maximize Recall@100 and Mean Average Precision (MAP).
According to Introduction to Information Retrieval, Precision (P) is the fraction of retrieved documents that are relevant:

Recall (R) is the fraction of relevant documents that are retrieved:

I want to know how to come up with ground truth(relevancy label) if it's not available? In other words, if I want to calculate precision or recall for the Semantic product search and if we don't have relevancy label available for input product query. In that case, how researchers calculate precision and recall? or how do they generate it?

I want to know how to come up with ground truth(relevancy label) if it's not available?

There's simply no way to properly evaluate a system if nobody knows what the output is supposed to be. However there are ways to work around a lack of annotated data:

Ask a panel of annotators to grade the quality of the output on a sample. Disadvantage: if a relevant instance is never predicted, the annotators are unlikely to notice it.
Compare the output to a state of the art system. Disadvantage: the evaluated system can only be as good as the reference system, any error by the reference system is considered correct.
Generate artificial data with an automatic method. Disadvantage: the evaluation relies on the quality of the artificial data, so in theory one has to prove that the artificial data is as good as real data... which is usually harder than actually collecting real data.

In that case, how researchers calculate precision and recall? or how do they generate it?

They can't. It would be like grading an exam paper without knowing the correct answers.
This is why benchmark datasets are so important for the research community and are published as proper scientific contributions.


DQN - how is it possible to train separate outputs for each action? I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action.
In the paper it recommends using a different output for each action

We instead use an
architecture in which there is a separate output unit for each possible action, and
only the state representation is an input to the neural network. The outputs correspond
to the predicted Q-values of the individual actions for the input state.

Since we can only visit one action, we only know the loss for that action (ie. a single output). But as far as I'm aware, we need to have values for all of the outputs in order to train the network. What black magic can you use to get the other output values?
It seems like a bad idea to get the network to predict the other action-values and feed them back, as it would affect the optimizer. And if you tried to ignore the other outputs and train it as if there were only the one you were currently focused on, you would still affecting the others as they would share edges.

DQN Paper

The output of your network should be a Q value for every action in your action space (or at least available at the current state). Then you can use softmax or epsilon-greedy (or other strategies) to select the final action. The network will learn to predict which action should return the maximum reward from your current state. Also we update the network after we have collected a specific amount of experience and we use batches from that experience buffer in order to update the network. We do not feed back any value in the vanilla DQN (no recurrence).

There are many good implementations online that can help you understand the algorithm.


counter vector fit transform cosine similarity memory error count_matrix = count.fit_transform(off_data3['bag_of_words'])

I have count_matrix shape with

  count_matrix.shape
  (476147, 482824)

cosine_sim = cosine_similarity(count_matrix, count_matrix)

I think the matrix size is too big to cause this memory error

  --------------------------------------------------------------------------- MemoryError                               Traceback (most recent call
  last)  in

  ~/venv/lib/python3.6/site-packages/sklearn/metrics/pairwise.py in
  cosine_similarity(X, Y, dense_output)    1034     1035     K =
  safe_sparse_dot(X_normalized, Y_normalized.T,
  -> 1036                         dense_output=dense_output)    1037     1038     return K

  ~/venv/lib/python3.6/site-packages/sklearn/utils/extmath.py in
  safe_sparse_dot(a, b, dense_output)
      135     """
      136     if sparse.issparse(a) or sparse.issparse(b):
  --> 137         ret = a * b
      138         if dense_output and hasattr(ret, "toarray"):
      139             ret = ret.toarray()

  ~/venv/lib/python3.6/site-packages/scipy/sparse/base.py in
  mul(self, other)
      479             if self.shape[1] != other.shape[0]:
      480                 raise ValueError('dimension mismatch')
  --> 481             return self._mul_sparse_matrix(other)
      482
      483         # If it's a list or whatever, treat it like a matrix

  ~/venv/lib/python3.6/site-packages/scipy/sparse/compressed.py in
  _mul_sparse_matrix(self, other)
      514                                     maxval=nnz)
      515         indptr = np.asarray(indptr, dtype=idx_dtype)
  --> 516         indices = np.empty(nnz, dtype=idx_dtype)
      517         data = np.empty(nnz, dtype=upcast(self.dtype, other.dtype))
      518

  MemoryError:

Any tips to avoid this memory error when You have large matrix?

It's not clear to me what is your data and what you are trying to do with it, but from what I gather you are trying to calculate cosine similarity for each pair in a cartesian product, right?

If yes then you might want to use "blocking" to reduce the number of comparisons, see https://datascience.stackexchange.com/a/54582/64377.


Autoregression with multiple factors I am not sure if this is the right place to ask this question. Anyway, I am working on a Forecasting using spending data. Using autoregression, I am able to predict the following number decently well but I would like to improve this. I have two other factors that don't predict nearly as well as the lagged factor but could still be useful in the model.

My question is this: would it be valid to include those factors in the model even though I am using a lagged version of the predictor? Does an autoregression imply that the only factor to be used is the lagged version of the predicted value?

I can't show the data as it is sensitive and my question is more about how autoregression and forecasting work in general.

Thanks for any help.

The idea of adding external variables is not new. ARIMAX model (https://robjhyndman.com/hyndsight/arimax/) was specifically created to include them.

If you ask whether it is ok to include the external variables of some particular period then the answer is that you need to understand when they are available for prediction. If you predict a profit of the company at an end of a year based on today's stock price then it is ok. The profit and loss statement appear much later than the information is available in stock price. And stock price should have some info on expected profits based on the financial theory. If you want to predict tomorrow's stock price based on tomorrow's number of sun spots then it is a bad idea. The number of sun spots tomorrow is not available today.

And generally you can use any combination of lags for different variables. Taking today's stock price and last year's profit to predict this year's profit is totally valid and even has financial theory supporting the relationship.


Finding dominating attributes with in the clusters generated I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and give names to these clusters. But I want to map demographic features on these clusters to validate my clusters whether they make sense or not.

Now for example I have original data set of 100 customers with 90 females and 10 males.

I suppose find 4 clusters and each is of size 25. In Cluster No: 1,2 and 3, I have all males and no females. In Cluster No: 4, I have 15 males and 10 females.

How to map genders to these clusters to describe clusters as male or female dominating ?

Am I making sense here ?

One way would be:

Cluster data.
Within each cluster, count the number of males and females.
Label each cluster with the majority gender.

Repeat steps 2-3 with each demographic feature of iterest.


Recommendations to generate a training set of images We are about to start a series of experiments in physics which will generate 8.5 thousands of images per month. This is a study on microscopic material reactions so the images are not 'normal' objects commonly found in sets available online (cats, cars, bicycles, etc.). The objective of the study is to generate those images to document our other measurements, and the the 2ry objective is that those images can be used to train an algorithm and later use that algorithm to classify objects in the rest of the images (including future studies, this is very important). So, given that we still did not start to acquire the images, we have the opportunity to "make it right" (if this even exist) from the beginning. We intend to generate and use those images in the following few years to come.

I am new to the ML world, although I have some experience in image analysis. We still did not decide which algorithm/approach we will use. Could be YOLO, DeepLearn, TensorFlow, OpenCV, etc. In principle it would be the easiest possible for non specialists (we will rather apply ready-made solutions instead of writing our own algorithms).
Since both image acquisition and object detection/recognition solutions change so fast all the time, and assuming that the train set should be ideally very similar to the images I plan to analyze with the trained algorithm, I have a few concerns (below) for a long-term projects. Could you give me your opinion and suggestions about the weight of each one (maybe I’m worrying too much) and how to circumvent (if possible) some of them?

1) What happens if I start the project with a given camera (eg Nikon DSLR D3500, 24.2 MP, 6000x4000 px, Full HD ) and then I switch to another one quite different (eg Raspberry Pi camera V2, 8MP, 3280x2464 px, Full HD)? If the 1st images are aquired by camera 1, then I assume I will lose accuracy of the object recognition algorithm due to different image resolution, is that so? How bad is this? What should I do to minimize this? Can I eg mix images from both cameras into the training set to avoid large deviations of the accuracy? I suspect this must be an old and very commented problem in object recognition. If I generate a training set with camera A, then can the algorithm work well in images acquired with camera B? For example, would it help if I take images from camera A and lower the resolution to match that of camera B (or just lower the resolution, since I not necessarily always know which camera we will use in the future)?

2) I read that one can make substantial changes to the training set (posterize, flip, decrease color depth, blur, etc.) to broaden the application of the algorithm. Would this enhance the accuracy of the ML algorithm to make it robust against camera changes?

3) As mentioned above, we still did not decide which ML technique we'll be using. Is this a decision we need to make before starting the training set generation?

4) Any other suggestions of ‘good practices’ on generating the training set and prepare for future changes in image acquisition technology?

1) wouldn't recommend switching cameras, could consider some type of resolution mapping to help though.
2) Likely not, but resolution alterations likely would.
3) Deeplearning => require more data, use transfer learning so you can use a less though.
4) try to keep the data as similar to it would be in production and try to keep the images general to avoid bias. (example: image recognizer predicting any image w snow to be a wolf)


Predicting survival on Haberman's dataset I am currently involved in analyzing a particular dataset called Haberman Survival Dataset.
The dataset contains cases from a study that was conducted between 1958 and 1970 at the University of Chicago's Billings Hospital on the survival of patients who had undergone surgery for breast cancer.

Attribute Information:

Age of patient at time of operation (numerical)
Patient's year of operation (year - 1900, numerical)
Number of positive axillary nodes detected (numerical)
Survival status (class attribute),
 1 = the patient survived 5 years or longer,
 2 = the patient died within 5 year

For this dataset I have first processed/cleaned the dataset using R, and now I am planning to collect a sample from this population and then train the sample data (training set) with some suitable machine learning algorithm and build a suitable model (1-3 are predictor variables, 4 is the response variable), and then again collect the sample (test set) from the population and then apply the model that I have built to test set and predict the outcome.

What machine learning models can be used to achieve this objective?

You can use any classification algorithm. Read up on Logistic Regression and Support Vector Machines, if you have sufficient data, you can try neural networks. The threshold score is of extreme importance here for good precision and recall. So lookup, precision-recall curves and ROC curves which help in determining a reasonable threshold.


Relationship between two sets of dates My wife has seizures a few times a month and takes medication for the seizures once a month.  So I have two sets of dates - the dates of the seizures and the dates she got her medication.

I'm trying to determine whether there is any sort of pattern in the timing of the seizures vs. the medication.  I hope to get some clues as to whether the medication is helping, whether she needs it more often, etc.

I'd appreciate any suggestions as to how to analyze these two sets of dates for any pattern/relationship.  I'm not a data scientist, but I'm pretty good with Excel, so I'm hoping for something I can model in Excel.

Thanks!

A few simple ideas to observe whether there are any patterns:

Plotting the days with seizure/without seizure together with the days when medication is taken across time. It's quite general but it might show some patterns.
for each seizure, difference with the date of last seizure, i.e. number of days without a seizure. Plotting this value (simple plot of the number of days over time) might show whether there is any regularity and whether there is an evolution in the frequency.
for each seizure, difference with the date of last medication. Plotting the distribution of this number (i.e. how often it's 1,2,3,...) might show if there is a direct relation between the two, e.g. if the number of days is generally more than 10 then it's likely that the effect of the medication wears down within 10 days. The medication might also have a delayed effect, which would be visible in the distribution.
In case there is a long period of data available, you can calculate the average (for instance of days without seizure) over the last N days and plot the evolution of this average across time.

Probably you and your wife can think about other things which could be relevant. The general idea is to represent a potential parameter numerically and then observe how this value behaves.


What is the largest public wearable accelerometer dataset? I'm looking for public datasets of people wearing a device with an accelerometer (and potentially other sensors eg gyroscope or magnetometer).  What are some of the largest available datasets like that?

This has obvious applications to machine learning: a good dataset will help develop good models for activity recognition and event detection from sensor data, same as the effect ImageNet / COCO / YFCC100m datasets have had in the visual field. Sadly, I think the very large datasets in this field are all private.

For my purposes, I don't care what people are doing; a totally random sample of activity is okay (but a broader sample is better than a specific sample).  I also don't care where the device is (wrist, pocket etc) or whether it is a smartphone or some other device (watch, actigraph, IMU etc). Finally, I don't care if the data is labelled / annotated.

I do however want the largest possible size: as many different people as possible, and as many total hours of recording as possible.

What I've found so far...

A few collections of datasets: https://arxiv.org/pdf/1707.03502.pdf and http://mobilize.stanford.edu/data-sources/

Some specific datasets:

UK Biobank has recordings from 100k people x 24 hours each.  I believe these are actigraph (1-minute resolution) not raw accelerometer data (can anyone confirm?).  It is also not open.
NHANES 2003 7k people x 7 days each.  Definitely actigraph, no raw accelerometer data.
LTMM  71 people x 72 hours each
PAMAP2 9 people x ~1 hour each
MHEALTH 10 people x ~15 minutes each
UCI-Smartphone

You can check these additional ones:

HAR: http://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones
It is collected by attaching a smart-phone and it has accelerometer and gyroscope data.
It has 30 participants conducting 6 different activities. The acceleration is at 50 Hz and they are using 3D accelerometers.

USCHAD: https://dl.acm.org/citation.cfm?doid=2370216.2370438 Have a look at this paper. Here you will find the data as well.

It has 14 subjects (7 male, 7 female) and there are 12 different activities. Additionally, it has accelerometers and gyroscopes, and it has a ground truth labeling by an observer standing nearby. They are sampled on 100Hz. Again they are using 3D accelerometers.

I would strongly suggest feature-engineering by defining time-windows where you segment the time-series data. After defining the small windows of a few seconds, you can have one activity per window. Then you can easily extract some time-dependent features such as mean, correlation, acceleration, orientation, etc.. Also, Fast Fourier transform on these small windows can be applied so you can extract many frequency domain features like energy, coherence, etc.


Confused about polynomial regression with multiple variables I'm trying to create a multivariable polynomial regression model from scratch but I'm getting kind of confused by how to structure it.
So, I have an array of feature vectors such that each vector can be displayed like so:
[height, weight, age]

I know with multivariable linear regression I would create an algorithm like so:
y=B0+B1*x0+...Bn*xn
Where x0 would be the first element of each in the feature vector.
So for multiple variable polynomial regression would it go something like this:
y = B0+B1*x0+B2*x1**2+...Bn*Xn**d
Where d is the degree of the polynomial. Apologies if this is painstakingly obvious and formatted badly, I'm just a small bit lost.

Not exactly,
Polynomial regression means that the dataset is not linear and we have to transform it to a specific polynomial degree based on the dataset, so that we may map the Linear model
 Decide a polynomial degree first, let's say 2
$y=b_0+b_1*x_0^2+b_2*x_1^2+...b_n*x_n^2$
If we want to add feature interaction,
$y=b_0+b_1*x_0^2+b_2*x_1^2+b_3*x_0*x_1....$
Basically, you have to implement this Class of Scikit-Learn
PolynomialFeatures


How shouild I save model weights in Pytorch? ### Create / load model

# Faster - RCNN Model - pretrained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
num_classes = 2

# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features

# replace the pre-trained head with a new one
model.roi_heads.box_predictor =  FastRCNNPredictor(in_features, num_classes)

Training the model doesn't use model.fit() function, it uses cycles.
    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

How can I save the model?

PyTorch has a state_dict which stores the state of the model (in this case, the neural network) at any point in time. Saving it would involve dumping those states into a file which is easily done with:
torch.save(model.state_dict(), PATH)

When reloading the model, remember to first create the model class with its default weights and load the state dict from the file.
Here's a link to saving/loading pytorch modules (https://pytorch.org/tutorials/beginner/saving_loading_models.html).


Reshape features in rows intro columns python? My features are arranged in rows, Can anyone tell me please how can I transform them into columns with theirs corresponding times and values??
I mean each features has a set of corresponding times and values ..
a view example of   dataset is :

Feature                      TIME                     Value
    YAM_B55P113BDPMN  2015-06-14T04:05:18.000Z            0
    YAM_B55P113BDPMN  2015-06-14T04:06:18.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:10:08.000Z            0
    YAM_B55P113BDPMN  2015-06-14T04:13:18.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:26:28.000Z          Bad
    YAM_B55P113BDPMN  2015-06-14T04:31:48.000Z            0
    YAM_B55P113BDPMN  2015-06-14T05:07:28.0
        .
.......
   YAM_LIK13.SET.TTCI  2017-05-28T03:36:22.000Z          42
  YAM_LIK13.SET.TTCI  2017-05-28T07:23:31.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-28T11:30:12.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-29T03:16:42.000Z           42
  YAM_LIK13.SET.TTCI  2017-05-29T06:56:12.000Z           42

Reflecting the DataFrame over its main diagonal by writing rows as columns and vice-versa is known as "Transpose". if your pandas dataframe is named "df" then

df_transposed = df.T

Check out this
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.transpose.html


Use machine learning to predict next schedule meeting for sales officers I have a project with data of sales field officers who visit their customers and enter the progress details. Visit can be an order or any kind of customer interaction.

Let's say one sales guy has around 1000 customers, It's only natural that he might skip some customers which can result in loss overtime. So I have the data for the visits done by him since the customer was onboarded. What model should I use to check the old frequency of visits done by him and factors which are stated below so that I can say like:

"Do you want to get in touch with "this" customer. ?"

So basically by checking his previous interaction rate, In case if he forgot to visit, I need to recommend like that.

Data points are like:

date/time - date when visit done
remarks - what was discussed
there is a status which is related to internal
and some other customer details.

So what model should I use or which technique do you think is best suited for this problem. I'm fairly new in machine learning and kind of learning it by doing.

What I understood is, since your sales person skipped the customer meeting you want to display details or send a notification like:

  Last date of conversation
  Discussion points and so on.

such that sales person decide her/his priority based upon the importance of discussion.

If so, I'm not sure how much it's possible with ML. But one thing is possible, where you can trained RNN(Recurrent Neural Network) based time series model. This model will predict the next meeting time, which is based upon the pattern of historical meeting data.

In that way you can book sales person calendar along with the last meeting Key-Discussion points or send an update in case sales person hasn't attended that person.


How to give a higher importance to certain features in a (k-means) clustering model? I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher importance for these dummy variables because multiple dummy variables represent one categorical variable.

For example, I have a categorical variable Airport that will result in multiple dummy variables: LAX, JFK, MIA and BOS. Now suppose I also have a numeric Temperature variable. I also scale all variables to be between 0 and 1. Now my Airport variable seems to be 4 times more important than the Temperature variable, and the clusters will be mostly based on the Airport variable.

My problem is that I want all variables to have the same importance. Is there a way to do this? I was thinking of scaling the variables in a different way but I don't know how to scale them in order to give them the same importance.

You cannot really use k-means clustering if your data contains categorical variables since k-means uses Euclidian distance which will not make a lot of sense with categorical variables. Check out the answers to this similar question.

You can use the following rules for performing clustering with k-means or one of its derivates:

If your data contains only metric variables:

Scale the data and use k-means (R) (Python).

If your data contains only categorical variables:

Use k-modes (R) (Python).

If your data contains categorical and metric variables:

Scale the metric variables and use k-prototypes (R) (Python).


Maximum Entropy Policy Gradient Derivation I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the derivation on Maximum Entropy Policy Gradients (Section 4.1)

Note that in the above derivation, the term H(q(thetha(at|st))) should have been log (qthetha(at|st)), and that log refers to log base e (i.e. natural logarithm). In the first line of gradient, it should have been r(st,at) - log(qthetha(at|st)).
In particular, I cannot understand how the second summation term from t'=t to t'=T in the second line arises in the derivation. I managed to derive the formula by expanding the definition of expectation, but the result that I got do not have the second summation term. Can anyone give me ideas on where this second summation term comes from mathematically?

Writing the objective in a different way may be helpful.
If we only focus on the reward part and ignore the entropy part since the focus in the question is about the second summation, the original objective is
$$
J(\theta)=\sum_{t=1}^T\mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}[r(s_t,a_t)]
$$
If we look at it from the perspective of the full trajectory $\tau$, just like the ELBO in the paper Eq.19, the objective is
$$
J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^Tr(s_t,a_t)\right]
$$
Then
$$
\begin{aligned}
\nabla_\theta J(\theta)&=\nabla_\theta\sum_\tau q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\sum_\tau \nabla_\theta q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\sum_\tau q_\theta(\tau)\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^Tr(s_t,a_t)\right]
\end{aligned}
$$
Since $q_\theta(\tau)=q(s_1)\prod_{t=1}^T q(s_{t+1}|s_t|a_t) q_\theta(a_t|s_t)$,
$$
\nabla_\theta \log q_\theta(\tau)=\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t)
$$
Then
$$
\nabla_\theta J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t=1}^Tr(s_t,a_t)\right]
$$
That is where the second summation comes from.
Since policy at time $t'$ cannot affect reward at time $t$ when $t<t'$ (causality),
$$
\begin{aligned}
\nabla_\theta J(\theta)&=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]\\
&=\sum_{t=1}^T \mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}\left[\nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]
\end{aligned}
$$


Replacing null with average in pyspark I have a problem during upsampling operation in PySpark. My dataframe is:

df_upsampled.show()
+-------------------+------------------+
|                 et|           average|
+-------------------+------------------+
|2018-08-15 00:10:00| 4.165999948978424|
|2018-08-15 00:15:00|              null|
|2018-08-15 00:20:00|3.6580000072717667|
|2018-08-15 00:25:00|              null|
|2018-08-15 00:30:00|0.9999999925494194|

What I want to do is that by using Spark functions, replace the nulls in the "sum" column with the mean value of the previous and next variable in the "sum" column.

Wherever there is a null in column "sum", it should be replaced with the mean of the previous and next value in the same column "sum".

In this case, first null should be replaced by (4.16599 + 3.658)/2 = 3.91 and so on for the rest nulls..

What would be a good way to do this?


Is the k-center problem part of machine learning or computational geometry? I am currently working especially with the k-center problem, which is e.g. used to determine optimal locations for k warehouses. This is done by defining k circles that cover a given set of points (here for example inhabitants), and at the same time, the radii of the circles should be minimal. The centers of these k circles would then be the optimal locations for the warehouses.

The question arises whether this method belongs to machine learning or computational geometry. Since it is a clustering method I would actually think that it belongs to machine learning, more specifically to unsupervised learning. On the other hand, there are many algorithms to solve this problem which are based on computational geometry.

How about your opinion?

Where is the "learning"? Where is the "training", and "becoming better"?

It's an optimization problem, but optimization is everywhere.

Fits computational geometry much better than ML.

But as a rough guideline, check which books discuss this well. Can you name a book that belongs into machine learning that discusses this is sufficient depth?


Definition of Type 1 and Type 2 Errors in Classification Why are the type 1 and type 2 errors as defined in bankruptcy prediction, different from type 1 and type 2 errors based on confusion matrix?

In bankruptcy literature:

Type 1 error: predicting a bankrupt company as a nonbankrupt one.

Type 2 error: predicting a nonbankrupt company as a bankrupt one.

In confusion matrix:

Type 1 error: predicting a negative case (nonbankrupt company) as a negative (bankrupt) one.

Type 2 error: predicting a positive case (bankrupt company) as a negative (nonbankrupt) one.

Thanks in advance.

What you said is true. Both articles look contradicting and arrive at opposite Type 1 and Type 2 errors.

I have had the same confusion several times.

Let me try to tell you what i concluded, though i am not 100% sure that my conclusion is right.

Definition of Type 1 error depends on False Positive.
Definition of Type 2 error depends on False Negative.

Above two statements are always correct.

Then why there is confusion?

Confusion arises by the definition of Null Hypothesis and Alternate Hypothesis functions.

In statistics generally Null Hypothesis is the one that is mostly true and everybody accepts and doesn't contain much information and Alternate Hypothesis is the one that unsettles status quo and if true is some big news.

So, in your banking example you can consider

Null Hypothesis = Bank is Normal (Solvent)
Alternate Hypothesis = Bank is Bankrupt

So, if this is the definition, then bankruptcy literature is correct.

But, in general in Machine Learning/Data Science we don't define two hypothesis, we just say our problem definition as predicting if a bank is bankrupt or not. In this scenario, our

True Positive is Bank is Bankrupt and we predicted it as Bankrupt.
False Positive is Bank is Solvent and we predicted it as Bankrupt.
False Negative is Bank is Bankrupt and we predicted it as Solvent.

So, again by the above statements, definition by confusion matrix is also right.

Then, why contradicting errors??
Answer is just the difference in the way we define hypothesis.

In banking literature, they defined true as solvent bank and false as bankrupt bank.
In confusion matrix, we assumed true as bankrupt bank and false as solvent bank.

Have a look at the following for more detailed explanations.

https://www.khanacademy.org/math/ap-statistics/tests-significance-ap/error-probabilities-power/v/introduction-to-type-i-and-type-ii-errors

https://www.abtasty.com/blog/type-1-and-type-2-errors/


How to create a Time Series Training Dataset with variable sequence length I have time series data with variable sequence lengths.
So something like:

date        value label
2020-01-01  2     0     # first input time series
2020-01-02  1     0     # first input time series
2020-01-03  1     0     # first input time series
2020-01-01  3     1     # second input time series
2020-01-03  1     1     # second input time series

how is it possible to create a training dataset (numpy arrays) of shape [samples, time_steps, n_features] when time_steps is not consistent?

Additional Info: The model that is going to be trained is an LSTM which is capable to handle variable input lengths.

I solved it the following way:

zero padding all time series that are smaller than the longest one
adding a Masking()layer with masking_value=0. which ignores all zero values before feeding the network.

The part of the model with the Masking layer looks like the following:

model = keras.Sequential()
model.add(layers.Masking(mask_value=0., input_shape=(None, 1)))
model.add(layers.LSTM(100))

Additional Info: Implementing the Masking Layer for a Model with two separate Inputs was kind of unconvinient with a the Keras Functional API, therefore i implemented that part of the model with the Keras Sequential API and connected it to the rest of the model which is implemented with the Functional API.


Is CRF suitable for multi-words Named Entity Recognition? I've a problem where I should create a custom NER by using sklearn CRF. In the official tutorial, they are using CoNLL2002 corpus is available in NLTK where the entities are represented with a single word but in my problem, an entity can be formed with multiple words ex: United States of America, Cinema at Miami, etc.
Can CRF handle this?

Absolutely. If you look at the training tutorial, it implies that this isn't an issue at all. When using multi-word entities, you typically need to use a IOB or BILUO tagging schemes, which helps your model train better.
But from a mathematical perspective, there aren't any restrictions for a CRF, as CRF models the likelihood of particular sequences/transitions. Often, people set restrictions for particular transitions if you know in advance that they aren't possible. But by default, all transitions are allowed. In sklearn-crf, allowing all transitions is done by using the all_possible_transitions=True argument.


Pattern Recognition, Bishop - (Linear) Discriminant Functions 4.1 Please refer "Pattern Recognition and Machine Learning" - Bishop, page 182.

I am struggling to visualize the intuition behind equations 4.6 & 4.7. I am presenting my understanding of section 4.1.1 using the diagram:

Pls. Note: I have used $x_{\perp}$ and $x_{p}$ interchangeably.

Equations 4.6, 4.7 from book: $$\mathbf{x} = \mathbf{x_{\perp}} + \textit{r}\mathbf{\frac{w}{\Vert{w}\Vert}} \tag {4.6}$$ Muiltiplying both sides of this result by $\mathbf{w^{T}}$ and adding $w_{0}$, and making use of $y(\mathbf{x}) = \mathbf{w^{T}x} + w_{0}$ and $y(\mathbf{x_{\perp}}) = \mathbf{w^{T}x_{\perp}} + w_{0} = 0$, we have $$r = \frac{y(\mathbf{x})}{\Vert{\mathbf{w}}\Vert} \tag{4.7}$$

  Questions:

  Is $y(\mathbf{x})$ the (orthogonal) projection of $(\mathbf{w^{T}x} + w_{0})$ along $\mathbf{w}$, the weight vector?
  Are the lengths normalized to express them as multiples of unit vector $\mathbf{\frac{{w}}{\Vert{w}\Vert}}$. If so, can the distance
  $r = $$\frac{y(\mathbf{x})}{\Vert\mathbf{w}\Vert}$ exceed 1.
  Given that, $$ y(\mathbf{x}) = \mathbf{w^{T}x} + w_{0}$$ i.e. $y(\cdot)$ has two parts: $\textit{orthogonal component above/ below}$
  decision boundary, and the $\textit{bias}$. And so, I'm calculating
  $y(\cdot)$ as:

  $$y(\mathbf{x}) = \mathbf{\frac{w^{T}x}{\Vert{w}\Vert}} + \frac{w_{0}}{\Vert\mathbf{w}\Vert}$$ while the book gets it as:
  $$y(\mathbf{x}) = {\frac{y(\mathbf{x})}{\Vert\mathbf{w}\Vert}} + \frac{w_{0}}{\Vert\mathbf{w}\Vert}$$
  I am struggling to visualize how do we get the first term in the equation above (in book, eqn. 4.7)

Alternatively, presenting my doubt/argument w.r.t. to book eqns 4.6 & 4.7; by substituting  $r$ (eq. 4.7) into eq. 4.6 we get:
$$\mathbf{x} = \mathbf{x_{p}} + y(\mathbf{x}) \qquad {(\Vert{\mathbf{w}}\Vert^{2} = \mathbf{w})}$$

which again seems to be incorrect - by triangle rule of vector addition.

  Given the context, where am I losing track? Request your inputs.

No, $y(\mathbf{x})=\mathbf{w}^T\mathbf{x}+w_0$; it is a scalar.  The dot product $\mathbf{w}^T\mathbf{x}$ is $\|\mathbf{w}\|$ times the length of the projection of $\mathbf{x}$ onto $\mathbf{w}$.  $w_0$ in your figure would be negative, and has the property that $y(\mathbf{x})=0$ whenever $\mathbf{x}$ is on the decision boundary.
No normalization appears to be necessary.  Certainly $r$ can be arbitrarily large (either positive or negative), when $\mathbf{x}$ is far away from the decision boundary.
As mentioned previously, the first term is actually a length from the origin; the bias serves to shift this so that $y$ itself is the orthogonal (scalar) component [up to scaling by $\|w\|$...But that's what we're out to show when we pass from Eq(4.6) to (4.7)].

The text's approach is to decompose $\mathbf{x}$ into components relative to the decision boundary: $\mathbf{x}_{\perp}$ on the boundary, and something perpendicular to the boundary.  Being perpendicular to the boundary, it is in the direction of $\mathbf{w}$, but we don't know how far, so they introduce its length as the unknown $r$.  (There's some standard geometry stuff here that could also get us to the conclusion, but I'll explain their approach.)

Now, as mentioned before, $y$ is zero on the boundary, so they have $y(\mathbf{x}_{\perp})=0$.  And now, just to fill in some of the details of what they say,
$$\begin{align*}
\mathbf{x} &= \mathbf{x}_{\perp} + r \frac{\mathbf{w}}{\|\mathbf{w}\|}\\
\mathbf{w}^T\mathbf{x}+w_0 &= \mathbf{w}^T\mathbf{x}_{\perp}+w_0 + r \frac{ \mathbf{w}^T\mathbf{w} }{ \|\mathbf{w}\| } \\
y(\mathbf{x}) &= y(\mathbf{x}_{\perp}) + r \frac{\|\mathbf{w}\|^2}{\|\mathbf{w}\|} \\
y(\mathbf{x}) &= 0 + r \|\mathbf{w}\|,
\end{align*}$$
and so $r=y(\mathbf{x})/\|\mathbf{w}\|$ as desired.

[I'm not sure what you meant in your last few lines; at least some of it seems to have been a typo?  Feel free to follow up.]

EDIT: Regarding your addition about substituting $r$, you should get $$\mathbf{x}=\mathbf{x}_{\perp}+y(\mathbf{x})\frac{\mathbf{w}}{\|\mathbf{w}\|^2},$$
but $\|\mathbf{w}\|^2$ is not equal to $\mathbf{w}$; the former is a scalar, and the latter a vector!  Rewriting, we have
$$\mathbf{x}=\mathbf{x}_{\perp}+\frac{y(\mathbf{x})}{\|\mathbf{w}\|}\frac{\mathbf{w}}{\|\mathbf{w}\|}.$$
This now looks correct: from the origin, go to $\mathbf{x}_{\perp}$, then along the unit vector $\mathbf{w}/\|\mathbf{w}\|$ for a distance of $y(\mathbf{x})/\|\mathbf{w}\|$ (which, per your bubble3, is the correct distance).


How to access an embedding table that is too large to fully load into memory? I'm currently trying to find a way of loading/deserializing a .json file containing Flair word embeddings that is too large to fit in my RAM at once (>60GB .json with 32GB of RAM). My current code for loading the embedding is below.

def get_embedding_table(config):
    words_id2vec = json.load(open(config.words_id2vector_filename, 'r'))
    words_vectors = [0] * len(words_id2vec)
    for id, vec in words_id2vec.items():
        words_vectors[int(id)] = vec

    words_vectors.append(list(np.random.uniform(0, 1, config.embedding_dim)))
    words_embedding_table = tf.Variable(name='words_emb_table', initial_value=words_vectors, dtype=tf.float32)

The rest of the code that I am trying to reproduce with a different word embedding can be found here.

I wonder if it is somehow possible to access the embedding table without deserialization of the entire .json file, for example: by sequentially reading it, somehow splitting it, or reading it directly from my disk. I would greatly appreciate your input!

There are a couple of options:

Incrementally parse JSON with something like ijson. Munge and append each part to tf.Variable.
Reduce the precision of the numbers dtype=tf.float32 to
dtype=tf.bfloat16


fast ML algorithms for binary classification with (large+sparse) binary input data I'm sorry that this is so very broad, but as a non-ML scientist it feels to be almost impossible to keep up with recent developments (esp. in deep learning etc.). Hence, I'm asking for guidance on how to handle this specific use case:

The goal is to predict a binary output from ~50,000 binary input variables (the input data being rather sparse with about 1,000 1s on average). The training dataset includes several thousand (fairly balanced) labeled samples. I already have a non-ML solution to this giving good results but it is computationally expensive. Thus, my questions:

Which ML algorithms work well (i.e. train reasonably fast on a small HPC-cluster) on binary data of that scale.
Do they allow to extract information about the inputs (i.e. the magnitude of loadings of the individual binary variables).
How large are the performance advantages of having binary data? As opposed to using the 50k binary input variables I could run a PCA and use the first couple hundred PCs (it takes about 500 to recover 90% of the variance) for training/prediction. What would the advantages/caveats be?

The order of the input variables is not really "random", but their importance might be. Hence I think CNNs would not be the best idea, but are non-convoluting NNs even feasible at this scale? Additionally, it is usually only a few input variables that mostly decide the output, if that makes any difference in model selection.

I have worked with ML in the past, but this is several years ago and my theoretical knowledge is more than rusty. Also, the variety of NN architectures / frameworks etc. has exploded since then, hence I wanted to ask for some input before blindly trying out everything.

I mentioned in a comment that boosting might be a option. However, after second thoughts, I guess (as far as I can tell from the description of your data)  that you might be better off starting with logistic regression with regularization (lasso, elastic net, ridge). Why?

You want to learn „fast“ (not much tuning etc)
Your data tends to be of high dimension
You don‘t know what feature(s) are good predictors

This together makes me think, that 1) trying lasso would be the thing to start. If this fails, 2) go on with ridge.

Lasso can shrink the impact of features to zero (good in high dimensional data). However, this may lead to a situation in which „too much“ regularization happens. So you can try ridge, in which case features are „shrunken“, but will never become zero.

I don‘t know if you work with R or Python, but lasso/ridge is available in both. Make sure you find the right lambda (tuning parameter for regularization) by cross validation (this is not too expensive).

Here is a good R tutorial: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

For Python, there are also good tutorials around.

Some background can be found in „Introduction to statistical learning“, for which R and Python code is online. http://www-bcf.usc.edu/~gareth/ISL/


Different X, Y input shapes I'm creating a CNN model in keras to categorise data. There's 150 features (represented as columns), each of which has a label of 1 to 9. The issue is I'm confused about how to use this in Keras since there's uneaven numbers of features to labels, with the features being able to be shown with dimensions of 100, 150 and the labels with dimensions of 100,1. Help understanding how to properly train this data on a Keras model would be appreciated.

#Reshape data to add new dimension
X_train = X_train.reshape((100, 150, 1))
Y_train = X_train.reshape((100, 1, 1))
model = Sequential()
model.add(Conv1d(1, kernel_size=3, activation='relu', input_shape=(None, 1)))
model.add(Dense(1)) model.add(Activation('sigmoid'))
model.summary()
model.compile(loss='categorical_hinge', optimizer='adam', metrics=['accuracy'])
model.fit(x=X_train,y=Y_train, epochs=200, batch_size=20)

The current code works, but the Conv1d layer doesn't seem to change the accuracy of the model. How can I make the model get above the 20% accuracy it currently reaches and utilise the Conv1d layer?

try to add more conv layers, and also play a bit with hyperparameters to improve accuracy.
Currently, the learning rate is being used as default, try to tweak it from 0.01 to 0.0001.


Are all 110 million parameter in bert are trainable I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below?
By trainable I understand they are initialized with random weight and during pretraining these weights are backpropagated and updated.

All these parameters are trainable.
Note that in normal Transformers it is typical to have fixed (non-trainable) positional embeddings, but in BERT they are learned.
Note also the "pooler" component, which is an extra projection that was not mentioned in the paper, but which the authors commented on later.


Unsupervised clustering without of Data which is supposed to be on a linear function When I have a dataset where each datum has x and y,
and the (x,y) has a relation of one of y = a_i*x + b_i (i=1,2,...).

Is the process written below available? and which algorithm does it belong to?

The process is.....

I have many points (x, y).
The machine finds 2 linear functions which represents the points.
The machine eliminates points which are far from the 3 lines.

In this case, I think I put parameters the number of linear functions and criterion to judge if a point is on a line or not..

The first figure is my dataset.

I want to have a machine which accepts the number of line (3 in this case)
and finds 3 lines(as the second figure (the lines in the figure are just put for idea without computation)), and then finally suggests points which may belong to neither of them. (In this case, for example, (71.6, 22))

Should I, for instance, extend the k-means algorithm to achieve this procedure?///

I would rather look at mixture models. Or, if there is additional noise besides the lines, use computer vision algorithms, for example, some variation of the Hough transform.


How can I classify text considering word order, instead of just using a bag-of-words approach? I've made a Naive Bayes classifier that uses the bag-of-words technique to classify spam posts on a message board. It works, but I think I could get much better results if my models considered the word orderings and phrases. (ex: 'girls' and 'live' may not trigger a high spam score, even though 'live girls' is most likely junk). How can I build a model that takes word ordering into account?

I've considered storing n-grams (check-out-these, out-these-live, these-live-girls), but this seems to radically increase the size of the dictionary I keep score in and causes inconsistency as phrases with very similar wording but different order will slip through.

I'm not tied to Bayesian classification, but I'd like something that someone without a strong background in statistics could grok and implement.

There is a very simple hack to incorporate word order in an existing bag-of-words model implementation. Treat some of the phrases, such as the frequently occurring bi-grams (e.g. New York) as a unit, i.e. a single word instead of treating them as separate entities. This will ensure that "New York" is different from "York New". You could also define higher order word shingles such as for n=3,4 etc.

You could use the Lucene ShingleFilter to decompose your document text into shingles as a pre-processing step and then apply the classifier on this decomposed text.

import java.io.*;
import org.apache.lucene.analysis.core.*;
import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.shingle.ShingleFilter;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.util.*;
import org.apache.lucene.analysis.util.*;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.charfilter.*;
import org.apache.lucene.analysis.core.WhitespaceTokenizer;

class TestAnalyzer extends Analyzer {

    TestAnalyzer() {
        super();
    }

    protected TokenStreamComponents createComponents( String fieldName, Reader reader ) {
        String token;
        TokenStream result = null;

        Tokenizer source = new WhitespaceTokenizer( Version.LUCENE_CURRENT, reader );
        result = new ShingleFilter(source, 2, 2);

        return new TokenStreamComponents( source, result );

    }
}

public class LuceneTest {

    public static void main(String[] args) throws Exception {

        TestAnalyzer analyzer = new TestAnalyzer();

        try {
            TokenStream stream = analyzer.tokenStream("field", new StringReader("This is a sample sentence."));
            CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);

            stream.reset();

            // print all tokens until stream is exhausted
            while (stream.incrementToken()) {
                System.out.println(termAtt.toString());
            }

            stream.end();
            stream.close();
         }
         catch (Exception ex) {
             ex.printStackTrace();
         }

    }
}


High / low resources language : what does it mean? In NLP, languages are often referred as low resource or high resource.

What these terms mean ?

High resource languages are languages for which many data resources exist, making possible the development of machine-learning based systems for these languages. English is by far the most well resourced language. West-Europe languages are quite well covered, as well as Japanese and Chinese. Naturally low-resource languages are the opposite, that is languages with none or very few resources available. Unsurprisingly many languages from Africa and other poorest parts of the world are low-resources. But this is also the case for some extinct or near-extinct languages and many local dialects. There are actually many languages which are mostly oral, for which very few written resources exist (let alone resources in electronic format); for some there are written documents but not even something as basic as a dictionary.

There are many different types of resources which are needed in order to train good language-based systems:

a high amount of raw text from various genres (type of documents), e.g. books, scientific papers, emails, social media content, etc.
lexical, syntactic and semantic resources such as dictionaries, dependency tree corpora, semantic databases (e.g. WordNet), etc.
task-specific resources such as parallel corpora for machine translation, various kinds of annotated text (e.g. with part-of-speech tags, named entities, etc.)

Many types of language resources are costly to produce, this is why the economic inequalities between countries/languages are reflected in the amount (or absence) of language resources. The Universal Dependencies project is an interesting effort to fill this gap.


How to delete a row if a values in a column is not NaN In the given code it displays only the rows without NaN values but i want only the rows with NaN values in a provided column everything has to be removed

df1 = df1[pd.notnull(df1['column_name'])]

If I am correct, you are looking to keep only the NaN values. You can use pd.isnull() to check which rows are NaN and then only keep those. Try this:

df1 = df1[pd.isnull(df1['column_name'])]


building a prediction model using CNN I have an input array  X, which is of the shape (38000,32,1); the output array Y is of (38000,1), the element of Y can be 0 or an numerical value, such as 0.040139 or 1.075341, or some other numerical values, etc. A typical vector in X looks like as follows

I would like to build an model to predict Y based on X. This is the model, two convolutional layer, followed by max-pooling, then one convolutional layer, one pooling, then a fully connected layer. The detailed architecture is as follows.

outX = conv_filter(20, 50, strides=1, activation='relu')(inputs)
outX = conv_filter(5, 60, activation='relu')(outX)
 outX = MaxPooling1D(2)(outX)
outX = conv_filter(2, 70, activation='relu')(outX)
outX = MaxPooling1D(2)(outX)

outX = Dense(300, activation='relu')(outX)
outX = Flatten()(outX)
predictions = Dense(1,activation='linear')(outX)
model = Model(inputs=[inputs],outputs=predictions)

The loss function is 'mean_squared_error'.The result does not look very good. One of the reason I can think is about the numerical values for the input vector, which tends to be very small. Can it cause any numerical issue? Are there any suggestions to pre-process them? Besides, are there any improvement can be made on the network architecture? Any suggestions would be highly appreciated.

If you haven't figured out what's going on already then you could try a couple of things.

1) Since you mentioned that your input values are very small, you could try normalizing them. I agree that very small (in your case, really small) values might mess up the weight updation. When you do, make sure you normalize first and then do training-testing split

2) Can you tell me how your loss looks during training? Do you see overfitting? If you do, did you use dropout? Did you try L2 regularization on the weights? (only use them on weights, not on biases)

There could be a million things going on and a lot of room for improvement may be there in the network, but I need to have a look at how your loss decreases as the training proceeds. Also, you need to elaborate on how bad your results look? (I assume that your training is quite successful but your testing error is high?). Also , if you're not bound to use CNN, did you try out with an ANN with a few hidden layers and see how that model does?


Will I be able to fetch a Data Analyst job if I learn everything in this image? India

Asking such a question because I am interested learn everything myself and I am confused about what to read.
And I am interested by reading books, will that suffice? or should I compulsorily attend a course and get certificate(because some say I might require certificate compulsorily)??
Please can somebody throw some light on this.(Please do if you are an Indian based Data Analyst or Data Scientist)
Thank you.

**I AM A COMMERCE GRADUATE AND INTERESTED IN DATA SCIENCE

I don't think someone can learn about everything in the image and be good at all of them. Especially when you get out of school !

You need to master:

Fundamentals
Statistics
Programming
Machine Learning
Visualization

The rest depends on what you want to specialize into.


Polynomial Regression Not Overfitting as Expected I am trying to implement polynomial regression in Python and have it mostly working, but it is not overfitting enough for higher degree polynomials. This is a weird thing to be upset about, I know, but it may point to an underlying problem with my implementation.
In this case I'm using a toy example where I'm sampling uniformly in the region $[0,1]$ and setting output values to $\sin(2\pi x) + \mathcal{N}(0, 0.3)$, i.e. just sin with added Gaussian noise with variance $0.3$.
I am using the normal equation form rather than gradient descent, so solutions should be optimal least-squares solutions. By Lagrange's interpolation theorem, we should have a polynomial of degree n fits n + 1 points exactly. However, for 10 samples and a 9th-degree polynomial (or even higher degrees), my implementation does not fit every point exactly.

You'll notice in my code that I'm not representing all the values of x in the graph exactly (i.e., I may be off after the ten-thousandths place), but the effect should not be as noticeable as it is if that is causing it. Because I'm sampling the x values uniformly in $[0, 1]$, I could have two overlapping x values within my error region which would cause some possible poor fitting in the graph, but this is extremely unlikely for low sample sizes (see my own question and solution to this problem here )
    import numpy as np
    import scipy as sp
    import matplotlib.pyplot as plt
    import math
    import random
    plt.rcParams['figure.figsize'] = [10, 5]

    def sample(n_samples):
        ''' Samples uniformly from x axis and returns y values given by sin(2pi)
        with Gaussian noise'''

        x = np.random.uniform(0, 1, size=n_samples)
        y = np.sin(2*np.pi*x) + np.random.normal(0, .3, n_samples)

        return [x,y]

    def poly_regress(data, n_param):
        ''' Implementation of polynomial regression with n powers. Uses normal
        equation rather than gradient descent.

        data: 2 by n matrix with data[0] the x values and data[1] the y values.
        n_param: degree of returned polynomial
        '''

        coeff = np.zeros(n_param) # Coefficient vector initialization

        # Calculate the vandermonde matrix of input x
        vand_x = np.vander(data[0],N = n_param + 1, increasing=True)

        # Computes the coefficient vector via the normal equation
        coeff = np.linalg.pinv(vand_x.T @ vand_x) @ (vand_x.T @ data[1])

        # This is for plotting purposes. If we wanted to predict a particular x
        # or range of values, then we'd use an almost identical form.
        x = np.arange(0,1, .0001)
        y = 0
        for i in range(n_param + 1):
            y += coeff[i]*np.power(x, i)

        # plots the true value of x. This only works for this toy example.
        true_x = np.arange(0,1, .01)
        true_y = np.sin(2*np.pi*true_x)

        plt.plot(true_x, true_y, color='g', label= "True Curve")
        plt.plot(x, y, color='b', label="Prediction")
        plt.plot(data[0], data[1], 'r+', label="samples")

        plt.ylim(-2,2)
        plt.legend()
        plt.show()
        return [x,y]

Any help is greatly appreciated. This is just base case testing before extending to more general forms of regression, so I want it to be functioning ideally at this point.
Cubic case:

High degree case with no noise:

With similar results for lower degree polynomials.

Good news is that your code seems to be theoretically correct.
Bad news is that the numpy.linalg.pinv has numerical stability problem when fitting polynomials of high order. The problem is driven by decreasing determinant of the AT*A matrix.
Alternative approach would be to use numpy.linalg.lstsq which tend to provide slightly better results.
However, according to my tests with your sample data (shown below), even numpy.linalg.lstsq cannot consistently fit the points exactly as soon as number of points exceeds 10.
For example for only 15 points (and polynomial of degree 14) the determinant of AT*A decreases to ca 10E-150 making the matrix almost singular.
In the figure shown below I plot the log10 of determinant as a function of average distance between x-points across 1000 experiments: the pattern is very clear, the smaller the average distance the smaller the determinant. It appears plausible: the closer the x-points the more linearly dependent become the equations.
However, in practice, we are often interested in cases when the order of polynomials is significantly lower than the number of observations. In such cases the results are more numerically stable.

import numpy as np
import matplotlib.pyplot as plt
import math
import random
plt.rcParams['figure.figsize'] = [10, 5]
#*************************************************************
# added random seed for reproducibility
np.random.seed(123456)
#*************************************************************
##############################################################
def sample(n_samples):
    x = np.random.uniform(0, 1, size=n_samples)
    y = np.sin(2*np.pi*x) + np.random.normal(0, 0.3, n_samples)

    return [x,y]
##############################################################
def poly_regress(data, n_param):
    #coeff = np.zeros(n_param) # Coefficient vector initialization
    # Calculate the vandermonde matrix of input x
    vand_x = np.vander(data[0],N = n_param + 1, increasing=True)
    # Computes the coefficient vector via the normal equation
    coeff_pinv = np.linalg.pinv(vand_x.T @ vand_x) @ (vand_x.T @ data[1])
    #*************************************************************
    # added alternative more precise approach
    coeff_lstsq = np.linalg.lstsq(vand_x, data[1])[0]
    #*************************************************************
    pred_pinv = 0
    pred_lstsq = 0
    for i in range(n_param + 1):
       pred_pinv += coeff_pinv[i]*np.power(data[0], i)
       pred_lstsq += coeff_lstsq[i]*np.power(data[0], i)
    return np.array([
        np.average(np.diff(np.sort(data[0]))), np.linalg.det(vand_x.T @ vand_x),
        np.average(abs(pred_pinv - data[1])), np.average(abs(pred_lstsq - data[1]))])
##############################################################
# evaluate results
for num_pts in [3, 5, 15]:
    iter = 1000
    res = np.zeros(iter*4)
    res.shape = (iter,4)
    poly_order = num_pts - 1
    for ii in range(iter):
        # this function returns:
        # [0] average distance between x points: np.average(np.diff(np.sort(data[0])))
        # [1] determinant of the A^T * A matrix: np.linalg.det(vand_x.T @ vand_x)
        # [2] MAE of linalg.pinv-based approach
        # [3] MAE of linalg.lstq-based approach
        res[ii] = poly_regress(sample(num_pts), poly_order)
    plt.plot(res.T[0], np.log10(abs(res.T[1])), 'x', color='tab:grey', label= "log10(abs(Det))")
    plt.xlabel("avg(diff(sort(data[0])))")
    plt.legend()
    plt.title("Num data points: " + str(num_pts) + ", poly degree: " + str(poly_order) +
              ". Calculations repeated " + str(iter) + " times.")
    plt.show()
    plt.plot(res.T[0], (res.T[2]), 'rd', color='tab:red', label= "MAE linalg.pinv")
    plt.plot(res.T[0], (res.T[3]), 'rP', color='tab:blue', label= "MAE linalg.lstsq")
    plt.xlabel("avg(diff(sort(data[0])))")
    plt.legend()
    plt.title("Num data points: " + str(num_pts) + ", poly degree: " + str(poly_order) +
              ". Calculations repeated " + str(iter) + " times.")
    plt.show()


Can we cross-validate the neuron no. and hidden layer no. of a feed-forward neural network? Confused about tons of relevant info on the web, so I appreciate if anyone of you can clarify for me. I will use the model to binary classify for your reference.
As for the problem, let's say in a feed-forward network with hidden layer no. to be decided, I want to know how many neurons I should put in (each) hidden layer(s), so

What are the most commonly referred metrics to decide the no. of the neurons (in each hidden layer)? Recall, accuracy, precision or...?
Can we use some kind of cross validations (e.g. Stratified K-Fold) to testify the performance of the network with different no. of hidden neurons and different no. of hidden layers in terms of the above metrics?

Thanks a lot in advance.

Yes. validation is the best way to choose model hyperparameters such as neurons number.
Validation with neural net is not substantial different from validation with other models.
The metric should be chosen according to the task the network performs. The validation method should be chosen according to the amount of data, training time, certainty level you want to achieve and compute and time resources you have.


How to handle categorical features in K-means? I am working on clustering algorithms. I am working with titanic dataset. It contains 6 categorical features. I used k-means algorithm on this dataset. I am using label encoding for categorical features. But I found that categorical features should use euclidean distance. It should use Hamming distance. So, how to make k-means work finely on mixed features? I don't need other algorithm. I just want to work with k-means only on mixed features dataset.

Label encoding is not a good idea if the nature of categories are not ordinal (it is actually not my favorite anyways). Use one-hot encoding and see how it works. You may apply a feature extraction on top of it, e.g. PCA, to reduce the noise coming from sparsity. The other idea is to label categories by their fraction in the feature, for example:
[a,b,b,c,a,a] --> [3/6, 2/6, 2/6, 1/6, 3/6, 3/6]


how to do Time Based splitting of Amazon fine food reviews dataset I want to do time-based splitting on Amazon food reviews dataset (https://www.kaggle.com/snap/amazon-fine-food-reviews ). But I don't understand the time format and also how can I divide the data after it is sorted according to time

Those dates are on timestamp. try to convert them useing:

from datetime import datetime
datetime.fromtimestamp(1540574790)

where "1540574790" is a moment ago for example.. for more, check https://en.wikipedia.org/wiki/Unix_time


Entity Recognition in Stanford NLP using Python I am using Stanford Core NLP using Python.I have taken the code from here.
Following is the code :

from stanfordcorenlp import StanfordCoreNLP
import logging
import json

class StanfordNLP:
def __init__(self, host='http://localhost', port=9000):
    self.nlp = StanfordCoreNLP(host, port=port,
                               timeout=30000 , quiet=True, logging_level=logging.DEBUG)
    self.props = {
        'annotators': 'tokenize,ssplit,pos,lemma,ner,parse,depparse,dcoref,relation,sentiment',
        'pipelineLanguage': 'en',
        'outputFormat': 'json'
    }

def word_tokenize(self, sentence):
    return self.nlp.word_tokenize(sentence)

def pos(self, sentence):
    return self.nlp.pos_tag(sentence)

def ner(self, sentence):
    return self.nlp.ner(sentence)

def parse(self, sentence):
    return self.nlp.parse(sentence)

def dependency_parse(self, sentence):
    return self.nlp.dependency_parse(sentence)

def annotate(self, sentence):
    return json.loads(self.nlp.annotate(sentence, properties=self.props))

@staticmethod
def tokens_to_dict(_tokens):
    tokens = defaultdict(dict)
    for token in _tokens:
        tokens[int(token['index'])] = {
            'word': token['word'],
            'lemma': token['lemma'],
            'pos': token['pos'],
            'ner': token['ner']
        }
    return tokens

if __name__ == '__main__':
sNLP = StanfordNLP()
text = r'China on Wednesday issued a $50-billion list of U.S. goods  including soybeans and small aircraft for possible tariff hikes in an escalating technology dispute with Washington that companies worry could set back the global economic recovery.The country\'s tax agency gave no date for the 25 percent increase...'
ANNOTATE =  sNLP.annotate(text)
POS = sNLP.pos(text)
TOKENS = sNLP.word_tokenize(text)
NER = sNLP.ner(text)
PARSE = sNLP.parse(text)
DEP_PARSE = sNLP.dependency_parse(text)

I am only interested in Entity Recognition which is being saved in the variable NER. The command NER is giving the following result

The same thing if I run on Stanford Website, the output for NER is

There are 2 problems with my Python Code:

1. '$' and '50-billion' should be combined and named a single entity.
Similarly, I want '25' and 'percent' as a single entity as it is showing in the online stanford output.
2.  In my output, 'Washington' is shown as State and 'China' is shown as Country. I want both of them to be shown as 'Loc' as in the stanford website output. The possible solution to this problem lies in the documentation .

But I don't know which model am I using and how to change the model.

The Stanford CoreNLP released by the NLP research group at Stanford University. It offers Java-based modules for the solution of a range of basic NLP tasks like

POS tagging (parts of speech tagging)
NER (Name Entity Recognition)
Dependency Parsing, Sentiment Analysis etc.

Before doing all above task you should first setup stanford CoreNlp.

If you want to have clear picture about stanford coreNlp starting from setup core nlp for python, NER , POS to sentiment, you can have a look at below link.

Advanced Natural Language Processing with Stanford CoreNLP


K-modes clustering: Estimating which features were most impactful on clustering? I have entirely categorical data (survey results from users), so I've used k-modes clustering to better understand my users.

I'm not an expert at clustering methods at all. Is there a way to known way of estimating the importance of a feature (or combination of features) in deciding which cluster a user falls into?

It boils down to counting.

Compute which feature value has the highest probability of predicting a particular cluster. It's a straightforward application of Bayes' formula.


Substituting nan values with mean code for x in num_cols:
    imp = SimpleImputer(missing_values=np.nan, strategy='mean')
    imp.fit(np.array(ds[x]).reshape(-1,1))
    ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))

Here, you are substituting the missing values (nans) with something, it can be either the most frequent data, median, average(mean), whatever. mean modifies the nan values with the average of not-nan values.

for x in num_cols:
    imp = SimpleImputer(missing_values=np.nan, strategy='mean')
    imp.fit(np.array(ds[x]).reshape(-1,1))
    ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))

  imp = SimpleImputer(missing_values=np.nan, strategy='mean')

For transforming missing values we use SimpleImputer, here, the missing values are np.nan and we want to use the mean strategy for transforming these nans. Thereby, their values would be substituted by the average value of each feature.

  imp.fit(np.array(ds[x]).reshape(-1,1))

You have an $n * m$ (array (np.array(ds[x]))), by reshape(-1,1)) you convert it to shape $(m*n, 1)$ -1 here stands for vertical array and 1 stands for having one column.

Then you fit your transformer (imp) to your data (np.array(ds[x]).reshape(-1,1))

  ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))

Here, the transformed data would be substituted in ds[x]


What is the shape of conv3d and conv3d_transpose? I want to do a GAN with coloured pictures. This means I need a three dimensional input and therefore I like to use conv3d and conv3d_transpose.

Unfortunately in the TensorFlow documentation, I can't find any formula for the output shape. Can anyone give me a hint on how to find the shape of the function's results?

Although your input data is three dimensional, you have to use Conv2D for your task. I guess Conv3d is used for data with temporal characteristic, yours is just a simple picture. To illustrate why you should Conv2D, suppose your input image is 224 * 224 * 3 and you employ a Conv2D layer with 10 filters. You have to specify stride and padding in order to specify the output shape. You have to specify dimensions to illustrate the height and width of you filters, also known as kernels, filter size will affect the output size if you assign padding to 'valid'. Here, there is a point. Suppose you have specified the filter size a 10 * 10 filter, then if the input shape was 224 * 224 * 1, each filter would be of size 10 * 10 * 1 to fit the input area. Now that the input is of size 224 * 224 * 3 the size of each kernel is 10 * 10 * 3 to fit the input volume. Consider in all cases the output of each convolution operation, better to say cross correlation, is a scalar. For more information take a look at videos here and for your case I encourage you watching Convolution Over Volume.