Trained Tensorflow model performs poorly on inference I trained an image classification model using Keras with Tensorflow backend. The model got good accuracy on validation dataset as well as on the testing data, I save the entire model to .h5 format, here is my checkpoint callback.

checkpoint = ModelCheckpoint(model_name+".h5", monitor='val_acc', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1).

As I was hoping to use this model on Android so I refreeze the model to binary protobuf (.pb) using keras_to_tensorflow.

When performing inference using the model on mobile I noticed the model gives very wrong and random predictions. I have tried exploring other reasons why this could be the case like I found here still, it seems clear the issue is not with loading the images.

Also, performing inference with the converted model on Tensorflow Python still gives the same wrong/random predictions. Here is my code for performing inference in Python.

def model_predict( model_path, image_path, model_input, model_output, class_names ):

    with tf.Graph().as_default() as graph: # Set default graph as graph

        with tf.Session() as sess:
            # Load the graph in graph_def
            print("load graph")

            # We load the protobuf file from the disk and parse it to retrive the unserialized graph_drf
            with gfile.FastGFile(model_path,'rb') as f:

                print("Load Image...")
                # Read the image & get statstics
                np_image = Image.open(image_path)
                np_image = np.array(np_image).astype('float32')/255
                np_image = np.resize(np_image, (224, 224, 3))
                np_image = np.expand_dims(np_image, axis=0)

                # Set FCN graph to the default graph
                graph_def = tf.GraphDef()
                graph_def.ParseFromString(f.read())
                sess.graph.as_default()

                # Import a graph_def into the current default Graph (In this case, the weights are (typically) embedded in the graph)

                tf.import_graph_def(
                graph_def,
                input_map=None,
                return_elements=None,
                name="",
                op_dict=None,
                producer_op_list=None
                )

                # INFERENCE Here
                m_input = graph.get_tensor_by_name(model_input) # Input Tensor
                m_output = graph.get_tensor_by_name(model_output) # Output Tensor

                print ("Shape of input : ", tf.shape(m_input))
                #initialize_all_variables
                tf.global_variables_initializer()

                # Run model on single image
                Session_out = sess.run( m_output, feed_dict = {m_input : np_image} )

                print("Predicted class:", class_names[Session_out[0].argmax()] )

How do I perform inference using Tensorflow Python/Android with a save .pb model?

Others have suggested I save the session used for training and load them to Tensorflow when performing inference. If this is the case how do I load the saved session in Tensorflow android?

I am sure the model did not overfit, it performs very well when using Keras.

I believe the samples that you are using for inference belongs to a different distribution of data from training and test set. So, I would suggest you to verify if the above case is true and if it is then you should fine-tune your model for the examples that you are trying to carry out inference.


Are CNNs applicable on structured data? I can use CNN to classify MNIST images, but I don't know whether CNNs are applicable on iris data as well? If not, why?

CNNs are applicable wherever the input signal contains spatial information. For instance, you can think of the following list:

1-D signal: Recorded voices are examples of one-dimensional inputs where there are relations between adjacent entries of inputs. This means adjacent entries have patterns which are valuable for different tasks, such as classification. You can employ 1-D convolutional layers for these input signals.
2-D signal: Images are an example of this kind, albeit they may have different channels, like RGB. It's clear that adjacent pixels are roughly like each other, and other than that, the adjacent pixels share patterns which may be repeated over the entire image multiple times. Consequently, 2-D convolutional layers can be employed for these signals.
3-D signal: Video frames are examples of this kind. Other than the similarity you can find inside each frame, the different frames that exist one after another in the time axis can have similar meaningful patterns which are replicated. You can employ 3-D convolutional layers for these signal.

About structured data where you can find them as the rows of relational databases which each column belongs to a specified feature, it does not have any meaning to use convolutional layers. The reason is clear. They do not have any spatial information. The adjacent rows should not share a common concept otherwise they would be redundant. Moreover, they are not spatially related to each other. For structured data, people utilise dense layers.


What is the relationship between Markov Random Fields and Conditional Random Fields? In Neural networks [3.8] : Conditional random fields - Markov network by Hugo Larochelle it seems to me that a Markov Random Field is a special case of a CRF.

However, in the Wikipedia article Markov random field it says:

  One notable variant of a Markov random field is a conditional random field, in which each random variable may also be conditioned upon a set of global observations o.

This would mean that CRFs are a special case of MRFs.

Definitions

Markov Random Field

Again, according to Wikipedia

  Given an undirected graph $G=(V,E)$, a set of random variables $X = (X_v)_{v\in V}$ indexed by $V$  form a Markov random field with respect to $G$  if they satisfy the local Markov properties:

  Pairwise Markov property: Any two non-adjacent variables are conditionally independent given all other variables:
  $X_u \perp\!\!\!\perp X_v \mid X_{V \setminus \{u,v\}} \quad \text{if } \{u,v\} \notin E$

  Local Markov property: A variable is conditionally independent of all other variables given its neighbors:
  $X_v \perp\!\!\!\perp X_{V\setminus \operatorname{cl}(v)} \mid X_{\operatorname{ne}(v)}$
  where ${\textstyle \operatorname{ne}(v)}$ is the set of neighbors of $v$, and $\operatorname{cl}(v) = v \cup \operatorname{ne}(v)$ is the closed neighbourhood of $v$.

  Global Markov property: Any two subsets of variables are conditionally independent given a separating subset:
  $X_A \perp\!\!\!\perp X_B \mid X_S$
  where every path from a node in $A$ to a node in $B$ passes through $S$.

Please note if you know a citable source which gives a good definition.

Conditional Random Fields

According to Wikipedia:

  Lafferty, McCallum and Pereira[1] define a CRF on observations $\boldsymbol{X}$ and random variables $\boldsymbol{Y}$ as follows:

    Let $G = (V , E)$ be a graph such that

    $\boldsymbol{Y} = (\boldsymbol{Y}_v)_{v\in V}$, so that $\boldsymbol{Y}$ is indexed by the vertices of $G$. Then $(\boldsymbol{X}, \boldsymbol{Y})$ is a conditional random field when the random variables $\boldsymbol{Y}_v$, conditioned on $\boldsymbol{X}$, obey the Markov property with respect to the graph: $$p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \neq v) = p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \sim v)$$ where $\mathit{w} \sim v$ means that $w$ and $v$ are neighbors in G.

  What this means is that a CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets $\boldsymbol{X}$ and $\boldsymbol{Y}$, the observed and output variables, respectively; the conditional distribution $p(\boldsymbol{Y}|\boldsymbol{X})$ is then modeled.

Question

What is the relationship between Markov Random Fields and Conditional Random Fields?

Conditinal Random Fields (CRFs) are a special case of Markov Random Fields (MRFs).

  1.5.4 Conditional Random Field

  A Conditional Random Field (CRF) is a form of MRF that defines a posterior for
  variables x given data z, as with the hidden MRF above. Unlike the hidden MRF,
  however, the factorization into the data distribution P (x|z) and the prior P
  (x) is not made explicit [288]. This allows complex dependencies of x on z to
  be written directly in the posterior distribution, without the factorization
  being made explicit. (Given P (x|z), such factorizations always exist,
  however—infinitely many of them, in fact—so there is no suggestion that the CRF
  is more general than the hidden MRF, only that it may be more convenient to
  deal with.)

Source: Blake, Kohli and Rother: Markov random fields for vision and image processing. 2011.

  A conditional random field or CRF (Lafferty et al. 2001), sometimes a
  discriminative random field (Kumar and Hebert 2003), is just a version of an
  MRF where all the clique potentials are conditioned on input features:
  [...]

  The advantage of a CRF over an MRF is analogous to the advantage of a discriminative
  classifier over a generative classifier (see Section 8.6), namely, we don’t need to “waste resources”
  modeling things that we always observe. [...]

  The disadvantage of CRFs over MRFs is that they require labeled training data, and they are slower to train[...]

Source: Kevin P. Murphy: Machine Learning: A Probabilistic Perspective


How to load a csv file into [Pandas] dataframe if computer runs out of RAM? I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB.

I wanted to try some primitive batching but in order to one hot encode I need to find number of all unique values, which i can't do without loading data into a dataframe first.

Are there any other tools that I can use to attempt loading the file into a dataframe? Does Pyspark also have a limit of when it starts crashing? I know that its capable of breaking down operations into stages, does that help with RAM management or just execution?

As noted many times by the writers of pandas, the ideal size of memory for analyzing with pandas is around 5-10 times the load of data you are giving.

That being said, if you can afford to load data is chunks for pre-processing and dump only the columns that are needed, I recommended pandas.read_table option to load data in pieces (check option chunks).

pyspark as you mentioned might be good to go with. But there is dask which is built around python stack of pandas and numpy for distributed work loads which has support for pandas.DataFrame and numpy.array . But I never followed anyone who was successful in using it in production (or atleast mentioned that they had used it). May be some of the people here can vouch for it. There is another library called sframe to load the data way out of memory as the author suggested in one of his presentations. It keeps on serializing data onto disk.

Hope I summarized it well.


How can I calculate AUC from the ROC curve for the classification? Based TPR and FPR, I have generate ROC curve for my binary classification model. I do not know, how to calculate AUC value. I would be very help for me if you can help me to calculate AUC value.

Welcome to the community!

As you know, AUC is just the area under ROC curve. So the question is more about numerical methods as you have a set of points and you would like to calculate the area under it.

Riemannian Sum

Trivial solution. Simply make rectangles from points you have. The area of each rectangle is simply the product of edges. Then sum them up! You probably don't like it do you?!

Trapezoidal Method

After Riemannian, the simplest and most naive algorithm to do this. You simply have a set of points and you just calculate the trapezoidal area between each pair and sum them up like what you see in the figure below. It has the maximum computation error as it simplifies the problem a lot.

Simpson (1/3) Method

Much better when we are talking about curves! Let's keep it simple and to the point. You can model your function in every interval using a quadratic ($y=ax^2+bx+c$) and having 3 data points. Using your three data points, you can calculate $a$, $b$ and $c$. Then the area under curve is not that difficult, but we have a better solution! Trust me or not, the value of this integration is simply

$$\frac{b-a}{6} (f(a)+4\times f(m)+f(b))$$

where $(a,f(a))$ and $(b,f(b))$ are endpoints of interval and $(m,f(m))$ is the midpoint. See the image below from here to compare these methods.

Romberg Methods

Simpson and/or Trapezoidal methods can be recursively applied to achieve a more accurate calculation. It's called Romberg method. Accuracy of these methods were in the length of interval. Smaller intervals give more accurate integration. Romberg uses this fact to iteratively get closer to more accurate answer.

And of course tones of more algorithms to do that.

PS: You certainly have libraries and functions in different languages to calculate it for you. Scipy offers for Python for instance.

Hope it helps! Good Luck!


Why can't ARIMA model large lags and/or long range dependence? ARIMA cannot model large lags(obtained from autocorrelation plot) and long range dependency(hurst exponent H > 0.5). Why is it so?

An ARIMA model with order of differencing d equal to 1 can model a time series with a unit root, which is a form of long-range dependence. If d=0 one has an ARMA model, which cannot model long-range dependence unless the number or AR or MA parameters is very high. To model long range dependence in a parsimonious manner, look at fractionally integrated ARMA models, known as ARFIMA models.


How to scale or standardize data that is mostly 0 (ranges from 0-1)? I am relatively new to data science and big data munging in general. I currently have various columns of data that range from $0-1$, but most of the values in each column are $0$. The data represents certain attributes about a customer, including the proportion of a certain customer's purchases in a particular category (so that the sum of each of these proportions is $1$), in addition to other data like the number of visits, the time between visits, etc. A set of sample rows could look like:

columns = [other_stuff,'C1','C2','C3','C4','C5','C6']
row = [1945,0.45, 0, 0, 0, 0.3, 0.25]
another_row = [438,0, 0.24, 0, 0.01, 0.5, 0.25]

A sample histogram of one of the "proportional" variables looks like:

As of now, I am struggling to find ways to scale the data to make it usable for clustering with other data with much different units/orders of magnitude. Should I:

Scale all the other variables to a 0-1 range using a Min-Max method and keep the "proportional" variables the same
Scale all variables using the variables' means and standard deviations, even though the "proportional variables" clearly do not follow a normal distribution (also cannot apply a log transformation since most of the values are indeed 0)
Keep everything as is, but perform dimensionality reduction on all variables and perform clustering based off the principal components (if Principal Component Analysis is used)
None of the above; use a completely different set of algorithms/methods

I am currently using option 3 using sckit-learn and Python 3.7. If there are packages in R that could also help, please throw them my way. Thank you.

You don't specify, but assuming you are doing K-means clustering, the way the algorithm calculates similarity, or more technically distance, using Euclidean Distance, the distances on the dimension with the greatest magnitude will overwhelm distances of those where the values are orders of magnitude smaller.

To overcome this I would normalize the data.  I am not sure the method you use is such a big deal.  Use min-max if you are comfortable with it.  Be careful though because it is possible, if I remember correctly, in sk-learn to scale the whole data set in one go which doesn't fix the magnitude differences.  Scale each column individually or even possibly just the one that is vastly different if all the others are nominally between 0 and 1.


What is a good way to transform Cyclic Ordinal attributes? I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far.

One way I could think is to do transformation: min(h, 23-h)

Input: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]

Output: [0 1 2 3 4 5 6 7 8 9 10 11 11 10 9 8 7 6 5 4 3 2 1]

Is there any standard to handle such attributes?

Update: I will be using superviseed learning, to train random forest classifier!

The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sink with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24).

You need both variables or the proper movement through time is lost.  This is due to the fact that the derivative of either sin or cos changes in time where as the (x,y) position varies smoothly as it travels around the unit circle.

Finally, consider whether it is worthwhile to add a third feature to trace linear time, which can be constructed my hours (or minutes or seconds) from the start of the first record or a Unix time stamp or something similar.  These three features then provide proxies for both the cyclic and linear progression of time e.g. you can pull out cyclic phenomenon like sleep cycles in people's movement and also linear growth like population vs. time.

Hope this helps!

Adding some relevant example code that I generated for another answer:

Example of if being accomplished:

# Enable inline plotting
%matplotlib inline

#Import everything I need...

import numpy as np
import matplotlib as mp

import matplotlib.pyplot as plt
import pandas as pd

# Grab some random times from here: https://www.random.org/clock-times/
# put them into a csv.
from pandas import DataFrame, read_csv
df = read_csv('/Users/angus/Machine_Learning/ipython_notebooks/times.csv',delimiter=':')
df['hourfloat']=df.hour+df.minute/60.0
df['x']=np.sin(2.*np.pi*df.hourfloat/24.)
df['y']=np.cos(2.*np.pi*df.hourfloat/24.)

df

def kmeansshow(k,X):

    from sklearn import cluster
    from matplotlib import pyplot
    import numpy as np

    kmeans = cluster.KMeans(n_clusters=k)
    kmeans.fit(X)

    labels = kmeans.labels_
    centroids = kmeans.cluster_centers_
    #print centroids

    for i in range(k):
        # select only data observations with cluster label == i
        ds = X[np.where(labels==i)]
        # plot the data observations
        pyplot.plot(ds[:,0],ds[:,1],'o')
        # plot the centroids
        lines = pyplot.plot(centroids[i,0],centroids[i,1],'kx')
        # make the centroid x's bigger
        pyplot.setp(lines,ms=15.0)
        pyplot.setp(lines,mew=2.0)
    pyplot.show()
    return centroids

Now lets try it out:

kmeansshow(6,df[['x', 'y']].values)

You can just barely see that there are some after midnight times included with the before midnight green cluster.  Now lets reduce the number of clusters and show that before and after midnight can be connected in a single cluster in more detail:

kmeansshow(3,df[['x', 'y']].values)

See how the blue cluster contains times that are from before and after midnight that are clustered together in the same cluster...

QED!


Recommendation - item based vs user based I have one clarification -

First the definitions-

User-based: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.

Item-based: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed offline.

Now the question - in item based CF the similarity between items are tracked via user behavior - since user behavior is changing will it not impact the similarity between items?

It depends on what data you are using to calculate similarity between your items.

If you are using data from a user interaction with an item - like viewing a web page or buying an item - then each time there is a user interaction with an item, the similarity between it and other items will change. If you are doing this calculation offline, then your model won't take that change into account until the next time the data set is updated and your recommendations are recalculated.

However, if you are using item meta-data to calculate similarity, then user behavior won't make any difference. For example, if you took the number of shared tags between two items, or the number of shared inbound links, etc.

I have always used user browsing data to build content recommendation engines, but it is certainly possible to calculate similarity in many different ways. Have fun!


Is image sharpening a good idea for data augmentation? I'm training segmentation networks and while the dataset is somehow decent (~5k images) I wanted to augment it, so far I'm trying:

RandomFlip
RandomRotate
RandomBrightness changes
RandomShadows

Due to constraints of the problem I can't do random crops or shifts. Other than those augmentations I was looking into image sharpening, and was wondering if it could be a good candidate for dataset augmentation. I could find it in some web articles and many augmentation projects on Github, but I can't find any solid papers that refer to it as a possible augmentation technique. Does anyone have some experience/tips on the matter?

At first, I have to mention that $5k$ cannot be considered as a large dataset for training a deep neural network. Anyway, about the question. In general, yes you can, but you have to be aware of some points. Data augmentation can be helpful or it can damage your entire predictions. The reason for each is that whenever you utilize data augmentation, you are somehow changing and manipulating the distribution of your data in hand. Because neural networks are considered to deal with random processes which their behaviour is iid, your data samples should not be dependent on each other. There is also another perspective. Your training data should have the same distribution as your test data. If you are sure that while testing what you are going to face can be sharpened images, so go ahead and carry out that, but if you're sure that your sensors are placed in a noisy situation, something you can encounter in self-driving cars, you can be pretty much sure that your raw data is blurry, and it is almost impossible to have sharp images due to velocity unless you take a preprocessing step and after that you feed it to your network.


Any differences in regularisation in MLP between batch and individual updates? I have just learned about regularisation as an approach to control over-fitting, and I would like to incorporate the idea into a simple implementation of backpropagation and Multilayer perceptron (MLP) that I put together.

Currently to avoid over-fitting, I cross-validate and keep the network with best score so far on the validation set. This works OK, but adding regularisation would benefit me in that correct choice of the regularisation algorithm and parameter would make my network converge on a non-overfit model more systematically.

The formula I have for the update term (from Coursera ML course) is stated as a batch update e.g. for each weight, after summing all the applicable deltas for the whole training set from error propagation, an adjustment of lambda * current_weight is added as well before the combined delta is subtracted at the end of the batch, where lambda is the regularisation parameter.

My implementation of backpropagation uses per-item weight updates. I am concerned that I cannot just copy the batch approach, although it looks OK intuitively to me. Does a smaller regularisation term per item work just as well?

For instance lambda * current_weight / N where N is size of training set - at first glance this looks reasonable. I could not find anything on the subject though, and I wonder if that is because regularisation does not work as well with a per-item update, or even goes under a different name or altered formula.

Regularization is relevant in per-item learning as well. I would suggest to start with a basic validation approach for finding out lambda, whether you are doing batch or per-item learning. This is the easiest and safest approach. Try manually with a number of different values. e.g. 0.001. 0.003, 0.01, 0.03, 0.1 etc. and see how your validation set behaves. Later on you may automate this process by introducing a linear or local search method.

As a side note, I believe the value of lambda should be considered in relation to the updates of the parameter vector, rather than the training set size. For batch training you have one parameter update per dataset pass, while for online one update per sample (regardless of the training set size).

I recently stumbled upon this Crossvalidated Question, which seems quite similar to yours. There is a link to a paper about a new SGD algorithm, with some relevant content. It might be useful to take a look (especially pages 1742-1743).


What should I observe when choosing which optimizer suits my Deep Neural Network model? I have trained my neural network model with optimizers such as RMSProp, AdaGrad, Momentum, and Adam.

Currently, after running the code, I have printed out the Train and Test Accuracy of every epoch (50 in my case). However, I would like to know how should I determine which of these optimizers performs the best?

Does a higher train accuracy at the last epoch determine which is best or would a higher test accuracy do so? Also, I observed that when using the Momentum optimizer, the model train accuracy reached its' highest around 0.91 in the 16th epoch compared to the other optimizer.

Hence, would that conclude that the Momentum optimizer performs best in this case?

High training score is not an indication of model performance, high test score is. Also, a faster convergence to the same, or better test score is an indication of optimizer performance.

Therefore, if Momentum optimizer reaches a better test score faster, it definitely means that it is the best out of those tested optimizers.

As a side note, be careful about the choice of "score", for example using "accuracy" for imbalanced classes is not a good choice, since it equates %1 error in a class with 100 members to 10% error in a class with 10 members. If classes are equally important, i.e. 1% error is equally bad for all classes, macro-f1 and AUC would be better replacements.

An important note: when we use a score to select a hyper-parameter or an optimizer, final model will be affected, thus, that score would be called validation score, not test score.


Assign new point to a class using spectral clustering Say I used spectral clustering to cluster a data-set $D$ of points $X_0 - X_n$ into a number $C$ of clusters. How can I efficiently assign a new single point $X_{n+1}$ to his convenient cluster?

Do I have to do the classification from the beginning (destroy all the clusters and apply the algorithm to the data-set $X_0 - X_{n+1}$), or is there an optimized way to extend to the point $X_{n+1}$?

I think you do not, since I was able to find several papers that proposed algorithms for the same:

Incremental spectral clustering by efficiently updating the eigen-system
Incremental kernel spectral clustering for online learning of non-stationary data
A Fast Incremental Spectral Clustering for Large Data Sets

Such algorithms are called sequential, incremental, streaming, or online. Armed with this knowledge, you can find more papers on your own.


clustering people according to answers on survey Hi I am finding it hard to find online the best clustering algorithm for clustering people according to answers they gave on 20 question survey. There are four categories which each of these answers can fall into. I want to cluster the respondents according to their category answers, assuming it is multiple choice questions on the survey, with questions only having two options. If person A chose the same or similar answers to person B in a similar category then they should be clustered together.

What is the best algorithm? Sorry, I found it hard to do google searches for this question.

Maybe you can

assign numeric values to the response of each question (e.x. 0 disagree, 1 for neutral, 2 for agree) if there is a natural order in them,
perform one hot encoding for the rest where no such order exist (ex. country of origin)

and then try your luck with a classical algorithm like k-means or hierarchical clustering. There exist a lot of bibliography against that approach but maybe you can create clusters that make sense to your case.


Problems with accuracy.score sklearn I am learning python and trying myself out at mashine learning. I am reproducing a super simple example - based on the infamous iris dataset. Here it goes:

from sklearn import datasets
iris = datasets.load_iris()

X = iris.data
y = iris.target

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .5)

from sklearn import tree
nordan_tree = tree.DecisionTreeClassifier()

nordan_tree.fit(X_train, y_train)

from sklearn.metrics import accuracy_score

I get the following error message:

Traceback (most recent call last):
  File "tree3.py", line 17, in <module>
    print(accuracy_score(y_test, predictions))
NameError: name 'predictions' is not defined

I don't get it. As far as I understand, predictions is the vector containing all the predictions produced with  DecisionTreeClassifier? What am I doing wrong?

You have not defined the variable predictions anywhere. You will need to get them from your classifier somehow. You have fit your nordan_tree on your training data, now you can use the fitted nordan_tree to generate the predictions, for example like this:

predictions = nordan_tree.predict(X_test)

Then your line of:

print(accuracy_score(y_test, predictions))

should work.


Clustering for high dimensional data I am have a data set with 52 variables. Most of them have zeros, it resembles a sparse matrix. How can I cluster this kind of data and are there any special types of clustering? I am attaching pca plot

It doesn't require any special method. The algorithm of choice depends on your data if for instance Euclidean distance works for your data or not.

Generally, you can try Kmeans or other methods on your X or PCAs; but Hierarchical Clustering may be a good choice for visualizing the clusters for high dimensional data.

Please check here if you can read/write python code.


How to train ML algorithm with multiple values in target data? I am new to data science and machine learning and looking for some help.
I am trying to train a machine with following data set:

Here, L3 is the target variable.
As it can be seen that the target variable contains a set of possible values, I am wondering how I can train the machine with such a data.
In my test data, I might want to predict any of the one or more values given one value.
Example test case :

20    c     aa

'aa' is given value and I want to predict other values based on the values 20, c and 'aa'.
Is it possible to do what I am trying to achieve?

Thank you.

Because your test cases may have values from L3 as input, this strikes me as more of a recommender system; consider encoding each possible value in L3 as a new binary column (thinking of them as products that each row/user has purchased / expressed interest in), and look into different kinds of recommender systems to see what seems most appropriate for your data.

(I'm not too familiar with recommenders.  AIUI, these are all clustering algorithms, but with different notions of distance and focusing on either the products or the users.  sklearn does not appear to have any built-ins specifically for recommendations, but you could use it for an underlying model.  There are also some other Python packages specifically for recommenders, but I can't recommend any [see what I did there?].)


In CS231n lecture, can't the linear classifier be softmax itself? I am little bit confused on why the scoring function that is the $f(X,W)$ is chosen to be $W,X$ while they talk about Softmax and SVM loss in this.

Can't they take Softmax classifier or SVM classifier and then explain the losses?

Was there a particular need of taking the above mentioned scoring function?

The notation they're using is a bit funny. $f$ is just the dot product between the input and weights, $w^Tx_i$. The loss function is what differentiates those classifiers. Each loss function takes that dot product as an input and uses it in different ways. All of those loss functions compare the output of $f$ in some way with an associated $y$ value: you are computing your loss relative to a target variable to understand how well your model is performing, and then you can compute the gradient of the loss function to improve those predictions. If you ignore the loss function, you're just left with linear regression predictions (and even linear regression needs a loss function, MSE, to fit the weights) which aren't being evaluated relative to anything.


Keras custom layer using tensorflow function I was wondering if there is any other way to write my own Keras layer instead of inheritance way as given in their documentation? My colleague is more comfortable in tensorflow and he gave me a tensorflow function that does the job of the layer. Do I necessarily have to rewrite it as Keras' inherited layer?

you can overwrite tensorflow classes and add you function as layer as colleague mentioned but Keras is just high level API to tensorflow and that layer can be called from keras.


Machine Learning - Input Prepocessing - NLP email classification model So I created a model which classifies emails into different categories, just like a spam filter. I deployed the model as a webservice, no problem with that but I can’t get my head around how I would use it to predicht the output category of a new email. How do I preprocess the new email (subject and message body) to match the input format of the model/webservice ? The model I trained has about 1000 features, corresponding to the 1000 most frequent words in the training dataset. Do I vectorize the new email ? Do I just search for the features/words in the new email ?
There is something obvious I’m missing, I think.

I used python, sklearn and pandas/numpy to preprocess and train the model

So, from what I have gathered, you are asking how to preprocess the new (I suppose unobserved) emails, which do not appear in the training set.
In that case, you should convert your email text into 1000-dimensional vector, where each value corresponds to a particular feature value.

I am going to go with the basis that you are simply counting the number of times any of the most frequent 1000 words occur in a new email (let's call it $x^{(1)}_{test}$).

To convert the email into a vector form, here is one way of doing it:

import pandas as pd
import numpy as np

words = ["blah", "tea", "tetra", "pak"]

def vectorise_email(words, e_mail):

    """Vectorise email to a vector of word counts based on a list of words.

    :param words: (List of Strings) List of frequent words in training set
    :param e_mail: (String) E-mail string

    :return word_counts: (Numpy Array) containing counts of words based on words list.
    """

    e_mail = pd.DataFrame(e_mail.split())

    e_mail = e_mail[e_mail[0].isin(words)][0].value_counts()
    word_counts = np.zeros(len(words))

    for w_idx, word in enumerate(words):
        word_counts[w_idx] = e_mail.at[word]

    return word_counts

print(vectorise_email(words, "this is a blah tetra pak tea tea blah blahh"))

Here we firstly tokenise sentences into words (I used the standard string split method, but you could use nltk's tokenise methods [https://www.nltk.org/api/nltk.tokenize.html]). Then we convert this list into a pandas DataFrame to use the value_counts method (Ref: https://stackoverflow.com/questions/22391433/count-the-frequency-that-a-value-occurs-in-a-dataframe-column) to get the word counts for those words which appear in the word list.
We then complete the vectorisation process by mapping these counts into a Numpy Array where each element in the array corresponds to a particular word count in the input e-mail.

Hope that helps


XGBoost feature importance has all features but decision tree doesn't I have used XGBoost to train a model with 400 features. My understanding is that since the max_depth is default at only 6, and 2^6 < 400, not all features will end up in the tree.
How come when I output the feature importance chart, it shows every single feature with above 0 importance? The decision tree output clearly shows that not every feature has been used in the final tree.

XGBoost iteratively trains many trees (=boosting). So there is not only one tree.
For example you can plot a single tree, see: https://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/
A basic decision tree algorithm creates just one tree. If you apply pruning to the tree not all features would be present in the tree. The first split would be the one with the highest importance, ...


how to find parameters used in decision tree algorithm I use a machine learning algorithm, for example decision tree classifier similar to this:
from sklearn import tree
X = [[0, 0], [1, 1]]
Y = [0, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
clf.predict([[2., 2.]])

How to find out what parameters are used?

Just type clf after defining it; in your case it gives:

clf
# result:

DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                       max_depth=None, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, presort='deprecated',
                       random_state=None, splitter='best')

i.e. all arguments with their default values, since you did not specify anything in the definition clf = tree.DecisionTreeClassifier().
You can get the parameters of any algorithm in scikit-learn in a similar way.
Tested with scikit-learn v0.22.2
UPDATE
As Ben Reiniger correctly points out in the comment below, from scikit-learn v0.23 onwards, we need to set the display configuration first in order for this to work:
sklearn.set_config(print_changed_only=False)


Finding number of common elements between different columns of a DataFrame I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings.

I would like to find, for each column, what is the number of common elements present in the rest of the columns of the DataFrame.

For example: say I have a dataframe like:

       0      1      2        3
0    cat     ox   bull    horse
1   lion    dog    cat    zebra
2   bull     ox  horse    tiger
3  horse  eagle   bull  giraffe

Starting with first column, I need answer as 2,3 as 2 seems to have most number of common elements with the column 0 after that column 3 as it has comparatively less number of common elements.

You can build something like below. You can leverage set().intersection() to find the intersection between list. You need to loop one column with other columns.

You would Notice that I changed the values of df columns into a list in order to use a set

import pandas as pd

df = pd.DataFrame({'A': ['cat','lion','bull','horse'], 'B': ['ox','dog','ox','eagle'],
                   'C': ['bull','cat','horse','bull'],'D': ['horse','zebra','tiger','giraffe']})

df_out = pd.DataFrame()

for col in df.columns:
    # making a list of all column which has to be compared with col
    other_col = [x for x in df.columns if x!=col]
    for oCol in other_col:
        #using a set we can find a intersection between 2 list and count them
        count = len(set(df[col].values.tolist()).intersection(df[oCol].values.tolist()))
        #storing all count with their respective column in separate df
        df_out = df_out.append([(col,oCol,count)],sort=True,ignore_index=True)

df_out.columns = ['Column','Comparison','Count']

df_out.sort_values(by=['Column','Count'],ascending=[True,False],inplace=True)

Output would look something like this:

Column Comparison  Count
1   A   C   3
2   A   D   1
0   A   B   0
3   B   A   0
4   B   C   0
5   B   D   0
6   C   A   3
8   C   D   1
7   C   B   0
9   D   A   1
11  D   C   1
10  D   B   0


Change date in linear regression model Im trying to run a linear regression model with Years as the x variable and temperature on the y variable but I keep getting errors. I manage to run a regression model using the below code and also predicting for a future date 2040. But what Im struggling with is to change the data so its running from a specific date, 1950 instead of 1880 which is the original start of the data. How do you change the date range in a linear regression model?

sns.regplot('Year','No_Smoothing',data=df)

x=df.Year
y=df.No_Smoothing
np.array(x).reshape((-1,1))

from sklearn.linear_model import LinearRegression
reg =LinearRegression()
reg.fit(df[['Year']],df.No_Smoothing)
reg.predict([[2040]])

Slice your DataFrame accordingly, and then apply your previous routine (you may want to put this in a function) to this part of the data:

df2 = df.loc[df.Year >= 1950]


XGBoost - feature importance just depends on the location of the feature in the data I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data is by far most important and then second is second, etc.
It seems like a sign that the model is not working properly as its not really learning anything... any advice on what could be going wrong?
UPDATE:
Correlation Matrix
https://ibb.co/3shDJjD
Model Code:
params = {
  'subsample':0.5,
  'learning_rate': 0.3,
  'max_depth':8,
  'num_parallel_trees' : 20,
  'objective': 'reg:squarederror',
  'verbosity':0,
  }
watchlist = [(train, 'train'), (test, 'val')]
reg = xgb.train(params, train, num_boost_round=5, early_stopping_rounds=5, evals=watchlist)

Results:
[0] train-rmse:0.274535 val-rmse:0.27431
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 5 rounds.
[1] train-rmse:0.273472 val-rmse:0.273653
[2] train-rmse:0.272796 val-rmse:0.27341
[3] train-rmse:0.272318 val-rmse:0.27334
[4] train-rmse:0.271943 val-rmse:0.273346
[5] train-rmse:0.271604 val-rmse:0.273374
[6] train-rmse:0.271218 val-rmse:0.273442
[7] train-rmse:0.270927 val-rmse:0.273529
[8] train-rmse:0.270641 val-rmse:0.273561
Stopping. Best iteration:
[3] train-rmse:0.272318 val-rmse:0.27334

Feature importance (note that 0 and 1 are first). If I change the order of the columns in the xtrain, the feature importance will also change and first two columns will always be two most important features.
https://ibb.co/QcHwbNg

You can use this chunk of code to plot the feature importance of your data. It is also possible that your data is columned with decreasing importance.
from xgboost import XGBClassifier
from xgboost import plot_importance
from matplotlib import pyplot
# load data
X =
y =
model = XGBClassifier()
model.fit(X, y)
plot_importance(model)
pyplot.show()


Combining 'class_weight' with SMOTE This might sound a weird question, but I could not find enough details in sklearn documentation about 'class_weight'. Can we first oversample the dataset using SMOTE and then call the classifier with the 'class_weight' option? As my testing set is highly imbalanced, I want to penalize misclassifications for minority classes. Thank you!

I tried different classifiers using a combination of SMOTE and class_weight, the results are almost the same as using only the SMOTE approach, and this new config made almost no difference (which could be expected, following the logic behind the class_weigh approach).

PS: I have a pretty large dataset with multiple classes. This might result in different performance in different contexts.


Quasi-linearity in deep learning regression problems (sports betting) I’m attempting to build a sports betting model that aims to predict final scores for games.  I’ve had some promising early results for US college football just by using linear regression to form team rankings where each team’s coefficient represents how much it would win or lose by against an average team.

With football specifically, but perhaps all major sports, I think outcomes are very dependent on individual matchups, i.e., the specific strengths and weaknesses of one team versus the strengths and weaknesses of another team as teams will try to exploit relative weaknesses of their opponents.

Here’s a hypothetical set of matchups between the offense of one football team, Team A, and the defense of another team, Team B.

Team A Offensive Ratings:
  Passing: +7
  Rushing: -3

Team B Defensive Ratings:
  Passing: -3
  Rushing: +7

What this means is that Team A’s passing offense contributes an average of 7 points more to its final score than an average team, and its rushing offense contributes 3 points less than an average team.  In total, assuming that passing and rushing are the only two ways to score, Team A scores 4 more points per game than a hypothetical average team.  This same thinking would apply to defenses (i.e., the defensive rating also measures how a team’s defense contributes to its average margin of victory or loss).  In practice, it’s not as simple as looking at the nominal statistical outputs for each team since different teams play different levels of competition, but that’s a somewhat separate topic.

With the example above, it first appears that Team A’s offense is even with Team B’s defense because both are 4 points better than average, but that’s probably an oversimplification.  Given that Team A has a relative advantage in passing and a relative disadvantage in rushing, we can expect Team A to pass more than it rushes and perhaps pass more than it usually does due to Team B’s relative weakness in pass defense and relative strength in rushing defense.  The fact that teams and coaches will be at least somewhat rational in their decision making should, in theory, lead to some degree of non-linearity in final scores based on specific strength profiles of each team.

I’m just getting started with data science and have to admit that I’m yet to code a neural network (although I’ve read a lot about them).  This is fundamentally a regression problem, but most of what I’ve seen indicates that using neural nets for (linear) regression is largely done for efficiency purposes for very large datasets, which is not the case here.

Seemingly, I can create whatever activation functions I want, even with different functions within the same layers as long as I can find their derivatives.  Generally, I think the activation functions should respect linearity regarding the net ratings difference for each stat component, but the network needs to account for any non-linearities that exist as result of interactions between stats.

I realize this is very open ended.  I could design a large number of ad hoc activation functions and see what works and what doesn’t, but I’m wondering if there’s a more generic way to approach the problem.

Any suggestions are appreciated.

There are several principles in Data Science and Statistics which you will have to know before undertaking a project of any kind.

One of them is what is called parsimony principle (you could know him with other words) but it says that if you have two models with similar results, the simpler, with less parameters model should be chosen.

As you must have read, the neural networks have A LOT of parameters which of course might allow to learn your database (not generalize your model to not-seen-before data).

My recommendation is you use linear regression if you have a linear problem.

The neural networks can represent a linear problem too: Can neural network represent a relation that is a derivative?

But, if you want to do neural networks anyway, I will have to tell you that they work by trial and error. There are rules and suggested architectures, but you will have to find the best by yourself


Can a linear regression model without polynomial features overfit? I've read in some articles on the internet that linear regression can overfit. However is that possible when we are not using polynomial features? We are just plotting a line trough the data points when we have one feature or a plane when we have two features.

It sure can!
Throw in a bunch of predictors that have minimal or no predictive ability, and you’ll get parameter estimates that make those work. However, when you try it out of sample, your predictions will be awful.
set.seed(2020)

# Define sample size
#
N <- 1000

# Define number of parameters
#
p <- 750

# Simulate data
#
X <- matrix(rnorm(N*p), N, p)

# Define the parameter vector to be 1, 0, 0, ..., 0, 0
#
B <- rep(0, p)#c(1, rep(0, p-1))

# Simulate the error term
#
epsilon <- rnorm(N, 0, 10)

# Define the response variable as XB + epsilon
#
y <- X %*% B + epsilon

# Fit to 80% of the data
#
L <- lm(y[1:800]~., data=data.frame(X[1:800,]))

# Predict on the remaining 20%
#
preds <- predict.lm(L, data.frame(X[801:1000, ]))

# Show the tiny in-sample MSE and the gigantic out-of-sample MSE
#
sum((predict(L) - y[1:800])^2)/800
sum((preds - y[801:1000,])^2)/200

I get an in-sample MSE of $7.410227$ and an out-of-sample MSE of $1912.764$.
It is possible to simulate this hundreds of times to show that this wasn't just a fluke.
set.seed(2020)

# Define sample size
#
N <- 1000

# Define number of parameters
#
p <- 750

# Define number of simulations to do
#
R <- 250

# Simulate data
#
X <- matrix(rnorm(N*p), N, p)

# Define the parameter vector to be 1, 0, 0, ..., 0, 0
#
B <- c(1, rep(0, p-1))

in_sample <- out_of_sample <- rep(NA, R)

for (i in 1:R){

  if (i %% 50 == 0){print(paste(i/R*100, "% done"))}

  # Simulate the error term
  #
  epsilon <- rnorm(N, 0, 10)

  # Define the response variable as XB + epsilon
  #
  y <- X %*% B + epsilon

  # Fit to 80% of the data
  #
  L <- lm(y[1:800]~., data=data.frame(X[1:800,]))

  # Predict on the remaining 20%
  #
  preds <- predict.lm(L, data.frame(X[801:1000, ]))

  # Calculate the tiny in-sample MSE and the gigantic out-of-sample MSE
  #
  in_sample[i] <- sum((predict(L) - y[1:800])^2)/800
  out_of_sample[i] <- sum((preds - y[801:1000,])^2)/200
}

# Summarize results
#
boxplot(in_sample, out_of_sample, names=c("in-sample", "out-of-sample"), main="MSE")
summary(in_sample)
summary(out_of_sample)
summary(out_of_sample/in_sample)

The model has overfit badly every time.
In-sample MSE summary
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  3.039   5.184   6.069   6.081   7.029   9.800

Out-of-sample MSE summary
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  947.8  1291.6  1511.6  1567.0  1790.0  3161.6

Paired Ratio Summary (always (!) much larget than 1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  109.8   207.9   260.2   270.3   319.6   566.9


Why should re-sampling change the value of model's coefficients? I have the code below in python to create LinearRegression model. When I train the model with re-sampled data, I get different values for its coefficients. I can't understand why that happens. Can you help me in this please?

[Update]

I assume that resampling is the same as shuffling. And that means the order of the data is changed but not the data itself.
In the use case presented, the number of rows are are the same as I inspected it and as I understand the order of the data is changed.

Thanks!

from sklearn.linear_model import LinearRegression
from sklearn.utils import resample

model = LinearRegression(fit_intercept=False)

model.fit(X, y)
print('model.coef_',model.coef_)

model.fit(*resample(X, y))
print('model.coef_',model.coef_)

model.fit(*resample(X, y))
print('model.coef_',model.coef_)

In each sampling, your data is going to be different from the previous sampling. For each sampling, you are going to find the best line which describes your sample with the least error value. Consequently, for each sample you are going to find a model which may be different due to the fact that it reduces the cost for each sample.


Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent. I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian multiplied by the negative of the Jacobian, to determine the step-size.

Any help with the intuition behind using the inverse of the Hessian would be much appreciated.

It's not an intuition, it's mathematics. And more precisely, the quadratic expansion of a function.

Say you have $f(x)=f(x_n+\Delta x)=f(x_n)+f'(x_n)\Delta x + f''(x_n)\Delta x^2$ and you want to get to a point where $f'(x) = 0$, then you will use $\Delta x=-\frac{f'(x_n)}{f''(x_n)}$.

Generalizing to n dimension, $f''$ is the Hessian of the equation.

So of course this works perfectly for least squares, it will only "work" close to the solution for non quadratic functions.


How to improve regex while fetching record dynamically The data is as follows:

COL1    COL2

 12    :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh
 57    :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh

I am creating a third column field_info like:

 COL1  COL2                                                    field_info

 12   :402:agsh,hhjd,:45:hghghgruru,:12:fgh,ghgh,:22:hhhh      fgh,ghg
 57   :42:agshhhjd :57:hghgh,gruru:120:fghghgh :12:hhhhhh    hghgh,g

I am using a regex function as follows:

df.loc[:,'field_info']=df.col2.replace(regex=r'.*'+ df.col1.astype('str') +':(.{15}).*',value="\\1")

I have 2 columns col1 & col2. col1 has some value which I am searching in col2 dynamically and extracting the next 15 characters from that. However, it's taking a lot of time. Can anyone suggest a faster way of doing this?

Based on your sample data, I replicated it 50000 times and the result is as follows-

>>> df = pd.DataFrame({'COL1':[12 ,57],'COL2': [':402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh',':42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh']})

>>> for _ in range(50000):
        df = df.append({'COL1':12,'COL2': ':402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh'}, ignore_index = True)
        df = df.append({'COL1':57,'COL2': ':42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh'}, ignore_index = True)

>>> df.shape
(100002, 2)

Then I defined a custom function and applied to the columns-

>>> def somefunc(x,y):
        res = []
        for i in range(len(x)):
            ix = y[i].find(x[i]) + len(x[i])
            res.append(y[i][ix+1:ix+8])
        return res

>>> df['col3'] = somefunc(df['COL1'].astype(str),df['COL2'])
>>> df.head()
    COL1                                   COL2             col3
0    12  :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:2...  fgh,ghg
1    57  :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:...  hghgh,g
2    12  :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:2...  fgh,ghg
3    57  :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:...  hghgh,g

I did not use regex and this function took nearly 5 seconds to complete on 100000 rows.


Network Visualization - R code - Which Package is needed? I'm trying to create a network visualization to study the flights to and from a certain airport.

My data consists of 123000 rows in this format:

Origin_ID    Destination_ID    Frequency
  1726            3504           40000
  3504            4517              40
  5616            7205              38
               ...

I'm trying to create a image similar to this:

Basically, I want to be able to see the strength of the relationship (based on the frequency) and identify which combinations are most common in that airport.

Can it be created in R using this format? Or do you suggest any other open source platform to create this visualization?

The visNetwork R package is the best I've worked with.  It renders with the vis.js Javascript package right in your RStudio window.  Nodes and edge visualization is fully customizable and may be data-driven.  They implement click-functionality on your graphs, so you can rearrange them or highlight a node and its neighbors.  I've found people really enjoy this in presentations, and the highlight functionality is great for making static PowerPoint presentations or figures.


In first visit monte carlo are we assuming the environment is the same over episodes? Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general:

The first time a sate is visited:

increment N(s): N(s) = N(s) + 1
increment total state's return function by current episode's return so far S(s) = S(s) + G_t

State's value is estimated by mean return over many episodes:

V(s) = S(s) / N(s)

by law of large numbers, V(s)-->V_true(S) as N(S)-->inf

My question is - should the environment always behave the same over different episodes, or can it change randomly, and we sill have the true value over a large enough number of episodes?

Yes, an assumption in the MDP formulation in RL is stationarity: the dynamics of the environment do not change over time (within or across episodes).

Note that this almost always false in the real world, but this assumption simplifies matters, so we have it.


Clustering evaluation metrics with subquadratic time complexity It exists many evaluation metrics but often they are quadratic or more on number of data points preventing any application on massive data sets as RAND or Silhouette indexes.

For the moment i used :

External metrics

Normalized Mutual Information, NMI

Internal metrics

Davies Bouldin, which is in $O(n.c^2)$ with $c$ the number of clusters

I implemented both under Spark framework in Scala here for those who are interested but i would be happy to know if others exists, especially concerning internal metrics because even if Davies Bouldin could be a good evaluation metric, it is in my opinion, only with elliptic clusters.

Rand index is not more expensive than NMI, it's O(n) when implemented correctly.

There are like 40 internal evaluation indexes. I am fairly certain there are several that are not quadratic in runtime. Just go through the list, and document your findings for the next one with the same question.

Spark sucks for clustering. It can do k-means and GMM but both are very slow compared to other tools. They don't scale. The attempts at implementing DBSCAN for Spark are even worse. They were found to return incorrect results to crash, and to scale worse than single-core implementations such as ELKI (not to speak of parallel HPC implementations):

  Helmut Neukirchen.
  Survey and Performance Evaluation of DBSCAN Spatial Clustering Implementations for Big Data and High-Performance Computing Paradigms

It's just not what Spark is good at. And given all the "extensions" for it - that often have a very low quality - you quickly run into problems and don't know who is to blame and where to fix them.

Make sure to benchmark.


Why Orange "Predictions" and "Test & Score" produce different results on the same data? It is not very clear what is the difference between the following two schemes:

From help docs:

Test & Score widget: tests learning algorithms on data.
Predictions widget: shows models’ predictions on the data. Which model? Pretrained one? Where is it taken from?

Obviously the results of Confusion Matrix are different. So how should I train and test my model is not clear.

If can easily do this:

What's the point of a separate single Predictions widget connected to some learner? What does it actually test?

Perhaps I can elaborate a bit on this.

Test&Score is used for evaluating a model. You provide T&S with training data and the learner (e.g. Random Forest) and then the widget performs 10-fold cross-validation on training data, leaving out 1/10 of a data for testing. Results of all 10 fold are then combined into a single evaluation result.

Predictions, however, does not perform cross-validation. It doesn't test the data provided. Instead, Random Forest passes a model to Prediction, not a Learner. A model is built on train data. Predictions widget does not consider the real class, it uses the model from RF and applies it to the (new) data.

These are two different procedures, thus you are getting different results.


Plot 2D function: do I need to pass all the values to the function? I'm new with Octave and I need to plot this function:

% Computes the value of the Powell Sum benchmark function.
% SCORES = POWELLSUMFCN(X) computes the value of the Powell Sum function at
% point X. POWELLSUMFCN accepts a matrix of size M-by-N and returns a vetor
% SCORES of size M-by-1 in which each row contains the
% function value for the corresponding row of X.
%
% Author: Mazhar Ansari Ardeh
% Please forward any comments or bug reports to mazhar.ansari.ardeh at
% Google's e-mail service or feel free to kindly modify the repository.
function scores = powellsumfcn(x)
    n = size(x, 2);
    absx = abs(x);

    scores = 0;
    for i = 1:n
        scores = scores + (absx(:, i) .^ (i + 1));
    end
end

I have copied from BenchmarkFcns.

I have no idea about how to plot it.

Do I need to pass all the possible values to the function plot it?

How about defining a plot function like:

function meshfcn(fcn, x, y)
    [X, Y] = meshgrid(x, y);
    Z = arrayfun(@(xx, yy)(fcn([xx, yy])), X, Y);
    mesh(X, Y, Z);

and then use it:

x = -2:0.1:2; % the range of 'x' values to plot. change to your needs
y = -2:0.1:2; % the range of 'y' values to plot. change to your needs
fcn = @powellsumfcn; % the function needs to be on the path
meshfcn(fcn, x, y)


which is better : F1-score of 'N' in imbalanced data or 'N+3' in balanced data? I have a dataset with 4712 records. Label 1 is 1558(33%) and Label 0 is 3154 (67%)

a)  Currently when I run the model and analysis as is (without sampling techniques), I get an F1-score of 71-77. I chose F1-score and AUC score as the metric as my dataset is imbalanced. (Atleast that's what I felt looking at class proportion). My AUC also ranges between 80-83 for tree based models. Screenshot of models with imbalanced data is given below

b) When I under-sample the majority class, I get all the metrics like F1-score/accuracy/AUC above 80 but less than 85

screenshot below

Now I am not sure which one should I consider? I know it's all about trade-offs.

My objective is to avoid/minimize the misclassifications

Based on your experience in ML projects, what would you guys suggest?

Can someone enlighten me with some reasons why to choose one over the other?

Second.
why? you said you objective is to minimize misclassifications which equates to maximizing
F1 score, and you achieve that in inbalaced situations when you down-sample the majority class (one approach to maximize)
Potential Pitfall:
It could be that you just got rid of of some important infomration in the majority class and you are over-fitting on what is left.
In some cases 1500 datasamples would suffice, but I it could be that it doesn not have enough information. I would rather advise to upsample the smaller class, for example using ADASYN, SMOTe , knnSMOTE etc...
Only you can answer this question, by knowing some meta-information about your dataset, quality rules, are you expecting huge covariate-shift, is the dropped data really that different in distribution, etc...
Then do the analysis. If there is one thing you should always ask for is diverse and informative dataset, and more data will (almost) always beat better algorithm.


How can we interpret biplot? This is not a question as such but more likely to be verification (enhancement) of my current understanding. With the thought that it may help future visitors as well, I am taking liberty to make this post.

Question

Is the interpretation mentioned in attached image correct ?
As per my interpretation, Movie A doesn't seems to have much Action, nor Drama. What does that mean? Is it boring movie or what?
I would like to give some intuitive name to PC0 and PC1, what seems to be suitable name for it?

In short, Answer to your questions:

Yes
Movie A is near to center of the plot. Its the most balanced movie from your collection. (Biplot's center reflects the average of PCA scores: neither positive nor negative).
PC1 scores are positively correlated with with Romantic/ Comedy  and negatively with Action movies. PC0 is mostly negatively correlated with Drama movies.


what is init_score in lightGBM? in the tutorial of Boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I wonder what does the init_score mean? In the help page, it says, "init_score: initial score is the base prediction lightgbm will boost from ;". Another thing is what does boost mean in lightGBM?

Pardon me if I asked something not precisely. I am a newbie to lightGBM & machine learning seeking for help eagerly.
Thanks.

Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of models.

The init_score value represents the prediction that you are trying to correct with the first boosting iteration. By default this probably predicts the majority class for all examples or a randomly selected class, but you can input the prediction that was outputted by any other model here if you like.

The first tree that is learned by LightGBM will try to correct the errors of this initial prediction. In the example that you linked, they train a LightGBM model for one iteration, and use the score to continue training another LightGBM model. So in practice, what they have done in this example is kind of pointless because they could have just trained a single LightGBM from the start, but I suppose it is just to illustrate the concept and how it works.


Is it my Sales growth is depend on Commisions/Discounts? I prefer this model in R

We are capturing sales data by time series (Month by month). Some of items have commissions and some have Discounts and others have both commissions and discounts. Is it Commissions or Discounts or commissions + Discounts have impact on my sales growth? Or is it my sales are growing because of those commissions or discounts or discounts +commissions Can you suggest me best model to solve my use case? I am thinking multiple regression. But I want to double check with experts like you.

Thanks for your all your help

Sample Data set: (5 variables)

Year-Month -Product -Sales  -Commission -Discounts
2013-01 Milk    300 No  Yes
2013-02 Milk    400 No  Yes
2013-03 Milk    200 No  Yes
2013-04 Milk    150 No  Yes
2013-05 Milk    500 No  Yes
2013-01 Bread   800 Yes No
2013-02 Bread   879 Yes No
2013-03 Bread   790 Yes No
2013-04 Bread   459 Yes No
2013-05 Bread   600 Yes No
2013-01 Cheese  400 Yes Yes
2013-02 Cheese  350 Yes Yes
2013-03 Cheese  600 Yes Yes
2013-04 Cheese  590 Yes Yes
2013-05 Cheese  720 Yes Yes

Multiple regression sounds appropriate in this case. Real question is what variables to use. Definitely, you should include indicator variables commisions and discounts (and possibly their combination - commisions_and_discounts - as a separate variable). Date and time information may play important role as well, though treating them is a little bit harder. At the very least, it's worth to include year (because there may be a global trend), month and day_of_week (many patterns are repeated periodically).

Product type is a little bit more complicated. Obviously, some products will always have higher sales than others, so first idea is to include a set of product_type_x dummy variables. But product type may affect sales not additively, but instead multiplicatively, i.e. not as

sales ~ beta_0 + beta_1*product_type_x + ...

but instead as

sales ~ beta_1 * prodcut_type_x * (beta_0 + ...)

In this case better solution would be to create separate regression models for each product type.


Data visualization on three factors I have three factors,

Income(from source A)，continuous variable
Income(from source B), continuous variable
Happiness index，continuous variable

Suppose I have 500 samples. My goal is to show the influence of both Income(from source A) and Income(from source B) on Happiness index. Meanwhile, I want to show the distribution of Happiness index. I think a boxplot on Happiness index is a good choice. I can use color to represent Income(from source A), however, how can I represent another continuous variable, i.e, Income(from source B) here?

What you can do is discretize happiness index using cut() and assign labels to it.
For plotting then you can use groupby boxplot which will show effect of both income sources on a particular happiness index bin.
Refer GroupbyBoxplot
If you want distribution of happiness index then you can try:
Color Scales


How word embedding work for word similarity? I am trying to apply word2vec/doc2vec to find similar sentences. First consider word2vec for word similarity. What I understand is, CBOW can be used to find most suitable word given a context, whereas Skip-gram is used to find the context given some word, so in both cases, I am getting words that co-occur frequently. But how does it work to find similar words? My intuition is, since similar words tend to occur in similar contexts, the words similarity is actually measured from the similarity among contextual/co-occuring words. In the neural net, when the vector representation for some word at the hidden layer is passed through to the output layer, it outputs probabilities of co-occuring words. So, the co-occuring words influence the vectors of some words, and since similar words have similar set of co-occuring words, their vector representations are also similar. To find the similarity, we need to extract the hidden layer weights (or vectors) for each word and measure their similarities. Do I understand it correctly?

Finally, what is a good way to find tweet text (full sentence) similarity using word2vec/doc2vec?

I think you have it mostly correct.

Word embeddings can be summed up by: A word is known by the company it keeps. You either predict the word given the context or vice versa. In either case similarity of word vectors is similarity in terms of replaceability. i.e. if two words are similar one could replace the other in the same context. Note that this means that "hot" and "cold" are (or might be) similar within this context.

If you want to use word embeddings for a similarity measure of tweets there are a couple approaches you can take. One is to compute paragraph vectors (AKA doc2vec) on the corpus, treating each tweet as a separate document. (There are good examples of running doc2vec on Gensim on the web.) An alternate approach is to AVERAGE the individual word vectors from within each tweet, thus representing each document as an average of its word2vec vectors. There are a number of other issues involved in optimizing similarity on tweet text (normalizing text, etc) but that is a different topic.


Multi-label compute class weight - unhashable type Working in a multi-label classification problem with 13 possibles outputs in my neural network with Keras, sklearn, etc...

Each output can be an array like [0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1 ,0].

I have an imbalance dataset and i trying to apply compute_class_weight method, like:

class_weight = compute_class_weight('balanced', np.unique(Y_train), Y_train)

When i try to run my code, i got Unhashable Type: 'numpy.ndarray':

Traceback (most recent call last):
 File "main.py", line 115, in <module>
   train(dataset, labels)
 File "main.py", line 66, in train
   class_weight = compute_class_weight('balanced', np.unique(Y_train), Y_train)
 File "/home/python-env/env/lib/python3.6/site-packages/sklearn/utils/class_weight.py", line 41, in compute_class_weight
  if set(y) - set(classes):
  TypeError: unhashable type: 'numpy.ndarray'

I know that is because i working with arrays, already tried add some dict,

i.e.:

class_weight_dict = dict(enumerate(np.unique(y_train), class_weight))

Well, i don't know what to do, tried others strategies, but no success... Any ideas?

Thanks in advance!

You're seeing this error because your Y_train data is a 2d array, where compute_class_weights expects a 1d array.

compute_class_weights can be used for multiclass classifications, but apparently not multi-label problems like yours.

You could try using compute_sample_weight instead, which is slightly different but handles multi-label output problems such as this.


General programs/libraries for studying user search behavior? Are there any general open-source programs or libraries (e.g., a Python library) for analyzing user search behavior?  By "search behavior", I mean a user's interaction with a search engine, such as querying, clicking relevant results, and spending time on those results.  I'd like something with the following properties - it doesn't have to be all of them, but the more the merrier:

Models individual user behavior (aggregate and time-based)
Models group user behavior
Simulates individual user behavior, given a model
Is easily extensible (to accept data input formats, user models, document models, etc., that end-users define)

Links are a plus!

I know of few though they do not model user behavior, they just try to present best search results and compare to user ideal. lemurproject.org has books based on it though it is old and in java. For example the user activity recorder works with firefox pre big numbers versioning. That is something like 10 years ago. Maybe has a modeler but almost certainly out of date. Apache Lucene is simpler though made for bigger data sets and regularly updated. Terrier IR platform is similar.


Is there room to improve the model? if the train data accuracy is 99.8% but test data accuracy is 90%? I understand this is a wide question.

But there can be some suggestions.

I can try some methods which I do not know.

I think the model is already prefect on train data. But the test accuracy is a little low. So do you think there still can be some improvement on the deep model?

Your model is overfit: it learns patterns on the training data which are specific to the training data, hence the big difference in performance on the test data. This is bad, because it means that your model wrongly assumes that some patterns are good indication for the class even though it's just noise.

It's possible that by solving the overfitting problem the performance would improve. However nobody can give you any advice without knowing anything about your actual problem and data. Btw you don't explain why you use accuracy as evaluation measure, it might or might not be optimal for your problem.


What is the most used format to save data with type information I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R.

Currently, I am using csv files to save the data. Everybody supports csv. But csv does not support type information. This makes it sometimes cumbersome to load a csv file because I must check all the column types. This seems unnecessary because the SQL database already specifies the types of the columns.

I want to know if there is a broadly accepted file format to save data that also specifies the type of the columns.

Currently I am working with SQL databases, FME ETL'ing and R but I think this is an issue for every data tranfer.

Parquet and Avro both support data types (strings, integers, floats, etc). These are the primary file types used for "big data" projects, although your data doesn't have to be big. Apache Spark is able to read both with ease. Having said that, I'm a big fan of keeping data in a SQL database (e.g., MySQL or Postgres) because that is what they are built for. If you can't re-use the database you're pulling from, could you make your own database locally or on a separate server? I would try using a relational database until your data exceeds 50 GB (an arbitrarily "somewhat large" size), and then I would use Avro or Parquet.


How to convert model.h5 to model.pb? I wrote in google-collab to get the model using keras, but I have to do predictions in Visual Studio using tensorflow

I've search for a method converting models from keras .h5 to tensorflow .pb, but none of them worked as it should.

Keras does not include by itself any means to export a TensorFlow graph as a protocol buffers file, but you can do it using regular TensorFlow utilities. Here is a blog post explaining how to do it using the utility script freeze_graph.py included in TensorFlow, which is the "typical" way it is done.
However, I personally find a nuisance having to make a checkpoint and then run an external script to obtain a model, and instead prefer to do it from my own Python code, so I use a function like this:
def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
    """
    Freezes the state of a session into a pruned computation graph.

    Creates a new computation graph where variable nodes are replaced by
    constants taking their current value in the session. The new graph will be
    pruned so subgraphs that are not necessary to compute the requested
    outputs are removed.
    @param session The TensorFlow session to be frozen.
    @param keep_var_names A list of variable names that should not be frozen,
                          or None to freeze all the variables in the graph.
    @param output_names Names of the relevant graph outputs.
    @param clear_devices Remove the device directives from the graph for better portability.
    @return The frozen graph definition.
    """
    graph = session.graph
    with graph.as_default():
        freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))
        output_names = output_names or []
        output_names += [v.op.name for v in tf.global_variables()]
        input_graph_def = graph.as_graph_def()
        if clear_devices:
            for node in input_graph_def.node:
                node.device = ""
        frozen_graph = tf.graph_util.convert_variables_to_constants(
            session, input_graph_def, output_names, freeze_var_names)
        return frozen_graph

Which is inspired in the implementation of freeze_graph.py. The parameters are similar to the script too. session is the TensorFlow session object. keep_var_names is only needed if you want to keep some variable not frozen (e.g. for stateful models), so generally not. output_names is a list with the names of the operations that produce the outputs that you want. clear_devices just removes any device directives to make the graph more portable. So, for a typical Keras model with one output, you would do something like:
from keras import backend as K

# Create, compile and train model...

frozen_graph = freeze_session(K.get_session(),
                              output_names=[out.op.name for out in model.outputs])

Then you can write the graph to a file as usual with tf.train.write_graph:
tf.train.write_graph(frozen_graph, "some_directory", "my_model.pb", as_text=False)

Credits: jdehesa
You can find more at:

How to export Keras .h5 to tensorflow .pb?

Keras as a simplified interface to TensorFlow: tutorial


Neural network for MNIST: very low accuracy I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:

# install
Pkg.add("MNIST");
using MNIST

# training data
X,y = traindata();
m = size(X, 2);
inputLayerSize = size(X,1);
hiddenLayerSize = 300;
outputLayerSize = 10;

# representing each output as an array of size of the output layer
eyeY = eye(outputLayerSize);
intY = [convert(Int64,i)+1 for i in y];
Y = zeros(outputLayerSize, m);
for i = 1:m
    Y[:,i] = eyeY[:,intY[i],];
end

# weights with bias
Theta1 = randn(inputLayerSize+1, hiddenLayerSize);
Theta2 = randn(hiddenLayerSize+1, outputLayerSize);

function sigmoid(z)
    g = 1.0 ./ (1.0 + exp(-z));
    return g;
end

function sigmoidGradient(z)
  return sigmoid(z).*(1-sigmoid(z));
end

# learning rate
alpha = 0.01;
# number of iterations
epoch = 20;
# cost per epoch
J = zeros(epoch,1);
# backpropagation algorithm
for i = 1:epoch
    for j = 1:m # for each input
        # Feedforward
        # input layer
        # add one bias element
        x1 = [1, X[:,j]];

        # hidden layer
        z2 = Theta1'*x1;
        x2 = sigmoid(z2);
        # add one bias element
        x2 = [1, x2];

        # output layer
        z3 = Theta2'*x2;
        x3 = sigmoid(z3);

        # Backpropagation process
        # delta for output layer
        delta3 = x3 - Y[:,j];
        delta2 = (Theta2[2:end,:]*delta3).*sigmoidGradient(z2) ;

        # update weights
        Theta1 = Theta1 - alpha* x1*delta2';
        Theta2 = Theta2 - alpha* x2*delta3';
    end
end

function predict(Theta1, Theta2, X)
    m = size(X, 2);
    p = zeros(m, 1);
    h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);
    h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);
    # 1 index is for 0, 2 for 1 ...so forth
    for i=1:m
        p[i,:] = indmax(h2[:,i])-1;
    end
    return p;
end

function accuracy(truth, prediction)
    m = length(truth);
    sum =0;
    for i=1:m
        if truth[i,:] == pred[i,:]
            sum = sum +1;
        end
    end
  return (sum/m)*100;
end

pred = predict(Theta1, Theta2, X);
println("train accuracy: ", accuracy(y, pred));

What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];

to

delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);

similar to the way you calculate delta2.


Unable to interpret this 100% stacked area chart I suffer from "graphical dyslexia". I have trouble interpreting charts. So I pore over them for long periods of time. For the life of me, I can't figure out how the conclusion from this chart is:

  the bottom 80% of men are fighting over the bottom 22% of women and the top 78% of women are fighting over the top 20% of men (Source)

Please help me understand.

The statement is that the point (22,80) is on the barrier between the regions marked "Male advantage" and "Female advantage".


how to disable query from beeline results I am occuring a strange hive-client beeline behavior.
In the outputed file with query results there is also a queary at the beggining and at the end. Is there any option to disable such behavior? I can't see such option in the beeline -help

-bash-4.2$ beeline -help
Usage: java org.apache.hive.cli.beeline.BeeLine
   -u <database url>               the JDBC URL to connect to
   -n <username>                   the username to connect as
   -p <password>                   the password to connect as
   -d <driver class>               the driver class to use
   -i <init file>                  script file for initialization
   -e <query>                      query that should be executed
   -f <exec file>                  script file that should be executed
   --hiveconf property=value       Use value for given property
   --hivevar name=value            hive variable name and value
                                   This is Hive specific settings in which variables
                                   can be set at session level and referenced in Hive
                                   commands or queries.
   --color=[true/false]            control whether color is used for display
   --showHeader=[true/false]       show column names in query results
   --headerInterval=ROWS;          the interval between which heades are displayed
   --fastConnect=[true/false]      skip building table/column list for tab-completion
   --autoCommit=[true/false]       enable/disable automatic transaction commit
   --verbose=[true/false]          show verbose error messages and debug info
   --showWarnings=[true/false]     display connection warnings
   --showNestedErrs=[true/false]   display nested errors
   --numberFormat=[pattern]        format numbers using DecimalFormat pattern
   --force=[true/false]            continue running script even after errors
   --maxWidth=MAXWIDTH             the maximum width of the terminal
   --maxColumnWidth=MAXCOLWIDTH    the maximum width to use when displaying columns
   --silent=[true/false]           be more silent
   --autosave=[true/false]         automatically save preferences
   --outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv]  format mode for result display
                                   Note that csv, and tsv are deprecated - use csv2, tsv2 instead
  --truncateTable=[true/false]    truncate table column when it exceeds length
   --delimiterForDSV=DELIMITER     specify the delimiter for delimiter-separated values output format (default: |)
   --isolation=LEVEL               set the transaction isolation level
   --nullemptystring=[true/false]  set to true to get historic behavior of printing null as empty string
   --help                          display this message


What is the best technique/algorithm to compare trees changes? I have a problem I would like to solve using machine learning. I would like to use some sort of classification to know if a just added change in a tree data structure is "good" or is "bad".

Let's say I have this tree:

        (A)
        / \
       /   \
     (B)   (C)

And I apply a change to it (a "good" change, so the algorithm should associate this change with the "good" changes). The updated tree would be like this:

       (A)
       / \
      /   \
    (D)   (C)
    /
   /
 (B)

Added a certain node (D) above another node (B) would be classified as a "good" change.

So when I have the learner with the correct data, the algorithm should be able to know that if I add a node of type D above a node of type B, it is a "good" change.

I would like to work with XML files that keeps the tree structure, a simple classifier like a naive bayes would not work, because it wouldn't be able to recognise if a node is added above another one, it only would be able to know that a node has been added.

I don't know how which algorithm/technique I should use and I don't know how I should pass the data to the learner, because the context in this scenario is important.

What is the best technique/algorithm to compare trees changes?

Most of the machine learning algorithms are designed to work with data in a tabular format. That mean, each data instance is contained in a single row, and the values from each column are the observed values for a specific instance for a given variable. There are few reasons why the most ML algorithms are designed to work on this kind of data. An important factor is that the structure is very simple and various operations can be done with ease. A second reason is that even if looks like a inflexible structure, some sort of additional structure in your data still can be represented on tabular format (using redundancy). Another reason would be that an algorithm designed to work for a specific structure of the data will be constraint to work on a much smaller set of problems.

So, the main point is that "If the mountain won't come to Muhammad then Muhammad must go to the mountain" (note there's nothing religion related here). So what you have to do would be to fabricate the features yourself in a tabular format.

I will give you an example on how I see a starting point. Consider an instance a row in a table. Each row will be a change. A change has a label, it is good or bad. So, you can add a feature used as target feature called class. We go further by noting that a change is an insertion of a node. If your changes are of multiple type you can add a feature called operation-type having values: insert, delete, change, etc. Now, a node has a type also. You can add a new feature called node-type, which could be A, B, etc. What you have to do would be to invent those features by noting what is important for you or your business and eventually select only those features which would be relevant enough. I really hope it was clear enough.


build a classification model under constraint Suppose I have n features $(x_1, x_2, ...., x_n)$ (all float) and want to classify $y$ (0 or 1)

For now I have a legacy expert system to do the classification.

Expert system rules:

Categorize all $x_i$

If $x_1 \lt 1$, set $x_1' = a_{11} $,

If $x_1 \in [1,10]$, set $x_1' = a_{12} $,

If $x_1 \gt 10$, set $x_1' = a_{13} $,

...

If $x_n \gt 100$, set $x_n' = a_{nk} $,

If $w_1 x_1'+...+w_n x_n' \geq z$, then predict $y = 1$, otherwise $0$

$a_{11}, a_{12}, .... a_{nk}, w_1, ..., w_n, z$ are determined by intuition instead of calculation, and I wanted to improve the expert system. How to do it? logistic regression cannot be simply applied on this case...

As Peter said, this is exactly the sort of problem logistic regression can work on directly. But, there are a few more things to say.

The binning that happens on your raw variables suggests there's some strong nonlinearity. You could do some univariate analysis to find suitable transformations, but in particular you could try binning, possibly with weight-of-evidence scoring and/or tree-based recursive partitioning.

If you want to insist on preserving the binning structure and the $x'$ values, then it's just straightforward logistic regression with these transformed predictors.

If you insist on preserving the binning structure but not their $x'$ values, you could again go with WOE or, one-hot encode for these bins and learn the $w_i x_i'$ products as the coefficients of the encoded variables in a logistic regression.

Finally, of course, you can replace almost any time I mentioned logistic regression with your favorite classifier.


Why is Distributional DQN faster than vanilla DQN? Recently I learned about Distributional approach to RL, which is a quite fascinating and break--through algorithm.

I have 2 questions:

What is it that makes it perform so much better during runtime than DQN? My understanding is that during runtime we will still have to select an action with the largest expected value. But to compute these expected values, we will now have to look at distributions of all possible actions at $x_{t+1}$, then select an action with a highest expected value. This would actually mean during extra work during runtime

What is the explanation for its faster converge than that of vanilla DQN? As I understand, the policy hasn't changed, we are still selecting the best action from state $x_{t+1}$, then use its best action's distribution for bootstrapping (adjusting) the distribution of our current state's best action.

Where does the Distributional part come into play and make the network be smarter about selecting the actions? (currently we still always select highest expected action as "the target distrib").

This is meant to be a comment, but I can't comment since I have insufficient reputation.

As for the second question, intuitively speaking, instead of taking a scalar value for an action, which initially, may be highly inaccurate and noisy, taking a distribution instead would be more accurate. I'd recommend https://flyyufelix.github.io/2017/10/24/distributional-bellman.html which explains the intuitive reason for using a distribution

In terms of convergence, actually, there is no guarantee of convergence. In the paper, however, explains that for distributional DQN to guarantee to converge, the gamma-contraction must be satisfied, which would be true if you measure the the distance between the distributions using wasserstein distance, but it would be impractical to try to minimize that distance, so distributional DQN uses cross entropy instead which you can find the gradients of, and perform backpropagation....etc

You may be interested in "Distributional Reinforcement Learning with Quantile Regression"
https://arxiv.org/pdf/1710.10044.pdf
which aims to improve the original distributional DQN algorithm


Help with training XGB I work for a highly-regulated entity, so I have to obfuscate what I'm working on; I'll provide the following as examples on what I'm doing.

I am training an XGB model for NLP comments about breeds of dogs (Tensorflow, sklearn). I have a dataset about dogs that I'm using to train the algo and cross-validation testing is promising.

However, my test data is not just about dogs, it's about animals in general. So that data can be about dogs, cats, elephants, whatever. Right now, the XGB model is giving all of the animals it does not recognize a default value (a category). How can I make my XGB just say "I don't know" instead? Is there a way to assign a default value to test values that don't make sense in any category?

BTW, I realize that another possibility is to create a model to classify the test data but I want to see if there is a way to solve this within the XGB model.

You could e.g. use as objective "multi:softprob" instead of "multi:softmax". The model is identical, but you get a probability for each label instead if the most probable label.  Then you can specify a threshold, e.g. 0.5 and return "unknown" if no probability is above the limit.


Does this line in Python indicate that KNN is weighted? Does this line in Python indicate that KNN is weighted?

clf = KNeighborsClassifier(n_neighbors=5, metric='euclidean', weights='distance')

Are the weights the inverse of the distance?

Can anyone also give an example of how weighted KNN works mathematically?

Yes, the line indicates that KNN is weighted and that the weight is the inverse of the distance. All of this can easily be found in scikit-learn's documentation

Also, pro-tip, you can find an object's documentation using the help function. In this case:

from sklearn.neighbors import KNeighborsClassifier

print(help(KNeighborsClassifier))

As to how weighted KNN works, I actually made a learning unit on my website about this.

In short, weighting the instances means that instead of each neighbor having an equal vote, their vote is weighted. This means that if for instance, in my 3 nearest-neighbors, 2 of them are of class A but have low weights, and 1 of them is of class B and has a high weight, KNN might still pick class B even though more neighbors are of class A.


Twitter Data-Analyse: What can I do with the data? I retrieve data to a specific topic from Twitter and did my sentiment analysis on it. I never did anything in NLP, etc. So what else can I do with that? "Main goal" would be to find out if the Twitter community is against this "topic" or not.
I am also struggling with cleaning the data and I mean by that, that I am unsure how much should I clean on that Tweet.
I would be also glad to get any advise on books, articles, communities, videos...

Community analysis implies graph analysis.
here is a short list of things you can work on:

People often reshares tweets among a certain social group. Minimum-cut method, Girvan–Newman and Modularity maximization are someof the starting algorithms to extract these type of substructures.
You can try and find different hierarchies among the groups sharing a particular topics
You can try and analyse the lifetime of tweets for particular topics (survival analysis)

Analysing tweets is closer to graph analytics rather than NLP.
Here is a great overview on community analysis. For coding and algorithms, please check graphX Spark library. If your data is not too large, networkX is easier.
For survival analysis, lifeline is one of the easier options.


How much can the AUC improve comparing the raw dataset and the feature engineered dataset? Let's say I put the following two datasets in the best possible model (same model for both):

A raw dataset, the variables as they came just from the query.
A feature-engineered dataset, with hundreds of created variables, which came from the same raw dataset I just mentioned.

Could the difference between both AUCs be high? How much?

Yes, the performance can vary a lot using feature engineering.

Example: suppose a dataset where the response variable $y$  is true if $x$ is odd.

x    y

346  F
13   T
178  F
64   F
987  T
...

Most learning models will fail to identify the pattern and will perform poorly, usually falling back to always predicting the majority class. However simply adding a feature $x \% 2$ to the data will allow any model to perform perfectly.

Of course this a toy example, but the point is that a single well chosen feature can drastically change the performance. Naturally the increase in performance totally depends on the data and the nature of the features added.