How to version data science projects with large files I am working on a project with large data files (~300MB). I want to version my work along with the data files so that it is always available online. I tried using git-lfs but it has a 1GB/month bandwidth limit, beyond which you're blocked for a month.
What are versioning tools using by data scientists for projects that use >100MB data files (both static and generated)?

I have used dvc.  It has data versioning though I do not use that often. I tend to use the makefile (pipeline) features.


How To Merge Features in the Dataset Forest Cover Type Classification Problem? The forest coverType Dataset Contains the following attributes distributed in many boolean features.

Wilderness_Area (4 binary columns) / qualitative / 0 (absence) or 1 (presence) / Wilderness area designation
Soil_Type (40 binary columns) / qualitative / 0 (absence) or 1 (presence) / Soil Type designation

Training the SVC classifier with RBF kernel without attributes re-engineering takes too much time. Can we merge all 40 binary columns for form a Soil_Type Attribute representing those columns. What are the pros and cons of this approach ?

Also, do we have any direct approach to achieve this in weka or sklearn ?

On Some Analysis I found out the columns were mutually Exclusive and therefore created one column named Soil_Type(0..40 possible values) replacing 40 binary columns and Wilderness_Area(0..4 possible values) replacing 4 binary columns. This did improve performance of the classifiers (Random Forest, Decision Tree ) with respect to time and prediction accuracy as well.


Stackoverflow API Structure data storage I'm downloading Stackoverflow questions & answers for a specific tag using Stackoverflow API and Python.

The goal is to perform document clustering to find relevant terms across the documents and find similarity among them.

Example:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'http://api.stackexchange.com/2.2/questions?page_number=1&pagesize=25&order=desc&sort=activity&tagged=amp-html&site=stackoverflow&run=true'

This is the procedure I'm using with Python 2.7 requests library:

Request Question  via API
Create Question object and assign API response to body property after using BeautifulSoap.
Find associated answers in Question.
Request Answers via API
Create Answer array
Assign Answer array to parent Question object.
For each Answer object assign API response to body property after using BeautifulSoap

API question request:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'https://api.stackexchange.com/2.1/questions/37745529?site=stackoverflow&filter=withbody'

API response example:

{"items":[{"tags":["html","input","amp-html"],"owner":{"reputation":314,"user_id":5426326,"user_type":"registered","accept_rate":39,"profile_image":"https://www.gravatar.com/avatar/cbff9d2f96733be04cb022f8a724ad49?s=128&d=identicon&r=PG&f=1","display_name":"Pranav Bilurkar","link":"http://stackoverflow.com/users/5426326/pranav-bilurkar"},"is_answered":true,"view_count":47,"accepted_answer_id":37787460,"answer_count":1,"score":1,"last_activity_date":1465850038,"creation_date":1465553340,"last_edit_date":1465797429,"question_id":37745529,"link":"http://stackoverflow.com/questions/37745529/input-tags-elements-in-amp-html","title":"Input Tags/Elements in AMP-html","body":"
we are implementing our current site in AMP version and i am new to this. I have below queries regarding AMP -
\n\n
\n
How to get user input from user in AMP-HTML.
\n
In the above mentioned site [desktop version] we have comment section at the bottom of the page. We need to implement the same functionality in AMP.
\n
Are there any websites which are build and developed in AMP? If yes, i need links to check.
\n
\n\n
Any suggestion and ideas related to above queries would be appreciated.\nThanks.
\n"}],"has_more":false,"quota_max":300,"quota_remaining":298}

Since questions and answers contain HTML, URLs, special characters, quotes, single quotes, commas, etc in body property. I'm looking to convert this text into structured data which can be represented as a single text chunk and be analyzed as tokens and then I use TF-IDF.

Details:

2 Objects: Question and Answers
Question object has an array of answers objects as properties.
Each Question and Answer objects contain a body property which is a single string containing each text.

What's the recommended way to store this information?

What do you want to define as your document? You could define a document as a single question, each question-answer pair, or a question with all its answers. How you define a document depends on why you want to cluster documents and how you plan to use this information. From reading your question, my guess is that you want to define a question with all its answer as a document.

Next, it sounds like you want to apply bag of words techniques to extract features (e.g. tf-idf). If so, you would first want to concatenate the question title, question body, and each answer body into one string object. Then I would suggest using a Python package like scikit learn or nltk to do the other preprocessing steps. For example in scikit learn you can then apply TfidfVectorizer to convert a string to a vector.

This is unrelated to your question, but it may be easier to use the data explorer instead of the StackOverflow API. The data explorer allows you to download questions and answers in a large batch and does not have a quota. Unless you need to collect data in real time, the data explorer may be more efficient. It is updated every week and you can use SQL queries to output data tables as csv.


Formatting categories of data with pandas in Python I have around 9500 cities that I want to put in a pandas dataframe, then store in a file for later use. For example, below, these are 3 cities that I have data for. Some range in size based on year. Lovell_Wyoming only has 9 years of data points, corresponding to the years, as opposed to the 15 that Wheatland and Worland have.

My original idea was putting the quantitative data (Arson..Year) in a map, then putting the city name, as a key, into a larger map, with that quantitative data. Building the larger map that way. Then, converting the map to a dataframe, then to a csv. I am a bit inexperienced with pandas so I am not sure how to do this correctly, if this is even the best way to do it.

At the end of the day, I would like this data in a csv file that is easily accessible by loading it into a dataframe and calling on whatever value I need.

City 'Lovell_Wyoming'
Arson [0, 0, 0, 0, 0, 0, 0, 1, 0]
Assaults [6, 6, 3, 4, 3, 28, 3, 2, 2]
Auto_thefts [1, 1, 1, 0, 0, 1, 2, 0, 1]
Burglaries [6, 11, 5, 2, 0, 15, 11, 7, 7]
Murders [0, 0, 0, 0, 1, 0, 0, 0, 0]
Rapes [0, 0, 3, 0, 0, 1, 1, 0, 1]
Robberies [0, 0, 0, 0, 0, 0, 0, 1, 0]
Thefts [23, 49, 35, 39, 28, 37, 54, 35, 10]
Year [2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2014]

City 'Wheatland_Wyoming'
Arson [0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0]
Assaults [9, 2, 6, 5, 6, 6, 2, 4, 2, 4, 3, 11, 5, 4, 8]
Auto_thefts [4, 8, 3, 3, 4, 4, 5, 3, 4, 6, 4, 8, 12, 7, 3]
Burglaries [17, 17, 14, 9, 10, 17, 12, 26, 51, 12, 15, 21, 32, 31, 13]
Murders [1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0]
Rapes [0, 0, 0, 4, 2, 1, 2, 0, 2, 1, 1, 0, 2, 0, 0]
Robberies [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Thefts [109, 95, 146, 81, 108, 100, 82, 85, 106, 128, 48, 85, 66, 56, 47]
Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]

City 'Worland_Wyoming'
Arson [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Assaults [12, 17, 19, 17, 11, 15, 16, 2, 9, 1, 4, 7, 2, 1, 3]
Auto_thefts [2, 1, 2, 1, 1, 8, 1, 1, 1, 0, 1, 0, 0, 0, 1]
Burglaries [6, 10, 10, 10, 9, 10, 10, 0, 6, 1, 0, 2, 0, 11, 18]
Murders [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Rapes [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3]
Robberies [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Thefts [44, 41, 47, 29, 30, 25, 27, 27, 23, 30, 36, 45, 54, 46, 43]
Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]

I apologize in advance if the way I formatted this is a bit weird! Please let me know if you would like any more information.


When to use which multiple testing correction? There are a large number multiple testing p-value correction methods. e.g.:
   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)
I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.
Any ideas? How do I decide which multiple testing correction I should apply?

The Bonferroni and Holm methods lead to the same FWER and disjunctive power when analysing multiple primary outcomes. This is because both methods adjust the smallest p-value in the same way. Similarly, the Hochberg and Hommel methods lead to same FWER and disjunctive power when two primary outcomes are analysed and differences between these methods arise when analysing three or more outcomes
You can read about in this article Link


In a Time Series Problem, is it possible to forecast quantities by learning the patterns of other items? What are my options? Suppose I own a store that sells a variety of apples and I have the following stats each month.

Report Date
Type of Apple (TA)
Quantity Available(QA)
Quantity Sold in the Past 30 days(QS30)
Quantity Shipping In (QSI)
Quantity Needed to Order(QN)

Lets make the following assumptions/givens:

There are three types of apples: red apples, green apples and yellow apples.
T(1) denotes the first month and T(60) denotes the 60th month.
QA @ T(i + 1) = QA@T(i) + QSI@T(i) - QS30@T(i+1)

I can provide all the data from T1-60 for each apple.

I can also provide all the data for T61 besides QN for each apple.

My goal is predict QN at T61, or the 61st month for each apple.

If I am only concerned about Red Apples, I can just use ARIMAX with data pertaining to only red apples or VAR right?

But what if I suddenly introduce a new apple type such as orange apples and only have a history from T1-4?

Given I need to use T1-4 for the orange apples...
Is it possible to use other data from the red, green and yellow to aid in properly calculating a QN value for orange apples at T5?

If I am only concerned about Red Apples, I can just use ARIMAX with data pertaining to only red apples or VAR right?

Yes, is possible, you will lose information (perhaps valuable) but you can work with it.

But what if I suddenly introduce a new apple type such as orange apples and only have a history from T1-4?

Given I need to use T1-4 for the orange apples... Is it possible to use other data from the red, green and yellow to aid in properly calculating a QN value for orange apples at T5?

You will be limited to the knowledge of the orange apples but the other variables might help you obtain enough information until you could have a more robust model, 4 entries might not be enough, but the model might help you obtain an estimation


How are samples selected from training data in Xgboost In Random Forest, each tree is not fed with the full batch of training data, only a sample.

How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?

In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement).
While on the other hand, Random Forest the samples for each decision tree are selected via bootstrapping; sampling a dataset with replacement.
Particularly for xgboost (see paper here)the ratio of sampling of each tree can be modified with the following hyperparameter:

subsample [default=1]

Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration.

There are more parameters such as colsample_bytree, colsample_bylevel, colsample_bynode.
Bootstrapping techniques can vary with the implementation and. Normally I use Catboost implementation for several reasons.
You can read about the sampling of cb that you can use in the documentation here
Moreover, a paper that I found interesting a bit ago is Minimal Variance Sampling in Stochastic Gradient Boosting . I will quote one of the sentences from the abstract

Different sampling approaches were proposed, where probabilities are not uniform, and it is not currently clear which approach is the most effective...it leads
to a new sampling technique, which we call Minimal Variance Sampling (MVS)...The superiority of the algorithm was confirmed
by introducing MVS as a new default option for subsampling in CatBoost

Here the Catboost development team explains what is the new sampling technique that is the default in their algorithm.


Twitter Retweet Network Visualization I am trying to visualize retweet network in order to find out which users are most likely to have most influence on other users. Here is my code:

    import networkx as nx

    G_retweet = nx.from_pandas_edgelist(translated_iranian_tweets,
                                        source = "userid",
                                        target = "retweet_userid",
                                        create_using = nx.DiGraph())

    print('There are {} Nodes inside Retweet Network'.format(len(G_retweet.nodes())))
    print('There are {} Edges inside Retweet Network'.format(len(G_retweet.edges())))

import matplotlib.pyplot as plt

#Size varies by the number of edges the node has (its degree)
sizes = [x[1] for x in G_retweet.degree()]

    nx.draw_networkx(G_retweet,
                     pos = nx.circular_layout(G_retweet),
                     with_labels = False,
                     node_size = sizes,
                     width = 0.1,
                     alpha = 0.7,
                     arrowsize = 2,
                     linewidths = 0)

    plt.axis('off')
    plt.show()

There are 18631 nodes and 35008 edges inside this network. Visualization is horrible, you cannot see anything. Does anyone have any suggestions what should I do about it? Should I try to extract specific type of users with specific tweets in order to reduce size of my dataset and then try to visualize the network, or something else?

For answering questions from graph, you should not visualize it. Visualizing graphs is for sake of having an overview on how it looks like in general. There are Graph Visualization techniques that show graph for sake of getting some initial insight (e.g. if there are visually obvious communities or not).

Your question has an analytic answer. After constructing your graph object, you can get the adjacency matrix as a Python array or matrix. This matrix will be asymmetric as outgoing and incoming degrees are different of course. In Networkx you can get in-degree and out-degree information as seen below. Then the main story starts!

The most influential of a social network can be seen as the most central node. Centrality measures capture them for you. The simplest one is degree centrality. It simply says the node with highest degree has most influence in graph. In your case be sure how you model "the guy who is retweeted"as he has the most influence. If being retweeted is in-degree in your modeling then this code get them for you:

import networkx as nx
import operator

g=nx.digraph.DiGraph()
g.add_edges_from([(0,1),(1,2),(2,3),(4,2),(4,3)])
dict_deg = {ii:jj for (ii,jj) in g.in_degree}
print('In-Degree Dictionary\n',dict_deg)

m = max(list(dict_deg.values()))
print('The node(s) number',[i for i, j in enumerate(list(dict_deg.values())) if j == m],'have the most influence!')

Output:

In-Degree Dictionary
 {0: 0, 1: 1, 2: 2, 3: 2, 4: 0}
The node(s) number [2, 3] have the most influence!

And the graph itself:

pos = nx.spring_layout(g)
_=nx.draw(g,label=True,pos=pos)
_=nx.draw_networkx_labels(g,pos=pos)

I hope it answered your question. If not please drop a comment so I can update.


Max depth for a decision tree in sklearn I know there is a partial answer here but my question is slightly different.
I have implemented a decision tree in sklearn. Say I have $2^n$ different values for a feature, with just one feature. I was expecting that a good decision tree should be able to keep the depth at or below $n$. Let's say the $2^n$ values for the features are indexed ($1 \dots 2^n$), if I make a split in the middle, that is the most effective way, and I can get a leaf with just one element at the end of a a n-deep tree.

Of course, it could be less, if, for example, I only have two categories, and all the elements below $2^{n-1}$ are in one class, and the other elements in the other class. But, in the worst case, I thought I should have depth $n$. However, when I implemented a solution for just around 1 million examples with one feature that has about 100,000 values, I got a tree with depth 125! (and I was expecting about 17).

Why is that?

There is an assumption in the statement if I make a split in the middle. The assumption is the tree will always split at the middle value = the median value so half the values go right and the other half go left. The split could be anywhere. And importantly the split focuses on the number of records at each value (think histogram).

Lets take the 100K values and 1m records. The root split could be, the records with the max value goes right, all other records go left. Then the next split on the left says the records with the max value of those 99,999 values go right, all other records go left. And so on. In this case it peels off one value, and all records with that value, at each split.

That is an extreme example but it illustrates the point. The split could be the min, max or anywhere in between. The number of records at each value is what is important.

You can research the answer to your tree by exporting the tree which will show you the splits and the split statistic for each node.


ensuring that dependent variable decreases monotonically with independent variable I have one key relationship between a numeric independent variable X and a numeric dependent variable Y, which is like a negative exponential function determined by 2 parameters. There are other independent variables. One numeric, the rest are categorical.

At the moment I am using quantile regression forests. They work pretty well. However, the aforementioned negative exponential relationship is not guaranteed.

Is there a way to achieve the above. For example, could this be modeled like a hierarchical model. Here the parameters of negative exponential function over X are adjusted depending on the other features. In fact, could one model this as a ANN (e.g. using KERAS) where the negative exponential function is the output layer - similar to the softmax function?

Any feedback would be very much appreciated!

There are multiple algorithms that support monotonicity, i.e., can learn a monotonic decision rule.

It is easy to enforce a monotonicity constraint when using linear regression and logistic regression; you simply enforce that the corresponding coefficient for that variable is non-negative during optimization.

xgboost supports monotonicity constraints.  However, beware that it is "best-effort" and does not guarantee that the model will be globally 100% monotonic for the entire space of data values.

Neural networks can support monotonicity constraints, by enforcing that the weights are non-negative.

I don't know how to enforce monotonicity with decision trees or random forest classifiers.  See, e.g., https://cs.stackexchange.com/q/69220/755 for a counterexample showing that even if your training set is monotonic, the resulting decision tree might learn a non-monotonic rule.

Some references to check out: https://stats.stackexchange.com/q/257049/2921, https://stats.stackexchange.com/q/342651/2921, https://stats.stackexchange.com/q/341422/2921.


How does Tensorflow compute gradients of reduce_min operation? With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation?
If so, can I retrieve the analytical computation for a specific gradient?

A minimum operation is differentiable, or at least you can easily express the partial derivatives w.r.t. its inputs:

$f = min(x_1, x_2, x_3 ... x_n)$

$ \frac{\partial f}{\partial x_i} =
\begin{cases}
    1,& \text{if } argmin_i(x_i) = i\\
    0,              & \text{otherwise}
\end{cases}$

This does not hold strictly when multiple values share the same minimum value. However, that is not a problem in practice for gradient-based optimisers in TensorFlow, which can simply set all tied indices to have partial derivative of 1 (or a fraction $\frac{1}{n_{min}}$), with little impact to the eventual result, because ties for values will happen rarely. Ties may happen frequently enough in a ReLU-based network that the TensorFlow developers have considered a best response for them - I don't know specifically what TensorFlow does for that situation.


Collaborative filtering when multiple items are rated multiple times by same user When trying to model as a recommendation problem the selection of an item that can be selected (and rated) by the same user many times, I can't find references of previous work.

For example User1 can select item A (and rate it) then item A again (and give it a different rate) and then again item A can be selected n times.. then the same for item B, but item A could be again selected at any time point, and that could happen for any of the A, B, C... Z items. Could this be modelled somehow as a recommender system?

When asked in SO I was told that can't be modelled even as a multi-modal recommneder, then I think my question should be extended to how could I model this for prediction of next rate?

SO question

I don't think there is any academic work on the subject, at least that I know of.

One simple way of using that data would be to use the mean of the ratings or other average like measures such as a moving average, a time weighted average, the median, etc.

But this approach is probably not exactly what you're looking for.

Try to look at collaborative filtering approaches with temporal dynamics, there might be something interesting for you.


Dynamic graph dataset I'm working with dynamic graph embedding but I can not seek the dataset for my problem. Can you help me find dataset with some requires:

A series of graph which represent for dynamic graphs. Each graph in series of graph is static graph with both directed and undirected
Label for each node (vertex) of graph to evaluate the embedding algorithm.

Thank for helping!

I guess it is too late, so I will be short.
You may search the SNAP graph dataset repository. There, each dataset has a set of tags like Temporal, Directed or Weighted, for instance.
It is one of the largest such repositories, so I hope you will find what you need there.


Can I completely cancel the effects of using a smaller batch size by reducing the learning rate? I'm having the problem that the data from a regular sized batch (e.g., 32, 64) doesn't fit in my GPU. Among other solutions, I'm considering reducing the batch size, as is normally suggested. Of course, this will make the neural network more unstable, so other things need to be considered.

For instance, I'm thinking about reducing the learning rate in order to offset the instability brought by the smaller batch size. Would the higher instability be completely canceled by using a smaller learning rate? Or, does this bring other problems of its own, and does not solve the problem completely?

Are you aware of this article/tutorial?

https://machinelearningmastery.com/how-to-control-the-speed-and-stability-of-training-neural-networks-with-gradient-descent-batch-size/

I did not study for this answer, but I would guess that when you assure that the batches are randomized, a smaller learning would give the same result.

My guess would be that you divide the learning rate by the square root of the batch size reduction factor.


Why does batchnorm1d in Pytorch compute 0 with the following example (2 lines of code)? Here is the code
import torch
import torch.nn as nn
x = torch.Tensor([[1, 2, 3], [1, 2, 3]])
print(x)
batchnorm = nn.BatchNorm1d(3, eps=0, momentum=0)
print(batchnorm(x))

Here is what is printed
tensor([[1., 2., 3.],
        [1., 2., 3.]])
tensor([[0., 0., 0.],
        [0., 0., 0.]], grad_fn=<NativeBatchNormBackward>)

What I am expecting is the following:
Using hand calculation, let $x = (1,2,3)$, then $E(x) = (1+2+3)/3 = 2$ and $Var(x) = (1^2 + 2^2 + 3^2) /3 - (2)^2 = 0.9999...$, so that the final output looks like $y \approx (1,2,3) - 2/\sqrt{1} = (-1, 0, 1)$
So, I am expecting the output to the batchnorm be
tensor([[-1., 0., 1.],
        [-1., 0., 1.]])

Can someone please explain where I went wrong?

BatchNorm works along dim = 0. You might want to use LayerNorm instead.


How do I create a complex Radar Chart? So, I want to create a Player Profile Radar Chart something like this:

Not only the scale of each variable different, but also I want a reversed scale for some statistics like the 'dispossessed' stat, where less actually means good.

One solution for the variable scale for each statistic maybe is setting a benchmark and then calculating a score on a scale of 100?

But, How do I display the actual numbers on the chart then? Also, how do I get the reversed scale for some of the statistics.

Currently working in Excel. What is the most powerful tool to create a complex chart like this?

Wow, this was bit challenging but I was able to make one of these plots in python.
The two main components are:

plotting multiple radial axes on a polar plot
remapping radial axes for variables with reversed scales

code:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns # improves plot aesthetics

def _invert(x, limits):
    """inverts a value x on a scale from
    limits[0] to limits[1]"""
    return limits[1] - (x - limits[0])

def _scale_data(data, ranges):
    """scales data[1:] to ranges[0],
    inverts if the scale is reversed"""
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        assert (y1 <= d <= y2) or (y2 <= d <= y1)
    x1, x2 = ranges[0]
    d = data[0]
    if x1 > x2:
        d = _invert(d, (x1, x2))
        x1, x2 = x2, x1
    sdata = [d]
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        if y1 > y2:
            d = _invert(d, (y1, y2))
            y1, y2 = y2, y1
        sdata.append((d-y1) / (y2-y1)
                     * (x2 - x1) + x1)
    return sdata

class ComplexRadar():
    def __init__(self, fig, variables, ranges,
                 n_ordinate_levels=6):
        angles = np.arange(0, 360, 360./len(variables))

        axes = [fig.add_axes([0.1,0.1,0.9,0.9],polar=True,
                label = "axes{}".format(i))
                for i in range(len(variables))]
        l, text = axes[0].set_thetagrids(angles,
                                         labels=variables)
        [txt.set_rotation(angle-90) for txt, angle
             in zip(text, angles)]
        for ax in axes[1:]:
            ax.patch.set_visible(False)
            ax.grid("off")
            ax.xaxis.set_visible(False)
        for i, ax in enumerate(axes):
            grid = np.linspace(*ranges[i],
                               num=n_ordinate_levels)
            gridlabel = ["{}".format(round(x,2))
                         for x in grid]
            if ranges[i][0] > ranges[i][1]:
                grid = grid[::-1] # hack to invert grid
                          # gridlabels aren't reversed
            gridlabel[0] = "" # clean up origin
            ax.set_rgrids(grid, labels=gridlabel,
                         angle=angles[i])
            #ax.spines["polar"].set_visible(False)
            ax.set_ylim(*ranges[i])
        # variables for plotting
        self.angle = np.deg2rad(np.r_[angles, angles[0]])
        self.ranges = ranges
        self.ax = axes[0]
    def plot(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)
    def fill(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

# example data
variables = ("Normal Scale", "Inverted Scale", "Inverted 2",
            "Normal Scale 2", "Normal 3", "Normal 4 %", "Inverted 3 %")
data = (1.76, 1.1, 1.2,
        4.4, 3.4, 86.8, 20)
ranges = [(0.1, 2.3), (1.5, 0.3), (1.3, 0.5),
         (1.7, 4.5), (1.5, 3.7), (70, 87), (100, 10)]
# plotting
fig1 = plt.figure(figsize=(6, 6))
radar = ComplexRadar(fig1, variables, ranges)
radar.plot(data)
radar.fill(data, alpha=0.2)
plt.show()


Question about graphing the clusters in K means I've used K means to cluster my data. Before using K means, I had used StandardScaler on my data to standardize the data. Now, I'm wondering how can I show the clusters of the original data. Scikit-learn gives the labels on the standardized data but I want to have the labels on the original data and show the clusters of the original data on the graph.

StandardScaler subtracts the mean from each variable and then divides it by the standard deviation. It's a common preprocessing step, certainly for k-means because this algorithm heavily depends on the scaling of the data.

If I understand correctly you want to visualize the original data and make use of the labels from k-means by doing so. You could either add the labels to the original data (assuming the order of the records did not change):

original_with_label = numpy.concatenate(original, labels, axis = 1)

Or you could transform the data back to its original scale:

transformed_back_to_original = scalar_fit.inverse_transform(transformed_data)


People detection methods In my problem I want to distinguish people from other shapes in images e.g I want to accurately know how many people are in specific region of image (at least for small number of people, for crowded places it is reasonable to get worse results).

In OpenCV there are three methods with pretrained datasets. Two basic HOGs + SVM algorithms (one trained on inria and the other on daimler dataset) and DPM method that I used also with inria dataset.
Models were trained on images of pedestrians of size 64x128(inria) and 48x96(daimler).

I've been doing my tests on a video file with two people in it, but those people are standing or sitting.
From my observations I can tell that:

DPM is the best algorithm, but it is really slow. It can detect human in sitting position.
HOGs are very dependent on the scale e.g. svm trained on daimler dataset works better with small resolution(180p in my case, because camera was close to the people), but for inria results are worse
HOGs works better when full human shape is clearly visible
Daimler gives a lot of false positives

The results coincide with expectations, because models where trained on standing pedestrians, but even when people were standing the accuracy is really bad.
Basically I need to train my own models, but I am concerned about such a strong dependence on the scale and doubt in the sense of using these methods.

I was thinking about using CNN's, but my target device has ARM cpu (https://www.mediatek.com/products/homeNetworking/mt7623n-a).
I don't need this detection to work in realtime and I'll be feeding the network with only a part of image (smaller than 480p).

Do you think that such a network can work with decent performance on such cpu?
Do you have any suggestions on type of the network, library (c++) that I could try?

UPDATE

I am experimenting with OpenCV's DNN module and yolov3-tiny. It processes whole frame in ~2s and doesn't use a lot of ram (on my target device). I am really satisfied with predictions. Of course, full yolo is better than tiny version, but much slower and consumes a lot of ram.

I am confused about number of models that exists and datasets they were trained on. Caffe Zoo has a lot of them, but I found that e.g GoogleNet doesn't have person class, so this is way I chose yolo for tests. But Yolo also is available for COCO or VOC and I just need to classify people not dogs etc.

I still have to experiment with input parameters in opencv to see how they affect performance and accuracy.

I am also thinking about retraining yolo for humans only maybe in this way I'll reach accuracy of full yolo and performance of the tiny one. What do you think? In general it is better to use COCO or VOC dataset?

If you don't need it to work in real time you should not worry about your CPU that much.

There are a few models for face detection using Res Net 10, with portability to OpenCV, those might be enough if you the people you are trying to count are facing forward.

Else, you can use a Res Net 10, it runs up to 100 FPS on a Intel i5 7200u, which is not that big deal.

Update

Oh, this Face Detection library has optimization form ARM processors  using Tengine which may help your case. About face/person I've seen a detection algorithm using OpenCV that detects human heads in any position, including the back of the head. If you want to do person count I think that should suffice. I will try to find it and post here.


To be useful, doesn't a test set often become a second dev set? I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:

You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally "done" you evaluate it on your test set. Your test performance comes back much worse than your dev performance.

So now you conclude, "My dev set must be too small, causing me to overfit my hyperparameters." So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.

But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.

To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set "too much" in the above way (increasing the dev set size in a cycle again and again), it is still "mostly" unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.

Do you think this is an accurate take?

Incidentally, I'm unsure if there's anything else you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options?

You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the Reusable Holdout. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.


How to recognize a two part term when the space is removed? ("bigdata" and "big data") I'm not a NLP guy and I have this question.

I have a text dataset containing terms which go like, "big data" and "bigdata".

For my purpose both of them are the same.

How can I detect them in NLTK (Python)?

Or any other NLP module in Python?

There is a nice implementation of this in gensim: http://radimrehurek.com/gensim/models/phrases.html

Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase "big data" comes up a lot, then the class will learn to combine "big data" into a single token "big_data". There is a more complete tutorial-style blog post about it here: http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/


Gini impurity in decision tree (reasons to use it) In a decision tree, Gini Impurity[1] is a metric to estimate how much a node contains different classes. It measures the probability of the tree to be wrong by sampling a class randomly using a distribution from this node:
$$
  I_g(p) = 1 - \sum_{i=1}^J p_i^2
$$
If we have 80% of class C1 and 20% of class C2, labelling randomly will then yields 1 - 0.8 x 0.8 - 0.2 x 0.2 = 0.32 Gini impurity value.
However, assigning randomly a class using the distribution seems like a bad strategy compared with simply assigning the most represented class in this node (in above example, you would just label C1 all the time and get only 20% of error instead of 32%).
In that case, I would be tempted to simply use this as a metric, since it is also the probability of mislabeling :
$$
  I_m(p) = 1 - \max_i [ p_i]
$$
Is there a deeper reason to use Gini and/or a good reason not to use this approach instead ? (In other words, Gini seems to over-estimate the mislabellings that will happen, isn't it ?)
[1] https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity

EDIT: Motivation
Suppose you have two classes $C_1$ and $C_2$, with probabilities $p_1$ and $p_2$ ($1 \ge p_1 \ge 0.5 \ge p_2 \ge 0$, $p_1 + p_2 = 1$).
You want to compare strategy "always label $C_1$" with strategy "label $C_1$ with $p_1$ probability, and $C_2$ with $p_2$ probability", thus the probability of success are respectively $p_1$ and $p_1^2 + p_2^2$.
We can rewrite this second one to:
$$
p_1^2 + p_2^2 = p_1^2 + 2p_1p_2 - 2p_1p_2 + p_2^2 = (p_1 + p_2)^2 - 2p_1p_2 = 1 - 2p_1p_2
$$
Thus, if we substract it to $p_1$:
$$
p_1 - 1 + 2p_1p_2 = 2p_1p_2 - p_2 = p_2 ( 2p_1 - 1)
$$
Since $p_1 \ge 0.5$, then $p_2 ( 2p_1 - 1) \ge 0$, and thus:
$$
p_1 \ge p_1^2 + p_2^2
$$
So choosing the class with highest priority is always a better choice.

EDIT: Choosing an attribute
Suppose now we have $n_1$ items in $C_1$ and $n_2$ items in $C_2$. We have to choose which attribute $a \in A$ is the best to split the node. If we use superscript $n^v$ for number of items that have a value $v$ for a given attribute (and $n^v_1$ items of $C_1$ that have value $v$), I propose we use the score:
$$
\sum_v \frac{n^v}{n_1 + n_2} \frac{max(n^v_1, n^v_2)}{n^v_1 + n^v_2}
$$
As a criterion instead of Gini.
Note, since $n^v = n^v_1 + n^v_2$ and $n_1 + n_2$ doesn't depend on the choosen attribute, this can be rewritten:
$$
\sum_v max(n^v_1, n^v_2)
$$
And simply interpreted as the number of items in the dataset that will be properly classified.

Misclassification error will not help in splitting the Tree.
Reason-We consider the weighted dip of error from parent Node to the child node and misclassification error will always result in 0(Other than pure splits).
Let's consider an example
Data = 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
Parent Classification error= 4/10 = 0.4
Parent Gini Impurity = 1-(0.4x0.4+0.6x0.6) = 0.48
Case - I
Split - 1, 1, 0, 1  Vs 1, 0, 1, 0, 1, 0
Classification error= 0.4x0.25 + 0.6x0.5 = 0.4, split not possible
Gini Impurity = 0.45
Case - II
Split - 1, 1, 1, 0, 0  Vs 0, 1, 0, 1, 0
Classification error = 0.5x0.4 + 0.5x0.4 = 0.4, split not possible
Gini Impurity = 0.48
Case - III
Split - 1, 1, 0, 0, 1, 1  Vs 1, 0, 1, 0
Classification error = 0.6x(2/6) + 0.4x0.5 = 0.4, split not possible
Gini Impurity = 0.477
Pure splits
Split - 1, 1, 1, 1, 1, 1  Vs 0, 0, 0, 0
Classification error = 0, split possible but no further splits
Gini Impurity = 0
Reference-
Sebastian Raschka Blog


How to Implement a Custom Loss Function with Keras for a Sparse Dataset My dataset is composed of an idle system that, at some time instants, receives requests. I'm trying to predict these instants through a clock. Since the requests are sparsely distributed (I've forced them to last for a while so they don't get too sparse), I wanted to create a new loss function that would penalize the model if it only gives out a zero prediction for everything. My implementation attempt is just a penalty for the standard logits:

def sparse_penalty_logits(y_true, y_pred):
    penalty = 10
    if y_true != 0:
        loss = -penalty*K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))
    else:
        loss = -K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))

    return loss

Is it correct? (I have also tried it with tensorflow). Every time I run it I either get a lot of NaN's as the loss or predictions that are not binary at all. I wonder if I'm doing something wrong at setting up the model also because binary_crossentropy is not working properly either. My model is something like this (the targets are represented by a column with either 0's or 1's):

model = Sequential()
model.add(Dense(100, activation = 'relu', input_shape = (train.shape[1],)))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(optimizer = 'adam', loss = sparse_penalty_logits)

If I run it, as I said, I get very strange results (boy, do I feel like I've messed up real bad...):

From the mentioned problems you are facing, this seems like a problem of exploding gradients. The exploding gradients problem can be identified by:

The model is unable to get traction on your training data (e.g. poor loss).
The model is unstable, resulting in large changes in loss from update to update.
The model loss goes to NaN during training.

More about Exploding gradient problem can be found at this article

I would suggest you to use some gradient clipping technique in you code and this will remove the NaN generation during model training.


Accessing directory of small files as one file I have a job that results in a directory of part- files. I'd like to read it as if it were one file. Specifically, I'd like to read it that way over a web interface.

How can I do either one of these things? Is there a hadoop component which makes this easy?

hadoop fs -getmerge <hdfs-output-directory> <local-file>

This command can be used to concatenate the HDFS files into a single local file.

Ref : http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#getmerge


Uniformity of color and texture in an image I am new to the field of deep learning and have a problem in determining whether two images have uniform color and texture. For example, I have a

Master image -

Now, with respect to this image i need to determine whether the following images have uniform texture and color distributions -

image 1 -

image 2 -

image 3 -

I need to develop an algorithm which will evaluate these 3 images with the master image. The algorithm should approve the image 1 and reject image2 because of its color. And reject image 3 because of color and texture.

My approach for the problem was directly analyzing image for texture detection. I found that Local Binary Patterns method was good among all texture recognition methods (but I am not sure). I used its skimage implementation with opencv in python and found that the method worked.

from skimage import feature
import numpy as np
import cv2
import matplotlib.pyplot as plt

class LocalBinaryPatterns:
    def __init__(self, numPoints, radius):
        # store the number of points and radius
        self.numPoints = numPoints
        self.radius = radius

    def describe(self, image, eps=1e-7):
        # compute the Local Binary Pattern representation
        # of the image, and then use the LBP representation
        # to build the histogram of patterns
        lbp = feature.local_binary_pattern(image, self.numPoints,
            self.radius, method="uniform")
        (hist, _) = np.histogram(lbp.ravel(),
            bins=np.arange(0, self.numPoints + 3),
            range=(0, self.numPoints + 2))

        # normalize the histogram
        hist = hist.astype("float")
        hist /= (hist.sum() + eps)

        # return the histogram of Local Binary Patterns
        return hist

desc = LocalBinaryPatterns(24, 8)

image = cv2.imread("main.png")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
hist = desc.describe(gray)

plt.plot(hist,'b-')
plt.ylabel('Feature Vectors')
plt.show()

It detected the features and made a histogram of feature vectors. I plotted the histogram using matplotlib and clearly found that image 1 and image 2 texture features were almost similar to the master image. And image 3 texture features were not matching.

Then I started analyzing images for their color. I plotted the color histograms using opencv as -

import cv2
from matplotlib import pyplot as plt

def draw_image_histogram(image, channels, color='k'):
    hist = cv2.calcHist([image], channels, None, [256], [0, 256])
    plt.plot(hist, color=color)
    plt.xlim([0, 256])

def show_color_histogram(image):
    for i, col in enumerate(['b', 'g', 'r']):
        draw_image_histogram(image, [i], color=col)
    plt.show()

show_color_histogram(cv2.imread("test1.jpg"))

I found that color histogram of image 1 matched with master image. And color histograms of image 2 and 3 did not matched. In this way I figured out that image 1 was matching and image 2 and 3 were not.

But, I this is pretty simple approach and I have no idea about the false positives it will match. Moreover I don't know the approach for the problem is the best one.

I also want this to be done by a single and robust algorithm like CNN (but should not be computationally too expensive). But I have no experience with CNNs. So should I train a CNN with master images?. Please point me in the right direction. I also came across LBCNNs, can they solve the problem?. And what can be other better approaches.

this is pretty simple approach

Firstly, can you evaluate your scripts on more images to get an idea of how well it performs? If you get an acceptable classification accuracy (or e.g. a good F1 score), then there is no need to try out a CNN!

  I have no idea about the false positives

Actually if you cannot evaluate your method like that, then a CNN is also not possible!

If you still want to try a deep learning method, bear in mind that you will generally need a lot of images, let's say at least 1000 in your "master image" training set - then hopefully a good percentage of images to test again (hold-out set / test set).

  I don't know the approach for the problem is the best one

Your current method seems reasonable to me. There probably is "the best" method in general, so don't be too worried.

Somebody has written An Analysis of Deep Neural Networks for Texture classification - maybe that contains some ideas to get you started  :-)

Here is a short video intro to one approch that introduced the Local Binary CNN, which was originally used for image classification, but perhaps could be adjusted to your problem.

Instead of aiming for target classes, you would simply need to map input to the labels, and could even relax the focus on colour/texture - the CNN would extract what it needs to learn the mapping.

NOTE: the downside of an end-to-end system like that is that you would no longer know what your model is using as its features to make its prediction! Colour? Texture? Raw RGB values in the top left corner?

This is actually where the LBCNN could shine, because its sparse (stochastically) binary kernels are much less likely to overfit to your training data, compared to a standard CNN.

While I couldn't find the code of the authors from the video above, here is an LBCNN implementation for face detection.


Is there any problem with dropping only part of the OneHot generated features? The one hot encoder adds more columns to the data, one for each category in the encoded feature. In the example below, the column City was transformed into 4 other columns.
Suppose a Decision Tree is ran on a dataset the below is part of and City_Chicago and City_New_York appear to be in top most important features while City_Detroit and City_SanFrancisco in the least important. Would there be any problem if I drop City_Detroit and City_SanFrancisco from my dataset, but keep City_Chicago and City_New_York or do I need to keep all city features as they are part of one initial feature?
|---------------------|------------------|-------------|---------------|---------------|
|      City           | City_SanFrancisco| City_Detroit| City_New_York | City_Chicago  |
|---------------------|------------------|-------------|---------------|---------------|
|     San Francisco   |         1        |      0      |       0       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     Detroit         |         0        |      1      |       0       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     New York        |         0        |      0      |       1       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     Chicago         |         0        |      0      |       0       |     1         |
|---------------------|------------------|-------------|---------------|---------------|

I think you can keep as many as you want, and it'll be alright. Sometimes it's even worth to delete the very rare classes to have more stable features.
In addition, for linear regression, you shouldn't include all of them, as you might have a collinearity issue.
To sum up, no problem with not keeping them all.


ML in R (caret-package) missing hyperparameters I have a pretty specific question regarding the caret package however I still hope to finde help here.
I recently worked with the caret package and trained a multilayer perceptron with method = 'mlp'. I looked up the github page of Max Kuhn (developer of caret), and it says that you only need to tune one hyperparameter: the size (number of neurons in the hidden layer). Which is really convinient.

However it further states that caret for the training builds on the RSNNS Package (by Bergmeier). The mlp model implemented in this RSNNS package has additional tunable parameters over just the size hyperparameter (i.e. learnFunc,hiddenActFunc,Std_Backpropagation, maxit).

So I asked myself what values caret uses for those parameters? Default values or are those optmizied?

It appears that the defaults are used, except for lin, which is inferred from the type of the target variable: [source code]

Note too that you can set any of the other RSNNS parameters through the dots.


Backpropagation for noobs I am trying to understand neural networks and how they work, by programming my own one from scratch in nodejs. Currently, i managed to build a network, that has weights, layers and neurons. I also understood what an activation function is and i am using the sigmoid function.

I've now come to the point of back propagation with the gradient descent algorithm. My Problem is, that i only know 10th grade Math and every tutorial/explanation i managed to find, uses complex functions and math which i cannot manage to understand.

I would really like to finish this project and get back propagation to work, so if someone could explain me how to use the back propagation without using too complex math (e, derivatives, functions, etc.) or explaining the complex math it would be greatly appreciated.

My source Code: https://gitlab.com/milan44/node (Training is happening in Network.prototype.train)

How about this video, which explains backpropagation intuitively?

https://www.youtube.com/watch?v=Ilg3gGewQ5U

Title: "What is backpropagation really doing? | Deep learning, chapter 3"

Author: "3Blue1Brown"


Gender identification task on instance or user level? I'm working on a task which is gender identification. Given a user account (e.g. Twitter account) with its documents (e.g. 100 tweets), the user should be classified as a male or a female.

The dataset that I have contains around 500 users for each class (label).

There are 2 different ways to approach this task, but I don't know which one is "more correct":

1) I aggregate the documents of each user into one large document, and then feed the final document into a classifier.

2) I project the user label (class) into her/his documents, and then feed each single document of the user into a classifier. At the prediction time, I apply averaging on the probabiltities of the users' documents to get the labels of the users (e.g. larger or smaller than 0.5).

Probably an answer to this question could be that both ways are task-dependent, but I want to know if there is a scientific explanation behind any of the solutions, especially in my task.

BTW, some of the documents for many users are not important (e.g. Hello all :D!).

This depends heavily on your intended use case:

When 1) is better

If you want to identify the gender as best as you can while always having access to a data set containing multiple documents/tweets then this is the way to go.

There are multiple features that you can only engineer from the full data set (like average length of tweets, tweet frequency, etc.) that might be very predictive.

When 2) is better

If however your use case is to predict the gender of user based on a single tweet (maybe even without knowing the id/origin of the tweet, only the tweet text) then you need to train a model that can make a prediction out of a single tweet.

In this case I would also encourage you to get a diversified data set which contains more tweets from several different users instead of a large class of tweets from few users.


TensorFlow MLP loss increasing When I train my model the loss increases over each epoch. I feel like this is a simple solve and I am missing something obvious but I cannot figure out what is it. Any help would be greatly appreciated.

The neural network:

def neural_network(data):
    hidden_L1 = {'weights': tf.Variable(tf.random_normal([784, neurons_L1])),
                'biases': tf.Variable(tf.random_normal([neurons_L1]))}

    hidden_L2 = {'weights': tf.Variable(tf.random_normal([neurons_L1, neurons_L2])),
                'biases': tf.Variable(tf.random_normal([neurons_L2]))}

    output_L = {'weights': tf.Variable(tf.random_normal([neurons_L2, num_of_classes])),
                'biases': tf.Variable(tf.random_normal([num_of_classes]))}

    L1 = tf.add(tf.matmul(data, hidden_L1['weights']), hidden_L1['biases']) #matrix multiplication
    L1 = tf.nn.relu(L1)

    L2 = tf.add(tf.matmul(L1, hidden_L2['weights']), hidden_L2['biases']) #matrix multiplication
    L2 = tf.nn.relu(L2)

    output = tf.add(tf.matmul(L2, output_L['weights']), output_L['biases']) #matrix multiplication
    output = tf.nn.softmax(output)

    return output

My loss, optimiser and loop for each epoch:

output = neural_network(x)
loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y) )
optimiser = tf.train.AdamOptimizer().minimize(loss)

init = tf.global_variables_initializer()

epochs = 5
total_batch_count = 60000//batch_size

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(epochs):

        avg_loss = 0

        for i in range(total_batch_count):

            batch_x, batch_y = next_batch(batch_size, x_train, y_train)

            _, c = sess.run([optimiser, loss], feed_dict = {x:batch_x, y:batch_y})

            avg_loss +=c/total_batch_count

            print("epoch = ", epoch + 1, "loss =", avg_loss)

    sess.close()

I have a feeling my problems lies in the either the loss function or the loop I wrote for each epoch, however I am new to TensorFlow and cannot figure this out.

You are using the function softmax_cross_entropy_with_logits which, according to Tensorflow's documentation, has the following specification for logits,

logits: Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized log probabilities.

Hence, you should pass the activations before the non-linearity application (in your case, softmax). You can fix it by doing the following,

def neural_network(data):
    hidden_L1 = {'weights': tf.Variable(tf.random_normal([784, neurons_L1])),
                'biases': tf.Variable(tf.random_normal([neurons_L1]))}

    hidden_L2 = {'weights': tf.Variable(tf.random_normal([neurons_L1, neurons_L2])),
                'biases': tf.Variable(tf.random_normal([neurons_L2]))}

    output_L = {'weights': tf.Variable(tf.random_normal([neurons_L2, num_of_classes])),
                'biases': tf.Variable(tf.random_normal([num_of_classes]))}

    L1 = tf.add(tf.matmul(data, hidden_L1['weights']), hidden_L1['biases']) #matrix multiplication
    L1 = tf.nn.relu(L1)

    L2 = tf.add(tf.matmul(L1, hidden_L2['weights']), hidden_L2['biases']) #matrix multiplication
    L2 = tf.nn.relu(L2)

    logits = tf.add(tf.matmul(L2, output_L['weights']), output_L['biases']) #matrix multiplication
    output = tf.nn.softmax(logits)

    return output, logits

Then, outside your function, you can retrieve the logits, and pass it to your loss function, as in the example bellow,

output, logits = neural_network(x)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits,
                                                             labels=y))

I remark that you may still be interested in the outputs tensor, for calculating your network's accuracy. If this substitution doesn't work, you should also experiment with the learning rate parameter on your AdamOptimizer (see the documentation here).


What is the use of torch.no_grad in pytorch? I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?

The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official PyTorch tutorial.
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
    print((x ** 2).requires_grad)

Output:
True
True
False

I recommend you to read all the tutorials from the link above.
In your example: I guess the author does not want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.


How could a considerable increase in loss leads to an improvement in accuracy? I'm experimenting with NLP and at the moment, I'm trying to come up with a translator model for converting English sentences to French counterparts. I'm using this dataset (not that it's relevant):

https://github.com/udacity/deep-learning/raw/master/language-translation/data

which is composed of more than 137K sentences. My model is an encoder-decoder LSTM with attention implemented in Keras. And here are my plotted validation loss and accuracy charts:

The two accuracy metrics are custom ones developed by myself. But they are based on the same categorical_accuracy from Keras.

Now, my question is why I'm getting an improvement for the accuracy while the loss value is getting wrose?

Also, is such a model trustworthy?

Check whether padded values get accounted while computing accuracy, which shouldn't be the case. You should create mask for padded values and use that while computing accuracy. For instance, while the y_true value for padded region is 0, and 0 as well for y_pred, then y_true == y_pred counts as a correct prediction and impacts the overall accuracy. This is wrong, and what you should instead do is define a custom accuracy metric that does the following:
correct = y_pred == y_true
mask = tf.cast(tf.cast(y_true, tf.bool), tf.int16)
accuracy = tf.reduce_sum(correct * mask)/tf.reduce_sum(mask)

Hopefully this should give you a correct measure.


Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution I am working on a binary classification problem with unbalanced data (17% for positive class).
The problem is as following:
My three individual models when predicting on the test set (for which I don't have the labels) gives quite similar distribution as for Train set.
But ensemling these models, while giving slighltly better result (F1-score), it drastically changes the distribution on Test set going from ~20% to 5%.
My question is :
I am confused between choosing the best individual model which maintains almost same distribution but lose some efficiency
Or
The ensembled one who gives really different distribution
And I have no Idea about the Test set distribution.
Thanks for any help

Depending on the size of the test and training set, and how they were sampled, their distributions may not be the same. if the sets aren't very big or weren't randomly sampled then their distributions won't necessarily be the same, and may not correspond to the distribution of the population
One way to test this is to compare the distributions of the other variables in the test and training sets.
In terms of selecting individual models or the ensemble for prediction, I would recommend trying the individual models and the ensembled combination in k-fold validation, and selecting whichever approach results in the best performance. This way you are using the entire training set for training and validation, and the assessment of this performance should be the best approximation of test set performance.


Am I justified in dropping this independent variable? I'm currently doing churn prediction in R and during EDA, I discovered that a variable, say gender, has 1720 males who don't churn, and 280 males who do. Also, it has 864 females who don't churn, and 136 females who do. So roughly 14% of both genders churn and 86% don't. From what I can see, this variable does nothing to explain churn. Am I justified in dropping this variable before I put it into a predictive model? (I have conducted stepwise logistic regression and this variable isn't included in the final call)

I'm sorry if this seems like a very basic or obvious question. I've just started with ML and didn't find a question like this elsewhere.

EDIT: Gender is highly uncorrelated with other independent variables

If you think it isn't a variable that explains your target variable (churn), you are of course allowed to leave it out - it is in a way then a hand-crafted feature, because you are deliberately telling the model to ignore it (by omitting it).

There are models that would be able to take it into consideration and then decide whether to use it or not, such as component-wise gradient boosting (CWGB) or neural networks. CWGB would simply end up assigning the variable a coefficient of zero if it decides that it doesn't help explain churn rate.

In general, which the percentage split that you gave for males and females, I personally think it could be a valuable feature - even if the model is able to deduce that roughly 14% of all people churn. There might also be correlation between the gender and other features in your dataset that, together, are able to give a stronger signal regarding churn probability.


Sentiment analysis using sources other than the IMDB data I'm curious as to whether the training data for a sentiment analysis tool needs to be specifically geared toward the domain it's being using in. For example, the IMDB movie review data makes sense if I want to gauge sentiment on movie reviews, but would it still be okay to use if I want to gauge sentiment on what the general public thinks about a company's stock?

If I want to gauge sentiment on a company's stock, should I create my own training set that uses data from financial news/social media reaction to companies?

One of the most fundamental assumptions in machine learning is that the training data is 'similar' to the test data. Training makes no sense otherwise.

So the question is: How similar are reviews of, say, movies and stocks? Maybe somewhat, but not too much.
Your movie-trained algorithm would certainly be able to deal with statements like 'This company is awesome' or 'Warning, do not buy this stock'.

But what about 'The stock price will explode/implode' or 'Concerning this stock, I am bearish/bullish'? The words explode, implode, bearish and bullish have probably never been expressions of sentiment in movie reviews.


What's the best way to predict weekly selling data? I am trying to create a model to predict the units that will be sold for different grocery items say in the next week.

I am structuring the problem in a three-step procedure.

Group together the selling data from a macro-category (e.g. if I want to predict "egg box XYZ" I will consider all the items from category "eggs") summing the selling data from all items of that category for each day. Then I use prophet to predict the selling data for next week for the whole category. I am doing this step as the "whole category" data have more history and should be more robust compared to single items data.
Implement a prophet model on the specific items. I expect the error to be higher since the data are less consistent.
Implement a regressor (could be GradientBoost, RandomForest etc.) on the single item using as input variable for a specific day: the previous week selling data for that item, the forecasted sold units for the whole category, the forecasted units (again with prophet) for the specific item, whether we are in a particular time of the year (e.g. boxing day), promotions applied etc. The idea behind this regressor would be to dampen the error of the second step. However I cannot lower the error from the second step.

How could I approach this problem?

There is an ongoing competition on Kaggle right now which is about sales prediction.
https://www.kaggle.com/c/m5-forecasting-accuracy/notebooks

The user kyakovlev have created a series of notebooks where he creates features and creates an lgbm model:

https://www.kaggle.com/kyakovlev/m5-three-shades-of-dark-darker-magic

The notebook linked is the last in the series.
I've had success in applying a very similar strategy on a real-world project for a client.

You probably want to engineer more features and add more lagging variables.

The prophet model could possibly be a problem if the patterns in the data aren't clear. I have had examples where it is completely of. This could be due to the trend. The Prophet model also easily overfit on holidays due to the lack of data. This is due to the nature of the model.
You should definitely inspect what the model is learning.

Models such as XGBoost or LightGBM are great for this type of problem!


How to compare two clustering solutions when their labelling differs I am planning to test the reliability of a clustering approach for some data. My plan is to repeatedly (with replacement) draw a number of random subsample pairs (e.g. 2x 10% of the total data), run the clustering on both individually, and then compare the results. The issue is that I am using HDBSCAN, which not only creates a non-fixed number of clusters (for different sets of data but same params), but it also therefore labels clusters differently since k is not defined, and the input data will always have slightly different structure due to variability.
I tested this by using the same HDBSCAN parameters on two subsamples (A, B) of my data, and my issue is quite easy to see. The cluster labels with corresponding samples for A were:
{-1: 4306, 0: 1737, 1: 2999, 2: 72068, 3: 20628, 4: 3120}
while for B they were:
{-1: 4478, 0: 1711, 1: 3048, 2: 72089, 3: 3123, 4: 20408}.
From this, it seems that the solution is very close until we compare label 3. It looks like label 3 of A corresponds to label 4 of B.
My initial thought was that I could just relabel them both in order of each cluster's sample size. But this assumes that the two solutions will be similar across many tests (which is ultimately the whole point of the testing in the first place). So my next thought is I could set the constraints that (1) there should be a "similar" number of samples in the noise group, and (2) there should be the same number of clusters found. If these two conditions are met then I could relabel the clusters by order of their sample size, and then make my comparison using ARI or AMI.
I am doubtful that this is good, because I don't believe it is necessarily true that (even given the two constraints) two clusters labelled the same on the basis of their sample should necessarily correspond to the same "global" cluster. It therefore seems problematic to me but I can't think of an alternative.
Is the above approach generally reasonable? If not, is there something else I could do to assess the reliability/stability of HDBSCAN solutions? As an alternative, would it be better to just compute the DBCV score, %noise, and the number of clusters, and then use this as an indication of the quality of the clustering?

This is only a partial answer since I'm not familiar with HDBSCAN, hopefully somebody else can provide a more complete answer.
As far as I understand you need to find which cluster in A corresponds to which cluster in B, i.e. an alignment between the clusters labels of A and of B. It's not recommended to match based only on the size, since it could happen by chance that a cluster in A has the same size (or similar size) as a cluster in B.
Since the instances are different, you would have to rely on how the method represents the clusters.

For example probabilistic clustering methods represent each cluster as a distribution over the features, so one can use a distance/similarity measure between these distributions.
With k-means one would compare the centroids and match the pairs of clusters for which the distance is the shortest.

I'm not familiar with HDBSCAN so I don't know how the clusters are represented inside the model: whatever this is, the idea would be to compare each internal representation of a cluster in A vs. the same in B.


One-Dimensional Convolutional Neural Network Can someone explain how 'One-Dimensional Convolutional Neural Network' works. I do understand the 2-D for image but for 1-D how is the filer created. is it fixed 1-D filter within a specific time interval or the operation is the same as we convolve a signal with a filter in signal processing y = f*x

is it fixed 1-D filter within a specific time interval?

Yes. The same as filters in 2D. Adjacent filters may even have no overlap with each other.

1D CNN is almost the same as 2D CNN both mathematically and visually by setting the second dimension (either the horizontal or vertical one in visualizations) to 1. This way, 1D filters are placed (possibly with some overlap) in one dimension instead of 2D filters being spread in two dimensions.

The below image shows a filter set with shared parameter $W$ covering the overlapping regions of the input.

By shared parameter we mean $f_i=\mbox{ReLU}(\mbox{sum}(W \odot  \mbox{region}_i))$, where $\odot$ is a point-wise product between a region of input and $W$.


Filling missing values for Embedded List in Python3 I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough.

I have a dataset consist of text data. I store them in a list and every row of a list consists of a string value. But every row length is not equal. I want them to be equal, so I can use them in a self-attention model.

The sample of my dataset

In [8]: myList
Out[8]:
[
['the first line of my dataset'],
['the second line'],
['the 3rd'],
['the 4th'],
['the 5th'],
['the 6th'],
['the 7th'],
]

So as you can see the first one is longer than the rest of them. I want to fill with a certain value like # to equalize the word count.

The sample output I'd like to do

In [8]: myList
Out[8]:
[
['the first line of my dataset'],
['the second line # # #'],
['the 3rd # # # #'],
['the 4th # # # #'],
['the 5th # # # #'],
['the 6th # # # #'],
['the 7th # # # #']
]

If this would be a dataframe I could use fillna() function of Pandas library. I tried to apply this:

train_X = pd.Series(train_X).fillna("#").values

but since it is an embedded list(I guess) it didn't work. Is there a better way to do that?

Any recommendation is appreciated.

According to the suggestion, @bkshi gave to me I come up with a solution here below:

Also since texts_to_sequences() function convert my list to sequences starting from 1, I could use pad_sequence() and use 0 instead of a string value.

This solution satisfies my requirements so I used a number as padding instead of a string value.

import pandas as pd
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

my_list = [['the first line'],
       ['the 2nd line'],
       ['the 3r line'],
       ['the 4th line'],
       ['the 5th line'],
       ['the'],
       ['the 5th line, this is']]
max_features = 10 #how many unique words you're using

tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(my_list)
my_list = tokenizer.texts_to_sequences(my_list)
my_list = pad_sequences(train_X, maxlen=None, dtype='int32', padding='post', truncating='post', value=0.0)


Dot product and linear regression I'm studying PCA and my professor said something about finding the linear regression by doing the dot product of both axis. Could someone explain to me why? The dot product returns a number. What's the relationship between that number and the linear regression?
In my example, I have two vectors
$stat\_grade = [0,1,3,7,10]$
$physics\_grade = [1,5,8,10,10]$
The first step is normalizing them:
$ \frac{stat\_grade - mean(stat\_grade)}{std(stat\_grade)}  = [-1.69131435 -0.52489066  0.34992711  0.93313895  0.93313895]$
$ \frac{physics\_grade - mean(physics\_grade)}{std(physics\_grade)} = [-1.11613741 -0.85039041 -0.3188964   0.7440916   1.54133261]$
Computing the dot product between the normalized vectors will return $4.355129045906131$
I can't understand how a single number can help me to find the linear regression
Thanks for the response!

I think the professor might have meant the closed-form solution of linear regression.


prediction for a linear sum I am learning about SVMs in particular linear SVMs through many questions here. However, one problem i faced is that there seems to be no indepth explanation on how does linear SVM works in terms of predicting new data.

I understand that the main purpose of SVM is to find linear separating hyperplane $w^Tx+b$ and a linear SVM is actually a set of super long equation.
Let's consider a 2 class problem : A and B. Suppose $(w^*,b^*)$ are the minimizing hyperplane parameters for a fixed choice of $\lambda$.

Then how we classify a new, unlabeled test point $x_{test}$? Simple way that I thought would be reasonable is to have $(w^*)^Tx+b^*>0$ to class A and $(w^*)^Tx+b^*<0$ to class B. But how do we assign if it's 0 and what if there is an outlier in other class for example? IS this a good way of labeling test data?

If it's exactly zero, you might classify it in either class. When using floating point numbers, there's a nearly 0% probability that an observation would ever have a predicted value of exactly zero. There's always some chance that observations will get the wrong label - that's how statistical models work, if you can find some rule that will with 100% certainty give you the right result, you shouldn't be using a statistical model in the first place.

I'm not sure why you want to "label test data". Test data is by definition labeled data that you set apart in order to get a hopefully unbiased evaluation of your model performance. Perhaps you mean making new predictions?


How can I picture an unfolded RNN as a normal Feed Forward Network? I am currently working on a Transformer architecture.
Trying to picture an RNN (or Encoder) as a normal Feed Forward network really confused me after looking at the following image in an article:
(Image 1)

I am usually used to seeing it like this:
(Image 2)

In Image 1, it shows that the input goes in ALL at once, whereas in Image 2, we see only a single input at a timestep.
My two questions:

Does this mean that Image 1 is a single node in Image 2?

How can I picture an RNN-architecture as Image 1?

I cannot be sure whether this graphical representation is fully correct or wrong, but your misunderstanding looks like it revolves around what is an LSTM cell and LSTM neurons.
In the second figure, the nodes depicted should be LSTM cells. A cell is in essence a forward neural network consisting of neurons and so an analytic representation of it may look like what the first figure shows, although admittedly it looks very confusing.
See a recent discussion on that exact matter here What is the difference between a "cell" and a "layer" within neural networks?.


Tensorflow / Keras Python CNN I'm doing a project where a Python script used a convolutional neural network to determine if a plant is healthy, and then water it based on that. While training the CNN, it seems to get up to 100% accuracy quite early, although it isn't accurate. I only have a little less than 2000 images, and was wondering if I didn't have enough, or it was my model, which is here
        self.model = Sequential()
        self.model.add(Conv2D(numFilters, filterSize, activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)))
        self.model.add(Conv2D(numFilters * 2, (3, 3), activation='relu'))
        self.model.add(MaxPooling2D(pool_size=poolSize))
        self.model.add(Dropout(0.25))
        self.model.add(Flatten())
        self.model.add(Dense(numFilters * 4, activation='relu'))
        self.model.add(Dropout(0.5))
        self.model.add(Dense(2, activation='softmax'))
        self.model.compile(loss='categorical_crossentropy',
                      optimizer = 'adam',
                      metrics=['accuracy'])

I would just like to know the reason why it doesn't train well.

Not sure if your model is too simple, I have used the same number of layers before and had very good results. Guess it depends on many factors, such as the training image dataset, etc. The high accuracy is coming from learning the training data quickly, but the low accuracy means the model you built doesn't apply to the test data, so the model is over-fitting. Some possible causes:

1) not enough training data

2) training images are too similar to each other, but different from the test data

3) model is too complex and does not generalize well
4) etc

Possible fixes:

1) If your training data is too small, you can use public pre-built models (VGGnn, etc) as a starting point. This will work best on models that were built with similar data.

2) If your images are too similar to each other, you can alleviate this by using image augmentation. The keras.preprocessing.image.ImageDataGenerator library is good for this.
Here is an example on how to use:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

3) If the model is too complex, you can try to simplify by changing filter sizes, or add regularization


Using data agumentation for a frozen pre-trained model I was following the following article with regards to doing transfer learning:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

In the section, Using the bottleneck features of a pre-trained network: 90% accuracy in a minute, the authors mentioned that: "Note that this prevents us from using data augmentation"

I am not very clear about this; is there a rule that discourages the use of data augmentation when the pre-trained model is totally frozen?

No, you can definitely use data augmentation when the layers of the pre-trained model are frozen.

In this article, the author refers to the size of VGG16 which is quite a large CNN. Also, he trains the model on his CPU which slows down training even more. Therefore, he does not want to use data augmentation, as that would increase the training time even more:

  Running VGG16 is expensive, especially if you're working on CPU, and we want to only do it once. Note that this prevents us from using data augmentation.


Splitting into multiple heads -- multihead self attention So, I have a doubt in Attention is all you need:
The implementation of transformers on tensorflow's official documentation says:

Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads.

However, The paper mentions:

Instead of performing a single attention function with dmodel-dimensional keys, values and queries,
we found it beneficial to linearly project the queries, keys and values h times with different, learned
linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of
queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional
output values.

There is no mention of splitting Q,K and V to obtain heads. Instead, the paper says that they are passed through 'h' different dense layers to convert d-model dimensional vectors to 'h' different dk,dk and dv dimensional vectors respectively. So basically the pseudocode, from what I understood, should look something like this:
Q,K & V are d-model dimensional vectors.

for i in range(h):
   Qi = Dense(dk)(Q)
   Ki = Dense(dk)(K)
   Vi = Dense(dv)(V)
   Ai = Attention(Qi, Ki, Vi)

A0, A1, A2, ..., Ah are then concatenated.

Is this right? or am I missing something here?

In principle, the pseudocode is correct, but it is not how it is implemented. The projection and dot-product attention can be done efficiently using matrix multiplication only for all heads simultaneously.
Instead of looping over the heads, you can use only one dense layer for queries, one for keys and one for values. E.g., for the keys, you would do a dense layer of dimension $h \cdot d_k$ and then reshape it.
Assuming you have batch size $b$, sequence length $l$ and model dimension $d_m$:

The input of the dense layers is of shape $b \times l \times d_m$.

The output of the dense layer is has shape $b \times l \times hd_k$ (or $d_v$ respectively).

Then you can reshape the queries and keys to have shape $b \times l \times h \times d_k$.

Now, if you permute the dimensions, such that the queries have shape $b \times h \times l \times d_k$ and the keys have shape $b \times h \times d_k \times l$, you can do the batch matrix multiplication in the last two dimensions and you end with attention energies of shape $b \times h \times l \times l$. Then, if you do softmax in the last dimension, you get the attention distribution for each head and each query.
By doing the batch matrix multiplication with projected values that now with transposed dimensions to $b \times h \times l \times d_v$, you get the weighted average. So finally, the "concatenation" of all heads is in fact just another tensor reshaping.


Using historical label as a feature in my ML model? I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of 0.1 and so basically when the rate of change is above a certain threshold it gets a label 1, if below -1, and if not then 0.

Because I am using an exponentially smoothed moving average for the label, am I still allowed to use historical labels as a feature in my dataset? So for example if my predictive variable is the direction for t+5, could I use label at point t? Because there is an overlap in the data used

Correct me if I'm wrong, but it sounds like you are using an EMA to generate the price that you will be using to label as either 1,0, or -1. If so, that is not a problem to do so with historical data, as EMA calculations are only affected by data previous to whatever record the EMA is being based off of. Just make sure that your historical data is calculated as if it was the present moment for every record.


What does 1024 by 3 model mean? I was watching this video  and Sentdex mentioned he had to switch around 1024 by 3 model. What does he mean by 1024 by 3 model and what did he change around?

Edited: Youtube link with the timestamp where he talks about this topic

At 3:30 in video, he mentions this switching. What he is referring to by 'switch around' is that he is specifying the size of the LSTM being used. The 1024 refers to number of cells (or size) of the LSTM, and the 3 is the number of layers.

In the tutorial page being referenced I the snippet he is referencing is:

python translate.py
  --data_dir [your_data_directory] --train_dir [checkpoints_directory]
  --size=256 --num_layers=2 --steps_per_checkpoint=50

For details on why LSTMs are useful for this task, check out the paper.


What is the most straightforward way to visualize color-coded clusters along with the cluster centers? I have applied the kMeans Clustering algorithm to a dataframe and have gained cluster labels for each row. I had selected only two features.
There are 4 clusters.
I want to visualize the datapoints in 2D plane with color-coded clusters which I want to look like this-

Ignore the labels. I would like to plot cluster centers instead.
I have looked in many blog posts, articles etc. None was helpful.
What is the most straightforward to achieve this?

There can be multiple ways, one can be -
- Plot the points with hue=cluster_number
- Plot the Centroid with a different markers
Code for 3 Clusters on 2 Iris Features -
from sklearn import datasets
iris = datasets.load_iris()
X = iris.data
y = iris.target
X=X[:,:-2]
X = (X - X.mean())/X.std()

def create_cluster(k=3):
    from sklearn.cluster import KMeans
    kmeans = KMeans(n_clusters=k)

    kmeans.fit(X)
    return kmeans

kmeans = create_cluster()
y_pred = kmeans.predict(X)
centroid = kmeans.cluster_centers_

_, ax = plt.subplots(1,1,figsize=(10,6))
color = ["#e74c3c", "#34495e", "#2ecc71"]
sns.scatterplot(X[:,0],X[:,1], hue=y_pred, palette=sns.color_palette(color),ax=ax)
sns.pointplot(centroid[:,0], centroid[:,1],markers='^',join=False,ax=ax)

$\hspace{2cm}$Output - Ignore the convergence quality
$\hspace{2cm}$


Spike Slab in r, bad output I've successfully used Spike Slab in the past, but with this data it seems like something is going wrong. My code is:

>require(spikeslab)
>set.seed(2)
>model1_ss<-with(data_use,spikeslab(gdeaths_per_100_thousand~gdealers_per_100_thousand+Physically.Unhealthy.Days+Mentally.Unhealthy.Days, na.rm=TRUE))
>summary(model1_ss)

the first 10 lines of data_use is

 Physically.Unhealthy.Days  Mentally.Unhealthy.Days gdeaths_per_100_thousand    gdealers_per_100_thousand
 5.2        4.1     19.70210418  30.11828855
 4.9        4.1     21.6723146   32.83851989
 5.1        4.3     23.52217417  39.4076506
 5.1        3.6     NA           38.03625359
 3.4        4.1     17.78983533  27.28191219
 3.4        4.2     19.4576324   32.2906761
 3.3        3.8     13.9248167   40.75855495
 3.3        3.8     12.05513916  36.2194827
 5.9        3.9     NA           31.52340694

And it returns this, which is totally unlike what I expect from from Spike Slab:

    Length Class      Mode
summary           4  data.frame list
verbose          10  -none-     list
terms             3  terms      call
sigma.hat         1  -none-     numeric
y              2132  -none-     numeric
xnew          12792  -none-     numeric
x             12792  -none-     numeric
y.center          1  -none-     numeric
x.center          6  -none-     numeric
x.scale           6  -none-     numeric
names             6  -none-     character
bma               6  -none-     numeric
bma.scale         6  -none-     numeric
gnet              6  -none-     numeric
gnet.scale        6  -none-     numeric
gnet.path         2  -none-     list
gnet.obj         16  lars       list
gnet.obj.vars     6  -none-     numeric
gnet.parms        6  -none-     numeric
phat              1  -none-     numeric
complexity      500  -none-     numeric
ridge           500  -none-     list
model           500  -none-     list

Is there a way to interpret this that I don't know about? Or is there something wrong with how I'm running it? Thanks so much for any help, I have no idea what to try.

There seems to be no summary method for objects of class spikeslab, so the default summary method is used which prints the internal list elements.

Just print the object to get a nice summary - maybe that's what you want? Is this the output you expected?

> model1_ss
-------------------------------------------------------------------
Variable selection method     : AIC
Big p small n                 : FALSE
Screen variables              : FALSE
Fast processing               : TRUE
Sample size                   : 7
No. predictors                : 3
No. burn-in values            : 500
No. sampled values            : 500
Estimated mse                 : 7.5765
Model size                    : 2

---> Top variables:
                            bma  gnet bma.scale gnet.scale
Mentally.Unhealthy.Days   2.290 2.331    13.003     13.236
Physically.Unhealthy.Days 0.926 0.951     1.084      1.113
-------------------------------------------------------------------


Combining spatial input with a label as input for CNN using Keras I also asked this question on Stack Overflow. However, it has not yet been answered and I think this is a more suitable platform to place it.

I'm trying to implement a network set-up similar to this Google Deepmind paper. Their network set-up is as follows:

Mθ is a convolutional network, so I was wondering how they concatenate the input frames with the viewpoints? As far as I knew, the CNN takes into account the spatial information right? So does it even make sense to concatenate the frames with the viewpoints as an input for the CNN?

Thanks in advance!

Edit

I was thinking about it and they might have concatenated the viewpoint on the dense layers behind the convolution layers. Might that be the case?

Here the viewpoint ($v_i$) and corresponding frame ($f_i$) are not concatenated. $v_i$ is simply an index of $f_i$. As mentioned in their section 3.1, $v_i$ is the timestamp and $f_i$ is the actual frame (image). The convolution network $M_\theta$ is applied on $f_i$, not $v_i$.

After the convolution network $M_\theta$, the output of different frames $r_i$ are added together (notice the "+" in the figure, as well as the equation $r=r_1+...+r_m$ in section 3.2.


Learning with groups of sequential data Say I have a data set such as the following:

person, Time, Value, Event
person1, 2010-07-02 00:00:00, 5.4, 0
person2, 2010-07-02 10:00:00, 12.7, 0

We have a current model in place at work that doesn't take into account the temporal aspect of our data. In that implementation, the model was trained with only unique values for 'person', and it throws away the time variable. However, it has come to our attention that we can look at our data as a sequence instead. This starting time is unique for each person, and clearly associated with only that person, so merely pretending each person is independent and just treating each row as an individual data point wouldnt make any sense. The following is what I've restructured the data as:

person, Time, Value, Event
person1, 2010-07-02 00:00:00, 5.4, 0
person1, 2010-07-02 00:00:15, 3.6, 0
person1, 2010-07-02 00:00:30, 2.4, 0
person2, 2010-07-02 10:00:00, 12.7, 0
person2, 2010-07-02 10:01:15, 12.8, 0
person2, 2010-07-02 10:01:30, 13.1, 1

This sequence for each person would continue until and 'event' or 'non-event'. I'm totally unfamiliar with machine learning on time series data. All of the examples I've read with different models treat the data as one big sequence corresponding to one entity, while our data clearly doesn't work like that. Is the way I've structured the data the right way to approach a time series model? And if so, what would be an appropriate model to consider?

The data that you are showing is typical survival data. If you want to model the event depending on time and value you should look into survival models with time dependent covariates. If you want to do this without any assumptions on the distribution you can start with a Kaplan Meier estimate as explained here. If you want to use parametric models you can look at Weibull or Gamma regression.

If you are new to this topic I can highly recommend to browse the examples of the packages in the CRAN Survival Task View.


How to estimate the not available observation in time series data? Suppose, I have a 30 seconds time-step observations of sports data, in some of the intervals the game was partially/fully stopped. I'm trying to prep the data for a time series analysis. Is it justified to take it as zero when it was stopped fully? or I have to interpolate the value...

Here is the sample of data created without taking DeadBallMin(game paused) into account...

** columns A and B are actual data observed during the time-step.

** Exp_Win_A and Exp_Win_B are monotonic increasing. And assume all the features are uniformly distributed within the time-step.

    A      B    Exp_win_A   Exp_win_B   DeadBallMin
0   1      0    0.891713    1.074992    0.000000
1   0      1    0.893859    1.076465    0.000000
2   0      1    0.930300    1.077941    0.036633
3   0      1    0.932539    1.112289    0.000000
4   0      0    0.934783    1.122372    0.907834

From the above table, in the second row, the game was stopped for 33.67% of the time-step.

Question

Any suggestions on how to incorporate the 'DeadBallMin' time into  Exp_win_A & Exp_win_B while keeping the behaviour?

One of the most common ways for this purpose is to generate artificial data for that intervals. One of the most novel and powerful algorithms for this purpose is Generative Query Network (GQN).

In this algorithm the basic application is to create a 3D model of an object by observing a few 2D samples of it. So, the network by receiving the angle of the camera beside the 2D image, try to construct the 3D model.

But in this case for time series data, time can play the role of camera angle. So, you can generate high precision artificial data in the case of data lost.

Here is a brief description of this algorithm.

And also here is an implementation of it using Pytorch.

For more information you can read it's respective paper.


Validation loss much higher than training loss I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is:
model = Sequential()
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
model.add(Conv1D(128, 5, activation='relu'))
model.add(GlobalMaxPooling1D())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(25, activation = 'relu'))
#model.add(Dropout(0.2))
model.add(BatchNormalization())

model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

Any help would be appreciated.

Your model is completely overfitting. The training loss is constantly decreasing but the validation loss isn't. This means that the your current model is complex enough to 'memorize' the patterns in the training data. In such situations, you need to regularize your model.

To regularize your data you may try any or all combination of the following:

1) Reduce the layers of the neural network.

2) Reduce the number of neurons in each layer of the network to reduce the number of parameters.

3) Add dropout and tune its rate.

4) Use L2 normalisation on the parameter weights and tune the lambda value.

5) If possible add more data for training.

6) Adding more data may not always be economically viable option so 'Data Augmentation' can play a big role here. Data Augmentation depends on the type of input to the neural nets. For images they comprise of rotation, translation, affine transformation like shear etc.


So, from what I understand you are trying to predict the target column given all your feature columns and the information from the past. This feels like a time-series forecasting problem to me too (You can check variations online).

Vanilla RNN tends to vanish or explode the gradients especially given that you have such a large number of timestamps. I'd suggest you to try an LSTM in order to avoid it as much as you can. Ideally yes, it is supposed to remember what happened earlier and learn from that.