Reducing search iteration over millions of data The problem goes like this, with a story.

You have an application that contains a search field. When you search some input, there's an auto-complete component that pops up showing up similar results to your input.

Each result is a location in the database and this is the structure of location:

id | name                    | lat  | lon
-----------------------------------------
0  | New york street test 51 |34.123| 38.245

Every location has lat and lon coordinates.

Your system has a function that knows how to receive 2 locations and return the distance in km.

You also have a table called stores which contains id, name, lat, lon of the shop.

When you select a place, your application finds 20 closest stores to the location that you have selected.

The problem

Everything works fine for a start, but now the application has grown and filled up with new stores in many locations and you have over one 500 million stores registered in the database. Every search is heavy and overloading the system and it takes a few seconds to a minute to get a result.

How can you make it better, efficient?

Spoiler - My idea to answer the question

   My idea is to split the world map into regions matrix of size Constant where constant is set to 10km x 10km. Each region will have its ID and every shop you add through your platform will get the region id of where you add it to. When you search for lat and lon, the system will go through all regions searching what region contains that coordinate and return the region id, and then you can grab all shops of that region id. But wait, you have a problem, what if there's a closer store in the neighbor regions? Simple, you have 4 points of your region's corners, you can look up for all neighbor regions and compare if there's a closer store. And if you don't have enough stores and you must get 10 stores out of all regions, use the 4 coordinates of the corners as-well and go recursive on all regions until you reach 10 closest.

I am open and happy to hear everyone's thoughts. Note this is not a real story, it was asked in some interview and I took it very interesting to see if there are better and more interesting solutions.

You are on the right track. In an interview situation, this should be a good answer.  Another answer would be to pre-calculate list of 20 stores in ETL (or CRUD services or DB triggers) and store list of N closest stores in master-detail table.

I have implemented these two solutions in projects :

In database, create indices on lat and lon columns AND add a where clause with a bounding box.

SELECT distance_function(cur_lat,cur_lon,lat,lon) as distance

WHERE (lat between cur_lat -1 and cur_lat +1) AND (lon between cur_lon
      -1 and cur_lon +1)

(+Order by, top 20 rows clause depending on SQL dialect)

This solution cuts search space to a bounding box. This operation in log(n) in time complexity (due to B+ tree indices in DB)

For 100 million rows, results should be available in less than 1 ms.

Choice of +-1 four bounding box is arbitrary in this example. In reality, you will have to store this box with each row (1000s of stores in 1 mile in NYC, London etc; 1 store in 100 miles in other locations)

Implement QuadTree https://en.wikipedia.org/wiki/Quadtree . In this context it is similar to your solution but takes care of extreme cases where density of stores varies across geographical regions.


How to count categorical values including zero occurrence? I want to count number of code by month.
This is my example dataframe.

        id    month  code
0     sally    0  s_A
1     sally    0    s_B
2     sally    0   s_C
3     sally    0   s_D
4     sally    0    s_E
5     sally    0   s_A
6     sally    0    s_A
7     sally    0   s_B
8     sally    0   s_C
9     sally    0   s_A

I transformed to this Series using count().

df.groupby(['id', 'code', 'month']).m.count()

id      code   month  count
sally  s_A      0    12
                1    10
                2     3
                7    15

But, I want to include zero occurrence, like this.

id      code   month  count
sally  s_A      0    12
                1    10
                2     3
                3    0
                4    0
                5    0
                6    0
                7    15
                8    0
                9    0
                10   0
                11   0

Based on the short example DataFrame you provided, this block of code will include all of the months. It is based on using the Series.reindex method and creating a new MultiIndex with the additional values for the months:

import pandas as pd
# Load example data into DataFrame
df = pd.read_table("categorical_data.txt", delim_whitespace=True)

# Transform to a count
count = df.groupby(['id', 'code', 'month']).month.count()

# Re-create a new array of levels, now including all 12 months
levels = [count.index.levels[0].values, count.index.levels[1].values, range(12)]
new_index = pd.MultiIndex.from_product(levels, names=count.index.names)

# Reindex the count and fill empty values with zero (NaN by default)
count = count.reindex(new_index, fill_value=0)
print(count)

Printing the result, I get something like this (only showing the first entry for sally/s_A):

id     code  month
sally  s_A   0        4
             1        0
             2        0
             3        0
             4        0
             5        0
             6        0
             7        0
             8        0
             9        0
             10       0
             11       0


What makes binary cross entropy a better choice for binary classification than other loss functions? I'm reading this
post where I came across this quote "Cross-entropy is the default loss function to use for binary classification problems."

But what about it makes it the default and presumably best loss function for binary classification?

As mentioned in the blog, cross entropy is used because it is equivalent to fitting the model using maximum likelihood estimation. This on the other hand can be interpreted as minimizing the dissimilarity between the empirical distribution of training data and the the distribution induced by the model (measured as Kullback-Leibler divergence, hence the name cross-entropy).

Now you may ask why maximizing the likelihood. Maximum Likelihood estimators have nice asymptotical properties (they are the best estimators in terms of convergence rates), they are consistent and statistically efficient.

Another advantage is that you don't need to think much about how to the define a cost function. The ML framework gives you the cost function right away as long as you have a model that specifies $P(y|x)$.

Also, directly optimizing the cross-entropy leads naturally to well-calibrated probabilities compared to, say, a random forest. That means you can interpret the resulting probabilities as you would expect, namely that of all the cases with about x% predicted probability, x% should belong to the positive class.

There is another important point. Given the prevalence of gradient-based learning algorithms (especially in deep learning), the logarithm in the cross-entropy will undo any exponential behavior that is often given through popular activation/output units like the sigmoid/softmax. Thus, the logarithm will avoid that the gradient saturates for extreme values. Large gradients are important for gradient-descent algorithms to make sufficient progress in each iteration. (There are more details, for example, in the "deep learning book" in chapter 6.2)


fix first two levels of decision tree? I am trying to build a regression tree with 70 attributes where the business team wants to fix the first two levels namely country and product type.To achieve this,I have two proposals:

1.Build a separate tree for each combination of country & product type and use subsets of the data accordingly and pass on to respective tree for prediction.Saw here in comments.I have 88 levels in country and 3 levels in product type so it will generate 264 trees.

2.Build a basic tree with two variables namely country and product type with appropriate cp value to generate all combination as leaf nodes(264).Build a second tree with rest all variables and stack tree one upon tree two as a single decision tree.

I don't think the first one is the right way to do.Also, struck on how to stack the trees in second approach, even if it is not the right way would love to know how to achieve this.

Please guide me to approach the problem.Thanks.

Depending which tree algorithm you want to use you could manually construct the two first levels of the tree. You can just follow the pseudo code explained for example here for the C4.5 tree. Once you have done this you can remove the two features from the data set and create trees for the remaining part of the tree. If you want to create a rpart object you would be required to take some parts of the source and this may be a bit more demanding. Depending on what tree algorithm you use you will just have a binary split at both levels so you will only need to build 4 separate trees and not 264. Note that you may not have the optimal decision tree since after stepping through the first two levels, the country and product type may still be variables that cause a split. But without seeing the data is impossible to tell.

Side note, it may be valuable to explain the business that country and product type are not the most sensible variables to have in the top of the decision tree. Sometimes it is better to educate the end users than to force machine learning to do something inaccurate. In my experience end users prefer to have a correct solution than a solution that works because people have a gut feeling that it should be in a certain way.


Ideal gas data needed I want to do a quick computation in R that involves estimating the ideal gas constant from experimental data.  Ideally I'd have data from a monatomic gas at different pressures, volumes, temperatures and moles (or just pressure and volume, keeping the others fixed).  Anyone have an idea of where I might find a data set like this?  An R data frame would be nice, but I'll take excel, tab delimited, or whatever.

The National Institute of Standards and Technology (NIST) has ideal gas datasets here


What is the human performance on ImageNet, the top 1 error? (not top 5 error) What is the human performance on ImageNet, the top 1 error? (not top 5 error).
The best of the machine learning algorithm top 1 error is 11.6% ,according to thie website https://paperswithcode.com/sota/image-classification-on-imagenet

.

The ImageNet top-1 error rate for humans is unknown.  Russakovsky et al and Andrej Karpathy only discussed the ImageNet top-5 error rate for humans.


R and Python, memory differences yielding performance differences I think there are numerous posts regarding which one to use: R or Python. However, I'm curious about how their architecture differences yield differences in speed performance, not which one to use.

This blog post performs a small test between R and python to show that the (optimized) python code was 2x faster than R code.* And I've read in this post that R tends to put everything in memory, which is why computations on large datasets is generally slow.

But what makes python's low level memory management so much different than R, which helps it yield these benchmarks?

*Though python was 2x faster in this test than R, I'm not saying that python is generally 2x faster than R.

It depends on the usage of various packages (Numpy/Scipy etc), which are written in C and be incredibly fast also Python can be complied using JIT.  Here is an excellent comparison between R and Python : https://learnanalyticshere.wordpress.com/2015/05/14/clash-of-the-titans-r-vs-python/


Need help understanding the structure of this convoluted neural network I'm trying to grasp the structure of this convoluted neural network.

(Source)
I understand the first layer is a 6x6 conv with stride 2 followed by 3x3 max pool and then 6 5x5 convs and another 3x3 max pool. After this, however, outputs from a fully connected layer of 64 neurons are "tiled over the special dimensions of the response map of pool2".
I don't understand what this means. The output of pool2 should be 64 (because of 64 filters) 18x18 arrays. In the first 18x18 array do I add output1 to each of the 18*18=324 values, and in the second array add output2 to each of the 324 values, etc.?
TLDR
What do I do with the 64 outputs (each is a 18x18 array) and the 64 outputs from the fully connected layer?

The so-called motor command $v_t$ (I don't know what it means but it looks to be some scalar feature although it would work the same if it's a vector) is fed into a layer that builds 64 representations of this value, one for each feature map in the convolutional layer that we are going to add it too. The conv layers have a spatial resolution however, and this representation is only one number for the corresponding feature map. What they do is tile this number so that this whole motor command representation has the same dimensions as the max pooling layer after the convolutional layer (pool2). Now that the dimensions match we can use an element-wise addition operation to inject this information into the convolutional network.


Methods for Determining Possible Causation Between Two Time Series I'll set this question up as a simplified example:

We hypothesize that there is a causal relationship between average gasoline prices and road traffic in a particular city.  The data cover the same time period at the same frequency (e.g.: weekly) and contain no missing values. We'll put aside the idea that there could be confounding variables at work, presume that we've made the time series stationary (via differencing), and determine there is no autocorrelation in the series.

What are reliable methods to determine that if a shock occurs in the price data that a corresponding shock occurs in the traffic data at some time lag?

I've looked a ;little bit into Dynamic Time Warping, but I suspect that this is better suited to determining how to map one series to another rather than for determining causality.

I find two possible solutions: Granger Causality and Convergent Cross Mapping (CCM).

Granger Causality is based on the t-test of lagged value of first variable with the second variable. The limitation is that it could not eliminate the confounding effect.

The CCM first build the manifold for the first variable X. Then for each time point, weights of neighbors are estimated, and used to estimate Y. Then correlation between estimated Y and Y are tested.


Building Timeseries models for stock trading having multiple stocks I have gone through some of the tutorials on the timeseries and all of them have taken one stock for the timeseries and tried to forecast it. My dataset contains many stocks for the time period(each day). Do i need to choose one stock for or some of them or all of them for training the model?

This is a question about Univariate versus Multivariate Time Series analysis. Both can be valid approaches.
However, for stock price forecasting I would suggest that Multivariate will probably be better, with the important caveat that your choice of time series should have some fundamental justification. Randomly choosing to analyse stock A and B together from a pool of 'N' thousands of potential stock pairs/triplets etc. will not be as effective as choosing stock X and Y if you believe X and Y are related in some way, i.e. they are in the same industry sector.


How to find possible subjects for given verb in everyday object domain I am asking for tools (possibly in NLTK) or papers that talk about the following:

e.g. Input: Vase(Subject1) put(verb)

Ans I am looking for: flower, water

Is there a tool that can output subjects (objects) that can be associated to this verb? (I was going through VerbNet but didn't find anything)

If you want something quick, I think pattern is the best tool for the job. It provides a ready-to-use multilingual parser that you can use in the following way:

import pattern
from pattern.en import parse
s = 'I put water in the vase'
s = parse(s)
print s
# output = I/PRP/B-NP/O put/VBP/B-VP/O water/NN/B-NP/O in/IN/B-PP/B-PNP the/DT/B-NP/I-PNP vase/NN/I-NP/I-PNP

Once you have a string like output above, you only need regex parsing to extract every sequence of tokens whose tags match the sequence [B-NP, B-VP, B-NP].

NP stands for "noun phrase" and VP stands for "verb phrase". In English, virtually every sequence consisting of a noun phrase, a verb phrase, and a second noun phrase, all in strict adjacency, is a subject-verb-object sequence, so this should give you what you're looking for.

pattern's parser will also be able to handle some non-strict adjacencies (e.g. intervening adverbs and adjectives between the three phrases in the subject-verb-object sequence).

However, pattern is not terribly sophisticated -this will give you some Precision and some Recall, but not terribly high numbers. If you need high-quality parsing, you should try the Stanford parser's Python implementation or spacy.

Hope this helps!


What is the difference between data singular and data plural ? I don't understand the difference between these terms and google didn't help much. Please answer the question even if you have any slightest idea.

Data is plural, as in 'These data are depicted below'. Singular is 'datum' but it's rare one would have occasion to use the term.


Spectral Networks and Deep Locally Connected Networks on Graphs I’m reading the paper Spectral Networks and Deep Locally Connected Networks on Graphs and I’m having a hard time understanding the notation shown in the picture below (the scribbles are mine):

Specifically, I don’t understand the notation for the matrix F. Why does it include an i and a j?

k is a fixed parameter indicating the input layer and i,j refer to the index of the matrix F , used to traverse throught the matrix.


Consensus clustering: how to choose the final cluster based on the consensus matrix? I've been reading about consensus clustering and the consensus matrix in this article. I understand how the consensus matrix is made after re-sampling and clustering parts of your data H times. I understand that the consensus matrix is used to determine the optimal amount of clusters (k) and allows for making a nice heatmap. What I don't understand is how this consensus matrix results in your final clustering.
Say I have a consensus matrix of 4*4 (so we have 4 items to be clustered) where each value between 0 and 1 in the matrix represents the number of times items i and j are assigned to the same cluster, divided by the total numer of times both items are selected for clustering. We could have the following consensus matrix after 4 iterations of 80% sub-samples (taken from here). We kept track of all 4 clusters that were made in the process of obtaining this consensus matrix. How do we select the final clustering based on this consensus matrix?

Consensus clustering is an additional evaluation technique (i.e. how much consensus is there between clusters), like the Silhouette score, rather than explicit clustering.
The clusters themselves would drop out earlier in the processing. The consensus metric is used to test for the best K value, in the K-means clustering algorithm.


Silhouette calculation in k-means I am trying to compute Silhouette with k-means. However I have the value really close to 0 and the clusters are very clearly separated. Do you know where can be the problem? This is the code:

n_samples, n_features = data.shape
n_digits=2
labels=data[:,-1]
data=data[:,:-1]
sample_size = 300

print("n_digits: %d, \t n_samples %d, \t n_features %d"
      % (n_digits, n_samples, n_features))
print(79 * '_')
print('% 9s' % 'init'
      '    time  inertia    homo   compl  v-meas     ARI AMI  silhouette')

def bench_k_means(estimator, name, data):
    t0 = time()
    estimator.fit(data)
    print('% 9s   %.2fs    %i   %.3f   %.3f   %.3f   %.3f   %.3f    %.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=sample_size)))

pca = PCA(n_components=n_digits).fit(data)
bench_k_means(KMeans(init=pca.components_, n_clusters=n_digits, n_init=1),
              name="PCA-based",
              data=data)

print(79 * '_')

The obtained Silhouette is 0.052 and this  is the obtained k-means clustering.

Thanks,

Laia

Since the points in each cluster are fairly spread out along the direction of the separating line, the average distance within a cluster is probably not that much smaller (as you might hope) than the average distance to points in the other cluster. A small positive value could be plausible. (Maybe look at the distribution of the points along the second principal component. A big spread could be the problem.)


pandas: How to impute the categorical column by the nearest neighbors? I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in the neighborhood. In the beginning of the input signal you can see nans embedded in an otherwise continuum 's' episode. My definition as to what characterizes an episode is occurrence of the corresponding symbol at least 5 times in a row. Also, in my interest to be given more weight to 'r' and 'l' when tied with 's'.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
oput = ['s','s','s','s','s','s','s','s','s','s','s','s','s','r','r','r','r','r','r','r','s','s','s','s','s','s','s','s','s','s','l','l','l','l','l','l','l','l','l']

I tried knn as following but it is rather suitable for numerical column and also imputing nans with zeros.
I was hoping for some ideas how to tackle this problem.
from fancyimpute import KNN
knnimpute = KNN(k=5)

>>>x = np.array([0,np.nan,1,1,1,np.nan,2,2,2,2,np.nan,2,3,3,3,3,np.nan,1,1,2,2,np.nan,1,3,3,3,3])
>>>x2 = knnimpute.fit_transform(x.reshape(-1,1))
>>>x2
>>>
array([[0.],
           [0.],
           [1.],
           [1.],
           [1.],
           [0.],
           [2.],
           [2.],
           [2.],
           [2.],
           [0.],
           [2.],
           [3.],
           [3.],
           [3.],
           [3.],
           [0.],
           [1.],
           [1.],
           [2.],
           [2.],
           [0.],
           [1.],
           [3.],
           [3.],
           [3.],
           [3.]])

The following script will give the value of the most frequent item to the nan value. It is a list of 7 items, since it checks the three samples before the nan, the nan itself and the three after the nan samples.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
for i in range(len(iput)):
    if type(iput[i]) is float:
        iput[i]=max(iput[i-3:i+3],key=iput[i-3:i+3].count)


Create data visualization for unstructured data - Basket Market Analysis I have this dataset (just a sample):

product1,product2,product3
product1,product4
product1,product2
product4,product3,product1,product2

The products are grouped by transaction. I want to create some data visualization using this dataset but I don't any tool or any type of visualization that allows creating some visualization with this structure...

Anyone can suggest me an option?

Thanks!

I feel desperate because they can not find anything that suits this data structure

Considering data stored in CSV format like below without headers you can use below R code to plot simple bar chart. It will plot occurrences of transactions grouped by unique transactions.

product1,product2,product3
product1,product4
product1,product2
product4,product3,product1,product2
product1,product2,product3
product1,product4
product1,product4

R Code -

transactions <- read.csv("filepath/transactions.csv", header = FALSE)
transactions$V1
plot(transactions$V1)

Bar Chart-


Undestanding Bayesian network with OpenMarkov I downloaded OpenMarkov software for  probabilistic graphical models and tried it on mtcars dataset.

The mtcars.csv data looks like this:

In OpenMarkov GUI, I went to Tools > Learning and loaded mtcars.csv dataset. I then adjusted preprocessing settings to have Discretize with Equal width intervals for all variables.

I then chose Hill Climbing algorithm (default) and Automatic learning options. On learning, the result was as follows:

My question is what exactly does this figure represent? Does it represent a Bayesian network or some other type of probabilistic graphical models? Also, do arrows mean that hp affects cyl and carb; and cyl in turn affects disp and carb and so on?

First off, I did not know OpenMarkov. Anyway, from its website it has a particular focus on learning Bayesian networks (Bayes nets). Thus, I assume your figure represents a Bayes net, yes. Syntactically, it also qualifies since it is a directed acyclic graph.

The arrows (edges) represent influences (conditional dependencies) observed in the data.
For instance, the conditional probability distribution of carb, P(carb | hp,cyl,disp), is defined by the values for hp,cyl,disp. However, arrows do not necessarily represent causal relationships.


Is it possible to customize the activation function in scikit-learn's MLPRegressor? Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway.

Is it possible to change the activation function of the output layer in an MLPRegressor neural network in scikit-learn?

I would like to use it for function approximation.  I.e.

y = f(x)

where x is a vector of no more than 10 variables and y is a single continuous variable.

So I would like to change the output activation to linear or tanh.  Right now it looks like sigmoid.

If not, I fail to see how you can use scikit-learn for anything other than classification which would be a shame.

Yes, I realise I could use tensorflow or PyTorch but my application is so basic I think scikit learn would be perfect fit (pardon the pun there).

Is it possible to build a more customized network with MultiLayerPerceptron or perhaps from individual layers (sknn.mlp)?

UPDATE:

In the documentation for MultiLayerPerceptron it does say:

  For output layers, you can use the following layer types: Linear or
  Softmax.

But then further down it says:

  When using the multi-layer perceptron, you should initialize a
  Regressor or a Classifier directly.

And there is no example of how to instantiate a MultiLayerPerceptron object.

I see in the code for the MLPRegressor, that the final activation comes from a general initialisation function in the parent class: BaseMultiLayerPerceptron, and the logic for what you want is shown around Line 271.

# Output for regression
if not is_classifier(self):
    self.out_activation_ = 'identity'
# Output for multi class
...

Then during a foward pass this self.out_activation_ is called (defined here):

# For the last layer
output_activation = ACTIVATIONS[self.out_activation_]
activations[i + 1] = output_activation(activations[i + 1])

That ominous looking variable ACTIVATIONS is simply a dictionary ,with the keywords being the descriptions you can choose as a parameter in your MLP, each mapping an actual function. Here is the dictionary:

ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,
               'relu': relu, 'softmax': softmax}

With all of this information, you might be able to come up with a few ways of putting in your custom function. Off the top of my head, I can't see a quick way to simply provide a function. You could for example:

define your function where all the other activation functions are defined
add it to that ACTIVATIONS dictionary
make self.out_activation_ equal to your custom function (or even a new parameter in MLPRegressor
cross your fingers it doesn't break something somewhere else
run it and solve the inevitable small adaptations that will be necessary in a few places

I'm, afraid I have never looked at the source code of that library before, so cannot give more nuanced advice. Perhaps there is a beautifully elegant way to do it that we have both overlooked.


Classifying job titles I have a dataset of about 10000 unlabeled job titles (mostly very short titles) such as head of mobile or lead iOS developer.

I would like to perform classification of those job titles in two different ways :

The 1st classification is sorting job titles according to the type of job it represents (i.e. marketing, IT, healthcare, legal...)

The second classification would be about the level of seniority of a specific job (i.e. executive, manager, associate, trainee...)

I have tried an approach based on Word2Vec (vectors were taken from the Google Word2Vec set)  :

I cleaned and stemmed job titles
I created a vector representing the job title as the average vector of the words that compose it
I created a list of vectors representing the target values (executive could be the average vector of +executive, +chief, -assistant for example)
Find the least cosine distance between the job title vector and the target vectors.

While this approach gets decent result (about 70-80% accuracy) it is not enough for the task I'm planning to do.

So I was wondering if a better approach could be used (except labeling the data by hand and using that to train some algorithm)

I'm surprised you got 80% accuracy and think you will have to do some labeling if you want to improve. Even people have problems with this! For example:

Sales Executive - executive can be misleading, this might have been called "Salesman" 50 years ago
VP at a technology company vs VP at a bank. In the latter case, VP is more of an honorific and might not include any actual management.

If you don't want to mass label data, I would try to assign weights to common words that are featured in titles. For example, replace senior/junior with +1 and -1 in a "seniority" value. If you do that for the top 100 words, I bet it will help over a totally unsupervised approach.


How to extract specific parts of text from a string? For example:
Here is a textual input:

"ALL imagery SINCE 1952
20 MULE T
aerial BOOSTER
& Multi-Purpose Neutralizer.MAY BE â€œAHMFLL |-E SWALL """""" -5 NETWT4LBS1 DZ (65 OZ) 1.84 kg

"
Output desired:
4 LBS
65 OZ
1.84 kg
"

It is a specific pattern which can be detected by numbers and after that a unit name with two or three letters.
In the other words, detect numbers first, and then find numbers which after of them there is two or three letters.
Moreover, you can check these letters are in a predefined list of units or not.


How to model a binary classification problem in a evolving environment Suppose you have a problem with two classes YES or NO. While the YES class is fixed, in the sense that the observations do not evolve, the observations of class NO may evolve during time and may change to the class YES. I would like to predict the probability of a given observation change to YES.

What kind of ML models could be used here? Could Bayesian Networks be a solution?

I would also like to observe what changed in the variables (or which variables were most important) that turned a NO into a YES.

Thank you

If you are trying to predict something that looks like probabilities, but don't actually need a "real" probability (for instance, if you aim at ranking observations to find those who are the most likely to switch to no), you could probably use logistic regression over the dataset. This would require the model to be refitted every once in a while.

If you are looking for actual probability, it has to be a probability to switch from NO to YES over a period of time. In that case, you are falling into time series analysis. This is a more complicated problem.

My intuition would be to consider the state change as a physical random phenomenon, such as a radioactive decay, where the probability of having decayed over a $\Delta t$ time period is $e^{-\lambda \Delta t}$. Based on cases who have switched in the past, you could build a rgression model to predict a $\lambda$ value for each observation. Of course you can use other decay functions than the exponential (find the best one through the validation step). This will eventually allow predicting a switch probability over any given time period.


Classifying Java exceptions We have a classification algorithm to categorize Java exceptions in Production.
This algorithm is based on hierarchical human defined rules so when a bunch of text forming an exception comes up, it determines what kind of exception is (development, availability, configuration, etc.) and the responsible component (the most inner component responsible of the exception). In Java an exception can have several causing exceptions, and the whole must be analyzed.

For example, given the following example exception:

com.myapp.CustomException: Error printing ...
... (stack)
Caused by: com.foo.webservice.RemoteException: Unable to communicate ...
... (stack)
Caused by: com.acme.PrintException: PrintServer002: Timeout ....
... (stack)

First of all, our algorithm splits the whole stack in three isolated exceptions. Afterwards it starts analyzing these exceptions starting from the most inner one. In this case, it determines that this exception (the second caused by) is of type Availability and that the responsible component is a "print server". This is because there is a rule that matches containing the word Timeout associated to the Availability type. There is also a rule that matches com.acme.PrintException and determines that the responsible component is a print server. As all the information needed is determined using only the most inner exception, the upper exceptions are ignored, but this is not always the case.

As you can see this kind of approximation is very complex (and chaotic) as a human have to create new rules as new exceptions appear. Besides, the new rules have to be compatible with the current ones because a new rule for classifying a new exception must not change the classification of any of the already classified exceptions.

We are thinking about using Machine Learning to automate this process. Obviously, I am not asking for a solution here as I know the complexity but I'd really appreciate some advice to achieve our goal.

First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind.

Any supervised ML algorithm consists of at least 2 components:

Dataset to train and test on.
Algorithm(s) to handle these data.

Training dataset consists of a set of pairs (x, y), where x is a vector of features and y is predicted variable. Predicted variable is just what you want to know, i.e. in your case it is exception type. Features are more tricky. You cannot just throw raw text into an algorithm, you need to extract meaningful parts of it and organize them as feature vectors first. You've already mentioned a couple of useful features - exception class name (e.g. com.acme.PrintException) and contained words ("Timeout"). All you need is to translate your row exceptions (and human-categorized exception types) into suitable dataset, e.g.:

ex_class                  contains_timeout  ...   | ex_type
-----------------------------------------------------------
[com.acme.PrintException, 1                , ...] | Availability
[java.lang.Exception    , 0                , ...] | Network
 ...

This representation is already much better for ML algorithms. But which one to take?

Taking into account nature of the task and your current approach natural choice is to use decision trees. This class of algorithms will compute optimal decision criteria for all your exception types and print out resulting tree. This is especially useful, because you will have possibility to manually inspect how decision is made and see how much it corresponds to your manually-crafted rules.

There's, however, possibility that some exceptions with exactly the same features will belong to different exception types. In this case probabilistic approach may work well. Despite its name, Naive Bayes classifier works pretty well in most cases. There's one issue with NB and our dataset representation, though: dataset contains categorical variables, and Naive Bayes can work with numerical attributes only*. Standard way to overcome this problem is to use dummy variables. In short, dummy variables are binary variables that simply indicate whether specific category presents or not. For example, single variable ex_class with values {com.acme.PrintException, java.lang.Exception, ...}, etc. may be split into several variables ex_class_printexception, ex_class_exception, etc. with values {0, 1}:

ex_class_printexception  ex_class_exception  contains_timeout | ex_type
-----------------------------------------------------------------------
[1,                    , 0                 , 1              ] | Availability
[0,                    , 1                 , 0              ] | Network

One last algorithm to try is Support Vector Machines (SVM). It neither provides helpful visualisation, nor is probabilistic, but often gives superior results.

* - in fact, neither Bayes theorem, nor Naive Bayes itself state anything about variable type, but most software packages that come to mind rely on numerical features.


Using Diebold-Mariano test I've got predicted results from two different types of neural networks. Now I would like to run significance testing on both of the results to prove that they do not have equal predictive accuracy. I've learnt that the only tool in the game for this is Diebold-Mariano test. What tool I can use to run this testing (Matlab? R?)

So you want to do a Diebold-Mariano test eh? How about the Diebold-Mariano test dm.test function in the forecast package of R?

dm.test {forecast}

Diebold-Mariano test for predictive accuracy
Package:
forecast
Version:
6.2
Description

The Diebold-Mariano test compares the forecast accuracy of two forecast methods.
Usage

dm.test(e1, e2, alternative=c("two.sided","less","greater"),
    h=1, power=2)

(Took me ten seconds to find)


Problems with Linear Discriminant Analysis Classifier I wrote two functions for determining the linear discriminant classifier of a EEG data-set. The data set consists preprocessed EEG data ∈5×62×5322 and stimulus labels ∈2×5322 during a copy-spelling paradigm with a P300 speller. The data matrix X contains 5 selected time windows of EEG activity at 62 electrodes after a visual stimulus was presented on the screen in front of the subject. If the first row of  is 1, the stimulus was a target stimulus, if the second row of  is 1, the stimulus was a non-target stimulus. The first function returns the weight vector and the bias term. The second function is a graph class to show the result

def lda_fit(X,Y):

# class means

unique_classes=np.unique(Y)
mu=np.zeros((len(unique_classes),X.shape[1]))
for i,name in enumerate(unique_classes):
    mu[i,:] = X[Y==name,:].mean(axis=0)

mupos=mu[1]
muneg=mu[0]
mupos=mupos.reshape(155,2)
muneg=muneg.reshape(155,2)
Xneu=X[0].reshape(155,2)

# D-by-D inter class covariance matrix (signal)
Sinter = np.dot((muneg-mupos),(muneg-mupos).T)

# D-by-D intra class covariance matrices (noise)
Sintra =np.dot((Xneu-mupos),(Xneu-mupos).T)+np.dot((Xneu-muneg),(Xneu-muneg).T)

# solve eigenproblem
eigvals, eigvecs = sp.linalg.eig(Sinter,Sintra)
w = eigvecs[:,eigvals.argmax()]
# bias term
b = (w.dot(mupos) + w.dot(muneg))/2.
# return the weight vector
return w,b

I get the following error: "ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 155 is different from 310)"

I know it has something to do with the shape of the matrices, but I really stuck.

As you say; it has something to do with the shape of the matrices. In this case, you have matrices in the same of (155,2) - whereas it's expecting them to be of (310, 1).

Try looking into the shape of each matrix as you transform them (tip: insert a print statement wherever a transformation occurs); and make sure everything stays consistent.

Also, can you provide us with more information, such as exactly where this error occurs? It looks to be an issue with the dot product.


Retrain final layer of Inception model I'm trying to retrain Inception model final layer for a binary classification.

My training image set contain 2000 images in class 1 and more than 6000 images in class 2.

Will this huge difference in number of images of each class in training set affect my classification?

The short answer is yes.

While your classes are imbalanced, model will be more likely to "learn" class 2 during the training, especially when it comes to mini-batch updates. (even though each mini-batch is unlikely to have only class 2 samples)

The common solution is to use weighted loss function or feed weighted samples to mini-batch.

In my practice, I always use weighted samples if I have sufficient data, even when my classes are not that skewed. Weights always help, at least it never ruins the model.


Where is my error in understanding gradient descent calculated two different ways? The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical justification for this.

For one weight, this is w' = w - r*dC/dw.

Second, we also have this equation deltaC ~= sum(dC/dw(i) * deltaw(i) ), which is just the definition of linearity of C near the point that its derivative is calculated. For one weight, this is the same as deltaC/deltaw ~= dC/dw, e.g., the definition of derivative.

Let there be only one weight, and let s = -deltaC. Then we have -s = dC/dw * (w'-w), where we've split up deltaw into the original and perturbed value. Then w'-w = s * (1/ (dC/dw)), and w' = w -s * (1/dC/dw). (Since we want to reduce the cost, we want deltaC to be <= 0, so s is >= 0, and looks like a normal positive learning rate.)

What I haven't been able to understand is why I get two different answers for what appears to be the same operation, updating the weights to lower the cost. In one case I use dC/dw, and in the other, I use 1/(dC/dw.) In both cases, r and s are small positive numbers.

What am I missing?

Your second function is not an update rule, it is just a re-statement of the approximation between using finite differences to measure a gradient and the infinitesimal ones used in calculus. Most importantly, $s$ is not a learning rate or parameter that you are free to change. It is, by definition, the (negative of the) change observed in $C$ when you alter $w$ to $w'$.

The second equation you have derived is the same as:

$$w + \Delta w \approx w + \Delta C \frac{dw}{dC}$$

In addition, for some reason you decided to drop the approximation symbol partway through and treat your manipulations as if you had created a new weight update rule. Be careful when manipulating expressions on either side of an approximation, they do not always follow the same rules as for an equality, and treating them as such can lead you down some false paths.


Python: Detect if data of a time series stays constant, increases or decreases i need to analyse and later try to improve (integrate a filter) for measurement data that i compare to accurate reference data with python.

First i want to calculate the mean offset and the standard deviation of the measurement data overall, during halt, increasing and decreasing state.

How can i automatically detect and mark sections in which the reference system data stays constant, increases or decreases and use this information later for the statistical analysis.

I already tried to write a simple algorithm myself but the problem is though the reference data is given precise enough so you can calculate the difference of two neighbouring data points and it is zero if no change takes place this case is also in the majority during sections of change (increase, decrease). So the algorithm can't really mark the state of the data points reliable.

I already researched a little bit and here are some keywords that seemed interesting for my problem: time series analysis, anomaly detection, novelty detection, change point detection, structural changes, One-Class Support Vector Machines

So my question would be if sb. could give me a pointer towards the right direction (correct method, python package, tutorial, example) so i can solve this problem myself.

First i want to calculate the mean offset and the standard deviation
  of the measurement data overall, during halt, increasing and
  decreasing state.

I would recommend checking windows of your dataset (let's say 10 samples in a row). A very convenient method to do that is pandas.DataFrame.rolling(), please take a look at it :)


Name Anonymization Software Although I have seen a few good questions asked about data anonymization, I was wondering if there were answers to this more specific variant.

I am seeking a tool (or to design one) that will anonymize human names from a specific country: particularly first names in unstructured text. Many of the tools that I have seen have considered the wider dimensions of data anonymization; with an equal focus on dates of birth, addresses, etc.

An imperative aspect is that it needs to have near absolute recall. The major pitfalls, as far as I can see, are diminutive variants ("Tommy" instead of "Thomas", "Ben" instead of "Benjamin", etc.) and typos. These two factors prevent a simple regex based on a database of names (based on censuses, etc.)

You have a few problems here.  The first is cleaning your data.  That's a whole separate issue form anonymization and belongs in another question if you're still having problems with it.

The second is your anonymization.  After you have some sort of identifier you're satisfied with (sounds like you're using people's real names), try hashing their names to generate a new id.  This id is useful because you'll always be able to take the original name and figure out what id it is but won't be able to derive the real names from just the hashed id (providing your hashing algorithm is good).

Further reading:

https://security.stackexchange.com/a/61878
https://stackoverflow.com/a/21563966/4435034


difference between supervised learning and imitation learning I find these two concepts confusing because I feel that imitation learning is just a 'subset' of supervised learning. But after thinking hard enough, I could not think of any difference and feel that they are the same.
Are they the same? or different from each other?

In supervised learning each data example has a label.
Imitation learning is mapping from observations to actions and is generally considered part of reinforcement learning.
The primary difference between supervised learning and reinforcement learning are when the labels / rewards are available. Reinforcement learning labels / rewards are often time delayed. In supervised learning, the labels / rewards have to be available concurrently with all the data. In reinforcement learning, the agent might make several actions before receiving a reward.


Large categorical dataset for regression I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not find any suitable one.

Does anybody know of any such dataset, or of any additional dataset repository on the Internet?

I would like to recommend to check the following open data repositories and meta-repositories (they are not focused on categorical data, but I'm sure that many data sets, listed there, contain such data):

http://www.kdnuggets.com/datasets
http://www.data.gov
http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free

Also check built-in data sets in the open source software Parallel Sets, which is focused on the categorical data visualization: https://eagereyes.org/parallel-sets.


human level performance on ImageNet, top-1 or top-5? Anyone have pointers to where the human level performance on ImageNet comes from?

I found a reference to 5.1% accuracy (top-1? or top-5?) from here.

It comes from this paper: https://arxiv.org/abs/1409.0575

O. Russakovsky "ImageNet Large Scale Visual Recognition Challenge" 2014


Multiple linear regression, fMRI Say I have an fMRI experiment where I have a task that is suppose to measure response time, and the corresponding "activation" in the brain. I then have participants complete this task. Then I measure some variables through tests or assessments, and questionaries such as IQ, working memory, age, and years of education.

In fMRI experiments, as I am aware typically voxels (the name for the 3d fmri representation of a pixel) that remain "active" after statistical tests, while the participant completes a task, say something about how that brain region where those voxels are modulates that cognitive task.

Now, what if I want to use the variables that I measured, IQ, age, working memory, and years of education in a multiple linear regression test, with the brain activation to this task as the response variable, or dependent variable. What does this tell me?

To put it simply, it will give you a equation (less error relative to data-set presumed to be correct equation). Depending on data-set, equation has constants value for each variables which will provides the ratio of one variable with other variables.
You might find equation on similar format as this:

y = (b0 + b1.x1 + b2.x2 + b3.x3) + c

where x1  , x2 , x3 can be different characteristics (as variables which can be IQ, age, working momory, etc.) and b0 , b1, b2, b3 are the constant calculated from given data-set.

You can find exact value of y or x1 or x2 or x3 if you have exact value of other 3 variables.

Lets say, it was the data-set provided to an android/robot as in sci-fi movies; that android/robot will make it's decision based on this equation as law/fact/pattern and judge a person response time (if every other variables are known) or IQ (if every other variables are known) etc. by this equation.

There is also a lot of details in this.

Note: This logic will only practical with Multiple Linear Regression Method.


What are the main distribution semantics based algorithms? I am aware that LSI, RRI and word embeddings are distributional semantics models. However, I am not certain if the below mentioned are also distributional semantic models.

Non-Negative Tensor Factorisation
Singular Value Decomposition (SVD)
Vector Space Model (VSM)

Please let me know if the above mentioned algorithms are also distributional semantics models. Moreover, please also let me know the other distributional semantics based algorithms.

The main idea here is: "birds of a feather flock together" That is, words that appear near each other inform the "function" of a word.

More importantly, I think of the techniques you mentioned as "methods," rather than "models." The reason why is it seems possible to violate the definition of what a distributional semantic model is without appropriate data preprocessing.

SVD for example, is typically a dimensionality reduction technique, or the pre-cursor to a clustering technique depending on feature engineering and usage of the methodology. In this case, if you were counting the co-occurrence of words across documents -- rows = documents, cols = words, cells = # of times that word appeared in that particular document -- and then you ran SVD on that, you might have the pre-cursor to a "model" that may eventually be something you can call distributional semantic.

Another example could be Word2Vec, where one typically uses a neural network to train a shallow layer, and extract the weights. Word2Vec can be trained via Skip-gram, or continuous bag of words. Since the "meaning" of a word is derived from the co-occurrence and/or proximity to neighboring words, it may be considered a distributional semantic model. FastText is probably even more so, since it explicitly uses the distribution of words in documents to perform a similar vector operation.

Latent Dirichlet Allocation may be another example. Perhaps even Naive Bayes, if used in the appropriate manner.

So ultimately, the answer is, it depends on data pre-processing/feature-engineering and usage, rather than the technique.


Loss Function for Probability Regression I'm trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in the context of a binary classification problem where the labels are $\{0, 1\}$, but in my case I have an actual probability as the target. Is one of these options clearly best, or maybe are they all valid with just minor differences around the extreme 0/1 regions?

Assuming $x$ is the output of the final layer of my model.

Cross Entropy:
$target * -log(sigmoid(x)) + (1 - target) * -log(1 - sigmoid(x))$

Mean Squared Error with Sigmoid:
$(sigmoid(x) - target)^2$

Mean Squared Error with Clamp:
$(x - target)^2$

When I use the output I clamp the values between $[0, 1]$.

Given that you are trying to predict a scalar probability value, the cross-entropy formula you listed in the question is only valid if the target variable is discrete. So if your question is "predicts the odds of drawing a matching card from a deck", it will be fine.

The main difference between cross entropy and MSE will be how they penalize wrong predictions. Let's say given a target of 1 but prediction with 0. The cross-entropy is actually undefined in this case, but as the prediction gets closer to 0, the cross-entropy loss gets exponentially larger. On the other hand, your MSE is only 1. Which one is better, it depends on your application, if you want to avoid large margin of errors, it would seem the cross-entropy is more appropriate and vice versa.

The second and third approach only differs in how they make sure the prediction is within [0, 1], one uses a sigmoid function and another uses a clamp.
Given you are using a neural network, you should avoid using the clamp function. The clamp function is the same as the identity function within the clamped range, but completely flat outside of it. So the gradient of that function is 1 within the clamp range but 0 outside of it. Because of this, you are more likely to run into the dead neural problem in the same way when people talk about "dead relu".


Dealing with big data I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have found many tools with which I can analyse the data where each tool has its own advantages and disadvantages. Big data being really new to me, I find it very difficult to choose a platform to start with. There should be other people here who should have dealt with similar situations.

What criteria should I mainly take into account before selecting a
tool for analysing big data?

Some of the Criteria that I have taken into account : Visualization, Interaction, Security, Data Access and integration, Speed of response, Integrated Data Mining, Pattern Matching, Ease of use etc. As you can see the list that I have made for the criteria comes from the extensive reading of different articles on the topic. But I can't narrow down the list nor find the individual contribution of these criteria in the various tools available for analysis.

Let me also list some of the tools that I found after googling : Knime, Statistica 2, Rapidminer, Orange, WEKA, KEEL, R and RATTLE.

On what basis could I choose a tool from a list of tools that perform similar tasks?

UPDATE based on Comment

Aim : To develop a software that analyses the data coming from the wind mills and generate reports. The software should be able to predict when a wind mill can fail based on the analysis.

The project is still in the phase of gathering User Requirements. Maybe i am so early to come into conclusions about what tool should be used.

Someone else suggested that I should be finalise the requirements and then think about a tool that can help me get things done. So is it possible that I find what and how things should be analysed before finding a tool? And is it also possible that I find an algorithm for predictive analysis without knowing what would be the results of the tool after analysis.

Syncfusion provides a Big data platform which is an easy to use Hadoop distribution for Windows. It can help you get started quickly. Syncfusion also provides a PMML processing library using which you can execute predictive analytic models. There is a Dashboard platform also available that can help visualize the data.

All of the above are available for free through the community license if you qualify.

Note: I work for Syncfusion.


Machine Learning Dataset: Easy enough for fully connected, but not easy enough for logistic regression I was wondering if someone could direct me to a dataset for a classification task with the following conditions:

Multinomial logistic regression alone does not learn a good classifier
A series of fully connected layers is able to learn a good classifier
The task is not MNIST

Thank you

Here's a one-dimensional problem that should be impossible for logistic regression:

Generate x uniformly on [0, 10]
Let y = sin(2 * pi * x)

If you want a classification problem define y_disc as:

0 if y > 1/sqrt(2)
1 if 1/sqrt(2) > y > -1/sqrt(2)
2 if y < -1/sqrt(2).

This is non-linear, so logistic regression should do poorly. Further, if you decide to put higher powers of x into your logistic regression classifier, I think you'll need a lot of powers to accurately represent the Taylor series near 10.

If fact, you could try adding a small noise term to y before discretizing - I suspect that will result in your logistic regressor overfitting long before it's able to accurately approximate the behavior near x=10.


How to export one cell of a jupyter notebook? I'm currently working/prototyping into a Jupyter notebook. I want to run some of my code on a standalone iPython shell.

For now, I export my iPython code (file --> download as) and then execute it in my iPython (with %run). It works, but I would like to export only one cell or set of cells. So, that I can run only what I modified in my Jupyter notebook.

See this stack question

You can use %%capture Jupyter notebook' magic command to catch output of cell and then paste it to your text file with

with open('output.txt', 'w') as out:
   out.write(cap.stdout)

if you want to cell code to specific file for example code.txt you can use magic function %%writefile

%%writefile example.txt
ab = 'This is code'
a = 5
print(a+ 2)

Also if you want to append to file you must use -a parameter


A Machine learning model that has a undefined input size but a fixed output? I don't know too much about ML but I can seem to figure out how to train something like this. If you guys could list some possible ways to do this, thank you.

For Deep Learning (neural networks), there techniques like Adaptive Average Pooling and Adaptive Max Pooling which outputs a fixed number of values irrespective of the size of the input to these layers.


Problem with overfitting I make small CNN from scratch to classify barcodes. I have two classes: one for images with barcodes and second for all what isn't barcodes (items, animals, landscape, furniture, people). I got good training accuaracy (80%+ after 50 epochs) but my validation accuracy is constant and around 50% and it is my main problem. I am aiming in around 80% for validation accuracy. I use some technique as dropout, decrease number of parameters, data augmentation, etc. My train set is 480 images (240 - barcodes, 240 the rest) and my validation set is 120 (60\60). What should I do to achieve about 80% accuracy on the validation set. I use keras and tensorflow framework. Here my CNN:

model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 1), padding='same'),

tf.keras.layers.MaxPool2D(3, 3),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),

tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),

tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Flatten(),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=
['accuracy'])

If your training accuracy is significantly higher than your validation accuracy, you are overfitting. You might want to try increasing the amount of dropout or using batch normalization.
Your test set is very small, leading to high variance in your results. Your model might actually be better than it seems due to this variance. Try cross validation so you can test on more examples.
Make sure your train and test set are randomly split. Otherwise your train and test examples might look different, leading to lower performance on the test set.
Do error analysis to try to identify what mistakes the model tends to make. Does it have trouble finding bar codes at certain angles, at a certain size, or on a certain background color? Error analysis by itself won't help but it might help you identify why your performance is worse than you want it to be.
Use a more powerful model (more parameters). Counterintuitively, bigger neural nets that have more potential to overfit sometimes generalize better than smaller neural nets. We don't yet understand why this is the case but the phenomenon is well known so you might want to see if it's the case here.


Random Forest Stacking Experiment for Imbalanced Data-set Problem In order to solve a Imbalanced Dataset Problem, I experimented with Random Forest in the given manner (Somewhat inspired by Deep-Learning)

Trained a Random Forest which will take in the input data and the predict probability of the label of the trained model  will be used as a input to train another Random Forest.

Pseudo Code for this :

train_X, test_X, train_y, test_y = train_test_split(X,y, test_size = 0.2)
rf_model = RandomForestClassifier()
rf_model.fit(train_X, train_y)
pred = rf_model.predict(test_X)
print('******************RANDOM FOREST CM*******************************')
print(confusion_matrix(test_y, pred))
print('******************************************************************')
predict_prob = rf_model.predict_proba(X)

X['first_level_0'] = predict_prob[:, :1].reshape(1,-1)[0]
X['first_level_1'] = predict_prob[:, 1:].reshape(1,-1)[0]

train_X, test_X, train_y, test_y = train_test_split(X,y, test_size = 0.2)
rf_model = RandomForestClassifier()
rf_model.fit(train_X, train_y)
pred = rf_model.predict(test_X)

print('******************RANDOM FOREST 2 CM*******************************')
print(confusion_matrix(test_y, pred))
print('******************************************************************')

And I was able to see considerable improvement in the recall. Is this approach mathematically sound. I used the second layer of the Random Forest such that it would be able to correct the error by the first layer. Just looking to combine the principle of boosting to Random Forest Bagging Technique.Looking for thoughts.

The underlying idea is fine, but you've fallen into a common data leakage trap.  By recombining the data and then resplitting, your second model's test set includes some of the first model's training set.  The first model knows the labels on those datapoints and, especially if overfit, passes along that information in its predictions.  So the score you see for the ensemble is probably optimistically biased.

The most common approach to fixing this is to use k-fold cross-validation to produce out-of-fold predictions on the entire training dataset for the second model.

Note that sklearn now has such stacked ensembles builtin:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html


Reason for better performance of variants of SGD when local minimas of Neural Nets are equivalent? From some neural net article I read that if you scale up the Neural Net architecture the differnce in different local minimas in the loss surface diminishes. Essentially all local minimas become equivalent.

If that is the case then why do different variants of SGD (like Adagrad, ADAM etc.) works better than plain SGD? I believe reason for using these variants of SGD is to solve "bad" local minima problem, but if all local minimas are more or less the same then what is objective of using these variants?

I believe reason for using these variants of SGD is to solve "bad" local minima problem

That is not accurate. The variants are mostly about accelerating steps of gradient descent when faced with shallow or rapidly changing gradients, or with gradients that need adaptive learning rates because some parts of the network get stronger gradient signals than others. They do this by weighting or adjusting the step sizes in each dimension, based on additional knowledge to the current gradients, such as the history of previous gradients.

Shapes like saddle points or curving gullies, in the cost function "landscape" can cause difficulties for basic SGD. Take a saddle point as an example - there is no "bad" minima, a saddle point can be quite high up in a cost function. But the gradient values can be very low, even if they pick up again if you can take steps away from the saddle point. The trouble for SGD is that using just the gradient is likely to make updates oscillate up and down the steep parts of the saddle, and not move in the shallow direction away from the saddle point. Other difficult shapes can cause similar problems.

For a visualisation of the difference in behaviours of some of the optimisers at a saddle point, take a look at this animation (I'd like to find a creative-commons variant of this and include in the answer), and a blog that references it.

In addition, for deep learning network, there is a problem that gradients can "explode" or "vanish" as you work back through the layers. Using ReLU activation or similar can help with that, but is not always possible (think of RNNs which need sigmoid activations inside LSTM modules). An optimiser like RMSprop deals with this by normalising gradients used by the weight update steps based on recent history. Whilst SGD could more easily get stuck and fail to update the lower layer weights (weights closer to the input) - either not updating them much at all, or taking too large steps.


Using R and Python together I'm new in this field and I started working with data by using R. Because of that, I find R much easier to approach a data project. However, apparently an employer wants you to know an object-oriented programing language (a language like Python).

So it would be smart to think I can use Python just when I need to deal with a complex programming process, like replacing the na's in Titanic/kaggle project with the average based on name and to use R for anything else? So use them both interchangeably?

Beside the fact that Python is more programming oriented I don't see why somebody would use it over R...

Several clarifications:

you can program with object-oriented (OOP) concepts in R, even though OOP in R has slightly different syntax from other languages. Methods do not bind to objects. In R, different method versions will be involved based on the input argument classes (and types). (Ref: Advanced R)
you can also replace nans with mean / any stat. / value in R using a mask if you store the data in a dataframe (See SO post)
There is no problem using them interchangeably. I use R from Python using the package RPy2. I assume it is equally easy to do it the other way round.

At the end of the day, any language is only as good as how much the users know about it. Use one that you are more familiar with and try to learn it properly using the vast online resources online.


What makes a Tree-Structured Parzen Estimator "tree-structured?" From what I understand the Tree-Structured Parzen Estimator (TPE) creates two probability models based on hyperparameters that exceed the performance of some threshold and hyperparameters that don't.

What I don't fully understand is why TPE is "tree-structured." Is the simple clustering of y > thresh and y < thresh sufficient to call it a "tree?" When I envision a "tree structure" I usually think of many different "forks," perhaps some that are not binary and some that are forks of forks. Although a tree with just one fork is still technically a tree, the name "Tree-Structured Parzen Estimator" seems like it would describe something much more complex.

My question is whether or not I am missing some other deeper conceptual "forks" inherent in TPE that would turn it into a "tree" when you look at the bigger picture.

It means that your hyperparameter space is tree-like: the value chosen for one hyperparameter determines what hyperparameter will be chosen next and what values are available for it.

From a HyperOpt example, in which the model type is chosen first, and depending on that different hyperparameters are available:

space = hp.choice('classifier_type', [
{
    'type': 'naive_bayes',
},
{
    'type': 'svm',
    'C': hp.lognormal('svm_C', 0, 1),
    'kernel': hp.choice('svm_kernel', [
        {'ktype': 'linear'},
        {'ktype': 'RBF', 'width': hp.lognormal('svm_rbf_width', 0, 1)},
        ]),
},
{
    'type': 'dtree',
    'criterion': hp.choice('dtree_criterion', ['gini', 'entropy']),
    'max_depth': hp.choice('dtree_max_depth',
        [None, hp.qlognormal('dtree_max_depth_int', 3, 1, 1)]),
    'min_samples_split': hp.qlognormal('dtree_min_samples_split', 2, 1, 1),
},
])

From the original paper:

  In this work we restrict ourselves to tree-structured configuration spaces. Configuration spaces are tree-structured in the sense that some leaf variables (e.g. the number of hidden
  units in the 2nd layer of a DBN) are only well-defined when node variables (e.g. a discrete choice of how many layers to use) take particular values.


Train and predict two labels in a single process I have a python program that makes predictions using scikit-learn RandomForestClassifier. The label is called "default" and it's the default status of a loan. This works fine.
What I need now is to extend this model, and have another label called "Prepayment Percentage" that needs to be trained using the same data as the "default" label. Ideally, the model will be trained once and the predictions will also run only once for both labels. Is this possible with RandomForestClassifier?

This would not be possible since the two variables you are trying to predict are of a different type. You are first predicting the default label, which would be yes/no, so this is a classification problem. The second variable you are trying to predict is the prepayment percentage, which is a continuous variable, this is therefore a regression problem. You are not able to combine the two (i.e. a regressor and a classifier) into one model using RandomForestClassifier. You might be able to create a single model yourself that combines the two using the BaseForest base class.


In which CRAN mirror is the quadprog package available? I was trying to find the accuracy of some predicted data and I came to know that I can use the forecast package to do this. When I installed it and gave library(forecast), it said it needs quadprog package to work with forecast package.

I searched it in many CRAN mirrors [US(CA), US(IN), IRAN, MALAYSIA] and in all these only quadprogXT is available and with this package the forecast package is still not working.

In which CRAN mirror is this quadprog package available?

The quadprog package is available on all CRAN mirrors, but it requires the latest R version: R 3.6.0 (released on Apr 26, 2019), see also https://cran.r-project.org/web/packages/quadprog/index.html

> quadprog: Functions to Solve Quadratic Programming Problems
>
> This package contains routines and documentation for solving quadratic
> programming problems.
> Version:    1.5-6
> Depends:    R (≥ 3.6.0)
> Published:  2019-04-26


Feature addition/ subtraction and SVM model accuracy I am working on a text classification problem where I would like to improve the accuracy of my model. Presently, I am using SVM with linear SVC and OneVsRestClassifier. The model should correctly predict all of the subcategories for a parent category.

For entered account name, I should get the right description for my test data file that I will use to test the model later.

If I add another feature, +1 or -1, for debit and credit to my Account_Name field, will there any improvement in the accuracy?

Or, is there anything else that will help improve the accuracy of the model?

It is currently at 74% which is low.

It is hard to say without trying it. But you should add features as much as you can. When you have so many features, according to your success metrics, you can do future selection / regularization for both increasing the accuracy and speeding up the convergence of gradient descent.

Also you can also try (after adding more features) other classification algorithms such as RandomForest, GBM, Xgboost etc - Tree based models are more accurate in recent years.

Hope this is helpful!


Searching documents at scale: how to maintain cleaned documents? I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).

How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.

Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:

Read in raw documents.
Stem words, remove stop words.
Perform TF-IDF
Train model on cleaned up data.

Now when it comes time to score:

Load your saved model.
Repeat steps #2 and #3 from above.
Send the cleaned data into your model for a prediction/recommendation.

Here is an example in Python of something like that:

http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html


How to solve XNOR problem with neural network? How do we design a neural network with one hidden layer, two hidden neurons and an output neuron that implements an XNOR function? The truth table of XNOR is given below:

And how to provide weight and bias coefficients of this network?

Combine a NN for x1 AND x2 [your a1] with a NN for (NOT x1) AND (NOT x2) [your a2] to get the XNOR. You end up with inputs +1, x1, and x2 going to your hidden neurons a1 and a2, then +1, a1, and a2 into your output neuron.


Interactive plots in Jupyter I'm looking for Jupyter extension to plot interactive graphs. For example I need to plot twenty time series lines with order to examine data. Only plugin which I've found is bqplot https://github.com/bloomberg/bqplot but its documentation is too complicated.

Could you advice some other free tool or help with bqplot?

I can't plot lines with different colors.

unitIds = commonSeries['UnitId'].unique()
commonSeries = commonSeries.sort_values(by=['Date','UnitId'], ascending=True)
commonSeries = commonSeries.set_index(pd.DatetimeIndex(commonSeries['Date']))

plt.clear()
x_ord = OrdinalScale()
y_sc = LinearScale()
lines = []

for unitId in unitIds:
    timeseries = commonSeries.loc[commonSeries['UnitId'] == unitId]
    line = plt.Lines(x=timeseries.index, y=timeseries['Value'], scales={'x': x_ord, 'y': y_sc}, colors=['red','green','blue'])
    lines.append(line)

plt.Figure(marks=lines)

Plotly is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too.

There are tutorials on Plotly's docs which help you integrate it with Jupyter. Another tutorial.


Python: how to handle categorial values in dataset to build models I have a training dataframe dfTrain and the output of dfTrain.head() is shown below:

    C0 C1                              C2   C3 C4  C5               C6
0   1  73                 Not in universe   0   0   0  Not in universe
1   2  58  Self-employed-not incorporated   4  34   0  Not in universe
2   3  18                 Not in universe   0   0   0      High school
3   4   9                 Not in universe   0   0   0  Not in universe
4   5  10                 Not in universe   0   0   0  Not in universe

There are total 38 features and they are both categorical and numerical. Ignoring C1 and scaling numerical features, I am trying to build a Logistic Regression model. Since, the dataframe has categorical features, I am creating another dataframe which has dummy variables.

X = pd.get_dummies(dfTrain)

The shape of X now has 160 features which is much more than that of dfTrain.

Then I pass X and y (where y is target variable) to Logistic Regression Classifier

modelLogistic = LogisticRegression(C=10**-2, class_weight = 'balanced')

modelLogistic.fit(X, y)

The reason to use class_weight = 'balanced' is that there are 17 classes in y and highly imbalanced.

My question is: is my approach correct? Am I missing anything?

Yes, your approach seems to be right. The only thing that I would like to point out is that while it is desirable to convert categorical features to dummies but if you don't have enough memory then you can even consider to factorize your categorical variables. You can read about it more here.
Also, just make sure that after converting to dummies, the number of features are reasonable compared to the number of training samples.


TypeError: float() argument must be a string or a number, not 'function' I am trying to clean the data. But I don't know how to remove a function from a column in data frame.

At row number 473 it show column N has a function . How it should be filtered out ?

a generic way to convert pandas column to numeric dtype:

df['col_name'] = pd.to_numeric(df['col_name'], errors='coerce')

this will replace all values, that couldn't be converted to numeric values with NaN's (Not a Number).

PS you may want to analyze how and why do you have a reference to a lambda function in your cells. Fixing the source of that problem would be a more appropriate solution.


Similar objects same labels Example: red paprica and green paprica. The output I need is just paprica, should i label them the same or give them two labels "green paprica" and "red paprica" and leater on just treat them as one? Other example: 5 cent coin, "heads" and "tails". Output i need is just 5 cent coin. From machine learning point of course. Are there downsides and upsides of splitting labels or keeping them together in terms of accuracy?

If you only need to predict whether an object is a pepper or not why bother predicting whether it is a red pepper or a green pepper, you are making your problem more complicated than it needs to be. In my opinion you should try to make the problem you want to solve as simple as possible so I would advise using the shortest label possible as long as it solves your problem.


Is it conscientious to use a threshold for a model output in order to play on the recall and precision? I have just finished reading an article about the F1 score, recall and precision. Everything was clear except the fact that the author, in his example (see https://towardsdatascience.com/beyond-accuracy-precision-and-recall-3da06bea9f6c#a9f0), is using a threshold for his model output to play on the F1 score, recall and precision values.

So my question is: is it conscientious to use a threshold as the author did in his article?

In my opinion, this is like a kind of manual over-fitting, but maybe there is something I misunderstood...

Good question. Using a threshold is perfectly fine and is not "manual overfitting".

It is not manual because this is a step that can (and should) be done automatically. It is not overfitting as it doesn't modify the model itself. It modifies how you interpret the model's output.

What the user did is actual called cost-sensitive learning. It is a technique where you define which error is more costly and these costs are reflected in the performance metric you use.

I can see why it may feel like overfitting but it is important to understand that all you are doing here is handcrafting a performance metric with which you will evaluate your model. Your model will simply try to optimize this metric.


Is Neural Network Architecture independent of Data? If I change my dataset (let's say it is always images), should I change the architecture of my neural network?

Because your two questions are phrased a bit differently, I'm going to answer the one in the title.

  Is Neural Network Architecture independent of Data?

Generally speaking, yes.

One simple reason why you might have to change the architecture is if your new task has a different number of classes. If so you must change the last layer of your network.

Another reason is that some deep CNNs perform multiple stages of spatial subsampling (e.g. through pooling). This requires the input to be larger than a minimum shape. E.g. InceptionV3 and InceptionResnetV2 require the input to be at least $75 \times 75$, Xception $71 \times 71$, and so on.

Even if you don't necessarily have to, sometimes you should. An architecture that does well in MNIST, won't necessarily do well on ImageNet, because the latter is much harder. It requires models with a higher capacity.


Understanding Bernoulli Trials, Bayesian Setting I am required to complete a project on ML applications. I guess there is a lot of statistics in ML, not helpful for a non-maths background.

I am getting too bogged down by notations. There are too many notations.

I am trying to read about what Bernoulli trials are and I can't relate to it.

What is a Bayesian setting and why is Bayesian thing everywhere? What makes it such an omnipotent distribution?

Are there any theorems/theories that I must know ? Any book/notes/resources where I could learn about these stuff in a relatable way (I have very little maths background, but I can learn)?

What is a Bayesian setting and why is Bayesian thing everywhere?

In very simple terms:

  Bayesian is a statistical setting, where the likelihood of an event
  happening (called the posterior) depends on the prior trials or
  observations (called the prior(s)).

Bayesian networks is an extension of the above, forming a chain or a network of inferencing.

  Are there any theorems/theories that I must know?

For understanding the Bayesian paradigm, you need to know the Bayesian theorem/relation, which is basically:

$$P(\theta|d) = \dfrac{P(d|\theta)P(\theta)}{P(d)}.$$

  Any book/notes/resources where I could learn about these stuff in a
  relatable way (I have very little maths background, but I can learn)?

I would highly recommend "Doing Bayesian Analysis" by John Krushke


How to handle performance degradation due to domain transfers in data I am training a classification CNN on a labeled dataset $\langle x,y\rangle$. The network reaches a 0.92% accuracy rate on testing and validation sets. After this process, I pre-process the data to simulate some reasonable conditions by potting it through some process $\hat{x}=\Psi\{x\}$ and getting a new dataset $\langle\hat{x},y\rangle$ with the same labels. An example of this process is a conversion between color domains, for example, $x\in RGB$ and $\hat{x}\in grayscale$.

In the example I gave, The process may reduce the performance of the network if it using color features that are lost. In my case, no data is lost though the data may be disfigured. After applying $\Psi$ I am training a new network with a degradation in performance with only 0.81% accuracy. I was guessing that I am experiencing a concept drift, though I am not sure how to show or visualize this. Also, I do not know how to mend concept drift in NN.

Unfortunately, I cannot reveal the exact case I am working with. Is there a way to analyze or and show the data distribution has changed in the new domain?

How do I address the case described and what can I do to try and improve the performance (regardless of the specific task)?

There are multiple ways to detect covariate shift, for example Kullback–Leibler divergence but my favorite is adverserial validation. For images I would also look at the shap explainer, if its activating on different random places in the test set, you can suspect something fishy in concept.

Remedy concept shift with NN- try batch normalization if you havent. Why? Weights of the deeper layer, learned by the neural net, will be relatively less dependent on the weights learned on the shallower layer, hence avoiding shift.


How to keep only significant weights in an ANN My weights are store in a two dimensional matrix. Row i refers to node i in preceding layer and columns in that row are the neurons node i is connected to. I only want to keep some nodes. How do I pick 3 max weights and store it in a separate array while keeping track of which neuron it belonged to. Moreover, is it tested in theory that some weights contribute more than the others?

I'll address your last question first:

  is it tested in theory that some weights contribute more than the others?

When learning about NNs I thought it would be better to have a really large number of nodes in the first hidden layer, and then reduce down over subsequent hidden layers.  It is conceivable that many of the weights in this configuration are approaching zero, but this misses two main ideas: the values sent into the activation function are additive, so even small values can still make a difference if the input is large, whereas, 0 times anything is 0; the nodes are not representative or assigned a specific responsibility, so I assume, unless you start with exactly the same initial weights, a node may learn different features every time you train the model.

Subsequently, I have come to the conclusion that, if many of the weights are approaching zero, I probably have too many nodes in my hidden layer.  This does not necessarily take the activation function into account, but it is how I approach model architecture.

Having covered that, you could just convert the matrix to a sparse matrix, sort the sparse matrix based on values and delete everything below your cutoff. This would accomplish what you asked I believe.

However, I would ask why you want to save just three of the weights? It is inconceivable to me how reducing the size of the weight matrices is of any benefit.  If you can get the same performance under this concept, I think you should address your model architecture.  I am not aware of any specific research that would address whether this is an acceptable practice, but it just seems like a waste of effort to train a model and then throw out a majority of the model.  Simplify the model.

HTH


How is a policy expressed? In my work in behavioural cloning, I have been asked 'how is your policy expressed?' and I didn't know the answer to this. I was trying to create apply a behavioural cloning algorithm from the context of driving. I wanted to know what is the answer to the question 'how is your policy expressed'?
Am I supposed to give an analytical function? or can we simply say 'mapping from states to actions' as indicated by the demonstrations, which we regard as true?

I wanted to know what is the answer to the question 'how is your policy expressed'?

Typically when this is asked, what the asker wants to know is something like:
How do you represent the action choice in the output?
For example, if there are N discrete action choices, and your policy is stochastic, you could say something like: "The policy is expressed as a vector size N, with each value being the probability of choosing a matching action." You may also want to enumerate the actions - to say which index of the vector corresponds to which specific action. That depends on what need the person communicating with you has for understanding your policy.
Plus maybe:
How do you represent the state in the input?
The same thing here, give the state representation required to make the policy function work according to your implementation. Again, whether you should supply this depends on why the person asking you wants to know.

Am I supposed to give an analytical function?

With the right level of detail, that should be fine. The input and output domains plus maybe the index map (of which input and output features are in which positions). Note that because you are asked for how the policy is expressed, you would usually stop at the point that you have defined the probability distribution over the action space.


How to add a new category to a existing trained deep learning model? i have trained my deep learning model initially with 5 classes now i want to add another class without training the whole model over again for those 5 classes. How can I do that?

You cannot do that without re-training at least part of the model.

You will have to replace the existing output layer with a new output layer that has the desired number of neurons. That, of course, means that you will have to retrain at least the last layer or the last few layers. But you can freeze the weights of all the other layers.

In keras you can freeze the weights of a specific layer by setting its trainable property to False.


The upper range of a collected dataset is most likely accurate, but the rest may suffer biased omissions: How to call this phenomenon? Background: In collecting a dataset of a specific unit ordered by a numeric variable, it is possible that the upper 'cloud' of the dataset is correct, while the 'tail' seems inaccurate.
I can thus trust the upper bounds of the dataset, while I deem the mid- and lower ranges to be rather questionable.
Question: Is there a data scientific term for this phenomenon? If so, how is it called?
Example: I use various sources to gather a list of major publishers of scholarly journals. In total, I found 150 publishers that publish at least 50 journals.
At the top, I found the publisher AAA with 3.000 journals, BBB with 2.500 journals, CCC with 1.900 journals, DDD with 1.500 journals etc.
With my industry knowledge,  I can confirm that the top 20 is likely to be accurate.
However, in the lower ranges, there are publishers like XXX with 51 journals, YYY with 50 journals, ZZZ with 50 journals etc. Many of them are rather obscure, and even as an expert I may have never heard of them. I can imagine that the 'tail' is rather inaccurate with large-scale omissions, such as publishers from the Global South.
I thus tend to trust the top 20 or so of that list, but not the ones that rank between #21 and #150.

The case described can be cast as imbalanced dataset problem, or rare events problem.
Even more generally as highly non-uniform underlying distribution problem (which is an umbrella term for both cases).
References:

Machine Learning Tips: Handling Imbalanced Datasets
Handling imbalanced datasets in machine learning


How to scale exponential data for a regression problem? I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For instance, if I am training a RNN on time-series data to perform demand forecasting, the product I am forecasting may be in a growth stage that will yield higher demand numbers in the future months than the past months. Is there a way to normalize data but still allow for larger values in the future?

Yes, there is. Instead of Min-Max scaling, that shrinks any distribution in the [0, 1] interval, you can Scale the variables (in statistics, they are called Z-scores). The formula is:

(x - mean(x)) / stdev(x)

This will zero-center the distribution, and scale all the variables to a standard deviation = 1.

I strongly prefer this technique to Min-Max Scaling, since the latter is too sensitive to outlier observations and generates problems unseen, out-of-scale datapoints.

Additionally, if your analysis allows for that, consider taking the log() of a data distribution that grows exponentially. That would coerce it to a more linear growth. (I'm not sure this is what you need, it's just a possibility.)


Which insights a data scientist could derive from text-analysis? I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging.
Since it is my first time in text mining and manipulation, I would like to know which kind of insights I could get from this information and what the best approach to present this analysis would be.

For example: if I had many texts like the following:

Hemingway =["When spring came, even the false spring, there were no problems except where to be happiest. The only thing that could spoil a day was people and if you could keep from making engagements, each day had no limits. People were always the limiters of happiness except for the very few that were as good as spring itself.","Most people were heartless about turtles because a turtle’s heart will beat for hours after it has been cut up and butchered. But the old man thought, I have such a heart too.","Perhaps as you went along you did learn something. I did not care what it was all about. All I wanted to know was how to live in it. Maybe if you found out how to live in it you learned from that what it was all about.","The people that I liked and had not met went to the big cafes because they were lost in them and no one noticed them and they could be alone in them and be together."]

Shakespeare=["These violent delights have violent ends And in their triump die, like fire and powder Which, as they kiss, consume","Let me not to the marriage of true minds Admit impediments. Love is not love Which alters when it alteration finds, Or bends with the remover to remove. O no, it is an ever-fixed mark That looks on tempests and is never shaken; It is the star to every wand'ring barque, Whose worth's unknown, although his height be taken. Love's not Time's fool, though rosy lips and cheeks Within his bending sickle's compass come; Love alters not with his brief hours and weeks, But bears it out even to the edge of doom. If this be error and upon me proved, I never writ, nor no man ever loved.","O serpent heart hid with a flowering face!Did ever a dragon keep so fair a cave? Beautiful tyrant, feind angelical, dove feather raven, wolvish-ravening lamb! Despised substance of devinest show, just opposite to what thou justly seemest - A dammed saint,honourable villain!","Lord Polonius: What do you read, my lord? Hamlet: Words, words, words. Lord Polonius: What is the matter, my lord? Hamlet: Between who?  Lord Polonius: I mean, the matter that you read, my lord."]

Would it be useful a word frequency distribution, POS tags distribution? I have never presented any results, so I would like to know more on how a data scientist/analyst may 'tell' and look at this data.

Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples:

I once worked with a book distributor that tagged each book they sold with keywords (Fantasy, Horror, etc). You can automate the tagging process if you have a sufficiently large dataset of already labeled books. You can do the same with your phrases (Inspiring, Funny, etc) but probably you don't have labeled data. It would be nice to be able to ask an app for a phrase of certain type :).
Something easier is sentiment analysis: are most of the phrases positive? Negative? You don't need labels for this.
Style transfer: maybe you have several phrases by, let's say, Shakespeare. You can try transferring his style to Einstein's phrases. This is hard but feasible: look up Generative Adversarial Networks.