Find correct forms of a word I'm trying to find all grammatical forms for words from a dataset. I'm new to NLP.

I'm using a combo: spaCy (to get a base form of a word), word_forms (to get all possible forms). Sadly, it seems that they are not able to get everything right. Examples like 'wildest' or 'scariest' seem to be out of reach.

Maybe I'm expecting too much (in most of the cases the solution works really well).

Is there a decent tool I could use? I'm aware of nltk lemmatizer, its performance is worse than word_forms and spaCy. I'd appreciate some kind of advice.

Out of curiosity, have you made any effort to quantify the accuracy of spaCy?  The way that it performs lemmatization leverages the syntactic similarity hypothesis, and as such 100% accuracy is an unrealistic expectation.

With that said, if there is a particular subset of words that are essential for you correctly classify, and you're willing to create a labeled training set, spaCy allows you to train custom models. I believe that you are able to train these models based on the existing language models, and so the amount of training data need not be prolific.


How to go about obtaining data I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of data I need. However, Google doesn't seem to provide access to historical data. There are several other companies that provide analytics based on traffic data to their clients but raw data is never provided. How would one go about obtaining this kind of data?

Edit: the kind of data I need would be spatio-temporal. Take a bunch of N sensor locations on an inner city road network and give me the driving speed of the cars passing through each one of them at T equally spaced time points during a day. So we would have N*T data points per day. The denser the locations the better.
If you look at Google Maps traffic overlay you can see traffic flow speeds for almost every small street. I think they take this data from GPS in driving cars as well as stationary sensors. Google doesn't provide historic data but maybe one could record this from the website directly?

You should try contacting cities about getting the sensor data from their Traffic Controller Systems. These usually have several detectors per lane at intersections (they also control the traffic lights so you get data on those as well). Some of these data are property of the companies that supplies these system, in which case you'll need to contact the company. These companies are generally not keen on sharing the data with commercial parties, but generally are interested in sharing it with educational parties as outreach.

I know one public dataset of this kind of data, the city of Utrecht in the Netherlands provides a sample of traffic density data. Unfortunately the metadata is mostly in Dutch.


How to mask or crop IDW results My question is: how can I mask or crop the results, using R, of IDW interpolation to only the area containing the original set of data points?

In the example below, 20 random points are used to interpolate a surface using the IDW function in gstat. A convex hull is obtained from the points set and plotted on the interpolated map. But, I would like to crop the map so only areas within the point cloud show the interpolation result.

Thanks for any suggestions!

###
#                             IDW in R
#
library(gstat)
#
set.seed(1234)
x <- rnorm(20)+10
y <- rnorm(20)+10
z <- rnorm(20)+10
#
xyz <- data.frame(x,y,z)
coordinates(xyz) <- ~x+y
xr <- range(x)
yr <- range(y)
b <- round((xr[2]-xr[1])/100,2)
#
grd <- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),
y=seq(from=yr[1],to=yr[2],by=b))
#
coordinates(grd) <- ~ x+y
gridded(grd) <- TRUE
#
idw<-idw(formula=z~1, locations=xyz, newdata=grd)
#
image(idw, col=terrain.colors(16))
contour(idw, add=T)
points(x,y, col=4, pch=19)
#
library(spatstat)
conv<-convexhull.xy(x,y)
plot(conv,add=TRUE)
#

Here I am answering my own question:

To obtain an IDW surface interpolation restricted to the area containing the original set of xyz data points:

Create a rectangular grid based on the xy point ranges using expand.grid {base}
Create a convex hull using the function convexhull.xy{spatstat}
Cut an irregular grid using the function inout{splancs} on the
rectangular grid masked by the convex hull
Perform the interpolation using the function idw{gstat} with the new grid.
Be sure to detach spatstat, because it has a different function for
IDW with the same name

Code:

###
# IDW in R
#
# Load gstat:
library(gstat)
#
# An example XYZ data set:
set.seed(1234)
x <- rnorm(20)*1000+3000
y <- rnorm(20)*1000+3000
z <- rnorm(20)+10
xyz <- data.frame(x,y,z)
#
# Calculate ranges and spacing:
xr <- round(range(x),1);xr
yr <- round(range(y),1);yr
b <- round((xr[2]-xr[1])/60,2);b
#
# Create a grid:
grd1 <- expand.grid(x=seq(from=xr[1],to=xr[2],by=b),
    y=seq(from=yr[1],to=yr[2],by=b))
plot(grd1)
#
# Load spatstat and splancs
library(spatstat)
library(splancs)
#
# Extract convex hull
w2 <- convexhull.xy(x,y)
#
# detach spatstat because it has a different "idw" function:
detach("package:spatstat", unload=TRUE)
#
# Create a new cropped grid:
grd2 <- grd1[inout(grd1,w2$bdry[[1]]),]
plot(grd2)
#
# Convert grids and dataset to sp objects:
coordinates(grd1) <- ~x+y
gridded(grd1)   <- TRUE
coordinates(grd2) <- ~x+y
gridded(grd2)   <- TRUE
coordinates(xyz) <- ~x+y
#
# Run the interpolations:
idw1<-idw(formula=z~1, locations=xyz, newdata=grd1)
idw2<-idw(formula=z~1, locations=xyz, newdata=grd2)
#
# Plot the results:
image(idw1, col=terrain.colors(16))
contour(idw1, add=T)
points(x,y, col=4, pch=19)
image(idw2, col=terrain.colors(16))
contour(idw2, add=T)
points(x,y, col=4, pch=19)
#


Maximum number of images to train a convolutional neural network I just wondered if there is a technical limit on the number of images to train a neural network.

I want to work with extremely high numbers of images, around 1,000,000 to 10,000,000 images. Is there a cap because of the graphics card memory? I tried to do some research on that issue, but I mostly find questions about how many images you need to get decent results for a convolutional neural network.

Thanks and best regards,

beinando

Theoretically speaking, there is no limit on how many data points you can use to train any neural network. This is true because neural networks that can be trained incrementally and do not need to "see" the entire dataset.

Obviously, it helps if your machine is able to store at least a single image and the network in memory.

To train on a large number of images, you will likely need to use batches. This means that each iteration happens on a subset of your data, which can be as small as a single image.


Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline? I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.
I am pretty comfortable with the sklearn Pipeline object that I use for pre-processing but I am not sure if I should include data cleaning, data extraction and feature engineering steps that are typically more specific to the dataset I am working on.
My general thinking is that the pre-processing phase would include operations on the data that need to be done after splitting it so as to avoid data leakage. These would typically be:

Scaling
Imputing (if not by constant value)
Encoding

On the other hand, data cleaning or feature engineering are operations that could be performed on the whole data set, for example:
def clean_price(data):
    """Clean the price feature."""
    data['Prezzo_EUR'] = data['Prezzo'].str.split('€').str[1].str.replace('.', '').astype('float')
    return data

def create_energy_class(data):
    """Create energy class feature."""
    data['Classe_energetica'] = data['Efficienza energetica'].str.extract(r'([A-G]\d?)')
    return data

These operations are very specific to the dataset and don't cause data leakage, and I don't really see any value in including them in an sklearn pipeline using the FunctionTransformer object for example. I could easily have 20+ simple operations like the ones shown above and what I do instead is build a custom pipeline connecting all of the functions:
def create_pipeline(list_functions):
    """Pipeline function for data cleaning steps."""

    def pipeline(data):
        out = data
        for function in list_functions:
            out = function(out)
        return out

    return pipeline

So I basically end up having two separate pipelines, one for operations to be performed on the whole dataset before splitting it (and that are very specific to the dataset), and one for operations after splitting (what I call pre-processing).
What is the approach to building scalable data pipelines that include data cleaning, data extraction, feature engineering, pre-processing before modeling? Are there better or more standard ways of going about this?
Also please correct me if you think my terminology is not accurate.

One of the benefits of using scikit-learn's Pipeline is to appropriately apply the operations to both training and prediction datasets. The "whole dataset" you mention that is only the training data you currently have available. Scikit-learn's Pipeline can automatically handle additional data, such as production data.
Having two different pipelines can create unnecessary overhead and might lead to bugs in the system. It is best practices to have a single pipeline that handles all operations.


NLP: Mapping Penn treebank and Brown corpus, to Universal PoS Tags I am experimenting with NLP and PoS tagging. I wish to build a large corpus, composed of Penn Treebank and Brown corpus, and possibly even more. Unfortunately, their PoS tags are not compatible.

Is there a way to map their tags to universal tagging? (Or any other tagging?)

NLTK contains options for retrieving brown, treebank corpora with universal tags, instead of their own tagging schemes.

nltk.corpus.treebank.tagged_words(tagset='universal')

instead of:

nltk.corpus.treebank.tagged_words()

Similarly:

nltk.corpus.brown.tagged_words(tagset='universal')
nltk.corpus.nps_chat.tagged_words(tagset='universal')
nltk.corpus.conll2000.tagged_words(tagset='universal')

As far as I am aware, none of the other tagged corpora support the universal tagset option.


Class weight degrades Multi Label Classification Performance I noticed something strange while I was conducting a multiple label classification problem via keras neural network. My data set consist of imbalance data with 12 features and 25 possible labels. When I instantiate my model with no class weight I get a precision of 97%, recall of 13%, subset accuracy of 14%, f1-score of 23% using the micro average.
When I apply class weight these scores are significantly reduced to the below.

('Accuracy', 0.1757093081134893)
('Precision:', 0.19632925472747498)
('Recall', 0.1637291280148423)
F1 -score 0.178553363682

Also I calculate the weights with below code that I copied and modify from a previous post:

def class_out(s):
      y_classes = s#.idxmax(1, skipna=False)

      # Instantiate the label encoder
      le = LabelEncoder()

      # Fit the label encoder to our label series
      le.fit(list(y_classes))

      # Create integer based labels Series
      y_integers = le.transform(list(y_classes))

      #print y_integers
      # Create dict of labels : integer representation
      labels_and_integers = dict(zip(y_classes, y_integers))

      print labels_and_integers
      class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
      sample_weights = compute_sample_weight('balanced', y_integers)

      class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
      class_sweights_dict = dict(zip(le.transform(list(le.classes_)), sample_weights))

      print class_weights_dict

      return class_weights_dict

Also see a sample of the model:

batch_size = 100

weights = class_out(df_all['tag'])

model = Sequential()
model.add(Dense(10, activation="relu", input_shape=(12,)))

#model.add(Dense(10, activation='relu'))

#model.add(Dense(8, activation='relu'))
#model.add(Dropout(0.50))
model.add(Dense(25, activation="sigmoid"))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy',precision,mcor,recall, f1])

model.fit(X_train, Y_train, batch_size=batch_size, epochs=15,class_weight=weights,
      verbose=1,validation_data=(test, target_test))

Is there a reason to believe that the model performance is best without class weights ?

Adding class weight but not changing the way you measure performance will usually degrade overall performance as it is designed to allow increased loss on lower-weighted classes.

I would recommend also weighting your accuracy measures. This is a bit tricky with accuracy/precision etc. so maybe calculated the weighted logloss and compare it to the unweighted logloss of the unweighted model.

Basically it comes down to the question: are you happy with a model that performs worse overall but better on your heavily weighted classes?


Does sum of embeddings make sense? Referring to the LightFM model from paper Metadata Embeddings for User and Item Cold-start Recommendations, the  model tries to learn $d$-dimensional user and item feature embeddings $e_f^U$ and $e_f^I$ for each feature $f$ ($U$ is the set of users, $I$ is the set of items).

The latent representation of user $u$ is given by the sum of its features' latent vectors:
$$
\mathbf{q_u} = \sum_{j\in{f_u}}\mathbf{e_j^U}
$$

The same holds for item $i$:
$$
\mathbf{p_i} = \sum_{j\in{f_i}}\mathbf{e_j^I}
$$

Does it really make sense to sum the latent embeddings to represent a set of features (user or item)?

Thinking of a sum as an average, this model sounds reminiscent of continuous bag of words (CBOW) word embeddings (i.e. word2vec). In that context, the words in a sentence are used to predict a missing word using the average of embedded vectors (see this question). This method works pretty well for words, so it might make sense that you could extend it to other types of embeddings like user/item compatibility.

The average makes some intuitive sense when you think of comparisons. To place a user/item in embedding space, you can average the feature embeddings. If a user/item has different but similar features their average should put the user/item in a similar region of embedding space. A sum should work the same if the feature number is fixed.

An issue with a sum would arise if feature number isn't fixed. Users/items could appear different because their embedding vector is a different length, even if it has the same direction. Normalizing by feature number would fix this.

Another possible issue with using a sum rather than an average is that the user/item embedded vectors aren't comparable to the feature embedded vectors. This doesn't seem to matter for the author, because they don't try to compare the user/item vectors to the feature vectors directly.

TL;DR Yes, a sum might make sense to represent a set of features, but an average might be better.


Machine Learning methods for finding outliers I have a csv files of thousands of lines, the data is put down into a Dataframe of columns.
Some of the column have text information while others might have numbers.
I want to detect anomalies or outliers inside this csv file.

Being relatively new to this and not knowing much I would appreciate if I get some help or a small guidance and how I should tackle this problem. What methods or what is the best approach to find outliers in a Dataframe.

My language of choice is Python

You should plot the data with matplotlib and seaborn to get a visual view of the data.  That is where I would start.  Especially with large datasets, this is a quick way to visually see any outliers.


Best languages for scientific computing It seems as though most languages have some number of scientific computing libraries available.

Python has Scipy
Rust has SciRust
C++ has several including ViennaCL and Armadillo
Java has Java Numerics and Colt as well as several other

Not to mention languages like R and Julia designed explicitly for scientific computing.

With so many options how do you choose the best language for a task? Additionally which languages will be the most performant? Python and R seem to have the most traction in the space, but logically a compiled language seems like it would be a better choice. And will anything ever outperform Fortran? Additionally compiled languages tend to have GPU acceleration, while interpreted languages like R and Python don't. What should I take into account when choosing a language, and which languages provide the best balance of utility and performance? Also are there any languages with significant scientific computing resources that I've missed?

This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are:

Performance: Basically boils down to how quickly the language does matrix multiplication, as that is more or less the most important task in data science.
Scalability: At least for me personally, this comes down to ease of building a distributed system. This is somewhere where languages like Julia really shine.
Community: With any language, you're really looking for an active community that can help you when you get stuck using whichever tool you're using. This is where python pulls very far ahead of most other languages.
Flexibility: Nothing is worse than being limited by the language that you use. It doesn't happen very often, but trying to represent graph structures in haskell is a notorious pain, and Julia is filled with a lot of code architectures pains as a result of being such a young language.
Ease of Use: If you want to use something in a larger environment, you want to make sure that setup is a straightforward and it can be automated. Nothing is worse than having to set up a finnicky build on half a dozen machines.

There are a ton of articles out there about performance and scalability, but in general you're going to be looking at a performance differential of maybe 5-10x between languages, which may or may not matter depending on your specific application. As far as GPU acceleration goes, cudamat is a really seamless way of getting it working with python, and the cuda library in general has made GPU acceleration far more accessible than it used to be.

The two primary metrics I use for both community and flexibility are to look at the language's package manager, and the language questions on a site like SO. If there are a large number of high-quality questions and answers, it's a good sign that the community is active. Number of packages and the general activity on those packages can also be a good proxy for this metric.

As far as ease of use goes, I am a firm believer that the only way to actually know is to actually set it up yourself. There's a lot of superstition around a lot of Data Science tools, specifically things like databases and distributed computing architecture, but there's no way to really know if something is easy or hard to setup up and deploy without just building it yourself.


Rules/Methods to use for determining user groups with no review data I’m working on an analytical problem where I have customers who deal with end consumers. End consumers can leave customers reviews based on their experience (Star ratings, text etc). I’m trying to segregate new/old customers into different quality categories: good, average, bad, for an online advertising platform. I’m looking for insights on rules/methods on how to do this

How to set a quantifiable threshold for calling a customer good, average, bad for business? There is consumer review data available (Imagine a table with customer, and the star rating they received, and/or textual reviews from consumer)
How to classify a new customer on the platform with no or minimal review data available about them particularly, while there’s also some customers who have many reviews available?

Start with some natural thresholds : >3* average, >4* good, >4,5*: excellent, 4,9+* perfect. Then you can correct your rating based on some averages, other metrics or even text (but that's hard). Honestly I am not sure it will work as ratings should be viewable and giving different status to customers with same average rating will get noticed.
Leave them blank, consumers like to know they are first to comment / know there is no reliable rating. No comment is an interesting information. This will push both customers and consumers to give a rating.


Does it make sense to visualize data with a linear relationship using tSNE? I have used tSNE several times to visualize high dimensional data for cluster analysis, and it has always worked quite well when data falls into clusters.

However, does it make any sense to use tSNE to visualize the independent variables in a dataset which has a linear relationship $y = X \theta + b$, where $y$ is the target value and $X$ are the independent variables (b is the bias)?

As I understand tSNE, it is used more for capturing local structures in high dimensional data, so it is probably not suited for visualizing data for linear regression - is this a wrong assumption?

Your assumption is right, the results are in general misleading.

Suppose your (linearly correlated) data have missing points in some range:

Than for t-SNE the two subset of data will be two different clusters, even if they lie on the same linear distribution:

But, if you are actually interested in the fact that those two structures are separated, then t-SNE is a good choice to visualize it.

So a proper answer should be: it depends on what you need.

P.S.

Here the code used for this example:

import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

def function(x, y):
    return 4+3*x + 4*y

x1=np.random.rand(100)
x2=np.random.rand(100)+1.2
X=np.concatenate([x1, x2])
Y=0.5+2*X

Z=function(X, Y)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X, Y, Z, c='r', marker='o')
plt.show()

data=[]
for x, y, z in zip (X, Y, Z):
    data.append([x, y, z])

data_embedded = TSNE(n_components=2).fit_transform(data)

plt.scatter([x for x, y in data_embedded], [y for x, y in data_embedded], color='r')
plt.show()


How to figure out if the problem is time series Forecasting or not? Recently I have encountered a time-series based, where I have a dataset which contains data for call-centers performance. The dataset contains information about the number of calls made by customers for each date which is further categorized region wise.

Time        Region  Number of Calls
2018-03-01  X       1245
2018-03-02  Y       1390

The task at hand is to forecast the overall performance of call-centers as well as forecasting each region performance as well for the next 3 months to see which regions will perform better than others.
I have a dataset with dates of June-2017 to Current Date

Questions

Is the data enough to perform a time-series analysis with credible
results for the next quarter?
How should I tackle the task to forecast Region Wise Performance as well

1 - Is the data enough to perform a time-series analysis with credible results for the next quarter?

This dataset is about 700 lines per Region, correct? Assuming you have 5 regions that is 3500 samples. Well this is surely not a large dataset but you can try to perform forecasting models on it as it is not small either. If the model is relatively simple your model may be able to run it. In DS the only way to know what will work is by trying or by having someone that already tried that saying it work, or it don't.

2 - How should I tackle the task to forecast Region Wise Performance as well

Region Wise, you have even less data, but still you can perform it in the same way, and probably you can get this as a bonus from the general model for all regions.

Your only challenge is to encode the Region into a way that can be used for ARIMA, NNs or other regressive models.

Maybe data from the Region is more relevant to a shallow model than the region itself (such as number of employers, population size, level of technological advance ...)


Creating Bag of words What is the best technology to be used to create my custom bag of words with N-grams to apply to. I want to know a functionality that can be achieved over GUI. I cannot use spot fire as it is not available in the organization. Though i can get SAP Hana or R-hadoop. But R-hadoop is bit challenging, any suggessions.

create my custom bag of words with N-grams to apply to

My initial recommendation would be to use the NLTK library for Python. NLTK offers methods for easily extracting bigrams from text or ngrams of arbitrary length, as well as methods for analyzing the frequency distribution of those items. However, all of this requires a bit of programming.

  a functionality that can be achieved over GUI

That's tricky. Have you looked into GATE? I'm not exactly sure if/how GATE does what you want, but it does offer a GUI.


How do I represent a hidden markov model in data structure? My task involves a POS Tagging using HMM. I am given a training data set (word/tag). I have to write a file with transition probabilities and emission probabilities. I am currently using a nested dictionary of the form {State1: {State2: count, State3 :count}}. However, while calculating the probabilities now via the counts in nested dict, my program is running very slow for mid size files (e.g. 2000 sentences and tags)

Is there a better way to store a HMM in python? For my project, I cannot use any external library that already does this, I must use standard python libraries.

With 29 states and 841 possible transitions to track whilst reading a file with 2000 entries (word, tag), then you should not be experiencing a speed problem when using a dictionary of dictionaries.

Assuming your data structure as described called transition_counts, and receiving data in pairs, (this_pos, next_pos) then running 2000 times:

transition_counts[this_pos][next_pos] += 1

takes only a fraction of a second. This is similar for code that calculates $p(POS_{t+1}|POS_t)$:

total_from_pos_t =  sum(transition_counts[pos_t].values())
prob_pos_tplus_one = transition_counts[pos_t][pos_tplus_one] / total_from_pos_t

This is very fast. Your problem is not with the representation.


how to apply MC dropout to an LSTM network keras I have a simple LSTM network developped using keras:

model = Sequential()
model.add(LSTM(rnn_size,input_shape=(2,w),dropout = 0.25 , recurrent_dropout=0.25))
model.add(Dense(2))

I would like to apply the MC dropout method. How can I enable dropout in the test phase in order to compute the uncertainty?

Thanks.

Well, in order to enable dropout during test phase you can do something like this:

keras.layers.Dropout(0.5)(x, training=True)

Then you'll probably want to run it multiple times. If you don't care about the inference time just run forward pass multiple times and at the end calculate mean and variance of your output.


How to improve performance of Tensorflow's CNN example? In the example in this link: https://www.tensorflow.org/tutorials/images/cnn, the performance is over 70%, how can I improve it? What parameters do I need to tweak?
Thanks.

Coming to Neural Networks We can tweak a lot of parameters that can improve performance of the network. Sometimes changing a small parameter value can result in a big performance increase and sometimes we have to tweak a lot to get good performance. This depends on the problem.
In general the following parameters are often important in terms of getting good performance:
Number of layers: change number of layers by adding more layers or removing some layers. Number of layers.
Number of Kernels
Kernel Size
activation function(most people use relu but we can try more like prelu, elu..etc)
weight initialization techniques: try different weight initializers with respect to activation function used.
Try dropout layers which we reduce overfitting
Try Batch Normalization
use pretrained models(Transfer Learning) which can increase performance very much.
These are some of popular parameters on which performance of model depends. Try tweaking them and get a good knowledge of these.


VC dimension for Gaussian Process Regression In neural networks, the VC dimension $d_{VC}$ equals approximately the number of parameters (weights) of the network. The rule of thump for good generalization is then $N \geq 10 d_{VC} \approx 10 * (\text{number of weigts})$.
What is the VC dimension for Gaussian Process Regression ?
My domain is $X = \mathbb{R}^{25}$, meaning I have 25 features, and I want to determine the number of samples $N$ I must have to archive good generalization.

The expressiveness of the gaussian process grows with a number of training points. So, the vapnik-chervonenkis dimension in fact is infinite (pretty much the same way it's infinite for k nearest neighbors) and unfortunately your rule of thumb is not applicable here.
You should probably rely on train/validation split to estimate the generalization. From my experience, GP generalizes much better than neural nets, but the exact required train size depends on data distribution complexity itself.


Autocorrelation of a sine wave I would like to know the meaning of an autocorrelation graph of a sine wave. When the time lag is 0, then the autocorrelation should give the highest value of 1 since a copy of the signal is completely correlated to itself. By this logic, after a time equal to the period of the signal, the correlation should again be maximum since the shifted signal is again the signal itself. However, when I plot the correlation in python, I get a function which keeps increasing which goes against my intuition that the correlation function should be periodic. Can anyone please explain why the autocorrelation shows this trend in its graphical form?

import matplotlib.pyplot as plt
import numpy as np
time = np.arange(0, 10, 0.1);
y = np.sin(time)
result = np.correlate(y, y, mode='full')
plt.plot(result[:int(result.size/2 )])
plt.show()

You actually do not have a problem here. The way np.correlate is implemented, you get the negative delay values first. If you plotted your entire signal, it would look as expected. Run plt.plot(result) and you will get

The maximum is at 100 and represents zero delay. That the maximum is so high, is because np.correlate does not normalize the function, but just computes the sum, as you can see in the documentation

  This function computes the correlation as generally defined in signal
  processing texts:

  c_{av}[k] = sum_n a[n+k] * conj(v[n])

  with a and v sequences being zero-padded where necessary and conj
  being the conjugate.

That the acf decays for your signal and for delays larger or smaller than zero is due to its finite length. For an infinite signal, you would be able to see the periodicity without decay.


Working with Data which is not Normal/Gaussian What happens if my data/feature is not normal? Can I still use machine learning algorithms to utilize such data for predictions?

I noticed in many data sciences courses, there is always a strong assumption of using a normal/Gaussian data. I have always wonder why this is so, and most people would say that due to central limit theorem, the data is always assumed to be normal.

However, what if the data that I am dealing with is not normally distributed?
Should I perform log/ exponential transformation on the data for the sake of getting normal distributed data?
Why is Gaussian data always best suited?

There are models that do not make assumption that the underlying data distribution is a normal distribution.

For example, support vector machine just cares about the boundaries of the separating hyperplane and do not assume the exact shape of the distributions. Decision tree models also do not make such assumption.

Gaussian distribution is popular and it has been analyzed frequently for its simplicity but there are other models as well.

If you know the distributions that you believed your data can follow, you can also build your own model, for example by maximizing the likelihood or posterior distributrion.

If you are able to convert your data to normal distribution and use a model that you are familiar with, you can try it and see how does it perform.


Please help select an Algo based on Accuracy and Confusion Matrix I am very new to Data Science would appreciate your advice big time.

Got a task: predict if a trade will be profitable or not, based on a set of data.
I have prepared, cleaned and tested data. Selected a few two-class algorithms since the answer requires a boolean result (true or false (1/0)). After testing selected two best from my opinion: Random Forest and Logistic Regression CV (cross validation). Below are the results. Can't decide which one to use in production. Also, please note: it is better not to trade than trade and loose money, therefore false negatives are better than false positives.

Random Forest results:

Logistic Regression CV (cross validation) results:

Added a sample of dataset for clarification:

There must be definitely some leakage in the data, which is why the Logistic Regression is predicting almost all as the positive class. You may want to check on that first before you proceed. The only reason Random Forest may seem better in that case is that it, by default, only takes a subset of variables.


Data science which is not part of AI? I have understood that data science is part of  AI. However, do we have some area in data science which we can say does not fall under AI? If yes, please name them.

I would not concern yourself too much about any structuring of knowledge that declares that one subject is categorised as one thing or another. These structures are often wrong and knowledge in general is more fluid and difficult to define than can be viewed as some kind of Venn diagram.

In addition, both Data Science and AI are poorly defined, and have more in common with marketing terms used to recruit staff and sell products than stricter academic subject areas such as Maths or Modern Languages. You will find Data Scientists working on "AI" projects and "AI Engineers" using basic statistics to analyse data.

Having said that, I would prefer to say that Data Science and AI share a common set of tools, especially when it comes to machine learning. Neither contains the other in my opinion.


How to approach data prediction problem I'm new to ML and data science. I would really like high level advice how to approach the following problem. I need to predict if an engine will fail, what I've is a sensor that give a certain value between 1 to 4 so a record might look like:

1243412312431444123123234234232423

Each record is 150 readings in length. I've a few millions of this records but in no particular order (i.e. the database doesn't contain explicit timestamp although by using some quirk database artifact I can make some records appear in order)

I've 200 different engines and a label of "failed" and "active" for each one.

So I'm looking for a predictive model that can predict (or detect) using the above data whether an engine failed or not. Any advice?

Thanks,
Eden

Are you familiar with deep learning? If yes, I would advise you to one-hot encode these records and use a 2D convolutional neural network for prediction. Since these are mere numbers and there is no way to engineer new features, I'm afraid conventional ML methods won't work. And the records have only 4 unique values, so 2D-Convolutional NN can handle it. Treat them as a picture and let the NN capture the patterns and predict the result.

Best of Luck, Eden.


multi channel feature classification using deep feed forward neural network on tensorflow Can any one suggest a dataset that contains multi channel/multi dimensional feature values that can be classified using a deep feed forward neural network?

Please ask me if the question is not clear enough.

You have not specified which deep net you are using. You can use any data set which is of the right format in TensorFlow. If you are using convolutional networks, You may need data which locality is important. In this case:

MNIST data set
ImageNet data set

may help you. If you are doing other tasks I highly recommend you taking a look at this question which is so much popular and has great answers.

If I want to quote, I quote the following answer. Network Repository an interactive data repository with over 600+ networks in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks my help you.

  Data Sets

  Academic Torrents
  Quora
  hadoopilluminated.com
  data.gov
  Quandl
  freebase.com
  usgovxml.com
  enigma.com
  datahub.io
  aws.amazon.com/datasets
  databib.org
  datacite.org
  quandl.com
  figshare.com
  GeoLite Legacy Downloadable Databases
  Quora's Big Datasets Answer
  Public Big Data Sets
  Houston Data Portal
  Kaggle Data Sources
  A Deep Catalog of Human Genetic Variation
  A community-curated database of well-known people, places, and things
  Google Public Data
  World Bank Data
  NYC Taxi data
  Open Data Philly Connecting people with data for Philadelphia
  Network Repository An interactive data repository with over 600+ networks  in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks, etc.
  A list of useful sources A blog post includes many data set databases

Data Sets From awesome-datascience


What does "linear in parameters" mean? The model of linear regression is linear in parameters.

What does this actually mean?

Consider an equation of the form

$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$

where $x$'s are the variables and $\beta$'s are the parameters. Here, y is a linear function of $\beta$'s (linear in parameters) and also a linear function of $x$'s (linear in variables). If you change the equation to

$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \epsilon$

Then, it is no longer linear in variables (because of the squared term) but it is still linear in parameters. And for (multiple) linear regression, that's all that matters because in the end, you are trying to find a set of $\beta$'s that minimizes a loss function. For that, you need to solve a system of linear equations. Given its nice properties, it has a closed form solution that makes our lives easier. Things get harder when you deal with nonlinear equations.

Assume you are not dealing with a regression model but instead you have a mathematical programming problem: You are trying to minimize an objective function of the form $c^Tx$ subject to a set of constraints: $Ax \geq b$ and $x\geq0$. This is a linear programming problem in the sense that it is linear in variables. Unlike the regression model, you are trying to find a set of $x$'s (variables) that satisfies the constraints and minimizes the objective function. This will also require you to solve systems of linear equations but here it will be linear in variables. Your parameters won't have any effect on that system of linear equations.


Should I build a different model for each subset I have a dataset which has categorical variable class. I am trying to solve a regression problem

I am not understanding whether I should build a model on entire dataset and consider variable class as one of the input variable or for each class should I build a different model altogether.

What are general rules which can help me deciding between two approaches.

This is a sample of how my data looks like

+------------+----+-----+-----------------+
|   Class    | X1 | X2  |     Speed       |
+------------+----+-----+-----------------+
| Class1     | 12 | 123 |              10 |
| Class2     | 14 | 120 |              32 |
| Class3     | 15 |  34 |              12 |
|   .        |  . |   . |               . |
|   .        |  . |   . |               . |
|   .        |    |     |                 |
|  Class 300 | 23 |  13 |              45 |
+------------+----+-----+-----------------+

Class is the input categorical variable and I have around 300 classes. The output variable is Speed. I am trying to predict the speed with variables Class, X1, X2.

Should I build a model for each Class separately. So when I know input type is Class1 I will select model built for Class1. When input type is Class2 I will use model built for Class2 and so on. Also the values in Class variable can repeat meaning Class1 can come 4 times, Class2 can come 8 times etc

Other way I was thinking is to include Class itself as a variable and just build one model.

I don't know which would be the correct way for it

Build one model including the class variable as a categorical feature.

Since it is a high cardinality feature, there are different techniques that you can use:

One hot encoding (would create around 300 variables based on different classes)
Label encoding (would be a single variable but will assign weights to different classes - not ideal)
Hashing trick - use a hashing function to reduce the number of features
Embeddings
Cluster the different classes together


how to apply feature selection on LSTM-RNN? am doing my research using lstm-rnn algorithm. i have time-series and non time-series features. how to apply lstm on my dataset? and also how to apply feature selection mechanism to select features?

Considering that Neural Networks (LSTM-RNN in this case), like the rest of deep learning methods, are like black boxes.

The transformation applied are too complex to determine how much a value, variable, weight affects the solution.

Although is a hot topic, there's no current method viable or wide-known used to understand which variables are better or have higher influence in the model.

However, what you can do is an EDA (Exploratory Data Analysis) between your explanatory variables and response variables. Their relationships can give you some skyview of which variables are more correlated or influenced by others.


NER: Extracting entities from an article Description : I have dataset of categorised articles and to extract specific values from respective categorised article I have regex created for each category.

Aiming for:

Nlp techniques which learns the context of the content and avoids/minimizes the use of regex
If some new (similar) article comes up, depending on the learning (from 1) it tries to give the specific values.

Steps taken:

Created a dataframe with various features like : 'Name of the author', 'published date', etc. and got the values from the dataset by using regex

Research_work:

I was considering these options ahead this stage :

Using CNN : it will classify new articles depending on the feature values it learnt on and then use regexs for entity extraction. (It wont achieve the first aim)
Using CRF (medium_article): making use of POS+IOB tagging

Is there any other way around ?
cons of above stated methods?

CRF is a standard for such case but bi-LSTM + CRF are said to be even better (e.g. https://arxiv.org/pdf/1508.01991.pdf). Not sure if you need POS as this is usually solved using the same techniques - may not help in the main task of entity extraction.

Depending how different are articles in specific categories and how much data you have for each of them, you may need to train each category separately and have some classifier in the beginning to decide category - like in your point 1. but word2vec-like classifier may be more robust here.


How to interpret the rec curve for a regression task? I am using forest fire dataset and applied neural network model. I tried to generate REC curve, this is how it looks like. Pretty weird!!!

I have also applied XGBoost but the REC curve is almost parallel to X-axis.
I don't understand how to make sense out of it?
I also want to understand how to interpret REC curve in general?

The title of the graph says "REC curve for various models" but this curve is for a single model. It shows as Y the percentage of "correct" predictions by the model depending on what "correct" means, which is given as X: for example if correct means an error less than 10 ha (on X axis), then the model is correct around 55% of the time (on Y axis). In other words the graph splits the instances between correct/incorrect ones based on whether the error value (difference between true and predicted value) is lower than a threshold X.
In particular this graph shows that:

this model predicts very few values with an error less than 8
this model very often (around 50%) predicts values which are 8-9 ha off the true value.
there are couple more values that the model predicts with an error around 15.
more than 40% of the predicted values are not visible on this graph, which means that their error is higher than 20.

I'd suggest plotting the true vs. predicted value, it's easier to interpret (although it can be harder to see where are the large groups of instances).


How to create pipeline for Neural Network models? I have tried creating a pipeline like this but I am getting error

- AttributeError: 'Pipeline' object has no attribute 'compile'

scaler = StandardScaler()
model = Sequential()

model.add(Dense(120, input_dim=46,activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(80, activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(40, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

pipeline = make_pipeline(scaler,model)
pipeline.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

pipeline.fit(X_train,y_train, epochs=50, batch_size=20, validation_data = (X_test,y_test))

# evaluate the model
scores = pipeline.evaluate(X_test,y_test)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

I am able to achieve this with below code

Thanks  @JahKnows

scaler = StandardScaler()
model = Sequential()

model.add(Dense(120, input_dim=46,activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(80, activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(40, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

pipeline = Pipeline([("scaler",scaler),("model",model)])
pipeline.fit(X_train,y_train, model__epochs=50, model__batch_size=20, model__validation_data = (X_test,y_test))

# evaluate the model
nn_pred = grid_search.predict(X_test)

print('Accuracy of NN on test is:',accuracy_score(y_test,nn_pred))
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


What is the reason for the speedup of transformer-xl? The inference speed of transformer-xl is faster than transformer.

Why?

If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?

For vanilla Transformer language models (Al Rfou et al), you process [1 2 3 4], predict 5, process [2 3 4 5], predict 6, and repeat.

For a Transformer-XL language model, you process [1 2 3 4], predict [5 6 7 8], predict [9 10 11 12], and repeat. Note that we don't have to reprocess [5 6 7 8] when predicting [9 10 11 12], because the representations can be reused.

https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html


Prepending Input layer to pre-trained model I'm trying to input numpy arrays of shape (1036800,) - originally images of shape (480, 720, 3) - into a pre-trained VGG16 model to predict continuous values.

I've tried several variations of the code below:

input = Input(shape=(1036800,), name='image_input')
initial_model = VGG16(weights='imagenet', include_top=False)
x = Flatten()(initial_model(input).output)
x = Dense(200, activation='relu')(x)
x = Dense(1)(x)
model = Model(inputs=input, outputs=x)

Previous variations of the above code yielded errors related to the input being the wrong dimensions, input_shape needing to have 3 channels (when using (1036800,) for that parameter in the initialization of VGG16), and the most recent error that results from running the above code is this:

Traceback (most recent call last):
  File "model_alex.py", line 57, in <module>
    model = initialize_model()
  File "model_alex.py", line 20, in initialize_model
    x = Flatten()(initial_model(input).output)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 596, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2061, in call
    output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2212, in run_internal_graph
    output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 164, in call
    dilation_rate=self.dilation_rate)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3156, in conv2d
    data_format='NHWC')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 639, in convolution
    input_channels_dim = input.get_shape()[num_spatial_dims + 1]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 500, in __getitem__
    return self._dims[key]
IndexError: list index out of range

Here is the full code. Here is the sample data file used in the script.

One of the possible approach towards fixing this might be to resize the raw image files to 224x224 and turn them into numpy arrays of shape (224, 224, 3) so they can be plugged into the pre-trained model's first layer. However, I don't want to warp the images or waste another night pre-processing data when I should already be training.

Besides that, I all I can think to do is Google my problem and try to adapt the found solutions or aimlessly tweak various shape related parameters and functions -- neither of which has gotten me very far over the past 4 hours.

The issue is that you shouldn't flatten the images into 1-dimensional vector because the VGG16 contains 2D convolution layers (e.g. spatial convolution over images), which require the input to have the shape of (number_of_images, image_height, image_width, image_channels), given that keras.backend.image_data_format() returns 'channels_last'. If your image_data_format is 'channels_first', change the input data shape to (number_of_images, image_channels, image_height, image_width).

Here is your fixed code (tested with Keras 2.0.4):

x_train = x_train.reshape((x_train.shape[0], 480, 720, 3))
x_test = x_test.reshape((x_test.shape[0], 480, 720, 3))

initial_model = VGG16(weights='imagenet', include_top=False)
input = Input(shape=(480, 720, 3), name='image_input')
x = Flatten()(initial_model(input))
x = Dense(200, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(1)(x)
model = Model(inputs=input, outputs=x)
model.compile(loss='mse', optimizer='adam')

model.fit(x_train, y_train, epochs=20, batch_size=16)
score = model.evaluate(x_test, y_test, batch_size=16)


How to replace short words into full words from tweets using python I am doing sentiment analysis on tweets. Most of the tweets contains short words and i want to replace them as original/full word.

Suppose that tweet is:

I was wid Ali.

I want to convert:

wid -> with

Similarly

wud -> would

i have 6000 tweets in which there are lots of short words. How i can replace them ? is there any library available in python for this task? or any dictionary of shorts words available online?

i read answer of Replace appostrophe/short words in python Question but it provides dictionary of appostrophe only.

Currently i am using NLTK but this task is not possible with NLTK.

I have worked on lexicon level token normalization on twitter data, which seems to be very similar to what you are trying to do. In our task, we were not only normalizing short word, but also "welllll"->"well", "lysm"->"love you so much", "2"-> to, etc. Refer to the task description here. At that time we did not find a good library directly for this job.

In our approach, we resort to buidling a lookup table to replace suspicious words. The lookup table is compiled from the following resources:

Huge List of Texting and Online Chat Abbreviations

Twitter Dictionary: A Guide to Understanding Twitter Lingo

Lexical normalisation of short text messages: makn sens a #twitter

Insertion, Deletion, or Substitution? Normalizing Text Messages without Pre-categorization nor Supervision

The latter two links are two relevant papers. They both include a dataset that we used in our lookup table.

The suspicious words that need to be replaced can be simply identified by whether they are OOV, or using a more complicated machine learning approach to determine based on context.


logistic regression : highly sensitive model I am a newbie to data science and ML. I am working on a classification problem where the task is to predict loan status (granted/not granted).

I am running a logistic regression model on the data. The accuracy of my model is 82%. However, my model is more sensitive (sensitivity = 97%) and less specific(specificity = 53%).

I want to increase the model's specificity. At this stage, after referring to a bunch of internet resources, I am confused about how to proceed.

Below is my observation :
In Testing data,

a percentage of 1's in the class label is 73.17073170731707.

Testing data has more 1's than 0's in the class label. Is this the reason behind model being highly sensitive.

I am attaching my data file and code file.  Please take a look at it.

Data sample :

Process :
 Data --> missing value imputation -->distribution analysis-->log transformation for normal distribution ---> one hot encoding --> feature selection --> splitting data --> model selection and evaluation

Code snippets :

Here I have selected "3 best features": Credit History, Property Area

How should I proceed? Any help (even if it's just a kick in the right direction) would be appreciated.

Actually, what is happening is natural. There is a trade-off between sensitivity and specificity. If you want to increase the specificity, you should increase the threshold of your decision function but note that it comes at a price and the price is reducing the sensitivity.


Are there databases specializing in scientific data I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository.

I know of proprietary solutions such as Oracle extensions, but are there open source/free systems for scientific data? I would define that as a system that would have the MKSA unit system integrated, and an extensive library of scientific conversions/operations.

SciDB is an open-source DBMS for scientific data.


What does mtry < 1 in TuneRF mean? I'm running a random forest model using the randomForest package in R, using the TuneRF function. I have the option to set the 'step factor', which is how much the mtry parameter is changed at each interaction. It allows me to set an step factor of <1 which doesn't make sense to me because I don't see how it can change the number of variables in a model by less than 1?
What is going on here?

It's a multiplicative factor, so nonintegers is OK.

But a value <1 shouldn't be used: while theoretically setting it equal to 0.5 should be the same as setting it equal to 2, the details under the hood may break:

      ...
      mtryCur <- if (direction == "left") {
        max(1, ceiling(mtryCur / stepFactor))
      } else {
        min(ncol(x), floor(mtryCur * stepFactor))
      }
      ...

(https://rdrr.io/cran/randomForest/src/R/tuneRF.R)
when stepFactor < 1, the ceiling(mtryCur / stepFactor) may exceed ncol(x), and the ceiling is the wrong direction to round, etc.

See also https://stackoverflow.com/q/27211765/10495893


Evaluating new features How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?

Actually in these cases I usually plot the data based on features two by two. This approach is so much similar to watching covariance matrix. In this way you should see the new added features whether they have correlation, whether they change linearly with each of previous features. Suppose that you have already one feature for a classification task. Then you may want to add another feature. You have to plot data with each feature as one axis and investigate whether they have linear correlation or not. If they don't have any correlation or their correlation is near to zero, you should add the feature because that may help you, they provide a kind of knowledge that the previous features didn't provide. If a new feature has correlation with the previous feature, it means that adding that does not help you have a new knowledge or perception of that concept. Although there are debates here I prefer not to add correlated features because of computation complexity. That will be so time consuming.

For illustrating more, suppose you have a data-set A = {X1, X2, y} in which X1 and X2 are the feature and y is the label and all are binary values. Also suppose that the covariance matrix between these is as follows.

We may infer that we can ignore X1 in the process of classification without loss of accuracy, if e is equal to zero which means label y is independent of X1, since they are binary valued. We can ignore X1 in classification, regardless of other values.
We may infer that we can ignore either one of features in the process of classification without loss of accuracy, if |d| >> 0 which means there is high correlation between X1 and X2 and hence information redundancy. So we may ignore one of them without problem.
We may infer that we can not ignore neither X1 nor X2 in the process of classification, if d = 0 and none of e or f are zero. Then the feature are independent and both are influential on label. So we may not ignore any of them.


In doc2vec, how to model correctly when many documents share the same label? Of all the examples I've found for doc2vec training, the documents are uniquely labeled. What happens when many documents share the same label?

TaggedDocument of gensim accepts a list labels for the same text. It implies we can have multiple labels for the same text. However, it's not clear to me if it's a good practice to have fragmented texts under the same label. You can still train and get the embeddings. But are they good?

For example, the question I am posting here has a title, a detailed description, and a list of tags. How do I model it for doc2vec to find similar questions?

Note that some of the tags are not in the title nor the description. What's the best way to include them in the doc2vec trainings. Shuffle them and concatenate with title and description? Or keep them as separate entries under the same label?

I've tried to explain the logic behind labels used in Document vectors in Doc2Vec - How to label the paragraphs (gensim)

To answer your questions.

1) when two documents share the same label, then doc2vec algorithm determines the semantic meaning of the label from both the documents. Note that doc2vec learns the semantic meanings of labels not individual documents.

2) Again, you are not learning documents. you are instructing doc2vec to learn the embeddings for the labels. So, if multiple labels are given for a document, all receives the same semantic meaning from the document and when a part of labels when used in other documents, keep on learning more semantic meaning from them.
For instance. doc1-> hunt, bite, eat, flesh doc2-> life, love, eat, money. It is clear that doc1 is about animals and doc2 is about human, the label eat will have semantic meanings from both of them.

3) If your goal is to find similar question such as this, then you should probably give just a single label for the whole question and then find a question with a label having closer cosine distance.

4) Never confuse labels with words. In word2vec, words learn embeddings and in doc2vec, labels learn embeddings from the words used in documents. If you would like to add some more semantic meanings to a document, then you could add it to the particular document as words. If you want to add semantic meanings to a label, the words have to be added to each document which carries that label if you want the label to have strong affinity (but adding words manually, doesnt sound like a good option to me personally).


ValueError: Error when checking input: expected dense_36_input to have shape (None, 12) but got array with shape (140, 2) create model

model=Sequential()
model.add(Dense(input_dim = 12, units = 10, activation='relu', kernel_initializer='uniform'))
model.add(Dense(units = 8, activation='relu', kernel_initializer='uniform'))
model.add(Dense(units = 1, activation='sigmoid', kernel_initializer='uniform'))

print('Training the model...')

Compile model

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

while running the model.fit()

Fit the model

model.fit(X_train,y_train,batch_size=32, epochs=10, verbose=1)

I am getting the Error

  ValueError: Error when checking input: expected dense_36_input to have shape (None, 12) but got array with shape (140, 2)

@Angel
Input dimetion means how many no of feature or columns your input is supposed to have.
The neural network will try to adjust to this input shape.
In your case you have specified input_dim = 12 but in your data (140,2) there are only 2 features or columns.
So there are two options

Set input_dim =2

Reshape input array to in multiples of specified input dim i..e 12.

In your case ONLY option 1 works because (140,2) cannot be reshaped in multiples of 12.
Note:: other valid input_dim for you can be 14 (reshape input to (20,14)), 10 (reshape input to (28,10))


How can I prevent this model to learn more(less) :))) As you can see, GradientBoostingClassifier overfit with more training example.
These are my parameter for the model:

{'learning_rate': 0.1, 'loss': 'deviance', 'max_depth': 6, 'max_features': 0.3, 'min_samples_leaf': 80, 'n_estimators': 300}

What should I to make my model better or stop training at 350?

Well, by making your model better, I suppose you mean enforcing some sort of regularization on it, so that it doesn't overfit. Early stopping, however can be viewed as a regularization method by itself.

I would suggest sampling the performance of your model more frequently (than the 200 examples you are doing currently) and if you see the validation score is not improving after a few iterations stop the training.

Early stopping can be implemented through the monitor parameter in the GradientBoostingClassifier's .fit() method.


How important is outcome variable scaling in SVM regression? Should I scale outcome variable for SVM regression? What is the magnitude of impact of outcome variable scaling in SVM regression?

This is a cross-posting from CrossValidated:
In support vector regression (with linear loss), we minimise the objective function:
\begin{align}
    \min_{\mathbf{w}, b, \mathbf{\xi}} \quad & \frac{1}{2}\| \mathbf{w}\|^2 + C\sum_i \mathbf{\xi}_i + \mathbf{\hat \xi}_i \\
    \text{s.t.} \quad & (\mathbf{w} \cdot \mathbf{x}_i + b) - y_i \leq \varepsilon + \xi_i \\
    \quad &  y_i - (\mathbf{w} \cdot \mathbf{x}_i + b) \leq \varepsilon + \hat \xi_i \\
    \quad & \xi_i, \hat \xi_i \geq 0
\end{align}
As you can see from the conditions, $\varepsilon$ and the slack variables $\xi_i, \hat \xi_i$ are on the same scale as the output variables $y_i$. So, as you scale $y$, you need to scale $\varepsilon$. On the other hand, in the objective function, $\mathbf{w}$ enters quadratically, and the slack variables only linearly. To keep the balance unchanged, you also need to scale the multiplicative factor $C$.
Assuming you've adjusted the hyperparameters accordingly, the scaling has no effect:
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm

def svmTest(x, y1, sc=1):
  N = x.shape[0]
  x = x.reshape([-1, 1])
  regr = svm.SVR(C=sc*N/100, epsilon=sc*.2)
  y2 = sc*y1
  regr.fit(x, y2)
  y2p = regr.predict(x)

  plt.plot(x, y2, '.', )
  plt.scatter(x[regr.support_], y2[regr.support_], s=80, facecolors='none', edgecolors='r')
  plt.plot(x, y2p, '-')
  plt.gca().set_title(f'Outcome variable scaled by {sc}')

x = np.arange(-1, 1, .01)
y = np.cos(2*np.pi*x)*np.exp(-np.abs(x)) + np.random.normal(0, .1, x.shape[0])
x = x.reshape([-1, 1])

svmTest(x, y, 1) produces:

while svmTest(x, y, 1e6) produces:


ValueError from statsmodels ExponentialSmoothing I've been having a frustrating issue with the ExponentialSmoothing module from statsmodels.

My data is a pandas series with 74 weekly data points that looks like this:

2017-12-31    6069
2018-01-07    8143
2018-01-14    6740
2018-01-21    6433
2018-01-28    6631
2018-02-04    6308
2018-02-11    5536
2018-02-18    6025
2018-02-25    5171
...           ...

When I call the following functions:

model = ExponentialSmoothing(data, trend='add',damped=True,seasonal='mul',seasonal_periods=52)
model_fit = model.fit()

I get:

Traceback (most recent call last):
  File ".\smoothingjuly.py", line 24, in <module>
    model_fit = model.fit()
  File "C:\Users\lhughes\AppData\Local\Programs\Python\Python37\lib\site-packages\statsmodels\tsa\holtwinters.py", line 641, in fit
    l0, b0, s0 = self.initial_values()
  File "C:\Users\lhughes\AppData\Local\Programs\Python\Python37\lib\site-packages\statsmodels\tsa\holtwinters.py", line 773, in initial_values
    b0 = ((lead - lag) / m).mean()
ValueError: operands could not be broadcast together with shapes (22,) (52,)

Why is this? It works if I decrease the number of seasonal periods, but that makes my model useless. Is 74 data points not enough for the model? If so what is the minimum?

Thanks

The error is raised from lead - lag; in initial_values, these are set as y[m:2m] and y[:m] respectively,  where m is the seasonality length (52 in your case). So lead is only getting 22 values, and hence the size mismatch complaint.
So yes, for this to work directly, you'd need at least 104 datapoints.  See the source, and docpage.
(N.B.  This is for version 0.10.  Some changes have been made in v0.12, in which the initialization takes more options.  But OP's context is version-specific, so I'll leave it at that.)
But, you can bypass this bit of the code by passing your own initial_slope to fit*.  I might suggest mimicking statsmodels's approach, but limiting the year-over-year slopes to the range you have available: (y[52:74] - y[:22]).mean() / 52
* this is one change in v0.12; these parameters are passed on model instantiation, not fit.  See the docs if you want to do this in a newer version.


interpreting multi linear regression results am very new to all of this and am taking baby steps learning this (so please be merciful).

I have imported my csv file into python as shown below:

data = pd.read_csv("sales.csv")
data.head(10)

and I then fit a linear regression model on the sales variable, using the variables as shown in the results as predictors. the results are summarised below:

model_linear = smf.ols('sales ~ month + weekend + holiday + prod_function + prod_regime + prod_listprice + discount + stockout', data=data).fit()
print(model_linear.summary())

this is the part I have problems with. how should I interpret and make sense of this result? for instance, what does this say about the impact discount have on sales?

There are a lot going on in a simple OLS model. I strongly encourage you to learn more about them from textbooks. One of the best place to start is the free online book An Introduction to Statistical Learning  (see Chapter 3 about Regression, in which it explains some of the elements in your model summary).

Also in this blogpost, they explain all elements in the model summary obtained by Statsmodel OLS model like R-Squared, F-statistic, etc (scroll down). There are also series of blogposts in blog.minitab, like this one about R-Squared, and this about F-test, that explain in more details each of these elements. You can even explore more and learn about Residual Plots for post analyses of your regression models (important one to check).

Please note, you do not often need to have all these elements to examine a regressor model (some of them like BIC or F-stat are just not that informative, unless someone explicitly ask for them).

The first table i.e. overall model statistics, you can learn about:

R-Squared, which is an estimate of the strength of the relationship between your model and the response variable that lies between 0 (worst) and 1 (best), is 0.278 for your model. While from experience this is quite low, but one can not jump into a conclusion by simply only looking at only R-Squared! I think you can still do better.
F-Stat: It is a statistical test that compare the fit of the intercept-only model with your model. In simple works, if P value for the F-Stat (here Prob (F-Statistic) is less than your significance level, one can reject the null hypothesis that a intercept-only model is better, meaning your model is better a better job (Not a big surprise honestly).

The second table i.e. summary of statistics of your model breakdown:

Gives a lot of information about each variable. Each coefficient with its corresponding standard error, t-statistic, p-value. For instance, for each coefficient if the t-statistic is large (or simply the p-value is ZERO or very very small < 0.05), that particular coefficient is trustworthy like discount or stockout, otherwise the error is large and unlikely any interpretation from them to be reliable like some of the sublevels of prod_function or prod_regime categorical variables in your model. *Just a rough thought about these variables in your model, maybe the type of categorical encoding you do on these is not performing well, that is why the errors here are large?!

How To Interpret Your Model:

This is an interesting part. Taking that your model is good enough (within the defined confidence interval), one can find out how each of these variables contribute to the dependant variable (herein sales). Read more about how Interpreting Regression Coefficients or see this nice and simple example.

Specifically for the discount variable, if all other variables are fixed, then for each change of 1 unit in discount, sales changes, on average, by 0.4146 units (the coefficient of the discount from your model).

Hope these points get you started!


LSTM for time series forcasting I manipulate the time series using the different structures of the neural networks in order to make a prediction, and I wonder if there is a way to choose the parameters of the networks intelligently? from the characteristics of the signal, namely (trend, seasonality ...) can we choose these parameters that will make learning better?

Indeed you can introduce some "unvariant" features to your LSTM network using Conditional RNN that use these features to create the initial hidden state:
https://github.com/philipperemy/cond_rnn
I hope this is what you are looking for.


Gradient descent formula I came across an interesting book about neural network basics, and the formula for gradient descent from one of the first chapters says:

  Gradient descent: For each layer update the weights according to
  the rule

  $w^l \rightarrow   w^l-\frac{\eta}{m} \sum_x \delta^{x,l} (a^{x,l-1})^T$

where $w^l$ is the weights matrix in layer $l$, and $x$ is the index of a specific training example.

I don't want to rewrite all formulas from the chapter, but the important part one is BP4 - equation for the rate of change of the cost with respect to any weight in the network:

$\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j$

Am I missing something or the first formula is incorrect? Shouldn't we use Hadamard product instead, like this?

$w^l \rightarrow
  w^l-\frac{\eta}{m} \sum_x \delta^{x,l} \odot a^{x,l-1}$

Thanks for help.

You can see the result by writing down what you get explicitly in both cases. What you want to get after summation in the update rule is:

$\sum_x \delta^{x,l} (a^{x,l-1})^T = \delta^{1,l} a^{1,l-1}+\delta^{2,l} a^{2,l-1}+...$

which is obtained by the form given in the text.

Keep in mind that Hadamard product can be done only for vectors of same dimension and result in another vector of the same dimension. I suppose you can treat $\delta$ as an $m$ dimensional vector (m being number of training examples) with vector components and take a Hadamard product of it with $a$, but you will end up with another $m$ dimensional vector. And at that point, summing over $x$ becomes meaningless. You can say "Do a Hadamard product and then sum the rows", and this will give the above result.

Edit:

I will try to further clarify the dimensions of the objects given. Let us assume that the layer l has j neurons, and the layer l-1 has k neurons. That makes $w^l$ an $(j x k)$ dimensional matrix. The update equation has to conserve the dimensionality, so the right hand side also has to be matrix of the same size.

Now, $a^{(l-1)}$ is a vector with k components, corresponding to the activations from the neurons of the layer l-1. Likewise, the error of the layer l has j components, one for each neuron.

To get a $(j x k)$ matrix, we take the outer product of these two vectors, that is why there is a transpose in the equation. And finally we sum over all training samples to finalize the calculation.


Regression performance varies hugely on shuffling training and testing data I'm working on a regression problem to predict a variable y based on an input vector X with about 10 columns. To split the data for training and testing, I use the test_train_split method:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = True)

Here, it can be seen that X and y are shuffled entirely along with their indices.
For the purpose of this example, I will consider only one algorithm, RandomForestRegressor:
regr1 = RandomForestRegressor(n_estimators = 200)
regr1.fit(X_train, y_train)
y_pred = regr1.predict(X_test)

When I train the regressor with X_train and y_train from the above method, I get certain results which I consider good. For my application, I would need the model to predict for cases with sequential data (as they would be if not shuffled).
Therefore, I tried the following method to shuffle the training data alone and keep the testing data as is.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = False)
train_data = pd.concat([X_train, y_train], axis = 1)
train_data = train_data.sample(frac=1)
X_train = train_data.iloc[:,0:-1]
y_train = train_data.iloc[:,-1]

In the above method, I first split the data into test/train indices and then shuffle the training data alone and keep the testing data with their original indices. When I train the regressor with the exact same parameters as before and test with X_test, I get significantly poorer results.
I have also tried with both train and test data without shuffling and got bad results as well. I want to be able to train the model so that it can predict for unknown values coming in a sequential order (as it would be in real time).
I'm not able to understand why the shuffling of the test data alone affects the performance, as the model should merely be predicting based on the trained parameters which entirely depend on the training data.

I can't add comments yet, so that's why I'm making a post.
It is natural that for a time series data (I assume that's the kind of data you have if you don't want to shuffle) without shuffling you get worse results. Imagine this toy example. You want to predict sales of a grocery store where your data is sales of the store for every day. If you shuffle you can get dates in your training set that are later than dates in test set. So for example if you have 2nd of March in your training set, it is easy to predict the sales for 1st of March if it's in the test set. However, this situation is not feasible in real life - you don't have data about future when making the prediction.


Deep Q Network gives same Q values and doesn't improve I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit of debugging, I figured out that the Q values the network predicts are the same every time. The action space is [up, right, down, left] and the network predicts [0, 0, 1, 0]. The training loss does go down over time, but it doesn't seem to make a difference. Here's the training code:

def train(self):
    tf.logging.set_verbosity(tf.logging.ERROR)
    self.build_model()
    for episode in range(self.max_episodes):
        self.current_episode = episode
        env = SnakeEnv(self.screen)
        episode_reward = 0
        for timestep in range(self.max_steps):
            env.render(self.screen)
            state = self.screenshot()
            #state = env.get_state()
            action = None
            epsilon = self.current_eps
            if epsilon > random.random():
                action = np.random.choice(env.action_space) #explore
            else:
                values = self.policy_model.predict(state) #exploit
                action = np.argmax(values)
            experience = env.step(action)
            if(experience['done'] == True):
                episode_reward += experience['reward']
                break
            episode_reward += experience['reward']
            self.push_memory(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
            self.decay_epsilon(episode)
            if self.can_sample_memory():
                memory_sample = self.sample_memory()
                X = []
                Y = []
                for memory in memory_sample:
                    memstate = memory.state
                    action = memory.action
                    next_state = memory.next_state
                    reward = memory.reward
                    max_q = reward + (self.discount_rate * self.replay_model.predict(next_state)) #bellman equation
                    X.append(memstate)
                    Y.append(max_q)
                X = np.array(X)
                X = X.reshape([-1, 600, 600, 2])
                Y = np.array(Y)
                Y = Y.reshape([self.batch_size, 4])
                self.policy_model.fit(X, Y)
        food_eaten = experience["food_eaten"]
        print("Episode: ", episode, " Total Reward: ", episode_reward, " Food Eaten: ", food_eaten)
        if episode % self.target_update == 0:
            self.replay_model.set_weights(self.policy_model.get_weights())
    self.policy_model.save_weights('weights.hdf5')
    pygame.quit()

Here's the network architecture:

    self.policy_model = Sequential()
    self.policy_model.add(Conv2D(8, (5, 5), padding = 'same', activation = 'relu', data_format = "channels_last", input_shape = (600, 600, 2)))
    self.policy_model.add(Conv2D(16, (5, 5), padding="same", activation="relu"))
    self.policy_model.add(Conv2D(32, (5, 5), padding="same", activation="relu"))
    self.policy_model.add(Flatten())
    self.policy_model.add(Dense(16, activation = "relu"))
    self.policy_model.add(Dense(5, activation = "softmax"))
    rms = keras.optimizers.RMSprop(lr = self.learning_rate)
    self.policy_model.compile(optimizer = rms, loss = 'mean_squared_error')

Here are the hyperparameters:

learning_rate = 1e-4
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = 1e-5
memory_size = 100000
batch_size = 2
max_episodes = 1000
max_steps = 100000
target_update = 100

I've let it train for the full 1000 episodes and it's pretty bad at the end. Am I doing something wrong with the training algorithm?

EDIT: Forgot to mention that the agent receives a reward of 0.5 for going towards the food, 1 for eating the food, and -1 for dying

EDIT 2: Added updated code for models and hyperparameters

I can see you are using "plain" deepQ, consider using double deep Q or duelling, but go first on your current implementation.

First of all, most of the times the problem is a bug, can you provide the history of the scores?. It is important to understand if the agent is learning or the best solution it can find is killing itself to minimize loses.

As per your hyperparameters, I will suggest two increase gamma first and decay epsilon slower.

Another tweak will be your alpha, reduce it a bit, seems too high.

Anyway, I need more details to help you, for example, Are you reading the screen ?, if so, did you reduce the image to something manageable?
Can you provide the full code?


Is there any optimal way on feature selection for more than one classification algorithms? I have a wine dataset with 13 features that indicates 3 different wine classes (target), and k-NN, SVM with linear kernel and SVM with rbf kernel algorithms to be tried with this dataset.

My goal is to obtain the best classification accuracy, and to obtain this accuracy:

Which classification algorithm (kNN, SVM with linear kernel or SVM with rbf kernel) should I choose?
Among all the features, which ones of them (based on backward elimination, maybe according to p-values) should be chosen?

I have thought of using GridSearchCV with 3 estimators for the algorithms above. But in this case, the problem is feature selection part as you guess. Is there any optimal way to achieve both? Thanks!

If you want to know what the feature importance of a dataset is, you can obtain it by training a random forest. After training the random forest, you can access the feature importance which is valid for any algorithm.

Note that the feature importance of other similar algorithms, such as some boosting algorithms, is strictly related to those algorithms; such thing is not true for random forest.

Hope this helps!

EDIT1:

The reason I said random forests give a sort of universal feature importance is that a rf is based on a lot of smaller decision trees in which each one uses bootstrap from the train set and a subset of attributes taken randomly. The bootstrap tries to avoid overfitting, whereas the subset of attributes helps determine which ones are the most important. A rf is capable of giving the importance of each feature averaging the oob precision from the trees that use the attribute. When a feature  is a great predictor, the trees that use it have better results that those that don't use it. With hundreds or thousands of trees, a rf can have a very good opinion on the predicting capacity of each attribute.

Note that each tree in a rf is a decision tree which bases the split on Gini impurity/information gain (entropy), or variance in the case of regression. This naturally selects the features that matter most in each case.

Here, an article which explores alternative ways of studying feature importance. Some more things are said about rf, but it's not limited to that method, so it may be useful for anyone reading this post.
https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e


Non-Convex Constraints for Classification Problems I am willing to create a hypothetical non-convex constraints for the purpose of practising nonlinear classification using an algorithm. I thought of such constraints in the form: $x^TAx + Bx \leq c$.

I am curious if this qualifies for non-convex constraints, and if the matrices $A$ and $B$ are necessarily PSD. Or could I possibly have more than this constraints?

I would like if someone explains this or refers me to any text/paper I could read up. My Mathematics is kind of rusty at the moment.

Notice that constraint  of the form of $g(x) \le c$ where $g$ is convex is convex.

To see this, consider $g(x_i) \le c$, then for $\lambda \in (0,1)$,

$$g(\lambda x_1 + (1-\lambda)x_2) \le \lambda g(x_1) + (1-\lambda) g(x_2) \le \lambda c + (1-\lambda) c = c$$

To make it non-convex, let $A$ be negative definite or indefinite.


RNN LSTM input conflict due to generator I am trying to do text classification with LSTM RNN on 255 length padded sequences.
My classification data looks like this
1, 'sequence 1'

The first column is the class label for the sequence. Since I am using custom word embedding (100 dims) I have created a custom data generator and used fit_generator api.
My data is converted into following format after preprocessing

X -> (batch_size, time_steps=255, 100) # 100 dim word embeddings
Y -> (batch_size, 2) # 2 for binary class labels

Following is what I have tried with custom generator
def data_generator(file):
    while True:
        with open(file, 'r') as f:
            reader = csv.reader(f)
            for row in reader:
                # Get embedded line encodes each sequence to its constituent word vectors
                # creates a 2D np array of size (max_len, 100)
                yield get_embedded_line(row[1]), to_categorical(row[0], num_classes=2)

input_layer = Input(shape=(255, 100))
lstm1 = LSTM(100, return_sequences=True)(input_layer)
dense1 = Dense(2, activation='softmax')(lstm1)
model = Model(inputs=input_layer, outputs=dense1)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(data_generator('data.csv'), steps_per_epoch=10)

The model defined above is not even compiling, it throws following error

ValueError: Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (100, 1)

get_embedded_line(row[1]) has to return a matrix with size (Batch size, 255, 100) but it is yeilding (100, 1) matrix. please check the implementation of "get_embedded_line" function.


Understanding orthogonal regression In orthogonal regression, we are trying to minimize the distance from each data point $(x,y)$ to the fitted model.

My question is, how come that there is a distinction between independent and dependent variables in orthogonal regression?

In my naive understanding, the fit we are trying to achieve does not distinguish between $x,y$ - we want to find the line that minimizes the sum of distance to each $(x_i,y_i)$.

Nice question.

Short answer: the distinction between independent and dependent variables may not matter to your loss/error function, but it might matter for your modeling function.

Think about fitting a parabola with orthogonal regression.  Here, it's really important which variable is dependent and independent, because the function is not one-to-one.  So I'd say the distinction matters more for "modeling."

In a simple case, such as fitting a line, then any distinction is probably more philosophical than practical.


Improve the loss reduction in a neural network model The following code is to train a neural network model of a given dataset (50,000 samples, 64 dim).

from keras import Sequential
from keras.layers import Dense
from keras.optimizers import Adam

X, y = process_dataset()
model = Sequential([
    Dense(16, input_dim=X.shape[1], activation='relu'),
    Dense(16, activation='relu'),
    Dense(1, activation='sigmoid')
])

'''
    Compile the Model
'''
model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy'])

'''
    Fit the Model
'''
model.fit(X, y, shuffle=True, epochs=1000, batch_size=200, validation_split=0.2, verbose=2)

In the beginning, you can see below that the val_loss gets reduced from one epoch to another very well.

Epoch 82/1000
 - 0s - loss: 0.2036 - acc: 0.9144 - val_loss: 0.2400 - val_acc: 0.8885
Epoch 83/1000
 - 0s - loss: 0.2036 - acc: 0.9146 - val_loss: 0.2375 - val_acc: 0.8901

When the model takes many epochs, the loss change becomes so small, especially when the number of epochs increases.

Epoch 455/1000
 - 0s - loss: 0.0903 - acc: 0.9630 - val_loss: 0.1317 - val_acc: 0.9417
Epoch 456/1000
 - 0s - loss: 0.0913 - acc: 0.9628 - val_loss: 0.1329 - val_acc: 0.9443

Kindly, I have two questions:

What does this phenomenon mean? i.e., the loss begins to decrease very well at the beginning but not much reduction by the time the training epochs takes a lot of iteration.
What is the possible solution for this?

Thank you,

Generally, the decrease in loss tends to be smaller, the longer you train your model. You can think about this in a way, that the model first makes good progress in learning, but later any further improvement becomes harder (thus slower). At some point the model stops to learn. This comes from the logic of gradient decent, a numerical optimization process which is behind most ML models. If the model has learned what it is able to learn, loss does not decrease any more.

What can you do about this? You can try to make your model „better“ in terms of learning capacity. You can increase the capacity of the model (more neurons) or add more layers. You can also adjust the learning rate during learning by „callbacks“ (ReduceLROnPlateau). In this case, you lower the LR automatically if learning progress becomes small. By doing so you can try to make the model to learn more detailed patterns. See callbacks for Keras: https://keras.io/callbacks/.

Here is a nice blogpost about how to train NN: http://karpathy.github.io/2019/04/25/recipe/


How can I use geo location for predictive modelling While determining the house price I don't want to throw away geo location as feature. At least I want to see if there is correlation. Geo-location is more like categorical variable. Is there a way to make it useful feature

Why is it categorical? You can use the longitude / latitude numeric coordinates as two features.


Accuracy drops if more layers trainable - weird I'm training a 4-class classifier. I load pre-trained weights from vgg16, and only set the last layer to be trainable. This gives me a loss (categorical crossentropy) of 0.9 and a training accuracy of 70%.

However, I want features for my images - the penultimate activations is one choice for getting features for every image. Hence I set the last and the penultimate layers to both be trainable, but then my loss jumps to 9.8 and training accuracy drops to 39%.

How can this happen? Here's the code I'm using:

vgg.model.pop()
for layer in vgg.model.layers:
    layer.trainable = False
vgg.model.add(Dense(trn_batches.nb_class, activation='softmax'))
opt = Adam(lr=0.001)

#only the last layer is trainable
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,
    validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

#make last 2 layers trainable (starting from -3 because there is a dropout layer)
for layer in vgg.model.layers[-3:]:
    layer.trainable = True
vgg.model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
vgg.model.fit_generator(trn_batches, samples_per_epoch=trn_batches.nb_sample, nb_epoch=1,   validation_data=val_batches, nb_val_samples=val_batches.nb_sample)

Accuracy is not a function of number of features. You might have heard about feature selection processes, and that is the reason we use feature selection or dimensionality reduction.
There is huge theory behind this, but in very generic terms, we need to find out the best features, to make our model least complex, still keeping all relevant features in the model.
One example, can be adding a new feature reduces accuracy, which is highly biased towards one out come and hence it creates inaccuracy in your results.

This may also happen, if all your features are not normalised, and hence the newly added features is biasing your results and hence introducing inaccuracy.


What model is recommended: I am using text features in a regression and want to interpret coefficients I am using the text of comments on a forum to predict how many upvotes it will get. I want to be able to say, "Reviews with X, Y, Z words are more upvoted". So to do this, I want to use text features in a regression. In particular,

What model should I use to maximize interpretability of coefficients?

I suppose you have a binary outcome (upvote: yes/no). In this case you could use simple linear (ols) regression (with lasso penalty). Each word (in a bag of words) is a „dummy“ here. If you look at the predicted coefficients, you can directly interpret them as „marginal effects“. Higher value means higher chance of getting an upvote (if the word is present). You can also directly see the magnitude of exp. increase in upvote probability.

One problem is that OLS is unconstrained wrt to y. So you can end up with predicted probability of upvote > 1. Use logistic regression if this bothers you. Under logit you will have similar results. Positive/negative coefficients indicate if a word increases/decreases the probability of an upvote. But because logit uses a transformation to squeeze y into a interval from zero to one, the coefficients are the log-odds and you cannot directly infer marginal effects. You would need to calculate this separately.

I tend to say: use OLS and get quick and dirty results if you are not interested in super precise estimates but if you merely look for robust estimates for which words are important.

However, if you really want to do this in a sound way, you would also need to think about „interaction“ of words (positive or negative effect on y), which are „masked“ on approaches as described above.


Notation for features (general notation for continuous and discrete random variables) I'm looking for the right notation for features from different types.
Let us say that my samples as $m$ features that can be modeled with $X_1,...,X_m$. The features Don't share the same distribution (i.e. some categorical, some numerical, etc.). Therefore, while $X_i$ might be a continuous random variable, $X_j$ could be a discrete random variable.

Now, given a data sample $x=(x_1,...,x_m)$, I want to talk about the probability, for example, $P(X_k=x_k)<c$. But $X_k$ might be a continuous variable (i.e. the height of a person). Therefore, $P(X_k=x_k)$ will always be zero. However, it can also be a discrete variable (i.e. categorical feature or number of kids).

I'm looking for a notation that is equivalent to $P(X_k=x_k)$ but can work for both continuous and discrete random variables.

As far as I am concerned, there is no distinction between a continuous and a discrete variable when it comes to notation. So $P(X_k=x_k)$ is perfectly fine for either.


How to use SMOTENC inside the Pipeline? I would greatly appreciate if you could let me know how to use SMOTENC.  I wrote:

num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,123:160]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))

pipeline=Pipeline(steps= [
    # Categorical features
    ('feature_processing', FeatureUnion(transformer_list = [
            ('categorical', MultiColumn(cat_indices1)),

            #numeric
            ('numeric', Pipeline(steps = [
                ('select', MultiColumn(num_indices1)),
                ('scale', StandardScaler())
                        ]))
        ])),
    ('clf', rg)
    ]
)

Therefore, as it is indicated I have 5 categorical features. Really, indices 123 to 160 are related to one categorical feature with 37 possible values which is converted into 37 columns using get_dummies.

I think SMOTENC should be inserted before the classifier ('clf', reg) but I don't know how to define "categorical_features" in SMOTENC. Besides, could you please let me know where to use imblearn.pipeline?

Thanks in advance.

As it follows, two pipelines should be used:

num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:120,121:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,120]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
cat_indices = [94, 96, 98, 99, 120]

from imblearn.pipeline import make_pipeline

pipeline=Pipeline(steps= [
    # Categorical features
    ('feature_processing', FeatureUnion(transformer_list = [
            ('categorical', MultiColumn(cat_indices1)),

            #numeric
            ('numeric', Pipeline(steps = [
                ('select', MultiColumn(num_indices1)),
                ('scale', StandardScaler())
                        ]))
        ])),
    ('clf', rg)
    ]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices), pipeline)


Using LSTM for multi label classification I am trying to use LSTMs to train and predict authors using reviews data and metadata

author  phone  country day     review
james   iphone chile   tuesday the book was really amazing

How do i pass all these features into the network?

Since the review of an author is bound to change in terms of the number of words being used in the review, I would suggest using a Keras Sequential() model to build an LSTM encoder for the review itself. The final hidden layer of the Review LSTM encoder can then be fed into another LSTM encoder with 3 words (phone, country and day). Think of the last LSTM encoder as a sequential 3 worded message. The final layer of this LSTM can then be joined with a softmax layer to predict the author.

The reason I suggested two different LSTM has something to do with word embeddings. Word embeddings are basically features that get modified in multiple hidden layers in an LSTM. Having two different word embeddings means that the same word in the Review Embedding may mean different in the other LSTM embedding, ie have different embedding vectors after training. It ensures that the features are independent. You can argue that you might need 4 different LSTM and Embedding Layers because of 4 different features. That is definitely a viable option but it might need a lot more time to train. Having more LSTM will add hyperparameters significantly and training time will be really high.

That being said you may not need an LSTM to solve this problem. You can easily convert the phone, country and day values into integers. There are 7 days in a week so the value of day can only be one from 0 to 6. That reduces the dimension of features significantly, reducing training time. You can probably try a TF-IDF for the review feature or a naive bag of word model and use a Random Forest or SVM Classifier for the prediction of Author. It would be interesting to perform multiple experiments with and without LSTM and observe results with accuracy and performance.