Looking for spatial clusters and anomalies. Is DBSCAN the right tool? I have a regular 2D grid of data points (X, Y) with each point having a value.
I'd like to identify clusters and then anomalies that don't belong to those clusters.
I'm trying to understand the best way to do this and it looks like DBSCAN might be an appropriate algorithm to use but I'm a bit confused.
I'm not clear whether I should collapse the 2D (X,Y) data down into 1 dimension and run DBSCAN on the 1D array. That would lose the spatial component of the data but would then identify clusters based on the value of the data point.
Or do I run DBSCAN on a 3D array (the X index, the Y index and it's corresponding data value for that X,Y point)? That sounds intuitively like it might be the better way to approach this but I'm not 100% sure.
If anyone could point me towards an example of using DBSCAN for spatial clustering that'd be really helpful.

Clustering in X,Y,Value
Clustering in 3d is great. But be careful with feature scaling in this case. Presumably X, Y have the same scales - so unless you want to treat the different directions differently, make sure not not apply any normalization. As that would distort the grid.
Your Value column on the order hand, might be on a very different scale from the X,Y values. If not corrected for this creates an implicit and usually quite arbitrary feature weighting, which is rarely desirable.
If you want to weight the Value column equally to X and Y, then scale the Values by the either mean,std or max,min of the X/Y columns.
If you do want to weight the individual attributes differently, then you can introduce additionally some weighting factors that scales them.
PS: remember that DBscan hyperparamters, such as epsilon is also given in the same unit as your feature space.
Anomalies in Value given X,Y location
Often an interesting question is "is this value anomalous compared to the normal for this location?". This can be done on X and Y axis individually, but often there are interactions between X,Y and one should consider the two dimensions together. Here are some approaches for that:
Binning the 2D (X,Y) space
Divide the X and Y axis into a grid. Typically with uniformly sized cells. Assign each grid cell an location identifier. Then use this identifier as a feature.
This approach is simple and with a lot of dense data it usually works pretty OK. But the grid can fit the data pretty poorly, datapoints very close to eachother end up in different cells, and then quite distant in feature space.
Clustering in the 2D (X,Y) space
An alternative location scheme is to use clustering. Cluster the values just based on X,Y - and then use the cluster number as the location identifier. You may also want to include distance from the cluster center as a feature - since some values might fall pretty far from a cluster.
Angle from the cluster can in some cases also be informative.
This has a much better shot at creating good location features for the data. But is more complex. One also should consider when one wants to re-cluster.
Aside
Make sure to plot your data and look at it! For example use a 2d-plot with X,Y as coordinates, and the values using color. You may also want to do the same kind of plot, but then to use Anomaly Score as the color. If you upload a plot here, it will also be much easier to give good advice.


Does Tensorflow support a Decision Tree Classifier? I am trying to implement decision tree classifier to classify my data set. I am using Python. Now it is easy to implement in scikit learn, but how can I implement this in tensorflow.

Basically I guess TensorFlow does not support decision trees. I quote from here,

  This is a big oversimplification, but there are essentially two types of machine learning libraries available today, Deep learning
  (CNN,RNN, fully connected nets, linear models) and Everything else
  (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc). The reason for
  this is that deep learning is much more computationally intensive than
  other more traditional training methods, and therefore requires
  intense specialization of the library (e.g., using a GPU and
  distributed capabilities). If you're using Python and are looking for
  a package with the greatest breadth of algorithms, try scikit-learn.
  In reality, if you want to use deep learning and more traditional
  methods you'll need to use more than one library. There is no
  "complete" package.

You can see from here that there are other learning algorithms implemented in TensorFlow which are not deep models.

You can take a look at here for tracking algorithms implemented in TensorFlow.


Loading pretrained model with Pytorch I saved my model with this code:
from google.colab import files
torch.save(net, 'model.pth')

# download checkpoint file
files.download('model.pth')

Then uploaded this way and checked on an image (x):
model = torch.load('model.pth')
model.eval()
torch.argmax(model(x))

And on the old session, it worked great, but then I started a new session and tried the code above, and got such an error:
AttributeError: Can't get attribute 'EfficientNet' on <module '__main__'>

Maybe somebody knows how to deal with it?

Got the very same error recently.
Your network is usually defined as a class (here class EfficientNet(nn.Module). It seems when we load a model, it needs the class to be defined so it can instantiate it.
In my case, the class was defined in the training .py file. So what I did to fix that error was just copy-paste (it seems importing it didn't work for me, so I had to copy paste) the whole class definition to the file where you load your model.
In my case the class definition looked something like that and needed to be pasted before using the torch.load function:
class FCNClass(nn.Module):
    def __init__(self, in_feat=1000, nb_classes=3, nb_hid=1, hidden_size=500, act=nn.ReLU):
        super(FCNClass, self).__init__()

        self.act = act()
        self.flat = nn.Flatten()

        self.fc1 = nn.Linear(in_feat, hidden_size)
        self.fcs = nn.ModuleList()
        for nb in range(nb_hid):
            self.fcs.add_module("hid" + str(nb), nn.Linear(hidden_size, hidden_size))
        self.out = nn.Linear(hidden_size, nb_classes)

    def forward(self, x):
        x = self.flat(x)
        x = self.fc1(x)
        x = self.act(x)
        for lay in self.fcs:
            x = lay(x)
            x = self.act(x)
        x = self.out(x)
        x = self.act(x)
        return x

torch.load('Model.h5')

The class basically defines the architecture of the model, if it is not defined, then the load function doesn't know where to put the weights. Mine was done with torch.load(path) and not model.load_state_dict(torch.load(path)) but it should not make a difference


multi regression for energy data I'm trying to develop a multi regression model to predict energy consumption during one day period.
X-set dimension is (10178, 52) and consist of 52-feature and Y-set dimension is (10178, 48) as output. I have used the following code:
xtrain, xtest, ytrain, ytest=train_test_split(X, Y, test_size=0.1)

in_dim = X.shape[1]
out_dim = Y.shape[1]
model = Sequential()

model.add(Dense(48*4, input_dim=in_dim, activation="relu"))

model.add(Dense(86, activation="relu"))
model.add(Dense(out_dim))
model.summary()
model.compile(loss="mean_absolute_error", optimizer="adam")
model.fit(xtrain,ytrain, epochs=100, batch_size=12,)

after compiling my model although my model's loss is very low but when I visualize my output the result is unsatisfying as follow:

any idea what I'm doing wrong?!? my initial guess is that since output dimension is high(48-dimension) compared to input dimension I need a lot more Data. or maybe I'm using wrong loss function or the model is too shallow. also it is noticeable that model's output at spark point is very poor.

As you can see, your predictions are able to catch the trends. In other words, the model is able to predict the direction of movement almost every day.
The only point that it is not able to catch is those high peaks, which can be treated as outliers. It is because due to seasonality or some other cause your daily data drastically changes on some of the days. This change is not normal for models to capture because those points deviate from the general characteristics of your time series.
It is quite normal having low energy consumption for 6 consecutive days but having a large energy consumption on the 7th day if you would consider that 6 days were sunny and suddenly the weather gets cold. This is just one single case where there might be lots of those.
To capture these anomalies you should have a variable to explain to them (e.g. Image that those anomalies are only due to weather conditions, then including weather variable would help you. However, it is too hard to find all those variables that explain all anomalies).
To solve the issue, you can decrease the frequency of your data. That is, instead of modeling daily data you can model weekly average or weekly end (means last day of every week), or even monthly average or monthly end.
If these outliers are consistent every week, averaging them will help you. In any case, I think decreasing the frequency of your data will help you.
To summarize, there is not a wrong thing with your model, it is just outliers make it unhandy in some predictions.


What kind of neural network structure is suitable for image to image learning? There exists a mapping from input image to out image. Say input image is a piece of paper with a square hole in the center, and output image is the shadow of the input image when light shines on the paper, which also looks like a square, but with some blurs and different in size. (The real pattern is far more complicated than this square hole) I'm trying to implement a neural network that can learning this kind of mapping, when given the input image, the network can predict the "shadow" of the input image. I've tried to use a convolutional autoencoder-like structure (like this https://blog.keras.io/building-autoencoders-in-keras.html) by replace the target image from the input image it self to the output image training set. But it didn't looks work well. I would like to know what kind of neural network is suitable for this task? And what kind of loss  function is suitable?

I would like to know what kind of neural network is suitable for this task?

A convolutional autoencoder is a good thing to try, but there is a minor issue with it for this type of problem: the latent representation has often lost spatial awareness (especially if FC layers were used at the end of the encoder). For a task like this, a pixel $p_{ij}$ in the input is likely to be far more related (and useful) to the corresponding (same position) pixel $\widehat{p}_{ij}$ in the output, compared to some other, far away pixel $p_{\ell k}$ in the input.

For this reason, I'd suggest using a U-net architecture, as this preserves spatial localities throughout the network.

References:

Long et al, Fully Convolutional Networks for Semantic Segmentation
Ronneberger et al, U-Net: Convolutional Networks for Biomedical Image Segmentation

  And what kind of loss function is suitable?

A good starting point is the standard mean $L_1$ or $L_2$ loss over the output image. Since your images are not natural images, I hesitate to recommend perceptual losses, but they could work as well.

Without more details about your architecture and data, I'm afraid I cannot say much more than these though.


How prevalent are traditional CS questions in data science interviews? I've started interviewing for data science roles and I've noticed that a lot of companies ask traditional computer science questions. I have no formal CS education. In fact, I was a business major in college which is pretty much the opposite. I don't know anything about traditional CS algorithms like binary search, bubble sort, or dynamic programming.

I have a very strong background in machine learning / data science but pretty much everything I know is through years of Googling, doing interesting ML side projects, Kaggle, and generally learning things the hard way. I know that a data scientist needs to be able to code. Ironically I'm a professional software engineer and I code all day. I just don't have a theoretical background. Do I need to read something like Introduction to Algorithms and brush up on CS undergrad work before I start applying to data science jobs in earnest?

Please note: I know that there are a lot of other questions on here about what to study to become a good data scientist, but this question is subtly different. I personally disagree that knowledge of traditional CS algorithms is relevant for most day-to-day data science work and that knowing these things makes you a better data scientist.

Most machine learning algorithms (gradient boosting, random forests, linear models, SVMs, neural nets, etc) are available in easy-to-use libraries like caret (R); scikit-learn, and TensorFlow (Python); or H2O, and MLlib (Scala/Java). There are also easy-to-use tools, like Spark, to make these algorithms scale. I feel that understanding how the learning algorithms (like gradient descent) work is relevant, but I don't agree that recursion or dynamic programming is relevant. Am I wrong? Should I be more open minded?

It depends on whether you're going to write production code or even pseudo-code that will be implemented by others. If so, yes you need computer science skills. If you're going to merely analyze data, you can technically get away without it, but some companies (e.g., Facebook) expect all technical employees to know computer science.

Computer science is useful even to pure analysts because it will help you implement algorithms not covered by your libraries.

I make use of recursion and dynamic programming, for example, but I wrote production code. That's my deliverable, after analysis and prototyping.


finding themes from text documents I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find keywords arranged in a cluster. I tried to auto summarize using markovify library in python but the summary it created or the sentences did not make much sense. Any suitable suggestions are welcome. Thank you

The best method to find themes in a collection of documents is topic modeling. Topic modeling finds the hidden (aka, latent) themes beyond just keyword counts.

There are many approaches to topic modeling. Latent Dirichlet allocation (LDA) is a standard topic modeling approach. LDA is a probabilistic graphical model that assumes that each document is a mixture of a small number of topics and that each word's creation is attributable to one of the document's topics. The number of topics is a selected hyperparameter.


what is difference between fit and fit_transform in sklearn while applying feature scaling I have seen few post related to this question but i am not quite clear about my confusions as mention bellow.
I have some confusion related to fit and fit_transform.
suppose, I have X_train and X_test data, and let my scaling function is standard scalar.
I am using following code for scaling,
sc_X =StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.fit_transform`
My question is, if i use same scalar on bot trainin and testing data, wont it creat problem of data leakage?
What if I use the code like below,
sc_X_train =StandardScaler()
sc_X_test =StandardScaler()
X_train = sc_X_train.fit_transform(X_train)
X_test = sc_X_test.fit_transform(X_test)
Does the both code give different results?

In real life problems you may not not have the actual “test data” to fit on, for example in some time series forecasting problems. My recommendation if you want to keep safe and avoid data leakage is to fit on training data and transform on test.


Predicting game scores using sklearn I am using onehotencoding and RandomForestRegressor to predict scores of a set of soccer games.
How can I use it into predict? I am sure I am doing it wrong at the moment as I am getting all predict values to be 1 (Probably because I am filling all NaN values as 1 for splitting and fitting)
What dataset should I pass as I am encoding a few columns and then transforming it?
My code is as below
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeRegressor

# Pandas needs you to define the column as date before its imported and then call the column and define as a date
# hence this step.
date_col = ['Date']
df = pd.read_csv(
    r'C:\Users\harsh\Documents\My Dream\Desktop\Machine Learning\Attempt1\Historical Data\Concat_Cleaned.csv'
    , parse_dates=date_col, skiprows=0, low_memory=False)

# Clean dataset by dropping null rows
data = df.dropna(axis=0)

# Column that you want to predict = y
y = data.Full_Time_Home_Goals

# Columns that are inputted into the model to make predictions (dependants), Cannot be column y
features = ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result']
# Create X
X = data[features]

# Split into validation and training data
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

# Define and train OneHotEncoder to transform numerical data to a numeric array
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(train_X, train_y)

transformed_train_X = enc.transform(train_X)
transformed_val_X = enc.transform(val_X)

# Build a Random Forest model and train it on all of X and y.
# To improve accuracy, create a new Random Forest model which you will train on all training data
rf_model_on_full_data = RandomForestRegressor()

# Define columns we want to use for prediction
columns = ['Home_Team', 'Away_Team']
test_data = test_data[columns]
# Renaming Column Names to match with training dataset
test_data = test_data.rename({'Home_Team': 'HomeTeam', 'Away_Team': 'AwayTeam'}, axis=1)
# Adding NaN columns to dataset to match the training dataset
test_data['Full_Time_Result'] = np.nan
test_data['Full_Time_Away_Goals'] = np.nan
test_data['Full_Time_Home_Goals'] = np.nan
# Aligning dataframe to model defined
test_data_features = test_data[features]
# Filling all NA values as Encoder cannot handle nan values
df = test_data.fillna(1)

# Define Y for Fitting
Y = df

# We need nY as that would be the column used for splitting
ny = df.Full_Time_Home_Goals

# We need to encode and transform dataset so we have converted all nan to 1 and we are defining a new model as the
# val_x values are confusing, we will use n_
train_n_X, val_n_X, train_n_y, val_n_y = train_test_split(Y, ny, random_state=1)

# Since we have text again, we will need fitting and transforming the data
enc.fit(train_n_X, train_n_y)
transformed_train_n_X = enc.transform(train_n_X)
transformed_val_n_X = enc.transform(val_n_X)

# Fitting and then we will be using predict
rf_model_on_full_data.fit(transformed_train_n_X, train_n_y)

# Predicting. This step needs correction as predict should be on the new dataset and not just on on column.
test_preds = rf_model_on_full_data.predict(transformed_val_n_X)

print(test_preds)

What should go into predict() to get the outcome that I want?
Files used here

If I understand correctly, you're trying to predict the target variable 'Full_Time_Home_Goals' based on the four features ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result'] (the first two being being categorical and one-hot-encoded).
In your test set, you only receive (or only use) the first two, and fill the last two (which are, naturally, the more important ones) with a value of 1.
Now, let's imagine that your model decides that the teams aren't important, and makes the prediction based solely on the two latter variables... but in your test set, those all have the same value - so they'll get the same prediction.
The lesson here is that when you're training a model, do not use features that are (mostly or completely) missing from the test set.


Jaccard Similarity with Binary Data I have 5400 rows of data and 3211 columns of attributes.

The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity measures.

Huge dimensionality, I know, but I wanted to (as a first step) just see how this data clusters and finds similarity between all attributes.

I converted all attributes values to "0" if there was no value, and "1" if there was a value. I thought it'd be an easy first step to get me started with a clustering visual and similarity metric if I converted the values to binary.

Jaccard similarity seems to be a good measure for binary, but I'm stumped as to how to implement this (in Python) when I don't have any lists for comparison. Am I supposed to hard code each variable into the algorithm (3207 variables)?

I'm not sure where to start. Also, if there's a better way of doing this, I'm all ears. I've been researching how best to tackle this problem and there's so many similarity metrics that can be used, but I'm stuck on how to start since there are so many columns that need to be used.

The DBSCAN clustering algorithm has a built-in Jaccard distance metric.

from sklearn.cluster import DBSCAN
db = DBSCAN( metric='jaccard' ).fit(X)
labels = db.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)

Where X is your dataset with the related columns you want to use.


What is the different between Fine-tuning and Transfer-learning? Usually the neural network training has at least 2 steps:

first trained on a large set of some standard data (ImageNet, ...)
and then the resulting weights are trained on a small set of my data (in this step we can train all layers or only one last layer)

What is the same of 2-nd step, is it Fine-tuning or Transfer-learning? And what is the different between Fine-tuning and Transfer-learning?

Generally, I would refer to this as transfer learning or network adaptation. That is, taking a network that has learned useful features from one domain and adapting that network and its developed features to another domain.

That said, there appear to be many sources that closely conflate fine tuning with transfer learning. Therefore, I would say the difference in terminology is primarily opinion-based and suggest closure of this question on those grounds.


What if My Word is not in Bert model vocabulary? I am doing NER using Bert Model. I have encountered some words in my datasets which is not a part of bert vocabulary and i am getting the same error while converting words to ids. Can someone help me in this?
Below is the code i am using for bert.
df = pd.read_csv("drive/My Drive/PA_AG_123records.csv",sep=",",encoding="latin1").fillna(method='ffill')

!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py

import tensorflow_hub as hub
import tokenization
module_url = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(module_url, trainable=True)

vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)

tokens_list=['hrct',
 'heall',
 'government',
 'of',
 'hem',
 'snehal',
 'sarjerao',
 'nawale',
 '12',
 '12',
 '9999',
 'female',
 'mobile',
 'no',
 '1155812345',
 '3333',
 '3333',
 '3333',
 '41st',
 '3iteir',
 'fillow']

max_len =25
text = tokens_list[:max_len-2]
input_sequence = ["[CLS]"] + text + ["[SEP]"]
print("After adding  flasges -[CLS] and [SEP]: ")
print(input_sequence)

tokens = tokenizer.convert_tokens_to_ids(input_sequence )
print("tokens to id ")
print(tokens)
```

The problem is that you are not using BERT's tokenizer properly.
Instead of using BERT's tokenizer to actually tokenize the input text, you are splitting the text in tokens yourself, in your token_list and then requesting the tokenizer to give you the IDs of those tokens. However, if you provide tokens that are not part of the BERT subword vocabulary, it will not be able to handle them.
You must not do this.
Instead, you should let the tokenizer tokenize the text and then ask for the token IDs, e.g.:
tokens_list = tokenizer.tokenize('Where are you going?')

Remember, nevertheless, that BERT uses subword tokenization, so it will split the input text so that it can be represented with the subwords in its vocabulary.


About batches in stateful RNN ..., to create proper consecutive batches, where the nth input sequence in a batch starts off exactly where the nth input sequence ended in the previous batch.
  Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (Kindle Locations 12018-12020). O'Reilly Media. Kindle Edition.

So data for a stateful RNN looks like:

Why not like the below?

Why does stateful use batches like the bottom one? As far as i know, it seems like the bottom one's hidden state has longer memory. (e.g. seq 4, seq 1-2-3, but the top one, seq 4, just seq 3)

Maybe some reasons is there?

It is not a good idea to keep the statefulness like your second figure, because this would prevent the parallelization of the computations.

A stateful RNN (LSTM, GRU, etc) saves the last hidden state and uses it as the initial state for the next batch. If you do it like in your first figure, all sentences in a batch can be computed in parallel. However, for the second figure, you need to compute the last hidden state in sequence 1 to use it for sequence 2, meaning that you need to compute the LSTM for sequence 1 only, then for sequence 2, then sequence 3, and so on, effectively making your batch size 1 and wasting GPU memory.


Generalization bound (single hypothesis) in "Foundations of Machine Learning" I have a question about Corollary $2.2$: Generalization bound--single hypothesis in the book "Foundations of Machine Learning" Mohri et al. $2012$.

Equation $2.17$ seems to only hold when $\hat{R}_S(h)<R(h)$ in equation $2.16$ because of the absolute operator.  Why is this not written in the corollary?  Am I missing something important?

Thank you very much for reading this question.

You are right. The relaxed inequality
$$R(h) \le  \hat{R}_S(h)+ \epsilon.$$
can be replaced with the complete inequality
$$\left |\hat{R}_S(h) - R(h) \right| \le \epsilon.$$
Actually, authors use this complete inequality for the follow up examples in the book. Again in Theorem $2.13$, they write the relaxed inequality, but prove for the complete inequality.

We could say that the relaxed inequality is written for the sake of readability and/or convention.

On the relation of inequalities

Let us denote:
$$A:=\hat{R}_S(h) - R(h) \le \epsilon$$
$$B:=\hat{R}_S(h) - R(h) \ge -\epsilon$$
thus,
$$\left| \hat{R}_S(h) - R(h) \right| \le \epsilon = A \text{ and } B$$
Equation $(2.16)$ states:
$$\begin{align*}
& {\Bbb P}(\left| \hat{R}_S(h) - R(h) \right| \ge \epsilon) \le \delta \\
& \Rightarrow {\Bbb P}(\left| \hat{R}_S(h) - R(h) \right| \le \epsilon) \ge 1 - \delta \\
& \Rightarrow {\Bbb P}(A \text{ and } B) \ge 1 - \delta \\
\end{align*}$$
knowing that ${\Bbb P}(B) \ge {\Bbb P}(A \text{ and } B)$,
$$\begin{align*}
& {\Bbb P}(B) \ge {\Bbb P}(A \text{ and } B) \ge 1 - \delta \\
& \Rightarrow {\Bbb P}(\hat{R}_S(h) - R(h) \ge -\epsilon) \ge 1 - \delta
\end{align*}$$
which is equivalent to
$$R(h) \le \hat{R}_S(h) + \epsilon$$
with probability at least $1-\delta$, i.e. equation $(2.17)$.


How to speed up optimization using Differential Evolution? My application is high frequency trading.  My data are time series of the bid and ask prices of a stock recorded on every tick (change in price). For each data point I also have a certain indicators that predict the future movement of the price. The indicators have different horizons of the predictions, some being optimal at few second intervals and others few minutes. I need to assign these predictors weights and based on whether the linear combination crosses a threshold, the decision will be taken to buy of sell the stock. So far I have tried the Differential Evolution (DE) method to figure out the weights. I use a black box model with the weights vector $w_i$ and threshold as inputs. For each data point I have a vector of indicators $\alpha _i$.
$$
total\_alpha = \sum\alpha _i*w_i
$$
If
$$
total\_alpha > threshold, BUY
$$
Else If
$$
total\_alpha < -threshold, SELL
$$
The output of the model is the sum of difference between each between the price of each consecutive buy and sell. This output is being optimised by the DE algorithm.

I am having trouble with the computational aspects. My data is very large (~$7 \times 10^8$ rows by 20 columns), and thus the execution time for the DE algorithm is unacceptable.

My question: Is there a better and a faster way to solve this problem?

Do you run your analysis algos in batch or live? Which programing language under which environment do you use?

At a first naive look, i would recommend to parallelize your code as each indicator calculation seems to be independent of the other's.


What type of neural network should I use to detect meteors in images? I am currently building a project that takes fisheye images from cameras and detects whether the picture contains a meteor, and if it does it tries to identify where the meteor is.

The images look like this:

I am currently building a project that takes fisheye images from cameras and detects whether the picture contains a meteor.

The images look like this:

The 'circle' in the middle is the actual sky, everything outside it is just black pixels

The meteor is the long flash of light in the middle, the small flashes of light on the 'sides' are lights that come from the cities nearby.

Using opencv's HoughLines I was able to create a program that detects the meteors, however it also detects the lights on the 'sides', so my idea was to train a neural network to only detect the meteors.

How would I go about doing this? I was thinking of using a U-Net neural network, using the images and masks of the images where only the meteor is 'detected' for training, sort of like this example.

This is not a NN solution but can you adjust the parameters to opencv's HoughLine? There should be a way to find just the straight lines. There should be a way to adjust the minimum line length value to be considered a meteors. Meteors will always appear straight. So only keep the results when the points have similar slopes and are near each other?

See this example:

import cv2
import numpy as np

img = cv2.imread('dave.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,50,150,apertureSize = 3)
minLineLength = 100
maxLineGap = 10
lines = cv2.HoughLinesP(edges,1,np.pi/180,100,minLineLength,maxLineGap)
for x1,y1,x2,y2 in lines[0]:
    cv2.line(img,(x1,y1),(x2,y2),(0,255,0),2)

cv2.imwrite('houghlines5.jpg',img)

Source


Word Frequency Analysis of Document Sets I'm doing some work trying to extract commonly occurring words from a set of human classified documents and had a couple questions for anyone who might know something about NLP or statistical analysis of text.

We have a set of a bunch of documents, and users have classified them as either good or bad. What I'd like to do is figure out what words are common to the good documents, but not necessarily the other ones.

I could, for example, use the (frequency within good documents / total frequency) which would essentially normalize the effect of a word being generally common. This, unfortunately, gives very high precedence to words that occur in only a few good documents & not at all in the other documents. I could add some kind of minimum threshold for # of occurrences in good docs before evaluating the total frequency, but it seems kind of hacky.

Does anyone know what the best practice equation or model to use in this case is? I've done a lot of searching and found a lot of references to TF-IDF but that seems more applicable for assessing the value of a term on a single document against the whole set of docs. Here I'm dealing with a set of docs that is a subset of the larger collection.

In other words, I'd like to identify which words are uniquely or more important to the class of good documents.

There are many algorithm to do classification: Naïve Bayes, logistic regression, SVM, decision tree..etc. My suggestion is to try Naïve Bayes first by calculating below probabilities which a new document belongs to $class_{good}$ or $class_{bad}$. (https://web.stanford.edu/class/cs124/lec/naivebayes.pdf)

$$
P(Class_{good} \vert document_{new}) = \frac{P(document_{new} \vert Class_{good}) \cdot P(Class_{good}) }{P(document_{new})}
$$

$$
P(Class_{bad} \vert document_{new}) = \frac{P(document_{new} \vert Class_{bad}) \cdot P(Class_{bad}) }{P(document_{new})}
$$

And generally, when we are doing text mining questions, we will do several preprocess on one document:

Tokenization(1-gram/bigram/...etc)
remove stop words ('a', 'the' 'at', ... etc)
Stemming: transforms a word into its root form. (studied => study)

My suggestion is to do above preprocesses and try more features not just the words in one document, if there are some metadataes.


How to obtain unique count of categorical variable based on another categorical variable? I have a dataframe with date and cat_1 variables among other variables. I want to get number of unique values of cat_1, per date. Here is an example: Below is a toy dataframe.

         date foo
0  2019-04-21   a
1  2019-04-21   b
2  2019-04-23   c
3  2019-05-01   c
4  2019-05-01   c
5  2019-04-23   a

I want to get is the following:

2019-04-21    2
2019-04-23    2
2019-05-01    1

I got it, but I think it is a bit convoluted.
My code:

a.groupby(by='date')['foo'].apply(np.unique).apply(len)

Is there any other straightforward way to get the same answer?

Thanks

Try nunique(). That should do it. Here is a toy example:

import pandas as pd

df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                   'B' : ['one', 'one', 'two', 'three',
                          'two', 'two', 'one', 'two']})

print(df)

gives

     A      B
0  foo    one
1  bar    one
2  foo    two
3  bar  three
4  foo    two
5  bar    two
6  foo    one
7  foo    two

and

print(df.groupby('A')['B'].nunique())

then gives

A
bar    3
foo    2
Name: B, dtype: int64

For more details check this question on SO.


How to construct a Decision tree in R where the training data has a frequency associated with each class Essentially, this is my data set

X Class    Sex   Age Survived Freq
1   1st   Male Child       No    0
2   2nd   Male Child       No    0
3   3rd   Male Child       No   35
4  Crew   Male Child       No    0
5   1st Female Child       No    0
6   2nd Female Child       No    0
7   3rd Female Child       No   17
8  Crew Female Child       No    0
9   1st   Male Adult       No  118
10  2nd   Male Adult       No  154
11  3rd   Male Adult       No  387
12  Crew   Male Adult       No  670
13  1st Female Adult       No    4
14  2nd Female Adult       No   13
15  3rd Female Adult       No   89
16  Crew Female Adult       No    3
17  1st   Male Child      Yes    5
18  2nd   Male Child      Yes   11
19  3rd   Male Child      Yes   13
20  Crew  Male Child      Yes    0
21  1st Female Child      Yes    1
22  2nd Female Child      Yes   13
23  3rd Female Child      Yes   14
24  Crew Female Child      Yes    0
25  1st   Male Adult      Yes   57
26  2nd   Male Adult      Yes   14
27  3rd   Male Adult      Yes   75
28  Crew  Male Adult      Yes  192
29  1st Female Adult      Yes  140
30  2nd Female Adult      Yes   80
31  3rd Female Adult      Yes   76
32  Crew Female Adult      Yes   20

If there were no frequency but only single valued data, then I know how to invoke rpart to construct a decision tree for me. How to do it considering the frequency of each class?

I am a beginner in R. Thanks

To use the column named Freq as your case weights, you can call rpart with the argument weights=Freq.


Do I have to standardize my new polynomial features? I have a vector X with n features previously standardized.

If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new features after the computing ?

Because knowing that my original features are already between 0 and 1, the new polynomial features will also be between 0 and 1. But the higher the degree is, the higher the "ratio" between the original one and the polynomial one will be.

Also,  would that be better to do the square computing on the values of the non standardized features, and then standardize these new features ?

By the way, when I say "standardization", I m thinking about substrat each value by the mean and divide it by the std.

Thanks in advance,

Regards.

It's better to do the square computing on the values of the non standardized features, and then standardize these new features.

The point of standardizing features is to put features in a similar scale, e.g. 0-1. If you didn't standardize these new features after the computing, although your new features are between 0-1, your new features will be one or more magnitude smaller than your old features.


How to get a similarity vector from two vectors? I want to make a classification model for 3 classes,  i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer.

inputs = Input(shape=(2,n_timesteps))
embedding_inputs = embedding_layer(inputs)

sentence1 = Lambda(lambda x: x[:,0,:,:])(embedding_inputs)
sentence2 = Lambda(lambda x: x[:,1,:,:])(embedding_inputs)

conv_sentence1 = Conv1D(filters=64, kernel_size=3, activation='relu',
input_shape=(n_timesteps,n_features))(sentence1)
conv_sentence2 = Conv1D(filters=64, kernel_size=3, activation='relu',
input_shape=(n_timesteps,n_features))(sentence2)

pooling_sentence1 = MaxPooling1D(pool_size=2)(conv_sentence1)
pooling_sentence2 = MaxPooling1D(pool_size=2)(conv_sentence2)

flat_sentence1 = Flatten()(pooling_sentence1)
flat_sentence2 = Flatten()(pooling_sentence2)

concat_senrences = concatenate([flat_sentence1,flat_sentence2])
dense_layer = dense(50)(concat_senrences)
dense_prediction = dense(3,activation='softmax')(dense_layer)

but i get an early overfetting, so i thought that the problem comes from "sentence 2", each observation has an unique "sentence 1", ,instead, "sentence 2" can be exists in several observations, in that case the neural network relies strongly on it, so i want to combine two sentences and apply an unique CNN layer, that's why i asked how to obtain a similarity vector.

thanks !!!

If I understand your question correctly, you have 2 sentences and you converted those sentences into 2 vectors, you want to know how those sentences are similar.
If this is the case use cosine similarity between 2 sentences and cosine similarity is a scalar not a vector its the dot product of your 2 embedding vectors.

from numpy import dot
from numpy.linalg import norm

cos_sim = dot(a, b)/(norm(a)*norm(b))

where a and b are your vectors.

There is direct formula as well:

from sklearn.metrics.pairwise import cosine_similarity

cosine_similarity(vector1,vector2)


In handwritten digit recognition problem using logistic regression, what changes needed to add another class "Not a Digit" In handwritten digit recognition problem using logistic regression, normal implementation would forcibly classify even a picture of dog or cat as a digit. To eliminate this, what changes are needed to add another class i.e. "Not a Digit" to already existing 10 classes (0 to 9) ?

Logistic regression is normally used to perform binary classification, which answers a yes or no question, e.g.:

Is this an 8 or not?
Will it rain today or not?

Perhaps I have misunderstood your explanation, but it sounds like you are trying to perform multi-class classification, i.e. to classify an image as one of a certain number of options. So a single image must contain (be classified as) a number from the list: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].

Usually, to add an extra class to a model that already does this, you would need to adjust the final output of the model to predict a one-hot vector that is simply one element longer!

In the specific case of the Scikit-Learn LogosticRegression class, it seems as though you don't need to specify anything - the class will automatically use a multinomial model and relevant optimiser as soon as it sees that data is not binary (i.e. a yes-no model as explained above).

Have a look at this official tutorial, which should a multinomial model trained on a dataset with a target variable that has a total of 20 classes. The number of features (n_classes is equal to 20) is not passed to the model at all, it is inferred.


Should I modify dataset? I am working on a dataset which contains more than 80 features, and thousands of instances. Among those features there are some nominal ones, such as IP SOURCE, IP DESTINATION, Flow ID, which don't have any signification for my machine learning model. My question is, should I remove those features manually, or do I have to replace their values with numerical ones?

You don't need to remove your features from your data set you may just drop the features which you don't want considered as features or lables.  Dropping a feature or label works by excluding the columns while fitting the data into the training set.


Can I modify a Logistic Regression classifier to out put more than one class based on the probabilty? I'm training a Logistic Regression classifier on text data. I found that many of my data points have more than one target class. Is it possible to modify my model to output more than one class based on the data.
I plan to split multi-class data points in my training set into distinct classes(i.e. if one x has 3 classes, I will split that text into three so that each different text has a unique class associated with it). Then when I predict on the test data if I will output n classes such that

Probability(Class_1)+Probabilty(Class_2)+...+Probabilty(Class_n)>0.95

I will use the prdict_proba method of LogisticRegression for this.
Is it a correct way of doing this?. your help is much appreciated.

We can't use the same data with a different label in the same model.
It will confuse the model.
What you need is multiple models e.g. OneVsRest strategy. i.e. one for each Class.
Little clarification-
What you are looking for is Multi-label, not Multi-class. Please check the internet on this. May read this SO Answer for a quick understanding.
You can implement an OneVsRest multi-label yourself but better use the Scikit-learn implementation Link.

OneVsRestClassifier can also be used for multilabel classification. To use this feature, provide an indicator matrix for the target y when calling .fit.

What it means is to encode your data in a multi-label format when calling fit().
You may use the from sklearn.preprocessing import MultiLabelBinarizer for this.
classif = OneVsRestClassifier(LogisticRegression())
classif.fit(X, Y)

Read this SO Answer for an example.
You may also use a sklearn.multioutput import MultiOutputClassifier wrapper for the same. Check this SO Answer


Selecting models for ensemble from large group of models with high uncertaintly I'm in a situation where many models have been created, and I have their cross-validation performances as well as performance on test data. I need to select models for inclusion in a simple bagging ensemble that are most likely to generalize to new data.
Conventional wisdom would dictate choosing models with high CV performances and low correlation with each other, since each individual model should have a good chance of generalizing and there will be an error-correction effect with the model diversity.
However, it seems like given a large number of models, the odds of having a high test AUC by chance are not insignificant, and therefore choosing uncorrelated models could actually be more dangerous since that lack of correlation could indicate that they've found vastly different mechanisms to reach a high CV performance, one of which could be incorrect/overfit. Perhaps the safer way is actually to pick models within a certain range of correlation (like Pearson or Spearman in the range of 0.7-0.9, for example), to maintain some error-correction effect while ensuring that the mechanism is fairly consistent (and therefore, perhaps reliable).
I've been searching for literature on this, and haven't been able to find anything. I'd really appreciate any guidance on how to approach this, or what papers to read - thanks!

The general principle of ensemble learning is indeed to rely on the diversity of the individual learners rather than their performance. Therefore it's ok to include models which perform well by chance (typically because they're overfit), since if there is sufficient diversity across the models it's very unlikely that two models would be overfit in the same way, i.e. that they would wrongly predict the same instances by chance. So if most models perform reasonably well in general through different mechanisms, it is expected that for any given instance most models will predict the right answer and only a few will be wrong.
The risk of using models which have a quite high correlation together is to defeat the purpose of ensemble learning and obtain a performance similar to the best individual learner.
Note also that the risk of overfitting is quite low if the individual learners have been tested with CV. It could also be worth checking the variance of the performance across CV runs (high variance indicating potential instability), but even that is not so useful in my experience: it's really the diversity of the learners which makes ensemble learning work optimally.
But as usual a lot depends on the actual data/task, so I'd suggest properly testing a few different approaches.


What is a good method for detecting local minims and maxims? I'm using kernel density estimation in order to compute
probability density function for item occurrence.
Using this output, i want to find all the local minims and maxims.
I'm interested in different methods for local-min/max extraction.

One way to do it is to calculate first derivative (difference in discrete domain) and find where there is a change in the sign. That indicates the existence of a local minimum or maximum. For example:

from numpy import diff, sign, cos, pi, arange
import matplotlib.pyplot as plt
t = arange(0,2,0.01)
x = cos(2*pi*t)
# Find derivative of x
first_derivative = diff(x)

# Calc. sign difference
sign_diff = sign(first_derivative[1:]) - sign(first_derivative[:-1])

# Find local min and max
local_max_index = [i for i,k in enumerate(sign_diff) if k == -2]
local_min_index = [i for i,k in enumerate(sign_diff) if k == 2]

# plot results
plt.figure()
plt.plot(t,x)
plt.plot(t[local_max_index],x[local_max_index], 'ro')
plt.plot(t[local_min_index],x[local_min_index], 'ro')

Hope it helps!


How to cross-validate a deep learning model for highly imbalanced datasets? I am working with a multi-modality classification problem (with Keras). I have 1000, 5000 and 10000 samples for three different classes. I would like to do a five fold cross-validation to select the best pre-trained deep learning model for deployment. I'm including class weights during model training to give more weightage for less-pronounced classes. For a given fold, I would be validating with 200, 1000 and 2000 samples from these three classes. Is accuracy a good prediction measure to be used in this case? Or do I have to measure the F1-score and Matthews correlation coefficients as well? Am I doing it right?

Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include roc_auc_score (see here) and average_precision_score (see here), both defined through scikit-learn.

If you're using Keras, I would recommend using class_weights (note this will not work well if you have a multi-label problem, although there are some workarounds, for example here).


sklearn and pandas in AWS Lambda I made a front end where I would like to make REST calls to an AWS Lambda interfaced with AWS API Gateway.

I dumped my model as a pickle file (and so my encoders) which I initially trained locally. I then stored these files in a S3 bucket.

The problem is that I cannot import libraries such as pandas and sklearn to make model predictions because the lambda console is unable to find them.

Does anyone have any suggestions to help solve this issue?

Using layers will allow the dependency to be more reusable and potentially easier to maintain and deploy.

The short version is:

create a requirements.txt from pip freeze or similar that looks like this:

pandas==0.23.4
pytz==2018.7

create get_layer_packages.sh bash script to be run by docker which looks like this:

#!/bin/bash

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

Run the stuff from above like this in the terminal:

chmod +x get_layer_packages.sh
./get_layer_packages.sh
zip -r my-Python36-Pandas23.zip .

I'm not so experienced with python and spent a decent chunk of time messing around with zipping pandas and virtual envs up, and have never really used docker for anything IRL, but this process is actually far more accessible (and better documented) than the venv > zip > upload process I was using before.


Evaluating Recommendation engines What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare with precision/recall/F measure of  different algorithms ? Is this the best way for evaluation ? Or is there any other way to compare results of various recommendation algorithms ?

The standard way to evaluate a recommendation engine is by using the RMSE (root mean square error) of the predicted values and the ground truth.

It is almost a SOP that, after finishing developing a recommendation engine, we will evaluate this engine by comparing its RMSE with other famous, common recommendation algorithms like SVD, tranditional CF, even RBM, etc.

Some terms mentioned above do not seem to be related with recommendation, but you can easily find on the internet how these techniques can be used in this topic.


What is Spectral clustering? What is spectral clustering?
I have little background in statistics. I have tried to search for notes online but they assume quite a lot of knowledge.

Would be good if you are able to find some notes online which teach the basics and the math foundations for spectral clustering

I have found notes like this one which require a lot of background knowledge which are not suitable for me

Spectral clustering is an algorithm for data clustering that uses the spectrum (eigenvalues) of the similarity matrix of the data to perform dimensionality reduction before clustering in fewer dimensions.

The similarity matrix includes the relative similarity of each pair of points in the available dataset.


MAE,MSE and MAPE aren't comparable? I'm a newbie in data science. I'm working on a regression problem. I'm getting 2.5 MAPE. 400 MAE 437000 MSE. As my MAPE is quite low but why I'm getting high MSE and MAE?  This is the link to my data

from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import Normalizer
import matplotlib.pyplot as plt
def mean_absolute_percentage_error(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100

import pandas as pd
from sklearn import preprocessing

features=pd.read_csv('selectedData.csv')
import numpy as np
from scipy import stats
print(features.shape)
features=features[(np.abs(stats.zscore(features)) < 3).all(axis=1)]
target = features['SYSLoad']
features= features.drop('SYSLoad', axis = 1)
names=list(features)

for i in names:
    x=features[[i]].values.astype(float)
    min_max_scaler = preprocessing.MinMaxScaler()
    x_scaled = min_max_scaler.fit_transform(x)
    features[i]=x_scaled

Selecting the target Variable which want to predict and for which we are finding feature imps

import numpy as np
print(features.shape)
print(features.describe())
from sklearn.model_selection import train_test_split
train_input, test_input, train_target, test_target =
train_test_split(features, target, test_size = 0.25, random_state = 42)
trans=Normalizer().fit(train_input);
train_input=Normalizer().fit_transform(train_input);
test_input=trans.fit_transform(test_input);

n=test_target.values;
test_targ=pd.DataFrame(n);

from sklearn.svm import SVR
svr_rbf = SVR(kernel='poly', C=10, epsilon=10,gamma=10)
y_rbf = svr_rbf.fit(train_input, train_target);
predicted=y_rbf.predict(test_input);
plt.figure
plt.xlim(300,500);
print('Total Days For training',len(train_input)); print('Total Days For
Testing',len(test_input))
plt.ylabel('Load(MW) Prediction 3 '); plt.xlabel('Days');
plt.plot(test_targ,'-b',label='Actual'); plt.plot(predicted,'-r',label='POLY
kernel ');
plt.gca().legend(('Actual','RBF'))
plt.title('SVM')
plt.show();

test_target=np.array(test_target)
print(test_target)
MAPE=mean_absolute_percentage_error(test_target,predicted);
print(MAPE);
mae=mean_absolute_error(test_target,predicted)
mse=mean_squared_error(test_target, predicted)
print(mae);
print(mse);
print(test_target);
print(predicted);

I'll be honest, I haven't thoroughly checked your code. However, I can see that the range of values of your dataset is approx [0,12000]. As an engineer, I see that:

sqrt(MSE) = sqrt(437000) = 661 units.
MAE = 400 units.
MAPE = 2.5 which means that MAE can be up to 0.025*12000= 250 units.

All three cases show similar magnitude of error, so I wouldn't say that "MAPE is quite low but you're getting high mse and MAE".

Those 3 values explain the results from similar yet different perspectives. Keep in mind, if the values were all the same, there would have been no need for all 3 of those metrics to exist :)


Why do convolutional neural networks work? I have often heard people saying that why convolutional neural networks are still poorly understood. Is it known why convolutional neural networks always end up learning increasingly sophisticated features as we go up the layers?
What caused them create such a stack of features and would this also be true for other types of deep neural networks?

Actually I guess the question is a bit broad! Anyway.

Understanding Convolution Nets

What is learned in ConvNets tries to minimize the cost function to categorize the inputs correctly in classification tasks. All parameter changing and learned filters are in order to achieve the mentioned goal.

Learned Features in Different Layers

They try to reduce the cost by learning low level, sometimes meaningless, features like horizontal and vertical lines in their first layers and then stacking them to make abstract shapes, which often have meaning, in their last layers. For illustrating this fig. 1, which has been used from here, can be considered. The input is the bus and the gird shows the activations after passing the input through different filters in the first layer. As it can be seen the red frame which is the activation of a filter, which its parameters have been learned, has been activated for relatively horizontal edges. The blue frame has been activated for relatively vertical edges. It is possible that ConvNets learn unknown filters that are useful and we, as e.g. computer vision practitioners, have not discovered that they may be useful. The best part of these nets is that they try to find appropriate filters by their own and don't use our limited discovered filters. They learn filters to reduce the amount of cost function. As mentioned these filters are not necessarily known.

In deeper layers, the features learned in previous layers come together and make shapes which often have meaning. In this paper it has been discussed that these layers may have activations which are meaningful to us or the concepts which have meaning to us, as human beings, may be distributed among other activations. In fig. 2 the green frame shows the activatins of a filter in the fifth layer of a ConvNet. This filter cares about the faces. Suppose that the red one cares about hair. These have meaning. As it can be seen there are other activations that have been activated right in the position of typical faces in the input, the green frame is one of them; The blue frame is another example of these. Accordingly, abstraction of shapes can be learned by a filter or numerous filters. In other words, each concept, like face and its components, can be distributed among the filters. In cases where the concepts are distributed among different layers, if someone look at each of them, they may be sophisticated. The information is distributed among them and for understanding that information all of those filters and their activations have to be considered although they may seem so much complicated.

CNNs should not be considered as black boxes at all. Zeiler et all in this amazing paper have discussed the development of better models is reduced to trial and error if you don't have understanding of what is done inside these nets. This paper tries to visualize the feature maps in ConvNets.

Capability to Handle Different Transformations to Generalize

ConvNets use pooling layers not only to reduce the number of parameters but also to have the capability to be insensitive to the exact position of each feature. Also the use of them enables the layers to learn different features which means first layers learn simple low level features like edges or arcs, and deeper layers learn more complicated features like eyes or eyebrows. Max Pooling e.g. tries to investigate whether a special feature exists in a special region or not. The idea of pooling layers is so useful but it is just capable to handle transition among other transformations. Although filters in different layers try to find different patterns, e.g. a rotated face is learned using different layers than a usual face, CNNs by there own do not have any layer to handle other transformations. To illustrate this suppose that you want to learn simple faces without any rotation with a minimal net. In this case your model may do that perfectly. suppose that you are asked to learn all kind of faces with arbitrary face rotation. In this case your model has to be much more bigger than the previous learned net. The reason is that there have to be filters to learn these rotations in the input. Unfortunately these are not all transformations. Your input may also be distorted too. These cases made Max Jaderberg et all angry. They composed this paper to deal with these problems in order to settle down our anger as theirs.

Convolutional Neural Networks Do Work

Finally after referring to these points, they work because they try to find patterns in the input data. They stack them to make abstract concepts by there convolution layers. They try to find out whether the input data has each of these concepts or not in there dense layers to figure out which class the input data belongs to.

I add some links which are helpful:

Understanding convolution operation
Understanding arithmetic behind ConvNets
Useful toolboxes for tracking what happens in these nets


Input normalization for ReLu? Let's assume a vanilla MLP for classification with a given activation function for hidden layers.

I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid is the activation function and -0.5 and 0.5 if tanh is the activation function.

What about ReLu ?

Should I normalise the network input between 0 and 1, -0.5 and 0.5, or -1 and 1

Any known best practices there?

I am not talking about normalisation of the input of the ReLu like using Batch Normalisation just before or just after the ReLu : https://arxiv.org/pdf/1508.00330

But I am talking about normalising the input of the whole network.

You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping to other small intervals near to zero may also be fine but the latter case usually takes more time than the other. If you use ReLU, again based on experience, you have to normalize your data and use standard initialization techniques for your weights, like He or Glorot methods. The reason is that your should avoid each activation to be so large, because your net would be so much dependent to that activation, and you may have overfitting problem. When you use ReLU because there is no limit for its output, you have to normalize the input data and also use initialization techniques that avoid having large values for weights. For more information I encourage you taking a look at here and here.


Tensorflow and Sklearn Is there a way to feed tensorflow tensors into a sklearn model? I have the following model to set up data compression:

with tf.name_scope('model'):   with tf.name_scope('setup'):
    token_size = X_train[0][0].shape[1]
    batch_size = 64
    dropout_1 = tf.placeholder_with_default(1. , shape=(), name='dropout_1')
    dropout_2 = tf.placeholder_with_default(1. , shape=(), name='dropout_2')
    # [None, 5, 19, 400]
    inputs = tf.placeholder(tf.float32, shape=(None, FLAGS.eval_beam_size, largest, token_size), name='inputs')
    label = tf.placeholder(tf.float32, shape=(None, FLAGS.eval_beam_size), name='label')

  with tf.name_scope('layer1'):
    hid_size_1 = 20
    # [400, 5]
    w1 = tf.Variable(tf.random_normal([token_size ,hid_size_1], stddev=0.01), name='w1')
    b1 = tf.Variable(tf.constant(0.1, shape=([hid_size_1])), name='b1')
    y1 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(inputs, w1, 1), b1)), dropout_1, name='dropout_1_op')

    tf.summary.histogram('layer1_weights', w1)
    tf.summary.histogram('layer1_bias', b1)

  with tf.name_scope('layer_2'):
    hid_size_2 = 5   # [400, 5]
    w2 = tf.Variable(tf.random_normal([hid_size_1, hid_size_2], stddev=0.01), name='w2')
    b2 = tf.Variable(tf.constant(0.1, shape=([hid_size_2])), name='b2')
    y2 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(y1, w2, 1), b2)), dropout_2, name='dropout_2_op')

    tf.summary.histogram('layer2_weights', w2)
    tf.summary.histogram('layer2_bias', b2)

  with tf.name_scope('layer_3'):
    hid_size_3 = 1
    w3 = tf.Variable(tf.random_normal([hid_size_2, hid_size_3], stddev=0.01), name='w3')
    b3 = tf.Variable(tf.constant(0.1, shape=([hid_size_3])), name='b3')
    y3 = tf.nn.dropout(tf.nn.tanh(tf.add(tf.tensordot(y2, w3, 1), b3)), dropout_2, name='dropout_3_op')

    tf.summary.histogram('layer3_weights', w3)
    tf.summary.histogram('layer3_bias', b3)

with tf.name_scope('support_ops'):   summed = tf.squeeze(tf.reduce_sum(y3, axis=2), axis=-1, name='summed')

From here I would like to be able to feed the outputs into a sklearn model and perhaps leave these layers to train based on the model output. Is this possible?

Yes it is possible. Once you actually return the results from the Tensorflow model, they will (by default) be returned as NumPy arrays. You can then use them as input e.g. to a SciKit Learn model.

Have a look at this thread, which shows some nice examples the types returned by TF models.

One last consideration: The model must return results in such a way that they are in the normal memory, reachable for the CPU - not somehow left in GPU memory.


Confusion around categorical cross entropy I understand the binary cross entropy formula for a problem with a single label 0 or 1. If we have more than 2 labels we sum this binary cross entropy over all these classes.

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

I always thought the above was categorical cross entropy but it turns out its the following:

$$ H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i) $$

I'm pretty sure the two formulas are actually different but I think both would work. I realise the second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's), but still I don't see why we can't just use the first formula for all problems whether the labels are mutually exclusive or not? What is the need for the second formula and why is it proffered over the first?

For binary classification, they are the same thing. Imagine $y' \in \{0,1\}$

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

$$  = -
\begin{cases}
     \sum_{i} \sum_{c} \log(y_{i, c}), &\text{if}&  y_{i, c}' = 1  \\
     \sum_{i} \sum_{c} \log (1-y_{i, c}), &\text{if}&  y_{i, c}' = 0
\end{cases}$$

  second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's)

This condition is multi-labels which is a different problem. In that case, you shouldn't even use Sigmoid or Softmax. See another answer for multi-labels loss

The second one is used for multi-class classification.

Binary Classification Loss

Cross entropy Loss


How to treat the datasets with unreliable labels I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the samples in class C is actually belong to class A and class B. For now, I need to run some supervised learning (logistic regression, decision tree and random forests) models. According to the confusion matrix, the classification between A and B is relatively accurate, but the classification between C and other two classes is not acceptable.
I wondered whether is any way to deal with this issue?

For now, I'm considering to use a clustering algorithm for the samples in class C before runnning the model. After dividing the samples in class C into 3 groups, try to find a relatively better group as the datasets for class C.

You have a common issue in which your learning data is noisy. Of course if you can do anything to make your data better/use more data this is the way to go. But I assume manual cleaning at large is not possible.

Reasons for the noise?

There are different approaches in machine learning to deal with noisy data. It depends a lot on where your noise is coming from. It helps to understand why your labels are incorrect. In your case it sounds like there is a structural mistake where you systematically misclassify under certain conditions.

ML methods to solve the noisy data issue

Besides simple rules (if applicable) there are more complex ways to approach the issue. There are many papers (e.g. this, this and this) that suggest different variants of the following:

Manually create a small subset of cleaned labels
Train the network on clean and noisy labels
Use an additional network (or layer) to either create a mapping from noisy to clean labels or to learn about the noise distribution

This way you try to make the network recognizing its own mistake. Some of these models are rather complex, others are quite complex. There is even a paper that trains the network to also classify the noise and than make predictions based on features + noise classification.

I am not sure if there are any best practises yet which architecture works best in which case. You may need to try out some alternatives.


how to reconstruct image from feature space of convolutional neural network? lets say i have fed an image into VGG19 pre-trained on imagenet as follows:
from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg19 import preprocess_input
from tensorflow.keras.models import Model
import numpy as np

base_model = VGG19(weights='imagenet')
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block5_pool').output)

img_path = 'C:/shared/a5/images/training/n01518878_8432.JPEG'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

features = model.predict(x)

i then extract the values at layer5 max pool layer, which are determined by the input image, and unique for every different image that is fed into the network.
does anyone know how to reconstruct the original image based on the features extracted from a specific layer? thank you

One choice is to train a neural network model to take these values and output original images.
Notice that usually some data is loss in this process so it might be impossible to reconstruct the image with perfection.
You could try inverting the functional form but:

CNNs usually use ReLu activation which is not bijective.
Pooling layers throws information away (it still could be fixed if we had redundant information)
The weight matrix might not be inversable


Is there a difference if input nodes have discrete or range value If you have some input nodes containing fruit values like {apple=0, pear=1, oranges=2} vs temperature values like {5, 10, 30, 50}.
Is there any difference on how you set up the neural network to learn the output?
I'm guessing on the first case the neural network only learn about the input you used to train. For example if you then try with banana the result will be something random.
But in the case of temperature if I input 35 I would expect a result similar to 30.
Also, I'm guessing for the first case would need much more data to learn than the second case. So wondering if have to take different consideration on each case.

If categorical values are 1-hot encoded, then there is a node for each item in the category. There would be an input note that indicate the presence or absence of a categorical value. There would be an apple node, a pear node, or a orange node. If apple is present, then the apple input node would take a value of 1; 0 otherwise.
Numerical values have one node per feature. The activation node have higher value if the numerical value is higher. Typically, these input node are rescaled between 0 and 1. The activation of an input node can take on any value between 0 and 1.
In the case of temperature, it depends on how temperature is measured. If temperature is measured continuously, then it can be encoded as a numerical value. If temperature can only take a limited subset of values, then can not be encoded as a numerical value. It would be measured on a ordinal scale.


Clustering unbalanced dataset The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did Clustering on the data along with the labels found in the previous task.

Is this a right way to go?

What you could try is to make a transformation of the price data, i.e. log(price). By doing so, very large values become smaller. The data is more "harmonic" after the transformation.


Extracting keywords from pdf file with python I have a pdf file(link below).

I have to extract keywords from it and also need have there frequency in pdf file. E.g. 'Java':42, 'inheritance':3.

I have to store the keyword with their weights in an excel sheet.

Can you please tell me some code in python to do it.

https://drive.google.com/file/d/1gZCnlhwVMBIE0SugUUxDIgQrfVz-cDQR/view

This is an interesting question!  Let's convert all your PDF files to Word Documents, so you can loop through each one, and read all the contents of each, and then report everything in MS Excel.

Sub OpenAndReadWordDoc()

Rows("2:1000000").Select
Range(Selection, Selection.End(xlDown)).Select
Selection.ClearContents
Range("A1").Select

    ' assumes that the previous procedure has been executed
    Dim oWordApp As Word.Application
    Dim oWordDoc As Word.Document
    Dim blnStart As Boolean
    Dim r As Long
    Dim sFolder As String
    Dim strFilePattern As String
    Dim strFileName As String
    Dim sFileName As String
    Dim ws As Worksheet
    Dim c As Long
    Dim n As Long
    Dim iCount As Long
    Dim strSearch As String

    '~~> Establish an Word application object
    On Error Resume Next
    Set oWordApp = GetObject(, "Word.Application")
    If Err Then
        Set oWordApp = CreateObject("Word.Application")
        ' We started Word for this macro
        blnStart = True
    End If
    On Error GoTo ErrHandler

    Set ws = ActiveSheet
    r = 1 ' startrow for the copied text from the Word document
    ' Last column
    n = ws.Range("A1").End(xlToRight).Column

    sFolder = "C:\Users\Excel\Desktop\test\"

    '~~> This is the extension you want to go in for
    strFilePattern = "*.doc*"
    '~~> Loop through the folder to get the word files
    strFileName = Dir(sFolder & strFilePattern)
    Do Until strFileName = ""
        sFileName = sFolder & strFileName

        '~~> Open the word doc
        Set oWordDoc = oWordApp.Documents.Open(sFileName)
        ' Increase row number
        r = r + 1
        ' Enter file name in column A
        ws.Cells(r, 1).Value = sFileName

        ActiveCell.Offset(1, 0).Select
        ActiveSheet.Hyperlinks.Add Anchor:=Sheets("Sheet1").Range("A" & r), Address:=sFileName, _
        SubAddress:="A" & r, TextToDisplay:=sFileName

        ' Loop through the columns
        For c = 2 To n
            If oWordDoc.Content.Find.Execute(FindText:=Trim(ws.Cells(1, c).Value), _
                    MatchWholeWord:=True, MatchCase:=False) Then

                    strSearch = ws.Cells(1, c).Value
                    iCount = 0

                    With ActiveDocument.Content.Find
                        .Text = strSearch
                        .Format = False
                        .Wrap = wdFindStop
                        Do While .Execute
                            iCount = iCount + 1
                        Loop
                    End With

            ws.Cells(r, c).Value = iCount
            End If
        Next c
        oWordDoc.Close SaveChanges:=False

        '~~> Find next file
        strFileName = Dir
    Loop

ExitHandler:
    On Error Resume Next
    ' close the Word application
    Set oWordDoc = Nothing
    If blnStart Then
        ' We started Word, so we close it
        oWordApp.Quit
    End If
    Set oWordApp = Nothing
    Exit Sub

ErrHandler:
    MsgBox Err.Description, vbExclamation
    Resume ExitHandler
End Sub

Function GetDirectory(path)
   GetDirectory = Left(path, InStrRev(path, "\"))
End Function

Test the code above and feed back.  My setup looks like the image below.


Public dataset for news articles with their associated categories for multilabel data classification I am wondering if there are any public datasets of news, like New York Times (NYT) or similar to various news categories such as politics, entertainment, lifestyle, general news, sports etc.
I want to use such a dataset for multilabel data classification of various sentences or paragraphs i.e a sentence could belong to politics, entertainment, sports, or all, I need the dataset to classify the data into more than one label. I was planning to train a classifier with such a dataset and use it for predictions. However, I couldn't find any. Are there any such known datasets available?
I want dataset something like this but for news category

I found one on Kaggle by searching "news categories". I believe this dataset should work for you. Its a JSON file which contains a link to the article, an associated category (e.g. "crime"), headline, authors, date, and a short description.
You may also want to check out the Open Data Stackexchange:
https://opendata.stackexchange.com/
Enjoy!
https://www.kaggle.com/rmisra/news-category-dataset


What are common ways to identify subject and object from a question? I am looking for a way to extract the potential  subject and object from a question in French.

For the moment, I am building some handmade rules.
Alternatively, I started to think of using an already trained model for the task.
I wanted to use StanfordNLP parser but unfortunately it is not free for commercial use.

So, here comes the questions:

Are they ML models used for this task?
Are they labeled data sets (in french) which can help create ML models used for this task?

[Edit] I am looking mainly for open source tools/ libraries.

Google SyntaxNe has pre-trained models for French.  It is open source and can be customized for a specific use-case.

https://github.com/tensorflow/models/blob/master/research/syntaxnet/g3doc/universal.md

With this, you can create a dependency tree for questions and evaluate fitness for this application.


Is it possible to detect drift with real time predictions? I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or not. (Requiring ground truth) is this the correct assumption?
Then I stumbled on Kullback-Leibler Divergence and JS-Divergence. Can I use these methods to detect data drift in real time? (ex: request comes into my models API and then the prediction is made. I then take the features and pass it to the function calculating the drift)
Some of my concerns are do I need the full training data to compare to? As I understand these algorithms need the same size of data to compare, so would I need a data set the same size as my training data? Even understanding what the inputs used to detect data drift vs concept drift vs covariate shift would helpful.

You may or may not use ground truth to detect drift.
According to google:

What is data drift? Data drift is one of the top reasons model accuracy degrades over time. For machine learning models, data drift is the change in model input data that leads to model performance degradation. Monitoring data drift helps detect these model performance issues.

In predictive analytics and machine learning, the concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.

Covariate shift is the change in the distribution of the covariates specifically, that is, the independent variables.

So I see data drift and covariate shift very similar if not equivalent. According to these definitions:

You need the ground truth to measure concept drift.
You don't need the ground truth to measure data drift.

In order to measure data drift:

You may or may not need all the training data. If you model the predictors (let's say I fit a gaussian variable to my feature $x_i$, and it has mean $\mu_i$ and standard deviation $\sigma_i$) and save the parameters of their distribution it might be enough to summarize the distribution and you don't need the full training data.
I don't think you need to have the same sample size for the serving data at all.
Data drift needs to be done in a batch way, so you can do it with your API as long as you store the results and analyze them after a period of serving time. It doesn't make sense to say that a single observation has drift unless in very extreme cases.


What to do if one out of 2 one-hot encoding variables have a very high p-value? I ran an OLS model on a dataset with 2 categorical variables. One of them was gender. The other one had 3 different categories. I used one-hot encoding for it during pre-processing before running my model.

Variables in the image: Embarked_C and Embarked_Q.
The results showed a p-value for Embarked_Q as 0.785. In this case, should I remove both Embarked_Q and C or just Q?

You should keep all of levels as they collectively describe the feature. Removing the insignificant ones will bias your coefficients and distort your interpretation (e.i. change the reference level).

Here are some stats.exchange references:

Is it advisable to drop certain levels of a categorical variable?
Can I ignore coefficients for non-significant levels of factors in a linear model?


Is there an open source implementation for bag-of-visual words? I'm not quite sure I understand the bag-of-visual-words representation, so I may misformulate my question.

What I'm currently looking for is an open source library (possibly with python API). I give it pictures as input, and its output is a set of (sparse) features, so that I can perform my stuff base on this features. Idealy, I would like this piece of software to work without internet connection (so that I can work with it while in the plane).

EDIT: I just learnt that facebook recently (summer 2016) released some of its image recognition code (namely multipathnet, deepmask and sharpmask)

There is one implementation of BoVW in openCV. You can find the documentation here :

http://docs.opencv.org/2.4/modules/features2d/doc/object_categorization.html


Hello, when i'm training my model with 80% data and testing with 20% data the accuracy is 49% and without split it's 99% Hello, when i'm training my model with 80% data and testing with 20% data the accuracy is 49%. And when i'm training my data without splitting it's giving around 99%. I'm confused. Please help me with this
The below code is with split which got 49% accuracy
data = pd.read_csv(r"dataset.csv")

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = data.select_dtypes(include = "object").columns

for feat in objList:
    data[feat] = le.fit_transform(data[feat].astype(str))

X = data.iloc[:, data.columns != 'Outcome'].values
y = data.iloc[:, data.columns == 'Outcome'].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

from sklearn.ensemble import RandomForestClassifier
randomforest = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 0)
randomforest.fit(X_train, y_train)

y_pred = randomforest.predict(X_test)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)
print(cm)

The below code is without split which got 99% accuracy
data = pd.read_csv(r'dataset.csv')

le = LabelEncoder()

objList = data.select_dtypes(include = "object").columns

for feat in objList:
    data[feat] = le.fit_transform(data[feat].astype(str))

X = data.iloc[:, data.columns != 'Outcome'].values
y = data.iloc[:, data.columns == 'Outcome'].values

file = open(r'finalized_model.pkl', 'rb')

data = pickle.load(file)

y_pred = data.predict(X)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y, y_pred))
cm = confusion_matrix(y, y_pred)
print(cm)

Total data is the same used for 2 codes.

When you split the data , you are training your model on 80 percent of the dataset , but you are finding the accuracy of the remaining 20 percent of the data.
And when you are not splitting the data your model is learning and calculating accuracy from a similar dataset , hence the accuracy is pretty high.
You should:
1.Always split the data and try to achieve high accuracy on the test set rather than the training set.
2.If your training data accuracy is quite higher than the test data this means your modell is overfitting.


How is the equation for the relation between prediction error, bias, and variance defined? I'm reading this article Understanding the BiasVariance Tradeoff. It mentioned:

  If we denote the variable we are trying to predict as $Y$ and our
  covariates as $X$, we may assume that there is a relationship relating
  one to the other such as $Y=f(X)+\epsilon$ where the error term
  $\epsilon$ is normally distributed with a mean of zero like so
  $\epsilon\sim\mathcal{N}(0,\,\sigma_\epsilon)$.

  We may estimate a model $\hat{f}(X)$ of $f(X)$. The expected squared
  prediction error at a point $x$ is: $$Err(x)=E[(Y-\hat{f}(x))^2]$$
  This error may then be decomposed into bias and variance components:
  $$Err(x)=(E[\hat{f}(x)]-f(x))^2+E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big]+\sigma^2_e$$
  $$Err(x)=Bias^2+Variance+Irreducible\ Error$$

I'm wondering how do the last two equations deduct from the first equation?

If:
$$Err(x)=E[(Y-\hat{f}(x))^2]$$
Then, by adding and substracting $f(x)$,
$$Err(x)=E[(Y-f(x)+f(x)-\hat{f}(x))^2]
$$
$$= E[(Y-f(x))^2] + E[(\hat{f}(x)-f(x))^2] + 2E[(Y-f(x))(\hat{f}(x)-f(x))]$$
The first term is the irreducible error, by definition. The second term can be expanded like this:
$$E[(\hat{f}(x)-f(x))^2] = E[\hat{f}(x)^2]+E[f(x)^2] -2E[f(x)\hat{f}(x)] $$

$$=E[\hat{f}(x)^2]+f(x)^2-2f(x)E[\hat{f}(x)] $$

$$= E[\hat{f}(x)^2]-E[\hat{f}(x)]^2+E[\hat{f}(x)]^2+f(x)^2-2f(x)E[\hat{f}(x)] $$

$$= E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big] + (E[\hat{f}(x)]-f(x))^2 $$

$$= Bias^2+Variance$$

Then the only thing that is left is to prove that the third term is 0. This is seen using $E[Y] = f(x)$.

Edit

I am not that sure on how to prove
$$E[(Y-f(x))(\hat{f}(x)-f(x))] = 0$$
If we assume independence between $\epsilon = Y - f(x)$ and $\hat{f}(x)-f(x)$, then the proof is trivial, as we can split the expected value in two products, the first of them being $0$. However, I am not so sure about the fact that we can assume this independence.


Difference between 1x1 Convolution and TimeDistributed(Dense()) Are these lines of code equivalent in Keras? From a few runs, they seem to be, and also intuitively since the channels dimension of my data is 1, my understanding is that a fully connected acts like a convolutional layer. Is one better computationally (they have the same number of parameters)?

Conv1D(filters=1, kernel_size=1, strides=1, padding='causal')
TimeDistributed(Dense(1))

For example, I have a regression problem, where I have (batches,64) points coming in and (batches,64) points coming out of the model.

My model is:

model = Sequential()
model.add(Reshape([64, 1], input_shape=(64,)))
model.add(BatchNormalization())
model.add(Conv1D(filters=4, kernel_size=8, strides=1, padding='causal', activation = 'relu'))
model.add(BatchNormalization())
model.add(Conv1D(filters=1, kernel_size=1, strides=1, padding='causal'))
model.add(Flatten())
model.compile(loss=loss, optimizer=optimizer)

My thinking for the layers is:
reshape dimensions to 3D.
variance scale.
conv layer, activation.
variance scale.
final layer: my question is whether is matters if its a TimeDistributed(Dense) or Conv in my situation.
reshape back to a 2D waveform.

Thank you in advance!

First thing is that TimeDistributed should not be useful for your case.
It helps when having 2D data consisted of timesteps (ex. in recurrent forms) and you only have one dimension : (64, 1).

Secondly to answer, yes, your intuition is right. Conv1D with size 1 and a channel size of 1 is equivalent to a Dense of size 1.


How does Google's 'showing results for' work? If I search 'I love to eate my food' on Google then Google will 'show results for' I love to eat my food.... How does this algorithm work?

First of all, no one knows how google search works except what google officially publishes. But I give you a simple algorithm for query correction (I have implemented this previously in production).
It is a simple prediction based on spell-check. First you detect the (possible) typo. Then you find the query which maximizes the likelihood. This search is done in the huge database of user queries plus some language models.


Text classification using deep learning with removing name, place address etc I want to prepare a deep learning model for text classification(Document classification), But in my training content many place have name, address, brand name, etc.. which will do confuse to model as deep learning is work with sequence of words, I was thinking if we will train model by removing these details, but in predict content when we get name, address, brand name then it also confuse...
Can someone give the idea how i handle this situation in deep learning?
And which deep learning algorithm and library should use for best accuracy and speed for text classification.

Normally if you have enough training data this shouldn't be an issue, because the model will realize that these named entities are not relevant for classifying the text.
Anyway if you want to remove them you would need to first detect them with Named Entity Recognition, there are many implementations. Then you can simply remove them.
Using DL classification as a first option is questionable imho:

it usually requires larger training data
it takes much longer to train than traditional models
it's harder to understand what happens when things go wrong

I'm old school so I'd recommend starting with simple methods, for instance TFIDF vectors with Naive Bayes or Decision Trees.


Classification Model and Their Accuracy I am trying to work with Classification model. I am planning to train and test my model with a large dataset (Training with 80% and testing with 20% volume - no under/oversampling).
What I understand about classification models is that it focuses on a binary TRUE/FALSE or YES/NO type of prediction. However, I should also be using some evaluation metrics e.g. Accuracy/Precision/Recall etc. Ideally, there would be a threshold which I need to elevate/lower to meet my expectations.
My question - if I am to use any bespoke software solution (e.g. some Python/R library) for this, what am I expecting to see from my model to output? Is it as simple as YES/NO output or with some form of accuracy/precision number?

Binary problems have the most amount of metrics to measure its performance.
They can be classified as accuracy metrics, probabilistic metrics and metrics depending on true/false positives/negatives. You are not expected to implement them from scratch, the last time I implemented an AUC (area under the curve) was in college so you will find these metrics already implemented in most machine learning libraries in all languages. This is a comprehensive list taken from the Keras library, check the links for more information about the metrics or the library.
About the matter of the outputs you can have a true/false 1/0 values or the probabilities of the observation to belong to a class which is a better way to measure the algorithm's performance. Do you need a threshold to evaluate results properly, it all depends on the data you have. For some problems high accuracy like above 99.999% is critical like clinical diagnoses but for others accuracy like 55% can be seen as great like the case of stock predictions. So it depends on your data.
Accuracy metrics:
These metrics, simple in design, are often raw percentages of the correctly labeled classes for one class or many classes.
Raw Accuracy
Binary Accuracy
Categorical Accuracy - Straightforward accuracy for all classes.
Top K Categorical Accuracy - Accuracy for the top K categories
Sparse Top K Categorical Accuracy
True False Positive Negative metrics:
More expressive metrics useful to determine the impact the algorithm has not only in its accuracy but also in the wrongly classified observations.
AUC
Precision
Recall
True Positives
True Negatives
False Positives
False Negatives
Precision At Recall
Sensitivity At Specificity
Specificity At Sensitivity
Probabilistic metrics:
These as often used as the function loss for machine learning algorithms, however, they can also be understood as a benchmark metric specially for a validation set.
Binary Crossentropy
Categorical Crossentropy
Sparse Categorical Crossentropy
KLDivergence


Where can I find resources and papers regarding Data Science in the area of Public Health I'm quite new to Data Science, but I would like to do a project to learn more about it.
My subject will be Data Understanding in Public Health.
So I want to do some introductory research to public health.
I would like to visualize some data with the use of a tool like Tableau.

Which path would you take to develop a good understanding of Data Science? I imagine taking some online courses, eg. Udacity courses on data science, but which courses would you recommend?
Where can I get real data (secondary Dummy Data) to work with?
And are there any good resources on research papers done in Data Science area with the subject of Public Health?

Any suggestions and comments are welcome.

I don't think that you will learn much about data science (meaning, acquire understanding and skills) by using software tools like Tableau. Such tools are targeting mainly advanced users (not data scientists), for example analysts and other subject matter experts, who use graphical user interface (GUI) to analyze and (mostly) visualize data. Having said that, software tools like Tableau might be good enough to perform initial phase of data science workflow: exploratory data analysis (EDA).

In terms of data science self-education, there are several popular online courses (MOOCs) that you can choose from (most come in both free and paid versions). In addition to the one on Udacity that you've mentioned (https://www.udacity.com/course/ud359), there are two data science courses on Coursera: Introduction to Data Science by University of Washington (https://www.coursera.org/course/datasci) and a set of courses from Data Science specialization by Johns Hopkins University (https://www.coursera.org/specialization/jhudatascience/1). Note that you can take specialization's individual courses for free at your convenience. There are several other, albeit less popular, data science MOOCs.

In terms of data sources, I'm not sure what do you mean by "Dummy Data", but there is a wealth of open data sets, including many in the area of public health. You can review corresponding resources, listed on KDnuggets (http://www.kdnuggets.com/datasets/index.html) and choose ones that you're interested in. For a country-level analysis, the fastest way to obtain data is finding and visiting corresponding open data government websites. For example, for public health data in US, I would go to http://www.healthdata.gov and http://www.data.gov (the latter - for corresponding non-medical data that you might want to include in your analysis).

In regard to research papers in the area of public health, I have two comments: 1) most empirical research in that (or any other) area IMHO can be considered a data science study/project; 2) you need to perform a literature review in the area or on the topic of your interest, so you're on your own in that sense.

Finally, a note on software tools. If you're serious about data science, I would suggest to invest some time in learning either R, or Python (if you don't know them already), as those are two most popular open source tools among data scientists nowadays. Both have a variety of feature-rich development environments as well as large ecosystems of packages/libraries and users/developers all over the world.

You might also find useful some of my other related answers here on Data Science StackExchange site. For example, I recommend you to read this answer, this answer and this answer. Good luck!


Given n ordered sets, each containing 6 numbers, generate the next set in the sequence I am facing the following problem in machine learning. Given n ordered sets, each containing 6 numbers, generate the next set in the sequence. The numbers in a set are not random.
For instance, it could be that these numbers are generated by an equation, e.g. $(1+x^y)$ so that the form the following sequence:

[{1, 2, 3, 4, 5, 6}, {1, 2, 5, 10, 17, 26}, {1, 2, 9, 28, 65, 126}, ..., {1, 2, 1 + 2**n, 1 + 3**n, 1 + 4**n, 1 + 5**n}]

But this equation or any other rule governing the order is not known. We are given a large n of these sets. What would be the most appropriate way to predict this in Keras?

This should just be a sequence prediction problem, and a multivariate one. You can use LSTMs or GRUs in Keras to handle the sequences. In the simplest architecture, the inputs will be batches of (it seems) n-1 steps of 6 numbers, and the output will be the final 6 numbers in the sequence. This is pretty much how any RNN works. The loss will be mean_squared_error, I presume.

But given how much is unknown here about the distribution of the sequences, it's unclear whether this simple approach would work well, or whether even a more complex network can learn whatever patterns these follow.