why is an optimizer used in something that is not a neural network? why is an optimizer used in something that is not a neural network?
in the neural style transfer architecture the image is updated instead of the neural network weights.
in stylegan encoder (find image in latent space), the latent space is updated instead of the weights of a neural network.
Why does it work in these cases, since it is not the weights of a neural network that is updated?

In general, there is more than one kind of optimiser. The gradient-based optimisers such as gradient descent, stochastic gradient descent, Adam, Adagrad, RMSProp, form one broad category of optimiser that can find minima or maxima of scalar functions provided it is possible to calculate a gradient.
In your question however, you are effectively focusing just on gradient-based optimisers - why can they be used to alter input images or latent spaces, as well as neural network weights?
The answer is that these optimisers are not specific to neural network weights, or even to neural networks. They are appropriate whenever:

You can measure success at a task with a scalar function - typically this is a loss or cost function to minimise, but it can also be a numerical score that you wish to maximise.

The task involves parameters that you can control. This is a set of variables that can be changed in order to solve a problem. The weights of a neural network are one example, but so are the pixels in an image if your goal is to find an image that matches some criteria (such as looking like a photo of a face).

You have a way to estimate gradients of that function with respect to the parameters that you can control. Usually some form of back propagation is used to get from the loss function to the parameters you care about. That is the case in style transfer or stylegan, but it is not a requirement.

The basic requirement is being able to estimate the gradient of your scoring function with respect to parameters that can be changed. Then you can minimise or maximise that function - or at least find local minima and maxima. Neural network weights are just one common example of the approach.
Likewise, using minibatches of input/output pair examples is one common way of estimating gradients. It is commonly used when learning how to generalise from a data set, but is not the only possible way to get gradients - which approaches are available and make sense depends on the optimisation problem you need to solve.


Dropping less frequently used categorical data? I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;

>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count')
                      count      mean
category
code__1                   1  0.000000
//readabity
code_36                   1  0.000000
code_37                   2  0.000000
code_38                   2  0.500000
//readability
code_103                 10  0.000000
code_151                 73  0.000000
code_175                706  0.247875
code_188              23145  0.032059
code_189              64414  0.074006

For the total dataset we have:

>>> df['target'].agg(['mean','count'])
mean          0.072536
count    159880.000000

I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with pd.get_dummies. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:

is this wise? will I introduce data leakage or mess up future datasets?
is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list cats, iterate over my category column and substitute all categories not in the cats list to dummy, this is reproducible over a future set or test set.
Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?

The most straightforward approach would be to consider these small categories as single 'unknown' category - if they're really small, your model shouldn't change drastically.

  Another approach would be to replace all categories to dummie where
  the mean is close to the total mean of target column? then I would
  ditch code_189 (the most frequent) because its varies barely from
  0.072?

This idea seems to be close to mean encoding approach. Mean encoding consists of creating a column where categories are encoded by mean of target for this category. This can incorporate knowledge of small categories naturally - you can just replace their means with global mean, or use smoothing. I encourage you to see this segment on mean encodings (the next video is on regularization).


what happens when a decision tree can't be split into further unit values? Suppose I have a dataset
A B C D
1 1 1 0
1 1 0 0
1 1 0 1
1 0 1 1

Here A,B,C,D are my independent features and D is my dependent feature. Now if I make a decision tree here.
A -> Yes/No = 2/2 - > Based on value 1-> 2/2 So we go further for B.
B -> Yes/No = 2/2 -> Based on value 1-> 1/2 So we go further for C
When B=1 & C = 1 then D = 0
When B=1 & C = 0 then D = 0 and as well as D = 1 too.

So here we have a 50/50 chance, Or we can see we can't go further to get unique value to predict. So how does decision tree solve this problem?

Well, sometimes the features simply do not provide enough information to get 100% accuracy (like in your case), even with a model a flexible as the Decision Tree.
The Decision Tree works by trying to split the data using condition statements (e.g. A < 1), but how does it choose which condition statement is best? Well, it does this by measuring the "purity" of the split (conditional statements split the data in two, so we call it a "split"). The purity is simply how well the statement split up the target variable (so, a good split would be [2, 0], which basically translates to "If I did this split, I would get two $D = 0$ instances, and zero $D = 1$ instances.", and a bad split would be [1, 1], saying "With this split I would get one $D = 0$ instance and one $D = 1$ instance"). If the model finds that no further splits can reduce the purity, it stops.
If you want to look into it further, there are a couple of measures for measuring purity (or rather, impurity), the main ones being gini and entropy.


PCA on acceleration time series data Has anyone attempted Principal component analysis on time series and acceleration data(ie. data from accelerometer and sensors) and tried compressing it as well as regenerating back the data with minimum error?

I wanted to know if this is possible using PCA. Kindly suggest alternate methods available to compress and regenrate my data.

PCA doesn't really make sense for a time series if it is left as a time series:

There are times when its useful to aggregate time series data into a rectangular representation in order to gain insight into interesting aspects of the data.  This is a feature engineering where the time series provides lots of the features.  For instance, you can create features for 'minute of hour', 'hour of day', 'morning', 'afternoon','evening', 'day of week', 'day of month', 'week of month', 'week of year', 'season', ...  You can even join this data with other data, like historical weather data or economic index, to see how daily temperature or economics affect your phenomenon. In these cases you could do all of this feature engineeting and then use PCA to find the most important features (radar plot) or simply to select the most important orthogonal features from the post PCA mashup (linear transformation).

Time Series Smothing:

There are a bunch of different methods to smooth and down-sample time series data. In this case, you are turning the quantized stochastic events and either turning them into regularly spaced data, or taking your already regularly spaced data and downsampling and smoothing it. Smoothing algorithms include Exponentially weighted moving average (EWMA) or Holt-Winters smoothing.  Connor Johnson has a nice blog write-up on the these.

ARIMA is a more complete algorithm that includes smoothing, de-seasoning, and forecasting. It is very powerful, but takes some time to master.

Hope this helps!


Prerequisites for openCV I am looking to start learning openCV to perform a car detection program. Should I know data science and/or machine learning stuff before going for openCV?

If not, where can I start learning openCV with Python?

OpenCV has a module to detect faces, eyes etc. With pretrained models you may also be able to detect cars.
https://www.pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/

There are many good tutorials online. Just have a look.

So if you are happy with this, you don‘t need to know much about data science. However, if you want to go a step further and train your own models, you will need to look into e.g. Keras (with pretrained models).
https://keras.rstudio.com/articles/applications.html

My view is that OpenCV is a cool tool and you can do a lot with it. However, off-the-shelf applications have their limitations (of course).


Improve line plot visualisation I want to improve the graphic

that I created with R's ggplot2. The graphic is for a scientific publication. These are the runtime of 12 algorithms. There are four at the top, four in the middle and four at the bottom. They are not necessarily related to each other. It is very hard to discern the different colours from each other.

The main point that I want to drive home is that there are four that are slow, four that are middle fast, and four that are fast. Indeed, the groups should be more or less equally fast, and the fact that they differ is mostly due to the fact, that we talk about runtime (which always differs).

How can I improve the plot such that I drive home the main point without being distracted by the many colour, but possibly still have differentiation within the three groups?

Maybe it is not smart to keep the differentiation in the first place?

I would like to keep the line plot, but I am open to alternative suggestions if there are really good ideas out there.

You already made it log scale which would have been my first idea. How about use lattice to make it three plots with different Y start and end? Each plot should have a center point that covers one of the three groupings you have and you can stack them 1x3 (vertically).


Long range forecasting with sequence-to-sequence models I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily observations going back 40 years at those sites. So I have a lot of data to train with. I am working with climate and geologic data, so the time series is strongly cyclical or periodic with some random variation.
I was trying to use a Sequence-to-Sequence LSTM network with attention to start with. I have 365 days input and trying to predict 365 days output. I am just starting to collect the results so far. But I was wondering if there are any specific models designed for long-range time-series forecasting? I thought sequence to sequence models still only worked well for say 20-30 day output, but I was not sure if there has been any developments on that front.
Thanks.

I am wondering why you chose a Neural Network approach in the first place. In general a recurrent Neural Network requires many more parameters to be fitted than for instance a SARIMA model. Therefore, they are more sensitive to spurious artefacts in the train sample. (This is the price you pay for being more flexible in the way the relationships between past and future values are modelled). Given the fact that you mentioned the strong periodicity in your data I think modelling the data using a SARIMA model or even a model with seasonal dummies may generalise much better out of sample, given the sample size that you have. In addition you could add a cyclical trend component to also incorporate other cyclical components in the data. Hope this helps. Both the SARIMA model and the addition of a cyclical component are often described in the econometric literature. Comparing your current model with a SARIMA with: {p=0; d=0; q=0; P=1; D=1; Q=1, s=365 and trend='c'} may be a good place to start (These arguments refer to the statsmodels SARIMA that can be used with Python "statsmodels.tsa.statespace.sarimax.SARIMAX").


What am I supposed to see on tensorboard images tab? I'm training an object detection model with Tensorflow and monitor the training task with tensorboard. I was expecting in the Images tab of tensorboard that displayed images would show a bounding box (at a specific point of training). What I see though is only images with an orange line drawn above the picture (the same orange that I expect for the bounding box). Am I missing something? Am I right when I say that a bounding box should appear or not? Picture of what I see is joined. Any help greatly appreciated.

This is what I see using data from Pascal VOC that was converted using this example script:
I am using Tensorflow 2. I think it's just the images preprocessed using the data_augmentation_options that were specified in your pipeline.config file.
I seem to recall seeing bounding boxes on the images in Tensorboard back when I was using the Object Detection API with Tensorflow 1. But I haven't seen any bounding boxes on the images using the API with Tensorflow 2. I have verified that the models subsequently worked. I just didn't see bounding boxes in Tensorboard. Note that I am just running training and then doing evaluation separately. If you specified eval_config or eval_input_reader in your pipeline.config, you might get different results.


Tips for checking data integrity / data sanity? I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets.

I am interested in knowing the steps to check such factors on datasets.

Kindly let me know if this is too broad, I can think of more specific questions.

Here are a few things to check in a data set, there could be a lot more but i'm sharing what i did myself.

Check the NULLs in your columns, count them to see if there is a column that contains too many NULLs and that you might want to eliminate
Plot your data distribution each column at a time to see the range of values taken by your data, to see if there is something particularly remarquable about its behavior.
And I'd recommend this famous kernel on Kaggle where a good exploratory data analysis is done.


Why use regularization? In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with infinite observations instead? What is the intuition?

Regularization is used to help smooth multi-dimensional models.
Take this example,
y = x_1 + eps*(x_2 + ... + x_100)

Let's say eps is very small. It  doesnt seem very useful to store those 99 coefficients, isn't it? How do we manage to fit a model in such a way that we drop negligible coefficients? This is exactly what L1-regularisation does!
Each other type of regularisation has another geometric intuition.


Cross-fold validation done on whole dataset or training set? I have a dataset of 77 samples with 302 features with two labels (0,1). I trained an SVM with gridsearch (cv=5) to perform binary classification.

In one run of my script, I do a test-train split, and then pass the training data into gridsearch. I am getting wide range of training accuracy (>70-90%) and test accuracy (40-75%). My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?

Rather than a train-test split, I would recommend n-fold cross-validation or, better yet, leave-one-out cross-validation.

  My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

The high variance in your error estimates is a consequence of a very small dataset.  When sampling from such a small dataset, it's super easy to wind up with train and test splits that aren't representative.  Cross-validation will give you a more stable error estimate.

  What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?

It's a very good idea to run your model evaluation routine several times and average the results.  It sounds to me like you're manually running a less-controlled version of cross-validation.

I'm not sure what you mean when you say "plot the error bars on the ROC curve".  Is area-under-ROC your accuracy metric?  Or are you computing the accuracy then somehow plotting that on the same axes as an ROC curve?

This wasn't part of your question, but it may be helpful nonetheless.  When you have a dataset with more features than examples, then you are incredibly likely to run into problems related to the curse of dimensionality.  In the 302-dimensional space where your features reside, your dataset of just 77 examples is incredibly sparse.

You may want to consider simpler models (logistic regression, shallow decision trees, etc) and employ some type of regularization to discourage your model from learning spurious relationships.  You could also try to use a dimensionality reduction technique to eliminate some of your 302 features.


How to architect ConvNet to ignore top half of image I'm building a convoluted neural network to teach a toy car, powered by a Raspberry Pi, how to drive based on incoming streams of frames from a webcam mounted on top of the car. The top half of each image is irrelevant. What matters is the curvature of the road, and this is in the bottom half.

I've generated a substantial amount of data (about 40k records) by driving the car around myself and recording what I do (commands are left, right, and straight) and what the frames are. However my trained ConvNets aren't giving me the performance I'd hoped for. My experimentation with one of the deployed models confirms that it is indeed tricked by changes in the top half of the streaming images.

A simple solution is to programmatically cut the frames in half so that the neural net only receives relevant portions. However, deep neural nets are praised for their ability to learn features with (ideally) zero human future transformations, so I want to avoid this approach. This project is meant to be a learning experience for me so I want learn how to architect the ConvNet more effectively. Each training session takes several hours to run, so rather than try everything under the sun, I'd figure I'd reach out to the community here to narrow my focus of exploration.

One thought I have is to put a fully connected (FC) layer in front so that subsequent layers effectively only convolve over the relevant portion of the image. I think this could potentially work if this up front FC layer learned to assign very small weights to pixels in the top half of the image. Could this work? Are there better architectures?

You could be right that ignoring top part of image would benefit the CNN. However, there is very little point in trying to architect this - if your premise that the CNN will ignore irrelevant details in the top half is correct, then that will occur anyway and there is no standard NN architecture that will help that other than disconnecting the top half of network, which is going to be logically exactly the same as programmatically slicing the image, with the disadvantage of storing and calculating with twice as many parameters.

You should either programatically cut the image in half or do nothing to the image and rely on the CNN's inherent ability to give low weights to irrelevant details. If you do the latter, you may be able to get around the learning of incorrect details in the top half by augmenting your data - e.g. add some noise to images*, especially in the irrelevant top half. Perhaps horizontally flipping a few images (and reverse relevant targets for the control class) might be another useful augmentation. Some augmentations could also be useful if you just take the lower half of the image.

* Noise should be something close to variations that could be seen when in use. E.g. slurring pixels left or right might be reasonable. Inserting "static" probably is not.


Data type detection in Tableau When you connect a data source, Tableau automatically infers the type of each column of your data.  Whether it's Number (decimal), Number (whole), String, Boolean, etc.  A few questions about this:

1) Has Tableau ever misclassified one (or more) columns of your data with their automatic labeling?  If so, would you please give details?

2) Do you feel that these data type options can be improved?  For example, Tableau doesn't seem to be distinguishing between unordered categorical data and ordinal data.

Thanks!

Yes, it sometimes does that. I think it depends on the data source. For RDBMS it is likely that it takes the data type from the database (so the column type is typically accurate). For text/csv, it probably does a scan of the top certain lines to guestimate the type. It is similar to the text import wizard of the SQL, etc. other similar tool.  (A Tableau user of 4+ years)
I am not sure if it is relevant to many of the Tableau users; many databases don't have that distinction either; I think it is more helpful for someone working in machine learning, etc.?


How does the random forest vote work? I have a question. How is the voting done in random forests. I can't understand rationally, since we have a bootstrap sample drawn, and have built dection trees based on them, where is the new data point taken from to do the voting and extract the results?

Always separate training a model from using the model to make predictions.  During training, bootstrap samples are drawn and trees built using those resamples.  Voting only occurs when predicting, and all of the trees are used, whether your datapoint is from the original training set or not.
There is a bit of a caveat here though, because we may be interested in the "out-of-bag" scores of the random forest, in which case we want to let the trees that didn't see a given training datapoint vote on the outcome for that datapoint.  In that case, we need to keep track of which bootstrapped samples contain which datapoints, and then predict for each using the appropriate subset of trees.


Activity recognition with binary sensors I have a bunch of streams coming from a set of 28 binary sensors around a SmartHome like this:

Where:

OBJECT: indicates the name of a binary sensor
STATE: is obviously the state of the sensor at that precise instant (you can think of it like Movement/Pressure/Open=1 and NoMovement/NoPressure/Close=0)
ACTIVITY: is the Label I would like to predict and indicates the activity the Human was doing
TIMESTAMP/HOUR: you know what they mean...

The activities to be recognised are 24 and are a bit complex. For example: making breakfast, making lunch, making dinner, having breakfast, having lunch, having dinner, going to the toilet, teeth brushing, dressing, watch TV, and so on...

Can you suggest a technique to use this dataset for AR?

First you need to find a way to formulate this as a classification problem, where you have pairs of features (from your sensor data) and labels (the activity).

Bag of features

A simple way is to cut up the observations into a series of time-windows, where inside each window you ignore the temporal parts of your sensor data (a "bag of features" representation). Keep track of the current (or most common) state of each sensor, and count the number of sensor transitions that happened (ex: C08open->close: 2 times).

To select the appropriate length of time-window you should do some Exploratory Data Analysis to find typical activity durations.

Both during training and prediction, the time-windows should be computed with overlap.
Say that each window is 5 minutes long. Then the next window contains the 4 last minutes, plus 1 new minute of data. Overlap is usually in a range of 50-90%.

Note that some activities are quite time-dependent. Like make dinner or make lunch. So you should include also a feature time_of_day which is seconds/minutes since midnight.

Temporal models

Some transitions between activities are probably much more common that others, and this is a source of information that can be used. Example: making dinner -> having dinner probably way more likely than making breakfast -> making dinner.
A Hidden Markov Model (HMM) would be the standard way to model that. It can give you the most probable sequence of activities for a whole day.

There are some ways to use CNNs and RNNs on these kinds of datasets, but unless the dataset is huge I would probably not try that.


How to predict the value in KNN? I am trying to build the KNN algorithm for IRIS dataset. First, I've computed the distance and stored it in 1d array. However, I am really struggling to build the prediction function. Therefore two questions arise:

What does this code do? classes = y[np.argsort(dist)][:k]
How should I change the last loop of the function given certain requirements?

def KNN(k, X, y, x):
    from scipy.spatial.distance import cdist
    """K nearest neighbors
    k: number of nearest neighbors
    X: training input locations
    y: training labels
    x: test input
    """
    N, D = X.shape
    num_classes = len(np.unique(y))
    dist = np.zeros(X.shape[0])   # <-- EDIT THIS to compute the pairwise distance matrix
    for i in range(len(dist)):
        dist[i] = np.linalg.norm(X[i]-x)
    print(dist)

    # Next we make the predictions
    ypred = np.zeros(num_classes)
    classes = y[np.argsort(dist)][:k] # find the labels of the k nearest neighbors

    for c in np.unique(classes):
        ypred[c] = y[c]  # <-- EDIT THIS to compute the correct prediction
        print(ypred)
    return np.argmax(ypred)
```

Here is one example for you to consider.
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"

# Assign colum names to the dataset
names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'Class']

# Read dataset to pandas dataframe
dataset = pd.read_csv(url, names=names)

dataset.head()

X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, 4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 1.4, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

The very last line of code will print this:  ['Iris-setosa']
Let's look at the very first row in the test set:  testSet = [[5.1, 3.5, 1.4, .2]]
Look at: dataset.head()
Do you see how y_pred is the class, given [5.1, 3.5, 1.4, .2]?

Hope that makes sense!  Now, let's change the testSet just a bit, and check the outcome.
classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 4.9, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

Result: ['Iris-versicolor']
The variable ('petal-length') changed from 1.4 to 4.9, and now we have a different 'Class'!!  Does it make sense?  If you look at the raw data, you will see that the testSet doesn't even exist, but nevertheless, the model can make a prediction about what the Class should be!
K-Nearest Neighbors Algorithm in Python and Scikit-Learn


Should I scale or normalise my dataset before clustering? So i have a dataset with variables with unit of measurement as milligrams, kgs and quintals. Should i use standard scaler or minmaxscaler to scale the dataset.

As often in Machine Learning, there is no clear answer. In fact, both are valid options [1, p. 116]. However, for k-means min-max-scaling is usually used in practice [2]. So min-max-scaling would be the default choice and it's what I'd recommend. But as so often you can simply try both and see which provides better results (i.e. better internal cluster validation measures, such as the Silhouette Index).
References:
[1] https://dbs.ifi.uni-heidelberg.de/files/Team/eschubert/lectures/KDDClusterAnalysis17-screen.pdf
[2] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173


How to run only part of the code in PyCharm, having some data in operating memory In Jupyter Notebook I have blocks of Python code and I can run them arbitrarily. What is super cool about that is I can spend 10 minutes training some ML model and then play around with it, without the need to run the training process again and again.
How can I achieve something similar using PyCharm? While I like this feature in Jupyter it is so less convenient to write code.

Pycharm allows you to highlight a section of run and execute that specific section in the console by right-clicking and choosing 'Execute selection in Python Console'. This section of code has now been executed and any variables you've defined are now stored in memory.
You can continue executing pieces of code in this way and reference objects previously created up until you refresh the console session.


Fraud risk propagation in large scale network What's the best approach to do some graph analytics and risk propagation in a network using python where multiple accounts are connected through a relationship and few of the accounts in the network are marked as bad accounts and the rest are unknown?

I tried using networkx but it seems to run forever. I have about 8MM edges and 40K nodes

You could try applying a graph convolutional network to do some semi-supervised learning. See Kipf and Welling's paper "Semi-Supervised Classification with Graph Convolutional Networks". It probably depends on how unbalanced your dataset is though. If the dataset is too large, you could train a sample of it, and train the GCN on that subset. I'd try to find some exemplar data points and create a train set from that.


How to interpret two continous variables output using GAM? I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (count variable). I have tried a lot but I am not able to understand how to interpret the graph and output that I am getting.
I tried this formula using mgcv package:
model1<- gam(cvd ~ s(templg0), family=poisson)
summary(model1)
plot(model1)

So here is the output for summary that I am getting:
Family: poisson
Link function: log

Formula:
cvd ~ s(templg0)

Parametric coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept) 3.195669   0.004877   655.2   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Approximate significance of smooth terms:
             edf Ref.df Chi.sq  p-value
s(templg0) 3.422  4.295  57.23 2.93e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

R-sq.(adj) =  0.0152   Deviance explained = 1.68%
UBRE =  1.016  Scale est. = 1         n = 1722

Can someone please explain the output in detail. What this output is explaining? and also can someone help what this plot (picture attached) is showing? Please be kind as I have invested a lot of time but can not find how to interpret this.

What you do here is to use smoothing splines regression. Have a look at the book "Introduction to Statistical Learning" Chapter 7.5 for a very good overview on the method.

The s-function in GAM allows you to specify how the GAM is fitted. You have not supplied k, so some default is chosen:

  the dimension of the basis used to represent the smooth term. The
  default depends on the number of variables that the smooth is a
  function of. k should not be less than the dimension of the null space
  of the penalty for the term (see null.space.dimension), but will be
  reset if it is. See choose.k for further information.

GAM basically does several linear regressions (specified by k) along the x-axis if you want to say so. Thus, GAM allows to model wild non-linearity. If you want to check linearity in your data, you should check different values of k and look at the plot.

GAM Example:

library(gam)
library(ISLR)
df = ISLR::Auto

# GAM with 10 knots
gam5= gam(horsepower ~ s(mpg, 10), data=df)
summary(gam5)
plot(gam5, se=T)

GAM Result:

Would you conclude that your model is linear? No. In the range between about ]mpg=0 to mpg=20[ there is a linear relation as well as betweeen ]mpg=20 to mpg=40[. But linearity does not hold for the entire range of data. So I would differentiate these segments, e.g. by dummy encoding and interaction terms.

Note that the y-axis is rescaled. So there is no natural interpretation of the y-axis here.

Comparison to non-parametric (NP) estimation:

To deal with non-linearity non-parametric regression is an obvious alternative. What happens if we do NP?

# Nonparametric regression
library(SemiPar)
fit <- spm(df$horsepower ~ f(df$mpg))
plot(fit)

NP Result:

As you can see NP delivers almost the same result. However, the y-axis in the figure has a natural interpretation, which can be useful.

What about your Problem:

First make sure that you check different values for k in s(...,k), so check different number of knots and see how the figure changes. Also have a look at the book to understand the background.

In your current figure, I see some kinks at about x=10 and x=20. However, I would not say that this is severe non-linearity (but there is non-linearity in the data). Generally, if you can draw a line over the plot range (along the x-axis), and if this line is not outside your confidence bands, you can claim for a linear relationship.


Ideas for model to predict wind speed in a local kitesurfing spot We have a sensor in a local kite spot that captures real time wind speed, we've found that we can somewhat predict if there's going to be "good" wind by looking at different forecast models and weather conditions.
I think it should be easy to train a model that takes input from different forecast models as well as real time data to predict with higher accuracy the speed in the spot.
Do you think this would be too crazy? if not how would you suggest I approach building this, having no experience with machine learning at all

Predicting weather is very complex, so it's unlikely you'll improve upon any predictions from services like WindFinder.
I would first try to prove that predictions from a service like WindFinder are inaccurate, i.e. collect historical predictions and compare it your recordings of sensor data at your local position.
After this, encode your heuristic model which allows you to predict the "good wind" based on other sources into code, but with current information I'm not sure where the ML fits in, rather it seems to just be aggregating sensor & weather site information into a single metric.


What are the pros and cons of Keras and TFLearn? What are the pros and cons of Keras and TFlearn? When is one library preferred over the other?

TFlearn is a modular and transparent deep learning library built on top of Tensorflow. It was designed to provide a higher-level API to TensorFlow in order to facilitate and speed-up experimentations, while remaining fully transparent and compatible with it. Even with TensorFlow, however, we face a choice of which “front-end” framework to use. Should we use straight TensorFlow, or TF Learn, or Keras, or the new TF-Slim library that Google released within TensorFlow.

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

Straight TensorFlow is really verbose while Keras and TfLearn both seem solid, but the TfLearn syntax seems a little cleaner. One drawback to Tflearn is the lack of easily integrated pre-trained models.

Actually there are so many answers for your question here and here and I quote some of them here.

TensorFlow is currently the mainstream of deep learning framework, they are all the wrapper of TF. Whereas, Keras was released at the age of Theano, and therefore having a good support from Theano’s users. While TensorLayer and TFLearn are both released after TensorFlow. A good reason to choose Keras is that you could use TensorFlow backend without actually learning it. Plus Keras tends to wrap up the model deeply, so you don't necessarily need to consider the backend to be Theano or TF, which is a big advantage of Keras.

It depend on what do you want to do, fast prototyping or something else?

Keras : Many people are using it, easy to find examples on github. Suitable for beginner. Capable of running on top of either TensorFlow or Theano. Tflearn : Why no one discuss it? It is also a famous library, transparent over TensorFlow. High running speed. TensorLayer: Just release (Sep 2016), transparent over TensorFlow. High running speed. Easy to extend, suitable for professional, its tutorial include all modularized implementation of Google TensorFlow Deep Learning tutorial. TF-Silm: Just release (Aug 2016) similar with Tflearn, but no RNN layer at the moment (Sep 2016).

The best deep learning framework is the one you know best.


Combining Classifiers with different Precision and Recall values Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is high and the recall is low, whereas the precision for B is low and the recall is high. Is there a way to combine these two models where I can get the precision from one and recall from the other, or possibly use the metrics from one to improve those of the other?

For example, let's say these are the metrics:

A

Precision: 0.91

Recall: 0.35

B

Precision: 0.44

Recall: 0.90

As these are binary classifiers, my labels are 1 and 0, and my class of interesting is 1 (so the metrics above are for predicting 1s).

Let's say model A predicts 10 1s, and model B predicts 70 1s.

It is safe for me to say that of the 10 1s that model A has predicted, 9 are true positives.

It is also safe for me to say that of the 70 1s that model B has predicted, 60% are false positives, but the rest are 90% of the true positives in the dataset.

My question is, is there some method for me to combine these results so that I can obtain all 50 true positives from the dataset?

You want to ensemble your two algorithms. The way to do that is to not just use e.g. sklearn's precision and recall metric functions, but to actually obtain probabilities for being positive (most models can output this with a simple function call), and take the average of those probabilities between your two (or more) binary classifiers. Then, test a range of probability thresholds beyond which you might label something positive, and evaluate the impact on recall and precision. This is a pretty common approach to obtaining better results through utilizing the strengths of different prediction algorithms.

Just some pythonic pseudo-code to demonstrate:

recall = []
precision = []
for threshold in [0.1, 0.2, 0.3, ..., 0.9]:
    df_of_mean_probabilities['label'] = df_of_mean_probabilities > threshold
    recall.append(TP/(TP + FN))
    precision.append(TP/(TP + FP))


How to find the most important attribute for each class I have a dataset with 28 attributes and 7 class values. I want to know if its possible to find out the most important attribute(s) for deciding the class value, for each class.

For example an answer could be: Attribute 2 is most important for class 1, Attribute 6 is most important for class 2 etc. Or an even more informed answer could be: Attribute 2 being below 0.5 is most important for class 1, Attribute 6 being above 0.75 is most important for class 2 etc

My initial approach to this was to build a decision tree on the data and find the node that had the largest information gain/gain ratio for each class and that would be the most determining factor for that class. The problem with this is that the decision tree implementations I have found don't give the information gain/gain ratio for each node and as this is time bound I don't have the time to implement my own version. My current thought is to create multiple datasets which are all one class vs the rest and then perform attribute selection (eg. information gain) on them  to find the most important attribute. Is this the right direction to go down or is their a better option?

If you must split the dataset for each class then I suggest you try PCA. Principle Component Analysis is basically used for dimensionality reduction in the sense that it gives you the subset of attributes which represent the distribution of data best. You could use that for all classes and in that way get the attributes which affect the class data distribution the best.


Association rule mining and Orange3 package Python how i use Orange3 package for association rule mining ,in Orange 2.7 version Orange.associate.AssociationRulesSparseInducer() method is present but its not available in Orange3 ..

In Orange 3, AssociationRulesSparseInducer is removed.

The documentation of Orange-Associate may help you on your task.

References

https://github.com/biolab/orange3-associate/issues/4

https://stackoverflow.com/questions/39667272/orange-data-mining-version-3-3-python-association-rules


LSTM with input of actual time step I'm working on an implementation of LSTM neural network to forecast energy consumption. I have a dataset with load, series of weather parameters and indicator of it's bank holiday or not.

I first did a network with input of 24 lag (using function from this tutorial).
So I have a dataset like this, but with 18 variables and from ($t_{-24}$)

   var1(t-1)  var2(t-1)  var1(t)  var2(t)
1        0.0       50.0        1       51
2        1.0       51.0        2       52
3        2.0       52.0        3       53
4        3.0       53.0        4       54
5        4.0       54.0        5       55
6        5.0       55.0        6       56
7        6.0       56.0        7       57
8        7.0       57.0        8       58
9        8.0       58.0        9       59

And I have my LSTM network:

model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='Adam')

And currently I use var1(t-1) and var2(t-1) to predict var2(t) value.
But var2 is correlated with value of var1. I have value var1(t) and I want to use it to predict var2(t), but I don't know how to do that.

I tried to fit my neural network with var1(t-1), var2(t-1) and var1(t) but it don't match with then input matrix which is (value, lagged hours, number of variables)

I'm not sure whether I understand the question, but if you want to include var1(t) along with the other lagged features var1(t-1), var2(t-2), etc. you can simply add var1(t) as an additional feature.
This means that your matrix with previous shape

(number of samples, number of timesteps, old number of features)

now becomes

(number of samples, number of timesteps, new number of features = old number of features + 1)

Judging from your code this can be simply done by adding var1(t) as a new column to your train_X matrix, like in the first three columns of your first code snippet.

However I'm not sure why you want to do this. Forecasting at time t usually assumes that, once your model is ready to work on real data, you won't have any data at t, only from the past. If you build your model based on data at time t without being able to retrieve these information once in production, then you've got a problem! Also, you write that var1 and var2 are correlated, hence the LSTM is already picking up this correlation as you are feeding past values of these variables together.


What is a good explanation of Non Negative Matrix Factorization? I am trying to find a resource to understand non-negative matrix factorization. Apart from Wikipedia, I couldn't find anything useful.

Non-Negative Matrix Factorization (NMF) is described well in the paper by Lee and Seung, 1999.

Simply Put

NMF takes as an input a term-document matrix and generates a set of topics that represent weighted sets of co-occurring terms. The discovered topics form a basis that provides an efficient representation of the original documents.

About NMF

NMF is used for feature extraction and is generally seen to be useful when there are many attributes, particularly when the attributes are ambiguous or are not strong predictors. By combining attributes NMF can display patterns, topics, or themes which have importance.

In practice, one encounters NMF typically where text is involved. Consider an example, where the same word (love) in a document could different meanings:

I love lettuce wraps.
I love the way I feel when I'm on vacation in Mexico.
I love my dog, Euclid.
I love being a Data Scientist.

In all 4 cases, the word 'love' is used, but it has a different meaning to the reader. By combining attributes, NMF introduces context which creates additional predictive power.

$"love" + "lettuce \ wraps" \ \Rightarrow \ "pleasure \ by \ food"$
$"love" + "vacation \ in \ Mexico" \ \Rightarrow \ "pleasure \ by \ relaxation"$
$"love" + "dog" \ \Rightarrow \ "pleasure \ by \ companionship"$
$"love" + "Data \ Scientist" \ \Rightarrow  \ "pleasure \ by  \ occupation"$

How Does It Happen

NMF breaks down the multivariate data by creating a user-defined number of features. Each one of these features is a combination of the original attribute set. It is also key to remember these coefficients of these linear combinations are non-negative.

Another way to think about it is that NMF breaks your original data features (let's call it V) into the product of two lower ranked matrices (let's call it W and H). NMF uses an iterative approach to modify the initial values of W and H so that the product approaches V. When the approximation error converges or the user-defined number of iterations is reached, NMF terminates.

NMF data preparation

Numeric attributes are normalized.
Missing numerical values are replaced with the mean.
Missing categorical values are replaced with the mode.

It is important to note that outliers can impact NMF significantly. In practice, most Data Scientist use a clipping transformation before binning or normalizing. In addition, NMF in many cases will benefit from normalization.

As in many other algorithmic cases, to improve matrix factorization, one needs to decrease the error tolerance (which will increase compute time).


Convolution operator yields negative index of matrix When I read about convolutional neural network from the internet, like this one, mostly I found that discrete convolution operator is defined as follow:
$$C=I*F$$
$$C(x,y)={\sum_{a=0}^{k-1} }{\sum_{b=0}^{k-1}}I(x-a,y-b)F(a,b)$$

Where the size of $F$ is $k\times k$, Suppose that the size of F is $3\times 3$ and size of $I$ is $9\times 9$, then $C(1,1)=\dots +F(2,2)I(1-2,1-2)+\dots$ that is does not make sense when $I$ has negative index, how do I compute matrix $C$ ? does we change the way the matrix indexed?

There are two approaches that can be taken:

Only use valid indices. Matrix C will then be smaller than matrix I, in your example it would be a 7x7 matrix (9 - 3 + 1 = 7). You may see this in neural network libraries as a convolution working with "valid" border.
Use synthetic values, usually just 0, for out-of-bounds indices in matrix I in order to calculate C. This produces the same size output as input, so some neural network libraries note this with "same" border.

Your formula suggests "same" border is being used*, because it would be unusual to work with a matrix C starting from an offset corner. If you do use "same" borders, it is important to normalise your input image data pixels to mean 0, otherwise the synthetic border will appear as a strong edge to the kernels.

If you are developing your own CNN code, I would suggest using "valid" border mode, because it is simpler, and a more common approach. It will work just fine for image classification tasks.

* Actually I think the formula you have posted is over-simplified or even wrong. I would expect to see the term $I(x+a,y+b)$ so that the top left index of C stays as (1,1) for a "valid" border. For a "same" border, I would expect to see a centered kernel.


Nesterov Accelerated gradient is in correct order? I've implemented Nesterov Accelerated Gradient (NAG) (link, section: "Nesterov Accelerated Gradient"), however not sure it's viable.

$$v_t = \gamma v_{t-1} +\eta\nabla_\theta J(\theta - \gamma v_{t-1})$$
$$\theta = \theta - v_t$$

I attempted the following (consider a minibatch consisting of merely 1 training case):

run a forward pass, and get the error
Apply the scaled previous momentum, (momentum_tMinOne * coeff) to the weights
run a backward pass using the error of each pass.
set momentum_tMinOne = currentGradient, but don't apply it to the weights yet
repeat forward pass, from 1.

Do I require to remember something like momentum_tMinTwo,  or just keeping hold of the previous momentum will suffice?

Am I correct that this way we don't need to keep two matrices for momentums, but only need momentum_tMinOne matrix?

The standard momentum would have these steps:

straight away, recompute the new momentum:
$$\mu_{t+1} = \mu_{t} \cdot (decayScalar) + (learnRate)\cdot \nabla$$

adjust the weights by this new momentum
$$\theta_{t+1} := \theta_{t} - \mu_{t+1}$$

Nesterov momentum has this:

Make a big jump: correct the weights by any $\mu$ we have so far in our posession:
$$\theta_{t+1} := \theta_{t} - \mu\cdot (decayScalar)$$

Compute the gradient $\nabla$ from the new weights $\theta_{t+1}$

Correct these weights by this gradient (right now without any momentum):
$$\theta_{t+2} := \theta_{t+1} -  (learnRate)\cdot \nabla$$

At the very end, re-compute the momentum as follows:
$$\mu := \theta_{t+2} - \theta_{t}$$

So, the momentum is updated at the very end. It becomes a vector from "weights before the big jump", pointing towards the "weights after the correction by the fresh gradient".
reference: Geoffrey Hinton Lecture 6C Corsera

Re-arranging:
To avoid sticking the gradient computation in the middle of our optimizer function (steps 2 and 3), we can instead re-arrange things as follows:

compute gradient for the weights we have thus far.

correct such  weights by the gradient (right now without any momentum), as follows:
$$\theta_{t+1}:=\theta_t - (learnRate)\cdot \nabla$$

update the momentum:
$$\mu := \theta_{t+1} - \theta_{cached}$$
$$\theta_{cached} := \theta_{t+1}$$

big jump
$$\theta_{t+2} := \theta_{t+1} - \mu\cdot (decayScalar)$$

Notice, this way steps 2,3,4 are all inside of our optimizer. We can compute the gradient, outside of our optimizer (during step 1) making our code much more readable :)
    size_t _numApplyCalled = 0;

    //Nesterov Accelerated Gradient.
    //Placed at the end of a backprop, should be followed by a usual forward-propagation
    // https://datascience.stackexchange.com/a/26395/43077
    //
    void apply( float *toChange,  float *newGrad,  size_t count ){

        float learnRate = get(OptimizerVar::LEARN_RATE);//scalar
        float momentumCoeff = get(OptimizerVar::MOMENTUM_1);//scalar

        const bool isFirstEver_apply = _numApplyCalled == 0;

        for (int i=0; i<_arraySize; ++i){
                //correction by gradient alone:
                toChange[i]  -= newGrad[i]*learnRate;

                // determine momentum:
                if (isFirstEver_apply){//nothing was cached yet.
                    _momentumVals[i] = 0.0f;
                }
                else {
                    _momentumVals[i] = toChange[i] - _weightsCached[i];
                }
                //caching, AFTER momentum calc, but BEFORE the jump:
                _weightsCached[i] = toChange[i];

                //jump:
                toChange[i] -= _momentumVals[i] * momentumCoeff;
        }//end for

        ++_numApplyCalled;//increments by 1
    }


What is major difference between different dimensionality reduction algorithms? I find many algorithms are used for dimensionality reduction. The more commonly used ones (e.g. on this page ) are:
Principal component analysis (PCA).
Factor Analysis
Independent component analysis (ICA).
Non-negative matrix factorization (NMF or NNMF)
Latent Dirichlet Allocation (LDA)

What are major differences between these? How does one choose one over the other? Thanks for your insight.

A detailed answer would require many pages of explanation, but I think a brief answer may point to the right direction for further research.
First of all the choice of dimensionality reduction algorithm depends on the problem and data at hand. There is no golden standard. Your problem requirements dictate the best option(s) to try out.
The main concept of dimensionality reduction is to find alternative representations for the data that have less "dimensions" while at the same time keeping most of the original information contained in the data.
Right from the start, one sees that there is some trade-off between space/dimensions used and information kept. The less space/dimensions kept means also that the less original information contained in the data is kept.
The trick is to try to leave out redundant and useless information for the problem at hand. This is why the choice of algorithm depends crucialy on the problem and data at hand. Then one can indeed reduce space/dimensions of the data while at the same time keeping all relevant information.
In order to do that and depending on the nature and characteristics of data one can try some approaches:
1. PCA and variants
This factors the data to "principal/decorrelated components" and keeps those with most variance and discards the rest (as irrelevant and noise). This decomposes the data according to statistical variance (ie 2nd-order statistics are used) usualy performed by variants of EVD/SVD on correlation matrix of data.
PCA is the best, in the mean-square error sense, linear dimension reduction technique.
2. ICA
This factors the data into "independent components" which means that it uses higher-order statistics than 2nd-order which imply decorrelation only. However depending on algorithm, some data may not work well with ICA, since they must not be normally distributed (since for normal random variables decorrelation implies independece as well). Note: PCA is a pre-processing step in most ICA algorithms (eg JADE, ..)
ICA is a higher-order method that seeks linear projections, not necessarily orthogonal to each other, that are as nearly statistically independent as possible. Statistical independence is a much stronger condition than uncorrelatdness.
3. Dictionary and variants
Note that the above algorithms result in a set of "basic" components which can form the basis for all the data. Like the basis vectors of a vector space, each datum is a specific combination/sample of the basic components. Or like a dictionary each datum is a combination/sample of elements from this dictionary. But what if one knows the basic dictionary before hand for a certain problem. Then one can try to find the optimum representation of each datum wrt this basic dictionary. Or can try to learn adaptively this basic dictionary using some adaptve learning method.
4. Factor analysis
Also note that the first 2 approaches extract a set of basic factors from the data (ie they are equivalent to data factors). But what if one assumes a more general probabilistic setting for extracting (linear or non-linear) data factors (factor analysis). For example PCA/ICA can be seen as a specific example of  factor analysis where factors are required to be uncorrelated/independent. This approach is the generic probabilistic form.
5. Other data factorisation methods..
One can get the idea, if the data have certain properties that can be exploited in learning the optimum minimum dimension for represetnation one can try variations of data factorisation methods exploiting those data properies.
6. Unsupervised clustering methods..
Being able to find optimum representations of data automaticaly is a great advantage. Unsuperivised clustering algorithms can be used to this end, in the sense that they can try to cluster the data in an unsupervised manner (no prior information given) and then the cluster representatives can be chosen as the dictionary or basis factors that best represent the data as a whole. This results in dimensionality reduction (eg k-means, vector quantisation, ..)
References for further research:

A survey of dimension reduction techniques

Advances in data collection and storage capabilities during the
past decades have led to an information overload in
most sciences. Researchers working in domains as diverse as
engineering, astronomy, biology, remote sensing, economics,and
consumer transactions, face larger and larger observations and simulations on a dailybasis. Such datasets, in contrast with smaller, more traditional datasets that have been
studied extensively in the past, present new challenges in
data analysis. Traditional statistical methods break down partly because of
the increase in the number of observations, but mostly because of the
increase in the number of variables associated with each
observation. The dimension of the datais the number of variables that are
measured on each observation. High-dimensional datasets present many
mathematical challenges as well as some opportunities, and are bound to give
rise to new theoretical developments. One of the
problems with high-dimensional datasets is that, in many cases, not all the
measured variables are "important" for understanding the
underlying phenomena of interest. While certain computationally expensive
novel methods can
construct predictive models with highaccuracy from high-dimensional data, it
is still of interest in many applications to reduce the dimension of the
original data prior to any modeling of the data.

A survey of dimensionality reduction techniques

Experimental life sciences like biology or chemistry have seen in the
recent decades an explosion of the data available from experiments.
Laboratory instruments become more and more complex and report
hundreds or thousands measurements for a single experiment and
therefore the statistical methods face challenging tasks when dealing
with such high dimensional data. However, much of the data is highly
redundant and can be efficiently brought down to a much smaller number
of variables without a significant loss of information. The
mathematical procedures making possible this reduction are called
dimensionality reduction techniques; they have widely been developed
by fields like Statistics or Machine Learning, and are currently a hot
research topic. In this review we categorize the plethora of dimension
reduction techniques available and give the mathematical insight
behind them.


LOOKUP using 2 dataframes in Python I have 2 dataframes:

df1

Id     CategoryId
1       A
1       B
2       A
2       E
2       F

df2:

Id   A    B   C   D   E   F
1
2

I want to do a lookup which will help me fill up the values in df2 based on the values of df1

If df1 has id = 1 and CategoryId = A then I want df2 row 1 Column A to say Y else N

Final df2 should look like:

 Id   A    B   C   D   E   F
 1    Y    Y   N   N   N   N
 2    Y    N   N   N   Y   Y

I am not sure how to do this in Python.
I would really appreciate any help on this!

Thanks

If you have a dataframe df1 with two columns Id and CategoryId you can chain get_dummies and groupby, e.g.

>>> df2 = df1['CategoryId'].str.get_dummies().groupby(df1['Id']).max()
>>> df2
    A  B  E  F
Id
1   1  1  0  0
2   1  0  1  1

It's not quite the format you wanted but it avoids the lookup.


Peformance evaluation of ranking algorithms I have three questions:

How can we assess (or measure) the performance of the ranking algorithms?
Are there any specific measures, or performance metrics, for this?
More specifically, how can we compare the performance of AHP-based ranking and probabilistic ranking algorithms?

Kaggle's famous competition Chess ratings - Elo versus the Rest of the World, that aimed "to discover whether other approaches can predict the outcome of chess games more accurately than the workhorse Elo rating system", used this structure

  Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games

A similar structure - starting from a complete dataset, using first part
 for training and last part to check the outcome - could be useful to measure the performance of the ranking algorithms.


Siamese network return same loss value while training I have ~9k training samples, the same pair vectors are labeled as 0 and not same pair samples are labeled as 1.

I trained it for 100 epochs for the first 3 epochs the loss value is fluctuating, then it returns the same loss value until the end.

Base network architecture:

def create_base_network():

    a = 'tanh'
    model = Sequential()
    model.add(Dense(self.INPUT_DIM, input_shape=(self.INPUT_DIM, ), activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(900, activation=a))
    model.add(Dense(1000, activation=a))
    model.add(Dense(5000, activation=a))
    model.add(Dense(1000, activation=a))
    model.add(Dense(900, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(self.INPUT_DIM, activation=a))
    return model

Euclidean distance:

K.sqrt(K.maximum(K.sum(K.square(x - y), axis=0, keepdims=True), K.epsilon()))

Initilization model:

    base_network = self._create_base_network()
    input_a = Input(shape=(self.INPUT_DIM,))
    input_b = Input(shape=(self.INPUT_DIM,))

    processed_a = base_network(input_a)
    processed_b = base_network(input_b)

    distance = Lambda(self._euclidean_distance, output_shape=self._dist_output_shape)([processed_a, processed_b])
    model = Model(inputs=[input_a, input_b], outputs=distance)

Compilation of model:

model.compile(loss='mse',
          metrics=['mse'],
          optimizer=optimizers.Adam()
          )

Training the model:

model.fit([self.train_d['vec1'], self.train_d['vec2']], self.train_d['label'], batch_size=128, epochs=self.args.epochs, shuffle=True)

Log statements:

Epoch 1/100
9544/9544 [==============================] - 6s 594us/step - loss: 0.4556 - mean_squared_error: 0.4556
Epoch 2/100
9544/9544 [==============================] - 4s 470us/step - loss: 1.0693 - mean_squared_error: 1.0693
Epoch 3/100
9544/9544 [==============================] - 4s 464us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 4/100
9544/9544 [==============================] - 4s 465us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 5/100
9544/9544 [==============================] - 4s 461us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 99/100
9544/9544 [==============================] - 4s 470us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 100/100
9544/9544 [==============================] - 4s 470us/step - loss: 0.7328 - mean_squared_error: 0.7328

Can someone help me to find out what's the issue here?

Thanks in Advance.

Finally, I found a solution to this problem. I completely modified the architecture, Then I reduced the learning rate to 0.00005. Then the loss value of the model falls continuously.

Latest Architecture:

a = 'tanh'
model = Sequential()
model.add(Dense(self.INPUT_DIM, input_shape=(self.INPUT_DIM, ),
activation=a))
model.add(Dense(600, activation=a))
model.add(Dense(self.INPUT_DIM, activation=a))


I am getting a Type Error in this Line Diff = [i - j for i,j in zip(text_features, author_signature)]

Diff is a List ,
text_features    =  [1, 2, 3] ,
author_signature =  [3, 2, 1]

You must be getting a type error because the elements of text_features and/or author_signatures are not able to be subtracted in the part i - j.

Depending on what they are, you could try converting them. For example, you could try making one or both of them numbers or strings like this:

Diff = [float(i) - float(j) for i,j in zip(text_features, author_signature)]   # both made into numbers

Diff = [str(i) - str(j) for i,j in zip(text_features, author_signature)]   # both made into strings (words)

To give better advice, you need to give an example of what your two lists are that are in the zip()

EDIT

Based on the update in the question, I see that it works as expected for me:

In [1]: text_features = [1, 2, 3]

In [2]: author_signature = [3, 2, 1]

In [3]: Diff = [i - j for i,j in zip(text_features, author_signature)]

In [4]: Diff
Out[4]: [-2, 0, 2]


Data preprocessing : Aggregation, feature creation, or else? I have a problem to name data processing step.

I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step name?
I have an attribute that contain 200 tweets of a twitter user. I make a new attribute that describe URL ratio in 200 tweets (number of URL in 200 tweet / 200). is it aggregation or feature creation or else?

You are looking for names to attribute to the two items listed? For (1) I would just call it a transformation as it is a straight mapping with no change in distribution of the feature. When working with data, make sure you make copies of your data transformation and do not alter the original data set.

For (2), since it is a single number per group, where group here is the full data set I would call it an aggregation. Likewise if you did a similar calculation per user. If however, you computed a new value from existing features for each record, this would be feature generation or creation.


algorithmic difference between image analysis and video analysis Is there algorithmic difference between analyzing video and an image, say for example,if I want object recognition? Or do I just have to analyze every frame of the the video just as an image?

Example, detecting an object in a single image is easy compared to video becuase time dimension is added to video. In addition, on video, in every frame, the object is most probably moving, which makes the frames in motion... So how can u handle the time factor and the "in motion" part in the video. Those are the problem I imagine in video, it would be nice if u add ur own thoughts on it. Thanks

This is a huge topic, so I will just give you a high-level overview and some pointers to more info.

Yes, there are definitely ways to work with video that are different from working with individual still images.

At the simplest level, it could be running an object detector such as HoG (or a sliding-window convnet) on each frame, and then some means of assigning nearby detections in adjacent frames to be the same object, and discarding detections which do not seem to have continuity in time.  Many of the algorithms in this field seem to look at a single frame as a building block to looking at the whole sequence, where perhaps the data from adjacent frames is combined, aggregated, and/or used to disambiguate the current frame.

Another approach is to first estimate object motion between frames (using optical flow, phase correlation, pyramid block matching or another method) and then treat areas of multiple frames that are collocated after accounting for motion as being the same object.  This is very powerful but limited by the accuracy of the motion estimation.

In newer research, there is a back-and-forth between finding where things are (detection) and how things move (tracking), where each task can help the other one, for example (Kalal 2010) or (Andriluka 2008), to the point that the two parts of the algorithm are no longer separable.  Kalal's TLD algorithm is a recently famous version of this.

There are also algorithms which work directly in the spatiotemporal (or sometimes just temporal) domain.  An example of purely temporal would be detecting vehicles by the periodicity of variation of their wheel spokes.

Some of the frequently studied model problems are:

People, vehicles or other objects detection and tracking throughout a video sequence.  For example, using the CalTech Pedestrians dataset or other standard benchmarks.
Hand gesture recognition, either for games, user interface, or sometimes sign language recognition.  Often using depth data, for example Kinect video.
Activity recognition, for example walking vs standing.  Due to the nature of this problem, it is much more common to see pure spatiotemporal algorithms here, for example (Sadanand and Corso 2012).
Simultaneous location and mapping (SLAM).  Often found in robotics, this problem is basically to build a 3D model of the environment from video of a single camera (or stereo or depth video) which is moving around; the environment is often assumed to be static.  This is often done by running some kind of feature detector such as SIFT or SURF, matching up features in successive frames, and then building point clouds from the relative 3D motion implied by features.

Here are some more references:

Shah, Mubarak, and Ramesh Jain, eds. Motion-based recognition. Vol. 9. Springer Science & Business Media, 2013.

Turk, Matthew. "Gesture recognition." Computer Vision: A Reference Guide (2014): 346-349.

Rosenfeld, Azriel, David Doermann, and Daniel DeMenthon, eds. Video mining. Vol. 6. Springer Science & Business Media, 2013.

Kalal, Zdenek, Krystian Mikolajczyk, and Jiri Matas. "Tracking-learning-detection." Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.7 (2012): 1409-1422.

M. Andriluka, S. Roth, B. Schiele. People-Tracking-by-Detection and People-Detection-by-Tracking. Computer Vision and Pattern Recognition (CVPR) 2008

Sadanand, Sreemanananth, and Jason J. Corso. "Action bank: A high-level representation of activity in video." Computer Vision and Pattern Recognition (CVPR), 2012


Preprocess image data to classify objects based on shape Currently I'm trying to build a neural network that is able to classify different types of bottles on an image solely based on the shape. The bottles have no label and at first I only used beer and wine bottles.
My goal is to compare a neural network that uses feature extraction to one that uses the raw images.

Example input image:

https://i.gyazo.com/c22151a4bee6146cc689fcb8358483dd.jpg

Using openCV to process the images, I tried to obtain only the edges of the bottles and classify the types based on the shape.
Currently, I am receiving the following result for the images:

https://i.gyazo.com/8554b06ebf6e8bd520d11222f70c77c5.png

In the next step, I cropped out each bottle in the image and split them into a 2d array of tiles (normally I try with 30x50 but in this image I show it with 10x20)

https://i.gyazo.com/28fb1bc0fa8d61b185ed2245db222cc8.png

For every tile I execute PCA to extract the eigenvectors. In the end I receive a matrix of 10x20 tiles, where each of them contains the direction of the line.
As you can see, the (0.0, 1.0) tuples are the vertical lines, and the (1.0, 0.0) tuples are the horizontal lines.

https://i.gyazo.com/bcd291535ea01b6392ab6aef6747de9d.png

  Sorry for the long post, here is my question:

Is it possible to use this data do classify between wine and beer bottles? The shape of the neck of the bottle is different but is it representative for building a neural network?
I tried to train a model by calculating the gradient (arctan from the tuple) but i didn't manage to get some successful results. Does anyone have a idea how to pre-process the data for a neural network?

Is it better to use the cropped out images of the bottles as image data for the neural network?

I am thankful for any help as this is my first machine learning project and I would really like to get it working.

Well, you are trying so describe the image by predominant line directions and that really sounds fair but a bit of too much work. If you want to you can try a bit of data augmentation to increase your numbers of samples (Mainly affine transformations)
Other possible useful feature would be descriptors such as:

SIFT
SURF
FAST
BRIEF

You could try to describe those shapes using other geometrical functions such as skewness (fill the inside of the shape, consider the eigenvalues to define the major axis directions on PCA)
Finally, bottles in the sample image look similar in shape and differ mainly in size, if your model uses resize it might become scale invariant and then your method will decrease too much of your shape information and give bad results.
------------------------------------------------------------------------------------
Answering the comment bellow:

This is why i tried to do it with my approach of cropping out each bottle and split them into a 2d array of tiles (e.g. 50x20)

This inserts a scale invariance to your model, that is good for processes where image-size vary either because of distance or camera resolution but: This throws alway scale information, that might not be helpful when the main discriminant on your image is scale-related.


Is it possible to get an ROC curve using Relu activation? Based on my understanding, given that Relu doesn't provide probabilities unlike Softmax, it's not possible to plot an ROC curve. However, is there some way to convert the output from a Relu to something like an ROC curve. I'm interested in doing so because it's important for my project (image classification) to be able to adjust the classification thresholds.

YES
The typical way to do a ROC curve is to have probability outputs, set a threshold, and calculate the sensitivity and specificity.
y | prob
----------
0 | 0.2
0 | 0.3
1 | 0.7
1 | 0.8

At a threshold of 0, we classify everyting as $1$, so sensitivity of $1$ and specificity of $0$. At a threshold of $0.25$, we get $0,1,1,1$, so sensitivity of $1$ and specificity of $0.5$. Et cetera...
Plot those $(1, 1)$, $(0.5, 1)$ points to start building your ROC curve.
This idea of setting a threshold and calculating the sensitivity and specificity does not assume probability outputs, so let's do the same with non-probability outputs.
y | output
----------
0 | 2
0 | 3
1 | 7
1 | 8

At a threshold of 0, we classify everyting as $1$, so sensitivity of $1$ and specificity of $0$. At a threshold of $2.5$, we get $0,1,1,1$, so sensitivity of $1$ and specificity of $0.5$. Et cetera...


What methods are available to evaluate similarity between different clustering algorithms? I am performing extensive customer segmentation analysis and so far implemented Gaussian Mixture Models, K-Means, and Hierarchical Clustering. For the most part, the algorithms agree on the structure of the clusters and well as the number (7-8). I would like to know if there is a common method to either...

compare similarity between clusters. Can you apply Adjusted Rand Index to two different clusterings of the same data (k-means clusters vs gmm)? I was under the impression ARI is used in instances where you know the truth of the data.
Find the common clusters within the clusterings. If all of the algorithms say one cluster is defined by high spending, then is there a way to determine the best centroid(s) to use for a "Master" cluster? Is it common to cluster the cluster results?

If you just want to see how similar the clustering is between 2 algorithms, using the sklearn.metrics.adjusted_rand_score() function is a good starting point. This will work for unsupervised learning, no need for a label.

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html

Or are you looking at choosing the best combined grouping overall? I think "clustering the cluster" results is not a common approach, however there are a few articles online on how this could be done. But I don't know of any packages that will do this for you.
One idea is to create a new column for each algorithm with the grouping chosen and then do some comparisions or calculations on these columns to get a final composite clustering.


Algorithm for multiple input single output ML As an ML newbie, I have a question. I have a set of data with 2 inputs and 1 output. I'm trying to predict the output.

input1 is an integer number, input2 is like a category between 1-5. Output is also a number.

input1=25 input2=2 output=25
input1=34 input2=2 output=35
input1=12 input2=5 output=29
input1=3 input2=4 output=48
input1=45 input2=1 output=36

With this data, I want to predict the output for input1=27 and input2=2

I have a small set of data (10-20 items). I wonder which ML algorithm should I learn for this kind of multiple inputs and single output small sets of data?

Edit

With a high probability, while calculating the output, there is a mathematical relation between input1 and input2 like:

output = (input1)*x + (input2)*y (x and y is unknown of course and the equation can be linear or logarithmic or something else. No idea.)

Since you believe the output can be predicted by a linear combination of the inputs, a reasonable approach to try is Linear Regression, specifically Multiple Regression since you have more than one input variable.

Linear regression will attempt to fit the best parameters $\beta_0$ and $\beta_1$ to model your output as a weighted sum of your inputs, ie $\beta_0*input_1 + \beta_1*input_2$. This is exactly the same as the expression you gave, but it's more standard to call the weights $\beta_i$s instead of $x$ and $y$.

The most standard form of linear regression using Ordinary Least Squares will find $\beta_0$ and $\beta_1$ that minimize the sum of the squared errors over your dataset, which are the differences between the actual values of output and the predicted values generated by computing $\beta_0*input_1 + \beta_1*input_2$ for each row.

EDIT: To answer your question in the comments:

It is always reasonable to try linear model first since it is simple and efficient, and it will give you a good baseline.

However, if you suspect there is a non-linear relationship between your inputs and outputs you can also try more flexible models such as gradient boosting regression trees or a neural network.

You do not need to know what the exact relationship is to use these models - they will learn it for you. In theory a neural network can fit any function.

As you use more complex models, however, you should be increasingly wary of overfitting.


what is buyer classification problem? Recently in a data analytic job interview for an e-commerce site, they asked me, do i have some knowledge of buyer classification problem. Unfortunately i heard this term for the first time.
After interview i tried to search a lot about it over google but didn't find something meaningful. Please any one let me know if you have heard this term before and paste some links explaining this concept. Thanks

Buyer classification is used to categorize users who purchase groups of items. Buyers are categorized in order to be targeted for advertising. When users buy similar items, they are more likely to buy similar items in the future. This is useful information when pricing items on a website.

A clear example, a website selling nutritional supplements may want to target different buyers by category. Men in their 20s are unlikely to purchase menopause supplements, and women in their 50s are also unlikely to buy creatine. So, by analyzing user purchase history and categorizing types of buyers, the site can send intelligent promotions - males in their 20s get ads for creatine while women in their 50s get ads for menopause supplements.

Also, if I want to run a sale to attract customers, I don't want to offer lower prices on items which are often purchased together. I'd rathe price one low to attract the buyer, and then hope they buy the complimentary item at full price. Categorizing buyers also helps with this problem.

You may want to read up on shopping cart analysis, which is not a new problem. Department stores have been analyzing shopping carts and classifying buyers long before online shopping was popularized. That's why you have to use membership cards to get the 'special' prices.

Fine tuning these details can increase revenue substantially.


Big array and MemoryError: Unable to allocate memory (in Python) I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.
The data (in a csv file) is in such format:
Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5
2021-04-13 11:03:13+02:00,3,3,3,12,12

My current code:
filename = 'test.csv'
df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0)
df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
u, v, w, x, y, z = df.values.T

X = np.asarray([v, w, x, y, z])
Y = np.asarray([u, u, u, u, u])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).
However now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.
Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?

quick fix would be to change the data format - I can't see how your data looks like so my suggestion stay theoretical without example

float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.

Timestamp - decides what precision of time is needed - if you are fine with days - don't store hours, minutes and others

alternatively usse DataTable https://github.com/h2oai/datatable - Pandas is not the most efficient library when it comes to work with big data. however, I think the first 2 points will fix your issue. Because (157673, 157673) is not too big but 185. GiB sounds too much for it !


ValueError: Error when checking target: expected dense_1 to have shape (7,) but got array with shape (1,) Hi everybody can somebody help me!! I don't know what's wrong in my code and why it gives me this error ???

This normally appears in the training.py/training_utils.py file from the keras package.

I was having a problem when I was using the LIME package on a keras model. (This is a work-in-progress, so the workaround mentioned here may not be the correct approach)

It appears that the shape of the data file differs from the one that the module is expecting. I did a workaround by collapsing the shape of the data object to the one required. Seems to have worked.


Coding Problem - Extracting values from a column and forming a new dataframe [edited] The problem statement requires extracting certain weather parameters for every hour in a particular date as denoted in the dataframe. The column 'hourly' consists of 24 lists in each entry, denoting weather parameters for each hour on that particular date. Is there a way that I can extract the parameter 'CloudCover' for all those 24 hours and form a new dataframe whose columns denote the hours in the day and the corresponding CloudCover value for a single date ?

Edit: Based on the suggestions given below, I tinkered my code and a new problem has come up. While the code suggested by @jahKnows works perfectly fine, It gives only the first value of any parameter in the hourly column entries. For eg. in the date 01-01-2016, the corresponding hourly column entry has 24 values of cloudCover. But the code suggested below gives only the first CloudCover value and moves on to the next date leaving out the other 23 CloudCover values in that particular date. Can you suggest me changes to tackle this problem ? I have attached the updated notebook link and the original dataset link below.

Notebook link(edited): https://anaconda.org/vishwa989796/okayishtrial/notebook

data set link: https://drive.google.com/drive/folders/1wMNOZapHib9AyYFaLdA1jdEdx9DdLnXx

I wrote you two small functions which you can use to unpack a dataframe.

The original dataframe looks like this

import pandas as pd
df = pd.DataFrame(data = temp['data']['weather'])
df.head()

The first one is simple, it takes a dataframe and the name of a column, and it will extract the column into a new dataframe.

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]
    df = pd.DataFrame(data = data)
    return df

df_astronomy = extract_col_as_df(df, 'astronomy')
df_astronomy.head()

df_astronomy = extract_col_as_df(df, 'hourly')
df_astronomy.head()

The fact that the extracted tables sometimes have a useless list of dictionaries with a single value bothered me. Of course you can use the same function as above to extract that column as another dataframe, but then you would have a dataframe with a single column, why not just unpack it in place. So I wrote another function which cleans up the extracted dataframe by unpacking a list of dictionaries with a single value.

A better version

This version also takes a dataframe and a column name to extract a dataframe from it. But from the extracted dataframe if a column contains a list of dictionaries with only a single value it unpacks it.

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]

    data = []
    for datum in df[column_name]:
        record = {}
        for i in datum[0]:
            # If the entry in the record is comprised of a list with a
            # dictionary containing a single value then unpack it
            if type(datum[0][i]) is list:
                if len(datum[0][i]) == 1:
                    key_name = list(datum[0][i][0].keys())[0]
                    record.update({i: datum[0][i][0][key_name]})
                else:
                    record.update({i: datum[0][i]})
            else:
                record.update({i: datum[0][i]})
        data.append(record)

    df = pd.DataFrame(data = data)
    return df

df_astronomy = extract_col_as_df(df, 'hourly')
df_astronomy.head()


Has anyone been able to run Tensorflow with GTX 1070 GPU on Ubuntu 16.04/15.10/15.04? I've been tearing my hair out for the last 3 days and I just can't get it to work.

I recently got a GTX 1070 and I went to a fresh install with:

Ubuntu 16.04/15.10/15.04
PPA drivers nvidia-367
Cuda 7.5

Somehow every step of the way I couldn't get CUDA to detect my GPU despite following various instructions:

https://askubuntu.com/questions/693145/installing-cuda-7-5-toolkit-on-ubuntu-15-10

https://www.pugetsystems.com/labs/hpc/GTX-1080-CUDA-performance-on-Linux-Ubuntu-16-04-preliminary-results-nbody-and-NAMD-803/

Has anyone been able to get 1070/1080 running CUDA and tensorflow correctly and can guide me thru this painful process?

(I can't do Ubuntu 14.04 as it doesn't register my wifi card, which then I can't do anything)

I was in the same boat, but now that I have figured it out, I have listed the steps for installing tensorflow 0.9 with cuda toolkit 8.0, cudnn 5.1, bazel 0.3 on Ubuntu 16.04 LTS here: http://abhay.harpale.net/blog/machine-learning/deep-learning/getting-tensorflow-to-work-with-gpu-nvidia-gtx-1080-on-ubuntu-16-04-lts/

Here's the gist

Install NVidia Cuda Toolkit
Install NVidia CuDNN
Install Tensorflow dependencies such as swig, python-dev, numpy, python-wheel, zlib1g, g++
Configure and build tensorflow using Bazel


Retraining last layer of inception By default this script will run 4,000 training steps. Each step chooses ten images at random from the training set, finds their bottlenecks from the cache, and feeds them into the final layer to get predictions. Those predictions are then compared against the actual labels to update the final layer's weights through the back-propagation process.

According to this paragraph from this article from tensorflow , once the bottlenecks features are calculated for newer classes, backpropagation algorithm is used to retrain the last layer given the output labels. My question is, are the bottleneck features of the 10,000 classes of imagenet present while this retraining progresses - or the retraining is done for the newer class labels only.

I'm running the script now.   It's just creating bottleneck files for the flowers images, i.e. the old bottleneck files are not present.

You can see that the bottleneck files are all created in the /tmp directory.

This makes sense as the point of the script is only to retrain the final layer of the network.   The idea is that the previously trained layers will work for an arbitrary set of images.


How to decide the shape of input features, when each data file is of different length? To help me understand the benefits and shortcomings of decision trees, KNN, Neural Networks, I wanted to build a simple classifier that classifies into 2 classes (Bird Sound and Non-Bird Sound) using all of the above 3 methods.  So I downloaded a sound dataset from kaggle and was exploring pysoundfile as a module to read the sound file. So the following stub:

data, samplerate = sf.read('xc94652.flac')

returns a numpy ndarray. The shape of data varies with each file, some being of (8637686,) and some being (3227894,). Since each file is of different length the shape of data varies with each file. What are some ways, I can make the shape of data equal? I was planning to have the shape of data set to the smallest length from all the files. But it definitely does not do justice to the sound file dataset. I could lose so many features and so the model accuracy at the end.

Data exploration

I would suggest exploring the data a little further, which might help decide what would be the best approach for this bird-song dataset.

For example, have a look at the spectrogram of each bird (there are only 66 different genus types), to see how you might extract more data from the samples. Here is the spectrogram of a sample taken from here:

We can see that there is clearly a repeating pattern! We can see those tall light green blocks intermittently appearing throughout. So although the sample is indeed just over 70 seconds of sound, the call of the bird seems to really only last about 2 seconds!

Either with a simple filtering algorithm or even building a model to find those chunks, you could extract those chunks and only work on those, perhaps along with the data regarding the gaps between those chunks.

That is just one example of data-specific pre-processing; I am sure there are many other ways to improve the information density.

Samplerate

This is another degree of freedom you could look at. One idea would be to accept different samplerates within your input to a model. One could adjust the sample rate to ensure the final samples all are all of the same length.

My idea would be to use the length of the shortest sample and then perform regular sampling of all longer sound snippets, such that they the resulting snippets are all of the same length as your shortest sample.

This method will obviously be compromising the quality of the data (irregularly across samples), but if the starting samplerate is sifficiently high, you might be able to get away with it!

Have a look at this useful article that describes many methods in (pre-processing) sound waves.

Two models

In your particular case, if you really only have two possible lengths: 8637686 and 3227894... it might be feasible to simple create two models, one for each sample length. It definitely isn't an optimal solution; however, it would allow for very quick development and model iterations, as you could use the same model and would only need to change on parameter to use both parts of the data.

Basics

As well as truncating your longer samples (cutting them to match the length of the shorter/shortest samples - you could using padding to simple make the shorted samples match the length of the longest sample.

Typically, this is done by simply adding zeros to the end of the vectors. You could also try adding zeros at the beginning and end to keep the information centered within each sample.

If you create a neural network using Keras, you would probably be best by first looking at the ZeroPadding1d layer.


How to create a training set and classify them as positive or negative I'm trying to create a Sentiment Analysis algorithm for a custom data (government dept specifc data) and not like any other social media data etc. The data exists but I need to categorise the data as positive or negative.
My requirement is to classify the test data as positive or negative using a sentiment analysis algorithm.
The big challenge for me is to take the similar set of government data which I can get. But I need to prepare the training data set. For this I need to classify the training data set as positive or negative.
On what basis do I need to categorise the data set as positive or negative. The reason I'm asking this question is that preparing a good training data set is very important to improve the accuracy of my sentiment analysis algorithm .
I have come up with my sentiment analysis algorithm but really need to prepare a robust set of training data(classified as positive or negative).

Experts - I'm a newbie to the machine learning area and I need advice from the researchers and experts out there.

The only way to obtain a high-quality dataset in your specific domain is to do it manually. There exists no other method that can give you the sentiment labels for texts in arbitrary domains. If there would exists such a method, why would you even bother to create your own model.

You should probably find/hire people that can do this work for you. Adding the sentiment meta-data to the texts could be as easy as creating an additional column in excel. But you could also smoothen the process by creating a little app that shows the sentence and let the user decide what sentiment it is (e.g. by swiping to left or right). Find a bunch of people that are willing to classify 10 sentences a day and your dataset will grow steadily. Even better, apply a little gamification and give people points or a rank and give an award to the best performing user to boost their performance.

Keep in mind that if these texts are really domain-specific you probably need domain-experts to do the classification.

When having enough training data you could learn a (simple) that predicts the output. Then you could let the human classifiers focus on the sentences with low confidence, assuming that high confidence automatic classifications are correct.


How to determine Number of neuron in hidden layer for classification I used Iris dataset for classification with 3 layer Neural Network
I decided to use :
3 neurons for input since it has 3 features,
3 neurons for output since it has 3 classes and
In the hidden layer
what is the best number of neurons to do classification in this case?

There is not a clear a answer for this question. You should try and evaluated how the models performs testing diffferents configurations. But, basically, the next two premises are followed:

More neurons per layer --> more complex model, and probably you will
obtain better accuracy.
More hidden layers --> more complex model, and again, probably you
will obtain better accuracy.
WARNING Both approaches increase the chance of overfitting by increasing the complexity of model. You need to carefully evaluate them through statistically robust model selection procedures e.g. cross-validation

Be aware that incrementing the complexity of the model will increase the computational process and , as mentioned, be careful with overfitting.
I recommend try a small number of neurons first, and then, try to increase step by step seeing if some improvement is achieved in the model.