What is an alternative name for "Unstructured Data"? I'm writing my thesis at the moment, and for some time - due to a lack of a proper alternative - I've stuck with "unstructured data" for referring to natural, free flowing text, e.g. Wikipedia articles.

This nomenclature has bothered me from the very beginning, since it opens a debate that I don't want to get into. Namely, that "unstructured" implies that natural language lacks structure, which it does not - the most obvious being syntax. It also gives a negative impression, since it is the opposite of "structured", which is accepted as being positive. This is not the focus of my thesis, though the "unstructured" part itself plays an important role.

I completely agree with the writer of this article, but he proposes no alternative except for "rich data", which doesn't cover my point. The point I'm trying to make that the text lacks a traditional database-like (e.g. tabular) structure of the data, with every piece of data having a clear data type and semantics that is easy to interpret using computer programs. Of course I'd like to condense this definition into a term, but so far I've been unsuccessful coming up with, or discovering an acceptable taxonomy in literature.


XGBoost ranking file format The xgboost package has two files that must be used for ranking:

train.txt with the data
train.txt.group with the group of each observation

I don't understand two things in each file:

What should I use as positive/negative classes? In ranking, there is no such thing as a positive/negative class...
What should I use as group? Say that I have observations x1 > x2 > x3. How do I express this lexicographical order using groups?

Since I asked this question, it seems the documentation was expanded:

We need to provide a ".group" file. For example, the file could be

2
3

  This means that, the data set contains 5 instances, and the first two
  instances are in a group and the other three are in another group. The
  numbers in the group file are actually indicating the number of
  instances in each group in the instance file in order.


What classifier is the best to determine if object was detected in the correct position? I have a dataset which contains information about objects from 20 different categories. These objects are manually labeled on 100 images. The objects are located in 3 rows on a square image, there can be multiple instances of the same class object. For a given object, I have the following data:

its class
bounding box center coordinates
bounding box size
information whether the object is in the correct position or not

The ultimate goal is to have a classifier, which determines if the position of the detected object is correct. I am not sure if I should treat a label or "correctness" as an output, what would you suggest? Also, what are the best candidates for classifiers in case of this data, given the following:

Each training image (with labelled objects) is warped by a small angle, perspective, translations etc.
There is one correct arrangement of objects on the image

Please look below for simplified illustration of the data:
number - label,
bold - incorrect position

===================

1 1 1   3 3  5

6  7   1 1  4 4

9 5 5 5 9 7 7 7

===================

It might actually be called Intersection Over Union.

  So how do you tell if your object detection algorithm is working well? In this video, you'll learn about a function called, "Intersection Over Union". And as we use both for evaluating your object detection algorithm, as well as in the next video, using it to add another component to your object detection algorithm, to make it work even better. Let's get started. In the object detection task, you expected to localize the object as well. So if that's the ground-truth bounding box, and if your algorithm outputs this ...


Doubt in interpretation of Bernoulli Naive Baeyes Algorithm We say Bernoulli naive bayes assumes gaussian distribution of all continuous features. What happens if I have categorical features also in the dataset?

What type of prior transformation in data is suitable before applying Bernoulli naive baeyes to a dataset ?

Bernoulli Naive bayes does not assume gaussian distribution of all continuous features, because it does not make sense. Gaussian Naive Bayes assumes gaussian distribution for continuous features and it is the appropriate way for using Naive Bayes approach if you have continuous features.

On the other hand, if you have binary categorical data then the appropriate approach is Bernoulli Naive Bayes. If your features are categorical but not binary then you could transform them into binary categorical using dummy boolean variables for each available value of the categorical features. The main point of Naive Bayes algorithm is the assumption of feature independence, which in some real world classification problems does not hold.

You need to specify a conditional probability p(x|y) of the feature value x given the class label y. Since Naive Bayes assumes that all features are conditionally independent given the class, you can mix different likelihood models for each feature considering any prior knowledge about it.

For example, considering a continuous feature you might assume that p(x|y) is normally distributed, then you can stimate the mean and variance for this feature under each class in the training set and after that you can use the PDF of the Normal Distribution to estimate p(x|y).
Considering another feature which is categorical, you can estimate p(x|y) using a Bernoulli or multinomial event model and multiply the two conditional probabilities together in the final prediction (since they are assumed to be independent anyway).


Why there is two output in Titanic case in tflearn quickstart? I am stuck in the tflearn quickstart guide.

I've tried tit and it works perfectly with accuracy around 78%, but the problem is I don't understand why the "labels" consist of two "survived" columns that are the opposite of each other. I've trained the same data but used only single "survived" column in the labels or the output, but with the same code I just get around 36% with 0 loss or even NaN loss.

So why is there two columns output/labels? And why can't I use only a single column?

Since you have only one binary target to predict (survived vs not survived) you can use only one output (that provide the probability of surviving). The problem with this approach is that you can not longer use softmax activation, since it will always be 1. Remember that softmax activation has the form
$$g(x_k)=\frac{e^{x_k}}{\sum_{i=1}^n e^{x_i}}$$
Since you have only one element to run the activation it will be normalized always as 1.
It is used mostly for categorical prediction (i.e. among several possibilities only one is correct).
You are getting $0$/$NaN$ loss because your model always return 1. Instead of using softmax activation you might try sigmoid activation, so you can interpret the result of the network evaluation as the probability of survive.


Shaping columns (and column headers) into multi-index rows using pivot/pivot_table Problem
I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the stacked data becoming columns (see image below for the output format) and the column headers (TP1, TP2, etc) becoming indices along with the dates.
I am new to Python and Pandas (C and MATLAB programming experience), initially I thought to loop through the data and append the relevant parts of each row into a new data-frame. It seems to me however that either the df.pivot() or pd.pivot_table() would be a more suitable way to achieve this as it would allow some aggregation at the same time.
Sample dataset:
df = pd.DataFrame({'POC_Code': ['ARA', 'ARA', 'ARA', 'BPE', 'BPE', 'BPE'],
                   'Trading_date': ['1/04/2018','2/04/2018','3/04/2018','1/04/2018','2/04/2018','3/04/2018'],
                   'TP1':[10120,11760,25930,3545,12749,11358],
                   'TP2':[10170,11790,25890,4329,13793,15448],
                   'TP3':[10200,11750,25860,3465,13943,16132]})

  POC_Code Trading_date    TP1    TP2    TP3
0      ARA    1/04/2018  10120  10170  10200
1      ARA    2/04/2018  11760  11790  11750
2      ARA    3/04/2018  25930  25890  25860
3      BPE    1/04/2018   3545   4329   3465
4      BPE    2/04/2018  12749  13793  13943
5      BPE    3/04/2018  11358  15448  16132

My Attempt
Using the following I get close what I want (the image below), but I am still not sure how to turn the TP1, TP2, into indices:
piv = df.pivot_table(values=['TP1','TP2','TP3'], index=['Trading_date'], columns='POC_Code')

                TP1           TP2           TP3
POC_Code        ARA    BPE    ARA    BPE    ARA    BPE
Trading_date
1/04/2018     10120   3545  10170   4329  10200   3465
2/04/2018     11760  12749  11790  13793  11750  13943
3/04/2018     25930  11358  25890  15448  25860  16132

I tried to access the column headers like this, but it's not valid which I get: piv = df.pivot_table(values=['TP1','TP2','TP3'], index=['Trading_date', df.columns.values[2:5]], columns='POC_Code')
Desired Output
I need the output to be a chronological format (by date & trading period) such as the multi-index way below, so that I can combine it into a single index of the form 2018-04-01-01 for export to another program. If this combination of the indices can be done in the same step, even better.

Is this possible with pivot/pivot_table, or would something like this (https://stackoverflow.com/questions/37430940/python-pandas-converting-column-headers-into-index) be a better approach (or a combination of both)? The full dataset includes 65 POC_Codes and 50 Trading periods, so I would like to use a pivot table as I also plan to sum these into various categories.
Full dataset can be found at: https://www.emi.ea.govt.nz/Wholesale/Datasets/Generation/Generation_MD/201804_Generation_MD.csv

For this I would perform two steps, first transforming the data from a wide to long format using pandas.melt (i.e. transform the TP columns over the rows) and then use pandas.pivot to get the desired format.
result = (
    df
    .melt(id_vars=["POC_Code", "Trading_date"])
    .pivot(index=["Trading_date", "variable"], columns="POC_Code")
    .reset_index()
)

result

# Trading_date variable  value
#                         ARA    BPE
#   1/04/2018      TP1  10120   3545
#   1/04/2018      TP2  10170   4329
#   1/04/2018      TP3  10200   3465
#   2/04/2018      TP1  11760  12749
#   2/04/2018      TP2  11790  13793
#   2/04/2018      TP3  11750  13943
#   3/04/2018      TP1  25930  11358
#   3/04/2018      TP2  25890  15448
#   3/04/2018      TP3  25860  16132

If you want to change the column names you can simply assign the new column names to df.columns:
result.columns = ["Trading_date", "TP", "ARA", "BPE"]


How to simulate customer walk-ins for a given period in a fast food chain in R I would really appreciate it if someone could tell me where I would start in tackling the following tasks in R.

I'm not a data-science expert and am trying to teach myself data-analysis from a background in physics. I have introductory statistics and have just finished Udacity's course in R programming and exploratory data analysis.

Tasks:

1 )Simulate customer walk-ins for a given period. The simulations should take into account peak volumes at different times of the days and differences in week days and weekends.

2) Simulate customer purchases based on the customer walk-ins. Make assumptions on the average dollar price and variations for different times of the day.

Thank you

You need either an initial data set or a large set of constraints to "fake" a data set.  if you have an initial data set you could make somewhat valid assumptions regarding traffic flow, purchasing behaviors and the likes.

If not, you are going to need to research these different metrics from existing companies that are like what you want data for and then I assume put together a little program with those metrics as constraints and simulate your data.  Once you have those constraints you could use simple averaging or weighted options for developing a constraint per variable.

Make sure they are semi accurate.  For example, if you were modeling company growth for a "really good idea" research growth trends for like companies and for companies you want to emulate and then scale BACK from there.

You can also always purchase a dataset:
http://datamarket.azure.com/browse/data


Drop all nan value in Series I have try to drop a nan value like so

import pandas
data = pandas.read_json("data.json")
lat = data["Latitude"]
lng = data["Longitude"]
shop = data["Store - Business Name"]
df2 = lat.dropna(axis=0, how='any')
df1 = pandas.to_numeric(df2, errors='coerce')

print(df1)

what am i missing, because i still get nan value in my panda series.

lat is a series in your data, if you have even only one nan value in your series ,lat, then you will loose all of your series.
Another problem might be in the lat column values might not be suitable for converting to pd.to_numeric such as "1,2" (not "1.2"), "a", "nan" etc. Then you will loose your series again.
Therefore:

Check:
len(lat.dropna()) == len(lat)
if it is True then check:

len(df2[df2.str.contains(",")]) > 0

I hope it works!


How to measure Entity Ambiguity? When using/building a system for Entity Linking, is there a well-known measure for "ambiguity degree" of an entity?

Some approach to compare named entities regarding how difficult to disambiguate?

Entity Linking is a type of supervised machine learning, thus many of the common performance metrics could be used. In particular, creating a confusion matrix would identify where one label was predicted but the ground-truth was different. Confusion matrices can be calculated with counts or normalized, a normalized data would be an estimate of "ambiguity degree" relative to the other labels in the current dataset.

Other classification measures such has F-score, precision, and recall could also be used. In particular, low precision for a label would suggest the model has trouble disambiguating entities from nearby text. "Cheap and easy entity evaluation" goes into more technical details.

Inter-rater reliability could also be used, the raters could be different humans or different models. If the joint-probability of agreement between different raters is low then entities could be regarded as difficult to disambiguate.

The performance also depends on the relative value of an exact match vs partial match.


The meaning of γt−t0 in Reinforcement learning with pytorch When reading pytorch tutorial:

  Our aim will be to train a policy that tries to maximize the discounted, cumulative reward Rt0=∑∞t=t0γt−t0rt, where Rt0 is also known as the return

I know γ is the discount factor, but I am not sure that what t-t0 ofγt−t0 mean?

Thank you.

I have no experience with reinformcement learning, however looking at the figure I think I understand what is meant. Gamma is the discount factor, which is taken to the power t-t0, i.e. the number of episodes starting from t. This gives the discount factor for a specific episode, which is then multiplied by the return of that episode, r_t, to get the discounted reward for one specific episode. The total return is then computed by summing all the future rewards for future episodes.


What are machine learning/deep learning models for generating contextually related words and synonyms? I have a task to work on models for finding synonyms and contextually related words.
For example, if I enter:

'car' it should generate -> 'vehicle'
'sun' and 'sea' could generate 'beach', or some other related word to the first two.

So I used so far word2vec and nltk to generate examples.
But since I am not an expert on NLP I really find it difficult to use other algorithms or to build my neural network architecture.
I would appreciate it if someone can give me other suggestions and some explanations, that could be useful.

For synonyms I would directly use WordNet.

[added] For contextually similar words the traditional approach is to extract a context vector for every target word:

for every occurrence of a target word extract the words within a -/+ N window (e.g. N=5).
for every target word aggregate all its context words in a single context vector over the whole vocabulary.

Finally once a context vector has been calculated for every target word a similarity measure can be used, for example cosine. That means for every target word, compare its vector against any other candidate.

The same approach can be used with word embeddings instead of context vectors.


Time Series Forecasting I work in the Oil & Gas industry.

I have been trying to build a ts forecasting model with covariates, and the model R code is as follows:

#Getting R libraries:
library(readxl)
library(ggplot2)
library(forecast)
library(timeSeries)
library(tseries)
library(MTS)

#Create a time series object:
myts <- ts(dataset, start = c(2005,1), end = c(2019,12), frequency = 12)

#Illustrate out of sample forecasting with covariates, splitting the data:
train <- window(myts, end = c(2018,12))
test <- window(myts, start = c(2019,1))

#Fitting the time series forecasting model:
covariates <- c("Income","Prices","Sites","Vehicles")
fit <- auto.arima(train[,"Volumes"], xreg = train[,covariates])

#Forecasting from test data:
mytsfcast <- forecast(fit, h = 6*12, xreg = test[,covariates])

autoplot(mytsfcast)

However, I have been trying to forecast the retail volume sales 12,24,36, etc months out. The model only generates the following result:

Model fit results:

Please may I ask that you kindly advise on how I can get my model to forecast beyond end = c(2019,12). I am missing something?

I have not been able to reproduce the blank model fit results.  But, I know that the forecast function will only forecast up to known xreg inputs. For instance, your test period starts at 1/2019 and ends 12/2019 (12 periods) however you're attempting to forecast  72 periods.  Try setting the h parameter in the forecast function equal to 12 and see if you get the same result.

To get around this problem in the past, I have forecasted the xreg series into the future (in your case an additional 60 periods for each of the four xreg variables).  You'll then need to add these 60 periods to the end of the test set for each respective variable.


Does the count of items in a transaction matter to apriori? When preparing my data for funneling into the Microsoft Association Rules algorithm, I was not sure if I should group by data by Transaction and Item, or have a record for every instance of an item in a transaction. Does the algorithm care and add weight if an item appears 3 times in a transaction  Or is it just looking for the existence of an item with another item, regardless of how many are present?

No it is not important and highly recommended to remove the duplicate items and sort the items in lexicographical order in each transaction. This is to improve the performance.

In association rule mining, an item is frequent iff it is repeated in multiple transactions not in a single transaction. This is why you don't need to have duplicate items in a each transaction.


Decision Tree Ensembling In order to ensemble a decision tree, let me explain the specific situation.
I have split the dataset in 5 sections, per protocol e.g. TCP, HTTP etc.
Now I've trained a decision tree for each one and run this against my test dataset.

How do I go about combining the 5 prediction models.
Do I....

a) combine the predicted output e.g. 1,0;1,1 for each section done separate and then run it against the test dataset to identify confusion matrix. This is actual class, predicted class.
or
b) do I take the tree build and then add the additional ones to the tree model, in effect combining them.

Which method should be suitable, would option a even be a good solution?

Popular ways to combine trained models are schema's based on voting: predictions are treated as weighted votes, the class with the majority of votes is selected as ultimate prediction, and/or stacking: predictions are treated as features in a newly trained model.

Depending on the domain at hand one of this approaches might help you. Stacking (depending on the metalearner of choice) introduces more freedom and might introduce a complexity that is not warranted by the concept you are searching for, while voting won't help much when identifying subdomain fitted for perticular problems in your search space.

Also when combining multiple classifiers it might be worth your while to try out some fundamentally different approaches.


Probabilities of a Poisson distribution not making sense I am trying to find probabilities of orders a restaurant gets on Sunday's. For last 6 months average orders are 1000 without any big anomalies like 700 or 1300. This is a case of poisson distribution & I used scipy library in python and plotted the probabilities.
The graph shows probabilities that of getting orders greater than values on x-axis, e.g. probability of orders>800 is 1, >950 is close to 0.95, >1000 is close to 0.5 & so on.
I am not sure if these probabilities are  correct esp. for high order counts, e.g. for >1050 orders prob is less than 0.1, for >1100 almost zero which I find odd because its quite likely to have 1100 or little more orders occasionally. So what are thoughts on probabilities calculated using Poisson distribution.
prob_arr=[]
orders_arr=[]

dist=poisson(1000) #mu=1000

for num_orders in range(800,1201,50):
    prob_arr.append(dist.sf(num_orders)) #survival function
    orders_arr.append(num_orders)

plt.figure(figsize=(12,4))
plt.plot(orders_arr,prob_arr,linewidth=2)
plt.xlabel("Orders Count")
plt.ylabel("Probability")
plt.title("Probabilities of Minimum Orders Count");

As suggested by @Romain Reboulleau tried truncated norm distribution and below graph shows probabilities of getting orders >than values on x-axis derived using  scipy truncnorm.

Problem is low probability of getting orders >=1100, it is 1.3% as per truncnorm vs only 0.09% with poisson.  This would make non-stats business managers laugh and I am not sure how to justify this.

This seems quite right to me. A poisson(1000) law has a standard deviation of $\sqrt{1000}$ which is something around $31.6$.

At such values, the Poisson law pretty much behaves like a normal distribution, so the probability of getting a value greater than 1100 would be very close to 0 (around 0.1%).

Perhaps what you need is not a Poisson law, but a truncated normal distribution?


How to remove outliers using box-plot? I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)?

All the ['AVG'] data is in a single column,
I need it for time series modelling.

Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas.

If we assume that your dataframe is called df and the column you want to filter based AVG, then

Q1 = df['AVG'].quantile(0.25)
Q3 = df['AVG'].quantile(0.75)
IQR = Q3 - Q1    #IQR is interquartile range.

filter = (df['AVG'] >= Q1 - 1.5 * IQR) & (df['AVG'] <= Q3 + 1.5 *IQR)
df.loc[filter]


need explanation on how an equation is being converted to cvxopt logic in solver.lq This is the equation that is given in the example:

and the code to replicate it in python is

>>> from cvxopt import matrix, solvers
>>> A = matrix([ [-1.0, -1.0, 0.0, 1.0], [1.0, -1.0, -1.0, -2.0] ])
>>> b = matrix([ 1.0, -2.0, 0.0, 4.0 ])
>>> c = matrix([ 2.0, 1.0 ])
>>> sol=solvers.lp(c,A,b)

Can anyone please explain how are the the data given to variables a, b, c?

Welcome to the site!

As Media has mentioned values of A,b,c are passed through those matrices.
To understand it better I'm naming the constraints from top to bottom i.e., 1st constraint as constraints-1,......constraint-4.

Firstly, lets talk about matrix b which consists of all the values on the right hand side of the constraints i.e., 1,2,0,4. He converted all the constraints to $⩽$ by multiplying with -1 on both sides. So he multiplied with -1 for constraints-2 and 3.

Now the constraints-2 has changed from $x_1 + x_2 ⩾ 2$ to $-x_1 - x_2 ⩽ -2$

Values of Matrix b would be 1,-2,0(-0),4

Now lets go through the constraints i.e.,

$-x_1 + x_2 ⩽ 1 $ for the coefficient of $x_1,x_2$ are -1 and 1. The values of Matrix A row 1 values are -1,1
Similarly for row 2 to row 4 as you have 4 contraints.

Finally our objective function coefficients, are the values of c i.e., 2 and 1.

Do let me know if you have any more doubts.


Counting the number of layers in a neural network I am going over the Udacity tutorial on Neural Networks.

Here's a diagram from the tutorial:

What makes this a '2 layer neural network'?

I was under the impression that the first layer, the actual input, should be considered a layer and included in the count.
This screenshot shows 2 matrix multiplies and 1 layer of ReLu's. To me this looks like 3 layers. There are arrows pointing from one to another, indicating they are separate. Include the input layer, and this looks like a 4 layer NN.

This is a 2-layer network because it has a single hidden layer and an output layer. We don't count the first layer. When we say 3 layers, we actually mean 2 hidden layers and an output layer. Perhaps this helps you?

EDIT: We don't count the input layer because there's no parameter (bias + weights). In actual implementation, it's not implemented. Netural network framework simply "connects" the input features to the first adjustable layer (eg: hidden layer).


how to work with a large dataset in python? for a training purposes, in order to start machine learning and data analysis with python I work on a pretty small dataset (20000 rows, 600MB) .But since couple days I decided to go further so  I have download a large data set of 1Go.

I wanted to do some analysis and apply machine learning on it so I tried to read the csv file with pyhton in jupiter nootebook and th file is still loading after more than 3 hours.

So I will would like to know  what are the best practices/process to follow when you have to work with large dataset ?

Before exploring more sophisticated tools like Spark or Dask, one option would be to read the data in chunks instead of loading the whole file. For example, if you are using pandas the read_csv method accepts chunksize argument.

The main idea is that often what you need to do is reduce each chunk down to something much smaller with only the parts you need (some average/count/sum etc.. of some classes) and store a series/array/dict etc accordingly.


Unable to open .json file in pandas I want to convert a json file into a dataframe in pandas (Python). I tried with read_json() but got the error:

UnicodeDecodeError:'charmap' codec can't decode byte 0x81 in position 21596351:character maps to <undefined>

I think I have some unwanted data in the json file like noise. The data is server generated.

This is a collection from the json file:

{"_id":{"$oid":"57a30ce368fd0809ec4d1b41"},"session":{"start_timestamp":{"$numberLong":"1470151881189"},"session_id":"8356bd90-20160802-153121189"},"metrics":{},"arrival_timestamp":{"$numberLong":"1470152028294"},"event_type":"OfferViewed","event_timestamp":{"$numberLong":"1470151943271"},"event_version":"3.0","application":{"package_name":"com.think.vito","title":"Vito","version_code":"5","app_id":"7ffa58dab3c646cea642e961ff8a8070","cognito_identity_pool_id":"us-east-1:4d9cf803-0487-44ec-be27-1e160d15df74","version_name":"2.0.0.0","sdk":{"version":"2.2.2","name":"aws-sdk-android"}},"client":{"cognito_id":"us-east-1:1d507b8f-857c-42a4-a705-8db07d46bc8f","client_id":"aa092911-b9a7-498a-82da-76318356bd90"},"device":{"locale":{"country":"US","code":"en_US","language":"en"},"platform":{"version":"5.1.1","name":"ANDROID"},"make":"Xiaomi","model":"Redmi Note 3"},"attributes":{"Category":"90000","CustomerID":"4077","OfferID":"11846"}}

You have to read the file line by line, you can find a detailed answer in this question of stackoverflow


Deploying ML/Deep Learning on AWS Lambda for Long-Running Training, not just Inference Serverless technology can be used to deploy ML models to production, since the deployment package sizes can be compressed if too large (or built from source with unneeded dependencies stripped).
But there is also the use case of deploying ML for training, not just inference. For example, if a company wanted to allow power users to retrain a model from the front-end.
Is this feasible for Lambda given the long training times?
Whereas latency wouldn't be issue (cold start delay is fine) the runtime could be fairly long (hours).

I've used ECS (Fargate) to train models, the retraining trigger could be the start of an ECS service. While ECS has a little latency, it handles well long runtimes.
You can then serve the model via a lambda.


Boundary conditions for clustering I have some data that I would like to cluster with k-means.

One of the features is the hour of the day.

The problem is that the hour '23' is considered far from the hour '0'.

How can I map the data so that the boundary will be cyclic?

Since you accepted another answer, which says this can't be done, I am editing this to include an example of it being done.  Hope this helps!

Original Answer:

The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sink with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24).

You need both variables or the proper movement through time is lost.  This is due to the fact that the derivative of either sin or cos changes in time where as the (x,y) position varies smoothly as it travels around the unit circle.

This method works really well for clustering and for keeping the distance between 15 minutes after midnight and 5 minutes before midnight "close" in Euclidean space.  All of the modulo suggestions don't accomplish this and the cyclic representation that they do accomplish is pretty clumsy.

Finally, consider whether it is worthwhile to add a third feature to trace linear time, which can be constructed by hours (or minutes or seconds) from the start of the first record or a Unix time stamp or something similar.  These three features then provide proxies for both the cyclic and linear progression of time e.g. you can pull out cyclic phenomenon like sleep cycles in people's movement and also linear growth like population vs. time.

Hope this helps!

Example of if being accomplished:

# Enable inline plotting
%matplotlib inline

#Import everything I need...

import numpy as np
import matplotlib as mp

import matplotlib.pyplot as plt
import pandas as pd

# Grab some random times from here: https://www.random.org/clock-times/
# put them into a csv.
from pandas import DataFrame, read_csv
df = read_csv('/Users/angus/Machine_Learning/ipython_notebooks/times.csv',delimiter=':')
df['hourfloat']=df.hour+df.minute/60.0
df['x']=np.sin(2.*np.pi*df.hourfloat/24.)
df['y']=np.cos(2.*np.pi*df.hourfloat/24.)

df

def kmeansshow(k,X):

    from sklearn import cluster
    from matplotlib import pyplot
    import numpy as np

    kmeans = cluster.KMeans(n_clusters=k)
    kmeans.fit(X)

    labels = kmeans.labels_
    centroids = kmeans.cluster_centers_
    #print centroids

    for i in range(k):
        # select only data observations with cluster label == i
        ds = X[np.where(labels==i)]
        # plot the data observations
        pyplot.plot(ds[:,0],ds[:,1],'o')
        # plot the centroids
        lines = pyplot.plot(centroids[i,0],centroids[i,1],'kx')
        # make the centroid x's bigger
        pyplot.setp(lines,ms=15.0)
        pyplot.setp(lines,mew=2.0)
    pyplot.show()
    return centroids

Now lets try it out:

kmeansshow(6,df[['x', 'y']].values)

You can just barely see that there are some after midnight times included with the before midnight green cluster.  Now lets reduce the number of clusters and show that before and after midnight can be connected in a single cluster in more detail:

kmeansshow(3,df[['x', 'y']].values)

See how the blue cluster contains times that are from before and after midnight that are clustered together in the same cluster...

QED!


Obtaining frequently occurring phrases using Word2Vec I'm rather new to Word2Vec, having started working on it about a week ago.
My question is this: Is there a way to obtain frequently occurring phrases in a large document using Word2Vec along with a score to denote the 'frequency'?

You may use gensim phrase vectorizer module available in Python.
You need to give threshold value which is some sort of pmi of words. The higher this value less are the number of phrases the default is 10. You can play around with this value to get results for your data.

phrase_threshold = 1
bigram = Phrases(sentences,threshold=phrase_threshold)

This is based on this based on the skipgram paper by Tomas Mikolov.


What to choose: an overfit model with higher evaluation score or a non-overfit model with lower one? For lack of a better term, overfit here means a higher discrepancy between train and validation score and non-overfit means a lower discrepancy.

This "dilemma" just showed in neural network model I've recently working on. I trained the network with 10-fold cross-validation and got overfitted model (0.118 score difference):

0.967 accuracy for training set and
0.849 for validation set.

Then, I applied dropout layer with dropout rate of 0.3 after each hidden layer and got "less overfitted" model (0.057 score difference):

0.875 accuracy for training set and
0.818 for validation set

which is supposedly good since have lower discrepancy thus have better reliability for unknown data. The problem is, it has lower validation set score. My uninformed intuition says that no matter how overfitted your model is, validation set score is what matters because it indicates how well your model sees new data, so I choose the first model.

Is that a right intuition? How to go for this situation?

What library are you using?
Dropout is used during training to prevent overfitting.

Make sure dropout is not applied for the validation (which is standard for Keras). This could artificially decrease your validation accuracy.

Also, accuracy is a bad metric to evaluate your performance. See this answer to find out why. Try ROC-AUC to evaluate your model performance.


How to represent target variable for chess AI Inspired by Google's recent AlphaGo project, I've decided that as a fun personal challenge I'd like to use deep learning and convoluted neural networks to build an algorithm that can beat an ordinary chess program when the difficulty is set to expert. I'm awful at chess and generally can't beat a computer beyond easy/medium, so making something that's a lot smarter than I am (at chess, at least) sounds fascinating.

Question:
What should the target variable look like for predicting the next best chess move?

I've been building predictive models for a long time, but I can't wrap my head around this question. Some things that have me confused:

I believe the target variable for Go is simply the entire grid of spaces on the board. In chess there are a limited number of legal moves, which continuously diminish as pieces are removed from the board throughout the game. How do I represent a shrinking target space?
Not all pieces have the same set of rules (ex: queen vs knight). Do I need to somehow explicitly code these rules or will deep learning automatically learn them?

Notes:

I know that alpha-beta pruning and quasi-brute-force search
techniques have been shown to do well in chess, however, I
specifically want to use deep learning instead.
AlphaGo started with supervised learning (SL) on documented expert human moves to make a policy network, among other things. In my case, I'm just trying to do the supervised learning part, and when I do that, the target variable is difficult for me to define. That's the root of my question.

See the following paper:

Giraffe: Using Deep Reinforcement Learning to Play Chess - Lai


How to implement handmade features in a Keras Sequential model? I wanted to know how to implement my handmade features in my Keras model.
Below I have some code where the embedding_matrix is the Stanford Glove NLP dataset.

def BidLstm(maxlen, max_features, embed_size, embedding_matrix):
   inp = Input(shape=(maxlen, ))
   x = Embedding(max_features, embed_size, weights=[embedding_matrix],
                 trainable=False)(inp)
   x = Bidirectional(LSTM(300, return_sequences=True, dropout=0.25,
                          recurrent_dropout=0.25))(x)
   x = Attention(maxlen)(x)
   x = Dense(256, activation="relu")(x)
   x = Dropout(0.25)(x)
   x = Dense(6, activation="sigmoid")(x)
   model = Model(inputs=inp, outputs=x)

I want to implement some extra features (preferably numpy array of features) into my keras model.

Any suggestions?

Very interesting  question , I am  wondering if there  are nough skilled  data science  entusiast   that  could   answer this question , because I am  interested in simmilar stuff as well.
Congratulations  for this  question  man


Why is large decision tree likely to overfit My lecture slide told me that if we don't prune the regression tree, then the tree likely to over-fit.
So, I wonder why would that happen?
Is that because if the tree grows too large, we would end up with very little instances on each leaf nodes of the tree so the estimated mean value on each leaf node will be not accurate?

Overfitting means that a model is giving a good fit on a dataset (whatever the measure you use to assess fit), but this is not a general case (i.e. when new data comes in or on another dataset, the error will explode. Or said otherwise, the model variance is high).
In the case of trees, adding a node to a leave based on one feature should be done only if the feature really brings information at this level. The feature could be random though and this would deteriorate greatly the fit.
As a simplistic example on a classification task; if we want to sort out apples and oranges based on some features, including one of the features that is a value, 0 or 1 chosen at random.
If it happens that in our dataset the values 1 correspond in 80% of the time to apples, then we would be tempted to add a node saying "if value is 1, then apple", but you can see that this is absolutely not a generality: the tree wouldn't fit at all to another dataset.
Hence this node shouldn't be added, i.e. the tree should be pruned.


Predicting Multiple Values Values Using Time Series Forecasting I want to illustrate my question with the following example:
I have a wholesale company through which I sell 200 products:
P1,P2,P3 .... P200
to a 1000 customers  C1,C2,C3 ... C1000
I keep my sales records in a database as a time series in which I keep the product_id,customer_id, amount and timestamp for each transaction over a period of 4 years.
My question is regardless of the forecasting algorithm (static or learning), can predicting the sales of all of these products made through a single model? or model for each product, or a model for each customer? If any or many of these choices can be valid why ?

Thanks for this question I think it is a nice use case to play with time series forecasting in all (or many) of its types.
As you suggest, there are several possible approaches, and all of them are valid hypothesis a priori to check and validate with your goal. Answering the question:

yes, it is possible to build a single model to predict the sales amount of your products at once (aka a multi-output forecasting model) where each of your products would be a feature to predict --> https://www.tensorflow.org/tutorials/structured_data/time_series#multi-output_models, and you can whether predict a single time-step or a multi time-step forecasting

I would say building a model for each product makes sense, as you will potentially have many more customers than products throughout of time; this way, for each product you will have several customers who bought it, and modelling the clients characteristics for a certain type of product sounds reasonable; in this case, you have a single-feature to predict (in the same source of info you have this model type, agsain for both single time-step or multi time-step forecasting: https://www.tensorflow.org/tutorials/structured_data/time_series#single_step_models


Do I use the mean vector from my training set to center my testing set when dimension reducing for classification? Please let me know if this is the right place to ask this (or if any of my tags are wrong) or if I need to write this any differently.

Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?

I am using the principal component analysis procedure to reduce the dimensions of the training set.  I build the classifier.  Then, before I classify the feature vectors from the test set, during the centering part of the dimension reduction, do I use the same mean vector from the training set, do I take the mean vector of the testing set and subtract that from the test set, or do I take the mean vector of the union of the training and test set and subtract that from the test set?

If the third option, does that mean I was also supposed to use the union of the training and testing set to center the training set as well?  No, (for the sake of generalizing to other testing sets) right?

Also, even though I am pretty sure the answer will be the same as above, can you please let me know if the same is true for using the covariance matrix from the training set to get an eigenvector matrix and multiplying the inverse (transverse) of it times the test set to reduce it.  Or, do we use the testing set or the union of the two to get the covariance and then eigenvector matrix to multiply times the testing set?

Please let me know if any of the premises are wrong.  This is my first time.

Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?: Yes.

Test set must not be combined with training set in any step of calculating the reduced dimension space. Characteristics of final space is determined by training set and test set just follows that i.e. the mean-adjusting step uses training mean.

You just calculate the final eigenvectors matrix $E$ (whose dimension is $d\times d$ at the beginning where $d$ is the dimensionality of data, and becomes $d_{reduced}\times d$ after choosing top vectors) and then your test data $D$ ($n\times d$) is just multiplied to that matrix, you get test data in the reduced space ($D^{'}$):

$$D_{n\times d}\times E^{T} = D^{'}_{n\times d_{reduced}}$$

where the dimensionality of $E^{T}$ is $d\times d_{reduced}$ as $T$ denotes matrix transpose (you mentioned inverse which is wrong).

NOTE: Depending on how you arrange samples in your data matrix, the matrix product will be totally different. Do not get confused if you see different things in literature. The standard form of data is usually $n_{samples}\times n_{features}$ which was assumed above as well. Each row is a sample and each column is a dimension.

I hope it helped. You can comment if you had any questions.


How to download COCO dataset images? I'm trying to download the COCO dataset images using the following COCO API command:

from pycocotools.coco import COCO
import requests
catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

... but getting the following error message. Any idea why is it so?

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-9cc4e2f62a0e> in <module>
----> 1 catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

TypeError: getCatIds() missing 1 required positional argument: 'self'

Previously I configured pycocotools using the following instruction:
https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/62381

COCO is a python class and getCatIds is not a Static Method, tho can only be called by an instance/object of the Class COCO and not from the class itself.

You can probably solve it by doing this instead:

a = COCO() # calling init
catIds = a.getCatIds(catNms=['person','dog', 'car']) # calling the method from the class

(answer based on reading the code from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py)


Correctly interpreting Cosine Angular Distance Similarity & Euclidean Distance Similarity As an example, let's say I have a very simple data set. I am given a csv with three columns, user_id, book_id, rating. The rating can be any number 0-5, where 0 means the user has NOT rated the book.

Let's say I randomly pick three users, and I get these feature/rating vectors.

Martin: $<3,3,5,1,2,3,2,2,5>$

Jacob: $<3,3,5,0,0,0,0,0,0>$

Grant: $<1,1,1,2,2,2,2,2,2>$

The similarity calculations:

+--------------+---------+---------+----------+
|              | M & J   | M & G   | J & G    |
+--------------+---------+---------+----------+
| Euclidean    |   6.85  |    5.91 |    6.92  |
+--------------+---------+---------+----------+
| Cosine       |    .69  |    .83  |      .32 |
+--------------+---------+---------+----------+

Now, my expectation of similarity is that Martin and Jacob would be the most similar. I would expect this because they have EXACTLY the same ratings for the books that both of them have rated. But we end up finding that Martin and Grant are the most similar.

I understand mathematically how we get to this conclusion, but I don't understand how I can rely on Cosine Angular distance or Euclidean distance as a means of calculating similarity, if this type of thing occurs. For what interpretation are Martin and Grant more similar than Martin and Jacob?

One thought I had was to just calculate Euclidean distance, but ignore all books for which one user hasn't rated the book.

I then end up with this

+--------------+---------+---------+----------+
|              | M & J   | M & G   | J & G    |
+--------------+---------+---------+----------+
| Euclidean    |   0     |    5.91 |    6.92  |
+--------------+---------+---------+----------+
| Cosine       |    .69  |    .83  |      .32 |
+--------------+---------+---------+----------+

Of course now I have a Euclidean distance of 0, which fits what I would expect of the recommender system. I see many tutorials and lectures use Cosine Angular distance to ignore the unrated books, rather than use Euclidean and ignore them, so I believe this must not work in general.

EDIT:

Just to experiment a little, I adjusted Jacob's feature vector to be much more similar:

Jacob: $<3,3,5,1,2,3,2,0,0>$

When I calculate Cosine Angular distance with Martin, I still only get .82! Still less similar than Martin and Grant, yet by inspection I would expect these two to be very similar.

Could somebody help explain where my thinking is wrong, and possibly suggest another similarity measure?

If you look at the definitions of the two distances, cosine distance is the normalized dot product of the two vectors and euclidian is the square root of the sum of the squared elements of the difference vector.

The cosine distance between M and J is smaller than between M and G because the normalization factor of M's vector still includes the numbers for which J didn't have any ratings. Even if you make J's vector more similar, like you did, the remaining numbers of M (2 and 5) get you the number you get.
The number for M and G is this high because they both have non-zeroes for all the books. Even though they seem quite different, the normalization factors in the cosine are more "neutralized" by the non-zeroes for corresponding entries in the dot product. Maths don't lie.

The books J didn't rate will be ignored if you make their numbers zero in the computation of the normalization factor for M. Maybe the fault in your thinking is that the books J didn't rate should be 0 while they shouldn't be any number.

Finally, for recommendation systems, I would like to refer to matrix factorization.


What is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis? I am a student and I am studying machine learning. I am focusing on the concept of Bayesian learning and I have studied the maximum likelihood hypothesis and the maximum a posteriori hypothesis.
I have seen that the maximum likelihood hypothesis  is the hyposesis that maximizes the likelihood of seeng the data, and it is defined as:
$h_{ML}=arg_h max P(D|h)$
while maximum a posteriori hypothesis is the hypothesis that maximizes the posterir probability of seeng the data, and it is defined as:
$h_{MAP}=arg_h max P(D|h)P(h)$
I am really confused by these two definitions, since I can't grasp what is the difference between the two.
I have understood that the maximum likelihood hypothesis is the one that, given some observed data, finds the parameters of the distribution such that I am most likly to understand the data.
But I cannot unserstand what is the MAP hypothesis.
I have tried to read some interpretations and definitions, mìbut I can't understand the difference between the two.
So, what is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis?

I'll try and provide some intuition for you here, instead of focusing on the mechanics of the math behind the methods.
Imagine you are evaluating whether a coin is fair or not, so you collect a sequence of heads and tails as your data set. In MLE, we simply look at the data we collected and find the maximum likelihood... this works well when we have no prior knowledge to leverage (i.e. we have no idea if the coin is fair or not).
By contrast, in MAP we take the same likelihood we used in MLE but now multiply by our prior knowledge. For instance, we may strongly suspect that our coin is biased and so we can influence our estimate with that knowledge through a prior distribution. This new estimate is a mixture of what we believe (our prior) and what we measured (our likelihood).
Thinking of two extreme cases here would be 1) if we very strongly believe in our prior then we would need to collect a lot of data to influence the resulting estimate away from the prior. Conversely, if we know very little up front (i.e. we have an uninformative prior) then finding the MAP estimate is equivalent to the MLE estimate because our prior did not influence the result.
Perhaps a visual representation can help too:

MLE is finding the maximum of the green curve.
MAP is 1) multiplying the blue curve by the green curve to create the red curve and 2) finding the maximum of the newly created red curve.


Pytorch dynamic forward pass Does there exist a fast and convenient way for handling such a problem:

class MyModule(nn.Module):
def __init__(self):
    super(MyModule, self).__init__()
    self.linears = nn.ModuleList([nn.Linear(10, 20) for _ in range(10)])

def forward(self, x, indices):
    x = self.linears[indices](x)
    return x

You see i want to access different layers in the network conditioned on an additional input, which is also a list. Further i want to process the whole batch at once and the output.shape != input.shape.

Here is my understanding of your problem:

# Import
from torch import nn

# Define custom class
class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 20) for _ in range(10)])

    def forward(self, x, indices):
        x = self.linears[indices](x)
        return x

# Intialize custom class
net = MyModule()

# Access networks layers based on additional input
additional_input = 1
if additional_input == 1:
    idx = 0
    print(net.linears[idx].in_features)


ML Approach for Graph Anomaly Detection Very new to ML. I am trying to create an anomaly detector. I have thousands of graphs like the one I have attached. I am interested in the pink line. If the pink line's behavior changes drastically from the overall pattern of the line (i.e. the black circle), the graph needs to be flagged. While I am only interested in the pink line's behavior, I have included the blue line because it seems in every graph when the pink line dips downwards, the blue line dips upward.

I was thinking to use KNN, but I am not sure.

Any tips or guidance would be very appreciated! Thanks :)

It really depends on your data. First, you should also keep in mind that you have to define what is a "drastic change" and if it is just a simple number or threshold then you might not even need an ML algorithm to solve this, you can easily have some conditions to check the values or do it threshold-based.

But, if you want to use an ML algorithm then I would suggest that you take a look at Novelty and Outlier Detection tools provided by Scikit-Learn here:

https://scikit-learn.org/stable/modules/outlier_detection.html

In particular, you can try "Isolation Forest" and "One-class SVM". For me in many cases "Isolation Forest" is the best fit, but try them out and see which one works the best for you.

Let me know if this is helpful.


Best Python library for statistical inference I'm curious if anyone has Python library suggestions for inferential statistics. I'm currently reading An Introduction to Statistical Learning, which uses R for the example code, but ideally I'd like to use Python as well.

Most of my data experience is with Pandas, Matplotlib, and Sklearn doing predictive modeling.

So far I've found statsmodels. Is this what is recommended or is there something else?

Thanks!

statsmodels is a good, and fairly standard, package to statistics.

For Bayesian interference you can go with PyMC - see as in Cam Davidson-Pilon, Probabilistic Programming & Bayesian Methods for Hackers.


predict the probability of buying a product I have a huge dataset with 3 variables Company_ID, Area_code, Product_ID each one of them is a categorical variable of levels 1500,50,15 respectively,where Product_ID is the product the Company_ID is using.Each company can be located in more than one area_code and can be using more than one product.Lets call this train data.

Given a combination of Company_ID, Area_code, Product_ID ,
example- comp_025, area_012, p_10 that is not present in train data,i have to predict the probability that a company Company_ID located in area Area_Code uses the product Product_ID. So that i can recommend that product to company.

I am a beginner and having difficulty in coming up with a model. Any input is highly appreciated.

Thank you.

Seems like Naive Bayes of e1071 package could help.

Given the model product_id ~ comp_id + area_id it will construct conditional probabilities that may reveal some pattern in your data. For specifics, however, some sample data is nescessary.


XGboost classification with very small data set I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets.

Normally I tune the n_rounds parameters by cross-validation, but what if you have too less observations to do proper CV? For example if I have 30 variables and 4000 observations in my training data, how can I find a nice value for n_round which is not over/underfitting the training data?

Are there any "best practices" for parameter tuning (also max_depth etc.) having small datasets?

You can use grid search or xgb.cv for find the best iteration. Run xgb.cv for example 500 trees and add early stopping criteria. Then you can use the best iteration for xgb.train. Have a look at these links.

Hypertuning XGBoost parameters

https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees


Is this a problem for a Seq2Seq model? I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation.

I have a multivariate dataset with n number of input variables each composed of sequences, and a single output sequence which is unrelated to any of the input variables (e.g. using weekly wind speed and humidity to predict temperature).

I've converted the features and label to batches with fixed time steps and n dimensions, however I'm confused which model should be used. Ideally the output should be a single sequence (e.g. annual temperature) however which model would achieve this? Is this something an LSTM could achieve, or is this a problem for a seq2seq model?

Any suggestions/insightful would be appreciated.

The type of your problem is time series regression. The common way of dealing with such problems is using ARMA models and Kalman filters with input (control vector). Look at this links:

https://www.sas.upenn.edu/~fdiebold/Teaching104/Ch14_slides.pdf

https://en.m.wikipedia.org/wiki/Kalman_filter

For time series regression using LSTM, check
Multivariate LSTM Forecast Model part from this link:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras


How should a neural network for unbound function approximation be structured? I've heard that a multilayer perceptron can approximate any function arbitrarily exact, given enough neurons. I wanted to try it, so I wrote the following code:

#!/usr/bin/env python

"""Example for learning a regression."""

import tensorflow as tf
import numpy

def plot(xs, ys_truth, ys_pred):
    """
    Plot the true values and the predicted values.

    Parameters
    ----------
    xs : list
        Numeric values
    ys_truth : list
        Numeric values, same length as `xs`
    ys_pred : list
        Numeric values, same length as `xs`
    """
    import matplotlib.pyplot as plt
    truth_plot, = plt.plot(xs, ys_truth, '-o', color='#00ff00')
    pred_plot, = plt.plot(xs, ys_pred, '-o', color='#ff0000')
    plt.legend([truth_plot, pred_plot],
               ['Truth', 'Prediction'],
               loc='upper center')
    plt.savefig('plot.png')

# Parameters
learning_rate = 0.1
momentum = 0.6
training_epochs = 1000
display_step = 100

# Generate training data
train_X = []
train_Y = []

# First simple test: a linear function
f = lambda x: x+4

# Second, more complicated test: x^2
# f = lambda x: x**2

for x in range(-20, 20):
    train_X.append(float(x))
    train_Y.append(f(x))
train_X = numpy.asarray(train_X)
train_Y = numpy.asarray(train_Y)
n_samples = train_X.shape[0]

# Graph input
X = tf.placeholder(tf.float32)
reshaped_X = tf.reshape(X, [-1, 1])
Y = tf.placeholder("float")

# Create Model
W1 = tf.Variable(tf.truncated_normal([1, 100], stddev=0.1), name="weight")
b1 = tf.Variable(tf.constant(0.1, shape=[1, 100]), name="bias")
mul = tf.matmul(reshaped_X, W1)
h1 = tf.nn.sigmoid(mul) + b1
W2 = tf.Variable(tf.truncated_normal([100, 100], stddev=0.1), name="weight")
b2 = tf.Variable(tf.constant(0.1, shape=[100]), name="bias")
h2 = tf.nn.sigmoid(tf.matmul(h1, W2)) + b2
W3 = tf.Variable(tf.truncated_normal([100, 1], stddev=0.1), name="weight")
b3 = tf.Variable(tf.constant(0.1, shape=[1]), name="bias")
# identity as activation to get arbitrary output
activation = tf.matmul(h2, W3) + b3

# Minimize the squared errors
l2_loss = tf.reduce_sum(tf.pow(activation-Y, 2))/(2*n_samples)
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum).minimize(l2_loss)

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})

        # Display logs per epoch step
        if epoch % display_step == 0:
            cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})
            print("cost=%s\nW1=%s" % (cost, sess.run(W1)))

    print("Optimization Finished!")
    print("cost=%s W1=%s" %
          (sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y}),
           sess.run(W1)))  # "b2=", sess.run(b2)

    # Get output and plot it
    ys_pred = []
    ys_truth = []

    test_X = []
    for x in range(-40, 40):
        test_X.append(float(x))

    for x in test_X:
        ret = sess.run(activation, feed_dict={X: x})
        ys_pred.append(list(ret)[0][0])
        ys_truth.append(f(x))
    plot(train_X.tolist(), ys_truth, ys_pred)

This kind of works for linear functions (at least for the training data, not so much for the testing data outside of the range):

However, it doesn't work at all for non-linear function $x^2$:

Why does this neural network not work for such simple function approximation? What do I have to change to make the same network topology work for both functions?

This does not answere your question directly, but might include some helpful pointers:
In the recent Deep Residual Learning for Image Recognition paper it is written:

If one hypothesizes that multiple nonlinear layers can asymptotically
approximate complicated functions[...]
This hypothesis, however, is still an open question. See [28].


Use 1 or 2 norm for Voronoi vector quantization? I have a script from a lecture. Basically it says that based on the Voronoi partitioning we identify the corresponding (nearest) class $w_k$ to a vector $x$ where $\left| {{w_k} - x} \right| = \mathop {\min }\limits_i \left( {\left| {{w_i} - x} \right|} \right)$ given the classes $w$.

The script uses the absolute value notation. This does not make much sense as we are using vectors. Which vector norms can or should be used? The 1 or 2 norm?
The 1 norm is faster to compute but maybe there are drawbacks I cannot currently think of.

In general the quality of the quantization is measured using the mean squared error (MSE) between the input vector $x$ and the reproduction value $q(x)$ ($w_k$ in the notation you used in the question). For the MSE the best partition is the one defined by Voronoi regions under the euclidean distance ($l^2$ norm), i.e:
$$V_j = \{ x \in R^d : \|x - w_j \|^2 \leq \|x - w_i \|^2\ for\ all\ i   \}$$

You can find the proof in here. Also here and here the euclidean distance is used.


features' range in logistic regression I use logistic regression. We know that it is a supervised method and needs calculated feature values both in training and test data. There are six features. Although the functions produce these features’ values are different and their maximum value can be 1, there are four features (both in training and test data) that have very low values. e.g. they range between 0 and 0.1 and are never 1, even more than 0.1!!!. Thus these features’ values are very close to each other. Other features are distributed normally (they range between 0 and 0.9). So the difference between these two kinds of features is high, I think this causes trouble in learning process for logistic regression. Am I right?! Does it need any transforming/normalizing these features? Any help would be highly appreciated.

You do not have to normalise features used in logistic regression, but sometimes it can help.

You should normalise features used in logistic regression, if you are using a gradient-based optimiser (e.g. SGD) to find the optimum weights. That is because the optimiser will perform better when partial derivatives of the cost function are of similar magnitude in each direction. When the derivatives vary too much, you will need a lower learning rate to compensate (making learning slower, and more likely to get stuck) or the optimiser will not converge - it may oscillate or start to diverge instead.


How to set limits of Y-axes in countplot? df in my program happens to be a dataframe with these columns :

df.columns
'''output : Index(['lat', 'lng', 'desc', 'zip', 'title', 'timeStamp', 'twp', 'addr', 'e',
'reason'],
 dtype='object')'''

When I execute this piece of code:

sns.countplot(x = df['reason'], data=df)
# output is the plot below

but if i slightly tweak my code like this :

p = df['reason'].value_counts()
k = pd.DataFrame({'causes':p.index,'freq':p.values})
sns.countplot(x = k['causes'], data = k)

So essentially I just stored the 'reasons' column values and its frequencies as a series in p and then converted them to another dataframe k but this new countplot doesn't have the right range of Y-axis for the given values.

My doubts happen to be :

Can we set of Y-axis in the second countplot in its appropriate limits
Why the does second countplot differ from the first one when i just separated the specific column i wanted to graph and plotted it separately ?

Countplot from seaborn will not work as you expect. When you calculate the frequencies, you want to plot the values in p.values as they appear. Countplot will take a dataframe where labels are not aggregated and then count each one of them, as it did in the first case.

So countplot will be appropriate for the case where your dataframe looks like:

index | reason |
0        EMS
1        EMS
2        Traffic
3        Fire
4        Fire
5        EMS
6        Traffic
...

In the second case you already have your frequencies:

index | reason |
EMS       10
Traffic   21
Fire      15

Then count plot will just count the lines and it will be one for each, that is why your plot looks like that.

To solve your problem you could just plot using .plot from pandas:

df['reason'].value_counts(normalize=True).plot(kind='bar')

Where the parameter normalize=True will show normalized frequencies instead of raw count values.


NEAT XOR Example gives different results at each run I have just started learn to use NEAT algorithm. I was thinking I understood the basics of NEAT when I read "Evolving Neural Networks through
Augmenting Topologies" paper and current python documentation. However, in practice I m in trouble.

I have run XOR examples from source (https://github.com/CodeReclaimers/neat-python.git). However, when I run any of "evolve-feedforward.py",  "evolve-feedforward-parallel.py" or "evolve-minimal" codes more than once, results are  changing. For instance,

When I run "evolve-feeedforward.py" it gives : 1

when I run same code for second time, it gives: 2

and it changes with each run. I have also deleted generated neat-checkpoint files before each run, but structure still changes. I would like to understand reason what cause this, I think I m missing some essential points here.

Provided each network implements the XOR function approximately, but within reasonable error bounds, then this is NEAT behaving as designed.

Neural networks allow for multiple equivalent solutions. Even using a fixed architecture and stochastic gradient descent to learn a simple function, there is a good chance of ending up with very different weights each time. With NEAT, it also explores alternative architectures, and the search includes many calls to random number generators in order to make decisions.

The main deterministic step in NEAT is the selection process - comparing population members with each other to rank them in terms of fitness. However, as multiple designs of neural network can all solve the XOR problem roughly equally well, it is unlikely this will find the same one each time.

If you need to exactly reproduce an experiment in NEAT, you can achieve that by setting the RNG seed. I think this could just be a call to random.seed() for the library you are using, but some libraries may have other RNG instances that need setting up too.


How to install tensorflow-gpu? I installed cuda-11.0 and cuDNN-8.2 and Tensor-RT 7.1. And then installed tensorflow-gpu. But tf.is_gpu_available() is showing False.
Actually, I installed tensorflow-gpu first. I have pip3 20.1.1 and Nvidia GForce 930MX and Ubuntu 20.04.
Here is the nvidia-smi command output:

Right now Tensorflow 2.1 doesn't support CUDA 11.0. Downgrade CUDA to version 10.1 and try again.
Reference: https://www.tensorflow.org/install/gpu


Is Maxout the same as max pooling? I've recently read about maxout in slides of a lecture and in the paper.

Is maxout the same as max pooling?

They are almost identical:

  The second key reason that maxout performs well is
  that it improves the bagging style training phase of
  dropout. Note that the arguments in section 7 motivating
  the use of maxout also apply equally to rectified
  linear units (Salinas & Abbott, 1996; Hahnloser, 1998;
  Glorot et al., 2011). The only difference between maxout
  and max pooling over a set of rectified linear units
  is that maxout does not include a 0 in the max.

Source: Maxout Networks.


Improve a regression model and feature selection I am working on Azure ML Studio and try to create a regression model to predict a numerical value. I will try to describe my features and what I have done until now.

My data with about 3 million rows :

Features:

8 integer features from 1 to 25
2 boolean features with 0 and 1
3 integer features from 1 to 10
2 integer feature from 0 to 500.000 (and 1.000.000 respectively) with about 4.500 unique values
1 integer feature from 20 to 50
1 integer feature from 1 to 15
1 integer feature from 0 to 100

Label:

Integer from 10.000 to 100.000.000 with about 5.000 unique values

What I have done:

Split the dataset to 80% (train) and 20% (test). Then I split the training dataset again to 60% (actual train) and 40% (validation).
Normalize the features with many unique values (4th bullet in the above list)
Train a model of Boosted Decision Tree Regression.
Use the Sweep Parameters module to find the best combination

I tried also Neural Networks, Bayesian Linear Regression, but BDTR gave the best score.

I tried to exclude columns and start with only a few (based on what I think it will affect the model) and then add more columns one by one.

However, the least MSE I could achieved was 1.500.000 (plus I had many negative scored values)

So, I was thinking what other techniques I could use to improve the model.

I think the next option you have to take is to add more features. You have a huge amount of training examples, which is good, but the number of features is very low. Adding more features is one of the most used methods to improve performance in machine learning.

Furthermore, it would be good to try to understand how your features affect your model. Imagine you have a linear model like y = theta1*feat1 + theta2*feat2 + theta3*feat3. If theta3 is near to 0, then feat3 is not affecting the model.


Processing data stored in Redshift We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data involved, ideally I'd want to run the computation in the same location as the data rather than shifting the data around, but this doesn't seem possible with Redshift. I've looked at MADlib, but this is not an option as Redshift does not support UDFs (which MADlib requires). I'm currently looking at shifting the data over to EMR and processing it with the Apache Spark machine learning library (or maybe H20, or Mahout, or whatever). So my questions are:

is there a better way?
if not, how should I make the data accessible to Spark? The options I've identified so far include: use Sqoop to load it into HDFS, use DBInputFormat, do a Redshift export to S3 and have Spark grab it from there. What are the pros/cons for these different approaches (and any others) when using Spark?

Note that this is off-line batch learning, but we'd like to be able to do this as quickly as possible so that we can iterate experiments quickly.

The new Amazon Machine Learning Service may work for you. It works directly with Redshift and might be a good way to start. http://aws.amazon.com/machine-learning/

If you're looking to process using EMR, then you can use Redshift's UNLOAD command to land data on S3. Spark on EMR can then access it directly without you having to pull it into HDFS.

Spark on EMR: https://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923


Guidelines to debug REINFORCE-type algorithms? I implemented a self-critical policy gradient (as described here), for text summarization.

However, after training, the results are not as high as expected (actually lower than without RL...).

I'm looking for general guidelines on how to debug RL-based algorithms.

I tried :

Overfitting on small datasets (~6 samples) : I could increase the average reward , but it does not converge. Sometimes the average reward would go down again.
Changing the learning rate : I changed the learning rate and see its effect on small dataset. From my experiment I choose quite big learning rate (0.02 vs 1e-4 in the paper)
Looking at how average reward evolve as training (on full dataset) goes : Average reward significantly does not move...

The only resource I could find so far :

https://github.com/williamFalcon/DeepRLHacks

For my specific case, I made a few errors :

Frozen some part of the network that shouldn't be frozen
Wrong learning rate

Even if I could overfit a small dataset, it didn't mean anything : while training on the whole dataset, the average reward was not going up.

You should look for a reward going up.

I'm not accepting this answer as I believe it is not complete : it lacks general and systematic guidelines to debug a Reinforcement Learning algorithm.


Word2Vec how to treat words that aren't in the vocabulary I'm trying to assign a vector to every word in my sentence.
Some words are not recognised even though very similar words are - for example:
Going, gone, go are recognised while goes isn't.

How should I go about assigning any logical values to the word goes or any word of this like? Please notice that I don't know in advance what words won't be recognised.

The words Going, Gone, Goes are considered to be similar in only one context i.e. they all have the same root word Go. This is known as Stemming/Lemmatization, both of which are special cases of normalization process. So, in order to assign some value to these words, you have to first lemmatize them. In order to get more details about this, I am attaching two links which can be helpful in your case:

http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization

https://stackoverflow.com/questions/771918/how-do-i-do-word-stemming-or-lemmatization


Why is my loss increasing in gradient descent? When the learning rate is 0.01 the loss seems to be decreasing whereas when I increase the learning rate even slightly, the loss increases. Why does this happen? Are the gradients calculated wrong?

Neural Network with 2 hidden layers ,128 neurons in the first hidden layers and 64 in the second hidden layer. The output layer consists of a single sigmoid Neuron

class FNN:
    def __init__(self):
        self.W1=None
        self.b1=None
        self.W2=None
        self.b2=None
        self.W3=None
        self.b3=None
    def sigmoid(self,x):
        return 1/(1+np.exp(-x))
    def forward_prop(self,x):
        self.Z1=np.dot(self.W1,x)+self.b1
        self.A1=np.tanh(self.Z1)
        self.Z2=np.dot(self.W2,self.A1)+self.b2
        self.A2=np.tanh(self.Z2)
        self.Z3=np.dot(self.W3,self.A2)+self.b3
        self.A3=self.sigmoid(self.Z3)
        return self.A3
    def back_prop(self,x,y):
        self.forward_prop(x)
        m=x.shape[1]
        self.dZ3=self.A3-y
        self.dW3=np.dot(self.dZ3,self.A2.T)/m
        self.db3=np.sum(self.dZ3,axis=1,keepdims=True)/m
        self.dZ2=np.dot(self.W3.T,self.dZ3)*(1-self.A2**2)
        self.dW2=np.dot(self.dZ2,self.A1.T)/m
        self.db2=np.sum(self.dZ2,axis=1,keepdims=True)/m
        self.dZ1=np.dot(self.W2.T,self.dZ2)*(1-self.A1**2)
        self.dW1=np.dot(self.dZ1,x.T)/m
        self.db1=np.sum(self.dZ1,keepdims=True)/m
    def fit(self,x,y,epochs=100,learning_rate=0.01,plot=True,disp_loss=False):
        np.random.seed(4)
        self.W1=np.random.rand(128,x.shape[0])
        self.b1=np.zeros((128,1))
        self.W2=np.random.randn(64,128)
        self.b2=np.zeros((64,1))
        self.W3=np.random.randn(1,64)
        self.b3=np.zeros((1,1))
        m=x.shape[1]
        loss=[]
        for i in range(epochs):
            self.back_prop(x,y)
            self.W1-=learning_rate*self.dW1
            self.b1-=learning_rate*self.db1
            self.W2-=learning_rate*self.dW2
            self.b2-=learning_rate*self.db2
            self.W3-=learning_rate*self.dW3
            self.b3-=learning_rate*self.db3
            logprobs=y*np.log(self.A3)+(1-y)*np.log(1-self.A3)
            cost=-(np.sum(logprobs))/m
            loss.append(cost)
        e=np.arange(1,epochs+1)
        if plot:
            plt.plot(e,loss)
            plt.title('LOSS PLOT')
            plt.xlabel('Epoch')
            plt.ylabel('Loss')
            plt.show()
        if disp_loss:
            print(loss)
    def predict(self,x):
        y=np.where(self.forward_prop(x)>=0.5,1,0)
        return y

F=FNN()
F.fit(x_train,y_train)
y_pred=F.predict(x_train)

Output

Learning Rate:0.01

Learning Rate:1

Based on your plots, it doesn't seem to be a problem in your case (see my comment). The reason behind that spike when you increase the learning rate is very likely due to the following.

Gradient descent can be simplified using the image below.

Your goal is to reach the bottom of the bowl (the optimum) and you use your gradients to know in which direction to go (in this simplistic case, should you go left or right). The gradient tells you in which direction to go, and you can view your learning rate as the "speed" at which you move. If your learning rate is too small, it can slow down the training. If your learning rate is too high, you might go in the right direction, but go too far and end up in a higher position in the bowl than previously. That's called diverging.

Also, good to note that it could be completely normal that your loss doesn't always decrease. This is particularly true if you use mini-batch gradient descent. In that scenario, your gradient may not always be completely accurate, and you might simply make a step in the wrong direction every once in a while.

I hope this explanation helps!


Studying machine learning algorithms: depth of understanding vs. number of algorithms Recently I was introduced to the field of Data Science (its been 6 months approx), and Ii started the journey with Machine Learning Course by Andrew Ng and post that started working on the Data Science Specialization by JHU.

On practical application front, I have been working on building a predictive model that would predict attrition. So far I have used glm, bayesglm, rf in an effort to learn and apply these methods, but I find a lot of gap in my understanding of these algorithms.

My basic dilemma is:

Whether I should focus more on learning the intricacies of a few algorithms or should I use the approach of knowing a lot of them as and when and as much as required?

Please guide me in the right direction, maybe by suggesting books or articles or anything that you think would help.

I would be grateful if you would reply with an idea of guiding someone who has just started his career in the field of Data Science, and wants to be a person who solves practical issues for the business world.

I would read (as many as possible) resources (books,articles) suggested in this post and would provide a personal feed back on the pros and cons of the same so as to make this a helpful post for people who come across a similar question in future,and i think it would be great if people suggesting these books can do the same.

I would recommend limiting yourself to a few tried and trusted algorithms. I would not recommend Elements of statistical learning ( as a first book).  It is too theoretical, aimed  at graduate students, with exercises asking how to prove X or Y...  I think ISL is more appropriate, with more practical advice ( in any case both books are free as pdf downloads).

Besides statistics, I would make sure you are comfortable with experimental design/AB tests, and with Business Intelligence/Visualisation.


How to derive the sum-of squares error function formula? I'm attending a Machine Learning course and I'm studying linear models for classification right now. Slides present approaches to learn linear discriminants (Least squares, Fisher's linear discriminant, Perceptron and SVM), more specifically, how to compute the weight matrix $\tilde{\textbf{W}}$ to determine the discriminant function:

\begin{equation}y = \tilde{\textbf{W}}^T \tilde{\textbf{x}} + w_0. \end{equation}

My problem is about least squares: I don't understand how the minimization of sum-of-squares error function:

\begin{equation}E(\tilde{\textbf{W}}) = \frac{1}{2} Tr\Bigl\{(\tilde{\textbf{X}}\tilde{\textbf{W}} - \textbf{T})^T(\tilde{\textbf{X}}\tilde{\textbf{W}} - \textbf{T})\Bigr\} \end{equation} (where $Tr$ is the trace).

is derived and how it is possible to reach the closed formula solution:

\begin{equation}\tilde{\textbf{W}} = (\tilde{\textbf{X}}^T\tilde{\textbf{X}})^{-1}\tilde{\textbf{X}}^T\textbf{T}\end{equation}

Can someone explain me the main steps in the simplest and clearest possible way to make sense of these formulas? I'm a beginner.

P.S. These formulas come from C.Bishop. Pattern Recognition and Machine Learning.

There are two interpretations of this formula that I explain one of them.

\begin{equation} Xw = y \end{equation}

\begin{equation} X^tXw = X^ty \end{equation}

The above is for making sure that you make a square matrix that it has an inverse. It is possible that $X^tX$ does not have any inverse but its chance for linear regression problems is not that much. The reason is that you have a matrix $X$ which belongs to $R^{m \times n}$ which $m$ represents the number of samples and $n$ represents the number of features. Usually, the number of samples is much more than the number of features. Next,

\begin{equation}(X^tX)^{-1}(X^tX)w = (X^tX)^{-1}X^ty\end{equation}

\begin{equation}w = (X^tX)^{-1}X^ty\end{equation}

Consequently, you have found a closed form for the $w$ linear regression problem which can be generalised to non-linear regression.

Be aware that $(X^tX)^{-1}X^t$ is called the pseudo-inverse of the matrix $X$. The reason is that $X$ is not a square matrix and it does not have inverse but by the mentioned formula you can find its pseudo-inverse. Just multiply it by $X$ and you will get Identity matrix.

There is another interpretation of this. You can find here.


RMSProp - To a layman? Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things:

1) Why is is different from SGD with momentum?
2) Why does it actually work?

The only literature I could find were the slides of the professor who wrote this optimization technique and the slides didn't make a lot of sense to me.

Thanks

What momentum does is to average out the derivatives in different directions in parameter space so that the directions that oscillate much will be close to zero, but for the parameters that do not oscillate this would not be the case - averaging will not vanish.

On the other hand RMSprop works as a dampener in all directions: larger the derivatives, larger the dampening rate. This means that the fast oscillatory modes with large derivatives will be dampened more. The learning in slow directions which you want to keep may also be dampened, but the rate will be smaller.

You can find more details on RMSprop and their combination Adam on the following Coursera course:

https://www.coursera.org/learn/deep-neural-network/lecture/y0m1f/gradient-descent-with-momentum


Signal classification with convolution neural network I want to classify 8 noisy signals with trainNetwork() function that is available in Matlab.  All Matlab examples were for image classification. The arguments are the data and its label, the CNN layers, in which the first layer is imageInputLayer([32 32 3]).
My question is can this function be used to classify 1-D signals? if yes, how can we create such a layer?

A CNN usually requires a lot of data. And of course even more data if you have  a very large feature set. And even more data if you have many output classes. The number of output classes makes the necessary amount of data samples to increase exponentially.

The most important part of any machine learning project is to extract the most telling features from your raw data. Ask yourself in your signal what are the telltale signs that should tell your model what class the signal was drawn from.

You should try to use some feature extraction techniques, then you should try to use a model better suited for the size of your dataset.

Data pre-processing and feature extraction

Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. n your case if you have a signal you might want to extract some information about the temporal and frequency domain transforms. You can try something like empirical mode decomposition (EMD).

Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.

In your case

As a general rule of thumb, the amount of data that I suggest to have for shallow machine learning models is $10 \times \#features$. For deep models I usually suggest $100 \times \#features \times \#outputs$. So you are limited by the size of your dataset. In your case a CNN is not recommended. Only recently a group published a quite controversial result of training a CNN with 100,000 examples. The community generally believes that this is way too little and it is not feasible to expect good results when training a CNN with very little data.

The Model

I suggest you use a general machine learning technique such as SVM. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.


weather and products sell relationship which one should be the target variable? Let say I have this dataset

kor_cat is the categories of the product like(noodle,cookie....)
 and the kor_qty is the number of product selled

I want to know the weather has a relation with the store's sale record

but I want to know which machine learning technique is fit here
and I have the categori of the product and number of selled of that product.

How should I solve this problem with ML.

Apparently, you have a target variable (y) and a set of input variables (input variables (X). This seems like a supervised problem. A regression analysis can be a solution to implement.

Feature engineering needs to be completed before running a machine learning model. Such as text field columns can be encoded with Label Encoder and One Hot Encoder.

Irrelevant input columns for analysis can be dropped. If the city variable is same all the way, meaning the all of the data is from a single city. Also N/A and null values need to be cleaned.

In some models, normalization of the data can produce better results.

As a Machine Learning model, starting from Linear Regression would be feasible. Later Random Forest Regressor, Lasso, Ridge Regression can be examined.


Calculating correlation of slightly out of sync data I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I  would like to calculate the correlation between two glasses on two different people. At the moment I cannot use df.correlate() because the timestamps are not identical and therefore the data looks something like this:

index | ts | r_person | l_person
-----------------------
0     | 23 | 3.0      | NAN
1     | 25 | NAN      | 3.2
2     | 28 | 3.1      | NAN
3     | 32 | 3.0      | NAN

I was wondering if there was still any way to calculate a correlation directly.

At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.

This would be less trivial than it seems because even at the start it wasn't always one data point R one data point L and after cleaning the data it has become less so. In other words multiple NAN values might appear in the same column as you can see in the example. I can still deal with that by just spreading out the average. My second technique was going to try and merge the values together if they were close enough. Bearing in mind the data was collected at 50hz.

My question is whether anyone has a quicker or better way of aligning the data without losing it or changing it too much?

At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.

The problem here is that the missing data point isn't necessarily associated with a timestamp that is the average of the timestamps above and below. If it were, the procedure you suggested would be equivalent to a linear interpolation, i.e. drawing a line between the points before and after the missing timestamp, and using that line to construct a local prediction for the missing observation. Which is what you should actually do instead.

For example in the case above, the line for the point your trying to interpolate is $\text{r_person}(ts) = 3.0 +\frac{.1}{5}(ts-23)$, so $\text{r_person}(25) = 3.04$ (I think... doing this in my head right now).


How to deal with unbalanced data in pixelwise classification? I'm trying to train a fully convolutional network to generate binary image showing roads from satellite photos. For a long time, I struggled to get the network to output anything but black - then I figured out the problem. I have plenty of training data, but of course most of the output pixels
will be black, since roads take only small percentage of space.

I've created a toy example showing the problem:

from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D
import numpy as np
import random
import sklearn.model_selection

THRESHOLD = 3
BALANCED = False

if BALANCED:
    X = [[
            random.randint(THRESHOLD + 1, 255) if
                random.randint(0, 1) else
            random.randint(0, THRESHOLD)
            for _ in range(10)] for _ in range(10**4)]
    Y = [[0] if x[0] <= THRESHOLD else [1] for x in X]
else:
    X = [[random.randint(0, 255) for _ in range(10)] for _ in range(10**4)]
    Y = [[0] if x[0] <= THRESHOLD else [1] for x in X]

X, Y = np.array(X), np.array(Y)
X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(
        X, Y, test_size = 0.2)

model = Sequential()
model.add(Dense(8, activation = "sigmoid", input_shape = (10, )))
model.add(Dense(1, activation = "sigmoid"))

model.compile(loss = "binary_crossentropy", optimizer = "adam")

model.summary()
model.fit(X_train, Y_train,
        batch_size = 10, epochs = 10,
        verbose = 2,
        validation_data = (X_test, Y_test))

Y_res = model.predict(X_test)
Ys0 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 0]
Ys1 = [y[0] for x, y, y_opt in zip(X_test, Y_res, Y_test) if y_opt[0] == 1]

print(np.mean(Ys0))
print(np.mean(Ys1))

Here, I artificially create data, with X being ten random bytes, and Y being boolean thresholding first byte on the value of three. That means normally only around 1/64 of all samples will be generated with y equal to 0, similarly to my original problem. There's also a second dataset, with the same decision boundary, but more evenly (class-wise) distributed data - set by changing BALANCED toggle in the code. The results confirm my suspicions.

For balanced set, actual 0's are given, on average, value of 0.025. 1's - 0.97. That's reasonable. On the unbalanced, raw dataset, 0's have mean of 0.87, and 1's - 0.98. That means the zeros are pretty much always classified as one anyway. On my real data, the effect is even worse, and the returned value is pretty much equal to percentage of black, regardless of real classification.

I tried adding class_weights = [100, 1] to fit call, but it didn't change much. Is there a good strategy for dealing with such unbalanced data? In this toy example I could probably sample the x-y pairs (e.g. keep only 5% of 1-classified data), but I don't think this would work for my  full problem - since I'm using FCN, I'm putting a whole image into the network at once. There are no images with 50% roads, so I cannot possibly sample images to make roads take enough space.

I also tried, quite randomly, changing loss functions, including using one for categorical data and one-hot encoding expected outputs (i.e. [1, 0] and [0, 1] instead of 0 and 1). This did not fix anything either.

Looks like the problem lied just in poorly chosen parameters. In the example given above, simply increasing epoch count significantly improved the results. In my actual image processing problem, I had to decrease learning rate, since with the default one, ReLU's were saturating very quickly, hindering further progress.


How to select features for Text classification problem I am working on a problem where we need to classify user query into multiple classes.

  Problem:

  Suppose we are running a website for selling products. The website has a form where the user can write any complaint or issue.

  In order to resolve users issue, we thought to classify issues into predefined classes so that we can understand what type of problems users are facing.

Issues classes can be like

Class 1: Payment Issues
Class 2: Registration issues
Class 3: Order booking issues
Class 4: Problem accessing website app
...

I am thinking to apply random forest but cannot decide what features should I select. Any suggestions would be of great help thank you

Since I am not sure where you are at the moment in your solution, let me give you a comprehensive, yet brief, view of what you should be looking to do.

Preparing the training data: You would be required to collect data with the correct classifications. Once you have this, you should incorporate text mining algorithms (bag of words for instance) and end up with a document-term matrix (or term-document matrix, whichever is appropriate). Think of this as a list of all the different words (or word collections) possible in a complaint. These would in turn serve as your features
Training the data: This is where the algorithms you have stated above (RandomForest for instance) come into picture. I would also suggest you to try other boosting algorithms here subject to data availability
Testing on a holdout sample: Test the model accuracy on a holdout sample and finalize on the models you would like to choose. Note that you might also have to tune the text mining algorithm in case you do not get appropriate features for your classification algorithm
Classifying new complaints: Apply the text mining algorithm to the newer data to generate the features as before and run it by the classifier

Hope this gives you a good starting point for building your text classification solution.