How does tensor product/multiplication work in TensorFlow? In Tensorflow, I saw the following example:

import tensorflow as tf
import numpy as np

mat_a = tf.constant(np.arange(1,13, dtype=np.int32), shape=[2,2,3])
mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])
mul_c = tf.matmul(mat_a, mat_b)

with tf.Session() as sess:
   runop = sess.run(mul_c)
   print(runop)

[[[ 88  94]
  [214 229]]
 [[484 508]
  [642 674]]]

How does the tensor multiplication work?

You may want to read the documentation.

output[..., i, j] = sum_k (a[..., i, k] * b[..., k, j]), for all indices i, j.

For instance, in your example

$~~88=1\times12+2\times14+3\times16,~~~94=1\times13+2\times15+3\times17$
$214=4\times12+5\times14+6\times16,~229=4\times13+5\times15+6\times17$


What is the difference between active learning and reinforcement learning? From Wikipedia:

Active learning is a special case of machine learning in which a learning algorithm can interactively query a user (or some other information source) to label new data points with the desired outputs.

Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment in order to maximize the notion of cumulative reward.

How to distinguish them? What are the exact differences?

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system learns to mimic the training data, ideally generalizing it to unseen but extrapolable cases. Active learning is applied normally in cases where obtaining labels is expensive so, we obtain new labels dynamically, defining an algorithmic strategy to maximize the usefulness of the new data points.
Reinforcement learning is a different paradigm, where we don't have labels, and therefore cannot use supervised learning. Instead of labels, we have a "reinforcement signal" that tells us "how good" the current outputs of the system being trained are.  Therefore, in reinforcement learning the system (ideally) learns a strategy to obtain as good rewards as possible.


What is meta- data and meta features? I want to know what is metadata and what is meant by meta features?
When I google Meta Features what I get is feature selection tool called "Meta-Feature". What is the function of feature selection tools ?
Also, what I want is the definition and meaning of the meta features ?

Gramatically "Meta" comes from greek and means beyond something. "Meta-Feature" means beyond the feature.
Normally refers to how the feature was obtained, the uncertainty, error measurements...
For example, in atmospheric modeling, for the features "Temperature" you can have a lot of different meta features like, with which tool are you measuring the temperature? what is the error of the tool? in which conditions is the measurement happening?
I understand the term meta feature as a generalization of this concept but in Data Science.


In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication? The example below is taken from the lectures in deeplearning.ai shows that the result is the sum of the element-by-element product (or "element-wise multiplication".  The red numbers represent the weights in the filter:
$(1*1)+(1*0)+(1*1)+(0*0)+(1*1)+(1*0)+(0*1)+(0*0)+(1*1)
= 1+0+1+0+1+0+0+0+1
= 4 $

HOWEVER, most resources say that it's the dot product that's used:

"…we can re-express the output of the neuron as , where is the bias
term. In other words, we can compute the output by y=f(x*w) where b is
the bias term. In other words, we can compute the output by
performing the dot product of the input and weight vectors, adding in
the bias term to produce the logit, and then applying the
transformation function."

Buduma, Nikhil; Locascio, Nicholas. Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (p. 8). O'Reilly Media. Kindle Edition.

"We take the 553 filter and slide it over the complete image and
along the way take the dot product between the filter and chunks of
the input image.  For every dot product taken, the result is a
scalar."

The best explanation of Convolutional Neural Networks on the Internet!

"Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity."

CS231n Convolutional Neural Networks for Visual Recognition

"The result of a convolution is now equivalent to performing one large
matrix multiply np.dot(W_row, X_col), which evaluates the dot product
between every filter and every receptive field location."

CS231n Convolutional Neural Networks for Visual Recognition
However, when I research how to compute the dot product of matrics, it seems that the dot product is not the same as summing the element-by-element multiplication. What operation is actually used (element-by-element multiplication or the dot product?) and what is the primary difference?

The convolution can be written as

$I(x,y) = \sum_{i=0}^n \sum_{j=0}^m k(i,j) I(x+i, y+j)$

where the kernel $k$ is of size $n$ by $m$. The image is size $N$ by $M$. Ok so in your example we are evaluating $I(0,0)$, our kernel is $n=3$ and $m=3$. Then our image is $N=5$ and $M=5$. If you go through the equation then you will see that you get exactly the result you showed.

$(1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4$

This is a dot product as defined as being a sum of products. We multiply element-wise and then we add them all together. The result is the answer you got $4$.

The entire convolution operation across the image is a matrix where each entry is this dot product.


Why is duplicating inputs bad? I am trying to predict an output value based on several continuously-valued inputs using a regression model.

I am not sure what approach is appropriate to scale/transform the input data for the regression. Let's just pretend that it is unlabeled data.

My most naive approach would be to add each input multiple times:

input
log(input)
sqrt(input)

and then let the regression model worry about finding which flavor of each input (if any) is significant.

What are the risks with using this approach?

The issue with building a regression model on all 3 of these is that you are potentially introducing multicollinearity into the model. Although log(input) and sqrt(input) are not linear functions of the input a quick test (using Matlab) shows they are still highly correlated (depending on the range)

input=rand(1,100);
input_log=log(input);
input_rt=sqrt(input);
corrcoef(input, input_log) %->0.8549
corrcoef(input, input_rt) %->0.9779
corrcoef(input_sq, input_rt) %0>0.9407

Multicollinearity will not reduce the predictive power of the model, but it will make the regression coefficients of these variables difficult to interpret since small adjustments in the data may cause the regression model to switch which of the 3 input variations it finds significant. It also adds unnecessary complexity to the model. I would train the models with the 3 separately if you want to observe the effects.

Additionally, applying things like sqrt and log to your output can be useful if the data currently can't be described by a linear relationship.


What's the best way to train a model when having a very big dataset split in batches? I was wondering what is the best way to train my Keras Model, in a binary classification context.
My full dataset is composed of 2.3M rows and 120 columns, which is really big.  As you would imagine, it doesn't fit into memory. To avoid memory errors, I have split this big dataset into 23 CSVs of 100.000 rows each. I have shuffled the full dataset prior to the splitting, so we have 23 i.i.d. samples.
I have two options:

Take the first CSV, train the model with say 20 epochs (with mini-batches of 256), then take the next CSV, and train the same model with 20 more epochs, and so on and so on 23 times...
Create a Python Generator that yields batches from the 23 CSVs in a way that at the end of the first epoch, the model has seen the whole 2,3M datapoints.

The main difference is the order in which the model sees the data. In both methods, the model will have seen 20 times the whole dataset.

In the first method, the model sees 20 times the same 100.000 datapoints, then moves to the next step.
In the second method, the model sees 1 time the whole dataset, then moves to the next step.

The reason I am asking this is because the first method is way easier to implement. A simple for loop can do the work. The second method requires Python Generators and mini-batch sampling from the already sampled CSVs.
Which of those way is the best to train my model ?

The first solution might give you a lot of headache as some convergence problem may arise from changing the dataset every n iterations. The second one seems the way to go, but before considering a more complex approach with generators, you might want to have some considerations at your memory usage.
Basically memory usage is : number of rows * number of columns * average memory usage. As it happens there are simple solutions to reduce each of those terms independently that often makes wonder as their reduction is multiplicative.
Average memory usage: this one has to be done on each datasets. Basically, working on your datatypes you can reduce the memory usage. This need to be done careffuly so as not to loose too much information. But basically reducing your float types from float64 to float32 already divide your memory usage by a factor of 2. This factor can go up to 8 as you could use small int8 or categories for some variables. (Plus it speeds up calculations, which might end up also giving you statistical performance boost). In the case of multiple csv files it is best to have a dictionnay of types that ensure general coherence.
Number of columns: with Machine Learning there is always the temptation to dump the whole database in the SotA algo and let it sort things out. This is not really a good approach. Usually it is best to start with a simple model (linear / logistic regression), including a smaller set of features. Selecting a subset of pertinent variables means that you will have to do some explanatory analysis. It will help you to handle your problem : having a better understanding of the relationship between features and target, observing problems like outliers and missing values, and being able to explain it. You can even use things like glmnet that do a bit of feature selection for you. With a dozen variables we are talking a factor 10 in memory usage reduction.
Number of rows : when building a model it is often practical to downsample it significantly, like at a 10% rate. This allows you to check you model building step by step without having to actually do all the calculations. Then you can reduce the downsampling. depending on your problem learning on 25% or 50% of your dataset might be more than enough, while giving a memory reduction of a factor 2-4.
Depending on your problem - if there is a variable that mainly drive the underlying behavior - you might want to build multiple models, one for each category of said variable (or buckets of a continuous variable). It's not always intuitive as most Data Scientist will prefer having one model, but sometimes it is more practical to use / handle / explain. And in your case it would allow for handling smaller databases.
All in all, those simple considerations would be enough to reduce your memory usage by a factor of 40-80.


Training a LSTM/any other deep learning model with temporal as well as non temporal attributes I'm working on a project to predict the usage of all the files(rough frequency of usage) in a filesystem(a company server on which 100s of company employees are active) in near future(say the next 1 month) based on the metadata of the file system for past 6 months. I've got the following attributes about the files with me :

The temporal sequence of file usage for last 6 months(whenever the
file was read/written/modified and by whom).
All the users who are on the server and can access the files.
Last modified/written/read epoch time and by whom.
File creation epoch time and by whom.
Any compliance regulations on the file(whether the file contains any
confidential data).
Size, name, extension, version, type of the file.
The number of users who can access the file.
File path.
The total number of times accessed.
Permitted users.

Now, I plan to use LSTM but for standard LSTMs, the input is temporal sequence only. However, all the attributes that I have seem significant in predicting the future usage of the file.

How should I also make use of the attributes of the file that I have?
Should I train a Feedforward Neural Network, disregarding the fact
that it usually fails on temporal sequences?
How should I proceed?
Does a variant of LSTM exist that can take into account the
attributes of the file as well and predict the usage of the file in
near future?
Do I need to use MLP and LSTM together like a hybrid?

Like you said, you can proceed your temporal data with a recurrent network like LSTM. I suggest you simply concatenate your other features to the output of the LSTM (with return_sequence = False). Then add some dense layers (1 could be enough) : this will output the probability of belonging to your classes.


problem of choosing right statistical method for scheduler prediction I am struggling to choose a right data prediction method for the following problem.
Essentially I am trying to model a scheduler operation, trying to predict its scheduling without knowing the scheduling mechanism and having incomplete data.

(1) There are M available resource blocks that can carry data, N data channels that must be scheduled every time instance i

(2) Inputs into the scheduler:

Matrix $X_i$ size M by N, consisting of N column vectors from each data source.    Each of M elements is index from 1 to 32 carrying information about quality of data channel for particular resource block. 1 - really bad quality, 32 - excellent quality.
Data which contains type of data to be carried (voice/internet etc)

Scheduler prioritizes number of resource blocks occupied by each channel every time instant i.

Given that

I CAN see resource allocation map every time instant
I DO have access to matrix $X_i$
I DON'T know the algorithm of scheduler and
I dont have access to the type of data to be scheduled.

I want to have a best guess (prediction) how the data will be scheduled based on this incomplete information i.e, which resource block will be occupied by which data channel. What is the best choice of prediction/modelling algorithm?
Any help appreciated!

Do you know if the scheduler has a memory?

Let us assume for a moment that the scheduler has no memory.  This is a straightforward classification (supervised learning) problem: the inputs are X, the outputs are the schedules (N->M maps).  Actually, if every N gets scheduled and the only question is which M it gets, the outputs are lists which channel (or none) is scheduled to each block, and there is only a certain possible number of those, so you can model them as discrete outputs (classes) with their own probabilities.  Use whatever you like (AdaBoost, Naive Bayes, RBF SVM, Random Forest...) as a classifier.  I think you will quickly learn about the general behavior of the scheduler.

If the scheduler has a memory, then things get complicated.  I think you might approach that as a hidden Markov model: but the number of individual states may be quite large, and so it may be essentially impossible to build a complete map of transition probabilities.


What is Sentiment Bias? How will it affect a Lexicon Based Sentiment Analysis? I am comparing deep learning and lexicon/rule-based models for sentiment analysis. When I was doing some research into the limitations of lexicon based models, I came across a journal article that mentioned sentiment bias. However, this article did not explain what exactly sentiment bias is.

I am providing the link here: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0202523

What the authors call sentiment bias is the tendency for such systems to give strong results, either positive or negative, as opposed to neutral. Very often sentences or documents are more or less sentiment-neutral, but accumulating the positive or negative weights associated to individual words makes it more likely to result in a non-zero "sentiment value".

Very simple example:

I am comparing deep learning and lexicon/rule-based models for sentiment analysis.
               +1   +1                   -1         +1         +1

Individually words such as "deep", "learning, "model", "sentiment" can be considered positive; "rule" can be considered negative. As a result your sentence would receive a strong positive score of +3, even though it's actually neutral.


Spark DataFrame "Limit" function takes too much time to display result import pyspark
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
import findspark
from pyspark.sql.functions import countDistinct
spark = SparkSession.builder \
.master("local[*]") \
.appName("usres mobile related information analysis") \
.config("spark.submit.deployMode", "client") \
.config("spark.executor.memory","3g") \
.config("spark.driver.maxResultSize", "1g") \
.config("spark.executor.pyspark.memory","3g") \
.enableHiveSupport() \
.getOrCreate()

handset_info = ora_tmp.select('some_value','some_value','some_value','some_value','some_value','some_value','some_value')
I configure the spark with 3gb execution memory and 3gb execution pyspark memory. My Database has more than 70 Million row. Show I call the
 handset_info.show()

method it is showing the top 20 row in between 2-5 second. But when i try to run the following code
mobile_info_df = handset_info.limit(30)
mobile_info_df.show()

to show the top 30 rows the it takes too much time (3-4 hour). Is it logical to take that much time. Is there any problem in my configuration.
Configuration of my laptop is:

Core i7 (4 core) laptop with 8gb ram

The reason is the way limit and the show is implemented under the hood. Show just reads the first 20 (first n) rows, which limit reads the whole data before showing it. Refer this answer on StackOverflow - link


R Programing beginner Hi I am new to Data analytics.I am planning to learn R by doing some real time projects. How should I stream line (set goals) my time in learning R and also I have not learnt statistics till data. I am planning to learn both side by side. I am mid level data warehouse engineer who has experience in DBMS Data-Integration. I am planning to learn R so that I can bring out useful analysis from the Integrated data.

If I be specific, I am beginning in R, so what are the basic statistical concepts I should know and implement it in R. If I want to be an expert or above average person in R how should I plan strategically to become one. Say if I can spend 2 hrs a day for 1 year what level I should reach. FYI am working for a SaaS company. What are the way s in which I can utilize R knowledge in a SaaS environment

I would start off with All Of Statistics by Larry Wasserman. This quickly gets you upto speed with statistics assuming you have some mathematical background. I think all it needs is introductory calculus and linear algebra.

R is pretty straightforward to pick up and there are a number of resources you can use. The R Programming course on coursera is an excellent short course to get you familiarized with R. Besides that there are a number of books and tutorials on the subject.

I would recommend you start working through All of Statistics and start playing around in R along with that book.

If you are at a SaaS company, there are a number of data sciency roles and responsibilities that should be available. Most SaaS companies will have analytical tools to provide basic insight. Once you start learning statistics and data science, you will be able to identify the gaps in the pre baked tools and ways to improve them. I would also read The Field Guide to Data Science, it's a short book that will give you a high level idea of data science and its utility.


How to predict product price range using machine learning algorithms I would like to run thru regression algorithms(linear, SVM, Random forest, Xgboost) thru historical data to predict the price range of a product.

To get the price range am going to use top predict value as upper bound and next best value as lower bound).

Ex: Product a price range preicttion - $1000 to $5000.

$5000 is from top predicted price.
$1000 is from next best-predicted price.

Is my approach correct?

Yet to receive data, so do not have much data sample to provide.

Please suggest.

If in your dataset you have two columns for the minimum and maximum value as columns n-1 and n, you can train the dataset for n-1 columns, where n-1th column is your dependent value, then for predicting nth column, the nth column would be your dependent value and n-1 others independent.


What are some significance tests to rank features(multiple) before training the data I have 8 features for a classification problem. The target value tells if there was an anomaly or not.

I want to run some significance tests to rank each feature, as being a distinctive feature of anomalies.

Could you please suggest some significance tests for this in Python?

One of the easiest ways to do this is with a correlation matrix. Variables that are correlated with the value you are predicting will probably be good predictors. Here is a good answer on how to do that in Python:

https://stackoverflow.com/questions/29432629/plot-correlation-matrix-using-pandas

However - be careful. If your variables are correlated with the response variable (the one you are predicting) AND they are correlated with each other it may be best to use just one of those variables instead of both.

The "best case" scenario is that you are predicting "y" which is correlated with "x" and "z" but "x" and "z" are not correlated at all with each other.

Because you are testing for anomalies (a binary response?) you can transform your predictor variables to be between 0 and 1 and then test the correlation between your transformed predictors and with your predictor variable.

If you want a significance test for which variables will be good predictors you will have to build some models. I know that in R the summary(...) function returns the statistical significance of regression model coefficients - there is probably a similar functionality in Python.


Plotting in Multiple Linear Regression in Python 3 So I'm working on linear regression. So far I've managed to plot in linear regression, but currently I'm on Multiple Linear Regression and I couldn't manage to plot it, I can get some results if I enter the values manually, but I couldn't manage to plot it. Below is my code block and dataset and error, what can i change to plot it?

Dataset:

deneyim maas    yas
0.5 2500    22
0   2250    21
1   2750    23
5   8000    25
8   9000    28
4   6900    23
15  20000   35
7   8500    29
3   6000    22
2   3500    23
12  15000   32
10  13000   30
14  18000   34
6   7500    27

Code block:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LinearRegression

dataset = pd.read_csv("multiple-linear-regression-dataset.csv",sep = ";")

x = dataset.iloc[:,[0,2]].values
y = dataset.maas.values.reshape(-1,1)

multiple_lr = LinearRegression()
multiple_lr.fit(x,y)

b0 = multiple_lr.intercept_
b1 = multiple_lr.coef_
b2 = b1

multiple_lr.predict(np.array([[10,35],[5,35]]))

array = np.array([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]).reshape(-1,1)
y_head = multiple_lr.predict(array)

plt.scatter(x,y)
plt.plot(array, y_head, color = "red")
plt.show()

It says ValueError: shapes (16,1) and (2,1) not aligned: 1 (dim 1) != 2 (dim 0) when I try to compile it.

You cannot plot graph for multiple regression like that. Multiple regression yields graph with many dimensions. The dimension of the graph increases as your features increases. In your case, X has two features. Scatter plot takes argument with only one feature in X and only one class in y.Try taking only one feature for X and plot a scatter plot. By doing so you will be able to study the effect of each feature on the dependent variable (which i think is more easy to comprehend than multidimensional plots).I think your issue should resolve.


How to deal with analyzing optional survey data Consider this scenario. There is a website that ranks every type of sandwich. For each type of sandwich, users fill out a survey about the following: A) how tasty it is (on a scale from 1-5); B) how filling it is (on a scale from 1 to 5); C*) if it is spicy (-1 or 1).

The website displays the average value of the answers to Question A, the average value of the answers to Question B, and the sum of the answers to Question C.

Question A and Question B are mandatory. Question C, however, is optional. I have the number of user-submitted ratings for each sandwich, and I have the sum of the answers to C.

I want to compare the spiciness of different sandwiches. Since different sandwiches have different rankings, I cannot simply compare the sum of the answers to Question C. I also cannot weight it as "Sum of Answers to Question C/Number of Ratings" since Question C is optional and some people won't answer it.

What would be the best way to go about comparing the spiciness of different sandwiches under these constraints?

Your best bet is to change the way you ask question C and the way you measure it. Giving a yes / no response for C will not let you gauge the actual spiciness so you need to adjust it to a scale and allow individuals to rank it. While all of your questions are fairly subjective, C is the most so as tolerance to spice plays a big role in if one thinks it spicy or not.

I suggest using a 1-5 scale, as you have for the others, and give comparative suggestions for each level of the scale e.g. 1 is eating bread while 5 is equivalent to eating a habnero pepper.


Graphical Dyslexia -- building intuition for common data visualizations? It took me years before I could understand what was going on in a histogram. I had trouble understanding the tails, skewness, and concepts such as "fat tails."

With a lot of practice, histograms have become more intuitive for me.

I believe I have some sort of "graphical dyslexia."

I want to build an inituitive understanding of basic, common types of data visualizations such as:

histograms
scatter plots
dotplots
stem and leaf plots
density plots

By intuitive, I mean that when I glance at one of these plots, I want to figure out the gist of it within a few seconds (rather than stare at it for minutes without anything registering in my mind, as is usually the case for me).

StackExchange: can you recommend to me books/resources that can help me with this?

There are two books that are our "bibles" in data visualisation.

Edward Tufte, The Visual Display of Quantitative Information
David McCandless Information is Beautiful

Tufte has a wealth of examples on good and bad visualisations. McCandless has some lovely, or even beautiful graphics and ways to display data.


Is Keras useful for professionals? I know that Keras is developed for quick deployment. Is it just for beginners or also useful in industry for professionals?

Keras is used in academia (see google scholar citations of Keras as a proxy for academy adoption) and hobbyists (see github stars or google results for keras in www.kaggle.com as proxies for hobbyist adoption).

It was recently bundled together with Google's tensorflow.

It is also used in industry, at least for prototyping (I know because I use it!), but I have found no source to back this statement.


Cocktail Party Problem using ICA If you already had a recording of two separate sound sources mixed, where one source was a person singing and the second source was a piano playing. Could ICA be used to separate the recording into 2 distinct sound sources?  If so, how would ICA be used to do this?

Yes, this can be done: here’s a paper from 2015 indicating recent developments and there’s a handy dandy tutorial here to get you started using scikit-learn’s ICA implementation. Enjoy!


What is the difference between Perceptron and ADALINE? What is the difference between Perceptron and ADALINE?

The Adaline (Adaptive Linear Element) and the Perceptron are both linear classifiers when considered as individual units. They both take an input, and based on a threshold, output e.g. either a 0 or a 1.

The main difference between the two, is that a Perceptron takes that binary response (like a classification result) and computes an error used to update the weights, whereas an Adaline uses a continous response value to update the weights (so before the binarized output is produced).

The fact that the Adaline does this, allows its updates to be more repesentative of the actual error, before it is thresholded, which in turn allows a model to converge more quickly.

Have a look at this really interesting history of neural networks, which contains a small section on Adalines, along with memistors - resistors with memory, as the neurons where figuratively perceived back in the 1960's.

There are also some other answers to a similar question here.


Why is it necessary to overfit a yolov2 on a subset of images before model training? I am using yolov2 for passport details detection with 8 classes? My object detectiion bounding boxes has errors though recognition being fine.

I want to know if object_detection by overfitting with a sample subset of training data, say 3 images, helps determine prediction performance? If yes how?

As far as I know, overfit a small sample of training data doesn't help predict performance. Normally, overfit a batch of training data is a debugging method.

It's often very hard to detect logical errors in a neural network. One easy way to test is if our neural network can overfit a small batch of training data, we can reasonably assume that it has no critical bug and start training. The reason is that a neural network can even fit a random labeling of training data, if it cannot fit a small batch, it's almost guaranteed that there's a bug in your network.

Keep in mind that this is only one test to perform, there might be some other subtle bugs that prevent you from getting a good result.


ValueError: Error when checking input: expected conv2d_13_input to have shape (3, 150, 150) but got array with shape (150, 150, 3) I am trying to train the model, I keep ending up with this ValueError:

  ValueError: Error when checking input: expected conv2d_13_input to have shape (3, 150, 150) but got array with shape (150, 150, 3)

How can I fix this? Should I use numpy.resize or cv2.resize to change the dimensions to (3,150,150). If so, how would I resize it in the generator?

Here is my code:

train_datagen = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2)

test_datagen = ImageDataGenerator(rescale=1./255)

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 150, 150),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

batch_size = 64

# this is a generator that will read pictures found in
# subfolers of 'data/train', and indefinitely generate
# batches of augmented image data
train_generator = train_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\train',  # this is the target directory
    target_size=(150, 150),  # all images will be resized to 150x150
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')  # since we use binary_crossentropy loss, we need binary labels

# this is a similar generator, for validation data
validation_generator = test_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\val',
    target_size=(150, 150),
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=2000 // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=800 // batch_size)
model.save_weights('first_try.h5')

Change this:

model.add(Conv2D(32, (3, 3), input_shape=(3, 150, 150),padding='same'))

to this:

model.add(Conv2D(32, (3, 3), input_shape=(150, 150, 3),padding='same'))

And read the doc: https://keras.io/layers/convolutional/#conv2d

In particular the section about data_format:

  data_format: A string, one of "channels_last" or "channels_first". The ordering of the dimensions in the inputs.  "channels_last" corresponds to inputs with shape  (batch, height, width, channels) while "channels_first" corresponds to inputs with shape  (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be "channels_last".


Why models performs better If normalize test data and train data separately? Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found like this one and this one that suggest that applying the normalization to the test set is not really required and it might depends on many factors such as the model used for training or the nature of the test data.

Now, personally, I am more inclined towards applying the normalization on test data as well. But the problem is this: I am working on a neural network model where:

If I apply normalization using the recommended way I get 79% accuracy, (and to be honest it's not interesting for me)
If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

So, I don't know what my neural network performs better on test unseen data if I use the second method. I really I want continue using the second method for this particular model, but I don't feel good about it and feels like it's wrong or cheating.

Now, I have one last argument. The last link I provided, have one answer that says this:

"..This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine..."

The dataset I use is a refined version of its predecessor (NSL-KDD dataset). The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

Ps: Sorry if this is long, it's a research ethics thing. I will follow the approach you guys recommend.

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a prediction for a single instance independently from any other instances, in which case there is no set of instances to obtain the mean/std dev from. More importantly, the prediction of the model for a given instance should always be the same. Normalizing on the test set breaches this principle, because it makes the prediction for a particular instance depends on the other instances in the test set.

I don't think "separate normalization" is unethical strictly speaking, because it doesn't imply using any of the test data at training stage (whereas normalizing before splitting the train/test sets would). However it's theoretically incorrect for the reasons I mentioned above.

The fact that you obtain such a big difference in performance by normalizing "separately" points to a very different distribution of the data between training and test set (or a bug somewhere along the process). I'd suggest investigating that, maybe there's some error in the data?


Classification of keystrokes A year or two ago I wrote a keylogger that has been quietly running in the background of my computer. Each line consists of a timestamp, a keycode, and a value representing modifier keys (e.g. ctrl, shift, etc.).

Separately, I have a web app that I use to manually make notes about how I spend my time: TaskRanger.com

As you can see in this screenshot, Task Ranger is basically a daily log of work done. Each day consists of a list of intervals. Each interval consists of a start time, a duration, and a string briefly describing the work I did during that period. The description also contains a twitter style hashtag that manually classifies the type of work I was doing.

I'm interested in combining both of these data sources to create a classifier that can take a raw log of keystrokes as input and output a sequence of tags indicating the type of work I did over time. For example, I might have been doing research from 11am to 1pm, then planning a project from 1pm to 2:30, then alternating between coding and debugging for the next few hours, etc.

The intent is to basically build a personal profiler. I want to become more productive. And I think I ought to be able to treat my workflow as a software program: figure out what the bottlenecks are, drill down into them, and focus my improvement efforts on those areas.

I tried to build some kind of naive classifier myself without much luck. One problem is I couldn't find an easy way to figure out when one kind of work ends and the next kind begins (segmentation I think it's called?). I think this whole classification task might be a lot harder than it initially seems. I have this vague idea that a Hidden Markov Model should work. Basically the type of work that I'm doing is a hidden state, and the keylogs provide clues as to that state. But I know next to nothing about signal processing or machine learning, so I'm not sure if I'm on the right track here.

Any advice or suggestions on how to build this would be appreciated. Some sample code demonstrating something similar would be especially useful.

Looks like you have a classification problem. A simple way to solve this is with a linear regression model. Here is how I would do that with the data you've provided:

1) Determine a "unit" of time, for example 1 minute of keystrokes. Once this process is complete you can tweak the unit of time to see if different intervals give you better results (5 minutes, 10 minutes, etc...)

2) Split all keystrokes into 1 minute chunks, and categorize those chunks as work, study, play, etc... Now you basically have a bunch of character strings that are categorized into activity types.

3) Create variables to describe your chunks. Word count, character count, wpm, # of times "lol" was typed, etc...Get creative.

4) Create a test set and training set of chunks.

5) Create a linear regression model to predict type of activity based upon the variables you created.

This is a pretty simple solution but if you can come up with good variables to distinguish your activity types it can work.


Find best machine learning for predicting category of products I have a dataframe that contains product and in this dataframe I have some features like: brand, cat1, cat2, cat3, city, desc, image_count, mileage, price, title, year.
The goal is predicting category of products. I have 1 billion training data and important features for prediction are title and description that are text type.
I like to know what algorithm is best for my prediction? I'm a beginner in machine learning and confused among different algorithms. Thanks

This seems like an ideal problem for deep learning. Your outputs will be the categories and you can pick the 3 activations (categories) with the highest probabilities.

You will need to vectorize your description features, you can do this using one of these techniques.


How to get same accuracy with identical models in Keras and Tensorflow? As we all know Keras backend uses Tensorflow and so it should give out same kind of results when we provide same parameters, hyper-parameters, weights and biases initialisation at each layer, but still the accuracy is different.

This maybe because of the batches of images which are fed at each step in both the models are not identical and as it gets shuffled randomly.

Is there any way in which we can make sure that the same batch of images are fed into the model while eliminating the randomness?

I have tried using all the same parameters, hyperparameters, same weights and biases initialization with seed values.

The accuracy of both the models are not same.

I created a rule to achieve reproducibility:

Works for python 3.6, not 3.7
First install Keras 2.2.4
After install tensorflow 1.9

And finally in the code:

import numpy as np
import random as rn
import tensorflow as tf
import keras
from keras import backend as K

#-----------------------------Keras reproducible------------------#
SEED = 1234

tf.set_random_seed(SEED)
os.environ['PYTHONHASHSEED'] = str(SEED)
np.random.seed(SEED)
rn.seed(SEED)

session_conf = tf.ConfigProto(
    intra_op_parallelism_threads=1,
    inter_op_parallelism_threads=1
)
sess = tf.Session(
    graph=tf.get_default_graph(),
    config=session_conf
)
K.set_session(sess)
#-----------------------------------------------------------------#


LIME visualization outputs padded regions as important - Mel-spectrogram (audio analysis) I am encoding audios as Mel-spectrograms and using these Mel-spectrograms as input to my deep learning model (Inception-ResNet V2). The input image is of size 256 X 256, made up of a 128 X 64 spectrogram (128 melbands) along with zero padding. Sample input:

I am using LIME to visualize the important regions of a Mel-spectrogram. In some Mel-spectrograms, LIME is labelling the padded regions as important, like in this image.

So, is this happening because of poor learning or is this a genuine problem with LIME when it encounters padded regions?

Feeding the neural network 75% irrelevant data (zero padding) seems less-than-ideal, both in terms of computational inefficiency and because it makes your model potentially vulnerable to this data.

I would consider a smaller model with 128x128 input input format (ex: MobileNet). You can either upscale the mel-spectrograms you have now from 128x64 -> 128x128 or extract windows that are 128 long (potentially adjusting FFT/hop size to make the window cover same area in time).


How google get the 'use over time for' data from the before internet years? Recently I am reading a book and I found there are lots of words I rarely saw. While I was searching their meaning, I found that for a single vocabulary google has a use over time for: statistic -- from 1800 to 2010.

I may imagine how google could gather the frequency that a vocabulary to be used in Internet. But what about before the Internet? How they know the frequency of the people in 1800 using a vocabulary? Does they first convert the chartaceous material into digital data then computing for the result or something else?

I am not sure if you are familiar with this, but in linguistics there is a term "corpus". It is a large collection of texts used for statisical analysis and other linguistic research, both historical and contemporary.

There are corpora of digitalized texts, both literature and other domains such as newspapers, bulletins, etc etc. I recon Google uses data from such corpora.

For example: https://www.sketchengine.eu for English.

Here's a somewhat robust list of corpora: https://en.wikipedia.org/wiki/List_of_text_corpora


API for Company Data Enrichment Suggestions I'm looking for API suggestions for enriching data on companies. Currently I use the Crunchbase API to look up a company's name or domain and I am trying to gather the domain/name (if I don't already have both), contact email (this one is a long shot), and the location of their headquarters.

This works incredibly well if Crunchbase has the company in their API, but I'd say this only happens about 25% of the time.

I'd love to get some suggestions on some free APIs that I could use along with Crunchbase. I'd also love to see if anyone has had positive or negative experiences with paid APIs!

Check out Clearbit.com -- https://dashboard.clearbit.com/docs#company-api

Super comprehensive company lookup. Not free, but 50 free calls a month.


Why not always use the ADAM optimization technique? It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.
There is often a value to using more than one method (an ensemble), because every method has a weakness.


Training a classifier with text and numerical features - what is the state of the art? I'm trying to build a binary classifier where the features are mostly numerical (about 20) and there are a couple of unstructured short text fields as well. What is currently considered the state of the art for combining these types of features?

I've tried building a separate classifier (logistic regression, TFIDF) using the text data alone, and then including that output score of that classifier as an additional when training using the rest of the numerical features (random forest, taking care to train each classifier on different folds of the data to prevent signal leakage). It works alright, but I think a better performance is possible.

A variant is to simply train two separate classifiers, one using text and the other using numerican features and then stacking those two. And finally, another idea is to use neural networks, have two input networks, one CNN/LSTM for the text, and another dense for the numerical features, and then combining them and having a single output.

Are there other approaches I haven't thought of that are worth a try?

Regarding the first question, to the best of my knowledge, there is no SOTA approach. It depends on the task.

For the second one, did you try to combine the numerical values with the TFIDF vectors, and then feed the final vector(s) to the classifier? Probably not. I usually try this way. But, if your vocabulary is too large, try to reduce it by considering top N-words, because combining small numerical vector with a huge TFIDF vector may not allow the numerical values to affect the results.


Text Dataset Help: Need to figure out how to get the publication date of a text I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the labels will be the publication/creation date. Since our project requires a significant amount of data we were planning to use the Gutenberg project. However we came upon the issue that the creation date was not in each file but rather on a website associated with the name (so we have to go to a website which ends with the text's file name). We are wondering what the best approach would be to get our data - should we somehow get the date using the website (even though they say they will ban ip's of scrapers), find a new dataset, or build some sort of tool which will search the internet for that text's creation date. Thanks and if you have any suggestions please feel free to comment. If our question should be further expanded upon please clarify. Thanks again.

I would hazard a guess that most books uploaded on Gutenberg also have a Wikipedia page. Especially the well-known books. There are a few Wikipedia related Python packages (i.e. Wikipedia) that can make it very easy to search for the title of the book, navigate to the book's page, and extract the publication date. An optimal example might be Alice's Adventures in Wonderland. It'd be trivial to extract the date from that page, which appears neatly in the info box, and convert it to a machine readable format in Python.

Or, you could attempt to scrape the info straight from Gutenberg. Perhaps consider Selenium, although make sure you limit your bot's rate so you don't adversely effect the site.

Lastly, trying to predict the publication date might be similar to predicting the author; both may be derived from writing style. There's a lot of work on this topic, a Google search or an arxiv search will help. Here's one of the first articles I opened.


what is the first input to the decoder in a transformer model? The image is from url: Jay Alammar on transformers

K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each decoder layer in the decoder.
The previous output is the input to the decoder from step 2 but what is the input to the decoder in step 1?  Just the K_encdec and V_encdec or is it necessary to prompt the decoder by inputting the vectorized output (from the encoder) for the first word?

At each decoding time step, the decoder receives 2 inputs:

the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks.
the target tokens decoded up to the current decoding step: for the first step, the matrix contains in its first position a special token, normally </s>. After each decoding step $k$, the result of the decoder at position $k$ is written to the target tokens matrix at position $k+1$, and then the next decoding step takes place.

For instance, in the fairseq implementation of the decoding, you can see how they create the target tokens matrix and fill it with padding here and then how they place an EOS token (</s>) at the first position here.

As you have tagged your question with the bert tag, you should know that what I described before only applies to the sequence-to-sequence transduction task way of using the Transformer (i.e. when used for machine translation), and this is not how BERT works. BERT is trained on a masked language model loss which makes its use at inference time much different than the NMT Transformer.


Is my data good for (DBSCAN) clustering? I have a particular dataset consisting of 50k elements with 40 features each. I want to try to cluster the data as it is, without any dimensionality reduction. The main algorithm I am considering is the DBSCAN since is the more versatile and I can accept some poits to result as noise. However how can I judge if the clustering is "significant" since I can't plot the clusters in comparison to the data?
Tring to select the paremeters for the DBSCAN I've done a k-nn analysis, but the results worried me. For example, the following is the third nearest neighbor plot.

As you can see the distances (y-axis) are pretty much "uniform" along the x-axis (object). Does it mean that the data is somehow uniformly sparse and, in this conditions, clustering is useless?

P.S. I have tried to cluster the data anyway, in particulary using epsilon around 2-2.5 and for different values of min_sample. The silhouette score however results very low, about 0.11, and the noiseless fraction of points is about 80%.

You can look at several clustering metrics (e.g., Silhouette Coefficient, Calinski-Harabasz index, or Davies-Bouldin index) and see if they revealing patterns in your clusters.
You can look at relative performance by trying different hyperparameters for DBSCAN and comparing DBSCAN to other clustering techniques.


Get number of correct predictions for each class in Keras I am having unbalanced dataset(1:93) and want to use kappa's metric.
However, for that I need to capture how many correct predictions are made for each class.

I have tried understanding from here and other google links.

Is it possible to capture class wise #correct predictions made?

That's true. For understanding how many correct decision your classifier has made, confusion matrix can be used. The main diagonal illustrates that. It depicts how many data samples are correctly classified and how many are mislabeled to which class. you can take a look at here.


Install Spark and Hadoop in the same machine I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it worked perfectly. But, when I started installing Hadoop and set the JAVA_HOME environment variable HDFS worked, but spark break down and shows: Files was unexpected at this time. when I launched it. When I deleted JAVA_HOME, Spark worked again but it's not the case for HDFS. What should I do in this case?

The Hadoop stack is difficult to setup and people complain that you can't trust any answers to problems over 6-12 months old. I would recommend getting a pre-configured Hadoop/Spark setup from Cloudera or HortonWorks. Both have free community editions.


How to approach a new data set with no dependent variable Basically I am taking a small course in data science.
And one of our first assignments are that we have gotten a pretty huge data set (hundred of thousands of samples, and roughly 150 independent variables). There is no response variable in this, but the assignment is pretty much: "Find something that are interesting". Which I obviously kind of broad.

My questions is: How would/should you/I approach this ?

Up until now I have just cleaned the data set as good as I can. Removed variables I don't think make much sense, made sure there are no NaN variables, and stuff like that. But what would be the next step in order to "find something useful" in a data set that may or may not contain anything interesting?

Your assignment is basically the process we call EDA - Explorative Data Analysis.

So what should you do? Simply explore!

What is the shape of your dataset?
How do variables behave, do they have a factor structure, correlate, etc.
What are the main descriptives of your dataset, to they tell an interesting story, etc.

And once you start doing this you will find something that might be interesting to explore a bit deeper depending on your dataset. Do not just use summary functions like mean, median, etc. but also try to build simple graphs and comment everything in a neat notebook!

My tip:

Look at some EDA notebooks on Kaggle for inspiration or watch this superior video by a master at work:

https://www.youtube.com/watch?v=go5Au01Jrvs

Also here is a beginner guide as well:

https://towardsdatascience.com/exploratory-data-analysis-eda-a-practical-guide-and-template-for-structured-data-abfbf3ee3bd9


What is multi-hot encoding? I was read and paper for machine learning, and i found this term "multi-hot encoding" without explanation.
Can you help me please?
the paper: https://arxiv.org/abs/2001.06917

Without having read the paper, from the name I would say a multi-hot encoding is a matrix $X \in \{0,1\}^{n \times m }$, with $\sum_{i = 1}^{n} X_{i,j} = 1, \forall j = 1,\ldots,m$.
You obtain the normal hot-encoding by setting $m=1$.


Determining completeness of dataset I'm hoping you have some research or experience with determining the completeness of a data set. I'm trying to use a twitter dataset I scraped myself and want to have an indication on the completeness.
Obviously, I will miss some data but I am wondering if there is a formula or method to calculate a probability on the completeness.

If you are talking about exploring your data for patterns of missing data you can try using Self-Organizing Maps [https://en.wikipedia.org/wiki/Self-organizing_map], which are a special flavor of neural networks. Here is a small research paper explaining the concept a bit. Here is another [link][3] with historical info and links to some original papers about the topic, specifically by Kohonen et al.

From Wikipedia:

  Self-organizing maps are different from other artificial neural
  networks as they apply competitive learning as opposed to
  error-correction learning (such as backpropagation with gradient
  descent), and in the sense that they use a neighborhood function to
  preserve the topological properties of the input space. This makes
  SOMs useful for visualizing low-dimensional views of high-dimensional
  data...

Essentially this method will allow you to look for patterns in your data. From there you can determine what level of completeness you are dealing with in your data set.

Specifically for your example for Twitter data, I would imagine there are many fields in the JSON data with missing values. Maybe some users choose not to fill in their gender or age etc. Visualizing your data as well calculating summary statistics will help you to paint an overall picture of your data. And when you have high dimensional data using tools to visualize it in a lower dimensional space can always be handy. Hope this helps you!


PCA & Clustering Confusion I have a question related to K-Means clustering and PCA. In my project, I have two target classes - 0 and 1- and I am trying to group the records that were predicted as 0 into 5 clusters.
I am using PCA strictly as a visualization technique since my data frame has 8 dimensions and I need to bring it down to 2-3 dimensions to see the clusters. My question is about the procedure I should follow~
First Way:

Extract all records with target = 0
Do PCA and KMeans on just those records

Second way:

Do PCA on all records (target = 0 and 1)
Extract PCA records with target = 0 (from the PCA data frame created in step 1)
Do KMeans on those records

The PCA1, PCA2, PCA3 values for the records(with target = 0) are different using these two ways. And since the PCA values are different, the cluster visualizations are also showing up differently. Which option should I follow?
Thank you so much!

My answer would be second option
I think the use of PCA is to represent original high dimensional information/data in lower dimension by calculationg the direction/axes along which there is maximum variablity in data.
In first case, where you filter for 0-labeled observations and then do PCA so PCA would measure variablity based on a smaller version of original data, and would make different axes than the second case, where the PCA would measure variablity accross entire data and hence the axes made in second case could be different. So in the first option the dataset you get after PCA is not a correct representation of high dimensional 0-labeled observations.


predict_proba returns different results on Python 2 & 3 I had some old code used to train a Random Forest Classifier (sklearn 0.17.1), for classification on two classes (spam/ham). I ran this in a docker container and sent it some data.

Sklearns predict_proba returns [0.3, 0.7].

I “rewrote” (literally changed a few lines for Python3 compatibility), trained with the same data, and sent it the same data, and now predict_proba returns [0.7, 0.3].

Did something change in sklearn.ensemble.RandomForestClassifier predict_proba or have I messed up somewhere, or does the new one still work?

Maybe...
Have a you seen if one of the predict_proba produces 2 columns?

I think Python 3 produces two columns and the one comparable is the second.


inner workings of Mobile-net resolution multiplier - what does it do? i have a question concerning the way Mobile Net's resolution parameter works. From the article itself and from the blog posts on the topic (1, 2) I wasn't able to find an answer to my question. It is said that “the resolution multiplier changes the input dimensions of the image, reducing the internal representation at every layer”. Does that mean it (the parameter):

A) breaks down each image into a, for example, respective 224x224 bits of pixels sub-images, over which the network runs its convolution?

OR

B) takes one single 224x224 slice of pixels from, for example, the middle of the image, over which the network runs its convolutions?

OR

C) Squishes the original image into a, for example 224x224 pixel representation of itself, over which the network runs its convolutions?

I contacted the authors of the mobile net architecture and papers and they provided me with the following answer:

The multiplier accomplishes what is being articulated in C): it acts as a re-scaling of the original input image to fit the value specified by the parameter. The default image size is 224x244, which corresponds to a resolution multiplier of 1.0. So 112x112 input image resolution corresponds to a 0.5 resolution multiplier which is achieved by simply downscaling the image to 112x112. Technically we scale the smaller side to target and crop the other dimensions.

I hope this proves as helpful to others as it was to me.


Support Vectors of SVM I have read somewhere that the value of slack variables of support vectors is not 0. Does that mean the points lying in the wrong region e.g a positive point lying in the negative region will also be a support vector? I have attached one picture as well which shows that points lying in the wrong region also support vectors. I am looking for an explanation of this phenomenon

Yes, they are support vectors. This is because they contribute (or serve as support) in the computation of the hyperplane that separates the positive and negative regions. This hyperplane is given by:
$$\mathbf{w}^T x + b = 0 $$
Where $x$ is our $\text{n-dimensional}$ input vector that contains the $n$ features of our problem (the input samples during the learning process), and $b$ and $\mathbf{w} = (w_1,w_2,...,w_n)^T$ are the parameters that are optimized. More concretely, $b$ is the intercept of the hyperplane and  $(w_1,w_2,...,w_n)^T$ is its normal vector (justification in Mathworld).
But why the misclassified points contribute?
More detailed justifications can be found in the Andrew Ng notes about SVM which I recommend.
To understand why they contribute, we need to have a look at the cost function, $J$, that is used when the data is not linearly separable, like the case of the question (this cost function is also used to prevent the influence of outliers):
$$J = \frac{1}{2}\Vert \mathbf{w}\Vert^2 + C \sum_{i=1}^m \xi_i$$
$$\text{subject to}\begin{cases}
y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\geq 1-\xi_i, \,\,\,\,\,\,\,\,\,\, i = 1,...,m\\
\xi_i \geq 0
\end{cases}$$
Where $\xi_i$ is the slack of the input sample $x_i$, being $\xi_i = 0$ only when the input sample $x_i$ is correctly classified and presents a functional margin $\geq1$.
In order to solve this in a efficient way, it can be proved that by applying the Lagrange duality to the problem presented before (minimizing $J$ w.r.t. $\mathbf{w}$ and $b$), we end up with a equivalent problem of maximizing the next function w.r.t. $\alpha$:
$$ \max_{\alpha}\,\,\,\, \sum_{i=1}^m\alpha_i - \frac{1}{2}\sum_{j=1}^m\sum_{i=1}^my^{(i)}y^{(j)}\alpha_i\alpha_jx_ix_j$$
$$\text{subject to}\begin{cases}
0\leq \alpha_i\leq C, \,\,\,\,\,\,\,\,\,\, i = 1,...,m\\
\sum_{i=1}^m\alpha_iy^{(i)}=0
\end{cases}$$
Where each $\alpha_i$ is the Lagrange multiplier associated with the input sample $x_i$.
Furthermore, it can be proved that, once we have determined the optimal values of the Lagrange multipliers, the normal vector of the hyperplane can be computed by:
$$ \mathbf{w}=\sum_i^m \alpha_i y^{(i)}x^{(i)}$$
Now we can see that only the vectors with an associated value of $\alpha_i\neq0$ will contribute to the computation of the hyperplane. This vectors are support vectors.
Now the question is: When $\alpha_i \neq 0$?
As explained in the notes linked above, the values of $\alpha_i \neq 0$ are derived from the KKT conditions which are needed to be satisfied in order to find the values of $\alpha_i$ that minimize our cost function. These are:

$\alpha_i = 0 \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\geq 1$
$\alpha_i = C \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\leq 1$
$0<\alpha_i < C \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)= 1$

So, in conclusion the vectors (samples) that lie on the margins (condition number 3 and condition number 2 when $=1$), the vectors that are correctly classified but lie between the margins and the hyperplane (condition number 2 when $<1$) and the vectors that are misclassified (also condition number 2 when $<1$) are the ones with $\alpha_i \neq0$ and therefore contribute to the computation of the hyperplane $\rightarrow$ they are support vectors.


validation_curve differs from cross_val_score? I'm trying to see how well a decision tree classifier performs on my input. For this I'm trying to use the validation and learning curves and SKLearn's cross-validation methods. However, they differ, and I don't know what to make of it.

The validation curve shows up as follows:

Based on varying the maximum depth parameter, I'm getting worse and worse cross-val scores. However, when I try the cross_val_score, I get ~72% accuracy reliably:

While I was using the default tree depth for clf here, it still puzzles me how the validation curve never reaches even 0.6, but the cross-val scores are all above 0.7. What does this mean? Why is there a discrepancy?

Code for reference below.

For the Validation curve:

import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import validation_curve

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()

param_range = np.arange(1, 50, 5)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", param_range=param_range,
    cv=None, scoring="accuracy", n_jobs=1)
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)

plt.title("Validation Curve with Decision Tree Classifier")
plt.xlabel("max_depth")
#plt.xticks(param_range)
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
lw = 2
plt.plot(param_range, train_scores_mean, label="Training score",
             color="darkorange", lw=lw)
plt.fill_between(param_range, train_scores_mean - train_scores_std,
                 train_scores_mean + train_scores_std, alpha=0.2,
                 color="darkorange", lw=lw)
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="navy", lw=lw)
plt.fill_between(param_range, test_scores_mean - test_scores_std,
                 test_scores_mean + test_scores_std, alpha=0.2,
                 color="navy", lw=lw)
plt.legend(loc="best")
plt.show()

For the cross-val scores:

clf = DecisionTreeClassifier(class_weight='balanced')
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
clf.score(X_test, y_test)

UPDATE
A comment has been asked about shuffling. When I shuffle the data by

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

I get:

Which makes even less sense to me. What does this mean?

First of all, you have to shuffle your data because it seems that the model has learned a special pattern in the training data which has not occurred in the test data so much. After that, suppose that you get a validation curve like the current one. As you can see, Increasing the value of depth, does not change the learning. The two lines are parallel. In cases which each of the lines may have intersection, the upper line has negative slope and the lower one has positive slope, in the future on seen levels, you may want to increase the number of levels, not in this case.

Having same error, means that you are not over-fitting. but as you can see the amount of learning is not too much which means that you are having high bias problem, which means you have not learned the problem so well. In this case means that your current feature space maybe has high Bayes error which means that there are samples which have same features and different labels. Actually the distributions of different classes overlap.

There is something to argue about decision trees. If you have numerical features which are continuous, you may not have exactly same input patterns but they have overlap in their range.


Training a model purely on weak labels I have read a couple of papers now use rules-based system to create weak labels and then train a BERT-based model only using these weak labels. Both studies have reported better performances on manually labelled gold-standard test data.
However, I just don't follow the logic here. I understand distant supervision and all that. It's been around for a while now. I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth", and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?
Even though the performances on test data are better. It doesn't convince me your ML model (from the 2nd step) have learnt something beyond the weak labels you fed to it.
The only argument I find has some merits is that you basically treat the BERT-based model already as a zero-shot classifier and you are fine-tuning it with weak labels. I am just confused. Can someone pls enlighten me? Am I missing something obvious here?

Based on a quick read of the paper linked in the comment:

I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth",

Correct, but only for the training: the model is trained to recognize the labels obtained with a "quick and dirty" method.

and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?

No, because the real ground truth they are interested in is not those from the "quick and dirty" method. If they were, it would indeed be sufficient to run their rule-based system. The goal is to predict the labels obtained in what the authors call the "Gold Standard Corpus", which was manually annotated and never seen by the model.
Typically the quick and dirty method will result in some classification errors. The point of tuning the model with these labels is to see if the model can extrapolate from these low-quality labels to high-quality labels. This ability to generalize beyond the training data is based on the underlying semantic information contained in the original BERT-like model. For example this model might be able to associate a specific sport like swimming with "physical activity", even though the weak supervision doesn't contain this association.


Resizing images for training with Mobilenets I have a script to download images, but the images are of different resolutions so I have written a script to shrink the image. I have two options:

size=(224,224)

with cv2

cv2.resize(img,size,interpolation=cv2.INTER_AREA)

with PIL

img.thumbnail(size,Image.ANTIALIAS)

after saving them I see cv2 doesn't maintain the original ratio where as PIL maintains the ratio.

My question :

Maintaining aspect ratio is important or not.
If yes (224,224) is a good choice or should I set it to higher
resolution.

Sorry if the question is naive, I am new to image processing.

Maintaining aspect ratio is important or not.

Yes, it's really important in most cases. As you can read from here, Why does aspect ratio matter? It’s all to do with the relationship of the main subject to the sides of the frame, and the amount of empty space you end up with around the subject. An awareness of the characteristics of the aspect ratio of your particular camera can help you compose better images. It also helps you recognise when cropping to a different aspect ratio will improve the composition of your image.

In deep learning tasks, it depends how you want to feed data to your network. It's better to train your network with real data. Consequently if you are going to face data with standard aspect ratio, you have too keep it during training.

  If yes (224,224) is a good choice or should I set it to higher resolution.

Depending on your task 224 may be good or not. In typical classification networks, it is an acceptable size, like the inputs of AlexNet or VGG and such customary nets. For localization tasks, not really. For instance, the input height and width of YOLO is more than that 1.


Quick guide into training highly imbalanced data sets I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test samples as a majority class.

Some good answers about sub-sampling and weighted random forest are given here: What are the implications for training a Tree Ensemble with highly biased datasets?

Which classification methods besides RF can handle the problem in the best way?

Max Kuhn covers this well in Ch16 of Applied Predictive Modeling.
As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data.
There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. Rusboost I think is only available as Matlab code.
I don't use Weka, but believe it has a large number of cost sensitive classifiers.
Handling imbalanced datasets: A review: Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas'
On the Class Imbalance Problem: Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, Guangtong Zhou


How do I store a question-and-answer combination for machine learning? I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the question word-by-word and have the algorithm note some of the previous words to implicitly learn how these words combine and affect the answer. For example, the question "How's the weather in Philadelphia?" gets broken up into "How's","the", "weather", "in","Philadelphia".

The problem is, not all my questions or answers are of the same length. I could have questions like, "How's the weather?" or "What color of pants am I wearing?" Plus, the vocabulary consists of about 300 words, making a one-hot encoding scheme based simply on these words impractical.

Typically, people do not use one hot encoding as it takes up a lot of space (vocabulary size explodes) and is difficult to learn as well. Instead, they project words into a high dimensional vector space where each dimension represents some sense or direction.

For example, if you consider the words red and green - they are opposites in the context of a traffic light but are similar in the sense that they are both names of colours. A vector representation of the word will incorporate such relations by having similar values for some dimensions and different values for the others.

For more details, you can read this paper.

As for your next question, standard practice is to limit the sentence length through truncation and pad smaller sentences with additional 0 vectors.


How can I get variable values from a plot? I would like to know if there is any way in python to automatically determine the values of point a and b marked in the graph below. Said a different way, finding the values when my plot shows a marked decrease or after a marked increase.

This is the source code of my plot if it is helpful:

#load data

min_duration = data['Time_Secondes'].min()
max_duration = data['Time_Secondes'].max()
xr_ = list(range(min_duration,  max_duration,  60))
y_ = []

for i in range(0,(len(xr_)-1)):
    Time_proba_ = np.logical_and(data['Time_Secondes'].values >= xr_[i], data['Time_Secondes'].values < xr_[i+1])
    Time_proba_1_ = np.logical_and(np.logical_and(data['Time_Secondes'].values >= xr_[i], data['Time_Secondes'].values < xr_[i+1]), data['outcome'].values==1)
    y_.append(sum(Time_proba_1_ )/sum(Time_proba_))

import matplotlib
matplotlib.pyplot.plot(xr_[1:len(xr_)], y_)

```

It seems what you are looking for is a function of your data, not of matplotlib. I would think of this as a second derivative problem -- you care about differences in slopes of successive lines.

Sort a dataframe [x,y] by x values, in increasing order.
Calculate the first discrete derivative delta1 = (y[i+1] - y[i]) / (x[i+1] - x[i]). This tells you the slope of each line.
Now you can do one of two things, depending on what you want to identify.

Calculate the second discrete derivative delta2 = (delta1[i+1] - delta1[i]) / (x[i+1] - x[i]).
Calculate the difference of successive slopes diff = delta1[i+1] - delta1[i]

Select all is for which absolute value of {diff or delta2} is greater than a certain threshold.

Selecting all x values for which np.abs(x[i]) > threshold will give you the x-locations of these large changes in slope.

Here's an example I just coded up,for identifying the jump in a jump function $x / |x|$.

import numpy as np
import pandas as pd

n_data = 100

x = np.random.uniform(-10,10,n_data)
y = x / np.abs(x)

delta = [0] * n_data
diff = [0] * n_data

data = pd.DataFrame({'x':x, 'y':y})
data = data.sort_values('x')

for i in range(n_data - 1):
    a, fa = data.iloc[i]
    b, fb = data.iloc[i+1]
    diff_quot = (fb - fa) / (b-a)
    delta[i] = diff_quot

data['delta'] = delta

for i in range(n_data - 1):
    a, fa, da = data.iloc[i]
    b, fb, db = data.iloc[i+1]
    dd = db - da
    diff[i] = np.abs(dd)

data['diff'] = diff

plt.plot(data['x'],data['y'])
plt.plot(data['x'],data['diff'])

threshold = 1
jumps = [dd > threshold for dd in diff]

Here is the jump function, whose jump we might want to locate

Here is a graph of the differences in slopes, where we see the jump is located near x = 0


Making sense of how R works internally As what type of "thing" are various objects and functions one deals frequently with in R stored internally?

To clarify: For languages like Python it is very easy to understand conceptually how your data is stored internally: Everything is stored as an object and this uniform way of handling data makes it very to do advanced stuff like assigning new meaning to methods or writing functions that take other functions as input. What I am looking for is a similar conceptual understanding for R (e.g. I presume not everything in R an object as well).

(One concrete point that I would hope to understand as a consequence of the answer to this question would be: why setting,e.g., the second column of a dataframe to NULL like sodf[2] <- NULL deletes it. This means: You don't have to answer this question particularly, but please answer in such a way that I understand how various recipes spread out on the internet for dealing dataframes actually make sense.)

Also, please don't just point me to the documentation. I'm sure somewhere deep inside it is the answer, but reading the documentation is just too time consuming.

I am not sure I can go as deep as you'd like, but I can give the basics.
The base types in R are C-structs. Taken from Hadley Wickham's Advanced R:

Base types
Underlying every R object is a C structure (or struct) that describes
how that object is stored in memory. The struct includes the contents
of the object, the information needed for memory management, and, most
importantly for this section, a type. This is the base type of an R
object. Base types are not really an object system because only the R
core team can create new types. As a result, new base types are added
very rarely: the most recent change, in 2011, added two exotic types
that you never see in R, but are useful for diagnosing memory problems
(NEWSXP and FREESXP). Prior to that, the last type added was a special
base type for S4 objects (S4SXP) in 2005.

One layer higher (or at least different in parallel to the C base types), R itself has a few Object Oriented Systems at work. Major data containers that you likely use, such as vectors, dataframe, methods are going to be of type S3. The newer major object system defines S4 objects. There is a great overview on Hadley's webpage
Things such as removing a column if it is set to NULL (as opposed to Panda's approach, of making each column take that value) I would guess are design choices. It is likely more common to drop a column than it is to fill it with null-values - in the context of 99% of the algorithms R packages the null values would simply be ignored anyway. So for convenience; not that DF.drop() in Pandas is a major inconvenience.
This behaviour is called an idiom in this blog. They also show that assigning NULL to a vector element also deletes that element. Additionally, NULL and NA behave differently: NA is a missing value, NULL is not!
IMO you should take the time to at least skim some documentation. With the help of search engines and control-F, you rarely need to drudge through pages of docs to find what you are looking for.


Classification problem: custom minimization measure Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods.

I'm trying to develop a classification model, where the desired outcome is the probability, rather than only the output class.

However, whatever variable/model combinations I try, the models are able to distinguish the bad cases quite nicely, but not the good cases. In other words, the distribution function of (smoothed) empirical outcomes vs the model estimated probabilities is not monotone, roughly having the shape of a tilted parabola (like letter $J$).

My question: What are the common  strategies in order to reshape any model to focus the estimation on the good cases? If it is possible at all?

Intuitively, it seems that a possible strategy would be to define a custom metric for the minimizer, that would have asymmetric weights for good vs bad cases. I.e., in the case of penalized linear regression, the variable selection could then be skewed towards distinguishing goods instead of maximizing total AUC. But so far I'm unable to find any realized similar solutions. I guess, implementing something like that is not a trivial task..

Alternatively, is it possible to achieve this by transforming the input variables in a certain way?

I think what you are dealing with is a type I vs. type II error discussion. It seems that you want to avoid Type II error.

One way to go is to consider $F_\beta$ for your performance metric. It is a modified version of $F_1$. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated in terms of type I /type II error. You then need to select a $\beta$ based on how much avoiding type II error is important to you. This can be done by considerations on the cost of type II errors compared to the cost of type I error.


Why i get OOM error although my model is not that large? I am a newbie in GPU based training and Deep learning models. I am running cDCGAN (Conditonal DCGAN) in tensorflow on my 2 Nvidia GTX 1080 GPU's. My data-set consists of around 32,0000 images with size 64*64 and 2350 class labels. If I set my batch size 32 or large I get OOM error like below. So I am using 10 batch size for now.

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]]
Caused by op 'discriminator/concat', defined at:
  File "cdcgan.py", line 221, in <module>
    D_real, D_real_logits = discriminator(x, y_fill, isTrain)
  File "cdcgan.py", line 48, in discriminator
    cat1 = tf.concat([x, y_fill], 3)

The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do help -n 1 nvidia-smi, I get the following output.

The GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas GPU:1 is always 0%-3% max. Performance for GPU:0 is always in P2 whereas GPU:1 is mostly P8 or sometimes P2. I have the following questions.

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

Model Details are as follows

Generator:

I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).
W1 is of the shape [X+Y, 16*16*128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively

Discriminator

It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).
w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [16*16*256, 1] respectively.

Session Configuration
I am also allocating memory in advance via

gpu_options = tf.GPUOptions(allow_growth=True)
session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

Yes, the batch size is probably the reason.

Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

  2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

By default, Tensorflow occupies all available GPUs (that's way you see it with nvidia-smi - you have a process 34589 which took both GPUs), but, unless you specify in the code to actually use multi GPUs, it will use only one by default.

Here is an official TF docs about how to use multi GPUs: https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus

Here is some tutorial for multi-gpu usage with more examples: https://jhui.github.io/2017/03/07/TensorFlow-GPU/


Difference between impurity and misclassificaton I am reading the gini index definition for decision tree:

Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.

This seems to be the same as misclassification. Is Gini index just a fancy name for misclassification? Or is there really some subtle difference? Thanks!

Is Gini index just a fancy name for misclassification?

No.

Note that Gini index definition doesn't involve predicted values, and also it involves some probabilities, which are not dependent on classifier.

Also in context of decision trees, Gini impurity corresponds to each region, and is not a single value, such as missclassification rate (technically you could also count missclassification rate per region, but then you'd also ).

See this notebook for a concrete example.


Handling time series data with gaps I am working on a dataset with physical measurements taken daily (weight, bmi, etc...) and I am working through the process to graphically represent it. I think it is worth noting that every day has a corresponding row, but if no measurements were taken, the values are the same as the day before.

Here is an example of the trend I am trying to manage:

Date, Weight, BMI
1/1/2016, 155.1, 21.9
1/2/2016, 155.1, 21.9
1/3/2016, 155.1, 21.9
--continued for several weeks--
3/1/2016, 170.2, 25.0
3/2/2016, 170.1, 25.0

edit: I should clarify that these repeated values are how the data is put together for missing values. The days that are repeated are days when no measurement was taken

If the value stays the same for an extended period, should there be a gap in any graphical representation? Should I keep the numbers as they were (155.1 & 21.9) until the next measurement was taken, or should the numbers increase over that time to "bridge the gap" - meaning they would increase by the difference in measurements divided by the number of days?

It feels like I should be increasing the value over time to account for what would happen in reality, but I don't know if this would have negative implications on the data.

I'd represent actual repeated values differently from missing data. The latter is straightforward; the former I'd interpolate with Gaussian process regression. This way you'll be able to get error bars like so:

Note how the sample functions (and thus error bars) expand and contract as you depart and approach the measurements, as you would intuitively expect.


Dimensionality reduction with prior knowledge of colinearity between features Let's say that I have sparse feature vectors and I'd like to use dimensionality reduction in order to visualize them more easily.

Dimensionality reduction techniques like PCA will estimate colinearity between the features given the data. What if I have some prior knowledge on the colinearity between my features? As in, I would be able to create an approximate distance matrix between my features, and therefore between my data points.

I am aware that if features are actually colinear, methods like PCA will find a way to reduce them, however, I am afraid that I don't have enough data points to infer that colinearity strictly from the data and that PCA would remove meaningful features and keep uninformative ones.

Let's say that my data looks like something like this:

$x_0 = \{1,NaN,-1,NaN\}$

$x_1 = \{1,NaN,NaN,-1\}$

In my case, I know that the 3rd and 4th features are colinear and that the distance between $x_0$ and $x_1$ is close to zero. Therefore, they should be mapped to a very similar data point in a lower-dimensional space.

Can PCA still do it with so few data points? Are there ways to "force" the known colinearity or distance measures? T-SNE perhaps?

One option is to use Multidimensional scaling (MDS) for dimensionality reduction. MDS can create a visualization of the relative positions for data based on the distance between the data points.

In your example, the data points that have no distance between them in several dimensions will be projected close to each other.


Is the maximum BDeu Bayesian Network always the empty network? I'm recently reading a paper about Scoring Mechanisms for Bayesian Networks.

For the BDeu score, it appears that the maximum possible score of BDeu for Bayesian Network structure learning is zero.

Does it mean that the best network is always the empty network?

Does it mean that the best network is always the empty network?

No.

The empty DAG (i.e. the DAG with no arcs) implies the factorization is simply $$P(X_1, \dots, X_n) = P(X_1) \dots P(X_n),$$ which is telling you all variables are independent.

This will be the best network (DAG) if all the variables are actually independent (Remember that independence for a dataset entails a certain tolerance), because the penalization coefficient will make the score lower as any arc is added.


Unstructured text classification I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than three). Does anyone have a suggested for how I might get started?

Is the "bag of words" approach feasible here? Later, I could add another classification stage based on document structure (perhaps decision trees).

I am somewhat familiar with Mahout and Hadoop, so I prefer Java-based solutions. If needed, I can switch to Scala and/or Spark engine (the ML library).

Let's work it out from the ground up. Classification (also known as categorization) is an example of supervised learning. In supervised learning you have:

model - something that approximates internal structure in your data, enabling you to reason about it and make useful predictions (e.g. predict class of an object); normally model has parameters that you want to "learn"
training and testing datasets - sets of objects that you use for training your model (finding good values for parameters) and further evaluating
training and classification algorithms - first describes how to learn model from training dataset, second shows how to derive class of a new object given trained model

Now let's take a simple case of spam classification. Your training dataset is a corpus of emails + corresponding labels - "spam" or "not spam". Testing dataset has the same structure, but made from some independent emails (normally one just splits his dataset and makes, say, 9/10 of it to be used for training and 1/10 - for testing). One way to model emails is to represent each of them as a set (bag) of words. If we assume that words are independent of each other, we can use Naive Bayes classifier, that is, calculate prior probabilities for each word and each class (training algorithm) and then apply Bayes theorem to find posterior probability of a new document to belong to particular class.

So, basically we have:

raw model + training set + training algorithm -> trained model
trained model + classification algorithm + new object -> object label

Now note that we represented our objects (documents) as a bag of words. But is the only way? In fact, we can extract much more from raw text. For example, instead of words as is we can use their stems or lemmas, throw out noisy stop words, add POS tags of words, extract named entities or even explore HTML structure of the document. In fact, more general representation of a document (and, in general, any object) is a feature vector. E.g. for text:

actor, analogue, bad, burn, ..., NOUN, VERB, ADJ, ..., in_bold, ... | label
    0,        0,   1,    1, ...,    5,    7,   2, ...,       2, ... | not spam
    0,        1,   0,    0, ...,    3,   12,  10, ...,       0, ... | spam

Here first row is a list of possible features and subsequent rows show how many times that feature occurs in a document. E.g. in first document there's no occurrences of word "actor", 1 occurrence of word "burn", 5 nouns, 2 adjectives and 2 pieces of text in bold. Last column corresponds to a resulting class label.

Using feature vector you can incorporate any properties of your texts. Though finding good set of features may take some time.

And what about model and algorithms? Are we bound to Naive Bayes. Not at all. logistic regression, SVM, decision trees - just to mention few popular classifiers. (Note, that we say "classifier" in most cases we mean model + corresponding algorithms for training and classification).

As for implementation, you can divide task into 2 parts:

Features extraction - transforming raw texts into feature vectors.
Object classification - building and applying model.

First point is well worked out in many NLP libraries. Second is about machine learning, so, depending on your dataset, you can use either Weka, or MLlib.


Can we use prior information to improve the results of a neural network? If so, how do we incorporate it into the learning process? Let $f_\alpha(x)$ be the function
$$
f_\alpha(x) = x^2 + \alpha\sin(x),
$$
on the interval $[-5,5]$. Suppose $\alpha = 2$, and our goal is to learn the function $f_2(x)$ using some form of neural network. $f_2$ looks like this:

We are given a set of noisy measurements of $f_2$ from which we would like to learn $f_2$. Specifically, we are given a set of random samples $\{y_i,x_i\}_{i=1}^n$ with
$$
y_i = f_2(x_i) + N(0,\sigma),
$$
where $N(0,\sigma)$ is normally distributed noise with standard deviation $\sigma$:

Now we can use the samples $\{y_i,x_i\}_{i=1}^n$ to estimate the unknown function $f_2$. However, before doing that, note that the general behavior of $f_2$ is captured well by the function $f_0(x) = x^2$:

Now suppose that instead of just being given the set of noisy samples $\{y_i,x_i\}_{i=1}^n$ of $f_2$ we are given both the noisy samples and the function $f_0$ which we are told is a decent approximation of $f_2$. In other words, we have some prior knowledge on what the function $f_2$ generally looks like.
Can we somehow incorporate this prior knowledge of $f_2$ into our neural network learning process so that we can get a better estimate of $f_2$ than estimating it based on just the noisy samples on their own?
If so, what are our options for incorporating this knowledge into a neural network? Does the type of neural network (CNN, RNN, etc...) affect the way we incorporate the prior information?
P.S. I am coming from statistics/mathematics and while I understand the general principles of neural networks I have only just started using them.
P.P.S. Here is the Matlab code for the images
rng(123);

N = 100;
x = linspace(-5,5,N);
alpha = 2;
sigma = 4;

f_0 = x.^2;
f_alpha = x.^2 + alpha*sin(5*x);
f_sigma = f_alpha + sigma*randn(N,1).';

figure
hold on, grid on
plot(x,f_0,'k--')
plot(x,f_alpha,'b')
plot(x,f_sigma,'r.')

What you may be referring to is transfer learning.
The process of transfer learning mainly boils down to using layers of big  pre-trained models (super set) in your smaller model (subset) during training.
Additional configuration may include freezing some of the pretrained layers (no updates during training)  and unfreezing others (update weights/biases during backprop).
It is very common in recurrent networks to use pre-trained word embedding layers (word2vec, GloVe).