Using time series to predict house prices vs. multiple linear regression Re-post.

Machine Learning Courses often teach house prices prediction using multiple linear regression - when we want to predict the value of a variable based on the value of two or more other variables. Indeed the one I did on coursera and that makes sense to me.

I note that others state that a techniques like time series is also a suitable method to predict the house price, given a set of input variable.

I am not clear as to why this could be so. Simply because there is no notion of selling of a house at a regular interval afaik. That is to say, the sale date (one of the variables) can be at any time, not at a reasonably regular interval. Or is it possible because it is not just one house but many houses that can contribute to such an interval?

So, looking for a clarification as to why time series could be an approach for house price prediction. To borrow from wikipedia: A time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. Examples of time series are heights of ocean tides, counts of sunspots, and the daily closing value of the Dow Jones Industrial Average.

I agree, the simple version of this problem isn't really a time series problem. There are some ways to deal with irregular intervals, but, houses are not all the same and we do not have a series of sale prices for it in most cases.

What may of course be very predictive is time. Housing prices probably have a long-term trend that's linear over the space of a few years, at least. There is probably a seasonality component, so month might be meaningful if used correctly.


How to train the model with for loop instead of the built-in epochs i want to use a for loop for epochs instead of the built-in ones. Does these parts are similar.
1)
epoch=0;
while epoch in range(0, 30):
        print('Epoch', epoch)
        results = model.fit(X_train, Y_train,  batch_size=16, epochs=1, callbacks=callbacks)
        epoch +=1

results = model.fit(X_train, Y_train,  batch_size=16, epochs=30,  callbacks=callbacks)

You can do it creating a custom training function. I have created a whole set of TensorFlow 2 tutorials about it. It's simpler than it looks like.
This is the code of some generic training function:
import tensorflow as tf

# This loss and optimizer are just examples, use the one you need
loss = tf.keras.losses.MeanSquaredError()
optimizer = tf.keras.optimizers.Adam(learning_rate)

@tf.function
def train_on_batch(X, Y):
    with tf.GradientTape() as tape:
        current_loss = loss(model(X), Y)
    gradients = tape.gradient(current_loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return current_loss

tf.GradientTape() and tape.gradient() record the gradient calculated on the loss function. Then, optimizer.apply_gradients() produces the actual weight update (the training).
Please keep attention at the @tf.function decorator. This is a feature available in TensorFlow 2.x called Autograph. It's a super powerful trick that transforms the whole training function into a TensorFlow op, making it an order of magnitude faster. To make it work, make sure it receives numpy arrays, and that it contains exclusively TensorFlow operations.
Custom training function are extremely useful when you need to implement some non-standard training. For example, I recently trained a GAN for the imputation of missing data, and using custom training made my code easier to implement and control in all its details.
Good luck!


Where can we find the application of bayes's theorem in Bayesian optimiation with gaussian processing I am trying to learn bayesian optimisation by following this tutorial.
However, until now I don't get the relation between bayes's theorem to the gaussian process formalism.

Any ideas?

It is a 49 page long paper, so following observations are based only on a cursory reading.

The optimisation is for finding best value of parameters for cost function of machine learning models.
Rather than finding a fixed value of the parameters, it is assumed that the parameters come from statistical distribution and the task is to find the nature/shape of this statistical distribution.
Bayes theorem tells you that if you have prior beliefs and evidence (data), you can go to posterior.
They start with the assumption that prior distribution of the parameters is Gaussian. The task is to then find the posterior.
Since there are multiple parameters and not single variable, Gaussian process comes into picture rather than single Gaussian distributed random variable.
Bayes theorem comes into picture while going from priors to posteriors. The optimisation is solved using sampling technique like Monte Carlo sampling. Reading up about MCMC sampling method will help you see the connection.


Predicting the speed of a car I'm working on the commaai speedchallenge. The goal of the challenge is to predict the speed of a car based on a dashcam video. So far all the examples that I found (example 1, example 2) use some kind of method that takes in the information at frames $x_{t-h}, \ldots, x_{t-2}, x_{t-1}$ when trying to predict the speed at frame $x_t$. I think this makes sense, because you can get an idea of the speed by looking at differences between two frames. Without a reference point (with just one frame) it would be very hard to predict the car speed..

Well apparently not! While looking for solutions I also found this repository. I'll summarise the approach that is used:

Rescale the frames to 50x50 images, and divide them into two sets: X_train and X_test.
Take X_train, X_test and pass them through the first two blocks of a convolutional network called VGG16 that is already trained, so that you get output X_train_features and X_test_features. The writer of the code explains that he wants to use transfer learning because his computer specs are not good enough to train a network himself.
Train a linear regressor based on the extracted features and compute the mean squared error.

If I understand the idea correctly, the author of this code just looks at each frame individually, passes the frame through the first two convolutional blocks of the network VGG16, and predicts the speed of the car based on the extracted features, without looking at earlier frames!

I've tested the code and it works, it has a mean square error just above 1!

Question: Am I missing something, or can you predict the speed of a car just by looking at the features of a single frame? If the answer is yes; then how?

I know that you can take the weights from the early layers of trained convolutional networks and use them for other purposes. This is possible because the early layers of convolutional networks usually just filter for very general/coarse shapes. What I just don't understand is how you could predict something that is relative to other frames based on just those low level features. If a human were to look at a single frame in the data, he would not be able to predict the speed of the car correctly, right?

There are some visual indicators of speed that should be present even in one frame. An easy example to think of is "blur", looking at a static image you could still determine "movement" (and therefore velocity) if a picture is blurred vs. absolutely sharp.

Additionally perspective, depth and shape of other objects might differ depending on velocity.

So it certainly could be possible to train a model based only on static images but it certainly should not outperform a model using both features of static images and differences between frames.

Edit:

I also looked at the repo and the presented method a bit. In the used method there could be a huge over-fitting problem if the video is only one single drive!

In this case the image detection would only identify "where" the picture was taken and since any point along the drive is associated with a single speed it could work from there i.e.: when the truck drove past a palm tree he was driving at 30 mph so identifying a palm in the picture should predict 30mph which works for a single drive data set but obviously is not able generalize.


Generating a sequence based on value in another column in Python I have the following data frame:

I would like to add a column with value equal to 1 if flag is 0 and incrementally add 1 in the following rows until the next 0 is encountered (as given in the example below).

I have been able to generate the sequence, but the code is extremely slow, so is there a faster way to generate the sequence?

I don't know how you have tested it the first time, here is my logic. It supposes the first element in flag is 0!

df = pd.DataFrame({'memberid': [1]*11,
                   'flag': [0,0,1,1,0,1,0,0,0,1,1],
                   })
df['seq'] = ""
for i in range(0, len(df)):
    df.loc[i, 'seq'] = 1 if df.loc[i, 'flag'] == 0 else  df.loc[i - 1, 'seq'] + 1
print(df)

Another solution using lambda:

df = pd.DataFrame({'memberid': [1] * 11,
                   'flag': [0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]
                   })

def f(flag):
    global previous_seq
    previous_seq = 1 if flag == 0 else previous_seq + 1
    return previous_seq

previous_seq = 0
df['seq'] = df[['flag']].apply(lambda x: f(*x), axis=1)
print(df)

I am not sure if it is faster than the first solution....


How to improve deep learning model having less data I have trained a deep learning model for regression. The accuracy of the model is poor. I am quite new to deep learning. How can I improve it? The target variable Y is obtained by multiplying the features X1 and X2.
DataSet(5800 rows)
X1       |      X2      | Y
1.000000    70.000000   70.000000
0.714286    29.045455   20.746753
0.000000    35.000000   0.000000
0.538462    22.071429   11.884615
0.000000    54.000000   0.000000

Model
#Define a larger model
def larger_model():
  #Create Model
  model = Sequential()
  model.add(Dense(2, input_dim=2, kernel_initializer='normal', activation='relu'))
  model.add(Dense(6, kernel_initializer='normal', activation='relu'))
  model.add(Dense(1, kernel_initializer='normal'))

  #Compile Model
  model.compile(loss='mean_squared_error', optimizer='adam')
  return model

#Evaluate Model

estimator = KerasRegressor(build_fn=larger_model, epochs=10, batch_size=5)
kfold = KFold(n_splits=10)
results = cross_val_score(estimator, X, y, cv=kfold)
print("Results: %.5f (%.5f) MSE" % (results.mean(), results.std()))

Output
Results: -83.81452 (170.38108) MSE

First have a look at your data scale. Deep learning models are sensitive to data scaling so it would be better to preprocess your data to keep in within accetable ranges:

scale [-2,2] - apply scaling so that all features and label are within this range
mean 0: try to center your data around 0

Second, deep learning models are regularised overfitting models. One thing you can try here is to progressively increase the size of your model (i.e. more hidden layers or hidden units) and also add regularisation.
Your options are:

weight decay - keras Dense layers can be augmented with weight decay. Have a look at the documentation

layer regularisation (batch norm, dropout, layer normalisation) these are a bit more advanced and work on a case by case basis - have a look at this paper as a start.


How to save a knn model? I need to save the results of a fit of the SKlearn NearestNeighbors model:

knn = NearestNeighbors(10)
knn.fit(my_data)

How do you save to disk the traied knn using Python?

import pickle

knn = NearestNeighbors(10)
knn.fit(my_data)

# Its important to use binary mode
knnPickle = open('knnpickle_file', 'wb')

# source, destination
pickle.dump(knn, knnPickle)

# load the model from disk
loaded_model = pickle.load(open('knnpickle_file', 'rb'))
result = loaded_model.predict(X_test)

refer: https://www.geeksforgeeks.org/saving-a-machine-learning-model/


Is there a python code to perform the following task in python Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than 60%.

This is the sample rating for the product. Where a 6 represents 60%. The data is currently stored in a pandas dataframe.

  9
  6
  2
  2
  8
  5
  4
  4
  5
  4
  4
  2
  4
  5
  3
  5
  7
  3
  5
  4
  1

dataset['Z32'].describe() # gives me a output

count        24114.000000
mean         4.327445
std          2.086072
min          1.000000
25%          3.000000
50%          4.000000
75%          6.000000
max         10.000000

counts = dataset['Z32'].value_counts()
print(counts)

5     5831
4     3722
3     3384
6     2874
1     2836
2     2218
7     1451
8     1040
9      416
10     342

If dataset means pandas.DataFrame then you can use

dataset['rate'] < 6

to get dataset with values True/False which can sum because True is converted to 1 and False is converted to 0

count = sum(dataset['rate']<6)  # 17

If you get number of all rows in dataset len(dataset) then you can calculate percentage value

percent = count/len(dataset)   # 0.8 (80%)

Working example

import pandas as pd

dataset = pd.DataFrame({
    'rate': [9, 6, 2, 2, 8, 5, 4, 4, 5, 4, 4, 2, 4, 5, 3, 5, 7, 3, 5, 4, 1]
})

count = sum(dataset['rate']<6)
percent = count/len(dataset)

print(count, percent, '{:.2%}'.format(percent))

Result

17 0.8095238095238095 80.95%


Multioutput regression with MLPRegressor - Does it work? I apologize if this has been addressed before, but I did not find a solution for this issue.

I am currently trying to solve a problem with four dependent variables and four independent variables, all non-categorial. They seem to have non-linear dependence and multicolinearity - standard multivariate regression yielded reasonable results, but I am trying to compare them with results from Random Forest and Neural Networks.

According to this link, it should be possible to get multi-output with the MLPRegressor. However, when I try to apply the .fit method, I get an error message:

fit() missing 1 required positional argument: 'y'

Here is the code that I am using:

# Data Preprocessing - Importing the Libraries
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('Rock_Properties_4T.csv')

y = dataset.iloc[:, 4:8].values
X = dataset.iloc[:, 0:4].values

# ANN
from sklearn.neural_network import MLPRegressor
regressor_ANN = MLPRegressor(tol = 1e-16)
MLPRegressor.fit(X, y)

There you go... Any idea of what could be the problem?

Not sure if this makes a difference, but I think you need to call fit from the initialized object. So for example, your fit line should be:

regressor_ANN.fit(X,y) instead of MLPRegressor.fit(X, y)

then predicting a new set will be:

regressor_ANN.predict(X_test)


what is meant by minimizing and maximizing in GANs? It is a subtle change that involves the generator maximizing the log of the discriminator probabilities for generated images instead of minimizing the log of the inverted discriminator probabilities for generated images.

This is a very tricky and confusing line to understand GANs loss functions, it is taken from machine learning mastery blog. My question is about the terms minimizing the log of the inverted discriminator probabilities, isn't the subtraction of probabilities from 1 give you same thing (referring to the formula of this part of loss function). please clarify it with some kind of analogy or example of some sort.

This basically boils down to the Vanishing Gradient Problem. It is a well known issue that vanishing gradients limit the ability of deep neural nets to learn. And playing the min-max-game for GANs can lead to this issue for G as well:
When you start to train a GAN G will not produce very good results, i.e. D can easily classify these as fake or real with a probability of them being real approaching $0$. Accordingly, the expression $\log(1-D(G(z))$ converges to $0$, too, and results in a vanishing gradient for G, i.e. the generator cannot really learn when it tries to minimize this.
In contrast, $\log D(G(z))$ does not converge to $0$ when $D(G(z))$ is very small (since $\lim \limits_{x \to 0} \log x=-\infty$). Therefore, G does have a gradient to learn from.
Also see the original paper where the authors write:

Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, $\log(1 - D(G(z)))$ saturates. Rather than training G to minimize
$\log(1 - D(G(z)))$ we can train G to maximize $\log D(G(z))$. This objective function results in the
same fixed point of the dynamics of G and D but provides much stronger gradients early in learning.

As you can see, this is a purely practical consideration in order ease learning for G and not about theoretical equivalence of expressions.


What kind of clustering would work better on such data? Would k-means work on such data? I have a dataset where datapoints are more or less spread like this:

What if I want to split the data in 2 data clusters, what would be a good choice? Would k-means work here?
Thanks.

Theoretically, Gaussian Mixture Model could identify [\, /] clusters.


Best practice - memory management when data wrangling I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:

df.read_csv() #one instance of a df
df1 = df.drop_duplicates #another instance of a df, total 2
df1_melt = df1.melt() #another instance of a df
df1_aggregated = df1_melt.groupby()... #another instance, total 4
df_mutual = pd.merge(df1_aggregated, df1_melted) #created another instance of a df, total 5

In the example above, we created 5 dataframes and stored them in memory, but we're really only interested in one. I have read that you can reduce the reference count to these variables to 0, and then when you delete the variable, it may be garbage collected. Is there a better way to program in a manner that I avoid creating multiple copies of the data in memory?

Programming best practices that are specifically usefull for ML / DS :

Know and handle your data types, with a dict for example, and pass this dict as and agrument to your read csv. You can get important memory usage reduction by using the right types of float / int and category instead of string. Float precision is rarely important in ML (and usually achieve better statistical performance trough faster calculations).

As a part of your model building process, it is usefull to do some variable selection (to reduce the number of columns) or sampling (to reduce the number of rows). Sometimes just for testing your pipeline, sometimes to get a baseline, sometimes as a long run solution.

You have to actively avoid multiplying you dataframes. I usually check for that in real time by seeing what memory is consumed. (Either trough command line interface like htop, or trough ad-hoc memory requests in the python environnement). This means you have to look into your functions which are duplicating your df. Then you have some work to do to find an alternative that is low memory (using vectorized calculation, using a 'in_place = True' option ... etc). During this step you also have to check that your operations are not upgrading the data types you specified in the first part.

Clean after yourself. You have to delete the objects you don't use (the raw data after you preprocessed them, the whole data you just splitted ... etc.). Basically using del() everywhere you can and the garbage collector gc.collect() often.


Ways to Encode context for text classification? I have a binary classification task which has the following specification:

Input: Chunk of text (not more than a few sentences, mostly a sentence).
Additional Input: For each input sample there is additional information available (which is also some text of similar length, max 2-3 sentences).
Problem: Classify text content using additional context
Problem Type: Binary classification

Essentially the task boils down to classifying the content conditioned on the context or p(content|context).

I was thinking of effective ways to encode the text for classification using a deep neural network. I searched for recent works but the existing literature mostly use the technique mentioned below.

I was wondering if there is a better way to encode context? Which has shown to be effective in some domain as compared to just concatenation.

Encode then concatenate: This involves encoding the context using ways similar to the content and then concatenating the feature representation before classification. This is the widely used technique. There are a lot of variants of this technique (different ways to encode text e.g. using tfidf rep, word embedding, LSTMs, CNNs ) which are widely explored.

Are there any better ways to bring context information during classification?

P.S: Recursive RNN is something that I have on my TODO.

A Recursive Neural Network (RNN) is best suited to solve your problem. And if you want to take a step further try Recursive Neural Tensor Network. You can generate compositions for entire sentences using RNN both for your content and context data. This will help you map your sentences to a higher dimensional space and as an output you will get numerical values (or embeddings) both for your content and context. Feeding these values into a simple logistic classifier will do your task. For a brief overview on RNNs take a look at this lecture by Richard Socher.


How to sort list by parameter in python? I have a list of employee records. Each tuple of the list represent a person's record, which includes his name, ID, and age.

For example,

emp_records = [('Karim',100, 45), ('Rahim',10, 30),('Salim', 300,60),('Abu',50,35)]

Now, I want to sort emp_records based on ID.

You can use lambda function to pass argument by which you need to sort the records.

For example,

sorted_list=sorted(emp_records, key=lambda emp : emp[1])   //Here 1 means ID
print(sorted_list)

You can also use the operator module. The operator module has the itemgetter function, which takes item as argument and sort based on the argument

For example,

from operator import itemgetter
sorted_list=sorted(emp_records, key=itemgetter(1)) // Here 1 means ID
print(sorted_list)


When training an image classifier, is it best practice to equally distribute the number of images in each category? When training the model, I understand

If I supply too many on a certain category, it may become overfitted and treat almost all predictions as the overfitted category. This can lead to false positives in the overfitted directory.
If instead, I supply the least amount of photos to equally distribute, is the probability of of a rarer category appearing is now equal to a category that would be common to find? This can lead to false positives in the rare category but may turn my false positives in #1 to be true negatives.
If instead I skip training the rare category, I get false positives in other categories.

How do we account for the natural distribution of our target classifications (for example, types of common lesion vs rare lesions)?

Is it best practice to equally distribute? If so, what should we do if we have so little of the rare category, the sample images are orders of magnitude smaller?

Depending on the method that you use, this answer might be different.

First, you split the data between training, validation, and test splits. Forget about the test set. Keep the validation data in its natural distribution of classes.

You have several choices with the training set data -

Balance the classes using down-sampling or up-sampling techniques
Keep imbalanced classes but attach weights to errors in each class (See som references here:
https://stats.stackexchange.com/questions/207977/balancing-classes-for-neural-network-training)


How can I get the diameter of each community I am trying to calculate the diameter of each community in my dataset, Zachary's karate club using Jupyter. I created a loop to iterate through, but it gives me the diameter of the whole network rather than of each community.

import pandas as pd
data = pd.read_csv('zachary.txt',sep =" ", header = None)
data_values = data.values
g = Graph()
new_data = data_values.tolist()
data_graph = g.Adjacency(new_data, mode = 'undirected')
s = data_graph.community_infomap()
print(s)
s_List = list(s)
print(s_List)
for ic in s_List:
    y = data_graph.diameter(ic)
    print(y)

I expect the output to be like "$1,2,2$" or "$1,3,1$" but the actual output is "$5,5,5$", which is the diameter of the whole community.

The method diameter doesn't appear to take vertex sets as a parameter.  Instead, call diameter on the appropriate induced subgraphs:

for ic in s_List:
    y = data_graph.induced_subgraph(ic).diameter()
    print(y)

Edit: Erm, I guess this depends on exactly what you want to measure.  While it's rather against the purpose of "communities," it is possible that the distance between a pair of vertices in your community is longer if the paths are restricted to the community than if they can pass through the rest of the graph.  Inducing the subgraphs as above would miss that.


Can neural network represent a relation that is a derivative? Let's say I measure people hiking uphill, and it happens that their hiking speed is related to the slope so they hike slower where the hill is steeper. My input variable is the hill's elevation and my output variable is their hiking speed. I fit a neural net to this data, so that I can later predict the hiking speed given new elevation measurements. Can a neural net represent the input's first derivative and so do a good job capturing this relation? What about representing a second derivative, i.e. if hiking speed was related to the curvature of the hill?

Interesting question.

What you are asking for is if a neural network is able to represent and/or approximate functions.

The answer is: Yes, the Neural Networks can represent derivatives of functions.

In this paper, the possibility is explained.

Approximation of functions and their derivatives: A neural network implementation with applications

Other author take the same conclusion:

Multilayer feedforward networks are universal approximators

Generalization and Approximation Capabilities of Multilayer Networks

Approximation Capability of Layered Neural Networks with Sigmoid Units on Two Layers


Doubt on scope of text classification problem I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,29,490 rows and 2 columns: Seller and brand)

eg:

I need unique instances for each seller. That is, in each row, I have to have a seller and all the information about the brands he is selling. My idea was to create make each unique brand as a feature and create a sparse dataset. However, we have almost 2 million unique brands.
Is it possible to do some text classification and come up with some text clusters and then make each cluster as a feature? I am not sure if this is a correct approach
I am stuck without an approach now.
Can anyone help me with how should I approach this problem?

Thanks in advance

In case you have a lot of cases where the same brand is described in different ways, e.g. "K.F.C", "KFC", "Kentucky Fried Chicken", then it might be worth using clusters based on textual similarity measures. This kind of problem is similar to record linkage.

But if this is not the case, then it would be a bad idea to try to merge brands based on their names since you would end up with groups of brands which have nothing in common except some part of their name.

At a more general level, it looks to me like the design of your task is flawed: why do you need a unique instance for every seller? A seller can have multiple brands, each from a different source, right? That would mean that there are multiple sources for each seller, how do you plan to deal with that? Are you sure you don't want to predict the source for a particular pair seller + brand instead?


What exactly are the data augmentation experimental Keras' layers doing? From what I gathered, data augmentation consists in increasing your number of instances in your dataset by applying some transfromations. Let's say I want to classify images. If I apply a random rotation to every image in a data set containing $n$ images, I will obtain a new dataset with $2n$ images, $n$ pairs of the original image plus it's random-rotated counterpart.
Assuming this is true, I don't understand what keras experimental layers related to data augmentation are doing.
Take tf.keras.layers.experimental.preprocessing.RandomRotation . In the image classification tutorial, it puts this layer inside the Sequential model like this:
model = Sequential([
  layers.experimental.preprocessing.RandomFlip("horizontal",
                                                 input_shape=(img_height,
                                                              img_width,
                                                              3)),
  layers.experimental.preprocessing.Rescaling(1./255),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes)
])

This is already kind of weird because a layer produces an output from an input (obviously) but it doesn't duplicate the image. Anyway, I decided to check this in the documentation, and, in effect, this is what is happening.
Init signature: layers.experimental.preprocessing.RandomRotation(*args, **kwargs)
Docstring:
Randomly rotate each image.

By default, random rotations are only applied during training.
At inference time, the layer does nothing. If you need to apply random
rotations at inference time, set `training` to True when calling the layer.

Input shape:
  4D tensor with shape:
  `(samples, height, width, channels)`, data_format='channels_last'.

Output shape:
  4D tensor with shape:
  `(samples, height, width, channels)`, data_format='channels_last'.

Therefore, I understand that I'm just randomly rotating the image, that is, changing each image in the dataset, but I'm not doing any data augmentation. However, I find this would make no sense, otherwise they wouldn't mention this as a data augmentation procedure. So what am I missing?

if you look into the code you can figure what's exactly happening.
Take tf.keras.layers.experimental.preprocessing.RandomRotation for example
def call(self, inputs, training=True):
  ...
  def random_flipped_inputs():
    flipped_outputs = inputs
    if self.horizontal:
      flipped_outputs = image_ops.random_flip_left_right(flipped_outputs,
                                                         self.seed)
    ...
    return flipped_outputs

You will see it's gonna to random_flip_left_right on your input images, and the comments inside it is
"""Randomly flip an image horizontally (left to right).
With a 1 in 2 chance, outputs the contents of `image` flipped along the
second dimension, which is `width`.  Otherwise output the image as-is.
When passing a batch of images, each image will be randomly flipped
independent of other images."""
Example usage:
...
>>> images = np.array(
... [
...     [[[1], [2]], [[3], [4]]],
...     [[[5], [6]], [[7], [8]]]
... ])
>>> tf.image.random_flip_left_right(images, 6).numpy().tolist()
[[[[2], [1]], [[4], [3]]], [[[5], [6]], [[7], [8]]]]

So basically what it is doing is each image will have a 50% chance to horizontally be flipped, so if you have 10 epochs, image_1 will have possibility to be [original, flipped, original, flipped, ...], and so on so forth to other images. Seeing from the outside would identify we have 10 different datasets(particularly on multiple data augmentation).


When projecting data with UMAP, should I use only the samples I need projected or the entire dataset? I have a neural network that maps my data samples to a 64-dimensional embedding. I wish to visualize a few of these embeddings (between 30 and 600) through a 2-dimensional projection, and I plan to use umap to do that. Would providing more embeddings sampled from the dataset along with the ones I want to project help the algorithm to identify the manifold and improve the quality of projection?

Yes, more data will improve the quality of the embedding UMAP can produce. While UMAP is somewhat robust/stable under subsampling in general you will get significantly better results with more data. It is also worth noting that most UMAP implementations are not designed for very small datasets (they make some optimization choices that assume a a reasonable dataset size). In practice it is probably best not to use UMAP with less than 100 or so data samples.


overfit a Random Forest I am trying to overfit to the maximum a random forest classifier using scikit-learn to make some tests.

Does somebody know what  hyperparameters I can tune to do that? Or does somebody know which other model I could apply to achieve a overfitted to the maximum a non-linear model?

Decision Trees are definitely easier to overfit than Random Forests. The averaging effect (see bagging) is meant to combat overfitting.

Other than that I think the default parameters will overfit.

Example:

from sklearn.tree import DecisionTreeRegressor

# Create a dataset
x = np.linspace(0, 10 * np.pi, 50).reshape(-1,1)
y = x + 3 * np.sin(x)
noise = np.random.random(50).reshape(-1,1)
noise -= noise.mean() # center noise at 0
noisy = y + noise * 2

# Define a Decision Tree (with default parameters)
dtr = DecisionTreeRegressor()
dtr.fit(x, noisy)
y_dtr = dtr.predict(x)

# Draw the two plots
plt.figure(figsize=(14, 4))
ax1 = plt.subplot(121)
ax1.plot(np.linspace(0, 10 * np.pi, 100),
         np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),
        color='gray', label='desired fit', zorder=-1, alpha=0.5)
ax1.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)
ax1.scatter(x, noisy, label='data')
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('Model Overfit')
ax1.spines['right'].set_visible(False)
ax1.spines['top'].set_visible(False)
ax1.yaxis.set_ticks_position('left')
ax1.xaxis.set_ticks_position('bottom')
ax1.legend()

ax2 = plt.subplot(122)
ax2.plot(np.linspace(0, 10 * np.pi, 100),
        np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),
        color='gray', label='desired fit', zorder=-1, alpha=0.5)
ax2.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('Same graph')
ax2.spines['right'].set_visible(False)
ax2.spines['top'].set_visible(False)
ax2.yaxis.set_ticks_position('left')
ax2.xaxis.set_ticks_position('bottom')

ax2.legend()

Running the code below will produce the following figure:


Survival regression with major event that won't happen I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "die" (probability of the survival function to be less than 0.1 for instance).

My problem here is that I have, in my training set, a variable that influences a lot my target variable, but is not available for the test set (and won't happen in real life).

Let's say my training data is the following:

id   status   poison_time  death_time     sex
 0     1          90           92          f
 1     0          90          150          f
 2     1          90           91          f
 3     1          60          130          m
 4     0          60          150          m
 5     1          60           62          m

With :

status = 1 for a dead person and 0 for a censored data
poison_time : time corresponding to the injection of a poison
death_time : time of the death or last follow-up
sex : sex of the individual (not relevant here, imagine a bunch of useful variables)

I can't just ignore the influence of poison_time: although for some individuals, the poison won't be as effective (individual with id 3, or individuals that ended up right-censored). It has a real impact on death_time.

In my test data the poison is not injected, but I still would like to have a good idea of "how long should it take before an individual most likely die", knowing my other variables (sex, etc.)

Is it possible to still have relevant results with such corrupted data as a training set?

If I understand your problem correctly, I think it's possible¹, but you have to do some extra work and you may be limited with what models you can use.

First, you have a time-varying dataset, so that must be handled correctly. Your poison comes long after birth, and that's important to model. Otherwise you are biasing your model. Ex: Suppose everyone is poisoned at age 90, if they live that long. Then just the act of being poisoned is a signal of living a long time, when in reality, it will hasten the death of the subject. Give this a read too, for many more arguments why it's important to model time-varying covariates.

Okay, so I took your dataset, and I modelled it into a time-varying dataset (see below). Notice:

A subject could have more than 1 rows (in your case, everyone will have at most two rows, one for prior to poisoning, one for after)
Each row for a subject is mutually exclusive (see start and stop columns). The status is True iff they died at the end of that interval.
poisoned is a boolean flag for activating after they were poisoned.

    start     poisoned sex   stop  id  status
0       0            0   f     90   0   False
1      90            1   f     92   0    True
2       0            0   f     90   1   False
3      90            1   f    150   1   False
4       0            0   f     90   2   False
5      90            1   f     91   2    True
6       0            0   m     60   3   False
7      60            1   m    130   3    True
8       0            0   m     60   4   False
9      60            1   m    150   4   False
10      0            0   m     60   5   False
11     60            1   m     62   5    True

The above matrix has the exact same information, but presented differently.

Okay, so we now need to pick a model that can handle time-varying datasets. Let's take a step back and ask "after fitting a model on the training data, can I "transport" that to the testing data (which has no poisonings)?" I argue that, yes, this is valid, and will be more valid you have individuals who died before being poisoned. What your model will do is understand the risk factors associated with your variables given poisonings / no poisoning. Hence, when you move to your test data, the risk factors are already learned and prediction can be done. Here's an analogy: I'm studying the student exam scores on Canadian and American students, and have other variables like lunch, grade, background etc and possibly interactions between them. The objective of "fitting" is to learn the relationship of lunch on exam scores given the other variables. So when I focus my model on only Canadian students in the test data, my model is still valid.

(However, if I had no Canadian students in my training data, then the model's validity can be questioned if I wish to focus on Canadian students in the test data. This is why I hope there are subjects in your training data who died before being poisoned).

We can fit the time-varying dataset using Cox's proportional hazard model. The (linear) model's output looks like:

---
             coef exp(coef)  se(coef)     z    p  -log2(p)  lower 0.95  upper 0.95
poisoned    17.60  4.41e+07   2670.86  0.01 0.99      0.01    -5217.18     5252.38
sex         -0.82      0.44      1.25 -0.66 0.51      0.97       -3.28        1.64
---

From your description, it sounds like your test data is not time varying. That's fine, it still fits into the time-varying framework: simply each test subject has a single row. You can choose to leave the poisoned variable out, or just leave it as 0 throughout. The latter is probably easier, especially if you have interactions.

¹ I could be missing some important gotcha that renders this all useless. Generally, working with time-varying data is easy to mess up causality/data leakage.


Why GPU doesn't utilise System memory? I have noticed that more often when training huge Deep Learning models on consumer GPUs (like GTX 1050ti) The network often doesn't work.

The reason is that the GPU just doesn't have enough memory to train the said network. This problem has solutions like using the CPU's cache memory for storage of things that are not being actively being used by GPU.

  So my question is - Is there any way to train models on CUDA with memory drawn from System RAM? I understand that the tradeoff would involve speed and accuracy but it JUST might work to train a BERT model on my 1050ti

When you are training one batch, the GPU memory is full, i.e. it is generally needed for calculation. Time of one batch processing is like 0.1 second or less. What is your CPU-GPU bandwidth, 2 GB/s? So without loosing speed you could send at maximum 0.1 GB back and forward... Or in terms of "I want anyway", you can extend your memory by 1 GB for the price of 10x slowdown.

There are techniques of throwing out some intermediate layers' activations and recalculating them during back-propagation. It's much better idea to sacrify say half of (enormous) GPU calculation power to save some memory. I saw reports that people actually use them. But don't know how and didn't see any supports in ML frameworks


Is it true that Machine Learning is basically the study of Pattern recognition along with Optimization? I study Pattern Recognition (Classification, Clustering, NeuralNet, Decision tree, and so on. But, excluding textual analysis), and Optimization (study of computational complexity, and trade off of computation and complexity of the target hypotheses, genetic algorithm, and so on) in my university.

If somewhere else somebody studies "Machine Learning", can I assume that they are actually studying Pattern Recognition + Optimization?

That is, is machine learning the study of pattern recognition along with optimization?

Maybe? As Wikipedia says:

  The terms pattern recognition, machine learning, data mining and
  knowledge discovery in databases (KDD) are hard to separate, as they
  largely overlap in their scope.

https://en.wikipedia.org/wiki/Pattern_recognition


Is it necessary to perform rolling-window on LSTMs? Let's say I have a set of n time-series with sequence length 8

  [[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]]

And let's define the input that LSTM expects as a tensor of shape (samples,sequence-length,features)

I want to predict the last value of each one of them.

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

  [a,b,c,d,e,f,g] -> [h]

In this case the tensor would be (samples,7,1)

Or I can do the classic rolling window, with a window size of, say, 2.

  [a,b],
  [b,c],
  [c,d],
  [d,e],
  [e,f],
  [f,g] -> [h]

In effect this shortens the length of the sequence.

And the input tensor would be (samples,2,1).

What are the trade-offs between performing rolling-windows or giving the "crude" time-series to the LSTM?

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

For typical LSTM, if your input sequence is [a,b,c,d,e,f,g] then your target sequence will be [b, c,d,e,f, g, h]. You will train your network with a long sequence. And after training, you can simply pass in [h] as input to get the estimate for [i].

  Or I can do the classic rolling window, with a window size of, say, 2.

There is no point in using LSTM if your window size is 2. You will end with one input and one output. There is no longer term dependency to be learned in this setting.

You might use a fixed window approach if your individual sequence is very long. You can slice your series using the window approach. The benefit of doing this

Reduce the length of the sequence. LSTM will still have problem learning dependency over very long steps due to gradient vanishing at the forget gate.
Augment the dataset in the similar way you doing rotation and scaling for an image dataset.

Summary

Say you have a single sequence [a,b,c,d,e,f,g,h,i]. Using a sliding window of size 6, your single squence will be transformed into 4 sequences

[
    [a,b,c,d,e,f],
    [b,c,d,e,f,g],
    [c,d,e,f,g,h],
    [d,e,f,g,h,i],
]

And you input X will

[
    [a,b,c,d,e],
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
]

And your corresponding target Y will look like

[
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
    [e,f,g,h,i],
]


How to make two parallel convolutional neural networks in Keras? I created two convolutional neural networks (CNN), and I want to make these networks work in parallel. Each network takes different type of images and they join in the last fully connected layer.

How to do this?

You essentially need a multi-input model. This can only be done through keras' functional api and can work with the pretrained nets in keras.applications. To create one you can do this:

from keras.layers import Input, Conv2D, Dense, concatenate
from keras.models import Model

1) Define your first model:

 in1 = Input(...)  # input of the first model
 x = Conv2D(...)(in1)
 # rest of the model
 out1 = Dense(...)(x)  # output for the first model

2) Define your second model:

 in2 = Input(...)  # input of the first model
 x = Conv2D(...)(in2)
 # rest of the model
 out2 = Dense(...)(x)  # output for the first model

3) Merge the two models and conclude the network:

x = concatenate([out1, out2])  # merge the outputs of the two models
out = Dense(...)(x)  # final layer of the network

4) Create the Model:

model = Model(inputs=[in1, in2], outputs=[out])


Transformation from Datawarehouse into Big Data structure What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it?
How to use the schema for such transformation, how to deal with different data types like facts and dimensions for instance. What is the criteria for data separation into machines, indexes or unique keys?

Based on your comments transitioning big data sets from on-premises systems to a cloud-based system is cumbersome and fraught with challenges. However, you can use Amazon RedShift:

  Amazon Redshift is a fully managed, petabyte-scale data warehouse
  service in the cloud. You can start with just a few hundred gigabytes
  of data and scale to a petabyte or more.

The first step is to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster.
After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

  OR

Use a Hadoop environment as the landing zone to pull in data from
various sources, process it, and transfer the processed data to the
existing data warehouse or other repositories.
Explore scenarios of different ways to implement a landing zone. Learn about the architecture of the zone and the tools and techniques for integrating it with various environments.


Time series: variations as a feature I am trying to predict clients comportement from market rates.

The value of the products depends on the actual rate but this is not enough. The comportement of the client also depends on their awareness wich depends on the evolution of rates. I've added this in model using past 6 month rates as features in polynomial regression.

In fact media coverage of rate mostly depends on rate variations and I wanted to add that in my model.  The idea would be to add a derivative/variation of rate as a feature. But I anticipated something wrong, example with only two month , my variation will be of the form $x_n - x_{n-1}$ that is a simple linear combination of actual and past rates. So for a 1d polynomial regression i will have:

$$ x_{n+1} = a * x_{n} +  b * x_{n-1} + c * (x_{n} - x_{n-1})$$

instead of:

$$ x_{n+1} = a_0 * x_{n} +  b_0 * x_{n-1}$$

wich is strictly equivalent with $ a + c = a_0 $ and $b-c= b_0$. Higher polynomial degree results in a more or less equivalent result.

I am thinking about a way to include derivative information but it seems not possible. So I am wondering if all the information is included in my curve. Is this a general idea ? all information is somewhat directly contained in data and modifications of features will result in higher order objective function ?

The derivative is a linear transform, and you're using a linear model. As you've demonstrated, nothing is gained by adding linear combination of other features (like the derivative) as a new feature in a linear model.

You may want a derivative coefficient to help interpret the model. You could use the normal model $x_{n+1} = a * x_n + b * n_{n-1}$ and then report the value $a-b$ to the client. A more general approach is to examine the frequency response of the model (the derivative is a high-pass filter, your model is a FIR filter).

Alternately you can rotate the basis of the input such that the derivative is a feature and the dimensions are still independent as follows:

You can think of the independent variables $x_n$ and $x_{n-1}$ as forming a standard basis for the two dimensions of your model's input, that is $\langle 1, 0\rangle$ and $\langle 0, 1\rangle$.
Neither $a\langle 1, 0 \rangle$ or $b \langle 0, 1 \rangle$ are orthogonal to  c$\langle1, -1\rangle$.

Therefore, if you want to use $c(x_n - x_{n-1})$ as a feature, I would also use the orthogonal feature $d(x_n + x_{n-1})$. That is,

$$ x_{n+1} = c(x_n - x_{n-1}) + d(x_n + x_{n-1}) $$.

You could think of this model as the derivative plus the integral.

The same logic applies to the choice of scaling function for the Haar wavelet.


Why n-split is not possible for a dataframe with KFold? On running below code on python 3.7, I am getting the following response:
'DataFrame' object has no attribute 'n_splits'.
How to get rid of this?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sklearn.model_selection as train_test_split
from sklearn.tree import DecisionTreeClassifier
%matplotlib inline
import seaborn as sns
sns.set()
from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

train=pd.read_csv(r"C:\Users\Pranjal Pathak\Desktop\kaggle-titanic-master\Digit recognizer kaggle\train.csv")
test=pd.read_csv(r"C:\Users\Pranjal Pathak\Desktop\kaggle-titanic-master\Digit recognizer kaggle\test.csv")

train1=train.drop('label',axis=1)
test1=train['label']

Kfold=KFold(n_splits=30,shuffle=True,random_state=0)

clf=KNeighborsClassifier(n_neighbors = 20)
scoring='accuracy'

score=cross_val_score(clf,train1,test1,cv=KFold,n_jobs=1,scoring=scoring)

There is a typo error in the below line. Change KFold to Kfold below and try.

score=cross_val_score(clf,train1,test1,cv=KFold,n_jobs=1,scoring=scoring)


How do I interpret the output of linear regression model in R? I have the following linear regression model and its analysis. There are a few errors, but I am not very sure about the errors. I have not succeeded in finding them so far.
First, the 95% confidence interval for the slope should be

So the calculation is wrong.
Second, I'm not sure about the interpretation of the confidence interval. How would you interpret it in the context ?

So, the question is centred around the meaning behind a confidence interval.
The main principle behind confidence intervals is the following:
It is very costly and time-inefficient (if not impossible) to sample the whole population (i.e. all UCLA students from China and Hong Kong) and measure their cultural adjustment. Therefore, we can take a sample from this population (i.e. 200 students).
From this sample we can then develop a linear model between input features (i.e. country) and the level of cultural adjustment and establish the slope of the model.
The slope alone only tells us the slope for the linear model trained on these 200 students. We want to have an estimate of the slope that we have at least 95% confidence that it lies in a particular range. Hence, confidence intervals.
Here is an article on confidence intervals for further reference: https://www.simplypsychology.org/confidence-interval.html


train NER using NLTK with custom corpora (non-english) must use StanfordNER? I have searched about customization NER corpora for trainig the model using NLTK library from python, but all of the answer direct to nltk book chapter 7 and honestly makes me confuse how to train the corpus with correct flow and data set that has structure like this below:
Eddy N B-PER
Bonte N I-PER
is V O
woordvoerder N O
van Prep O
diezelfde Pron O
Hogeschool N B-ORG
. Punc O

I have some questions:

I found so many article that if you will train customed corpora using NLTK, there uses StanfordNER library too, should it be? or we can use pure of NLTK library for it?
Should the grammar pattern be included if you want to apply it to other languages? How is the flow?

And please give me example of code to train custom corpora until give the tag of POS Tag and NER label output using data like data structure above if you have. Thank you.

It's true that the nltk book doesn't seem clear about this. Traditionally NER models are trained with Conditional Random Fields so I searched for "nltk crf" and found this SO question which points to this detailed example for NER.
To answer your questions:

nltk itself doesn't appear to propose a CRF model, the example above relies on an interface with CRFSuite (as mentioned in the SO question). It's probably possible to use other interfaces such as StanfordNER.
The tricky part is to define your own features, i.e. the conditions that the model uses as features at every step in the sequence. This is how you can specify any kind of specific "grammar rule" you want the model to use.

The example above looks complete but I didn't test anything.


A neural network that keeps learning as new training data becomes available I have a text-classification problem with a lot of training data. Running cross-validation takes a lot of time - several days or even weeks. In order to make the system more responsive, I am thinking of the following scheme:

Train a network on instances 1,...,100.
Test it on instances 101,...200; output the accuracy.
Train the existing network on instances 101,...200;
Test it on instances 201,...300; output the accuracy.
And so on.

Ideally, I would like that:

Training on each additional 100 instances will take a constant time (i.e, I will not have to re-train the network on all previous instances);
The trained network will have the combined "wisdom" of all previous instances (so its accuracy will tend to improve with time).

Is this possible to do with standard deep-learning tools (e.g. dynet)?

You can achieve that by training the network using batches of 100 examples. In this case, you train on a batch and update the neural network parameters, you predict on the second batch to estimate the accuracy and then you use the examples of this second batch to continue training the network and so on and so forth.

Batch training (a form of online learning) is implemented on every framework, see for instances here for Keras.


What the differences between self-supervised/semi-supervised in NLP? GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-supervised learning. So what the key differences between them?

Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn question answering right? Because you are able to retrieve relation between question and answer from data.
Or in modeling documents you need sentences which are similar and sentences which are dissimilar in order to learn document embedding but these detailed labels are usually not available. In this case you count sentences from same document as similar and sentences from two different documents as dissimilar and train your model (example idea: you can run a topic modeling on data and make similar/dissimilar labels more accurate). It is called self training.


Acurracy = 0 using tensorflow on tabular data I am new to machine learning and trying to apply this tutorial to my tabular data. My input is a pandas dataframe containing the features (I encoded the categorical columns as floats) and a pandas series containing the labels, which are ints.
This is my code:

def df_to_dataset(dataframe, labels, shuffle=True, batch_size=32):
   ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
   if shuffle:
      ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

feature_columns = []
# numeric cols
for header in list(X_train):
  feature_columns.append(feature_column.numeric_column(header))
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
batch_size = 32
train_ds = df_to_dataset(X_train, y_train, batch_size=batch_size)
val_ds = df_to_dataset(X_val, y_val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(X_test,y_test, shuffle=False,
  batch_size=batch_size)

model = tf.keras.Sequential([
   feature_layer,
   layers.Dense(128, activation='relu'),
   layers.Dense(128, activation='relu'),
   layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
          loss='binary_crossentropy',
          metrics=['accuracy'],)
          #run_eagerly=True)

model.fit(train_ds,
      validation_data=val_ds,
      epochs=5)
loss, accuracy = model.evaluate(test_ds)
print("Accuracy", accuracy)

First of all, when I print the shape of the datasets, all feature shapes are shown as (?,). I looked that up, but it didn't really get, what that meant, so I moved on.
When executing the model, I got the following output:

W0625 16:28:50.013361 140172694484864 deprecation.py:323] From
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180:
add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops)
is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Epoch 1/5
2437/2437 [==============================] - 12s 5ms/step - loss:
-368933040.3744 - acc: 0.0000e+00 - val_loss: -1374389959.0878 - val_acc:
0.0000e+00
Epoch 2/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-4239125012.7993 - acc: 0.0000e+00 - val_loss: -8055676778.8942 - val_acc:
0.0000e+00
Epoch 3/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-14449097654.0468 - acc: 0.0000e+00 - val_loss: -21844830544.8532 - val_acc:
0.0000e+00
Epoch 4/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-32560744568.1740 - acc: 0.0000e+00 - val_loss: -44181551604.6596 - val_acc:
0.0000e+00
Epoch 5/5
2437/2437 [==============================] - 11s 4ms/step - loss:
-60235093753.8022 - acc: 0.0000e+00 - val_loss: -76823729189.8015 - val_acc:
0.0000e+00
1219/1219 [==============================] - 3s 2ms/step - loss:
-78553874677.2896 - acc: 0.0000e+00
Accuracy 0.0

Obviously something is wrong, but I don't even know where to begin debugging this, so any tips are welcome!
Edit: the labels I am trying to predict are point scores from 80 to 100. Because of this, I switched from binary_crossentropy to mean_squared error. Now the loss is still very large, but positive and the accuracy is still 0

Accuracy is for classification problems (and usually not the best metric anyway). Now, with integer outputs you could treat this as a multiclass classification problem, but you'd need to encode things properly for tensorflow to understand. I'd guess (without really knowing your data, mind) that you'd be better off with regression (so no cross-entropy either).

You also can't use a sigmoid activation on the last layer; it's forcing output predictions between 0 and 1. If you go for regression, you won't need to adjust you true labels, you can use MSE as your metric, and you can just put a linear activator on the last layer.


Difference between a parameter and a weight? Is there any pedantic difference between referring to a specific "parameter" of a model as $\theta_1$ or referring to a specific "weight" of that model as $w_1$?

Are they always the exact same thing?

What you are referring to as parameter is rather called hyper-parameter. I also assume that you are talking about weights as in a neural network, so that would be your model's parameters.

Basically, the difference is that hyper-parameters are chosen by the user prior to learning (and affect the learning phase), while the parameters are computed by the algorithm during the learning process. Thus, the model's parameters depend on training data and the model's hyperparameters.

Note: Things are not always that simple, because optimizing hyper-parameters is often mandatory, and in many cases you could say that they are learnt from the validation phase (model assessment over the validation dataset), so they may also depend on the dataset.


Text vectorizer that capture feature offset in the text? I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification.
I believe the information I need tends to be in the beginning of the document, so I would like to somehow capture the offset of each feature per document (either of the first appearance, or the mean offset over of all appearances).
Is there some vectorizer that can do that? or some other method of extracting this information efficiently?

Thank you!

One approach is to create another matrix that stores this information. Scikit-learn stores text features in a  document-by-token matrix. The cells of this matrix would be the token index in a document. This matrix then could be used as features during modeling.

It would require writing a custom vectorizer which would be similar to scikit-learn's CountVector implementation.


How to use fine tuning of BERT when i have unlabelled dataset of text documents? I have gained a basic understanding of using BERT for various NLP/text mining tasks. When it comes to fine-tuning of BERT, I always see that fine-tuning is performed using some classification tasks. So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents? I'm aware that the BERT model is originally trained on unlabelled data, so there must be some way.

So, how should I refine the word/sentence embeddings vector given by
  the BERT model in the case when I have a set completely unlabelled set
  of documents?

What are you looking to achieve with these unlabelled documents? If you are looking to classify them, then there is no way of getting around getting labels and fine-tuning on them.

  I'm aware that the BERT model is originally trained on unlabelled
  data, so there must be some way.

If you don't need to perform any specific end task (like classification) with the data, and instead is just looking to train BERT using your own data, then there is a way. BERT is self-supervised, meaning that it takes unlabelled data and automatically generates labels that it can train on. The same methods can be used to train on any large language corpus. However, this will only train the embeddings and fine-tuning on labels will still always be needed for classification.

I think huggingfaces' blog post is a good starting point if you want train a general BERT on your own dataset.

You can read about how BERT is trained on generated labels from text here.


Euclidean distance for more than two datapoints I have n 3-dimensional vectors. Is there a way to find distance between all using euclidean distance?

You can use sklearn's euclidean_distances function.

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.euclidean_distances.html#sklearn.metrics.pairwise.euclidean_distances

In [1]: from sklearn.metrics.pairwise import euclidean_distances

In [2]: import numpy as np

In [3]: x = np.random.rand(10, 3)

In [4]: euclidean_distances(x)


A trick used in Rademacher complexity related Theorem I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11):

$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m}  g(z'_i)-g(z_i)\right]=E_{\boldsymbol{\sigma},S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right] \\
&\text{where } {\Bbb P}(\sigma_i=1)={\Bbb P}(\sigma_i=-1)=\frac{1}{2}
\end{align*}$$

It is also shown in the lecture pdf page 8 in this link:
https://cs.nyu.edu/~mohri/mls/lecture_3.pdf

This is possible because $z_i$ and $z'_i$ can be swapped.  My question is, why can we swap $z_i$ and $z'_i$?  In the book, it says this is possible because "we are taking the expectation over all possible $S$ and $S'$, it will not affect the overall expectation."  Unfortunately, I don't understand the meaning or intuition of this part.

Thank you very much for reading the question and for your time!

This requires hell of a derivation, but I liked the question :)

  My question is, why can we swap $z_i$ and $z'_i$?

The key insight is that notation $S \sim  \mathcal{D}^m$ is equivalent to $Z_1 \sim  \mathcal{D}, \cdots, Z_m \sim  \mathcal{D}$. This translates to
$$E_{S,S'}[.]=E_{Z_1,\cdots,Z_m,Z'_1,\cdots,Z'_m}[.]$$
which remains the same by reordering the $Z$s.

Therefore the swap argument can be shown by

Singling out an arbitrary term $k$ from $\sum_{i=1}^{m}$,
Switching $E_{Z_k,Z'_k}$ to $E_{Z'_k,Z_k}$, and
Renaming the variables.

That is,
$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right] \\
&\overset{\text{expand}}{=}
E_{\color{blue}{Z_1,\cdots,Z_m,Z'_1,\cdots,Z'_m}}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right]\\
&\overset{\text{separate}}{=}
E_{\cdots,Z_k,\cdots,Z'_k,\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}} \frac{1}{m}\left(\color{blue}{g(z'_k)-g(z_k)}+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] \\
&\overset{\text{switch}}{=}
E_{\cdots,\color{blue}{Z'_k},\cdots,\color{blue}{Z_k},\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(g(z'_k)-g(z_k)+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] \\
&\overset{\text{reorder}}{=}
E_{\cdots,Z'_k,\cdots,Z_k,\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(\color{blue}{-(g(z_k)-g(z'_k))}+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] (*)
\end{align*}$$
Now by renaming $Z_k \leftrightarrow  Z'_k$, and thus $z_k \leftrightarrow z'_k$ in $(*)$ we have:
$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right] \\
&\overset{\text{rename}}{=}  E_{\cdots,\color{blue}{Z_k},\cdots,\color{blue}{Z'_k},\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(-(g(\color{blue}{z'_k})-g(\color{blue}{z_k}))+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right]\\
&\overset{\text{collapse}}{=} E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(-(g(z'_k)-g(z_k))+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right]\\
\end{align*}$$

which corresponds to $\boldsymbol{\sigma}=(\sigma_1=1,\cdots,\sigma_k = -1,\cdots,\sigma_m=1)$.

We have proved this equality by switching the sign of arbitrary index $k$. Therefore, by defining
$$f(\boldsymbol{\sigma}):=E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right]$$
we have shown that
$$\forall \boldsymbol{\sigma_1}, \boldsymbol{\sigma_2},f(\boldsymbol{\sigma_1})=f(\boldsymbol{\sigma_2}).$$
For example:
$$\begin{align*}
f(\boldsymbol{1})&=f(\sigma_1=1,\cdots,\sigma_k=1, \cdots,\sigma_m=1)\\
&=f(\sigma_1=1,\cdots,\sigma_k=-1, \cdots,\sigma_m=1)
\end{align*}$$
Knowing that there is $2^m$ equi-probable vectors $\boldsymbol{\sigma_i}$, we finally have
$$\begin{align*}
E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m}  g(z'_i)-g(z_i)\right]&=f(\boldsymbol{1}) \\
&=\frac{1}{2^m} f(\boldsymbol{\sigma_1})+\cdots+\frac{1}{2^m}f(\boldsymbol{\sigma_{2^m}})\\
&=E_{\boldsymbol{\sigma}}[f(\boldsymbol{\sigma})] \\
&=E_{\boldsymbol{\sigma},S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right]
\end{align*}$$
$\square$


After a Feature Scaling do i have the same initial information? I'm studying the gradient descent algorithm for single hidden layer neural networks.
Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.

Why mathematically do the normalized features carry the same information of the initial features?

I assume by mean normalization, you mean scaling each feature by subtracting the mean and dividing by standard deviation:

$$
x_{\text{scaled}} = \frac{x - \bar{x}}{\sigma}
$$

where $x$ is a feature.

Even though you are changing all of the values of $x$, they are each being scaled by the same amount - initially, a translation by a constant (subtracting the mean), and then scaling by a constant (dividing by standard deviation).

Here's a two-dimensional, randomly generated dataset generated with scikit-learn's make_blobs function (left) and the scaled version using the above equation for the $x$- and $y$-coordinates (right):

The $x$- and $y$-values for each point have changed, but they are still all in the same place relative to each other. If you look closely, you can see that the structure of the data is identical, even though it has been scaled a bit, and you could go back to something that looks more like the original by simply 'zooming in' on the data. Because the structure of the data is the same, we say no information was lost.

Now consider a transformation where we only take the $x$-value, and set all $y$-values to $0$:

The structure of the data has changed, and there is no way to return to the original data by scaling or stretching space uniformly, so we say information was lost here. This is an extreme example, but hopefully it illustrates the point.

One way to think about it is to think if it is more or less difficult for a classifier to distinguish between the classes after the transformation. In the first case, we can draw a line that perfectly separates the two clusters just as easily with the original data or the normalised data, but in the second case, there is no such line that separates the transformed data.

By the way, if you normalise each example rather than each feature (as asked in your comment), for this data, you end up with something that looks like this:

where all points land on either $(-1,1)$ or $(1,-1)$. This makes sense, because normalisation makes the range of the values span from $-1$ to $1$. When there are only two dimensions, one of them has to become $-1$ and the other has to become $1$. Hopefully it's fairly obvious that information is lost here, and it's generally not a good idea to do this.

This is quite a hand-wavy explanation and doesn't really cover any actual information theory concepts, but hopefully it gives you some intuition for this. If you want to dive deeper into the mathematical side of things, have a look at the Wikipedia article for information theory.


Using sklearn knn imputation on a large dataset I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer.
Trying this off the bat I hit memory problems, but I think I can solve this by chunking my dataset... I was hoping someone could confirm my method is sound and I haven't hit any gotchas.
The sklearn KNNImputer has a fit method and a transform method so I believe if I fit the imputer instance on the entire dataset, I could then in theory just go through the dataset in chunks of even, row by row, imputing all the missing values using the transform method and then reconstructing a newly imputed dataset.
I'm wondering if there's an issue with this method regarding the chunksize or is the transformation on each new row independent?
50% of the dataset rows are fully populated... would it be better in terms of computation to fit the imputer object on only this portion of the dataset?

You could use a memmap
import numpy as np
from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'newfile.dat') # or you could use another dat file that already constains your dataset

# supposing your data is loaded in a variable named "data"

fp = np.memmap(filename, dtype='float32', mode='w+', shape=data.shape)
fp[:] = data[:]

You can check full documentation on this page, the code above is based on the documentation.
This way you:

only change the declaration of your data matrices, keeping you code as clean as possible
uses numpy built-in ndarray subclass without having to explicitly manage data retrieval from the disk


Machine Learning: Why do the error in cost function need to be squared? I have recently started Andrew NG's Machine Learning course on Coursera and I came across this cost function which is:

Why does the error in the cost function need to be squared? If its purpose is to eliminate the negative sign in the error then why don't we simply use the absolute function?

The simple answer is that it's conviences rather than necessities.  You're more than welcome to take the absolute value and in many cases it may be better to do so.  Squaring a function makes the math a bit happier and easier and for proofs has desirable properties.  While you can basically do the same proofs with an absolute value function, you may have to handle certain edge cases and just amounts to more writing.


Marketing Mix Model (MMM) using XGBoost? Does anyone know of any literature on Marketing Mix Modelling (MMM) using XGBoost? Is this a viable techniques for MMM modelling? What would the advantages/disadvantages be?

I think it would be a great option due to that interactions effects are dealt with in a good way, and the model also avoids inflated values. However, I am not sure of whether XGBoost is the best model for dealing with the multi colliearity often present in MMM data.

I would be happy for any thoughts on this.

Hi and welcome to the forum. As far as I know, MMM are causal models with the purpose of making statistical inference. This is the standard world of econometrics, where people stick to OLS or logistic regression since results are easy to understand based on estimated coefficients.

Boosting (XGBoost is only one implementation of specific type(s) of boosting), is a technique which is mainly applied in predictive modelling since you have no "easy to understand" coefficients. So if your MMM application is not primarily about making a prediction, maybe boosting is not a good idea. However, you can still calculate pseudo marginal effects based on predictions, so there might be a way to make boosting applicable for inference (similar like inference is done in logistic regression).

It is not clear what you mean when you say: "due to that interactions effects are dealt with in a good way". Since boosting (usually) is based on very small decision trees, there are no interaction terms. What boosting does is, it focusses on "hard to predict" cases by weighting. So with boosting you usually get much better results (lower SSR, higher ROC etc) compared to OLS, logit etc.

I don't know if and how boosting has been applied to MMM, but generally speaking, there are (in the moment) few applications of boosting in economics. So applying boosting can be innovative in this field, provided that you get it right. This requires a thoughtful definition of model requirements and objectives of the modelling approach.


What advantage does Guassian kernel have than any other kernels, such as linear kernel, polynomial kernel and so on? Guassian kernel is so important in SVM as we know. The parameter gamma is designed for this kind of kernel. My question is what makes Guassian kernel so unique? What advantage does it have over other kernels?

The advantange is that it is related to euclidean distance, and if your problem benefits from this type of feature, e.g. you have a highly non-linear problem where values nearby some vector point tend to be of a different class, it will be able to find relationships that other kernels would not, and create highly complex decision boundaries.


What to replace outliers with? (supermarket transaction data) I have a transaction dataset from a supermarket. Let's say the average spend is $50.
I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend.
For example:
John Doe's average spend is 150% of the overall average spend = "Gold Customer".
Jane Doe's average spend is 25% of the overall average spend = "Bronze"
etc
etc
Now to calculate the overall average spend of the supermarket, I want to get rid of outliers. This is a grocery store, but they may have a TV on sale every now and then. So there are some single transactions that may be $600+. I want to get rid of these.
The question is, what do I replace them with?
I look for transactions 3 standard deviations above the mean. These are my outliers.
I don't want to replace them with the mean / median. I could be losing some of my "Gold" customers if I erase their bigger purchases and replace them with $50.
Could I replace the outliers with mean + 3*std_dev ?
I'm using python, so the current code is:
# set threshold above which transaction will be labeled an outlier
# this is the average spend plus 3 times standard dev

value_threshold = (df['amount'].mean()+(df['amount'].std()*3))

# now replace any outlier with the value threshold.
# this will ensure any big spenders stay big spenders so I can rank them accordingly

df['amount'] = np.where(df['amount']>value_threshold ,value_threshold ,df['amount'])

Does my approach make sense or am I breaking any rules?
Should I be using the median instead with interquartile ranges to find outliers?
Distribution:

After removing the outliers using my method above. Notice we still have big spenders in the data to the right of the chart. (my gold customers)

Bonus
Boxplot of my transaction data before fixing the outliers. It's horrific:

Simplest approach
First a very fast and perhaps practical approach: just remove them without replacing them!
From your bar chart, it seems you have a lot of transactions - several hundred thousand. Removing a few hundred (I can't even see a bar for the > $600 transactions) and not replacing them wouldn't mean the remaining data is unusable. Replacing those transaction e.g. using the mean of the distribution, is basically trying to make sure they don't have any significant effect on the model; so what is the point in replacing them?
Looking at your frequency chart, I would be interested to know what appears in the final large peak. Could you just remove those transaction?
You could give this a go and move onto the modelling, see if you get sensible results - if not, come back to investigate further.

Back to statistical methods
Using 3 standard deviations isn't a bad approach - assuming your data is normally distributed, it means you only remove 0.3% of the data. The issue is perhaps that your distribution of transaction amounts does not look normally distributed - it looks more like a beta distribution(the orange line):

You could therefore try your approach of removing data outside the distribution, by computing the parameters of the beta distribution, and using those instead of the normal mean/std.
To do this, check out scipy.stats.beta, and be aware that you should probably normalise your transactions to the range [0, 1.0] (see Scikit-Learn's StandardScaler). Here is a thread on understanding the output of your beta distribution.

Impute
If you simple treat your outliers as missing data, there are some nice ideas and explanations about filling missing gaps in your data (a.k.a. data imputation) in this well-known book: Elements of Statistical Learning (pdf)$^1$ - see section 9.6.
$^1$Authors: Trevor Hastie, Robert Tibshirani, Jerome Friedman


How to get vectors of memory cell & the last output of $LSTM$ in keras? In this research paper the following paragraph appears,

  The state of every LSTM model is stored in two fixed-size
  vectors of real numbers called the memory cells and the last
  output. Since our LSTM model is trained to predict user’s
  behavior, elements of these vectors are the natural candidates for the user-dependent features. They can be extended
  by the resulting predictions (answers to the questions). This
  way 218 new features are obtained from the memory cells
  (100) and the last output (100) of the second LSTM layer
  and from the final output (18).

I am aware of getting weights of each layer. But how to get these two vectors?

Self answering after reading this article. Actually, the paragraph is saying that $218$ new features are obtained from -

Memory cells or Memory units of 2nd LSTM layer: It will give a vector of $100$ cell states (cell state corresponding to each memory cell). Note that the dimension of the vector is equal to the number of hidden memory cells.
Last output of 2nd  LSTM layer: It will give a list of $100$ cell outputs (cell output of each memory cell/unit).
Final output: Final output would eventually comprise of an $18$ dimensional vector. Dimensions are - $2$ for (a), $6$ for (b), $4$ for (c), $3$ for (d), $2$ for (e), $1$ for (f).

From the same article, it can be achieved in keras using return_state=True parameter.

from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
from numpy import array
# define model
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(2, return_sequences=True, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
# define input data
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
# make and show prediction
print(model.predict(data))

This ouputs,

[array([[[-0.00559822, -0.0127107 ],
    [-0.01547669, -0.03272599],
    [-0.02800457, -0.0555565 ]]], dtype=float32), array([[-0.02800457, -0.0555565 ]], dtype=float32), array([[-0.06466588, -0.12567174]], dtype=float32)]


xgboost and linear regression new feature analysis For linear regression, seems like a new feature has to be a linear relation with the target variable.

But If you make the new feature for the Xgboost, what do you have to consider to make a new feature for a xgboost or lightgbm ?

There is nothing that needs to be preconsidered when a new feature is presented to LightBGM or XGBoost.

Even though the two mentioned methods work are nonlinear methods, there is no restriction on use them, the method itself will select or not the new features.


Backpropagation derivation problem I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.

I followed this tutorial  in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

x_data = np.random.rand(500) * 15.0
y_data = [sin(x) for x in x_data]

ETA = .01

layer1 = 0
layer1_weights = np.random.rand(10) * 2. - 1.
layer2 = np.zeros(10)
layer2_weights = np.random.rand(10) * 2. - 1.
layer3 = 0

for loop_iter in range(500000):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    layer2 = layer1_weights * layer1

    # layer 3
    layer3 = sum(sigmoid(layer2) * layer2_weights)

    # error
    error = .5 * (layer3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2
    error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)
    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1
    error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights

    if loop_iter % 10000 == 0:
        print(error)

The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.

Here's the Tensorflow code it works flawlessly with:

x_data = np.array(np.random.rand(500)).reshape(500, 1)
y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)

x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])
W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))
hidden1 = tf.nn.sigmoid(tf.matmul(x, W))
W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))
output = tf.matmul(hidden1, W_hidden)

loss = tf.square(output - y_true) / 2.
optimizer = tf.train.GradientDescentOptimizer(.01)
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(500000):
    rand_index = np.random.randint(0, 500)
    _, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],
                                              y_true: [y_data[rand_index]]})
    if i % 10000 == 0:
        print(error)

sess.close()

I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).

What is important:

Added biases.
Altered your error_w such that they are correct.
Made some good random starting points for biases (np.random.rand(width) * 15. - 7.5) such that all biases are random points on the desired x-scale.
Made a plot that shows the initial guess and final.

Let me know if some parts are not clear:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

def guess(x):
    layer1 = x
    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3
    return a_3

x_data = np.random.rand(500) * 15.0 - 7.5
y_data = [np.sin(x) for x in x_data]

ETA = 0.05
width = 10

layer1_weights = np.random.rand(width) * 2. - 1.
layer1_biases = np.random.rand(width) * 15. - 7.5
layer2_weights = np.random.rand(width) * 2. - 1.
layer2_biases = np.random.rand(1)* 2. - 1.

error_all = []

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

epochs = 500000

for loop_iter in range(epochs):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    #TODO add the sigmoid function here

    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)

    # layer 3
    #TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3

    # error
    error = .5 * (a_3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2

    # error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)

    delta = (a_3 - y)

    error_wrt_layer2_weights = delta * a_2
    error_wrt_layer2_biases = delta

    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1

    # error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1
    error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1
    # error_wrt_layer1_weights = 0

    error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)

    # a = 0
    # while a ==0:
    #   a*0

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights
    layer2_biases -= ETA * error_wrt_layer2_biases
    layer1_biases -= ETA * error_wrt_layer1_biases

    error_all.append(error)

    if loop_iter % 10000 == 0:
        print(error)

# plt.plot(error_all)
# plt.show()

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()


Why linear regression feature coefficients become super large? Introduction

I've implemented linear regression using sklearn and after all calculations I've got results like this:

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
Feature: 2, coef: -2.85809
Feature: 3, coef: 1.50522
Feature: 4, coef: -1.07076

Data

My data is based on user visits in gym. All data normalized 0 <= x <= 1. Data set has 10k observations.

X:

feature_0: gym's rating
feature_1: gym's review(rating) count
feature_2: gym's one visit price
feature_3: gym's unlimited subscription price
feature_4: distance to gym from user's home | calculated min(x / 30, 1.0), because mean is 15.17

Y: user's visit count to that gym

Data sample

Code

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot
from numpy import loadtxt

# define dataset
x = loadtxt('formatted_data_x.txt')
y = loadtxt('formatted_data_y.txt')
# define the model
model = LinearRegression()
# fit the model
model.fit(x, y)
# get importance
importance = model.coef_
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: %0d, coef: %.5f' % (i,v))

Question

Why linear regression feature coefficients become super large? Is it okay?

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
...

P.S: I'm new to this "part" of StackExchange and ML\DS at all, so please if I do something wrong or I have to provide more information, let me know! Any help would be appreciated. Thanks in advance!

Large coefficients in linear regression are not necessarily a problem. They can be large becuase some variable was rescaled. You mentionned that you do some rescaling, but provide no details. Therefore it is not possible to tell what exactly is going on.

Here is a (general) example that explains how coefficients can get "large" (in R). Assume we want to model "visits" ($y$) contingent on "rating" ($x$):

# Data
df = data.frame(c(1,3,5,3,7,5,8,9,7,10),c(34,54,31,45,65,78,56,87,69,134))
colnames(df)<-c("rating","visits")

# Regression 1
reg1 = lm(visits~rating,data=df)
summary(reg1)

The regression results are:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   19.452     15.273   1.274   0.2385
rating         7.905      2.379   3.322   0.0105 *

This tells us, that visits increase by about 7.9 when rating increases by one unit. This is basically a linear function with intercept 19.45 and slope 7.9. Since our model is
$$ y = \beta_0 + \beta_1 x + u ,$$
the corresponding (estimated) linear function would look like:
$$f(x) = 19.45 + 7.9 x .$$

We can predict and plot our model. The results are just as expected, a positive linear function.

# Predict and plot
pred1 = predict(reg1,newdata=df)
plot(df$rating,df$visits,xlab="Rating",ylab="Visits")
lines(df$rating,pred1)

Now comes the interesting part: I do a linear transformation on $x$. Namely, I divide $x$ by some "large" number and I run the same regression as before:

# Transform x
large_integer = 10000000
df$rating2 = df$rating/large_integer
df

   rating visits rating2
1       1     34   1e-07
2       3     54   3e-07
3       5     31   5e-07
4       3     45   3e-07
5       7     65   7e-07
6       5     78   5e-07
7       8     56   8e-07
8       9     87   9e-07
9       7     69   7e-07
10     10    134   1e-06

# Regression 2 (with transformed x)
reg2 = lm(visits~rating2,data=df)
summary(reg2)

The results are:

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.945e+01  1.527e+01   1.274   0.2385
rating2     7.905e+07  2.379e+07   3.322   0.0105 *

As you see, the coefficient for rating is rather large now. However, when I predict and plot, I get basically the same results as before. The only thing that has changed is the "scale" of $x$ (the way $x$ is expressed).

Let's compare the coefficient for rating in both regressions.

In the first case it was:

# Relevant coefficient "rating" from reg1 (the "small" one)
reg1$coefficients[2]

  rating
7.904762

In the second case it was:

# Relevant coefficient "rating2" from reg2 (the "large" one)
reg2$coefficients[2]

 rating2
79047619

However, when I divide the coefficient rating2 by the same "large" number as I did to "rescale" the data, I get:

# "Rescale" large coefficient
reg2$coefficients[2]/large_integer

 rating2
7.904762

As you can see, the "rescaled" coefficient rating2 is exactly the same as the original coefficient for rating.

What can you do to check your regression:

Run the regression without any rescaling and see if the results make sense
Make a prediction from the regression
Rescale your data (i.e. "standardise"), which should contribute to get better predictions because data are less "wonky" in this case. However, coefficients have no natural interpretation any more
Compare standardised data to non-standardised to see how your data changed. Based on the discussion above, you should get a good idea if very small or large coefficients can make sense after standardisation
Make a prediction, compare to the prediction from above


Definition - center of the cluster with non-Euclidean distance I have cluster consisting of two points and I am working with non-Euclidean distance. I wonder if it is appropriate to talk about center of cluster in this case and how we can interpret it? We can start with simpler case when distance is weighted Euclidean.

Update: Here is clarification. Assume we are using Manhattan distance to find centroid of our 2 point cluster. As far as I understand centroid is not unique in this case if we use PAM algorithm. We need to work with whole set of centroids  for one cluster. Hence, clustering might produce random results on each iteration. It looks to me that problem is not well posed.

The easiest solution for a non-Euclidean cluster center is the medoid, as in the algorithm PAM. This works with arbitrary metrics, but unfortunately for a 2-element cluster the result by definition is random (by metric properties, each point is an equally good medoid).

Or you step back and rethink what the center is. It is the point which has the least squared error. So "all" you need to do is solve some math for your distance function, how to optimize this least-squares estimation. People have been solving such problems since Gauss. For example with the deviation abs(x-mu) the least-squares objective is minimized by the arithmetic mean, an the one dimensional result carries over to the n-dimensional case and thus Euclidean distance!

Obviously, this needs to be done for every distance function again.

So if you want to interpolate a center for an other distance function, solve that optimization problem (efficiently, but try to find an existing solution first, of course).


model to predict annual outcome based on previous years data I have below datasets for two years each holding about 10.000 records. Every week a new report is generated that shows the performance for the current or any previous month. Therefore a more recent weekly report would overwrite the monthly value of a previous report (for that specific month). I need to predict this year's performance based on the weekly data coming in.

I have two questions:

How would I need to structure my dataset for machine learning? If I
keep the above I would get about 150 features (3 per week) and I do not
know whether there is even a ML algorithm that can handle the many n/a (all
forthcoming weeks) for the year for which I need to run the
prediction. I can also transform this into a new dataset by combining
the three columns by week into three features (would result in long
narrow table). But that would replicate each entity and output value about 50 times (once per
week).
Based on the answer to 1. what would be suitable training
algorithms?

I guess you need to predict following week's performance based on previous month's performance. Well, I think its more or less like predicting tomorrow's weather based on previous data.

I guess you need to use Time series forecasting model (ARIMA or LSTM ) for the same by setting the lag for one month. You can build a simple Deep Learning model for the same by adding layers with lag as a parameter