Difference between text-based image retrieval and natural language object retrieval I'm working on creating a model that locates the object in the scene (2D image or 3D scene) using a natural language query. I came across this paper on natural language object retrieval which mentions that this task is different from text-based image retrieval in the sense that natural language object retrieval requires an understanding of objects in the image, spatial configurations, etc. I am not able to see the difference between these two tasks. Could you please explain it with an example?

Disclaimer: I can only answer for the NLP part since I'm no expert for image processing.
I assume that text-based image retrieval is the task of finding the image (or the part of an image) which corresponds to a short text which exclusively describes the object. Practically it means that any content word (i.e. excluding grammatical words like determiners) in the text refers directly to the object: "a bike", "a black cat", "the red car", etc. For a ML process it means that there's nothing to analyze in the text, every word can directly be associated with a characteristic of the image.
By contrast Natural Language object retrieval involves analyzing the text. For instance "the cat on the left of the picture" is different than "the picture on the left of the cat", even though the words are the same. Additionally there can be different ways to refer to the same object: "the book at the left of the shelf" may be the same as "the leftmost book" or "the book next to the green book". There are usually many ways to express the same meaning with language, and that makes the task much more complex. Additionally I would assume that mapping positional descriptions to the image characteristics can be tricky: "the man behind the tree" or "the second bridge" in a 2D image requires the model to "understand" depth. In a picture with two dogs, "the small dog" requires the model to "understand" size relation between objects. Humans intuitively know how to interpret these sentences, but for a machine Natural Language Understanding hasn't been solved yet (it might never be).


Are there neural networks packages that use complex numbers? Can you build complex (in terms of complex numbers) neural networks in Keras or Tensorflow or something similar?

This would mean the inputs, weights, activation functions, and outputs would all potentially use complex numbers. I know this can be done in theory, but all the software packages

I have come across seem to assume only real numbers are used for the weights and inputs.

Neural networks learn a function mapping from the input space to the output. This can be framed as a classification or a regression problem. In both of these cases the function determines the most information rich inputs which will lead to a minimized loss. We minimize the loss with respect to the input space.

If we have a complex valued input this is no problem. Let's assume two complimentary extreme cases. If we have a complex image, however, the complex part is completely randomly distributed. Then this problem would be equivalent to only using the real part of the image as the input. This is due to the complex part not having any pertinent information which would lead to the classification or regression. This is equally true, if the real part of the image part is completely random but the complex part contains all the information.

For the in between situation where we do not know the information distribution in the real and imaginary space then you can input of these components into the network.

For a 64*64 complex image. This will result in $2*64^2$ input nodes each accepting a floating point value, if the first layer is a densely connected layer. This is commonly used for MRI images. So the nodes can identify which real or complex values are needed to infer meaning and provide an output.

I have not seen any implementation that directly applies neural networks on complex values. But derivatives of complex values exist. You would need nodes which have complex functions such that the gradient is applied to both the real and imaginary part. From a guess, I suppose it could be possible. But the common method is just to split the real and imaginary part and feeding them to the network.


Labels as features in anomaly detection I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as a features? Is it an overfitting Risk?

It simply depends on what is the goal of the task:

If the final goal is still to predict Y after detecting anomalies (i.e. probably using the output of anomaly detection as a feature), then Y cannot be used since it wouldn't be available in a realistic test set.
If it's just a completely different task in which Y is available as an input, then why not use it.

With 500k instances, A single additional variable with 3 possible values has an extremely low risk of causing overfitting.
Note that since classification didn't work, it's likely that there is little relationship between the features and Y (otherwise there was some mistake in the classification experiment).


How to calculate compression ratio when using autoencoder in neural network For example, if I use an autoencoder to compress a 1000 dimensional data set to 25 dimensions. Is the compression ratio is 40:1?

Other info:
The dataset contains 5000 samples.
2 million parameters are needed for the encoder.
Another 2 million parameters are needed for the decoder.

Does these numbers have something to do with the compression ratio calculation?

Usually the compression ratio is measured on the files. So the correct way of computing it is to calculate the size of you data before compression, run them through the AE, store them to your drive and then see how much size they occupy after compression.

Then simply divide the two:

$$
Compression \; Ratio = \frac{Uncompressed \; Size}{Compressed \; Size}
$$

Theoretically in your case this should give a 40:1 compression ratio, but, due to overhead and the way you store your data it might not be so high.


Low prediction/classification accuracy due to imbalance in data feeding I am building the neural network for image analysis to do Chest Xray classification (Abnormal/Pass).

The classification accuracy for abnormal Xray is low, I guess it is due to the lack of abnormal samples to train.

Attached image is the schema I built. Is there any ways to improve the classification accuracy for abnormal Xray?

When the data is highly imbalanced, accuracy could be a bad measure of model performance. Instead consider precision and recall values, which separately takes into account the no of positive and negative samples. One idea to improve the performance is to use class weights (generally inversely proportional to the no of samples in that class) while calculating loss metric. This makes the model understand that class with less samples should be given more priority while training.


What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying I am really in need of some explanation, I am working on an NLP cyber-bullying detection tool which I will deploy to the web using Django framework, however, am stuck on some idea, can someone explain to me...What is the main concept of using lexical, linguistic, semantic or syntactic approach in (NLP) and how is applied in cyber-bullying or what are the step or why does people say I built this cyber-bullying system from a linguistic approach, I know POS-tagging is way of grouping word and look for dependency in other word, my idea of POS-tagging is a synonym of semantic because POS-tagging is a process linking words to it root and representation word in an understandable context correct me if wrong.

I read an article where a paper tackled a project using a predictive analysis approach with feature extraction techniques, Naive Bayes for classification and to train the model, in the discussion they also spoke on how other team used the Semantics approach to classifier cyber-bullying. I know of data cleaning, tokenization stemming and most of feature extraction model, however, am stuck on the problem of approach which is what is the relevant on lexical, semantic or syntactic and how are they been approached.

I suggest to check out some introductory books on NLP, e.g. Natural Language Processing with Python. This is a very accessible and practical introduction and useful even if you won't be working in Python. Another, more detailed text book is Speech and Language Processing by Jurafsky and Martin.

You need to understand the basics first, and these are independent of your particular application (cyber bullying). And I think this is too much knowledge to reasonably fit into a response here. But just to give you an idea:

Lexical: information about the words (meaning and structure)

Syntactic: the structure of the text (how words make up phrases, and how phrases make up sentences, the rules that govern that process)

Semantic: about the meaning of text (as it is constructed from the meaning of the words and their composition).

These are usually part of a pipeline, which means you need start processing from the words, then look how they form sentences, and then you can look at the meaning.

Your understanding of POS-tagging seems off the mark: it is the process of assigning words in a text their part of speech (POS), e.g. verb, noun, adjective etc. As such it is part of the syntactical processing (but requires lexical knowledge too), but is also useful for semantic analysis further down.


Dummy variable only for character value in a column (Neglecting float and integers) My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION)  contains all numerical value(around 2000 different values like 20,30,32....) but got one value as String e.g. 'No_information'.
When I create dummies(One-Hot Encoded) for the Feature (pd.get_dummies()), around 2000 new columns are created for the original column i.e. ESTIMATE_FAMILY_CONTRIBUTION.
What I want is to have only 2 columns to be created, one for 'No_information' and other with all the numerical values. How do I do it?

You have 2000 different values available for a dataset of 3000. I don't think you should treat that as a Categorical column.
Treat "No_information" as a NaN and impute using the best-suited strategy using the relation with other columns
Edit post comment
If you want to treat it as Categorical, I will suggest
- Create "bins" of values e.g. <50- Very Low, 50-500 - Medium, etc.
-  Then create One-Hot encoded data
-  With this approach too, "No_information" should be treated as NaN. The reason being that when Info is not available it could have been anything i.e. Low, High, Medium. Treating it as 4th value will have lesser information.
 - Try different bins/approaches and see which produce the best result
Dealing with High cardinality of Categorical feature -
- Search internet/SE with "Encode feature with very high cardinality". This is a known challenge, you will get plenty of resources.
 - Try other encoding approaches. See these links -
Beyond OHE
Library
kaggle post


Demand Forecasting for Multiple Products Across Thousands of Stores I'm currently working on a demand forecasting task, with data on tens of thousands of products across a couple thousand stores. More specifically,I have 3 years' worth of daily sales data per product in each store, and my goal is to forecast the future sales of each item in each store, one day ahead; then two days ahead, etc.

So far I've considered breaking down each product-store pair into a single time series, and doing a forecast for each time series as was done in Neal Wagner's paper, Intelligent techniques for forecasting multiple time series in real-world systems. In other words, I will use only the historical information of a particular store's sales of the product to forecast the future sales of that product in that store.

However, I've been browsing Kaggle and competitions like Corporación Favorita Grocery Sales Forecasting suggest a different approach, which is to use the information from all stores and all products to predict future sales. As I understand it, historical sales information of all products in all stores are dumped into the training set, from which the model will learn to forecasts future sales. It's very different from traditional time series methods, but apparently, based on the results of the competition, it works.

The latter method seems promising and more robust. However, there's the problem of having to process hundreds of millions of data points.

Which method is more appropriate for my task? For those who have worked on similar problems, which methodology would you recommend?

I was also in a similiar situation. I had a dataset with number of stores and product categories and I needed to predict weekly sales for each category. You may want to check gradient boosting alghorithm and random forest which give promising results in terms of forecast accuracy. Also, regardless of the method, the best thing to do is to create dummy variables as creating single time series is very time consuming.


Confused about steps_per_epoch and validation_steps in Keras According to the documentation in TF 2:

       steps_per_epoch: Integer or `None`.
            Total number of steps (batches of samples)
            before declaring one epoch finished and starting the
            next epoch. When training with input tensors such as
            TensorFlow data tensors, the default `None` is equal to
            the number of samples in your dataset divided by
            the batch size, or 1 if that cannot be determined. If x is a
            `tf.data` dataset, and 'steps_per_epoch'
            is None, the epoch will run until the input dataset is exhausted.
            This argument is not supported with array inputs.
        validation_steps: Only relevant if `validation_data` is provided and
            is a `tf.data` dataset. Total number of steps (batches of
            samples) to draw before stopping when performing validation
            at the end of every epoch. If validation_data is a `tf.data` dataset
            and 'validation_steps' is None, validation
            will run until the `validation_data` dataset is exhausted.

I don't understand this part:

Total number of steps (batches of samples)
                before declaring one epoch finished and starting the
                next epoch.

If I have 1000 samples, batch size = 100, then an epoch will take 10 steps to reach. Why is another steps_per_epoch is needed? If both are used, they are conflicting. Isn't it? If the batch size is 100, then 10 steps are needed. If also steps_per_epoch = 20, then it means in one epoch it needs 20 batches, which is conflicting with the '10' steps calculated through the batch size parameter 100.

Where am I wrong?

There is an additional functionality with this method, way to think about it is a chance to use mini-epochs if loss stagnates too much

Usually it is equal to n_samples // batch_size, BUT

steps_per_epoch give you the chance to feeding process of the NN when updating the learning rate using ReduceLROnPlateuau(). This callback checks the drop of the loss once each epoch has finished and if the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than n_samples // batch_size without affecting the overall number of training epochs of your model.


EDA for analysis of nominal variable with high cardinality I have a nominal variable (car model) with very high cardinality (~8500 labels) and I would like to analyse its relation with a binary target variable. While I can create logical groups and compare the distribution of target variable for each of the groups, can anyone suggest if there are any superior techniques/visualization tools for this type of analysis?

You can calculate mean target for each categorical variable and compare its values.
In pandas this can be done easily: df.groupby('categorical_feature').target.mean()

Then you can make a histogram to compare the approach. I also, seaborn has a catplot, where it do the same as above in a bar plot format, showing mean value for target variable based on each categorical one.


Difference between Ridge and Linear Regression From what I have understood, the Ridge Regression is just having the loss function for an optimization problem with the addition of the regularization term (L2 Norm in the case of Ridge). However I am not sure if the loss function can be described by a non-linear function or it needs to be linear. In this case, if the loss functions needs to be linear, then from what I understand the Ridge regression, is simply performing Linear regression with the addition of the L2-Norm for regularization. Please correct  me if I am wrong.

Yes, ridge regression is ordinary least squares regression with an L2 penalty term on the weights in the loss function. The loss function is not really linear in any of its terms, right? it's the squared residuals plus squares of weights. But there is no reason the loss function needs to be linear. It's helpful if it's convex, which it is here, but even that is not required to try to minimize it.


High error Arima model - Python I have a time series data. It has daily frequency.

I want to forecast the data for the next week or month with an ARIMA model.

This is a chart of my time series data:

First I use the method seasonal_decompose from stats model to check the trend/sessionality/residual looks like:

from statsmodels.tsa.seasonal import seasonal_decompose
result = seasonal_decompose(df['n_transactions'], model='add')
result.plot();

I check if my data is stationary:

from statsmodels.tsa.stattools import adfuller

def adf_test(series,title=''):
    """
    Pass in a time series and an optional title, returns an ADF report
    """
    print(f'Augmented Dickey-Fuller Test: {title}')
    result = adfuller(series.dropna(),autolag='AIC') # .dropna() handles differenced data

    labels = ['ADF test statistic','p-value','# lags used','# observations']
    out = pd.Series(result[0:4],index=labels)

    for key,val in result[4].items():
        out[f'critical value ({key})']=val

    print(out.to_string())          # .to_string() removes the line "dtype: float64"

    if result[1] <= 0.05:
        print("Strong evidence against the null hypothesis")
        print("Reject the null hypothesis")
        print("Data has no unit root and is stationary")
    else:
        print("Weak evidence against the null hypothesis")
        print("Fail to reject the null hypothesis")
        print("Data has a unit root and is non-stationary")

adf_test(df['n_transactions'])

Augmented Dickey-Fuller Test:
ADF test statistic       -3.857922
p-value                   0.002367
# lags used              12.000000
# observations          737.000000
critical value (1%)      -3.439254
critical value (5%)      -2.865470
critical value (10%)     -2.568863
Strong evidence against the null hypothesis
Reject the null hypothesis
Data has no unit root and is stationary

I use auto_arima in order to get the best parameters for my model:

from pmdarima import auto_arima
auto_arima(df['n_transactions'],seasonal=True, m = 7).summary()

I train my model with this paremeters:

train = df.loc[:'2020-05-12']
test = df.loc['2020-05-13':]

model = SARIMAX(train['n_transactions'],order=(1, 1, 1))
results = model.fit()
results.summary()

I calculate the predictions:

start=len(train)
end=len(train)+len(test)-1
predictions = results.predict(start=start, end=end, dynamic=False, typ='levels').rename('SARIMA(0,1,3)(1,0,1,12) Predictions')

ax = test['n_transactions'].plot(legend=True,figsize=(12,6),title=title)
predictions.plot(legend=True)
ax.autoscale(axis='x',tight=True)
ax.set(xlabel=xlabel, ylabel=ylabel);

But the model can't obtain good results, why?

Edit

I have used instead of counts the revenue that I obtain for this counts as you suggested me that may be this would be the problem:

But the model is not obtaining good results:

What conclusion can I extract from here?

your data looks like a count process. The default ARIMA parameters assume a normal distributed continues error term. So the standard ARIMA models are using this assumption. As far as I know there are some ARIMA models with poisson distribution as an error term, but I guess you just should google for timeseries for count processes. Something like pyflux could work.


How to process Dicom Images for CNN? I am building a disease classifier. I have Dicom scans for many patients.
The scans have different slice thickness, and different scans have different number of slices. However, the slice thickness for a single patient's scan has the same thickness for all slices.

For example:

Patient1's scan : 100 slices of 2mm thickness

Patient2's scan : 500 slices of 1mm thichness

The number of pixels for each slice is 512 x 512,
so currently the shape of nd array containing the information for

Patient1 is 100*512*512

Patient2 is 500*512*512

I want to pass all patients information into a CNN.
Do I need to resample the slices and make it uniform for all patients (like 512*512*n). If yes, how to do it, and what should be the value of 'n'?

Can you use zero padding? It's a pre-processing technique for CNNs, it consists in creating a frame of zeros around the image, so that all input image will have the same size.

The CNN will then learn autonomously to ignore the zeros. It's a common technique, Keras layers already have padding built-in arguments.


XGBoost results are not invariant under monotone predictor transformations? It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the arxiv paper later) that says whether it's invariant depends on how the split threshold is chosen (there are three methods), and according to this paper, xgboost would be invariant under transformations because it uses the sweep left method. This is mentioned in the last paragraph on pp.2 and first paragraph on pp.3.

But when I read the original xgboost paper by Chen, the split algorithm looks much more sophisticated than any method mentioned in the arxiv paper, and it looks like it should be sensitive to transformations. I've tried xgboost on regression for a few data sets, and if I have column subsampling turned on, I do see different results with predictor transformations.

Can anyone give me some confirmation on this topic? I'm confused mostly by the arxiv paper.

A quick experiment suggests that xgboost makes its splits at midpoints between values of a feature in its training set:

https://github.com/bmreiniger/datascience.stackexchange/blob/master/74634.ipynb

The history turns out to be relatively complex, and possibly ongoing.

In this github issue there's an indication that the midpoint splits are only for small-cardinality discrete features.  That code is gone in the current (at time of posting) version of the code.
It was removed in pull 5251, commit 969ebbc, with further discussion pushed into issue 5096.

In the opposite direction, the function AddCutPoints didn't always exist, but the same discrete/continuous optimization shows up here, commit Jan 2017.  So it seems that historically left-endpoints were used (at least for continuous data), supporting the arXiv paper, but that currently midpoints are used across the board.

The experiment above finishes with the Wisconsin Breast Cancer dataset, which has well over 16 values for the relevant features, and still has splits appearing at midpoints.

Finally, just a couple words about the monotonicity.  First, as the arXiv paper mentions, this is entirely a question of how the model will apply to unseen data and a feature value that lands between values in the training data.  Using left endpoints will always put such a value to the right side of the split, and so any monotonically increasing transformation won't affect that.  A monotonically decreasing transformation will exactly reverse that.  A linear(/affine) transformation will preserve the behavior when using midpoints (and also either left/right endpoints, depending on the sign of the transformation).


What Machine Learning Technique can I use to judge boxing fights? I want to build a machine learning model that judges the fights based on the results of each round. Any suggestions on what techniques can I use?

It depends by what you mean by the result of each round. In boxing after each round fighters receive from $7-10$ points in case if none of the fighters was KO, submitted or disqualified. An extract from Wikipedia :

The 10-point Must System is the most widely used scoring system since the mid-twentieth century. It is so named because a judge "must" award ten points to at least one fighter each round (before deductions for fouls). Most rounds are scored 10–9, with 10 points for the fighter who won the round, and 9 points for the fighter the judge believes lost the round. If a round is judged to be even, it is scored 10-10. For each knockdown in a round, the judge deducts an additional point from the fighter knocked down, resulting in a 10–8 score if there is one knockdown or a 10–7 score if there are two knockdowns. If the referee instructs the judges to deduct a point for a foul, this deduction is applied after the preliminary computation. So, if a fighter wins a round, but is penalised for a foul, the score changes from 10–9 to 9-9. If that same fighter scored a knockdown in the round, the score would change from 10–8 in his favour to 9–8. While uncommon, if a fighter completely dominates a round but does not score a knockdown, a judge can still score that round 10–8.

Whether the points for each round given, the decision is straightforward - simply a sum of all scores for each round and doesn't require any machine learning.
However, if the question is - how to judge a particular round, the question is more involved. One may collect the scores on referee cards with the number of significant strikes, strikes landed, knockdown's, control of the ring, e.t.c, or even attempt to implement an algorithm, which given the video, detects, whether the strike has landed on the opponent, was significant. But it seems to be rather a tough task, and the data mining for supervised algorithm would be quite expensive.


Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words? I want to compute the semantic similarity of two words using their vector representations (obtained using e.g. word2vec, GloVe, etc.). Shall I use the Euclidean Distance or the Cosine Similarity?

The GloVe website mentions both measures without telling the pros and cons of each:

  The Euclidean distance (or cosine similarity) between two word vectors provides an effective method for measuring the linguistic or semantic similarity of the corresponding words.

First of all, if GloVe gives you normalized unit vectors, then the two calculations are equivalent. In general, I would use the cosine similarity since it removes the effect of document length. For example, a postcard and a full-length book may be about the same topic, but will likely be quite far apart in pure "term frequency" space using the Euclidean distance. They will be right on top of each other in cosine similarity.


fast processing: changing the column value in a geospatial environment I have a table whose header looks like this: complaint_type, borough, street_name, incident_zip, latitude, longitude

1) I want to check if the "incident_zip" column of each row is in a specific list of zip codes and change the "borough" accordingly. There is a large amount of data and i cannot find any better code to do this. I am using python 3.6.
I want to change the borough where it is "unspecified". I used if statements along with replace but it is taking a lot of processing time (more than 3hrs) that I have to stop the kernel. There are five long lists of zip codes.

2) Is there any other way to update the "borough" column like latitude and longitude?

This is to get whether the incident_zip is in the list of zip code.

df.isin({'incident_zip' : zip_code_list)

If you have a zip code borough mapping in the form of dict. You can do

df['borough'].map(zip_to_borough)

For pandas dataframe never iterate say by rows unless as last resort because the performance will be terrible.

You can speed these operation multiprocessing if you want.


Here's my idea:

# dataframe example
b = [1,0,0,0,0,0,0-1,1,0,0,-1,0,0,0,0,0,0,1,0,0,-1]
minutes = ['2019-01-02 15:' + str(x) + ':00' for x in range(len(b))]
df = pd.DataFrame({'a': minutes, 'b': b})

# keep only rows which are equal to 1 or -1
df_tmp = df[df['b'] != 0]

# convert date column to datetime if needed
df_tmp['a'] = pd.to_datetime(df['a'])

# create a new column which is the time difference between each row
df_tmp['diff'] = df_tmp['a'].diff().astype('timedelta64[m]')

# keep only the rows equal to -1
df_tmp = df_tmp[df_tmp['b'] == -1]

# count the values which are smaller or equal to 5
(df_tmp['diff'] <= 5).sum()


Binary classification model with time series as variables This is probably a simple question. Assume I'm interested in modelling a binary variable, with various covariates, including ones that are time series observations. In the usual modelling approach, one can try searching for various features from the timeseries data, such as standard deviations, averages, max and etc, to make a flat model matrix.

My question: what are the tools/approaches that allow for a (relatively) simple inclusion of time series data to a classification problem?

I don't think panel regression would work since the time series data is very different among the rows, sometimes is very sparse and asynchronous.  Melting the data, due to the structure, obviously wouldn't work too. Descriptive statistics is the easy way, but there should be something else?

I'm not experienced in working with neural networks, but maybe there's a NN approach that could find meaningful structures in the time series data?

I'm also thinking about clustering different time series based on their similarities and check for significance, but again, is there something robust to different length/sparseness of the time series?

I think one of your statements is simply a misunderstanding of the relationship of data structure and analysis. When you say "Melting the data, due to the structure, obviously wouldn't work too." you seem to failing to understand that melting the data properly would require construction of a supplemental covariate that would encode the "column location" of the values. In longitudinal data this location (in the "wide" version of the data) would become the "time" variable in the "long" format. Regression and neural network methods can handle such a format.

I intended that my comment only be a suggestion for improving a search strategy,  since you had not described the task or the inputs in any great detail. So I'm "answering" with some links resulting from several variations on a search strategy along the lines of ("binary outcome" OR "signal detection" OR classification) AND ("logistic regression" OR "neural networks") AND longitudinal AND missing:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5401893/

http://www.stats.ox.ac.uk/~snijders/WrightLondon2009.pdf

http://www.drizopoulos.com/pdf/Slides/jsm_2015.pdf

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3834930/

https://dl.acm.org/citation.cfm?id=3167596

https://mlhc17mit.github.io/slides/lecture7.pdf

https://www.cs.cmu.edu/~epxing/Class/10708-17/project-reports/project8.pdf

https://arxiv.org/pdf/1802.04664.pdf


Basic ML based Prediction model I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python.

Say I have a training data of 1000 items as

Item_name, Attrib_1, Attrib_2, Attrib_3,....Attrib_N, Cost

And my aim is to create a model to predict cost for a new item given the attributes.

So where should I start and what are the different ways to prediction and solve this problem ? Also how to evaluate different methods ?

The type of ML problem you are trying to solve is a regression problem.  Essentially, given your attributes, predict a continuous variable.

There are multiple ways to solve your problem, but (assuming attributes are numerical) they all boil down to "plot your data points, then draw a line through them".  The different methods calculate the placement of this predictor based on various different measures of error; the most basic probably being least squares, which aims to minimize the sum of squared errors.

Metacademy is a great website for learning ML related stuff online, because they provide roadmaps that tell you what you need to learn before learning ______, as well as how to learn those prerequisites.

Here is a link to the metacademy page for linear regression, which is where you're going to want to start.  I'd recommend watching the coursera videos from Andrew Ng.  If you're just getting started with ML, his course is a wonderful introduction.  However, a common complaint for the coursera class is it's too shallow, and is to be considered as an overview of machine learning.  If you consider yourself a little more mathematically savvy, I hear the Stanford course (also taught by him) may be a more in depth resource for you.

You should implement your own basic linear regression at first, just to get a deeper understanding of how it works.  Afterwords, you can just use a python ML library called scikit learn to perform the regression (link here).  I strongly recommend you take this approach for all ML algorithms; strive to understand at least how the algorithms work at a basic level.  It is far too easy for someone just starting to learn about ML to see all the implemented algorithms in scikit learn go on to use the algorithms as black boxes without understanding what the algorithm is actually doing.

Lastly, in order to evaluate the different methods, you should split your dataset into two different parts; a training set with which to train your models on, then a testing set with which to test how accurate the predictions are.  The idea here being that your data is representative of the "wild" data that you'll see, so by purposely hiding some data, you can effectively simulate how the algorithm will perform in "real world" conditions.

Generally, people go with something along the lines of 70% of your data being used to train, and 30% of it being used to test.  Later on, as you get into complicated models with multiple parameters, it will help to split your data into 3 parts: training, used to train your model, validation, used to adjust model parameters, and finally, a test set, used to measure how accurate the model is.

Good luck!


XGBoost: how to adjust the probabilities of a binary classifier to match training data? Training and testing data have around 1% positives, but the model predicts only around 0.1% as positives.

The model is an xgboost classifier.

I’ve tried calibration but it didn’t improve much. I also don’t want to pick thresholds since the final goal is to output probabilities.

What I want is for the model to have a number of classified positives similar to the number of positives in the actual data.

The first (and easiest) option is to make sure that your model is calibrated in probabilites. In Python, it means that you should pass the option binary:logistic in your fitting method.
The alternative is to transform the output of your model into probabilities. There are different approaches for that.
This could be achieved with some sort of regression techniques to find the relationship between probabilities and your output. Python's isotonic regression should work for that purpose. However, without more information on your score ditribution it is possible it doesn't work well.
This can also be achieved with platt scaling : transforming your output into binary prediction (0 and 1) with a threshold, then calibrate a logistic regression on those new variables. It is relatively easy to do, but in my experience doesn't necssarily work well with unbalanced problems with non-linear relationships.
Finally, there are some approaches that just correct the output depending on your model. For logistic regression that would mean changing your bias variable so that the overall predicted proportion match the one of your data-set. This also can be used to counter the effects of rare events (see this). I have found this to work wuite well with logistic regression. However, I am not sure if it would directly be appliable to XGBoost, but it could be worth a try.


Feature engineering - house price prediction (small dataset) I am working on the task of predicting real estate prices. My dataset has only 10 variables described below. I'm thinking about feature engineering but nothing comes to mind.

Variables:

street

city

zip code

rooms

bathrooms

square feet

type

price

latitude

longitude

latitude, longitude could be used for clustering. Then you should set cluster_id to every house as a new feature.

Also it possible to use geo coordinates to calculate the distances to important points (to the city center for example)


What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another? I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding:

hessian regularisation blocks the individual trees ($f_t$) from growing (much like limiting max_depth)
loss regularisation blocks the individual trees ($f_t$) from being kept in the ensemble as it penalises number of trees

However, I have some troubles to find practical implications from that. I mean, is there a situation when hessian regularisation would perform better than gamma regularisation (or worse)? Or is it all dependent on the dataset and the other hyperparameter values? My feeling is that when my model loss is mainly caused by few awfully inaccurate prediction - then hessian regularisation might help more than gamma, and when my model is mistaken by approximetely the same amount for every observation then gamma would be better. But I'm struggle to figure out the proof for that feeling.

Loss regularization also prevents the trees from growing; it does not penalize the number of trees.
I think you're right that which is better (of course, you could use both) depends on the data and other hyperparameters.  You can think though about whether a given node will split, depending on the type of regularization:
If you have a large node with some potential split that would create two other large nodes, but that split doesn't decrease the loss much, gamma will prevent it but min_child_weight won't.  If you have a node with a potential split with one child being very small but that decreases the loss a lot, gamma will allow it but min_child_weight won't.  So, personal opinion (based on that thought experiment, and not experience tuning these two together or against each other): set min_child_weight large enough that you're comfortable that a node with that many samples (in the squared-loss case, and appropriately transformed in other cases) is big enough to absorb noisiness of your population [by which I mean: one sample is surely quite likely to be random noise; an average of 100 might be better, but might not be enough, depending on your data]; after setting that, tune gamma.  Better, tune both of them together, but still use the above intuition of your data to give a reasonable range for min_child_weight.


CNN gradients with different magnitude I have a CNN architecture with two cross entropy losses $\mathcal{L}_1$ and $\mathcal{L}_2$ summed in the total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. The task I want to solve is Unsupervised Domain Adaptation.
I have attested the following behavior:

The gradients coming from $\mathcal{L}_1$ have a different magnitude than those coming from $\mathcal{L}_2$ such that the supervision coming from the first loss is negligible.
$\mathcal{L}_1$ has a positive constant value and does not decrease during the training, while $\mathcal{L}_2$ does decrease.

How can I minimize $\mathcal{L}_1$ and how can I make the gradient from $\mathcal{L}_1$ more important? Currently I have two options:

Add a tradeoff parameter to one of the two losses $\mathcal{L} = \mathcal{L}_1 + \gamma \cdot \mathcal{L}_2$
Normalize the gradients at some step

The last option would be to leave everything as it is, with the motivation that one loss does not provide supervision to the task I want to solve. Do you have some advice on the road to follow?

If you are capable of optimizing the weighting hyperparameter γ, then the relative importance of the loss functions becomes an empirical question. That data will help guide the mixture of losses.
Normalizing the gradients is ad hoc. Thus simplifier but could over simplify the problem.


How do you overcome results oriented thinking from a client? I am working with a client on a special project that I am going to obfuscate in this question. Basically, I'm trying to overcome some short-term, results-oriented thinking from my client.

Let's say you have a model to forecast the performance of a racehorse. Your model tells your client to sell racehorse X because the probability of its performance is low (<10%). The horse is sold and said racehorse goes on to win 3 races. Your client says, "See, we should have never let go of that racehorse! The model is wrong!"

As data scientist we can understand that anomalies happen and that this horse may have also lost some races - we were just on the wrong side of probability this time. But how do you overcome those objections with the client? How do you turn short-term thinking into a long-term outlook of predictive modeling?

The scenario you described with one horse is basically anecdotal evidence. To objectively evaluate what you're doing, you need to agree with the client to make this an experiment.

  I understand you don't trust my horse predictions and that's totally
  understandable. It is human nature to worry about the ones that got
  away. Let's handle this objectively and run an experiment. I'll give
  you buy/sell ratings on 50 horses. We then track their performance for
  six months. If on average my ratings outperform a coin flip, we'll
  know we're onto something at a macro level even if a few got away. If
  my models end up being worse than a coinflip, we'll know the model
  needs to be improved or you need to fire me.


How to input LSTM output to MLP with concatenate? I am having a training data set for a time-series dataset like below where my target variable is var1(t) which is the value of var 1 at time=t.
import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]

I am inputting the LSTM network with the past 3 timesteps (t-3) to (t-1) of all 4 variables and then feed the output of the LSTM with the current timestep value of the var2,var3,var4 with an MLP with functional API in Keras.
So I prepared the inputs for the LSTM and MLP like below :
#subset the 3 previous timesteps of the 4 variables for the time sries part
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

Then I tried the below network :
#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[0],train_X_LSTM.shape[1]))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[0],train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = concatenate([lstm_output, mlp_input], axis=-1)
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit(x_train, y_train, batch_size=64, epochs=10, validation_split=0.2)

This throws the error of

ValueError: A Concatenate layer requires inputs with matching shapes
except for the concat axis. Got inputs shapes: [(None, 1), (None, 100,
3)]

which shows I am not combining them correctly. Any help is appreciated!

In Keras, there is no need to give the number of samples in your training data to your model.
Also, the way you defined your LSTM means that you consider each data sample as a timestep for the LSTM, instead of your t-1, t-2 and t-3 values.
So you can remove train_X_LSTM.shape[0] in your Input layers, and give X=[train_X_LSTM, train_X_MLP] and y=train_y_LSTM to model.fit so it matches what your model expects.
#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[1], 1))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)

In addition, from my understanding, you want to input (var1, var2, var3, var4) values vector at timesteps  t-3, t-2, t-1 to the LSTM.
In that case, you should also reshape the data you feed to the LSTM
train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)

This way, you will feed a sequence of 3 vectors containing (var1, var2, var3, var4) at the corresponding past timesteps.
Then, update the LSTM Input layer
lstm_input = Input(shape=train_X_LSTM.shape[1:])

Full code
import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]
#subset the 3 previous timesteps of the 4 variables for the time sries part
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

from tensorflow.keras.layers import Input, LSTM, Dense, Concatenate
from tensorflow.keras.models import Model
#lstm input shape
lstm_input = Input(shape=train_X_LSTM.shape[1:])
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)


Encode the days of week as numeric variable I would like to understand if there is the possibility to encode the days of the week as a single numerical column to preserve the ordinal relationship between the days.
My task is a classification task.
So, something like this:

Monday: 0
Thrusday: 1
Wednesday: 2
...
Sunday: 6

I wouldn't use a one-hot encoding for this scenario because I would like to preserve the ordinal relationship between the days.
However, I don't know if this is an allowed approach or not.

The answer is as follows:
If you are using a Linear Regression Model - then it will assume that Sunday (6) is bigger (or carries more weight) than all other days in the week. If that assumption makes sense for your problem domain - you are fine.
If you are using a model such as Random Forests - then it doesn't really matter (meaning you are good). Random Forests are typically scale invariant - They do not assume higher weightage for large values - They will figure out how best to split up the numeric values (0 - 6) to meet your objective.


What is the selection criteria to choose between XGBoost and Random Forest I am trying to understand - when would someone choose Random Forest over XGBoost and vice versa. All the articles out there highlights on the differences between both. I understand them. But when actually given a real world data set, how should we approach the problem to choose between these?

For eg: Is there a set of statistical tests for variance check, and then choose? Or is it simply like you have a number of features, and cannot really choose to do parameter tuning, so you apply Random Forest to get results?

Let´s say that the best way to choose is empirical. You run both algorithms in the dataset and check which one has better performance.

It's true that you can do a lot of theoretical analysis but at the end you have to try no matter what. They both use decision trees ensemble so the results should not be too different. By experience gradient boosting tends to achieve better results. Also, it is more mathematically complicated to understand.

Normally decision trees don´t require too much parameter turning, or at least less than other models.

There is no classical statistic test that will tell you will perform better. There are some heuristics but I find them overly complicated.


Requirements collecting How would gather requirements for a data science project? How do you convince stakeholders that whatever you develop is going to benefit them? How to build trust to make use of data science for the business operations?

An answer to the third question: "How to build trust to make use of data science for the business operations?"

It heavily depends on the type and structure of data that you want to get access and the application that you want to provide.

For example, if data owner has a concern about the privacy of its users, and you just want to execute some statistical queries over their data, you should design and propose a privacy-preserving method for accessing to their data. There are some techniques like Differential Privacy (or Randomized Response) which provide a well-defined privacy guarantee for data owner.


In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula? In Sergey Levine's lecture on policy gradients (berkeley deep rl course), he show that policy gradient can be evaluated according to the formula

In this formula, wouldn't longer trajectories get more weight (in finite horizon situations), since the middle term, the sum over log pi, would involve more terms? (Why would it work like that?)

The specific example I have in mind is pacman, longer trajectories would contribute more to the gradient. Should it work like that?

wouldn't longer trajectories get more weight?

Not necessarily. Gradient $\triangledown_{\theta}$ could be negative or positive (1D analogy), therefore, larger number of gradients could have a smaller weight, which makes sense. A consistent short trajectory is more informative (has more weight) than an inconsistent long trajectory with sign-alternating policy gradients.

  Why would it work like that?

If we are comparing two consistent trajectories, where most gradients are in the same direction, this formula makes sense again. A long consistent trajectory contains more useful information (more steps that confirm each other) than a short one. In real life, compare the informativeness of a successful week to a successful year for your policy learning.


Why might a neural network consistently underestimate its target? I have a neural network (MLP) that is consistently underestimating the target variable on the validation set, test set, and on the training set (by about the same amount as on the validation set and test set). In other words, the sum of the entries in the neural network regression output column (Y_hat) is something like 10% less than the sum of the entries in the target variable column (Y). The entries in the target column are greater than or equal to zero, with a decent number of zero entries. The right-side tail of the distribution of the target variable is long.

Fitting the neural network using many random seeds consistently leads to similar results (the neural networks are all biased in the same direction and by a similar amount).

The problem seems to be fairly robust to changes in important hyperparameters, including:
- early stopping
- learning rate schedule
- model complexity
- regularization (dropout, batch norm)

The problem is less severe for low-capacity neural networks.

Does anyone have any ideas as to why this persistent underestimation might be happening?

Consistently underestimating target could be due to the distribution of the target variable. If the target distribution has a negative-skew (i.e., a long tail towards lower values), then the neural network is just pattern matching to minimize those errors. For example, if the network has a squared loss function then large estimation errors in the tail are weighted more heavily than small estimation errors in the head, even though there are fewer data points in the tail. The model is trying to minimize the overall loss function and is "cheating to win" by consistently underestimating.


Categorical encoded variables in scikit-learn diabetes dataset When using sklearn.datasets import load_diabetes, the sex variable which is categorical, is scaled to continuous values. Is it even legal to scale such variables?

Yes, it is perfectly legal to do so. Scaling such binary variables does not change the shape of its histogram and the shape is all that matters to most models.


When to use Standard Scaler and when Normalizer? I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler.

I know when Standard Scaler is applied. But in which scenario is Normalizer applied? Are there scenarios where one is preferred over the other?

They are used for two different purposes.
StandardScaler changes each feature column $f_{:,i}$ to $$f'_{:,i} = \frac{f_{:,i} - mean(f_{:,i})}{std(f_{:,i})}.$$
Normalizer changes each sample $x_n=(f_{n,1},...,f_{n,d})$ to  $$x'_n = \frac{x_n}{size(x_n)},$$   where $size(x_n)$ for

l1 norm is $\left \| x_n \right \|_1=|f_{n,1}|+...+|f_{n,d}|$,
l2 norm is $\left \| x_n \right \|_2=\sqrt{f^{2}_{n,1}+...+f^{2}_{n,d}}$,
max norm is $\left \| x_n \right \|_\infty=max\{|f_{n,1}|,...,|f_{n,d}|\}$.

To illustrate the contrast, consider data set $\{1, 2, 3, 4, 5\}$ which consists of 5 one dimensional data points (each data point has one feature),
After applying StandardScaler, data set becomes $\{-1.41, -0.71, 0. ,0.71, 1.41\}$.
After applying any type of Normalizer, data set becomes $\{1., 1., 1., 1., 1.\}$, since the only feature is divided by itself. So Normalizer has no use for this case.
Also, when features have different units, e.g. $(height, age, income)$, Normalizer is not used as a pre-processing step; although, it might be used as an ad-hoc feature engineering step similar to what a neuron does in a neural network.
As mentioned in this answer, Normalizer is mostly useful for controlling the size of a vector in an iterative process, e.g. a parameter vector during training, to avoid numerical instabilities due to large values.


Terminology in machine learning: exogenous features vs external features I am currently writing a scientific paper and do not know whether to call some of my input features of my neural network either external or exogenous.
My neural network receives as input features like the outside temperature, which are completely independent of the mapped mathematical function.
Are these independent features called exogenous or external? When is which term used?

Exogenous simply means a value that is determined outside the context of your model & is then imposed on your model. Endogenous means the model determines the value.
I don't know about "external" as this word seems to depend on context. But you would be right to say these variables are exogenous.


MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks I know the concept of a neural network, and I followed the Machine Learning course by Andrew Ng on Coursera, so I also coded some simple ones. However, I miss all the new tricks which are useful to prevent overfitting of Deep networks, such as for example:

using ReLU instead than tanh neurons
using dropout
more advanced learning methods than just vanilla stochastic gradient descent

etc. I would like to follow a MOOC which tells me how to use Keras for Deep Learning (I like Keras very much because IMO is much easier to understand than other packages, but I'm open to suggestions). I would also be content with a book, but I'd really prefer a MOOC. Doesn't have to be free. Can you indicate me one? The application is Data Science, but generic Deep Learning would do.

EDIT: to provide more context to the question, my main applications would be Internet of Things Analytics, i.e., applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc. I don't need to develop the cloud platform: I just need to develop the "core" Analytics. Think of it as just applying Deep Learning to Time Series or to Classification problems. However, methods which can easily be retrained when new data arrive, without having to go through the full dataset again, would be preferred.

Since you like Keras, the main author has written a book on deep learning with Python. You can check out the sections to see if it covers all the topics you want.


ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0) I am running a multiple linear regression using backward elimination. Below is the code

import statsmodels.formula.api as sm
X = np.append(arr = np.ones((50, 1)).astype(int), values = X, axis =
1)
X_opt = X[:,[0,1,2,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,1,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3,4,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3,5]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()
X_opt = X[:,[0,3]]
regressor_OLS = sm.OLS(endog= y, exog = X_opt).fit()
regressor_OLS.summary()

But when I am predicting using the above regressor_OLS model,

X_new = X[:, 3]
y_pred2 = regressor_OLS.predict(X_new)

I am getting the below error:

    y_pred2 = regressor_OLS.predict(X_new)
    Traceback (most recent call last):

  File "<ipython-input-18-263dee38fc26>", line 1, in <module>
    y_pred2 = regressor_OLS.predict(X_new)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py", line 749, in predict
    return self.model.predict(self.params, exog, *args, **kwargs)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py", line 359, in predict
    return np.dot(exog, params)

ValueError: shapes (1,50) and (2,) not aligned: 50 (dim 1) != 2 (dim 0)

I tried X_new = X_test[:,3] but still same error.

Traceback (most recent call last):

  File "<ipython-input-19-5020d55a4448>", line 1, in <module>
    y_pred2 = regressor_OLS.predict(X_ne1)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/base/model.py", line 749, in predict
    return self.model.predict(self.params, exog, *args, **kwargs)

  File "/Users/ritesh.satapathy/anaconda/lib/python3.6/site-packages/statsmodels/regression/linear_model.py", line 359, in predict
    return np.dot(exog, params)

ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0)

X_ne1 = X_test[:,3]
y_pred2 = regressor_OLS.predict(X_ne1)


Scraping financial web data I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's clients. These information can come from various sources: annual reports, income statements, articles.... I will have to deal with two types of formats: HTML and PDFs. For now I will focus on retrieving the revenue of the company. After a month of research and tests, I realized a few things:
- NLP techniques are too slow to be used on annuals reports
The first step of the project will be the following:

Search for the annual report and scrape the HTML code: so far I managed to get all the google results and I'm using Beautifulsoup to get the HTML code. However I can't quite get the revenue of the company because each website has its own HTML structure. I first decided to focus on extracting tables (the goal is to find the company's income statement) but I realized that HTML tables are often used for layout (even if it's a bad practice). I can't rely on css selectors as I need to keep it as generic as possible. How can I achieve it?

I too would fall back on parsing either the HTML or the entities using regular expressions. My experience is though that this always gets unelegant quickly.

Do you have a somewhat clear idea of the relevant sources? If the
better part of the relevant data comes from a limited number of
pages, you could maintain a list of sources with matching wrappers.
Then within those relevant documents, I would search for the least
complex most valuable features to extract.

Example

For instance, if you'd be interested in the quarterlies of Alphabet, I would scrape this link. You're smart, you can figure out the next one.

A quick glance learns me that the first hit on revenue(s)  $ returns me the revenue for the quarter.

So something like this:

(?:revenue[s]?)(?:\s[\w]+\s)(\$[\d]+\.?\d\s[\w]+)

Testing that one the reports on the site seems to work on q1, q2 and q3 while q4 yields the annual revenue. Easy enough to fix.

My experience is that thee patterns hold for a while, and then change. No big deal, just add a couple of tests! Fi: Is the result not empty and is it in a believable range?


Threshold logic unit (McCulloch-Pitts 1943) vs. Perceptron (Rosenblatt 1958) I have searched various sources to find out what distinguishes the McCulloch-Pitts neuron from the perceptron invented by Rosenblatt. In most sources only one of these elements is considered, in others they are used as synonyms.
Can someone explain the differences in how they work?

MP model : 1) inputs are binary values; 2) has not weights
Rosenblatt model: 1) inputs can take any real numbers; 2) has weights.
Thank you, https://medium.com/@manushaurya/mcculloch-pitts-neuron-vs-perceptron-model-8668ed82c36


Software for automated database processing I faced a problem which I'd like to solve w/o any programming. And looking for a software to do this.

I have a dataset, for example: (brand-id, brand-name, product-class-name;)

0, Audi, economy business premium;
1, Rolls Royce, luxury;
2, Seat, economy;
3, Tesla, business premium;

And I'd like to automatically process this dataset, resulting in creating an additional table to classify parameters in column 3, like: (product-class-id, product-class-name, brand-id;)

0, economy, 0 2;
1, business, 0 3;
2, premium, 0 3;
3, luxury, 1;

Is there any database managing tool with features like this one? (preferably Mac)

Do you consider SQL to be programming? Because at the very least you're going to need to do some SQL manipulations to solve this. There's no visual tool that I know of that will do this for you (especially for OSX!)

I don't know why you have a barrier for a new language; if you were willing to do this in python it could be solved relatively quickly.


Confusion on Delta Rule and Error I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me.

At one point, he gives this equation for the error of a perceptron over a set of training examples.

$$E(\vec{w})\equiv \frac12 \sum_{d \in D}(t_d-o_d)^2$$

$O_d$ is the actual output of $ \vec{W} \cdot \vec{X}$, where $ \vec{X}$ is the input vector and $\vec{W}$ is the weights vector.

$t_d$ is the target output, what we want to get.

The sum over all the $D$ means we sum over every single  $\vec{X}$ we can input.

Okay, so far so good, I understand that.

However, he then gives this example:

But that is just not true!!!! That equation for the error does NOT give us a single minimum!!!

According to his previous rule, if we're considering the error for a single weight vector and a single training vector, the equation for the error would be:

$$E(\vec{w}) = \frac{1}{2}  (t_d - (w_0  x_0 + w_1  x_1))^2$$

Which has an infinite number of minimums!!! Every time $(w_0 x_0 + w_1 x_1) = t_d$

I graphed it here to show you:

In that picture, $x$ and $y$ are the two rows of the weight vector $\vec{w}$.

Please help! I've been confused about this for the last three hours!

Thanks

You are right, the least square solution need not be unique as you have illustrated.

In general, we are trying to solve for $w$ in the system $$Xw=t$$

which of course need not even be consistent (meaning has a solution). In the event that it has a solution, the uniqueness is also not guaranteed if the matrix $X$ has non-zero vector in the nullspace of $X$.

For the least square solution, we are trying to minimize

$$\frac12(Xw-t)^T(Xw-t)$$

of which the minimal is attained when $$X^T(Xw-t)=0$$

that is $$X^TXw=X^Tt$$

and it is only unique when $X^TX$ has full column rank. However, note that uniqueness is not needed for gradient descent to work.


DBSCAN on textual and numerical columns I have a dataset which has two columns:
title      price
sentence1  12
sentence2  13

I have used doc2vec to convert the sentences into vectors of size 100 as below:
LabeledSentence1 = gensim.models.doc2vec.TaggedDocument

all_content = []
j=0

for title in query_result['title_clean'].values:
    all_content.append(LabeledSentence1(title,[j]))
    j+=1
print("Number of texts processed: ", j)

cores = multiprocessing.cpu_count()

d2v_model = Doc2Vec(dm=1, vector_size=100, negative=5, hs=0,
min_count=2, sample = 0, workers=cores, alpha=0.025,
min_alpha=0.001)

d2v_model.build_vocab([x for x in tqdm(all_content)])

all_content  = utils.shuffle(all_content)

d2v_model.train(all_content,total_examples=len(all_content), epochs=30)

So d2v_model.docvecs.doctag_syn0 returns me vectors of all the sentences
I want to now perform clustering using DBSCAN but since I have the other price column which is numeric I am having some trouble fitting the final data to the model. I have a similar problem as described in Stackoverflow, one of my columns has an array of 100 sizes each row, and the other column is just numeric. Hence when I perform dbscan on the data I get the same error.
Is there any smart way to handle such cases? Combining doc2vec output with other numerical columns to prepare it for clustering? Something like this, where both_numeric_categical_columns is the desired input to the model:
clf = DBSCAN(eps=0.5, min_samples=10)
X = clf.fit(both_numeric_categical_columns)
labels=clf.labels_.tolist()

cluster1 = query_result_mini.copy()

cluster1['clusters'] = clf.fit_predict(both_numeric_categical_columns)

You did not mention which package you are using. If you using scikit-learn,
sklearn.pipeline.FeatureUnion concatenates results of multiple transformer objects.
Something like this:
from sklearn.cluster             import DBSCAN
from sklearn.pipeline            import FeatureUnion, Pipeline
from skearnsklearn.preprocessing import StandardScaler

pipeline = Pipeline([('feats', FeatureUnion([
                        ('doc2vec', d2v_model,        ['sentence1', 'sentence2']),
                        ('numeric', StandardScaler(), ['price'])
                    ])),
                    ('cluster', DBSCAN())
                    ])


Can I do bagging method as improvement technique to decision tree in research? Bagging use decision tree as base classifier. I want to use bagging with decision tree(c4.5) as base as the method that improve decision tree(c4.5) in my research that solve problem overfitting. Is that possible because some lecturers said not right as bagging is other classifier not hybrid between two?

Let's clarify a few things first:

The bagging technique is an ensemble method which is not specific to decision trees, it can be applied to any classification method.
It's worth noting that there is another ensemble method specifically for decision trees, it's called Random Forest. While it's not the same method, it is known to generally improve performance compared to a regular Decision Tree algorithm like C4.5.

These techniques exist because they have been proved to improve performance in general, but whether they would improve performance on a specific problem (and by how much) has to be tested.
Also just to be clear: these techniques are already established, so using them wouldn't be an original research contribution in the field of Machine Learning. Their application to a new problem in a specific research domain might be a contribution to this domain, but that depends on the specific context.


Found input variables with inconsistent numbers of samples: [30, 24] I'm using neural network machine learning and would like to see the result of my confusion matrix for my model. However, there is an error that I've got and don't know how to solve it.
from sklearn.metrics import confusion_matrix
cm=confusion_matrix(testY, testPredict)
print(cm)

then it give an error stated: Found input variables with inconsistent numbers of samples: [30, 24]
actually I've check the shape of the test and the prediction value and it shows the different shape. How can I make the shape of the test and prediction become same ?
the

I had the same problem, here is how I sloved it:
testX is a tf.data.Dataset, I guess.
In that case try the following:
# make the prdictions from your test set:
testPredictRaw = self.model.predict(testX)
testPredict = np.argmax(testPredictRaw, axis=1)

# Then, take all the y values from the prefetch dataset (thus changing the shape):
trueClasses = tf.concat([y for x, y in testY], axis=0)

# Calculate the confusion matrix using sklearn.metrics
cm = metrics.confusion_matrix(trueClasses, testPredict)

I used these answered questions to make it work for my case:

https://stackoverflow.com/questions/53620259/sci-kit-learn-confusion-matrix-found-input-variables-with-inconsistent-numbers
https://stackoverflow.com/questions/64622210/how-to-plot-confusion-matrix-for-prefetched-dataset-in-tensorflow


What is the interpretation for quadratic functions? I am working through the book Applied Predictive Modeling and came across something that was a bit confusing.

It discussed adding non linearity to a model to improve its fit - I get this part.

For example: $x^2 + 2x - 4$

What is the interpretation of these values though?

When we are using just a normal linear regression or a multivariate regression, we would say that the coefficients like 2 would suggest its relative importance relative to the other features being included in the model. However, what does this mean in the context of quadratic functions?

ie. Fuel efficiency of a car based on 2 Displacement + Displacement$^2$ -4

What exactly does displacement squared mean?

Any help would be greatly appreciated.

Thanks.

$Displacement$ component gives us a "line" to fit over the data points. To get more freedom add $Displacement^{2}$ a "curve" element. This adds to the flexibility - with same feature/ variable - to map the data points.
Please refer page 90, 91 on Introduction to Statistical Learning in R - Hastie, Tibshirani


Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model? I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the performance/accuracy of the multipose estimation model here.

What I would like to do next is to create a model similar to this one, except it should label each pose of the person(s) detected. There are multiple different implementations caffe/pytorch/tensorflow to choose from. I've thought about how to approach this and I have thought of a few different ways:

Create a completely new machine learning model and use the labeled output of the pose estimation model to train it.
Change or add layers to the machine learning model to change the output. (Not sure how this is done)
Ditch the pose estimate model and train a new model to directly estimate using raw images/labels of cropped people. This would rely on another method to detect each person.

I want to take the path of least resistance here but I also care about the time it takes to gather/process data, and most importantly the accuracy/performance of the model. Are there any experienced Machine Learning/Data Scientists who answer the following?

Which approach should I take? advantages/disadvantages
Which machine learning library offers the functions to do this.
My assumption is that option 1 or 2 would be more accurate than option 3. Am I correct?

I worked in a similar kind off object detection problem.so, my suggestion is Transfer Learning.

i)
The optimal way is to go with transfer learning.It allows us to build accurate models in a timesaving way.With transfer learning, instead of starting the learning process from scratch, you start from patterns that have been learned when solving a different problem. This way you leverage previous learnings and avoid starting from scratch.

A pre-trained model is a model that was trained on a large benchmark dataset to solve a problem similar to the one that we want to solve. Accordingly, due to the computational cost of training such models, it is common practice to import and use models from published literature (e.g. VGG, Inception, MobileNet).

Several pre-trained models used in transfer learning are based on large convolutional neural networks (CNN).

When you’re repurposing a pre-trained model for your own needs, you start by removing the original classifier, then you add a new classifier that fits your purposes, and finally you have to fine-tune your model according to one of three strategies:

1.Train the model.
2.Train some layers and leave the frozen.
3.Freeze the convolutional base.

Approaches for Transfer learning process:

1)Select a pre-trained model.
2)Classify your problem according to the Size-Similarity Matrix.
3)Fine-tune your model.

ii) Tensorflow offers a vision model named as PoseNet.

PoseNet is a vision model that can be used to estimate the pose of a person in an image or video by estimating where key body joints are.
From here you can freeze the model and fine tune it. orelse you can use other pretrained image model (e.g. mobilenet , resnet).

iii) The estimation directly from the raw images first request you to identify or detect the people , so it relays on another model.


Bert for QuestionAnswering input exceeds 512 I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long document, but there's no pretrained in spanish so, is there any idea get around bert.
What i tried is:
from transformers import BertConfig
config=BertConfig.from_pretrained(BERT_MODEL_PATH)
config.max_length=4000
config.max_position_embeddings=4000
config.output_hidden_states=True
model = MyBertModel(config)

but still gives me an error mismatch

RuntimeError: Error(s) in loading state_dict for BertModel:
size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([4000, 768]).

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum.
This limitation, nevertheless, is not arbitrary, but has a deeper justification: in a vanilla transformer, the memory requirements are quadratic on the input length. Therefore, limiting such a length is necessary for the model not to need too much memory.
There are variants of the Transformer architecture that are designed to overcome the quadratic memory problem, like Reformer, Linformer, Longformer, BigBird. They, however, are not "compatible" with the Transformer weights, so someone would need to train one of those models in a masked language modeling task (potentially with multitask learning on the next sequence prediction task) in order to reuse their weights in your context.


How should I construct a binary classifier for thousands of positive data and millions of unlabeled data? So far, I have stumbled upon many advices and papers on PU Learning and Unary classification.

TLDR: Does anyone have suggestions for specific algorithm or implementation for labeled data of only one class and unlabeled data that can be from either classes? And I'm unsure what is the proportion of Class A to B that exists within the unlabeled data.

The simplest answer I have found has been one-class SVM (Binary semi-supervised classification with positive only and unlabeled data set), but I have so many unlabeled examples compared to how many labeled ones I can find. And I am unsure if either the positive class or negative class are rare enough for anomaly detection.

One of the other suggested methods is the two-step process where I can figure out a set of reliable negative class data, but I cannot really identify a set of data as reliably negative (https://www.cs.uic.edu/~liub/publications/ICDM-03.pdf).

And another method suggests a weighted SVM (http://users.csc.tntech.edu/~weberle/Spring2011/CSC6910/Papers/posonly.pdf), but I am unsure if I can make the same assumption as the authors in that my positive data is a random subset of all the positive data, as I used a criteria to figure out which ones were positive, so I assume there is bias in the labeled data.

Overall, I have a lot of labeled data of positive class, that is to say the data of what I am looking for, but then I have many more unlabeled data. (Though in a way, the labeled data could also be considered data of a negative class.) And I am unsure what proportion of positive data and negative data exists within the unlabeled data, as there could be an equal distribution between the two classes. Or who knows, maybe data of the positive class could be rarer than the data of the negative class.

I think the problem you mentioned is sometimes countered as a deep one-class classification problem. You may want to Google for latest researches in this field.

One paper I've been reading is: Learning Deep Features for One-Class Classification.

Available on ArXiv.


Do adaptive learning optimizers follow the steepest decent? For adaptive learning optimizers such as Adam and RMSProb, the effective learning rate is not the same for all weight parameters. This means that we are not really following the direction of the steepest decent vector in the weights space, correct? If so, why then should we bother with computing the gradient (via backprop) in the first place?

Adam and RMSprop are both optimization algorithms which make the vanilla gradient descent more robust. You still need to calculate the gradient however, these algorithms will facilitate the descent in certain directions and punish the descent in others based on some criteria.

Before delving into the details consider this analogy; imagine you are on a mountain and you want to get back to the village. To do this you will follow the downhill slope until you get to the bottom of the mountain. However imagine a mountain that looks like the following

Evidently the best path down the mountain should just be straight down the snowy path. However, if we carefully calculate the gradient we will notice that it will always have a slight left and right component. This will cause us to oscilate from left to right while going down the mountain. This is a huge waste of energy, but it's not catastrophic. However, when we start reaching the base of the mountain the downward slope will start to subside and the oscillations will become more prominent. This may cause us to never truly reach the minimum due to the oscillations becoming dominant, or cause the learning process to be dragged on excessively thus wasting resources.

The optimization techniques such as Adam and RMSprop simply make adjustments to the gradient descent step such that we can more directly find the minimum of the objective function without distraction from oscillating gradients. This is achieved in RMSprop and Adam by maintaining statistics relevant to the past computed gradients. RMSprop keeps track of the mean and Adam keeps track of both the mean and the moment of the past gradients.

An amazing reference for the inner-workings of the gradient descent variants can be found here.

Gradient descent

Vanilla gradient descent is defined as

$\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta)$

where $\theta$ is the set of parameters which define our objective function $J(\theta)$. The learning rate is $\eta$ and $\nabla$ defines the gradient operator. We will extend this function.

Adding Momentum

Firstly, if we throw a ball down our cliff it will get some momentum as it falls. This will cause it to tend towards the steepest part of the gradient and left to right oscillations would be minimized. We can do the same to the gradient descent algorithm by introducing momentum $\gamma$ as

$\nu_{t} = \gamma \nu_{t-1} + \eta \nabla_\theta J(\theta)$

$\theta_{t+1} = \theta_{t} - \nu_t$

This is a good result but we can still do better.

Adagrad

It is preferable to have a distinct learning rate for each parameter such that we can adjust their impact on the gradient based on their past values. This is achieved by weighing the gradients by a sum of squares term of their past gradients up until the current time $t$. The matrix $G_t$ is a diagonal matrix where its entries are precisely this sum of squares of past gradients for each parameter. The Adagrad update is thus,

$\theta_{t+1,i} = \theta_{t,i} - \frac{\eta}{\sqrt{G_{t,ii} + \epsilon}}\nabla_{\theta_t}J(\theta_{t,i})$.

However, this method suffers from diminishing gradients when the sum of squares matrix makes it such that the past values are all too small and cause no more updates to be possible. Or for parameters which may be impactfull in the future to lose all relevance due to their low gradients at the start.

RMSprop

This is a solution to the diminishing gradients problem. We will use a running average of the past gradients until a time $t$ as defined by $E[g^2]_t$. The update rule is thus

$E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g^2_t$

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}}g_t$

where $g$ is the gradient for a single parameter.

Adam

In addition to keeping track of the running average of the past gradients we will also keep track of their second order moment. $m$ is the mean and $\nu$ is the moment. Thus we keep track of them as follows

$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$

$\nu_t = \beta_2 \nu_{t-1} + (1-\beta_2)g^2_t$

We will correct these terms to remove their bias that tends towards a zero vector as

$\hat{m}_t = \frac{m_t}{1-\beta^t_1}$

$\hat{\nu}_t = \frac{\nu_t}{1-\beta^t_2}$

Finally, the update rule is

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{\nu}} + \epsilon}\hat{m}_t$


TensorFlow / Keras: What is stateful = True in LSTM layers? Could you elaborate on this argument? I found the brief explanation from the docs unsatisfying:

  stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.

Also, when stateful = True is to be chosen? What are practical cases of its use?

This flag is used to have truncated back-propagation through time: the gradient is propagated through the hidden states of the LSTM across the time dimension in the batch and then, in the next batch, the last hidden states are used as input states for the LSTM.

This allows the LSTM to use longer context at training time while constraining the number of steps back for the gradient computation.

I know of two scenarios where this is common:

Language modeling (LM).
Time series modeling.

The training set is a list of sequences, potentially coming from a few documents (LM) or complete time series. During data preparation, the batches are created so that each sequence in a batch is the continuation of the sequence at the same position in the previous batch. This allows having document-level/long time series context when computing predictions.

In these cases, your data is longer than the sequence length dimension in the batch. This may be due to constraints in the available GPU memory (therefore limiting the maximum batch size) or by design due to any other reasons.

Update: Note that the stateful flag affects both training and inference time. If you disable it, you must ensure that at inference time each prediction gets the previous hidden state. For this, you can either create a new model with stateful=True and copy the parameters from the trained model with model.set_weights() or pass it manually. Due to this inconvenience, some people simply set stateful = True always and force the model not use the stored hidden state during training by invoking model.reset_states().


How many RNN units are needed for tasks involving sequences? I am training an RNN on the following task:
Given a sequence of thirty words, predict the next word.

Is there a benefit to having more than 30 cells (LSTM, GRU or plain RNN) in my network?
I've seen many examples online where similar networks are trained with multiple layers that each have 100 cells, but this does not make sense to me.
How does it help to have more cells than the length of the sequence? (in my case this length is 30)
I'm confused because from my understanding, each cell takes in two inputs
1. A new element of the sequence
2. The output from the previous cell So after 30 cells, there will be no new sequence elements to input into the cell. Each cell will just be processing the output of the previous cell (receiving no new info).

I am using LSTM cells for this task (however, I'm guessing the actual type of RNN cell used is irrelevant).

You don't need to assign an RNN cell to each entry in a sequence, you just need one.

A single cell can process sequences because of its recursive nature(its output feeds back into itself):

$$ h_t = f(h_{t-1}, W,  x_t) $$

where $h_t$ is the hidden state at step $t$, $W$ is the weight used for the cell, and $x_t$ is the current input at step $t$ .

Hopefully, you can see how it is only necessary for us one weight matrix, $W$, instead of multiple as your understanding would imply.

Where you got the notion of multiple RNN cells is probably when we feed the hidden state as the input, $x$, into another RNN cell. More layers of RNN cells, mean you can learn a more complex sequence.

Conclusively, your RNN cell cell can't work without new inputs, therefore it end when your sequence ends.
Also, it is recommended to use LSTM or GRU over vanilla RNN units.


Evaluating Logistic Regression Model in Tensorflow Following this tutorial, I have a doubt about the evaluation part in:

# test the model
n_batches = int(mnist.test.num_examples/batch_size)
total_correct_preds = 0
for i in range(n_batches):
    X_batch, Y_batch = mnist.test.next_batch(batch_size)
    _, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch})
    preds = tf.nn.softmax(logits_batch)
    correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y_batch, 1))
    accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) # need numpy.count_nonzero(boolarr) :(
    total_correct_preds += sess.run(accuracy)

print 'Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)

Note that this is done on the test set, so the goal is purely to obtain the accuracy using a previously trained model. However isn't calling the line:

_, loss_batch, logits_batch = sess.run([optimizer, loss, logits], feed_dict={X: X_batch, Y:Y_batch})

equivalent to re-optimizing the model using the test data (and labels)?
Shouldn't we avoid re-running optimizer and loss and just compute the predictions?

I think you are correct. The line should be

loss_batch, logits_batch = sess.run([loss, logits], feed_dict={X: X_batch, Y:Y_batch})


RBF neural network python library/implementation I want to use a Radial Basis Function Neural Network for my thesis.

Is there any library that implements it? And in the negative case, which is the best library to implement it?

You can simply use gaussian as the nonlinearity and use it in Tensorflow. I guess it is the easiest solution.


Problem with convergence of ReLu in MLP I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
The problem I observed depends on initial weights. When using sigmoid function it does not matter that much if weights are random numbers in ranges of [0,1] or [-1,1] or [-0.5,0.5]. But when using ReLu the network very often has a huge problem with ever converging when I'm using random weights in range [-1,1]. But when I changed the range of initialization of weights to [-0.5,0.5] it started to work. This only applies to ReLu activation function and I totally don't get it why it won't work for [-1,1]. Shouldn't it be able to converge with any random weights?
Also when I changed initial weights to normal distibution, it has no problem with convergence. I understand that normal distribution should work better and faster than random [-1,1]. What I don't understand is why it can't converge (error remains the same epoch after epoch) with [-1,1] and has no problem with converging with normal distribution... Shouldn't it always be able to converge just slower and faster with different initialization method?
PS. I'm using normal backpropagation with softmax as last layer and MSE as loss function

I will start with a toy example for the convergence part. Suppose that the loss function is $f(x) = x^4$ and we want to minimize it using the gradient descent. Clearly, the minimum is attained at zero and, in general, we would like the magnitude of the current approximation to decrease.
The update rule of gradient descent is
$$ x_{k+1} = x_k - \lambda \nabla f = x_k - \lambda \cdot 4x_k^3.$$
Simplifying the expression, we get
$$x^{k+1} = x^k(1 - 4\lambda x_k^2).$$
And now the combo initialization + learning rate start to appear. If $|1-4\lambda x_0^2| < 1$, then $|x_0| > |x_1| > |x_2| > ...$. The sequence will go to zero eventually. If $|1-4\lambda x_0^2| > 1$, then $|x_0| < |x_1|$. In this case, $|x_1| < |x_2|$ and so on -- the sequence will grow. Therefore, if the learning rate $\lambda$ if fixed, then the initial values $x^0$ determine if the grad descent converges or not.
When the gradient descent converge?
The math says that the gradient descent converge when the learning rate $\lambda$ and the gradient $\nabla f$ satisfy
$$\|\lambda \nabla f\| < 1$$
along the optimization path. Note that this condition does not need to be valid for all values of $x$, but at every $x_0, x_1, x_2, ...$
For many "good" functions, it suffices to require $\|\lambda \nabla f\| < 1$ only at $x_0$. The reason is that after the first iteration, we are closer to the local minimum, and for many "good" functions, it means that the gradient will be smaller.
What about weight choice in [-0.5,0.5] and [-1, 1]?
I think of it as follows: Suppose that we selected weights in $[-0.5, 0.5]$ (model 1) and then multiplied all weights by 2 to get uniform distribution in $[-1, 1]$ (model 2). Suppose that the learning rate is identical in both cases, and let's check how the SGD performs. For simplicity of the argument, I replace it with the gradient descent.
How does it transfer to the NN?
Note that a linear map (say, for the dense layer) has the following property. Suppose that all weights W are multiplied by 2, then
$$\|2W\| = 2\|W\|.$$
ReLU is quazi-linearly-scalable: for every $a > 0$,
$$a\cdot\text{ReLU}(x)=\text{ReLu}(ax).$$
Note that if your NN has $d$ layers, then, multiplying all weights by 2, your output increases in $2^d$ times (factor of 2 for each layer).
It is hard to compute the gradient for the NN, but using the product rule, I expect it to increase in approximately $2^{d}$ times. If model 1 satisfies the conditions for the gradient descent convergence, then we need to decrease the learning rate about $2^{d}$ times to guarantee the gradient descent convergence condition.


Which data science model is best for explainability for prediction problems? Imagine you have to create a model to explain to stakeholders e.g. to predict price, weight, sales etc.. Which regression models offer the best in terms of explainability and interprability? ... Which model is best for this?

Personally I think linear (through model's coefficients/weights) and tree-based models (gain importance) are the best for explainability
But this is not restricted to those models since you can use model agnostic techniques to explaine any model, even those consider as "black-box"
Like:

SHAP Values
Partial Dependence plot
LIME

You can check this good resource to learn more on that.
A final comment I'd say is that you cannot forget the fact that an essential part of model explainability is model performance, It does not make sense to have a model that is easily explainable but with a very low performance, since those structures found by the model do not generalize well. So a good model in performance will lead you to the "correct" conclusions when explaining it.


Representing a higher-dimensional chart I'm hitting an unusual roadblock in my quest to represent a set of data for the layman, and thought I'd ask for advice on how best to accomplish this task.

My data points are represented by a 4-float tuple (a, b, c, d), the sum of which is constant. These represent test conditions, to which an outcome z is assigned. This outcome is also a float, but that's largely irrelevant.

The fact that there are four effective x-axes really is throwing me off regarding the choice of visualization, and that's where I'd really like some help. If you were in my shoes, what would you render this as?

Just for completeness sake, here is a (small) extract of one of the sets:

0,3000,2500,750,101788
250,3000,2500,500,100458
500,3000,2500,250,99439
750,3000,2500,0,98573
0,3000,2750,500,101993
250,3000,2750,250,100834
500,3000,2750,0,99813
0,3000,3000,250,102370
250,3000,3000,0,101150

I think there is 2 ways to do it.

Dimensionality reduction

Using technique such as PCA, you can represent your data in lower number of dimensions.

This is a good approach if you want to represent all your data in one graph.

Dimension removal

Another way to visualize is simply to visualize only a few dimensions at a time. For example you can visualize a/b, c/d, a/c in 3 different 2D graphs.

While much simpler, this approach does not allow you to have a single graph visualization.


Is there an Orange server to deploy the models developed in Orange 3 I am new to Orange 3 and while the tool looks very promising, I was wondering if there is an Orange server where the model can be deployed for a production instance.

No. Orange is a stand-alone, desktop application and is not a tool for designing analysis pipelines that would on their own run on the servers. Instead, it fosters interactive visualizations and exploratory data analysis that all require human input.


forecast monthly shipment (time series) for 300 products individually in R I can build ARIMA model with regressor to forecast monthly shipment for one product. but I have 300 products and each of them needs a monthly forecast.

my question is, instead of building 300 models, is there another approach in R to deal with the goal I am trying to accomplish?

thank you!

It is difficult to answer this question with the limited information provided here. Nonetheless, I have following recommendations.

Do you have a (data backed) reason to believe that you do not need 300 different models? For example, are the shipments of different products correlated? Alternatively, can you instead model the problem at higher levels like product categories? If either case, you can take a look at hierarchical time series modeling. This journal article by Rob Hyndman, author of forecast package in R can get you started on hierarchical time series modeling.