Can I analyze Video in AI? For those who do AI or Machine Learning, is it possible to analyze video? For example, here’s a tidbit from an article I’ve been reading:

  in basketball, data can help coaches determine if a player executed a jump shot well because he was skilled, the pass was good or the defense was bad. If he is skilled, he can be used to lead key strategies. If the pass is good, it identifies players that work well together and use them for the same. If the defense is bad, coaches can strategize plays that take advantage of this weakness.

How would I measure this with AI? I’d like to find the true value of a high school player and how they’d affect their college team.

Yes, it's possible, especially using Deep learning.
To get an idea, you can see this github project:
 https://github.com/HuaizhengZhang/Awsome-Deep-Learning-for-Video-Analysis


How to generate synthetic dataset using machine learning model learnt with original dataset? Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset characteristics?
[original data --> build machine learning model --> use ml model to generate synthetic data....!!!]
Is it possible? Please point me to related resource if possible.

The general approach is to do traditional statistical analysis on your data set to define a multidimensional random process that will generate data with the same statistical characteristics.  The virtue of this approach is that your synthetic data is independent of your ML model, but statistically "close" to your data. (see below for discussion of your alternative)
In essence, you are estimating the multivariate probability distribution associated with the process.  Once you have estimated the distribution, you can generate synthetic data through the Monte Carlo method or similar repeated sampling methods.  If your data resembles some parametric distribution (e.g. lognormal) then this approach is straightforward and reliable.  The tricky part is to estimate the dependence between variables. See: https://www.encyclopediaofmath.org/index.php/Multi-dimensional_statistical_analysis.
If your data is irregular, then non-parametric methods are easier and probably more robust.  Multivariate kernal density estimation is a method that is accessible and appealing to people with ML background. For a general introduction and links to specific methods, see: https://en.wikipedia.org/wiki/Nonparametric_statistics .
To validate that this process worked for you, you go through the machine learning process again with the synthesized data, and you should end up with a model that is fairly close to your original.  Likewise, if you put the synthesized data into your ML model, you should get outputs that have similar distribution as your original outputs.
In contrast, you are proposing this:

[original data --> build machine learning model --> use ml model to generate synthetic data....!!!]

This accomplishes something different that the method I just described.  This would solve the inverse problem: "what inputs could generate any given set of model outputs". Unless your ML model is over-fitted to your original data, this synthesized data will not look like your original data in every respect, or even most.
Consider a linear regression model. The same linear regression model can have identical fit to data that have very different characteristics. A famous demonstration of this is through Anscombe's quartet.

Thought I don't have references, I believe this problem can also arise in logistic regression, generalized linear models, SVM, and K-means clustering.
There are some ML model types (e.g. decision tree) where it's possible to inverse them to generate synthetic data, though it takes some work.  See: Generating Synthetic Data to Match Data Mining Patterns.


How does tree-based algorithms handle linearly combined features? While I am aware that tree-based algorithms (e.g., DT, RF, XGBoost) are 'immune' to multi-collinearity, how do they handle linearly combined features?
For example, is there is any additional value or harm in including the three feature: a, b and a+b in the model?

If the sum of the two feature makes sense on the domain semantically, it might be a good idea.
But while trees can handle redundant features pretty well, increasing the number of features without adding any extra "value" or "information" can lead to lower performance in certain situations. For example, if there is no added value and you aggressively sample or restrict the number of features in each tree the two related feature will take the place of a useful features when they are both selected.
If you think the sum is a better feature than the partitions you might consider addig the sum only, or the sum and one of the components.


High overestimation on prediction data I am building lost sales estimation model for out of stock days etc. using XGBoost. I am using simple logic of training model on data of normal days with ample inventory (when sales and demand are same) and then using trained model to predict demand on out of stock days. For model building I am splitting the normal days data into train and test datasets.

However I am getting a peculiar problem of very highly overestimated sales values of out of stock days. prediction for both train and test days are fine but only out of stock days prediction gives problem. Any tips what might be going wrong and how I can debug the problem in gradient tree type of model.

Disclaimer: I am not 100% sure if my solution will be good for all cases but this solved my problem considerably.

For me changing to XGBoost based imputation worked wonderfully. In the earlier case I was doing multiple different type of imputation like mean, mode etc. From there I retained only those imputations on which I was very much confident. Imputations on which I didn't have much confidence I did let XGBoost do the job. After this change most of the overestimation cases vanished.

Another benefit I got from this change was: model fit parameters were direct reflective of the overestimation. So models in overestimation was still happening; all of those have had bad model fit Vs models with no overestimation always have good good fit.

I believe this is happening because with XGBoost also doing the imputation it uncovers and leverages deeply hidden patterns in data. With this knowledge it is able to do better imputation of the scenarios where my ways of ordinary imputation were not much useful.


How can i add weights in a bag of words model in text analysis? I have a twitter sentiment analysis using bag of word approach from the training set. Now i want to add weights to certain words so that they are considered more important than others.

One possible solution is to introduce prior counts for words (higher counts for words that are more important) that could be added to the term-document matrix.

An alternative solution is to compute tf-idf features (weights that modify word counts based on frequency) and apply additional weighting to tf-idf with higher weights corresponding to important words.


Classification Dataset which is linearly non separable I am trying to find a dataset which is linearly non-separable. I checked the Iris dataset and the UCI website says:

  The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable
  from the other 2; the latter are NOT linearly separable from each
  other

Source: https://archive.ics.uci.edu/ml/datasets/iris)

However, I need a two class dataset which is linearly non-separable.

Please give some pointers.

In Python you can use the in-built make_moons and make_blobs functionalities of scikit-learn, changing the parameters accordingly.

The "titanic dataset" and "mpgcars" also contain non-linearly separable categories, in addition.


Getting extra characters while loading data While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?

Try using endcoding=utf-8
Code:
import pandas as pd
mydf=pd.read_sas("\home\mypath\mydataset.sas7bdat",format = 'sas7bdat', encoding="utf-8")

If the sas dataset is small, I would recommend loading from CSV file instead of sas dataset. Performance loading from SAS using Python is poor.


Extracting individual emails from an email thread Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email message from a thread of emails.

For example consider the below message.

Hi,

Can you offer me a better discount.

Thanks,
Mr.X
Customer Relations.

---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------

Hello Mr.X,

Does the below work out. Do let us know your thoughts.

Thanks,
Mr.Y

Sales Manager.

Now the reason why we want to split the emails is because we want to do sentiment analysis. When we fail to split the email then the results will be wrong.

I searched around and found this very comprehensive research paper. Also found an implementation by mail gun called mail gun talon. But unfortunately it does not work well for certain kind of patters.

For example when the second message in the email thread breaks like

---------- Forwarded message ----------

instead of the above

---- On Wed, 10 May 2017 04:05:16 -0700 someone@somewhere.com wrote ------

My question is many people who are trying to do such stuff would have definitely faced such problems, but yet the area remains pretty shady. Is there any pretty solid implementation of the paper or something else that splits email pretty well.

First you can read about the format of ARPA internet text messages.

You're going to have to one up on Regex parsing if you are going to want to extract any meaningful info from the emails. I'd suggest the Oreilly book on regex, or reading over regular expressions.

There is also a tool to capture text from incoming emails and write to a text file.


What are the factors to consider when setting the depth of a decision tree? In scikit learn, one of the parameters to set when instantiating a decision tree is the maximum depth. What are the factors to consider when setting the depth of a decision tree? Does larger depth usually lead to higher accuracy?

Yes, but it also means you're likely to overfit to the training data, so you need to find the value that strikes a balance between accuracy and properly fitting the data. Deciding on the proper setting of the max_depth parameter is the task of the tuning process, via either Grid Search or Randomised Search with cross-validation.

This page from the scikit-learn documentation explains the process well: https://scikit-learn.org/stable/modules/grid_search.html


What is a sliding-window convolutional neural network? In the abstract of "U-Net: Convolutional Networks for Biomedical
Image Segmentation", the authors mention a sliding-window convolutional neural network. I've found several other articles mentioning this but no clean definition of it. How is it different from regular convolutional networks/what does the term mean?

Sliding window in this context is regarding what is given as input to the CNN. It is a sliding window of the input image. I have seen it being used in medical domain where the images are too large to fit into a network and reshaping them into smaller sizes doesn't help. So, a sliding window is done on the bigger image and the sliding window is fed into CNN for achieving whatever task you are trying to achieve.


Preparing Dataset Minority Class vs Majority Class I'm currently doing a binary classification for sentiment prediction. Currently I have the majority class (~90% of the data) as my positive class (labelled 1) and the minority class (~10% of the data) as my negative class (labeled 0). What I'd like to maximize in this experiment is the detection of negative sentiments, hence I'd like to maximize the precision (and recall) of my minority class.
However, in many similar datasets (in terms of prioritizing the detection of minority class) out there like credit card fraud detection, cancer detection, usually the minority class is set as the positive class and the majority class set as the negative class.
My question is: Does it matter if the minority class is set as the positive or negative label in relation to performance of training a model or affecting a loss function such as cross entropy?

My question is: Does it matter if the minority class is set as the positive or negative label in relation to performance of training a model or affecting a loss function such as cross entropy?

No it doesn't.
However in binary classification it's customary to call "positive" the main class of interest, so be careful to be clear about which one is positive/negative when/if you present your results to other people.
Also be careful that precision and recall are usually calculated for whatever is called the positive class, so don't inadvertently use the results of the majority class instead of the one you're interested in.


Segmentation of time series into one-second non-overlapping windows I need to split the time series dataset (accelerometer values (timestamp, X, Y, Z)) into segments in the form of a one-second window that does not overlap. I am trying to find an example of its implementation, but found nothing. I'm a beginner and don't know how to do this.

This answer has an implementation for audio spectrograms, but it is exactly the same for any multivariate time-series.
Here is the most relevant part of the code. To not have any overlap, set window_hop = window_size.
window_size = 64
window_hop = 30
start_frame = window_size
end_frame = window_hop * math.floor(float(frames.shape[1]) / window_hop)

for frame_idx in range(start_frame, end_frame, window_hop):
    window = frames[:, frame_idx-window_size:frame_idx]


'DataFrame' object has no attribute 'to_dataframe' I'm sure I have a small error here that I'm overlooking, but am having a tough time figuring out what I need to change.

Here is my code up until the error I'm getting.

# Load libraries
import pandas as pd
import numpy as np
from pandas.tools.plotting import scatter_matrix
import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analyisis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# Load dataset
names = ['action','reject','approve','occ','loanamt', 'suffolk', 'appinc','typur','unit','married','dep','emp',yjob','self','atotinc','cototinc','hexp']

# from azureml import Workspace
# ws = Workspace(
#      workspace_id='',
#      authorization_token='==',
#      endpoint='https://studioapi.azureml.net'
# )
# ds = ws.datasets['loanapp_c.csv']

ds = pd.read_csv('desktop/python ML/loanapp_c.csv')
dataset = ds.to_dataframe()

I was running this on Azure and am now trying to do it locally. Here is the error I'm getting:

AttributeError                Traceback (most recent call last)
<ipython-input-3-b49a23658806> in <module>()
     32
     33 ds = pd.read_csv('desktop/python ML/loanapp_c.csv')
---> 34 dataset = ds.to_dataframe()
     35
     36 # shape

~/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in_getattr_(self, name)
     4374 if self._info_axis.can_hold_identifiers_and_holds_name(name):
            return self[name]
  -> 4376 return object._getattribute_(self,name)
     4377
     4378 def _setattr_(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'to_dataframe'

Not sure what I have wrong.

The function pd.read_csv() is already a DataFrame and thus that kind of object does not support calling .to_dataframe().

You can check the type of your variable ds using print(type(ds)), you will see that it is a pandas DataFrame type.


Why does feature scaling improve the convergence speed for gradient descent? From this article, it says:

  We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.

For linear regresion, from the equation $a_{n+1}=a_n-\alpha\nabla F(a_n)$ it wasn't obvious to me why larger range variables converge slower or how/why an oscillation occurs.

The answer to this question simply lies in how one performs or applies the chain rule of differentiation. To understand this, first look at back-propagation which is simply a method to update weights by calculating the error between the predicted value and the exact value. When this error is back-propagated to optimize your objective function, derivatives are calculated in the subsequent layers and are multiplied by learning rate and the weights are then updated.

Now just imagine, if you have some features and range of one feature is from 0 to 1 and the other feature's range is from 10 to 1000. Now when you do differentiation for back-propagation for features with high range values you may get derivatives which are greater as compared to features which have low range values, this would lead to abnormal weight updating, and would add certain biasness to our model regarding those features, even when that feature as compared to the feature which has low range values may hold equal importance when statistical testing is done.

Thus to avoid this, introduction of biasness, feature scaling is used which allows us to scale features in a standard scale without associating any kind of biasness to it. This applies to various machine learning models such as SVM, KNN etc as well as neural networks. Another reason to scale features is to avoid large derivatives being back-propagated which increases the computation.

Another curious aspect of scaling is if you don't scale your features you will have different error surface (because of introduction of biasness when it is not there) as compared to scaled features error surface this may prevent you from reaching the optimum minimum value for your objective function thus that is why scaling helps you to ensure that you at least reach local minima thus it can be said scaling helps you to increase convergence speed while doing gradient descent.


Hadoop and input informations divided in splits Hadoop divides the input to a MapReduce job into fixed-size pieces
  called input splits, or just splits. Hadoop creates one map task for
  each split, which runs the user-defined map function for each record
  in the split. Having many splits means the time taken to process each
  split is small compared to the time to process the whole input. So if
  we are processing the splits in parallel, the processing is better
  load balanced when the splits are small

Why ?

All bigdata eco system works on something called parallel processing.

We have to process 100gigs of file. If we didnt split the file, then all the 100 gigs should be processed by single JVM(single map).

If we split the file into 1000 parts each of 100mb, then we can process each part with different JVM and apply the map function in less time.

MPP: Massively parallel Processing


What are some key strengths of BERT over ELMO/ULMFiT? I see BERT family is being used as benchmark everywhere for NLP tasks. What are some key strengths of BERT over models like ELMO or ULMFiT?

The reason you're seeing BERT and its derivatives as benchmarks is probably because it is newer than the other models mentioned and shows state-of-the-art performance on many NLP tasks. Thus, when researchers publish new models they normally want to compare them to the current leading models out there (i.e BERT).
I don't know if there has been a study on the strengths of BERT compared to the other methods but looking at their differences might give some insight:

Truly Bidirectional
BERT is deeply bidirectional due to its novel masked language modeling technique. ELMo on the other hand uses an concatenation of right-to-left and left-to-right LSTMs and ULMFit uses a unidirectional LSTM. Having bidirectional context should, in theory, generate more accurate word representations.

Model Input
BERT tokenizes words into sub-words (using WordPiece) and those are then given as input to the model. ELMo uses character based input and ULMFit is word based. It's been claimed that character level language models don't perform as well as word based ones but word based models have the issue of out-of-vocabulary words. BERT's sub-words approach enjoys the best of both worlds.

Transformer vs. LSTM
At its heart BERT uses transformers whereas ELMo and ULMFit both use LSTMs. Besides the fact that these two approaches work differently, it should also be noted that using transformers enables the parallelization of training which is an important factor when working with large amounts of data.

This list goes on with things such as the corpus the model was trained on, the tasks used to train and more. So while it is true that BERT shows SOTA performance across a variety of NLP tasks, there are times where other models perform better. Therefore, when you're working on a problem it is a good idea to test a few of them a see for yourself which one suits your needs better.


How do I know that model performance improvement is significant? Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one convolutional layer) and run the model again, leading to an accuracy of .8+x.
My question is how I would determine which in-/decrease in performance allows me to say that the new network architecture is better than my old one? I assume that x=.0001 falls within a reasonable margin of error, whereas x=-.2 is a significant decrease in performance - however, the use of "significant" here would be purely colloquial without any scientific backing.
I understand that some kind of hypothesis testing would in theory be appropriate here, but as far as I know, these require multiple samples (i.e. running the network many times), which in case of large ML models which take sometimes days to train isn't really feasible.

Your last paragraph is actually the answer but there is no need to train your model several times. You actually need to "validate/evaluate" your model several times which of course can be run on smaller sets.

Have several validation set. Quantity is not the main major but make sure they cover the distribution of data specially all classes.
run both models on those sets and keep results in two lists/arrays
simple/practical solution: If the difference between their means is larger than their individual standard deviations, then you are good to go
a bit more confident: You have two sets of numbers (performance of two models). use a simple anova and monitor the p-value to see if their differences is significant.


How to normalize the data correctly in spam dataset I'm working on the spam dataset to classify the inputs into binary classes.
my problem is that: the observations in the dataset are floats small numbers in the first 53 column, and the 54 is float larger numbers, while the last two columns are integers.

My Question:
How to Normalize this dataset
correctly, so all the observations have the same importance?
import pandas as pd
spam = pd.read_table("spambase.data",sep=',',header=None)

one proposed approach which didn't seem very convenient to me, because it normalize the whole row input is that:

#========================
# Normalization Function
#========================
def Normalize(x):
    '''
    ==================================
    Normalization Function
    ==================================
    -----------
    Parameters:
    -----------
    @Parameter x: Vector
    ---------
    Returns:
    ---------
    Normalized Vector.
    ================================
    '''
    norm=0.0
    for e in x:
        norm+=e**2
    for i in range(len(x)):
        x[i]/=sqrt(norm)
    return x

Normalizing so that "all the observations have the same importance" is kinda ambiguous and ill-defined. In any case, it would be strongly advised to avoid re-inventing the wheel, and use one of the several scalers available out there (e.g. in the sklearn.preprocessing module).
Here is an example using MinMaxScaler, which will re-scale your data in [0, 1] column-wise:
import pandas as pd
df = pd.read_csv("spambase.data", header=None)
print(df.head())
# result:
     0     1     2    3     4     5   ...     52     53     54   55    56  57
0  0.00  0.64  0.64  0.0  0.32  0.00  ...  0.000  0.000  3.756   61   278   1
1  0.21  0.28  0.50  0.0  0.14  0.28  ...  0.180  0.048  5.114  101  1028   1
2  0.06  0.00  0.71  0.0  1.23  0.19  ...  0.184  0.010  9.821  485  2259   1
3  0.00  0.00  0.00  0.0  0.63  0.00  ...  0.000  0.000  3.537   40   191   1
4  0.00  0.00  0.00  0.0  0.63  0.00  ...  0.000  0.000  3.537   40   191   1

[5 rows x 58 columns]

from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler() # define the scaler
df_scaled = pd.DataFrame(sc.fit_transform(df)) # fit & transform the data
print(df_scaled.head())
# result:
         0         1         2    3   ...        54        55        56   57
0  0.000000  0.044818  0.125490  0.0  ...  0.002502  0.006007  0.017487  1.0
1  0.046256  0.019608  0.098039  0.0  ...  0.003735  0.010012  0.064836  1.0
2  0.013216  0.000000  0.139216  0.0  ...  0.008008  0.048458  0.142551  1.0
3  0.000000  0.000000  0.000000  0.0  ...  0.002303  0.003905  0.011995  1.0
4  0.000000  0.000000  0.000000  0.0  ...  0.002303  0.003905  0.011995  1.0

[5 rows x 58 columns]

Keep in mind that normalization depends also from your choice of a model to use: it is practically necessary for neural networks and k-nn (and for k-means clustering), but is is completely redundant for decision trees and tree-ensemble models (Random Forest, GBM etc).


Cross-validation for model comparison: use the same folds? Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run both models through those folds? Then would the way to assess if M2 outperforms M1 be to do a paired t-test?

I'm mostly thinking about a situation where I have the results of a cross-validation that someone else did and want to see if my model can beat their average of 80% accuracy. In that case, I would not have their exact folds or perhaps not even how many folds they used, so a paired t-test would not be possible. What would be the pitfalls of comparing to their metric or to their 5 metrics on the 5 folds when I don't know exactly how they allocated the observations into folds?

There seem to be a few elements of confusion here. I'll try to answer clearly, but would definitely appreciate input from others.

1. What is cross-validation for?

Cross-validation is generally used as a mechanism for figuring out which of several models might be a better fit for data based on a training set before choosing a model to apply to the test set. This is commonly done when tuning hyperparameters (such as penalty factors in regularized regressions), and in other scenarios as well.

In such a case you would want to use the same folds for cross-validating each model, because you want to identify differences in performance attributable to the models themselves, and not due to effects of randomly partitioning observations into different folds. Importantly, cross-validation is part of the model training process, not final model evaluation.

Cross-validation is not for finding the "best" performing model overall. Model performance is based on application to a held-out test set of observations. The idea is that the model performance will often be artificially high in the training data due to idiosyncrasies among the training data set which are not present in observations overall, and application to the test data suggests how the model might generalize to totally novel observations. Even if cross-validation results are discussed in a study, they are not the appropriate value to compare a model prepared independently and trained on different data.

2. During cross-validation, how would one assess which putative model is "better"?

This question can be a bit controversial, namely in that accuracy (such as of classification) is not a very strong method of identifying the best model during cross-validation. It will work with varying degrees of quality depending on what you're modelling. But better measures are varieties of loss functions, which could be mean squared error, binomial deviance, or many others (it's very situation dependent).

How you would compare the output of those loss functions is usually fairly straightforward-- you want minimum error, minimum deviance, etc. Performing a paired t-test isn't really appropriate. Such a test would tell you something about the distributions underlying your variables for data points that are connected in time (like measures of some variable before and after a treatment is applied) in the context of a specific null hypothesis and alternative hypothesis set. That sort of paired relationship doesn't apply to observations subjected to different models one-at-a-time, and in a mutually exclusive way.

3. What should be done instead?

As outlined above, model performance is assessed on novel data rather than training data. In model development that will be the test set, but it can just as well be application of the model to totally new data. If the models to be compared are similar in complexity, a proper loss function is still ideal. But in many situations classification accuracy is necessary (if that's what your boss is asking for, that may be what you have to provide).

In that case simpler counts of correct classification (9 out of 15 were correctly classified) are appropriate. Better still are measures of model sensitivity and specificity (the model's ability to identify true positives and true negatives, respectively, as in binomial classification) and positive/negative predictive value (how reliable any given classification suggested by the model is) can be very effective.

tl;dr: Cross-validation is the wrong stage to be comparing an already-trained model to a different model that is being trained, and t-tests are not informative in comparing cross-validations. Loss functions or classification accuracy are more appropriate (to varying degrees in varying situations) and should be based on data not used in training.


How to implement an LSTM RNN with multiple input features EDIT: Now I didn't convert to list.

I am training LSTM for multiple time-series in an array which has a structure: 450x801. There are 450 time series with each of 801 timesteps / time series. The labels are classes with assigned integer from 1 to 6, so the dimension of the label is 450x1. This is my implmentation:

This is my code:

def readData():
labels = pd.read_csv('label.csv', header = None)
labels = labels.values
data = pd.read_csv('data.csv', header = None)
return data, labels

data, labels = readData()

data_train, data_test, labels_train, labels_test = train_test_split(data, labels)

model = Sequential()
model.add(LSTM(units=32, input_shape = (450,801,1)))
model.add(Dense(6, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Now I got this error:

Error when checking input: expected lstm_26_input to have 3 dimensions, but got array with shape (450, 801)

Any idea how to solve it?

For instance I can try to reshape:

data_train = np.reshape(data_train, (data_train.shape[0], 1, data_train.shape[1]))
data_test = np.reshape(data_test, (data_test.shape[0], 1, data_test.shape[1]))

And now the error is:

raise ValueError('Must pass 2-d input')
ValueError: Must pass 2-d input

It is becouse you convert your data to list data = data.values.tolist() But model input expects an array but not a list of arrays. So you need to convert your input list to the array.


Pyspark.read from MariaDB fetches duplicate header rows instead of data It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name.
If I don't use the query like below and instead just put the table, it errors out trying to cast the literal column name as its datatype, e.g. it will try to cast literally "dob" as a date. I'm posting because I don't see anything about similar issues elsewhere online.I've tried multiple versions of the JDBC driver, so I'm not sure where to go from here.
Code to reproduce (network and user info removed ofc):
##import required libraries
import pyspark

##create spark session
spark = pyspark.sql.SparkSession \
   .builder \
   .appName("Python Spark SQL basic example") \
   .config('spark.driver.extraClassPath', "/Users/username/Downloads/mariadb-java-client-2.7.3.jar") \
   .getOrCreate()

##read table from db using spark jdbc
patients_df = spark.read \
   .format("jdbc") \
   .option("url", "jdbc:mariadb:my_server:3306/SYNTH_EHR_NORM") \
   .option("dbtable", "(select fname, lname from SYNTH_EHR_NORM.PATIENTS limit 20) tmp") \
   .option("user", "username") \
   .option("password", "not_my_real_password") \
   .option("driver", "org.mariadb.jdbc.Driver") \
   .load()

##print the users dataframe
print(patients_df.collect())

which returns:
[Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname'), Row(fname='fname', lname='lname')]

Sometimes adding an alias to subquery confuses PySpark. Try:
.option("dbtable", "select fname, lname from SYNTH_EHR_NORM.PATIENTS limit 20") \


How to perform upsampling (and NOT interpolation) process theoretically modelled? As an example, I know that sampling a signal $s$ is modelled by multiplication of s by a dirac comb, which has the effect of convolving the Fourier Transform (FT) of $s$ by the FT of the dirac comb which is another dirac comb but with inverse spacing between peaks.
My question is what is the corresponding process for when one upsamples  (e.g., by a factor of 2),      , i.e., inserts zeros between samples,  and NOT yet applies interpolation since I know for interpolation you need to follow upsampling by a lowpassfilter whicg is a convolution a signal $s$?

If I understood your question right, you want a mathematical expression for the I/O (input/output) relationship of a signal expander (name of the block that expands (upsamples) -without interpolation filering- an input signal $x[n]$)
Below is a block diagram of signal expander by a factor of $L$:
$$ x[n] \longrightarrow \boxed{ \uparrow L } \longrightarrow x_e[n] \tag{1}$$
An expression for the expanded sequence $x_e[n]$ can be written as :
$$ x_e[n] = \begin{cases} { x[\frac{n}{L}] ~~~,~~~ n=m\cdot L ~~~,~~~ m=...,-1,0,1,... \\ ~~~ 0 ~~~~~~,~~ ~ \text{otherwise} } \end{cases} \tag{2} $$
An identical expression is also the following :
$$x_e[n] = \sum_{k=-\infty}^{\infty} x[k] \delta[n-L\cdot k] \tag{3}$$
where $\delta[n]$ is a unit sample (discrete-time impulse).
With $L=3$, an input $x[n]=[1,2,3,4]$ becomes output $x_e[n] =[1,0,0,2,0,0,3,0,0,4,0,0]$.


What's the best way to detect bible verse mentions in a text? I have a set of 10 verses from the Bible in English. I want to detect the occurrence of any of these verses in a text. What would be the best way to go about doing this?
Note that verses of the Bible are worded differently from one translation to another. For example, the verse "Cast all your anxiety on him because he cares for you" exists in other translations as:
KJV
Casting all your care upon him; for he careth for you.
ESV
casting all your anxieties on him, because he cares for you.
NLT
Give all your worries and cares to God, for he cares about you.
Also, people may make typos or make mistakes when writing a verse.
Would you say Semantic Search is the best approach for this? If so, do I use a pertained word embeddings model, retrain it on my set of 10 verses, and then search my text for the occurrence of these verses?
How would the search mechanism work? Do I get the word embedding sentence by sentence and compare each sentence's embedding one by one with my Bible corpus of 10 verses' embeddings?

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper
Each sentence in the text could be compared to the 10 Bible verses. The bible verses could be ranked based on similarity.


How to tune learning rate with HParams Dashboard on Tensorflow? In Tensorflow documentation, it is shown how to tune several hyperparameters but not the learning rate.I have searched how to tune learning rate using HParams dashboard but could not find much. The only example is another question on github but it does not work.Can you please give me some suggestions on this?Should I use a callback function?Or provide different learning rates in hp_optimizer as in the question in github? Or something else?

Parts of my code is below:

HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
HP_L_RATE= hp.HParam('learning_rate', hp.Discrete([0.0005, 0.001]))

def train_model(hparams):
    model= tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(None,43), dtype=tf.float64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], activation='relu')),
    tf.keras.layers.Dense(1)])

    model.compile(optimizer=hparams[HP_OPTIMIZER],loss='mae',metrics=['accuracy'])

    callback = tf.keras.callbacks.LearningRateScheduler(hparams[HP_L_RATE])

    model.fit(train_dataset, epochs=100,callbacks=[callback])  #, validation_data=test_dataset
    _,loss=model.evaluate(test_dataset)
    return loss

Problem is I cannot figure out how and where to insert hparams[HP_L_RATE] in the train_model().As you can see, I have tried to use a callback function to implement hparams[HP_L_RATE], but it does not work.

Thank you,

I guess the error in the example is that, instead of numbers objects were passed to HParams. You should handle the learning rate just as any other float parameter and just pass it to the optimizer. Actually hard to tell what you want exactly without a code example.


Mahout Spark shell not working I installed Hadoop, Mahout and Spark. I am able to see the Hadoop and Spark MasterWebUI. Moreover, I can also run the following command,

[hadoop@muildevcel01 mahout]$ bin/mahout

However, we I try running the spark-shell I run in the problem stated below,

[hadoop@muildevcel01 mahout]$ bin/mahout spark-shell

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/repl/SparkILoop
        at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.repl.SparkILoop
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 1 more

Question

Any suggestions how I could resolve my problem?

This error is common when the SPARK_HOME environment variable is not set.

In the shell type export SPARK_HOME=/path/to/your/spark


How to choose variables for regression I have a dataset of long/short equity hedge funds returns and their associated benchmarks (market indices).

I need to form multiple regression on the fund returns using the benchmarks returns as independent variables (i am allowed to form linear combination or manipulation of the indices or even non-linear combinations).

Of course, I do not know which independent variables to choose. Are techniques such as subset selection, Lasso, and Ridge supposed to be used in situation like this?

There are a few steps you can take to choose features for linear regression:

1 - Exclude variables that are highly correlated with each other. If variables are highly correlated you are essentially inputting the same information multiple times which can cause over-fitting and does not satisfy the properties no multi-collinearity for linear regression. You can create a Pearson correlation matrix and decided which variables are too highly correlated using some chosen threshold i.e only keep variables with a correlation coefficient of < 0.3

2 - If you have many variables you could perform principal component analysis (PCA) to reduce the dimensions of the data and use those as your linear regression features. The idea of PCA is reduce dimensions while holding all of the information. Each component from PCA are uncorrelated, satisfying the no multi-collinearity property.

3 - There is also a method known as stepwise linear regression. You allow all variables to enter the model and it will iteratively remove and add variables until the model with the highest R-squared (or whatever your chosen model metric is) is produced. You do have to be cautious using the stepwise  method as it can lead to overfitting, but it can give an indication on what features to use. Here's some info on stepwise: https://en.wikipedia.org/wiki/Stepwise_regression

4 - If you are using R, there is a brilliant package called "caret" that can help with feature selection. Here is a fantastic link to use as a guide: https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/

Hope this helps out as a starting point


PCA first dimension do not not capture enough variance I am doing a PCA as a data exploration step and I realize that the two first principal components capture only 25% of the variance, the ten first principle component capture about 60% of the information, does it worth to interpret those axis knowing that they do not capture enough information

You don't state how many original features there are? where there 10, 50, 50 million? The above 10 principle components capture 62.3% of the variance and so we can be sure that there are more that ten original features. However we will assume then that there aren't many more (lets say circa twenty), lets consider what is going on.

With PCA you are hoping to separate the useful information from the noise. By taking only the the first two components in you example you are discarding all other information as noise.

It depends upon the application but I would say that 25% is never enough (I think serious questions would be raised as why you had). So the question becomes then what is going on? Well lets consider the two extreme cases: A set of uncorrelated features and a set of fully correlated features. Consider first three uncorrelated features:

1 0 0
0 1 0
0 0 1

The total variance is 3. The eigenvalues are

1.000
1.000
1.000

Each component therefore represents 1/3 of the variance and they are equal. Now let use consider the other extreme, all features are highly correlated, i.e. a correlation matrix of

1.000 1.000 1.000
1.000 1.000 1.000
1.000 1.000 1.000

The eigenvalues are

3.000
0.000
0.000

Therefore the first principle component contains all the information and these three features could be characterised by a single latent variable.

So why is this relevant? Well in your data set your eigenvalues are all of similar magnitude. This suggests that the features in your original data are not very correlated (like the first example above, look at the off diagonal elements your original correlation matrix they are likely close to zero) and therefore it is expect that many principle components would be required since there is little shared information in the original feature space.

This of course assume that we only have a small number of original features. If you have 50 million features then capturing 62.3% in only ten latent variables is pretty good,


does entity recognition comes under classification problem? I want to extract named entities from a text but I don't know whether that comes under classification or not.if it comes under classification then how to prepare classes for recognizing entities in a text.(does the collection of few org names and person names and using them as my training dataset makes the solution?)

Simply put, Named Entity Recognition (NER) is a multi-class structured prediction (classification) problem, so you have a sequence of words and you want to label each one most of the time with these labels ( start-of-a-person-name, continue-of-a-person-name, start-of-an-org-name, continue-of-an-org-name, start-of-a-location-name, continue-of-a-location-name, other). Note that these are not the only classes, classes with different granularity can also be used.

If you want to solve this problem with SVM you can use StructSVM or other  variations of SVM for structure prediction. Though the common baseline for this task uses Maximum Entropy (Maxent)(log-linear) models.


When to normalize or regularize features in Data Science How can I know when I need to use normalization and when regularization of features.
I know that when I have big difference between min and max values in some feature / column that i need to scale data, also if I use SVM or KNN or Clustering i need to scale the data.

But how can I know when to normalize and when to regularize the features?
How can I know that if I have for example 200 features and 10 000 rows?
Is there any way that I can plot my data and see what can I use?

Step one would be understanding how the algorithms you are using work. Certain algorithms work better (distance based usually) when scaled, others don’t (like random forest). Knowing how an algorithm works will go along way to understanding when (and why) you should scale your data.


Keras LSTM Input Shape - Batch Size and Time Step So I have 82 different sets of data, each with varying length where each point has one feature and a label (0 or 1).

I'm trying to use Keras LSTM to be able to predict the class of a point depending on the previous values before it.

Currently I've padded each set to be of the same length and plan on using a masking layer.

What I'm confused with after reading many examples is how I should reshape my data and supply the correct input shape to the model.

If each set is of length 55, and I want to consider the previous 5 (just throwing a number out there) points to predict the class at each step, how would I do that?

Should I manually use a moving window to break each set into arrays of length 5, then have input_shape=(5, 11, 1)
since 55/5 = 11??

Is this the best approach? How would this work for the remaining 80+ sets?

Just to clarify:
say one of the 82 sets looks like this [1.1, 4.8, ... 2.1] (length of 55) I want the model to predict the class of each point [0, 1, ... 1] (x55) except for the padded points (-1).

I don't see much point in doing sliding windows with LSTM. The whole idea of this layer is to capture long-term dependencies and you're just throwing it away with the 5 step window limit.

The easiest for this case would be using a Conv1D with kernel_size=5 instead. Then the windows would go automatically with the kernel size. (But only one layer, if you add more, you will capture more steps). Other possibilities are 2 layers with kernel size = 3, or maybe 3 layers with sizes 3, 2, 2.

If you do want to use windows with LSTM, you will have to organize the data manually. This means you will loop your data and get segments of length 5 and treat each segment as an individual sequence.

In this case your input shape will be (5,1) and you will have far more than 82 samples. On the other hand, if all your sets are longer than length 5, you will need no padding at all.

Example loop:

originalData = load_a_list_of_samples()
windowData = []
for sample in originalData:
    L = len(sample) #number of time steps
    for segment in range(L - 5 + 1):
        windowData.append(sample[segment:segment+5])

windowData = np.array(windowData)

My suggestion though, unless you have this requirement for some special reason, is don't use windows at all. Let the LSTM do their job and capture long term dependencies.

Pad the data as intended, use the Masking layer and place the LSTM with return_sequences=True.

The results will be like (showing only the length dimension)

inputs = [step1 , step2 , step3 , step4 , step5 , step6 , step7 ]
outpus = [class1, class2, class3, class4, class5, class6, class7]

Make sure you mask your loss function as well, as I'm not sure Keras will do the masking job correctly there.


Data preprocessing for time series prediction I have a dataset that has the following structure
[
 [
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 3
                               :
 ],
 [
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 1
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 2
  [ product 2 ,shelf number, position on the tray, time of stay on the shelf, was sold?], # Hour 3
                              :
 ],
                              :
]

My goal is to predict for a newer product say product_n predict if it will be sold (3 hours earlier).
My question is how do I process it for a Recurrent Neural network  since the vector of prediction was sold? is available for each hour.
To say that in detail, since
[
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 3
                               :
 ],

is one observation  for the RNN how do I assign was sold? to it? Since len(X) should be equal to len(y)
was sold? is available for each observations, do I take max for 3 hours and asssign it to the obervation?
Like
X = [
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 1
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 2
  [ product 1 ,shelf number, position on the tray, time of stay on the shelf], # Hour 3
                               :
 ],
and
y = [max(was sold?)]

So the question is about how to represent this dataset so you can use this your classification task.
Firstly, we need to group the features for each product, if you are classifying on a product-by-product basis.
Secondly, a couple of things:

For categorical features (e.g. specific products), we normally represent these as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).
For numerical features (e.g. time of stay on the shelf [minutes/hours, etc.]) we can represent these as the value themselves.

Then, for the input into your model, then you simply concatenate these features together to form a "long" n-dimensional vector.


How to implement a Fourier Convolution layer in keras? I'm currently investigating the paper FCNN: Fourier Convolutional Neural
Networks. The main contribution of the paper is that CNN training is entirely shifted to the Fourier domain without loss of effectiveness. The proposed architecture looks as follows:

The authors state that the implementation was done in keras, however, it is not publicly available. I know I can define a Fourier transformation in the following way:

model.add(layers.Lambda(lambda v: tf.real(tf.spectral.rfft(v))))

But this is not a Fourier Convolution, right? How should I go from here?

An FFT-based convolution can be broken up into 3 parts: an FFT of the input images and the filters, a bunch of element-wise products followed by a sum across input channels, and then an IFFT of the outputs (Source).

Or as it is written in the paper:

So, for a Fourier Convolution Layer you need to:

Take the input layer and transform it to the Fourier domain:

input_fft = tf.spectral.rfft2d(input)

Take each kernel and transform it to the Fourier domain:

weights_fft = tf.spectral.rfft2d(layer.get_weights())

Note: The Fourier domain "images" for the input and the kernels need to be of the same size.
Perform element-wise multiplication between the input's Fourier transform and Fourier transform of each of the kernels:

conv_fft = keras.layers.Multiply(input_fft, weights_fft)

Perform Inverse Fourier transformation to go back to the spatial domain.

layer_output = tf.spectral.irfft2d(conv_fft)

Note: I used pseudo-code, it will probably needs some tuning for it to actually to work.


When should the Keras functional API be used? I have always been using the sequential model for deep learning and Convolution Neural Networks, image classification and prediction. But I am not getting good accuracy everytime. Should I use the functional API? Will it do any good?

Sequential API is simpler to write and to read. Functional API on the other hand gives you more freedom in the implementation of the ANN architecture. More specifically, Functional API is what you need when the layers of your Networks do not form a simple concatenation or sequence (i.e. the first sends a signal to the second, that sends a signal to the third, and so on...). Sometimes more complex things are needed.

I'll make you a personal example: in my current DS project I am building a seq2seq Neural Network that has two Encoders: one is LSTM, the other is Convolutional. These two blocks process the same input signal, then their outputs are concatenated and fed to a Decoder that is LSTM + Dense. In this case, the whole model cannot be defined as a simple, linear concatenation of layers. You need to link different layers in a less "linear" way. This could be done exclusively with a functional API.

In computer vision, Residual Networks (ResNet) are the main example of the need for a non-sequential API, because the way the signal traverses the network is not simple and sequential (no pun intended) as in classical CNNs.

That said, coming to your question:

  I am not getting good accuracy everytime. Should I use the functional API? Will it do any good?

You can still reach SOTA results with the Sequential API. There are so many hyperparameters to tweak in a classical CNN to work on that for months (if not more). Could you please exaplain your problem, your data, what you tried so far, and what outcomes you got?


How to interpret PR and ROC Curve for an unbalanced test set I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000.
All positive examples are correctly predicted, instead negatives elements correctly predicted are 99% of total negatives.

Plot of PR and ROC curves are those:

What can be inferred from these curves?
Those are my firsts works with classifiers and i'm confused.
I think that precision is always low, because the negatives that are wrong predicted as positive have an high score assigned by the classifier (close to 1).
ROC instead i think that is high because all positive examples are correctly predicted. These are my suppositions, correct me if I am wrong.

For binary-class classification problems and data being highly unbalanced,

Go for AUC and f1 score as metrics.
Plot the confusion matrix.
Split the data into train:valid:test::60:20:20 or 80:10:10, and do Cross-validation and hyperparameter tuning on train and valid sets. Then go for test set.
You could also try bootstrap resampling.


Item-Item similarity based on text We're build an item-item recommender based on the text descriptions of the items. Our initial approach was to calculate the TF-IDF vectors for each item. We used a hashing tf with 5000 possible hashes for the words. Then approximate all-pairs using a sampling technique (DIMSUM).

We have an mxn matrix where m is the number of words, n is the number of items. The naive approach of calculating all column cosine similarities won't work here since we have n=10^7 m=10^4.

Our first attempt was using DIMSUM http://stanford.edu/~rezab/papers/dimsum.pdf which is an all-pairs sampling technique. The problem with DIMSUM is it works for data where m >> n. Our matrix is short and wide n>m.

My Question:
What is a good approach for estimating all-pairs similarity of items based on words. Where number of items is 10^7, number of words is 10^4. We only want items pairs above a certain threshold of similarity.

We don't have to do it this way, our task is to recommend a small set of items given one item, we have ways of doing this using collaborative filtering, but we want to also handle new items that we have no user data for so we're trying to find a way to use tf-idf vectors for that case.

There are methods that approximate Jaccard similarity using hash functions, one of them MinHash. They scale well with the number of dimensions and  evaluate similarity of two items. If you want to do nearest neighbor queries (finding similar items given one item) you should look at locality sensitive hashing (LSH), which essentially maps similar items to the same hash value. There is a nice implementation in scikit-learn called LSHForest. It approximates cosine distance and scales well to large number of items and dimensions.


Multiple Features with the same categorical value In working with a telecommunications data set with multiple categorical variables which all depend on Internet Service, a separate categorical variable, I ran into the problem where 'No Internet Service' is dummy encoded several times in my training data. See the below pie charts for an illustration of the problem.
I was wondering what can be done to avoid this repetition of 'No Internet Service' as a feature value across several features, as it increases the dimensionality of my training data significantly and makes all of my features very highly interdependent.
Feature selection after encoding helps, but often leaves just the dummy variable 'No Internet Service' when removing the categories which don't contribute to the label.

The increase in dimensionality due to 'No Internet Service' could be handled maybe with a 2 layer model. If 'No Internet Service' is true, you run the first level model with a handful of other variables (excluding those which depend on internet service availability). The second layer model runs only when internet service available, with all the variables.


Are there any tools/ python packages for Fuzzy Grouping? I'm trying to get to a tool for Fuzzy Grouping as I do not have a reference column for matching the string. Is there any package on Python or R?
I looked at a package called textpack but the results aren't good.

found here:
https://pypi.org/project/textpack/

I'd really appreciate if someone could suggest a tool or a package so I can go ahead and research.

Probably there are more direct solutions, but you can try using some sort of fuzzy matching to get the most similar words to each word. That returns you an unweighted graph where the nodes are words and the edge indicates whether the world is similar. then, you can try clustering the resulting graph, for example with the Louvain's algorithm. Than would return non overlapping groups of related words.


hypeparameters tuning neural network according to loss vs according to scoring function During hyperparameters tuning we select a metric to measure performance of the model. Example of metrics : f1 score, precision, recall, AUC ...

In general, for the training of neural networks, back-propagation tries to optimize the weights of the model according to the value of the loss function.

Here comes the question:
Why don't people use the loss function as a main performance metric for neural networks optimization?

If you look for example at the L2 loss function:

$$
\sum (y - \hat{y})²
$$

It takes into account how much the predicated value differs from the actual value. Note that because of the squaring of errors, L2 is sensitive to outliers. So, as you can see it takes into account how far our predictions are.

And that is exactly the problem: Most of the time we are not interested in how far our predictions are from the ground truth.

Imagine a real world application where the model predicts an image label. The application won't give you n examples together with their probabilities. It will just output the image with the highest probability.

That means, in the case of a classification problem where we use a softmax together with cross-entropy-loss it is not important if the predicted probability for a class is 0.49 or 0.01. We only care about if our model got the class correct or wrong.

Also, accuracy is much more tangible and better to interpret. That's why accuracy is used much more often.


AlphaGo Zero loss function As far as I understood from the AlphaGo Zero system:

During the self-play part, the MCTS algorithm stores a tuple ($s$, $\pi$, $z$) where $s$ is the state, $\pi$ is the distribution probability over the actions in the state and $z$ is an integer representing the winner of the game that state is in.
The network will receive $s$ as input (a stack of matrices describing the state $s$) and will output two values: $p$ and $v$. $p$ is a distribution probability over the actions and $v$ is a value in $[-1,1]$ representing which player is likely to win the game.
For the training it will use the following loss function:

$$l = (z - v)^2 - \pi^T log(p) + c ||\theta||^2$$

Lastly, it evaluates the new network and it starts the self-play section again.

My questions

If the network receives only the state $s$ (represented as matrices) as input, how can it then calculate the loss function if the values $\pi$ and $z$ are needed?
If these values are indeed passed as input for the network, are they passed through the convolutional (and other) layers of the network? Because if this is true, there is no mention in the article (unless I missed it) of this information.

The best way to understand that part is by looking at figure 1 in the AlphaGo Zero paper.

The neural network (NN) minimizes the differences between its own policy $p_t$ and the MCTS policy $\pi_t$. The value of $\pi_t$ is produced by the MCTS self-play which in return uses the NN from the previous iteration.

The same goes for $v_t$ and $z$. In each iteration the weights of the NN are adjusted to minimize the distance between $v_t$ (output of the NN) and $z$ (output of the MCTS) as defined by the loss function. $z$ does not have a time index here as the full self-play produces just a single value for $z$ each time it is conducted.

TLDR for your first question: Both, $\pi$ and $v$, are being produced by the MCTS as input to the NN.

(The indexing in the paper is a bit confusing in my opinion so it is probably easiest to just look at it as stated above)

Now, with "input" I do not mean input on the input layer of the NN. As described in the appendix under "Neural network architecture" the input is a "19 x 19 x 17 image stack". which contains the following information:

The positions of player 1 for the latest 8 rounds (8 feature planes)
The positions of player 2 for the latest 8 rounds (8 feature planes)
A color feature indicating whose turn it is (1 feature planes)

And those 17 feature planes ($8+8+1$) combined with the $19\cdot19$ sized board is the $19\cdot19\cdot17$ input the NN receives thru its input layer. $\pi$ and $z$ are passed to the NN via the loss function only (i.e. they are the target values in this supervised learning problem!).

TLDR for your second question: $\pi$ and $z$ are not fed to the NN thru the input layer but just via the loss function as target values.


Use LSTM to predict the proportion of steps with nonzero feature values I am trying to do a simple regression for sequences. Each input $X_i$ is a $n=2000$ by 1 matrix, formatted as $n_i$ 0-s followed by $(n-n_i)$ 1-s. The output $y_i$ should be $n_i/n$, i.e. the proportion of steps with nonzero feature values.

The code for generating the data

ns = [1087, 280, 1013, 300, 311, 260, 960, 281, 417, 311, 102, 251, 748, 428, 802, 512, 61, 80, 1700, 693, 147, 577, 280, 196, 377, 281, 88, 359, 250, 330, 679, 201, 284, 1129, 151, 285, 722, 1014, 153, 202, 256, 99, 661, 415, 134, 269, 196, 137, 436, 617]
Xs = np.array([[[0] for i in range(2000-x)]+[[1] for i in range(x)] for x in ns])
Ys = np.mean(Xs,axis=(1,2))
trainX, testX, trainY, testY = Xs[:45], Xs[45:], Ys[:45], Ys[45:]

I tried to use LSTM and an output layer to do the prediction, i.e.

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add((LSTM(10,activation='tanh',unit_forget_bias=True,input_shape=(trainX.shape[1],trainX.shape[2]))))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'],)
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

Yet I found the model fails to predict what I want, instead it predicts every sequence with almost the same $y$ value. Just wondering if there's any solution to solve this?

First, you have too little data. 50 samples. Think of it. Of all possible 2001 sequences of that kind, you are only feeding 50 (less than 2.5%). Your question is actually a very good case to show the importance of big data for training a neural network.

And second, it is the problem stated in this arxiv article made by Uber empolyees - even a deep neural network with nonlinear units often cannot generalize to solution of a simple linear problem.

Your problem still can be approximately solved with

Reducing batch size

It is well known that increasing batch size leads to faster but poorer convergence. Particularly in your case, you should use line:

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

instead of

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

even knowing that it will be slooooow...

Modifying LSTM layer: removing unit_forget_bias=True and adding orthogonal initializer with small gain - explained in this article:

from keras.initializers import orthogonal
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    kernel_initializer=orthogonal(gain=.01),
))

Increasing coverage of the space of all possible samples

In the code below SEQ_LEN=2000 and N_SAMPLES=50 is your case. Here we generate random sequences like ones that your code above (although not the same sequences as yours). Now, try experimenting with values SEQ_LEN=200 and N_SAMPLES=80 and then SEQ_LEN=200 and N_SAMPLES=130. See if with N_SAMPLES<SEQ_LEN/2 your net converges at all, while in second case it will!

import numpy as np

SEQ_LEN = 2000
N_SAMPLES = 50
n_test_samples = int(.1*N_SAMPLES)
if N_SAMPLES>SEQ_LEN+1: raise ValueError("Can't create more samples of this type than SEQ_LEN+1 is.")
ns = np.random.randint(0,SEQ_LEN,size=(N_SAMPLES,))
all_Xs = np.triu ( np.ones(shape=(SEQ_LEN+1,SEQ_LEN)) )
np.random.shuffle(all_Xs)

Xs = all_Xs[:N_SAMPLES, ...]
Ys = Xs.mean(axis=1)
Xs = Xs.reshape(N_SAMPLES, SEQ_LEN, -1)
trainX, testX, trainY, testY = Xs[:-n_test_samples], Xs[-n_test_samples:], Ys[:-n_test_samples], Ys[-n_test_samples:]

import tensorflow # To prevent
import tensorflow as tf # to declarations to prevent cryptic errors on my Windows laptop

from keras import Sequential
from keras.layers import Masking, LSTM, Dense
from keras.optimizers import adam
from keras.initializers import orthogonal
from keras.layers import Flatten

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    input_shape=(trainX.shape[1],trainX.shape[2]),
    kernel_initializer=orthogonal(gain=.01),
))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'])
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

testY_PRED = model.predict(testX)
print (np.concatenate(( testY_PRED, testY.reshape(-1,1)), axis=1) )

I haven also tested case with 300 samples and sequence length of 500 (converges). However, it is rather long to test if network converges with SEQ_LEN=2000 and let's say N_SAMPES=1200 (don't hope it will with N_SAMPLES=50) - it is up to you :)


Running sklearn trained classifier in a windows machine without Python I have a classifier (and vectorizer) that I can export as a pickled model. I am using Python.

This is on my local machine. Now, in order to perform the classifications in "production" I need to be able to run this in another machine that does not have Python installed. Any idea how to do this? Is it possible to create a Python executable with the pickled model in it?

The simplest thing to do is to get Python installed properly on your production system. If you aren't able to do this for whatever reason, you can still run Python without installing it, but it's a bit more work to get everything to work properly, particularly if you don't have Administrator privileges.

If you truly can't install Python, your best bet would probably be to use something like py2exe to build a self-contained version of your application that includes a Python interpreter and required libraries.


Work experience in Data Science: Couple hours with A-level student Our company has accepted an A-level student (18-year old, just before uni) for some work experience. The student will be with me for a couple of hours. I do data analysis and I'm quite busy at the moment.
Not sure what I can show him in such a short time, and I would rather do it hands-on.

Any suggestion? Some online resources?

Since you're busy at the moment, perhaps you could take a section of a dataset that you've worked with previously (and have the relevant processing stored somewhere) and let them loose on it. Then in a week or two go back and see where they've got with it.. it has the two prong approach of throwing them in at the deep end and seeing what they can do as well as providing a ready-made learning opportunity for you.


Query data dimension import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.read_csv('Downloads/breast-cancer-wisconsin.data.txt',skiprows=1)
df.replace('?', -99999, inplace=True)
df.drop('id', 1, inplace=True )

X= np.array(df.drop(['class'],1))
y= np.array(df['class'])

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)

#clf = neighbors.KNeighborsClassifier()
clf = LinearRegression(normalize=True)
clf.fit(X_train, y_train)

accuracy= clf.score(X_test, y_test)
print(accuracy)

example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])
example_measures = example_measures.reshape(1,-1)

prediction = clf.predict(example_measures)     ##(example_measures)

print(prediction)

Problem arises when  I run the above command line at Ubuntu or Anaconda:

  ValueError: query data dimension must match training data dimension

How to solve that problem ? I am sure that by method of isolating individual commandline-- and find it appears Error at :

prediction = clf.predict(example_measures)

I try to use :

prediction = clf.predict(X_test).

It is ok.
I really want to predict the example I create. How can I change the code?

How many columns do X_train and X_test have?

I'd imagine (though can't confirm, as I don't have access to your data) that they have fewer (or more) than 18 columns.

This is because your code

example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])
example_measures = example_measures.reshape(1,-1)

produces an array of shape (1, 18).

EDIT: I've tried matching your dataset, and got the following:

import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_breast_cancer

X= load_breast_cancer().data
y= load_breast_cancer().target

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)

clf = LinearRegression(normalize=True)
clf.fit(X_train, y_train)

accuracy= clf.score(X_test, y_test)
print(accuracy)

If I call X.shape, I get (569, 30). So, if you want to make your own array to pass to clf, it needs to have 30 columns (one for each feature).


What are graph embedding? I recently came across graph embedding such as DeepWalk and LINE. However, I still do not have a clear idea as what is meant by graph embeddings and when to use it (applications)? Any suggestions are welcome!

What are graph Embeddings ?
"Graph Embeddings" is a hot area today in machine learning. It basically means finding "latent vector representation" of graphs which captures the topology (in very basic sense) of the graph. We can make this "vector representation" rich by also considering the vertex-vertex relationships, edge-information etc. There are roughly two levels of embeddings in the graph (of-course we can anytime define more levels by logically dividing the whole graph into subgraphs of various sizes):

Vertex Embeddings - Here you find latent vector representation of every vertex in the given graph. You can then compare the different vertices by plotting these vectors in the space and interestingly "similar" vertices are plotted closer to each other than the ones which are dissimilar or less related. This is the same work that is done in "DeepWalk" by Perozzi.
Graph Embeddings - Here you find the latent vector representation of the whole graph itself. For example, you have a group of chemical compounds for which you want to check which compounds are similar to each other, how many type of compounds are there in the group (clusters) etc. You can use these vectors and plot them in space and find all the above information. This is the work that is done in "Deep Graph Kernels" by Yanardag.

Applications -
By looking carefully, embeddings are "latent" representations  which means if a graph has a |V| * |V| adjacency matrix where |V| = 1M, its hard to use or process a 1M * 1M numbers in an algorithm. So, latent embedding of dimension 'd', where d << |V|, would make the adjacency matrix |V| * d and relatively easier to use. Another application could be - Consider a simple scenario where we want to recommend products to the people who have similar interests in a social network. By getting vertex embeddings (here it means vector representation of each person), we can find the similar ones by plotting these vectors and this makes recommendation easy. These are some applications and there are others. You can refer to a nice survey paper - Graph Embedding Techniques, a Survey.

Where from it all came ? There has been a lot of works in this area and almost all comes from the groundbreaking research in natural language processing field - "Word2Vec" by Mikolov. If you want to get started with the research on graph embeddings, I would recommend to first understand how Word2Vec works. You can find nice explanations - Word2Vec parameter learning explained and Stanford Lecture. Then you can jump to the papers that you listed. Those works can be categorized as:

Works based on "Vertex Embeddings": - DeepWalk, Node2Vec, LINE.
Works based on "Graph Embeddings": - Deep Graph Kernels, Subgraph2Vec.


Continuous variable not supported in confusion matrix I am using linear regression algorithm for a data set. And trying to compute confusion matrix between y_pred and y_test. I am getting "ValueError : continuous is not supported" error.

I have included the code below. Help to solve this problem.

x = data_set.iloc[:, :-1].values
y = data_set.iloc[:, 7].values

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 0)

from sklearn.linear_model import LinearRegression
from sklearn.metrics import confusion_matrix

regression = LinearRegression()
regression.fit(x_train, y_train)

y_pred = regression.predict(x_test)

cm = confusion_matrix(y_test, y_pred)

The confusion matrix is used to tell you how many predictions were classified correctly or incorrectly. You are looking at a regression model, which gives you a continous output (not classification).

So when you run confusion_matrix(y_test, y_pred) it will throw the ValueError because it expected class predictions, not floating point numbers.

Are you trying to predict classes, or really just a number output? If not, then you should not be using the confusion matrix.

If you want to predict e.g. 1 or 0 for your y values, then you would have to convert your linear regression predictions to either of these classes. You could say any value in y_pred above 0.7 is a 1 and anything below is 0.

cutoff = 0.7                              # decide on a cutoff limit
y_pred_classes = np.zeros_like(y_pred)    # initialise a matrix full with zeros
y_pred_classes[y_pred > cutoff] = 1       # add a 1 if the cutoff was breached

you have to do the same for the actual values too:

y_test_classes = np.zeros_like(y_pred)
y_test_classes[y_test > cutoff] = 1

Now run the confusion matrix as before:

confusion_matrix(y_test_classes, y_pred_classes)

which gives output:

array([[2, 3],
       [0, 0]])


LSTMs - Carousel of Error Is there any other materials that derives the LSTM back propagation and carousel of error except the original paper? (I could not understand it, sorry).

I tried deriving and got stuck, and asked the following question: LSTMs - Data Science Stack Exchange question, however, it doesn't seems that there are much people interested in hand derivation of LSTMs.

Thanks

One tool that may be helpful is Aiden Gomez's blog post. Its strength lies primarily in the fact that he runs through a toy/numerical example, which when paired with the original paper/thesis , serves as a great foundational tool.

I did take a look at the site you mentioned in your other question, it's actually an excellent resource. I'll hop on over and try and clarify what I can for you when I get a chance. It looks like you've misunderstood/overlooked notation which can happen since there are so many components involved.

It may also be worth taking a look at some code. Siraj Raval has a great video on LSTMs and includes the code in the link I've included. No libraries. I wouldn't dive too deep, but it's a great way to see the inner workings of the network.

As far as the CEC goes, there is a reddit post. If you're looking for a more rigorous handling of this topic you can either reference the original paper or often cited paper: On the Difficulty of Training Recurrent Neural Networks.


Is the distance in Nearest Neighbors a good measure of similarity? Let's train a Nearest Neighbor model with just one sample in it:

In [48]: nn = NearestNeighbors().fit([[0, 1, 0, 0]])

So this one sample has just one significant feature. Querying the model with the same sample returns 0 distance in the first array as expected:

In [50]: nn.kneighbors([[0, 1, 0, 0]], 1)
Out[50]: (array([[0.]]), array([[0]]))

But queries with samples of [0,2,0,0] and [0,1,1,0] both return the same distance value of 1:

In [51]: nn.kneighbors([[0, 2, 0, 0]], 1)
Out[51]: (array([[1.]]), array([[0]]))

In [52]: nn.kneighbors([[0, 1, 1, 0]], 1)
Out[52]: (array([[1.]]), array([[0]]))

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0]. Using Jaccard metric slightly improves on this issue:

In [56]: nn = NearestNeighbors(metric=scipy.spatial.distance.jaccard).fit([[0, 1, 0, 0]])

In [57]: nn.kneighbors([[0, 1, 0, 0]], 1)
Out[57]: (array([[0.]]), array([[0]]))

In [58]: nn.kneighbors([[0, 2, 0, 0]], 1)
Out[58]: (array([[1.]]), array([[0]]))

In [59]: nn.kneighbors([[0, 1, 1, 0]], 1)
Out[59]: (array([[0.5]]), array([[0]]))

But for my dataset Jaccard metric makes the kNN queries taking very long time, perhaps it is more suited for binary features. I have a set of readings from 52 sensors per row, nicely normalized with zero mean. I stumbled upon this issue when I fitted this set into sklearn.neighbors.NearestNeighbors and queried giving the first row of the training set as a sample and K=2, so it returned 0th index at 0 distance as expected and some other index with 0.02 distance. When I checked that other one I could not see ANY similarities, in fact most of the features where very dissimilar by value and/or by sign. I could get the same distance with a made-up sample of the first row from the training set where any one feature was increased by 0.02.

I am wondering now how to overcome this problem and if there is an easy way (ie. by tweaking parameters of NearestNeighbors) or a hacky way (ie. custom metrics, feature weights etc.) or should I rather use a different model? KMeans for example can converge clusters from my dataset pretty fast, but it uses NN internally and I do not fully like it because of the randomness in the init.

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0].

No this is expected, since the two points are exactly at the same distance in the Euclidean space. To see it take a simplified 2D version of your points:

A (1,0)
B (2,0)
C (1,1)

Both B and C are exactly at distance 1 from A.

  But for my dataset Jaccard metric makes the kNN queries taking very long time, perhaps it is more suited for binary features.

Jaccard actually assumes binary features, it will consider all non-zero values the same way. Its result is based on how many non-zero dimensions the two points have in common. I'm assuming that the implementation follows the original definition, there might be variants. Normally it's a very simple measure which doesn't require any heavy computation so it's surprising that it takes a long time.

  When I checked that other one I could not see ANY similarities, in fact most of the features where very dissimilar by value and/or by sign. A change of sign has no particular importance in the euclidean space, it's the distance which matters.

Well the only way to really check would be to calculate the euclidean distance, with 52 dimensions just looking at the values is not going to give a good indication.

  I could get the same distance with a made-up sample of the first row from the training set where any one feature was increased by 0.02.

I'm not sure I understand this part but again this sounds perfectly correct: changing any feature by 0.02 will move the data point in the space by a distance of 0.02 so... it's going to be at distance of 0.02 from the original point.

  I am wondering now how to overcome this problem and if there is an easy way (ie. by tweaking parameters of NearestNeighbors) or a hacky way (ie. custom metrics, feature weights etc.)

If there is any data-specific way to measure the distance between two points, you should definitely define a custom metric. It's not a hack, as you can see NN relies entirely on the metric. The default euclidean distance is a generic measure, it might not be fit for your purpose.


Clustering with cosine similarity I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number of clusters I expect.

I read the sklearn documentation of DBSCAN and Affinity Propagation, where both of them requires a distance matrix (not cosine similarity matrix).

Really, I'm just looking for any algorithm that doesn't require a) a distance metric and b) a pre-specified number of clusters.

Does anyone know of an algorithm that would do that?

First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems.

You said you have cosine similarity between your records, so this is actually a distance matrix. You can use this matrix as an input into some clustering algorithm.

Now, I'd suggest to start with hierarchical clustering - it does not require defined number of clusters and you can either input data and select a distance, or input a distance matrix (where you calculated the distance in some way).

Note that the hierarchical clustering is expensive to calculate, so if you have a lot of data, you can start with just sample.


Identifying users based on smartphone data (Google) I've heard that Google has now a technique to identify users based on the smartphone touch input (how the user is using the phone). I have found nothing about that online. Is there a paper or some article available explaining how this is done (e.g. what features are used)?

Even I was unable to find a paper on sucha topic by Google. But, I can discuss some features which Google might be using to uniquely identify its users based on smartphone data.

Fingerprint scanning: One of the preliminary methods of scanning users as the fingerprint is unique to every individual. Also, most devices are now equipped with a fingerprint scanner.
Home location: The Google account also keeps a track of the user's home location which can't be directly used to identify a user, but could be an important feature.
Data Personalization: Suppose a user has habit of viewing videos, blogs and websites related with the top tag "artificial intellgence". This personalized information could be used to track down a user.

  Data persoanlization could be applied to apps, videos and websites with which the user interacts. This data will be super complex and probably unique to each individual.

Device Usage: Every individual likes to personalize and use his/her device in their own way. Android 9 had this feature of tracking apps which are most used by the user and also at which times. This is another great factor for identification.
Preliminary device details: IP address, the device's model, Android version, account password are also stored with the Google account.

The above features could be brought together to uniquely identify users as they provide information in different dimensions regarding the user.

The weaker features ( Data personalization, device usage ) can be used with the powerful features ( IP address, passwords ) to uniquely identify a particular user.

All the above information was not mentioned in any paper.

Hope this helps. :-)


day is out of range for month import from csv file in python Here I have one csv file with different values and date_time. I wrote the code and run it. Then it gave me an error  time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M:%S'
I want to plot this different three values in one graph. Can anyone help me to solve this problem?
Here is my code:

condition = ""
date_time  = []
x1 = []
x2 = []
x3 = []
def convertTime(s):
  tm = time.strptime(s, " %d/%m/%Y %H:%M")
  return datetime.datetime(tm.tm_year,tm.tm_mon, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec)
with open('data43.csv','r') as csv_file:
csv_data = csv.reader(csv_file, delimiter=',')
 row_num = 0
for row in csv_data:
 if(row_num == 0):
  condition = row[0]
elif(row_num > 1): #Data starts here
  if(row[0] != ''):
    date_time.append(convertTime(row[0]))
  if(row[1] != ''):
    x1.append(int(row[1]))
  if(row[2] != ''):
    x2.append(int(row[2]))
  if(row[3] != ''):
    x3.append(int(row[3]))
row_num = row_num + 1
fig1 = plt.figure(1)
ax = fig1.add_subplot(2,1,1)
ax.plot(date_time,x1)
ax.stem(date_time,x2,'C1--','C1o',linefmt=None, markerfmt=None, basefmt=None)
ax.stem(date_time,x3,'C2--','C2o',linefmt=None, markerfmt=None, basefmt=None)
ax.legend()
ax.xaxis_date()
ax.get_xaxis().set_major_formatter(DateFormatter('%d/%m/%Y %H:%M:%S'))
plt.xlabel('t')
plt.ylabel('k')
leg = plt.legend( loc = 'upper right')
plt.draw() # Draw the figure so you can find the positon of the legend.
bb = leg.get_bbox_to_anchor().inverse_transformed(ax.transAxes)
xOffset = 0.3
bb.x0 += xOffset
bb.x1 += xOffset
leg.set_bbox_to_anchor(bb, transform = ax.transAxes)
plt.rcParams["figure.figsize"] = [20,20]
ax.plot(style='.-')
plt.show()

error :

error time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M'

Your problem is you have a leading single space. Change the match to '%d/%m/%Y %H:%M'


How to load large csv file from web in python My code is in jupiter notebook and my data set(400-800mb) is present on google drive/dropbox. My task is to load the csv file from another server into my jupiter notebook. How can I achieve this? Should I move my csv file to some other server?

Using requests

You can read the file line by line iteratively. You don't need to store the whole dataset into memory. This will only maintain a single row in memory at a time. This is very fast and memory efficient.

import requests
from contextlib import closing
import csv

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"

with closing(requests.get(url, stream=True)) as r:
    f = (line.decode('utf-8') for line in r.iter_lines())
    reader = csv.reader(f, delimiter=',', quotechar='"')
    for row in reader:
        print(row)

Using pandas

You can also do this with pandas, however this will store an entire chunk of the dataset in memory. However, this library is very powerful and can make your processing way easier.

import pandas as pd

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"
df = pd.read_csv(url)

print(df.head())


How to work with large amount of data overcoming RAM issues in python I use numpy arrays to work with deep learning images. But as the data gets bigger, I'm facing issue with RAM even before training the model when using techniques like data augmentation.

Can someone suggest me how to work with large data for eg. 30GB of data in my system which has 16gb ram.

P.S. I'm worried about RAM during preprocessing and training, while i do batch processing with my GPU

Dask is designed to manage these types of workloads. It provides an interface like NumPy, Pandas, or Python iterators for larger-than-memory operations. An example of using Dask with TensorFlow can be found here.


What exactly is the input of decoder in autoencoder setup I am reviewing various autoencoder setups for MNIST reconstruction, Seq2Seq translation and others.
My naive understanding of data flow is as follows:

Input -> [Encoder] -> Hidden Representation -> [Decoder] -> Output.

However, in case of Seq2Seq translation task similar to Sutskever et. al. decoder input is combined from hidden state + input sequence.

I wonder how is the input of decoder dependent on the target task? Why do we need to put input sequence besides the hidden state? Any high-level explanation is appreciated.

Your are mixing two different beasts. Despite both having encoder and decoder parts, the way in which a normal feedforward image transformation network (i.e. the autoencoder) and an autoregressive model (i.e. the seq2seq) are actually used is very different:

In the type of image transformation done in a vanilla autoencoder, you input an image and get another image at the output. That's it, both at training and at inference time.
In an autoregressive model like seq2seq, you feed the network with the input and the initial tokens of the expected output sequence (i.e. the prefix), and you get the next token as output. During training, those initial tokens are normally from the gold data (aka teacher forcing), that is, you feed as prefix real tokens from the training data. In inference, those tokens are generated by the model itself: you generate the first token, then you generate the second token given the first one you previously generated, then you generate the third token given the previous 2 generated tokens, and so on.


What's the input dimension for transformer decoder during TRAINING? For example, translate English sentence A to French sentence B.
During training with ith word in B, all previous words before B will be fed to decoder, whose length will change for different i. How this is handled so that it can fit into a fixed dimension in the final linear layer during TRAINING?

For feeding word one by one in transformer network we pass the whole sentence along with a mask to the network. And the mask will do the job by unmasking one new word at a time.


Two different pytorch networks, combined loss, back propagation and optimizer step So here is my network architecture as shown in below image.
I’ve two separate networks and loss_1 and loss_2 are coming from two separate nn.Modules networks and final_loss = loss_1 + loss_2.

Still only one final_loss.backward() would calculate grad in both networks?
I do have two separate optimizers to update grads in two separate networks respectively.

If not so, then what is the correct way to do back propagation in these individual networks on combined loss value.

A single final_loss.backward() would calculate the gradients but there are some strange things in your scheme (these may seem strange due to lack of information in your question):

The first strange this is: why having two separate optimizers? It could only be justified if you purposely want different optimization algorithms for each network.

The second and most strange thing is: why would you want to have a combined loss? The only reason that comes to my mind would be that module1 and module2 share some parameters. Otherwise, I see no point in combining two totally unrelated losses because they could have different scales, leading to one of the partial losses to have little effect. Minimizing each loss separately would lead to a much better result.


Why not use more than 3 hidden layers for MNIST classification? Many works use 2-hidden-layer neural networks for classifying MNIST handwritten digits sets.

To improve accuracy, other techniques (dropout, ReLU.. etc) have been used without increasing the number of hidden layers.

Is there any reason not to use more than three hidden layers?
for example, overfitting?

Empirically, the network performance does not increase much for a fully-connected network on MNIST when you add layers, but you can probably find ways to improve it on networks with 3+ hidden layers, such as data augmentation (e.g. variations of all inputs translated +-0..2 pixels in x and y, roughly 25 times the original data size, as a start).

I don't think this idea is pursued very far in practice, because CNNs offer a much better performance increase for the effort required. You hit the point of diminishing returns earlier with a basic MLP (around 96-97% accuracy) than you can reach easily with a CNN (around 99% accuracy).

The theory basis for this difference is not obvious to me, but very likely yes this is related to over-fitting. The weight sharing and feature pooling in a CNN is very effective way of processing image data for classification tasks, and avoids over-fitting by reducing the number of parameters, whilst re-using the parameters for the task in a way that makes very good sense given the nature of the inputs.


software for workflow integrating network analysis, predictive analytics, and performance metrics I am hoping that there is some existing software for what I want to accomplish, as I'm not a big fan of reinventing the wheel.

In general, I would like a software package that can serve as a workflow that integrates network analysis, predictive analytics, and performance evaluation.

More specifically, information about a criminal network along with other relevant data would be used to predict monetary flows across the network.  Law enforcement case data combined with information about network monetary flows would direct resource use in disrupting flows across the network.  These law enforcement performance metrics along with metrics related to criminal networks would be used for strategic decision making.  It would also be ideal to have dashboards showing some of the metrics of interest.

I have identified some potential software candidates: Dataiku DSS, RapidMiner, KNIME, Orange Data Mining, Watson, and Statistica.  I am not too familiar with them and I wonder if something else may immediately come to mind for those more familiar with these applications.

Any direction is greatly appreciated.

-Ted

In Orange, you can do something like this:

This takes the network, which is already containing class you'd like to predict, then training (or testing) the learner in Test & Score and evaluating it in Confusion Matrix. Then you can see misclassifications directly in the network graph.

There are a bunch of other learners and evaluation methods available. A big plus is also interactive data exploration (see how you can input wrongly classified data into Network Explorer?). However, there's no dashboard available yet. We make do with opening several windows side by side.

That's just my 2¢ on Orange. I suggest you to at least try all of them and see which one works best for you. :)