Is our data "Big Data" (Startup) I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.

In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths.

The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.

My belief is that a safer approach would be to use NoSQL storage such as "Azure Table Storage" in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.

I haven't done a significant amount of research in this field, so would appreciate your input.

Questions:

How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?
Would it be easy to initially start off with a simpler approach such as "Azure Table Storage" then onto Hadoop at a later point?
If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?
Is Hadoop necessary to invest from the get go as we will quickly incur major costs?

I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up.

Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year.

First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. The Hadoop ecosystem has HBase for NoSQL; I suspect Azure has some kind of stream processing story.

The bigger difference in your two alternatives is consuming a cloud provider's ecosystem vs Hadoop. The upside to Azure, or AWS, or GCE, is that these services optimize for integrating with each other, with billing, machine management, etc. The downside is being locked in to the cloud provider; you can't run Azure stuff anywhere but Azure. Hadoop takes more work to integrate since it's really a confederation of sometimes loosely-related projects. You're investing in both a distribution, and a place to run that distribution. But, you get a lot less lock-in, and probably more easy access to talent, and a broader choice of tools.

The Azure road is also a "big data" solution in that it has a lot of the scalability properties you want for big data, and the complexity as well. It does not strike me as an easier route. Do you need to invest in distributed/cloud anything at this scale? given your IoT-themed use case, I believe you will need to soon, if not now, so yes. You're not talking about gigabytes, but many terabytes in just the first year.

I'd give a fresh team 6-12 months to fully productionize something based on either of these platforms. That can certainly be staged as a POC, followed by more elaborate engineering.


Improving accuracy on highly imbalanced dataset I need some suggestions to improve my model accuracy.

The training data shape is : (166573, 14)

It has all int and float columns. I have dropped claims_daysaway column as most of values are NaN and replaced Nan value with mean for tier column.

X_train = train.drop(['outcome','testindex','claims_daysaway'], axis=1)
y_train = train['outcome']

As the values were on different scale, I have used StandScaler() to standardize values.

This dataset is highly imbalanced.

  train['outcome'].value_counts()

0    159730
1      6843

I tried SMOTE for oversampling.

from imblearn.over_sampling import SMOTE
smt = SMOTE()
X_train, y_train = smt.fit_sample(X_train, y_train)
pd.value_counts(pd.Series(y_train))

1    159730
0    159730

Lastly, I fit model using XGBClassifier but when tried this model on testdata and submitted it, it gives only 60% roc_auc_score.

Please suggest how to handle imbalanced dataset better.

I'm not very sure what you mean by "60% accuracy using AUC". Accuracy and AUC are two different metrics... I'm going to answer as if you're referring to classification accuracy, since that's in your title and the first sentence of your post.

First of all, don't use accuracy to evaluate performance on imbalanced data!

Your dataset has an imbalance ratio of 6843/159730 which is around 1/23. This means that if you make a dummy classifier that just predicts the majority class you'd get an accuracy of 96%. There are better options for imbalanced data such as the f1 score or any macro-averaged metric (you can read this post more information).

Secondly, I'm not sure what you're doing it but, just in any case, you shouldn't evaluate on the oversampled dataset.

As for ideas for improving performance, I don't have many because you are doing most things right. Tree-based algorithms (e.g. XGBoost) are good for dealing with imbalanced data. You are already oversampling the data, which helps a lot. Some other ideas are:

Try different oversamplers, undersamplers or perhaps a combination of over and under-sampling techniques.
Search to optimize the hyperparameters of your XGBoost. I can't tell by the information you gave, but maybe you're overfitting.
Try different algorithms (catboost, lightgbm, etc.), or maybe ensembles of those models (stacked models, etc.).


From development environment to production I have been working on a project as part of my master degree in participation with a firm.
I developed a predictive model in the past few months that is essentially a document classification model. The biggest limitation of the research and model is the lack of data available for training. I have a small data set of 300 documents, where as the features are in excess of 15000 terms (before feature selection).

How do we identify or estimate the number of data-points (documents) necessary to gain a level of confidence that will allow us to move forward with the testing stage into a production environment ? So how do I estimate the necessary dataset size required for the performance metrics to be viable enough to ensure it is a solid model?
Are there standards or metrics in the industry for when a model is sufficiently tested and should qualify for production?

I cannot find literature about this topic.

Thank you

There isn't a well established way of estimating the number of data points that you'll need. It's much more an art than a science. As you gain more experience, you'll learn some common sense lessons (in hindsight) along the way. For instance, you should never have more parameters than data points; if you're building random forests, you should not have more trees than data points. If you're doing deep learning, you should never have more neurons than data points (these are extreme examples). As a rule of thumb, try to avoid having 10 times more features than you have data points. 300 data points is very small, and you so you should probably limit yourself to linear models.

As Hobbes mentioned, cross validation and/or holdout sets are the proper way to judge the readiness of your model irrespective of the datapoint-to-feature ratio. If the error on the training set is 10% better than the error on the test/validation set, you're probably overfitting and your model is not fit for production. The 10% threshold is very much a rule of thumb.

Also worth mentioning, once you feel comfortable that your model is fit for production, you should never deploy it and then just walk away. If your model has value, you should do ongoing post production monitoring. In the real world, your model's performance will inevitably degrade over time (regardless of how strong/accurate it was at the time of the model build) as the environment that your model was trained for evolves. This is certainly true for finance and insurance industries.

Production monitoring can also be used to pull the plug if your model performs far worse than you anticipated. If you have a very low opinion of the potential stability of your model, deploy it silently (don't allow the predictions to go to downstream systems) and measure how your model would have done had you hypothetically deployed it.

A last piece of advice: when you do predictions on new documents you'll nearly always come across words that you have not accounted for in your training set. You should find a way to build this awareness into your model. For example, you could choose to use only keywords as predictors or you could create indicator variables for new words. Either way, you should not expect the vocabulary size to remain the same.


How to find out which features were used frequently by RandomForestClassiferer in scikit-learn? I have several random forest models that work well. Now, I would like to do feature selection based on this models. How do I find out which features were used frequently by a model e.g. RandomForestClassifier in scikit-learn?

Feature importance it is a property of the random forrest classifier.

See an example here


Detecting smoke and bluring/filtering using CNN in python I need some help to make an CNN via python to detection of fire-smoke and make the fire-smoke blur. What should I do where can I start some help please?

There is a book on Methods and Techniques for Fire Detection.

Computer vision based method for real-time fire and flame detection is one of the most cited papers on this topic. However, CNN is not a widely applicable method here.

Update:

Here is implementation of EXPERIMENTALLY DEFINED CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE
VARIANTS FOR NON-TEMPORAL REAL-TIME FIRE DETECTION.


Finding P value - Explain def get_pvalue(con_conv, test_conv,con_size,  test_size,):
    lift =  - abs(test_conv - con_conv)
    scale_one = con_conv * (1 - con_conv) * (1 / con_size)
    scale_two = test_conv * (1 - test_conv) * (1 / test_size)
    scale_val = (scale_one + scale_two)**0.5
    p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )
    return p_value

I have this function and I would like to know what it is actually doing and how it is actually calculating the p-value.

This is to find the difference between the conversion rate of control and test and group from an A/B test.

con_conv --> Conversion rate for control group
test_conv --> Conversion rate for test group
con_size --> population size for control group
test_size --> population size for test group

I understand that scale_one and scale_two are calculating the variance for each group, but I don't understand why they are adding both of them to calculate the standard deviation and why they are multiplying the cdf with 2 to get the p_value.

p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )

This is the key for your question: The p-value is the probability that the null hypothesis is true.

If the null hypothesis is true: Your model does not find any differences between groups.
If false: Your model finds differences between groups.

Given that you are using a model which its subyacent assumption is normallity (amongst others), the hypothesis test is to be tried comparing the probability in the context of a normal distribution.

The function stats.norm.cdf returns the probability of "lift being close to zero" if lift is supposed to be "normal". If lift is zero, then there is no difference between groups, so a p-value of <0.01 tell us that the probability that the groups are equal is almost 0, meaning that your groups are different.

The 2 is due to a concept called "two-tailed distribution": The difference between groups can be A greater than B or B greater that A, that's why you measure the difference in either two of the ways.

The addition between standard deviations obeys the concept of:
$Var(X+Y) = Var(X) + Var(Y)$ if $X$ and $Y$ are independent.


Opinions on an LSTM hyper-parameter tuning process I am using I am training an LSTM to predict a price chart. I am using Bayesian optimization to speed things slightly since I have a large number of hyperparameters and only my CPU as a resource.

Making 100 iterations from the hyperparameter space and 100 epochs for each when training is still taking too much time to find a decent set of hyperparameters.

My idea is this. If I only train for one epoch during the Bayesian optimization, is that still a good enough indicator of the best loss overall? This will speed up the hyperparameter optimization quite a bit and later I can afford to re-train the best 2 or 3 hyperparameter sets with 100 epochs. Is this a good approach?

The other option is to leave 100 epochs for each training but decrease the no. of iterations. i.e. decrease the number of training with different hyperparameters.

Any opinions and/or tips on the above two solutions?

( I am using keras for the training and hyperopt for the Bayesian optimisation)

First of all you might want to know there is a "new" Keras tuner, which includes BayesianOptimization, so building an LSTM with keras and optimizing its hyperparams is completely a plug-in task with keras tuner :) You can find a recent answer I posted about tuning an LSTM for time series with keras tuner here

So, 2 points I would consider:

I would not loop only once over your dataset, it does not sound like enough times to find the right weights.
I would rather control the number of possible hyperparams configurations as you said, which is something you can indicate in keras tuner via max_trials param
About using keras tuner with Bayesian tuner, you can find some code below as an example for tuning the units (nodes) in the hidden layers and the learning rate:

from tensorflow import keras
from kerastuner.tuners import BayesianOptimization

n_input = 6
def build_model(hp):
    model = Sequential()
    model.add(LSTM(units=hp.Int('units',min_value=32,
                                    max_value=512,
                                    step=32),
               activation='relu', input_shape=(n_input, 1)))
    model.add(Dense(units=hp.Int('units',min_value=32,
                                    max_value=512,
                                    step=32), activation='relu'))
    model.add(Dense(1))
    model.compile(loss='mse', metrics=['mse'], optimizer=keras.optimizers.Adam(
        hp.Choice('learning_rate',
                  values=[1e-2, 1e-3, 1e-4])))

return model

bayesian_opt_tuner = BayesianOptimization(
    build_model,
    objective='mse',
    max_trials=3,
    executions_per_trial=1,
    directory=os.path.normpath('C:/keras_tuning'),
    project_name='kerastuner_bayesian_poc',
    overwrite=True)

bayesian_opt_tuner.search(train_x, train_y,epochs=n_epochs,
     #validation_data=(X_test, y_test)
     validation_split=0.2,verbose=1)

bayes_opt_model_best_model = bayesian_opt_tuner.get_best_models(num_models=1)
model = bayes_opt_model_best_model[0]

You would get something like this, informing you about the searched configurations and evaluation metrics:


Is it okay to use training data for validifying the trained model? Currently, I have trained my model through 5-fold cross validation with very small amount of the sample (n=100).

I used whole data set to train and got quite low performance in terms of accuracy, which is bit higher than 70%.

However, if I put my data which was used for training back to trained model to validate, it gives me higher accuracy (80%).

So, my question is it okay to say that I have verified my trained model using training set and got 80 of accuracy? or should I have to stick with 70 % of accuracy that I received from 5-fold cross validation?

Scoring your model using the training data is not a best practice.  The reason is that you have already used this data to develop the "guess" at an accurate model.

This is akin to saying - 'Pick a random number between 1 and 10 (20 times)', here's a list of the last 10 numbers I picked to get you started.  You pick 20 new numbers and come up with 5 correct (25%).  What you are suggesting is that instead of reporting the 25% accuracy, that you would throw the first 10 back into the mix and report a 50% accuracy because now you've gotten 15 out of 30 right.

So there are two different things here.  The MSE of your initial model against the data you gave it - provides you and understanding of how well the model demonstrates history.  While the MSE (or other error metric) of the 'test' data shows how well you can predict the future.


Understanding the training phase of the tutorial "Using Keras and Deep Deterministic Policy Gradient to play TORCS" tutorial I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016.

The tutorial says:

  Then the actor policy is updated using the sampled policy gradient:
  $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$

which in the code corresponds to: actor.train(states, grads).

In the actor.train() method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.

I did read:

self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)

where self.action_gradient corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and tf.gradients(self.model.output, self.weights) corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no  multiplication.

Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?

Look at the documentation of tf.gradients and you will see that the third parameter is used to weight the gradient calculation of first param wrt to second param. This is phrased in the documentation as

  A user can provide their own initial grad_ys to compute the derivatives using a different initial gradient for each y (e.g., if one wanted to weight the gradient differently for each value in each y).

So it essentially uses the grad_ys parameter as an element-wise multiplication.

The parameter update gradient calculation in the project code is feeding in the already-calculated value of $\frac{\partial \mu(s|\theta)}{\partial \theta}$ as a placeholder, and using it for this third param.

So the multiplication of the two gradient factors occurs within tf.gradients.


Time Resolution Changes in Time Series Forecasting I am doing time series forecasting for estimating monthly sales of certain consumer goods SKU.

I have last 3 years of actual sales data, wherein first 2.5 years of data is monthly sales data, but last 6 months data daily sales data is available.

I can certainly add the daily data to get monthly data, but considering that total data points are then only 36 (for 3 months) and it may be slightly challenging to get various models like ARIMA, Exponential models.

Is there a way to use the daily last 6 months of data to improve the accuracy of the model?

I would do two separate analyses:

By day for six months
By month for the last 36 months (with the most recent 6 aggregated)

The 36 month analysis will be important for catching annual trends you wouldn't be able to see with the daily data, but you'll have a lot more fine-grain with the daily analysis to catch quarterly, monthly, and weekly cycles.


How can we get gradient with some other loss function apart from MSE? In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE.
Can we have an example, where we did not use derivative of MSE, but used derivative of some other loss function?
If so, please show the Python code.

Yes we have. MSE loss function is usually used for regression problems. We can have different loss functions according to our needs. Follow the article below.
https://towardsdatascience.com/common-loss-functions-in-machine-learning-46af0ffc4d23
If you wish to learn as to what does gradient does in the algorithm   you can visit this article:
https://towardsdatascience.com/wondering-why-do-you-subtract-gradient-in-a-gradient-descent-algorithm-9b5aabdf8150


Is the search space of Hyperparameters Continuous or Discrete? I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete?
My understanding of both those cases:
1. Continuous would make it 'easier' to look for hyperparameters combinations that are more fine-tuned to the problem.
2. Discrete would imply that a more brute-force (or random) approach would be helpful.

Subsidiary question: Am I mistaking on my understanding of either of those cases?

Thanks

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:

learning rate of SGD is continuous hyper parameter.
NN optimizer (SGD, Adam, RMSprop) is discrete parameter

From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:

grid search to initialize solutions
if your'e unhappy with the results:

run some hill climber on space, like simulated annealing
run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators


Handling huge dataset imbalance (2 class values) and appropriate ML algorithm I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem is that there are only 725 OLD instances in the train set and 95 in the test.

The only algorithm which succeeds for me to handle imbalance is NaiveBayes in Weka (0.02 precision for OLD class), others (trees) classify each instance as NEW.
What is the best approach to handle the imbalance and the appropriate algorithm in such a case?

Thank you in advance.

I'm not allowed to comment, but I have more a suggestion: you could try to implement some "Over-sampling Techniques" like SMOTE:
http://scholar.google.com/scholar?q=oversampling+minority+classes


IDE for Machine Learning workflow - AWS SageMaker Looks like we don't really use an IDE in any of Machine Learning workflow stages if we use AWS SageMaker.
Entire work is done in jupyter notebook. Is this correct?

Some ide you can use in your local machine :

Spyder
Jupyter Notebook
Jupyter Lab

Pycharm VSCode also has intregated support for notebook but i would suggest jupyter lab. It's really good
If you want to run on VMs then google colab and kaggle kernel is really good for free.


Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied onehotencoding to these columns.
As a result, I have a (3000*1835) sized array. This too I got after extracting only director's data from 'Crew' columns and applying PCA with 60% variance retention.
But, when I apply polynomial regression, I get the below mentioned error:

  $\lib\site-packages\sklearn\model_selection_validation.py:532:
  FitFailedWarning: Estimator fit failed. The score on this train-test
  partition for these parameters will be set to nan. Details:
  MemoryError: Unable to allocate 30.2 GiB for an array with shape
  (2400, 1686366) and data type float64

I am using the code as shown below for polynomial regression:

polyFeature = PolynomialFeatures(degree=2)
linearRegression = LinearRegression()
pipeline = Pipeline([('polyFeature',polyFeature),('linearRegression',linearRegression)])
score = cross_val_score(pipeline,XTrain,YTrain,n_jobs=4,cv=5)

I am using a system with 6 cores, 32 GB RAM.

You are allocating a float64 (2400, 1686366) array (very big!), no surprise you get a memory error.
You say that you have 1835 columns, but the error message says 1686366, so which one is the correct number of columns? If it's 1835 then there's an error before feeding your set to the model.

Also, are you sure you need a float64 data type? You can reduce the memory consumption by using float32 instead of float64 (unless you really need to work with that precision)


How to structure my data into features and targets for PCA on Big Data? I want to apply the PCA algorithm from Scikit-Learn.(https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) At the part where I have to separate the features and the targets I got a little bit lost. On the Iris dataset is easy and I understood, but now I am facing a real dataset. My problem is that part of the original data was meant to be into an array and as the database could not hold it then the data was split into columns. However, some columns are referring to the same thing. Please consider the example below.

To be more specific, I designed an example dataframe which is almost the same as mine.

Explained dataset: At a respective date, at a specific place, we had the same four kids being measured. Robin, Ted, Lilly and Jamie are their names. Each number next to their names refers to their height, weight, and other specific measurements. The database could not keep the array for each child with their measurements (height, weight, etc) so the data from the array was split into columns (Robin_1, Robin_2, Robin_3, etc.. for each child).

Specific problem: I do not know how to structure my features/targets. I selected the Date (which is also the index), Country, City and School as being the features. On them I apply the principal components (3 in number -> 3D plot). How should I select the target ? Is there any way on somehow grouping them before clustering? In the final result, I want to cluster each the full Robin, full Ted, full Lilly and so on. I have exactly four kids, but their attributes differ (each one can have different number of attributes (ex. Robin -2, Ted- 10, Lilly-7, etc)) --> four clusters
P.S.: I cannot cluster them by the height, weight, etc. Those are only example attributes.

A visualization image plot of the final wanted result:

Any help would be of use and I would highly appreciate any effort in helping me with this problem. I have been reading a lot but I have not met this problem before. If there is another way to structure/consider them please let me know. Please let me know if I have to be more explicit in a respective part if I did not explained/tell something important.

First of all, if the "weight" etc. attributes should not be used for clustering, then don't include them. It makes you question hard to follow. On the other hand, if you don't include them  there is obviously no difference between the kids... So I'm not certain you have understood what you want, need to do, or ask for.

PCA is not a magic tool that makes you data clusterable! It is a very specific tool to decorrelate data that should only be used when you have strong, known, linear correlations in your data that you need to ignore. It will not work when you have non-linear correlations (e.g., income and number of yachts), or when your data is not continuous on a linear scale. In particular you shouldn't use it on your first attributes such as date, state, etc.

What you first will want to do - and that has nothing to do with PCA nor clustering, is to pivot your data to separte the kids. Consider transforming your messy table into "tidy data" form first, this will make such cleanup much more structured and easier to do.


How modelling is affected by similar feature distributions across classes? I have grouped my dataset according to labels (good and bad customers), in an attempt to test how each feature is distributed within.
Along this, I found some feature has almost exact distribution in both groups and
I'm wondering what insight can I learn from this? Can it mean that the feature isn't that significant to data prediction and can be dropped?

EDA
What you are doing there, falls under Exploratory Data Analysis (EDA). A better way to investigate how your features between them and are distributed across classes is a correlogram, sometimes referred to as pairplot.
A correlogram helps with

investigating relationships between pairs of numerical features, via scatter plots for each pair of features
inspecting the distribution of each feature, using a histogram or a density plot in the diagonal of the pairplot.

You can easily create one of those in seaborn:
# library & dataset
import seaborn as sns
import matplotlib.pyplot as plt
df = sns.load_dataset('iris')

# Basic correlogram
sns.pairplot(df, kind="scatter")
sns.plt.show()

# Annotate classes with in different colours
sns.pairplot(df, kind="scatter", hue="species")

# Use regression instead of scatter
sns.pairplot(df, kind="reg", hue="species")

You can see how classes may or may not form clusters which an algorithm could potentially draw hyperplanes by combining and transforming features.
Your question

Can it mean that the feature isn't that significant to data prediction and can be dropped?

The y axis maximum value differs between the two plots you have posted, and so it is not easy visually inspect fairly.
One could potentially assume that the model may not "work" with that particular feature to begin with, should there be other variables in that dataset that are easier to separate. This assumption however (a) can be considered naïve given how ML algorithms work to transform the feature space in order to make classes separable and therefore (b) does not constitute grounds for dropping this feature.
A less useful feature in your dataset can still increase the performance of your model. In the case of unbearably large feature space you can use dimensionality reduction techniques (PCA, kernel PCA, autoencoders).
Dropping features usually relates to how well these relate to the ground truth e.g. noise, or how they may affect feature importance and model stability e.g. multicollinearities.

Image from: https://stackoverflow.com/questions/59212378/how-do-i-get-the-diagonal-of-sns-pairplot


how to calculate p value I need to calculate statistical significance between two time series, each with 4500 terms. My null hypothesis is that $H_0: \mu=\mu_0$ (value in time t of first time series is equal to value in time t-1 of second time series). How can i calculate p value? Is Z test approved for p value calculation? How to get p value after z test? I have $\alpha=0.05$.

I will rephrase you setup to make sure I understood it correctly.

  You have two time series $x = (x_1, x_2, .., x_n)$ and $y = (y_1, y_2,..,y_n)$, where $n=4500$. You want to understand if $x_t = y_{t-1}$, where $t \in [1,n]$ and by $=$ means approximately equal in statistical sense.

There is one way but you have to make some assumptions.

In general a time series can be thought as a signal plus additional noise. The signal is an explicit function which models some phenomenon and the noise is some alteration of the output which is random by nature. Using this model you split $x_t$ and $y_{t-1}$ into something like:

$$x_t = s^{(x)}_t + w^{(x)}_t$$
$$y_{t-1} = s^{(y)}_{t-1} + w^{(y)}_{t-1}$$

where $s^{(x)}_t$ is the signal of $x$, $w^{(x)}_t$ is the noise of $x$, respectively $s^{(y)}_{t-1}$ is the signal of $y$, and $w^{(y)}_{t-1}$ is the noise in $y$. Notice that $s^{(x)}_t$ and $s^{(y)}_{t-1}$ are well determined and noises are random.

Using this notation the idea of approximately equal in statistical sense means that the signal is the same and if there are difference in samples, than the differences are caused by random components (aka. noises). This means that if you assume that signals are the same and remains only the noise. So what your null hypothesis is $H_0 : x_t-y_{t-1}=0$ can be written as:
$$H_0 : s^{(x)}_t + w^{(x)}_t - s^{(y)}_{t-1} - w^{(y)}_{t-1} = 0$$
$$H_0 : w^{(x)}_t - w^{(y)}_{t-1} = 0$$

Now here comes the assumptions. You have two sorts of random variables (noises) which pose some difficulties when you are trying to apply tools based on central limit theorem.

One problem is that the random noises depends on time. You have to assume that the noise does not change with time, in both series. This is translated as independent and identically distributed or ($w^{(x)}_t = w^{(x)}$ and $w^{(y)}_{t-1} = w^{(y)}$).

So, if you assume that noises are independent and identically distributed your question can be worked out with a paired difference test. This means you can test if the differences depart significantly from $0$.

You can use a t-test dependent t-test for paired samples if you additionally assume that the difference is normally distributed. You cannot use z-test on paired differences because you do not know the variance of the difference's distribution (when you use sample variance for variance in a z-test, you fall into a t-test).

If you do not want to assume that the differences are normally distributed (that can be checked graphically at least with histograms or qq plots for skewness) you can use Wilcoxon signed-rank test, which should be fine for the number of samples you have. I would pick this test, for the same reason.

Attention: I restate again that those tests are valid if an only if the noises in both series are independent of time. Also you can test only if they different. You cannot prove that they are equal, you can only state that you do not have evidence that they are different, which for most cases should be enough.


Correlation among features (e.g. doc length, punctuation, ... ) in classifying spam emails I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value:

looking at the correlation with my target variable (1=spam, 0=not spam), using .corr() in python.
BT stands for binary text, e.g., and BS stands for binary summary, where I assign 1 or 0 based on the presence of a capital letter in the text/summary, or upper case word, or...
Do you think that features like these can be useful in model building? I cannot see very strong correlations, but I would like to determine if an email can be spam or not based also on features like these (number of character/text length; presence of !, upper case words,....).
I have around 1000 emails, but only 50 are spam (maybe too small to extract useful information).
However, I had to extract these information, so it is a new dataset, built on my own, so I could not get many more spam emails (and I would like to not use datasets from kaggle, for instance).
What do you think?

First about the features i  think you could add some such as :

the time when the letter is received,
number of links in the email,
the whole structure (does it follow typical structure for email),
number of words that contains numbers in it,
what is the whole mood of the email (sales,threats,info,...-for this purpose you can use sentiment analysis),
number of attachments ,
type of attachments and so on.

After that try with feature selection (you could read more about it here). For the imbalance data you need to resample the data. I would:

add copies of spam emails(oversampling)
try to generate new spam email (smote)

You can read more here.
I hope my answer will give you some clarity.


The added layer must be an instance of class Layer. Found: I don't know why BatchNormalization is giving the following error. Googled out but couldn't find any relevant answers

My code appears to be this:

How did you import "BatchNormalization" layer?
I had the same error and the solution was changing the way I was importing:

from tensorflow.keras.layers import Activation, Dropout, Dense, Conv2D, Flatten, Dropout, MaxPooling2D, BatchNormalization

I put the BatchNormalization with all the other layers, and it work in my case.

Hope it is useful.


How to make predictions of multiple input samples at once in tf 2 with keras I am quite confused on the output of model.predict

when after training I validate my model on around 6000 samples I use the following pseudo code:

model.fit(...)
predictions = model.predict(val_set)
len(predictions) # == len(val_set)

result: tensor array of shape=(len(tensor_array),14) (one prediction for each input sample)

in production I currently use the following code to ocr image numbers:

model = tf.keras.models.load_model('number_ocr_v2')
def predict(list_images):
    global model
    print("length:")
    print(len(list_images))
    #predictions = model.predict(list_images) # <- Same result
    predictions = model.predict_on_batch(list_images)
    print(len(predictions))
    print(predictions)

console Output:

  length:
  30
  {...tf loading text...}
  1
  tf.Tensor([[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]], shape=(1, 14), dtype=float32)

(one prediction for x input samples)

Apparently the Model delivers one array with only the prediction of the first input tensor.

Where are the rest of the 30 Input Tensors? What is the difference between my training code and my production environment that explains why training code delivers predictions[:] and production code only delivers predictions[0]?

A Time consumption benchmark suggests that all 30 digits are beeing processed. Timing fits with 30 Predictions a ~70us per image

Best regards,

Julian

In my case, the Input Tensor Array was of length 30 but faulty constructed.

Make sure your inputs are right, otherwise it can lead to inaccurate results


Can't understand this simple matrix multiplication in python Ok guys, I might be very tired here, but I can't figure out why this matrix multiplication by a scalar gives the following result (python)

  Matrix named 'dx'
        [ 1.6,  3.6,  0.4, 14.4, 25.6],
        [10. , 10. ,  0.4, 14.4,  3.6],
        [ 0.4,  0. ,  0. ,  1.6, 10. ],
        [ 6.4,  0. ,  3.6,  1.6,  0.4],
        [14.4,  0. , 25.6,  0.4,  6.4]

  10 * dx, in python, gives
        [ -40.,  -60.,  -20., -120., -160.],
        [-100., -100.,  -20., -120.,  -60.],
        [ -20.,    0.,    0.,  -40., -100.],
        [ -80.,    0.,  -60.,  -40.,  -20.],
        [-120.,    0., -160.,  -20.,  -80.]

From what I understand, each member should be multiplicated by 10, but it's clearly not the case.

I'm using Python 3.6 & numpy

What Am I missing ?
Link to matrix multiplication by scalar , pretty basic stuff :
http://www.purplemath.com/modules/mtrxmult.htm

Thanks !


What are some good papers that discuss Tufte's 'data density index' and 'data-to-ink ratio' in data visualisation? I've found a number of resources that mention Tufte's 'data density index' and 'data-to-ink ratio' when considering the analysis of particular visualisations and visualisation techniques, but I am yet to come across any papers that actually critique the measures themselves.

Are there critiques of these principles available?

There are critcism's of Tuft's principles.  Some examples:

Sometimes We Must Raise Our Voices, Stephen Few, Perceptual Edge

  Even after many years of working in the field of data visualization, which has involved a great deal of experience and study that has expanded my expertise into many areas that Tufte hasn’t specifically addressed, I have only on rare occasions discovered reasons to disagree with any of his principles. The topic that I’m addressing in this article, however, deals with one of those rare disagreements.

Minimalism in information visualization: attitudes towards maximizing the data-ink ratio

From the abstract:

  People did not like Tufte's minimalist design of bar-graphs; they seem to prefer "chartjunk" instead.


How to increase accuracy of classifiers? I am using OpenCV letter_recog.cpp example to experiment on random trees and other classifiers. This example has implementations of six classifiers - random trees, boosting, MLP, kNN, naive Bayes and SVM. UCI letter recognition dataset with 20000 instances and 16 features is used, which I split in half for training and testing. I have experience with SVM so I quickly set its recognition error to 3.3%. After some experimentation what I got was:

UCI letter recognition:

RTrees - 5.3%
Boost -  13%
MLP -    7.9%
kNN(k=3) -   6.5%
Bayes -  11.5%
SVM -    3.3%

Parameters used:

RTrees - max_num_of_trees_in_the_forrest=200, max_depth=20,
min_sample_count=1
Boost -  boost_type=REAL, weak_count=200, weight_trim_rate=0.95,
max_depth=7
MLP -    method=BACKPROP, param=0.001, max_iter=300 (default values - too
slow to experiment)
kNN(k=3) -   k=3
Bayes -  none
SVM -    RBF kernel, C=10, gamma=0.01

After that I used same parameters and tested on Digits and MNIST datasets by extracting gradient features first (vector size 200 elements):

Digits:

RTrees - 5.1%
Boost -  23.4%
MLP -    4.3%
kNN(k=3) -   7.3%
Bayes -  17.7%
SVM -    4.2%

MNIST:

RTrees - 1.4%
Boost -  out of memory
MLP -    1.0%
kNN(k=3) -   1.2%
Bayes -  34.33%
SVM -    0.6%

I am new to all classifiers except SVM and kNN, for these two I can say the results seem fine. What about others? I expected more from random trees, on MNIST kNN gives better accuracy, any ideas how to get it higher? Boost and Bayes give very low accuracy. In the end I'd like to use these classifiers to make a multiple classifier system. Any advice?

Dimensionality Reduction
Another important procedure is to compare the error rates on training and test dataset to see if you are overfitting (due to the "curse of dimensionality"). E.g., if your error rate on the test dataset is much larger than the error on the training data set, this would be one indicator.
In this case, you could try dimensionality reduction techniques, such as PCA or LDA.
If you are interested, I have written about PCA, LDA and some other techniques here and in my GitHub repo here.
Cross validation
Also you may want to take a look at cross-validation techniques in order to evaluate the performance of your classifiers in a more objective manner


How to predict user next purchase items I have an e-commerce website where customers can purchase items directly from the site. I have training data which includes order id, user id, order number, days since prior order, product id, add to cart order, reordered...

I am trying to predict, for each user, what items he will purchase on his next order. I tried to use Naive Bayes, average purchase items per user and the following equation: posterior ~ Bayes Factor x prior but the prediction outcome is not good and has many false positives and/or negatives.

Maybe I can try to first train on the number of items a user will purchase then train on the specific items he will get but not sure will it get better results.
I think this can go in the multi label classification but has not used multi labels in classification before.

I am using python with sklearn, pandas...

Any better models I can use and how to train and predict variable multi labels and whether I can do it in sklearn? Keep in mind that the data is large and predicting using some of the classification algorithms in sklearn unfortunately takes huge amounts of memory so, any ideas on how to reduce memory consumption would also be useful.

First of all you have to realize these kind of problems have large amounts of noise compared to signal, because predicting what someone will buy based on a very small window of information is difficult. That said, you are throwing away a lot of information with your current approach. Temporal aspects include a ton of information, for example the sequence in which items were bought etcetera. While this is a lot more complicated than what you are describing now, you could look into recurrent neural networks where you feed history up to the point of prediction as a sequence and predict the item they will buy next as softmax classification. This will depend on the amount of products that you offer whether this is feasible or not. Another advantage is that so-called 'out-of-core' training is relatively easy with neural networks due to the iterative training of batches. Multi-label is also clean, you can just add a number of labels at the end of your graph if necessary.


Deciding what type of model to use for predicting the bottom decile of student grades I have a large dataset which includes 36 variables (in %iles) to describe a student, and then the output is the students grades as a %ile. I am trying to predict, using the 36 variables, whether a student will fail out, ie. be in the bottom 2%ile.

My first attempt was a neural network with binary classification, forcing the model to predict that the student is either in the bottom 2%ile, or not. Having read a few posts on FHarrell.com, I was thinking that a binary logistic regression that outputs the probability of being in the bottom percentiles is better?

Or alternatively, should I just do a non-binary linear regression to directly predict the students %ile? The reason I thought that this wouldn't be as useful for my purposes is because I don't care about making predictions between any of the other percentiles. For example, I don't care about distinguishing between an A and a B student, I am just looking to identify F students. Therefore I would be losing accuracy in the bottom 2%ile and gaining it in the other irrelevant %iles?

Finally, I think that the neural network makes sense given that there is interaction between the variables. But if there is another model that I should consider please let me know.

My first attempt was a neural network with binary classification,
  forcing the model to predict that the student is either in the bottom
  2%ile, or not. Having read a few posts on FHarrell.com, I was thinking
  that a logistic regression that outputs the probability of being in
  the bottom percentiles is better?

I would recommend you start with logistic regression as a baseline. Then when you have your baseline I would compare the LR-model with a tree-based ensemble since they tend to work really well on tabular data. I would recommend random forest, it is easy to work with and is not that prone to overfitting. Then if you are feeling fancy you could try gradient boosting with XGBoost or LGBM. They tend to give marginally better results, but takes some more effort to work with.

  Or alternatively, should I just do a non-binary linear regression to
  directly predict the students %ile? The reason I thought that this
  wouldn't be as useful for my purposes is because I don't care about
  making predictions between any of the other percentiles. For example,
  I don't care about distinguishing between an A and a B student, I am
  just looking to identify F students. Therefore I would be losing
  accuracy in the bottom 2%ile and gaining it in the other irrelevant
  %iles?

I agree.

  Finally, I think that the neural network makes sense given that there
  is interaction between the variables.

Neural networks are generally not that good on tabular data. Unbalanced, which your have with a ration of 1:50, is especially hard for NNs. Then they tend to classify everything as the majority class.


Significance of comparing Receiver Operating Characteristic (ROC) curves An ROC curve plots the true positive rate (sensitivity) as function of the false positive rate (100-specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. This would permit one to compare the goodness of the model when varying the given parameter (e.g. number of trees in a Random Forest) to help optimize that particular classifier's parameters. Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)? I came across an instance of such an example on the second last slide here and was trying to understand the significance of these curves. As far as I am aware, the graph displayed appears to not be a necessarily fair comparison when comparing the area under the curve while tuning different parameters in different types of classifiers.

Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)?

Yes, because you can clearly see which model is performing best overall. The more you get close to the top left corner of the graph, the better your model is. (the blue line in your example, Random Forest)

If you compute the Area under the ROC curve in the example, you'll get:

AUC(RF) > AUC(MLP) > AUC(SVM)

So AUC is a good index to show which curve is "higher".

Now, sometimes you could be interested in a model that has a "steeper" ROC curve in the bottom left corner (higher chance of improving TPR, with low cost in term of FPR), and maybe that model isn't the best in term of AUC. (Not the case in your example)

In that case you would not choose AUC as a term of model choice, because it doesn't select the "best" model for you.

Overall the ROC curve between different models helps you a lot, showing not only the performance over different thresholds, but also comparing different solutions simultaneously.

Edit:

Each model gives a number for each observation that is the probability to belong in one class (a number p between 0 and 1).
Say you get p = 0.3 for one observation, then with a threshold of 0.5 you will label that observation to one of the two classes (assuming there are only two labels).

Your rule is:

If p < 0.5 (threshold) then obs is labeled to class "one"
If p >= 0.5 (threshold) then obs is labeled to class "zero"

With ROC curve, you basically have a "high" number of thresholds, say going from 0.01 to 0.99. For each model.

So for each model, you will assign each observation to the respective class, using for t = 0.01, then t = 0.02, and so on till t = 0.99. (t = threshold)


How would you describe cluster 2 from this output of a run of the EM program? My description:

Cluster 2 consists of 9511 instances, the age is around 42 (ranges between 29.7207 and 54.5257). Considering Age, Cluster 2 is very well separated from Cluster 1, with a distance of 18.9513. On the other hand, Cluster 2 and Cluster 0 are very close though, their centroids are withihn a distance of around 0.8248.

What else could be added?

Welcome to the community!

So, in clustering if the number of clusters you indicate Apriori, is not right (what is right indeed?!! it means the intrinsic number of clusters inside data) then some clusters will be broken down to more clusters and what you see here happens (and yes, you need to tell the number of desired clusters to most of clustering algorithms (including GMM that you use) Apriori!)

In GMM clustering using EM algorithm, you can simply plot the histogram of the data and try to count the number of single Gaussians, which summing together, build up the histogram. that is the best choice of number of clusters.

Histogram (he called it PDF because PDF is simply histogram divided by the integration of area under histogram curve) below is taken from this kernel in the Kaggle competition from which your data comes.
It simply shows (by arrows) that data inhibits 2 clusters intrinsically so using 3 clusters miss-partitions one cluster to two. What happened in your result.

Try the same run with two clusters and you will most probably see the problem solved :)

Hope it helped. Good Luck!


LSTM or other RNN package for R I saw some impressive result from LSTM models producing Shakespeare like texts. I was wondering if an LSTM package exists for R. I googled for it but only found packages for Python and Julia. (maybe there are some performance issue which explains why these programs are more preferred over R) Do You know about an LSTM (or at least an RNN) package for R? If exists are there any tutorials for using them?

Have a look at the rnn package (full disclosure, I am the author). It implements a multilayer RNN, GRU, and LSTM directly in R, i.e. not an underlying C++ library, so you should also be able to read the code and understand what is going on.

install.packages('rnn')

The CRAN version is quite up to date, but the GitHub version is bleeding edge and can be installed using:

if (!require('devtools')) install.packages('devtools')
devtools::install_github('bquast/rnn')


Scaling values for LSTM I have the following time series data set

Each row is a unique Item, and each column shows the amount purchased per day. There are a total of 33 columns.
I'm taking the first 32 columns(leaving out the last column, which will be my target) as my training set, and the last 32 rows (leaving out the first column) as my testing set
X_train = dataset[:, :-1] # taking all columns except the last column

y_train = dataset[:, -1:] # setting the last column to be the target

X_test = dataset[:, 1:] # taking all columns expect the first column

I'm going to feed X_train and y_train in my LSTM model, and use the model to perform prediction on X_test.
Now, I wish to performing minmax scaling on dataset before performing training,  but I have some questions:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_dataset = scaler.fit_transform(dataset)

This has obvious data leakages, because the scaler is fitted with the test values.
I thought of creating 2 scalers, one for the training set, and one for the target
scaler_x = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(-1, 1))
scaled_all_data = scaler_x.fit_transform(dataset[:, :-1])
scaled_y = scaler_y.fit_transform(dataset[:, -1:])

I'm not sure if that's the right approach. I've already tried searching for answers, but their situation is not quite like mine, or the questions are unanswered yet.
Any advice on how I should perform value scaling?

Why not scale the values as the very first step? Import your values, scale the values and then do your splits for training/test/etc.


Tensorflow API: What does the metric `tf.keras.metrics.TopKCategoricalAccuracy` do? According to the API doc, this metric

  "Computes how often targets are in the top K predictions."

But how come the following codes prouce the result 1?
0.95>0.9>0.8>0.1>0.05, both 0.95 and 0.8 lead to 1 in prediction, shouldn't the result be 2?

m = tf.keras.metrics.TopKCategoricalAccuracy()
m.update_state([[0, 0, 1], [0, 1, 0]], [[0.1, 0.9, 0.8], [0.05, 0.95, 0]])
print('Final result: ', m.result().numpy())  # Final result: 1.0

Result of tf.keras.metrics.TopKCategoricalAccuracy() will be between 0 & 1. Default value of the argument k is 5.
The result is 1 because for both the samples, the actual value is within the top 5 predictions.


Prediction algorithm for traffic I am working on making predictions about traffic trends in a big city. Traffic is very seasonnal, and depends on the time of the day (with peaks at 9am and 6pm), day of the week (Monday to Friday are alike compared to Sat and Sun), holidays (Xmas, school holidays etc). And I have 2 years (at hourly rate) of data.

I am using a RandomForestRegressor to make a prediction on the year to come. Is that a correct algorithm for this class of problem?

I think the best way to deal with this problem is to use blending (although it is not a bad idea to start with random forest just to get sense about the problem you have). If you are not familiar with the word blending you can check this page:
http://mlwave.com/kaggle-ensembling-guide/

But let me explain what is the idea:
I believe that there is a kind of relation between the features you have: you can group all the features related to Weather together, The day, time ..etc together. Each set of features can be used to train a random forest. The output of these random forests can be combined by either linear or non linear regression blending techniques.

If the database you have is very large it is better to read more about blending techniques and do some experiments before going to your problem to make yourself familiar with it


weight sharing among neurons at same depth I'm trying to understand some visual illustrations about the wight sharing in the Convolutional Neural Network as following:

In this picture we see that for different outputs different inputs share the same weights, as the weights are associated with the color (same color - same weight).
So in the Stanford example we can see it eather:

As for example, for the o(0,0,0)=5 and for o(0,1,0)=7 both we used the w0[0],w01,w2 weights,
and it fits the picture above.
Now there is this picture:

And I'm not fully understand the 3 estimation. It says that the same color means the same weight,
but as I can tell in the picture we see a filter, each neuron, let's mark it as $f$ refers to the  multiplication of $f(x)=wx(+b?)$, and as much as I can tell from the Stanford exmaple, the weight can be all different and not the same color, and the weight sharing isn't representable on the filter, but only in any view that includes the input, as in the first image, so isn't it a bit misleading ?
Or maybe I just didn't understand it and anyone could explain it more briefly ?

I agree that these pictures are a bit confusing. I think they are all hinting at the following:

As you know, CNNs involve splitting up the image into many small "blocks" and multiplying each block pairwise by a filter (actually, by several different filters). Weight sharing means that all the blocks use the same set of filters -- blocks in the upper-left corner, middle, and bottom of the image all use the same set of filters.
There is no weight sharing across the channels -- we don't use the same filter banks for the red channel and the blue channel. This makes sense because different classes have different signatures in each channel. There are various ways to think about this. Personally, I find it easiest to visualize that both the "block" and the filter are three-dimensional, and the pairwise multiplication is across all three dimensions.

So, to your pictures:

The top one shows that $x_{i-1}$, $x_i$, and $x_{i+1}$ are three different channels, and so use three different weights (colors). But when switching from block $i$ to block $j$, the same weights (colors) are reused.

The middle one has four columns:

The left column is the image itself -- three rows, one for each channel.
The second column is one filter -- also three dimensional. This also shows that each channel of the image is multiplied pairwise against each filter.
The third column I assume is a second filter bank, which works the same way as the original filter bank (CNNs usually apply many different filters at each stage of the network, so that multiple patterns can be learned).
Then the fourth column is the output, i.e., the next layer in the network.

The bottom one is not representing the filter bank, but actually represents the subsequent layer of the network (i.e., the output from the convolutional operation). In this subsequent layer, each "channel" refers to the output from a different filter in the preceding layer. Given this, the picture makes sense: everything within the same channel was formed by applying the same weights to different parts of the image.  And each channel is in a different color because they were formed using a different filter, and therefore different weights.


Creating a Graph from Address Data What's the best / easiest way to create a graph from address data?  For example, if I have 100 houses from all across a city is there any easy way to determine the shortest distance between two houses and all that good stuff?  Would this require changing the data into coordinates and using GIS software or can I get away with using Python or R?

Speaking as a Graph/Complex Networks guy I'd recommend Networkx package in Python. This is the main library I used for my master thesis and my research during last 2 years. As long as your graph is not gigantic (millions of nodes) you can handle it using Networkx.

But what you need is not only the library but a philosophy to convert your data into a graph. What is your data exactly? If you have some addresses, what would be the node and what would define an edge? This is important actually. Graph is not constructed from your addresses unless you have a definition for vertices and edges.

When you convert your data into a graph Networkx provides you all fancy algorithms for everything (shortest path, community detection, statistical analysis, etc).

If you provide info in the comment here I'll answer your question in details.


Python to sum values in a column I've created a Python code that reads the data from an excel file using Pandas.

Code for your reference:-

import pandas as pd

def myFunc():
   file = r"C:\Documents\myFile.xlsx"
   new_dataframe = pd.read_excel(file,'Sheet1')
   new_dataframe.fillna(value="No Data Found",inplace=True)
   print new_dataframe

myFunc

Current Output:-

             Name        date    amount_used
0             P1   2018-07-01          40.0
1             P1   2018-07-01          40.0
2             P1   2018-07-15          40.0
3             P2   2018-08-01          20.0
4             P2   2018-09-15          50.0
5             P2   2018-08-15          40.0
6             P3   2018-08-10          20.0
7             P3   2018-08-10          50.0
8             P3   2018-08-10          40.0

In the final output, I need to sum the amount_used column based on Name and date column.

Expected Output:-

             Name        date    amount_used
0             P1   2018-07-01          80.0
1             P1   2018-07-15          40.0
2             P2   2018-07-01          20.0
3             P2   2018-08-15          90.0
4             P3   2018-08-10         110.0

How can I achieve this using pandas ?

You can use groupby and then sum
Take a look at https://stackoverflow.com/questions/39922986/pandas-group-by-and-sum


Robustness of ML Model in question While trying to emulate a ML model similar to the one described in this paper, I seemed to eventually get good clustering results on some sample data after a bit of tweaking. By "good" results, I mean that

Each observation was put in a cluster with high probability (as opposed to having being 50/50 between two clusters or something similar)
A high proportion of the observations were put in the correct cluster, indicating that the model actually did work.

For example, if we had observation $a$ that belonged to cluster $A$, and observation $b$ belonging to cluster $B$, then the model might output (0.99, 0.01) for observation $a$ (where the 0.99 indicates a high probability for $a$ belonging to $A$ and 0.01 represents a low probability of belonging to $B$) and (0.02, 0.98) for observation $b$. (These specific numbers are chose randomly, but generally the good results give probabilities close to 0 and 1.) However, every few times I train the model, I get a weird result; one that seems to still 'cluster' the data in a sense, but is technically wrong. A bad result would give something of the nature (0.99, 0.01) for observation $a$ (which is good), but then give something like (0.65, 0.35) for observation $b$ . In this way, when I look at the data I can tell that it has been "clustered" as the observations that belong to $A$ give different cluster probability distributions than the observations of $B$, but the model has not accomplished either of the two goals above when actually assigning clusters.

I would like to make my model more robust so that I get "good" results more often, but I don't know what I could do to do this. One thing that would definitely work to help avoid this problem would be to do the cluster training many times and take the average result; the problem with this is that each training takes somewhere on the order of hours, meaning that doing it multiple times might take on the order of days, which I want to avoid.

If there are any (hopefully quick) fixes I can do to avoid getting these odd results, I would love to hear any advice on the topic. I should also be able to answer questions if you need more information, but the paper I linked should have most of the pertinent info.

Clustering is an unsupervised approach wherein unlike supervised learning its devoid of not knowing the class label or the response or dependent variable. In clustering, the idea is to determine patterns in data such that it can be grouped into cohesive clusters. Such groups will have data points that are similar to each other whilst dissimilar to other data points in other groups. Now, clustering algorithms can broadly be classified into two types, namely, partitional and hierarchical. Some well-known partitional clustering algorithms in the literature are k-means (that works for numerical data only), k-modes (that works for categorical data only), and the k-prototypes (that can work for mixed data (i.e. both numerical and categorical data)).
A fundamental problem associated with partition-based clustering algorithms is not knowing the initial number of groups. And as such researchers using this class of algorithms often fall into this trap by failing to address this fundamental issue. Thus their results are not reproducible.
In literature, there exist several approaches to have proposed to solve this issue of Estimating the possible number of clusters. A well-known approach is the Elbow method proposed by researchers (Kaufman & Rousseeuw, 2009). I fact a group of researchers developed an R package called NbClust that can be used to estimate the number of clusters using the elbow method. This package contains ~30 algorithms for determining the number of clusters.
Recommendation when building a partition-based clustering model
I would recommend the following steps for result reproducibility

setting a random seed value
choose a method to determine the number of clusters
run the experiment at least 5 to 10 times, note the result for each run, and then average the values. Report the averaged value as the final result.

Reference
Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: an introduction to cluster
analysis (Vol. 344): John Wiley & Sons


Extract feature vector of a CNN How can I get the feature vector of my dataset. I have a fine-tuned CNN model with my data. Now I want to feed the features of all my dataset extracted from the last layer of the CNN into a LSTM.

So far I have

cnn_model = load_model('weights/pre-trained_CNN.hdf5')
cnn_output = cnn_model.get_layer('fc7').output

I know I will have to feed all my dataset into this model but I have no idea on how to "save" the features.

In order to "tap" intermediate layers of an existing model you could do the following:

# get your feature layer
cnn_model = load_model('weights/pre-trained_CNN.hdf5')
feature_layer = cnn_model.get_layer('fc7')

# stack your LSTM and other layers below, e.g.:
lstm_layer = TimeDistributed(LSTM(...), input_shape = ...)(feature_layer)
output = Dense(...)(lstm_layer)

# create a combined model
model = Model(inputs = cnn_model.input, outputs = [output])


Neural network is getting partially trained So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am printing the output of the last layer it is definitely not the optimal one! Let me explain it a little further...

My training data is a pretty simple one

Input: {0,0,1},{0,1,1},{1,0,1},{1,1,1},{1,0,0},{1,1,0}
Expected output: {0},{1},{0},{1},{0},{1}

I have tried many variations, even some logical gates (AND, OR, XOR, etc.)! I have tried increasing the number of the neurons on each layer or increasing the layers. I tried both ReLu as the activation function and the common sigmoid one, even some combinations! I have played with the learning rate value but I am still getting the same results. On the above given example of the training set my network's output using ReLu is

0.001585
0.999990
0.000713
0.000109
0.000000
0.000000

and the error is

Error: 2.02248e-06

using the sigmoid function the output is

0.098032
0.840373
0.046706
0.036155
0.036184
0.059911

and the error

Error: 0.0289854

As you can see the expected output of the fourth and sixth element were supposed to be values close to 1 like the second element. I am facing similar problems with different data sets.

So how am I back-propagating the network?
I am calculating the output error like this

error = 2.0 * (expected_output - actual_output)

and I am updating the weights like this

w += learning_rate * transposed_layer_output * layer_delta

Am I missing something?

I can see a problem with your error function (blue). It is treating under-predictions and over-predictions the same. The error gradient should ideally direct you towards zero error.

For your error function:

$$
E = 2 \cdot (y_{expected} - y_{predicted})
$$

The derivative w.r.t prediction is:

$$
\frac{dE}{d\;y_{predicted}} = -2
$$

Note here that the error is the same whether you under-predict or over predict (i.e. the sign of error doesn't matter in the gradient). So the updates to the weights will be in the same direction (see arrows on blue line).

But if you use a quadratic error function (orange line), over- and under- prediction errors are treated differently (i.e. the gradient depends on the sign of the $expected - predicted$ difference).

$$
E = (y_{expected} - y_{predicted})^2 \\
\frac{dE}{d\;y_{predicted}} = 2 \cdot (y_{expected} - y_{predicted}) \cdot -1
$$


Working with more than one dummy variable in neural net I am building a neural net and am at the point of working with categorical variables.   Below you can see that I transformed the categorical variables into numeric variables.

#Encode "UserName"
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Encode "Token"
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Encode "ChildEXE"
labelencoder_X_3 = LabelEncoder()
X[:, 3] = labelencoder_X_3.fit_transform(X[:, 3])
#Encode "ParentEXE"
labelencoder_X_4 = LabelEncoder()
X[:, 4] = labelencoder_X_4.fit_transform(X[:, 4])
#Encode "ChildFilePath"
labelencoder_X_5 = LabelEncoder()
X[:, 5] = labelencoder_X_5.fit_transform(X[:, 5])
#Encode "ParentFilePath"
labelencoder_X_6 = LabelEncoder()
X[:, 6] = labelencoder_X_6.fit_transform(X[:, 6])

Now I am supposed to transform any that have more than 2.  For instance "UserName" has "Machine Name", "User", and "Local Service".  "Token" has about 10 options, "ChildEXE" has thousands etc.

X
array([[2, 0, 20788, ..., 46, 31, 24],
[2, 0, 19088, ..., 46, 31, 24],
[2, 0, 2840, ..., 27, 42, 15],
 ...,
[2, 0, 20148, ..., 17, 40, 32],
[2, 0, 20148, ..., 47, 23, 0],
[2, 0, 3176, ..., 48, 42, 32]], dtype=object)

Do I use the code below several times for each variable that has more that 2?

onehotencoder = OneHotEncoder(categorical_features = [0])
X = onehotencoder.fit_transform(X).toarray()

I found that this works:

onehotencoder = OneHotEncoder(categorical_features = [0,1,3,4,5,6])
X = onehotencoder.fit_transform(X).toarray()


Why mini batch size is better than one single "batch" with all training data? I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this.

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration; and epoch is a full turn when each of the training instances have been seen by the model. If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1.
In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point).
In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a single instance of the dataset. Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in [2]). The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution.
The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.
1 Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 (pp. 177-186). Physica-Verlag HD.
[2] Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015, June). Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. In COLT (pp. 797-842).
EDIT :
I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.)

Training with large minibatches is bad for your health.
More importantly, it's bad for your test error.
Friends dont let friends use minibatches larger than 32.
Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,
Dominic Masters, Carlo Luschi, Revisiting Small Batch Training for Deep Neural Networks, arXiv:1804.07612v1
From the abstract,

While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance ...
The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.


Why is a correlation matrix symmetric? I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix.

          Q1        Q2        Q3
Q1  1.000000  0.707568  0.014746
Q2  0.707568  1.000000 -0.039130
Q3  0.014746 -0.039130  1.000000

Why is it symmetric? Why is Q1:Q2, the same as Q2:Q1? Shouldn't they be inverses of each other? How do I read this and understand the relation?

Intuitively, the correlation matrix is symmetric because every variable pair has to have the same relationship (correlation) whether their correlation is in the upper right or lower left triangle. It doesn’t make sense to say the correlation between variables $X_1$ and $X_2$ is $\rho$, but the correlation between $X_2$ and $X_1$ is $\rho’\neq \rho$ if calculating a Pearson correlation (so correlation is symmetric).

Mathematically, correlation between two variables, $X$ and $Y$, is commutative: $Corr(X,Y)=Corr(Y,X)$.

In OP’s case, the correlation between Q1 and Q2 is the same calculation and therefore the same result as the correlation between Q2 and Q1. Therefore the correlation matrix will be symmetric.

There are more mathematical reasons and proofs why a correlation matrix of real valued variables has to be symmetric and positive semi-definite, but I’ve excluded them from this answer.


How is the linear regression cost function evolved? A couple of weeks ago I joined the Standford University machine learning course on Coursera. In that course, they directly gave the cost function formula without telling how this formula was evolved. Can anyone help me by telling how that cost function has been evolved?

What I remember is that they give you more insight in future lectures, but the main reason for that is the maximum likelihood with which you try to increase the chance of making the current data set by setting the parameters. It is a good choice for setting the parameters, but its weakness is that it may overfit to your training data. You may want to take a look at Maximum Likelihood Estimation.

The cost function which is discussed there is called minimum square errors. It is found by maximum likelihood. This means that you want to increase the chance of making your training set. In other words, you want to increase the chance of $P(D|\theta)$ where $D$ can be considered as your training set. Due to the fact that your data should be iid, you can write the previous probability as $\pi p(x_i|\theta)$. You then apply some simplifications and you finally find that cost function. You can take a look at MSE as Maximum Likelihood for exact justification.


Predicting outcome of MOBA team games MOBA team games have teams composed of a subset of 5 heroes from a larger set of possible heroes (say 100 heroes in the larger set)
For example, a game can be between a team with heroes 1,8,43,65 and 71 and a team with heroes 3,7,23,41 and 45.
What is the best way to train a model that predicts the outcome based on team compositions?
For example, one option would be to have something like this:
|Label  | Hero1 | Hero2 | Hero3 | Hero4 | Hero5 | Hero6 | Hero7 | Hero8 | Hero9 | Hero10 |
|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|
| 1     | 1     | 8     | 43    | 65    | 71    | 3     | 7     | 23    | 41    | 45     |

While another option could be instead of having 10 numeric columns for the hero selection, to have 200 boolean columns (100 for team A and 100 for team B) which would have a value of true if that hero was included in that team.
Which would be a better option? Or is there an even better option outside of those two?
Thanks!

If the order of the heroes doesn't matter, I would go for the boolean columns.
The reasoning is the following: We have two possible cases: we always order the player hero ids in a team, or we don't necessarily do that.

If we order hero ids: let's say a team has players 1,2,3,4,5 and another team has players 2,3,4,5,10. The teams are very similar (we've only changed one player), but the features are completely different. In machine learning we want similar entities to have similar representations in terms of features.
If we don't order hero ids: then, it's even worse, as the team 1,2,3,4,5 has the completely different features from 5,1,2,3,4, and they are the same team.

If we do the boolean encoding: if we reorder the heroes we have the same representation, and if we change only a player only a feature of the 200 changes.


Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions? I am just beginning to get into data science and have never posted here before, apologies if this question is worded incorrectly! I am curious if there is an unsupervised machine learning algorithm that can take data with more than 2 dimensions and cluster the data similar to how k-means clustering works with 2 dimensions. Ultimately, I would like to take financial data (e.g. P/E, operating margin, earnings growth) and divide different stocks into groups (e.g. growth stocks vs. value stocks).

I have tried googling to see if k-means clustering would work with a dataset that has more than just X and Y values but couldn't find anything super helpful. Also, I thought about using k-nearest neighbor but I don't have a large amount of "training" data so I think an unsupervised algorithm would work the best. Any and all help is greatly appreciated!

Ross Leavitt

k-means works with more than two dimensions (e.g., the sklearn implementation of k-means does not make any assumptions on the dimensionality of the data). Assuming that k-means is the "appropriate" way to deal with your data, you may consider normalizing your columns to have unit variance first.


Why running the same code on the same data gives a different result every time? I am using Keras in Jupyter Notebook.

I understood that for the same results, the random numbers should be produced from the same seed each time.

So, in the first of all my codes, I set random.seed as 1234 in a cell.

np.random.seed(1234)

Then other cells are the code for my model and the fit and evaluate code. But each time that I run the model cells, the loss values are different!

Why does it happen? How can I solve it?

It's not quite enough to set only the numpy random seed, as you've seen - the Keras documentation also notes that it's necessary to set:

the python hash seed
the core python rng
the random seed of your backend to keras
and force your backend to use a single thread.

The interaction of all of these will generally result in different loss values due mainly to changes in random weight initialisation, which data ends up in your train/val/test splits, and the order that data is passed to your model for training.

As others have mentioned, a small amount of variance is to be expected and it's certainly not worth hamstringing your performance over (by limiting yourself to a single thread); setting the seeds and rng should be enough to satisfy people your results are reproducible.

If your results still have unacceptable variance after taking those steps then it might be an indication that your model is brittle and may not generalise well to new data so you'll want to address that.

Final thought - when you say the values are different each time you run the model cell, are you deleting/overwriting the model you already trained? If you are running the model cell on existing model/weight values then that's the same as training for more epochs and would usually have a large impact on your results.


sklearn random forest and fitting with continuous features Does anyone know how the python sklearn random forest implementation handles continuous variables in the fitting process? I'm curious to know if it does any sort of binning (and if so, how it does the binning), or if a continuous variable is just treated as a categorical variable? I'm hoping it's not the latter...thanks! Also, I'd be open to using some R implementation if anyone knows about that.

To understand how a random forest treats continuous data it is imperative to understand how a random forest works. At the base of the random forest algorithm lays a tree construction. The default in sklearn is to split a tree based on the Gini coefficient (see sklearn documentation). This type of tree algorithm is referred to as CART trees. You can change the criterion to entropy to select ID3 and C4.5 trees. Without going to deep into the maths, the tree algorithm will seek to split the tree based on a cutoff that leads to the lowest Gini coefficient.

The random forest algorithm will build a large number of deep trees on your data and average over all the trained trees to give you the final prediction.

Depending on your requirements in terms of data size and necessity for parallelization I can highly recommend H2O. It is an open source machine learning software suite with APIs in Python and R. Their random forest implementation is very fast and leads to models with a higher AUC (see this page for a good comparison between different ML libraries).


How do you calculate how dense or sparse a dataset is? I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf

In order to select which CF algorithm should be used the paper refers to the density of the dataset. What it doesn't do is explain how you actually calculate the density of your dataset.

So in the context of that above paper can anyone help explain to me how I would calculate the density of a dataset? The paper refers regularly density in the 1-5% range.

It's actually defined on the first page:

  ... sparsity level (ratio of observed to total ratings) ...

In other words, the fraction of the user/item rating matrix that is not empty. Remember that the problem is that most user-item pairs have no rating, and we wish to estimate them.

Example:

Let there be three users and four products. The number of possible ratings is $3\times4 = 12$. If every user rates only one product each (regardless of which product), the density is 3/12 = 25%.


Publicly available social network datasets/APIs As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:

the name of the social network;
what kind of user information it provides (posts, profile, friendship network, ...);
whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);
whether it simply provides a snapshot of the whole dataset.

Any suggestions and further characteristics to be added are very welcome.

A couple of words about social networks APIs. About a year ago I wrote a review of popular social networks’ APIs for researchers. Unfortunately, it is in Russian. Here is a summary:

Twitter (https://dev.twitter.com/docs/api/1.1)

almost all data about tweets/texts and users is available;
lack of sociodemographic data;
great streaming API: useful for real time text processing;
a lot of wrappers for programing languages;
getting network structure (connections) is possible, but time-expensive (1 request per 1 minute).

Facebook (https://developers.facebook.com/docs/reference/api/)

rate limits: about 1 request per second;
well documented, sandbox present;
FQL (SQL-like) and «regular Rest» Graph API;
friendship data and sociodemographic features present;
a lot of data is beyond event horizon: only friends' and friends' of friends data is more or less complete, almost nothing could be investigated about random user;
some strange API bugs, and looks like nobody cares about it (e.g., some features available through FQL, but not through Graph API synonym).

Instagram (http://instagram.com/developer/)

rate limits: 5000 requests per hour;
real-time API (like Streaming API for Twitter, but with photos) - connection to it is a little bit tricky: callbacks are used;
lack of sociodemographic data;
photos, filters data available;
unexpected imperfections (e.g., it’s possible to collect only 150 comments to post/photo).

Foursquare (https://developer.foursquare.com/overview/)

rate limits: 5000 requests per hour;
kingdom of geosocial data :)
quite closed from researches because of privacy issues. To collect checkins data one need to build composite parser working with 4sq, bit.ly, and twitter APIs at once;
again: lack of sociodemographic data.

Google+ (https://developers.google.com/+/api/latest/)

about 5 requests per second (try to verify);
main methods: activities and people;
like on Facebook, a lot of personal data for random user is hidden;
lack of user connections data.

And out-of-competition: I reviewed social networks for Russian readers, and #1 network here is vk.com. It’s translated to many languages, but popular only in Russia and other CIS countries. API docs link: http://vk.com/dev/. And from my point of view, it’s the best choice for homebrew social media research. At least, in Russia. That’s why:

rate limits: 3 requests per second;
public text and media data available;
sociodemographic data available: for random user availability level is about 60-70%;
connections between users are also available: almost all friendships data for random user is available;
some special methods: e.g., there is a method to get online/offline status for exact user in realtime, and one could build schedule for his audience.


How to get predicted class labels in convolution neural network? I have built a convolutional neural network which is needed to classify the test data into either 0 or 1. I am training the CNN with labels either 0 or 1 but while running the below code I am getting the following result.

predictions = classifier.predict(x_test)

print(predictions)

[0.0128037 ]
 [0.01182843]
 [0.01042355]
 [0.00906552]
 [0.00820154]
 [0.00726516]......

logloss_score = log_loss(y_test, predictions)
print(logloss_score)
0.047878393431377855

How do I get the results between 0 and 1? What do I need to modify in the above code?

What you have are predicted class probabilities. Since you are doing binary classification, each output is the probability of the first class for that test example.

To convert these to class labels you can take a threshold:

import numpy as np

probas = np.array([[0.4],[0.7],[0.2]])
labels = (probas < 0.5).astype(np.int)
print(labels)
[[1]
 [0]
 [1]]

For multiclass classification  where you want to assign one class from multiple possibilities you can use argmax:

probas = np.array([[0.4, 0.1, 0.5],[0.7, 0.2, 0.1],[0.3, 0.4, 0.3]])
labels = np.argmax(probas, axis=-1)
print(labels)
[2 0 1]

And to get these as one-hot encoded arrays you can use LabelBinarizer:

from sklearn import preprocessing

lb = preprocessing.LabelBinarizer()
lb.fit_transform(labels)
array([[0, 0, 1],
       [1, 0, 0],
       [0, 1, 0]])

And for multilabel classification where you can have multiple output classes per example you can use thresholding again:

probas = np.array([[0.6, 0.1, 0.7],[0.7, 0.2, 0.1],[0.8, 0.9, 0.6]])
labels = (probas > 0.5).astype(np.int)
print(labels)
[[1 0 1]
 [1 0 0]
 [1 1 1]]

Some packages provide separate methods for getting probabilities and labels, so there is no need to do this manually, but it looks like you are using Keras which only gives you probabilities.

As a sidenote, this is not called "normalization" for neural networks. Normalization typically describes scaling your input data to fit in a nice range like [-1,1].


What's a difference between the neoperceptron and CNN? What's a difference (in terms of architecture) between the neoperceptron and CNN?

Both ANNs have hidden layers and scanners, as I understood, but many sources subdivide them in two classes.

According to the research paper, neoperceptrons are a class of CNN that are not sensitive to rotations.

One of the issues with traditional kernels (that was the case before CNN and it is still true with them) is that the rotation of the input image would lead to different results, because the neurons in the dense layer would have different levels of activations.

With these new neurons, you don't get an issue with orientation. So in theory, if you have a gradient in your image, no matter what the orientation is, you would get the same value.

For a traditional CNN, you would get maximum activation with the original orientation, inverse with a 180° rotated image, and no activation with 90° or 270°.


Genetic algorithms(GAs): to be considered only as optimization algorithms? Are GAs used in machine learning any way? As a quick question, what are genetic algorithms meant to be used for? I read somewhere else that they should be used as optimization algorithms (similar to the way we use gradient descent to optimize the best parameters search with a linear regression, neural network...).
If so, why are these GAs not so much present in machine learning (or at least I did not see it too much in the literature)?

Yes, as you found they (evolutionary algorithms such as genetic algorithms) are using for optimization tasks. One reason that they are not using mostly in machine learning could be their performance to converge to the optimum point. Also, implementation of the GAs for some domains could be problematic and it could not be generalized like gradient descent, as it's involved in at least 5 phases (mutation, crossover,  ...).


Conversion of Interval data to ordinal data I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both.
Lets say the three participants finished the race in 45 seconds,35 seconds and 30 seconds, this is a interval data since there is fixed order and 0 has no meaning and if we convert it to ordinal 30s becomes first 35s second and 45 second third,according to this option a perfectly fits. What am i missing here?

In my opinion etc. is playing key role here. Ordinal data must have the fixed length, which is not in this case as we can go on even further as 1st,2nd,3rd,4th,.. and so on. There is no end to this. You can argue that there must be fixed number of runners(say 50) and hence we have only 1-50 rank. But it is not mentioned in question hence this assumption is not valid.

If no of atheletes participating were mentioned then you're right.


Data enrichment of geographical records I have a user_data table with various fields, some of them are based on geography.

I'd like to enrich the data with additional columns, like expected_income_in_region, city_population, life_expectancy_in_state, etc...  for each user record.

I'd like to use these extended fields with my sql query (prior to additional python manipulations)

Is the standard way of getting hold of this data , or do I need to scrape from the US Census bureau and format it myself ?

If you are looking to use US Census data, the American FactFinder website: https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml allows you to search for Census data in different ways and then download tables in csv format.

For example, on the Community Facts site, you can enter a state, county, city, town, or zip code and obtain the population of the geography that you entered.  Then you can click on the subjects listed on the left side of the page (Age, Business and Industry, Education, etc.) to see other interesting information about your geography, or click the links under "Popular tables for this Geography" to see tables containing other data for your community.

Or you can use the Advanced Search feature to search all available data using all of the search features provided by the site.

Also, I don't know much about this myself, but there exists at least this one Github repository with "SQL scripts and other supporting files for importing recent American Community Survey releases from the US Census Bureau datasets".  The scripts are released under the GNU General Public License.


Same SVM configuration, same input data gives different output using Matlab and scikit-learn implementation of SVM, in a classification problem I have a classification problem with 60 data points in a 2-dimensional feature space. The data originally is divided into 2 classes. Earlier I was using Statistics Toolbox of Matlab so it was giving me fairly good results.It was giving 1 false negative and no false positive.

I used the following code:

SVMstruct = svmtrain(point(1:60,:),T(1:60),'Kernel_Function','polynomial','polyorder',11,'Showplot',true);

I am using a polynomial kernel and with polynomial order of 11.

But when I use same kernel configuration in scikit-learn SVC it does not gives the same result rather it gives very undesirable result with classifying all of them to single class.

I am using it as

svc = svm.SVC(kernel='poly', degree=11, C=10)

I have used with many values of C too. No major difference.

Why there is so much difference in results ?
How can I get same result as I got using Matlab ?
For me it is compulsory to do with python-scikit.

You have to be sure that the algorithm is the same and the kernel functions are really the same.
If you look at this python documentation page for kernels in scikit learn you will see there a description of poly kernel.

Notice that you have a gamma and a degree. Gamma is by default 'auto' which is evaluated at 1/n_samples. For the same kernel you have 'coef0' (a great name for a parameter), which is used in poly as the free term. I do not know how matlab put this values as defaults, but the usual formula for poly kernel in literature I found it to be $poly(x_1, x_2) = (<x_1,x_2> + 1)^{d}$. So no gamma and the free term is $1$. I think matlab uses that. (Anyway I found the 'improvements' in scikit learn to have a not so good smell).

Also in this SVC documentation page they state that there is a parameter called shrinking. I really do not know it's effect, but its auto, which means is enabled. Might be an issue.

Later edit:

I found this documentation page for svm in matlab which describes the kernel in the way I stated (no degree, free term with $1$). Also it states that 'SMO' is used by default, make it sure you use 'SMO' in python also.

On the other hand you have to understand that these kind of algorithms are solved by optimization methods which are usually iterative and to save some memory, or cycles their implementations can be different in small details, which will almost produce different results. I agree however that the results should be similar.