Parameter Adjustment based only on tagged predictions not sure that this is the best place to post this but if not, please let me know if there is a better stack community.

I have an anomaly detection method which has some parameters. I have some data that I need to find anomalies for. This data is not tagged. I use the anomaly detection method on the data using some default parameters. It predicts some anomalies. The user looks ONLY AT PREDICTED ANOMALIES and determines if he or she agrees with them or not (is the predicted anomaly truly an anomaly?) Now I need to adjust the parameters of the anomaly detection method so it can do better next time whether that is on the same data set or a new data set.

More specifically:

1) system predicts some anomalies

2) user looks at those predicted anomalies and says if they agree or disagree with them

3) system looks at those predicted anomalies and whether or not the user agrees with them and makes adjustments to the parameters based on this information.

The ``making adjustments" part is what I am looking for ideas about. As for why I am not listing a specific method, I was hoping for more generic ideas.

How do I go about this (the parameter adjustment) for either situation (same or new data set)?

This is more a comment than an answer but the adjustment part is likely to depend on the model you have.
So it will be difficult to give you a general method. If your model is a simple rule base one, the adjustment could consist in adding another rule, if it is a neural network then you'll go with backpropagation to adjust the weights of the nodes, if it's a Bayesian tree then you could grow another tree taking into account your new outliers, if it's Bob, your dear colleague, you'll have to make sure that he takes the new info into account next time he classifies something as being an outlier.


Production: TensorFlow and Keras I always here about TensorFlow is good because it is used for deploying and production. Does that mean that people don't use Keras for deploying models? If keras is now integrated into TensorFlow, does that mean that it can also be used for deployment and production?

Once model has been trained with Keras

It can be exported to TensorFlow model OR
With tf.keras it can be served as http service

Examples :

https://towardsdatascience.com/deploying-keras-models-using-tensorflow-serving-and-flask-508ba00f1037

https://medium.com/tensorflow/training-and-serving-ml-models-with-tf-keras-fd975cc0fa27

https://medium.com/@mr.acle/exporting-deep-learning-models-from-keras-to-tensorflow-serving-7d4a6e49ce3


Adjusting imbalance in classification problem reduce precision, accuracy but increase recall I've learned that adjusting imbalanced data when training a CNN affects model performance which got me thinking "what about in ML?" so I've done some testing on my own, you can check it out here -> https://haneulkim.medium.com/handling-imbalanced-class-in-machine-learning-classifiers-1b5c528f427f
I've ended up concluding that as we balance our data using random oversampling our precision and accuracy decrease. I want to know why this is the case...
If it requires me to do more research, link to resource would be greatly appreciated. Thanks!

Intuitively the problem of imbalanced data can be understood like this: if a classifier is not really sure how to classify an instance but it knows that most instances belong to class X, then whenever there's a doubt predicting class X is always the best decision. As a consequence, the classifier unavoidably assigns class X too often since all the "unsure" cases end up being labeled as the majority class X.
So first it's important to understand that resampling is not automatically the "cure" for imbalanced data:

Resampling doesn't provide the classifier with more information, it just presents it in a different way in order to force the classifier to pay more attention to the minority class.
In case the data is easy to separate, the classifier can do a perfectly good job without resampling. This means that whenever possible it's better to improve the features rather than using resampling, because that's what can actually help the classifier doing its job.

This being said, resampling is a useful technique in some cases. Assuming the standard choice of the minority class as the positive class, resampling will only increase recall at the expense of precision: as said above, the difference happens when the classifier cannot easily predict the class for an instance. In such a case it has two choices:

Assigning the negative majority class: more likely to be correct (True Negative), a small risk of False Negative error.
Assigning the positive minority class: more likely to be incorrect (False Positive), a small chance to be correct (True Positive)

Without resampling the classifier favors the first option, so it has few FP errors but quite a lot of FN errors. Therefore it can have quite high precision but low recall.
With resampling the classes are equal, so the classifier stops favoring the first option. Therefore it makes less FN errors but more FP errrors, which means that it increases recall at the expense of precision.


Python - Create many dummy variables from one text variable? I'm trying to create dummy variables for a variable that has text data in rows.

Data in 1st row is:
{"Wireless Internet","Air conditioning",Kitchen,Heating,"Family/kid friendly",Essentials,"Hair dryer",Iron,"translation missing: en.hosting_amenity_50"}

and Data in 2nd row is:
{TV,"Cable TV",Internet,"Wireless Internet",Kitchen,"Indoor fireplace","Buzzer/wireless intercom",Heating,Washer,Dryer,"Smoke detector","Carbon monoxide detector","First aid kit","Fire extinguisher",Essentials} and many more.

What I now want to do is, to create dummy variables out of that variable. For example from the above data:
one variable named Wireless Internet with 0 ans 1 in rows &
another variable named Cable TV with 0 and 1 in rows &
another variable named Kitchen with 0 and 1 in rows and so on.

sklearn for python has OneHotEncoder class which creates dummy variable named everything in a row considering all rows with unique values. That is not what I want to do here. I first have to split text in all rows and create dummy variables for them. How do I do that?

Expected results are, multiple columns like
Wireless Internet    Cable TV      Kitchen
1                     0             1
0                     1             1
1                     0             1

link to data(column named amenities) - https://www.kaggle.com/stevezhenghp/airbnb-price-prediction

On the kaggle page, there is a kernel available which focuses on exactly your problem. This is the code from user JAGADEESHWARA VARA PRASAD:

# load your dataset into df
df = pd.read_csv("../input/train.csv")

# trimm { } symbols and split at ',' and trimm " from each word
l = [[word.strip('[" ]') for word in row[1:-1].split(',')]
     for row in list(df['amenities'])]

# form a set of distinct text
cols = set(word for row in l for word in row)
cols.remove('')

# create and fill new data frame
new_df = pd.DataFrame(columns=cols)
for row_idx in range(len(l)):
    for col in cols:
        new_df.loc[row_idx,col]=int(col in l[row_idx])

print(new_df)


Facial recognition on offline images I want to do facial recognition on wide varieties of images captured at various ages of my family members. Below are some of the questions I have.

If a person uses glasses of different types, do I need to feed images with each glasses type in the training dataset ? What if the eyes are not visible after wearing sunglasses. ?
There are many pics which ranges from childhood to current age of approx 30. Do I need to train my model with pics of various ages to get good accuracy ?
Do I need to align the images of train dataset for better accuracy ?
Do I need to crop the background of the images in the training dataset just to keep the faces OR background doesn't matter ?

The idea behind my requirement is to scan through all the images on my hard disk (around 100 GB images) and classify the images in different folders for each person. I know similar thing is done on iphone and other mobile phones but I wanted to do it for the offline images stored on my external hard disk.

I was exploring facenet model from google, opencv face recognition. But stuck in the first step of dataset creation.

Glasses matter. So you will need those photos in the training dataset.
Yes. You will need to feed images of all ages for multiple people. This will help the model generalize how people age
Alignment should not matter
Background should not matter


Feasibility study of machine learning How to know whether machine learning is possible for a given data set. I have been given a data set, I should check whether machine learning is possible or not for that data set. How can I do that. How do you come to conclusion that machine learning can be performed for the given data set or not.

For this feasibility study, following will be high level steps :

For each feature perform PCA with rest of the features as train_x and feature as train_y. If you find a feature that can be predicted by other features; ML can be applied on Dataset
Can a Human solve it ? As a person; can you find patterns for a given feature, based on other features ?
Exploratory data analysis with Weka, Dataframe + Matplotlib or similar tools . https://datascienceguide.github.io/exploratory-data-analysis


Forecasting ticket sales and city I am learning data science. I have the following dataset for train tickets:

1. order_date_meduim

order,date,medium
95062,2017-09-11,35
171081,2017-07-05,39
122867,2017-08-18,39
107186,2017-11-23,
171085,2017-09-02,

2. order_ordercityA_ordercityB [some order has only 1 ordercity, I think ordercity means here which city is something like source and destination]

order,ordercityA,ordercityB
81773,4,11
105838,4,
76153,24,18
93058,12,
11623,24,3
3070,24,3

3. order_ticketcount,ticketclass

order,ticketcount,ticketclass
246783,1,pax
1693998,2,pax
1958576,1,other
673681,1,pax
1593899,1,pax
194035,1,pax

I need to forecast the ticket sales for a week and also the ordercity with medium of booking.

As I am new, could someone give a possible answer about how to create a prediction model that could predict the sales for 1 week? Also, I doubt the data is time-series data.

I code in Python.

You have got yourself a time series forecasting problem. And with multiple input variables it is called multivariate time series forecasting.

What Is Time Series Forecasting?

You can start with EDA on your data and find out if you can see any trend or seasonality. ( You might need to add or update your current features to get underlying trend/seasonality )

After EDA, you can start looking into following models, all of them are the go-to for time series prediction problem:

Classical, Statistical

ARMA for stationary data
ARIMA for data with a trend - Refer
SARIMA for data with seasonality
Holt-Winters Forecasting - Refer
Theta method - Refer
Fourier Transformation - Refer

Machine Learning

Quantile Regression Forest(QRF)
Support Vector Regression(SVR)
Recurrent Neural Networks(RNNs) (LSTM)

If you are not comfortable with Statistics then I would advise you to start with LSTMs for forecasting - Refer


Finding unpredictability or uncertainty in a time series I am interested in finding a statistic that tracks the unpredictability of a time series. For simplicity sake, assume that each value in the time series is either 1 or 0. So for example, the following two time series are entirely predictable
TS1: 1 1 1 1 1 1 1 1
TS2: 0 1 0 1 0 1 0 1 0 1 0 1

However, the following time series is not that predictable:
TS3: 1 1 0 1 0 0 1 0 0 0 0 0 1 1 0 1 1 1

I am looking for a statistic that given a time series, would return a number between 0 and 1 with 0 indicating that the series is completely predictable and 1 indicating the series in completely unpredictable.

I looked at some entropy measures like Kolmogorov Complexity and Shannon entropy, but neither seem to fit my requirement. In Kolmogorov complexity, the statistic value changes depending on the length of the time series (as in "1 0 1 0 1" and "1 0 1 0" have different complexities, so its not possible to compare predictability of two time series with differing number of observations). In Shannon entropy, the order of observations didn't seem to matter.

Any pointers on what would be a good statistic for my requirement?

Since you have looked at Kolmogorov-Smirnov and Shannon entropy measures, I would like to suggest some other hopefully relevant options. First of all, you could take a look at the so-called approximate entropy $ApEn$. Other potential statistics include block entropy, T-complexity (T-entropy) as well as Tsallis entropy: http://members.noa.gr/anastasi/papers/B29.pdf

In addition to the above-mentioned potential measures, I would like to suggest to have a look at available statistics in Bayesian inference-based model of stochastic volatility in time series, implemented in R package stochvol: http://cran.r-project.org/web/packages/stochvol (see detailed vignette). Such statistics of uncertainty include overall level of volatility $\mu$, persistence $\phi$ and volatility of volatility $\sigma$: http://simpsonm.public.iastate.edu/BlogPosts/btcvol/KastnerFruwhirthSchnatterASISstochvol.pdf. A comprehensive example of using stochastic volatility model approach and stochvol package can be found in the excellent blog post "Exactly how volatile is bitcoin?" by Matt Simpson.


What is the most efficient method for hyperparameter optimization in scikit-learn? An overview of the hyperparameter optimization process in scikit-learn is here.

Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid search is slow.

Random search is faster than grid search but has unnecessarily high variance.

There are also additional strategies in other packages, including scikit-optimize, auto-sklearn, and scikit-hyperband.

What is the most efficient (finding reasonably performant parameters quickly) method for hyperparameter optimization in scikit-learn?

Ideally, I would like working code examples with benchmarks.

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the underlying space within that black box. Based on a variety of sampling techniques, they probe the hyper-parameter space and attempt to build a function which represents the true underlying hyper-parameter space.

Bayesian optimization focuses on the surrogate model and how this model is constructed is crucial to BO. Also crucial to BO is choosing a good loss function.

I think performance between random search and Bayesian search varies from dataset to dataset, and model to model. Bergstra & Bengio (2012) made a strong argument for random search over grid search. Shahriari et al. (2016) make a strong case for BO. Model-based Hyperband strategies can potentially perform better than BO, especially for high-dimensions, however it is purely exploration, not exploitation. This can easily result in too early of stopping. However, there have been efforts to combine Hyperband and BO.

I've had good success scikit-optimize, despite there being quite a bit unimplemented. It's easy to prototype with and can easily interface with scikit-learn.

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb), 281-305.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & De Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.


How does binary cross entropy work? Let's say I'm trying to classify some data with logistic regression.

Before passing the summed data to the logistic function (normalized in range $[0,1]$), weights must be optimized for desirable outcome. In order to find optimal weights for classification purposes, relatively minimizable error function must be found, this can be cross entropy.

From my knowledge, cross entropy measures quantification between two probability distributions by bit difference between set of same events belonging to two probability distributions.

For some reason, cross entropy is equivalent to negative log likelihood. Cross entropy loss function definition between two probability distributions $p$ and $q$ is:

$$H(p, q)=-\sum_{x}p(x)\,log_e(q(x))$$

From my knowledge again, If we are expecting binary outcome from our function, it would be optimal to perform cross entropy loss calculation on Bernoulli random variables.

By definition probability mass function $g$ of Bernoulli distribution, over possible outcome $x$ is:

$$g(x|p)=p^{x}(1-p)^{1-x} \ \textrm{for} \ x\in [0, 1]$$

Which means that probability is $1-p$ if $x=0$ and $p$ if $x=1$.

Bernoulli probability distribution is based on binary outcome and therefore process of cross entropy being performed on Bernoulli random variables is called binary cross entropy:

$$\mathcal{L}(\theta)= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i)\log(1-p_i) \right]$$

Is this true? why are negative logarithm likelihoods associated with cross entropy? why does Bernoulli random variable perform so well?

In short, how does binary cross entropy work?

When doing logistic regression you start calculating a bunch of probabilities $p_i$ and your target is maximize the product of those probabilities (as they're considered independent events). The higher the result of the product the better is your model.
As we are dealing with probabilities we are multiplying numbers between 0 and 1, therefore, if you multiply a lot of those numbers you would get smaller and smaller results. So we need a way to move from probabilities multiplication to a sum of other numbers.
Then is when $ln$ function enters in to play. We can use some of this function properties such as:

$ln(a b) = ln(a) + ln(b)$.
When our prediction is perfect i.e. 1, the $ln(1) = 0$.
$ln$ lower than 0 are growing negative numbers e.g. $ln(0.9) = -0.1$ and $ln(0.5) = -0.69$.

So we can move from maximizing the multiplication of probabilities to minimizing the sum of the $-ln$ of those probabilities. The resulting cross-entropy formula is then:

$$
- \sum_{i=1}^m y_i ln(p_i) + (1-y_i) log (1-p_i)
$$

If $y_i$ is 1 the second term of the sum is 0, likewise, if $y_i$ is 0 then the first term goes away.
Intuitively cross entropy says the following, if I have a bunch of events and a bunch of probabilities, how likely is that those events happen taking into account those probabilities? If it is likely, then cross-entropy will be small, otherwise, it will be big.


How does time-series classification work? So I have the multiple time series data in this format --

id  Week    X1    X2   X3   y

1   201901  10.   34.9 52.8 0

1   201902  20.   44.1 12.6 1

1   201903  30.   54.2 84.3 1

1   201904  40.   64.4 77.1 0

2   201901  10.   64.9 62.8 1

2   201902  20.   54.1 22.6 1

2   201903  30.   34.2 54.3 1

2   201904  40.   44.4 17.1 0

I want to predict y for the next two weeks. How can I approach this problem?

One way I thought is to train on the data set and then the test data would look like this -

id  Week   X1   X2    X3
1   201905 20.  45.2. 33.2
2   201905 10.  65.2. 23.2

Is this the correct way to do such a classification? Or should I train the model for each time series independently.

Thanks!

To be clear, here, time-series classification refers to forecasting discrete values. In another context, time-series classification could refer to predicting a single class for the entire time-series (i.e. heart disease vs healthy heart). Thanks @mloning for pointing this out.

When it comes to forecasting discrete outputs, models are trained to predict the next value based on the previous ones, which means that

The input is the historical data up to timestamp t (in your scenario, the data up to week = 201904)
The output is the value at timestamp t + 1 (y when week = 201905)

If you want to predict more than 1 value into the future, you should perform predictions in a recurrent way, i.e.:

use data up to t to predict t + 1
add your prediction to your data
use data up to t + 1 (where t + 1 is your own prediction) to predict t + 2
and so on

How far into the future you want to look is called the horizon. And you are free to use as big of a horizon as you like. Of course, since every new step into the future is based on guesses and not actual data, it is expected that the further you look, the worse your predictions will become. Which makes perfect sense. That's why it's easier to predict what the weather will be like tomorrow than in 7 days.

Splitting data into training and testing is not very different than for a normal classification problem. The only constraint is that your test data should only contain data "in the future" compared to the training data. Here is a good blog about it: https://towardsdatascience.com/time-based-cross-validation-d259b13d42b8

should I train the model for each time series independently?

Short answer: try both approaches and see which one works best

Long answer:

All of this depends on whether you believe that each time-series represent a different "process" or "pattern". It could be that a single model could make good predictions for either time-series. It could also be that both time-series represent different processes and therefore, it might be wiser to train different models for each.

Think about it like a weather forecast. I could have time-series data for a bunch of cities. I could train a model specific to each city, but it might also help to train a model using data from all cities so that it can capture generic weather patterns. This can help if all of a sudden, I need to predict the weather for a new city I didn't know about. Since I didn't know about it, I don't have a model specifically for it, so I would need to reuse one, which ideally should be general enough.


Sentiment analysis of the target in articles The goal is as follows: I have a big article and I want to define the sentiment of the particular word. For example, the article describes pros and cons of bikes and cars and I want to find the sentiment of the word car.
In such an example I cannot use document-level SA as the article itself can be positive while the car was mentioned in a negative way.
So, I studied papers related to aspect-based sentiment analysis, but my constait is absence of data for training NNs. Hence, I concentrated on the approaches that basically do not involve training process. One of my attemts was to build sentiment analysis tool using word2vec and K-Means so that each cluster corresponds to one of three sentiments (pos, neg and neu).
It actually worked great but I found that for some reason one word can be at two clusters at the same time. Plus it generally goes not give sentiment for specific keyword but for all aspects found in the text.
Another problem is that basically cannot test the correctness of the output if only not to read the text by myself and check whether the keyword belonged to the correct cluster or not.
So I came to the decision to make summarization of the article first and then applying sentiment analysis (like sentiwordnet or similar).
Question 1
Are there ways to improve word2vec+KMeans approach? Is it even worse improving?
Question 2
Is it a good idea to go through text summarization before sentiment analysis?
Question 2
Is there better way to find sentiment of the particular word without training process (due to no training data and small amount of unlabeled data)?

I'm not sure I completely get the idea, but it looks to me like what you're actually interested in is the sentiment of a word in a particular context: a content word like "car" might not carry a stable sentiment by itself, but its usage in a specific context might.
So I'd suggest a method like this: for any target word you extract either the sentence or a context window, i.e. N words on the left and N words on the right of the target word. Then you could use predefined sentiment analysis tools to extract a sentiment value for this instance. From there you could:

measure the mean sentiment for a word by averaging over the instances
compare the distribution of sentiment or average sentiment for two different words


Given a machine learning algorithm, what is the minimum size of the training set for it? I understand that the more data we have, the more reliable is our model trained on that data. I also understand that the more parameters a machine learning model has, the more training data it requires. For example, deep neural networks need more data than linear regression or support vector machines.

Successful application of machine learning to texts and images requires millions of data points. I read this attempt (link) with a data set of 5000 quotes of thinkers to generate new quotes. The author had several failed attempts until he understood that his data set is too small for the task. He succeeded only by using transfer learning employing an AWD-LSTM model developed by Salesforce trained on 100 million tokens from Wikipedia.

On the other hand, I often see scientific articles that train machine learning models on just a few hundreds of data (or even less) and boast about their "great discoveries". For example, this paper (link) uses a data set of only 46 samples for which they use decision trees. This situation exists because it is an extremely rare in science that one can make millions of experiments with millions of materials to get large training data sets. Yet, machine learning is fashionable, so many research groups apply it to their tiny datasets just to get published. How reliable are their predictions given the size of their data sets?

Is there a formula, or an algorithm where I can insert the number of my training data, the number of parameters of my model, and maybe few other numbers that distinguish between the models (e.g. feed-forward neural networks, SVM, kernel ridge regression, decision trees) and this formula show that my training data size is sufficient or insufficient for a chosen method?

Is there statistical research that shows quantitatively that a certain machine learning model requires at least this amount of data?

You can have a model with a single training example.  The real question should be, how good is your model with only a single training example. The answer? A learning algorithm $h(\theta)$ is an approximation of the actual relationship between your features and your target. However, the performance on how good $h(\theta)$ relies on good data. Your model is only as good as your data. If your data shows clear patterns, you can have a good model with a few data points. That's why Exploratory Data Analysis is essential.

I'd recommend watching Statistical Learning theory on youtube, specifically the lecture "Is learning feasible". The class is offered by Caltech.


Keras LSTM accuracy stuck at 50% I'm trying to train an LSTM for sentiment analysis on the IMDb review dataset.

As input to the word embedding layer, I transform each review to a list of indices (that corresponds to word index in the vocabulary set). I thought of converting the text into one-hot/count matrix, but I will end up with huge sparse matrix (should I worry about this?).

Here is how I am creating the network architecture:

model = Sequential()
model.add(Embedding(
    input_dim=vocab_size,
    output_dim=word_embed_vector_size,
    input_length=sentence_len_max)
         )
model.add(LSTM(units=1))
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', 'binary_accuracy'])
model.summary()

Here is the model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_2 (Embedding)      (None, 1422, 4)           201764
_________________________________________________________________
lstm_2 (LSTM)                (None, 1)                 24
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 2
=================================================================
Total params: 201,790
Trainable params: 201,790
Non-trainable params: 0
___________________________

Now when I try to train the model I see accuracy stuck at 50%

losses = model.fit(
    x                = term_idx_train,
    y                = y_train,
    epochs           = epochs,
    batch_size       = batch_size,
    validation_split = 0.01
)

Here is the epochs output:

Epoch 1/10
25000/25000 [==============================] - 1148s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 2/10
25000/25000 [==============================] - 1156s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 3/10
25000/25000 [==============================] - 1149s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 4/10
25000/25000 [==============================] - 1110s 44ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 5/10
16800/25000 [===================>..........] - ETA: 6:10 - loss: 7.9816 - acc: 0.4993 - binary_accuracy: 0.4993

Changing the activation function to a sigmoid and the LSTM blocks to 32 didn't help mush (with 1 epoch):

Train on 24750 samples, validate on 250 samples
Epoch 1/1
24750/24750 [==============================] - 1186s 48ms/step - loss: 0.6932 - acc: 0.5022 - binary_accuracy: 0.5022 - val_loss: 0.6951 - val_acc: 0.0000e+00 - val_binary_accuracy: 0.0000e+00

Epoch 00001: val_loss improved from inf to 0.69513, saving model to sentiment_model

Looking at what the LSTM is predicting, I see:

count   25000.000000
mean    0.499023
std 0.000013
min 0.499010
25% 0.499010
50% 0.499010
75% 0.499010
max 0.499443

Any idea why it's doing this? and how I could fix the issue?

activation = ‘softmax’ should be used for multiclass classification whereas ’sigmoid’ for binary classification.

You can refer to: http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/

If changing the activation function does not help, I will be around for alternative solutions.


Algorithm for generating classification rules So we have potential for a machine learning application that fits fairly neatly into the traditional problem domain solved by classifiers, i.e., we have a set of attributes describing an item and a "bucket" that they end up in. However, rather than create models of probabilities like in Naive Bayes or similar classifiers, we want our output to be a set of roughly human-readable rules that can be reviewed and modified by an end user.

Association rule learning looks like the family of algorithms that solves this type of problem, but these algorithms seem to focus on identifying common combinations of features and don't include the concept of a final bucket that those features might point to. For example, our data set looks something like this:

Item A { 4-door, small, steel } => { sedan }
Item B { 2-door, big,   steel } => { truck }
Item C { 2-door, small, steel } => { coupe }

I just want the rules that say "if it's big and a 2-door, it's a truck," not the rules that say "if it's a 4-door it's also small."

One workaround I can think of is to simply use association rule learning algorithms and ignore the rules that don't involve an end bucket, but that seems a bit hacky. Have I missed some family of algorithms out there? Or perhaps I'm approaching the problem incorrectly to begin with?

C45 made by Quinlan is able to produce rule for prediction. Check this Wikipedia page. I know that in Weka its name is J48. I have no idea which are implementations in R or Python. Anyway, from this kind of decision tree  you should be able to infer rules for prediction.

Later edit

Also you might be interested in algorithms for directly inferring rules for classification. RIPPER is one, which again in Weka it received a different name JRip. See the original paper for RIPPER: Fast Effective Rule Induction, W.W. Cohen 1995


SVM prediction time increase with number of test cases I am using scikit-learn's SVM for the MNIST digit classification dataset. In order to improve the performance I extended the dataset by adding rotated samples. I was aware that SVM takes O(N^3) time to train the data, where N is the number of training vectors.

However even prediction seems to take increase polynomially, the number of test vectors is the same. Is there any explanation for this or some equation that relates prediction time to the number of training samples?

I am using a 3rd degree polynomial as the kernel with C=100.0.

Note: I am doing a group project to compare the performance of various methods so I can't use any other method as my teammates would have used those. I referred to a paper by Decoste and Scholkoph which uses Virtual SVM. However I don't think I can run this on my current system if I can't run a simple extended training set.

The number of support vectors must be increasing. The prediction time is proportional to that; after all, the kernel classifier is $f(x) = \sum_i \alpha_i k(x, x_i)$, where the summation is over the support vectors. With sklearn you can find out how many you have by inspecting n_support_


Image Captioning in Keras I am trying to implement demo of Image Captioning system from Keras documentation. From the documentation I could understand training part.

max_caption_len = 16
vocab_size = 10000

# first, let's define an image model that
# will encode pictures into 128-dimensional vectors.
# it should be initialized with pre-trained weights.
image_model = VGG-16 CNN definition
image_model.load_weights('weight_file.h5')

# next, let's define a RNN model that encodes sequences of words
# into sequences of 128-dimensional word vectors.
language_model = Sequential()
language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
language_model.add(GRU(output_dim=128, return_sequences=True))
language_model.add(TimeDistributedDense(128))

# let's repeat the image vector to turn it into a sequence.
image_model.add(RepeatVector(max_caption_len))

# the output of both models will be tensors of shape (samples, max_caption_len, 128).
# let's concatenate these 2 vector sequences.
model = Merge([image_model, language_model], mode='concat', concat_axis=-1)
# let's encode this vector sequence into a single vector
model.add(GRU(256, 256, return_sequences=False))
# which will be used to compute a probability
# distribution over what the next word in the caption should be!
model.add(Dense(vocab_size))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)

But now I am confused in how to generate caption for test image. Input here is [image, partial_caption] pair, now for test image how to input partial caption?

This example trains an image and a partial caption to predict the next word in the caption.

Input: [, "<BEGIN> The cat sat on the"]
Output: "mat"

Notice the model doesn't predict the entire output of the caption only the next word. To construct a new caption, you would have to predict multiple times for each word.

Input: [, "<BEGIN>"] # predict "The"
Input: [, "<BEGIN> The"] # predict "cat"
Input: [, "<BEGIN> The cat"] # predict "sat"
...

To predict the entire sequence, I believe you need to use TimeDistributedDense for the output layer.

Input: [, "<BEGIN> The cat sat on the mat"]
Output: "The cat sat on the mat <END>"

See this issue: https://github.com/fchollet/keras/issues/1029


How to fight underfitting in a deep neural net When I started with artificial neural networks (NN) I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even beat my score on random forest!

I'm seeking some very general or not so general advice on what should one do to make a NN start capturing trends in data.

For implementing NN I use Theano Stacked Auto Encoder with the code from tutorial that works great (less than 5% error rate) for classifying the MNIST dataset. It is a multilayer perceptron, with softmax layer on top with each hidden later being pre-trained as autoencoder (fully described at tutorial, chapter 8). There are ~50 input features and ~10 output classes. The NN has sigmoid neurons and all data are normalized to [0,1]. I tried lots of different configurations: number of hidden layers and neurons in them (100->100->100, 60->60->60, 60->30->15, etc.), different learning and pre-train rates, etc.

And the best thing I can get is a 20% error rate on the validation set and a 40% error rate on the test set.

On the other hand, when I try to use Random Forest (from scikit-learn) I easily get a 12% error rate on the validation set and 25%(!) on the test set.

How can it be that my deep NN with pre-training behaves so badly? What should I try?

The problem with deep networks is that they have lots of hyperparameters to tune and very small solution space. Thus, finding good ones is more like an art rather than engineering task. I would start with working example from tutorial and play around with its parameters to see how results change - this gives a good intuition (though not formal explanation) about dependencies between parameters and results (both - final and intermediate).

Also I found following papers very useful:

Visually Debugging Restricted Boltzmann Machine Training
with a 3D Example
A Practical Guide to Training Restricted Boltzmann
Machines

They both describe RBMs, but contain some insights on deep networks in general. For example, one of key points is that networks need to be debugged layer-wise - if previous layer doesn't provide good representation of features, further layers have almost no chance to fix it.


arima analysis after clustering time series I have a database of multivariate time series that I want to cluster in order to find natural grouping of features. I am thinking of taking each cluster points and perform an ARIMA analysis on its points. Within those clusters, what should be the best way to infer those groupings in the original mutltidimentional feature space?
a sample of my time series is :

TimeStamp       | Sensor0 | Sensor1| Sensor2
2015-02-05 11:30|<Min     | On     | off
2015-02-05 11:31|<Min     | on     | off
2015-02-05 11:32| Action2 | 10     | 0.0001
2015-02-07 11:33| Action2 | 10     | 0.00012
2015-02-07 11:34| Action2 | 10     | 0.00012
2015-02-07 11:35| Action2 | 20     | 0.00015

I have roughly 10 separate groupings and I don't think performing different ARIMA calculations would be the best efficient method. Any thoughts or ideas are also welcome!

Here's a reference for your problem. If you read the first two pages, you'll get a hint at what features to use for time series clustering.

https://www.jstatsoft.org/article/view/v062i01


What are the differences between Apache Spark and Apache Flink? Both Apache-Spark and Apache-Flink projects claim pretty much similar capabilities.

what is the difference between these projects. Is there any advantage in either Spark or Flink?

Thanks

Flink is the Apache renaming of the Stratosphere project from several universities in Berlin. It doesn't have the same industrial foothold and momentum that the Spark project has, but it seems nice, and more mature than, say, Dryad. I'd say it's worth investigating, at least for personal or academic use, but for industrial deployment I'd still prefer Spark, which at this point is battle tested. For a more technical discussion, see this Quora post by committers on both projects.


Unsupervised Classification for documents I'm trying to create a classifier in which there is less "manual" work for the user. For less manual work I mean that there won't be an initial phase of manual labeling of a training set, like in Machine Learning (supervised)

My dataset is composed by instances that are really different by class. They are documents in which there are orders for specified products for different clients. And every client got its template.

For example I got:

[Client A]
Image
Date: xxx  Order:
Products:
Table

[Client B]
Date: xxx
Order
Image
Products:
table
Image

Now I'm doing the classification doing a simple check on every document, for the presence of a specified feature, that's manually identified by a user (by area and using edit distance)

The classes are really different (in some cases), and trying an unsupervised classifier like an agglomerative clustering the classes are split really well. After that, using measures like TF/ICF often the features (in my case I use tokenized and normalized text as features) which got the greater values are the ones that are used in my manual classification.

The criterions that I use for stop the clustering iteration are different ( I got different configuration) like max distance, or max number of clusters.

After that, I thought that when the clusters will be created an user at the end will label each cluster identifying the class by the best TF/ICF (term frequency, inverse cluster frequency) features found in each cluster. And after that the clusters will be used like "classifier". I know that this approach will lead to worse classification, but It's not a problem.

The problem is that when two classes are really similar (I got classes in which the difference is only the customer code, for example) they are really difficult to split.

Any idea on how approach this problem?
And, there is a way in which my algorithm can find out if there is a "new class" in the flux?

If you have a good amount of instances for every class, you can try using a density-based approach for clustering, with algorithms like DBSCAN.

If you can label at least some of the documents, you can use semi-supervised learning. Usually, when SSL is used for clustering, you need to specify "cannot link" and "must link" constraints for some pairs of instances, which is basically labeling some instances. One algorithm that follows this approach is HMRF-KMeans (Hidden Markov Random Fields K-Means).


Is there any formal explanation for the sensitivity of AdaBoost to outliers? AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.

Consider a supervised problem with target values y either being -1 or 1. Let's say we are fitting function f(x) using our AdaBoost classifier. Intuitively we want positive margins, where a margin is defined as $y_{i}f(x_{i})$. A positive margin means we have correctly classified that sample since the sign of both f and y are the same. A negative margin means we have misclassified the sample.

The loss function for Adaboost is defined as an exponential:

$L(y, f(x)) = exp(-yf(x))$

This exponential then ends up more strongly penalizing points with a large negative margin (i.e., outliers) than other classifiers. Here is a plot of the loss function vs. the margin for several classifiers:

So, to summarize: Adaboost is susceptible to overfitting outliers due to the form of its loss function.

Source: Hastie, et. al, The Elements of Statistical Learning, 2017, pp. 343-347


How to find the class name of a new image from the pre-trained model I would just like to get the class names of the predictions. I can get the class names on the images that I trained the model. But if I predict an image (say which is not trained but already belongs to the pre-trained model (VGG16) I cannot get its class name).

My scenario:
I used VGG16 pre-trained and added new datasets (like logos, stars, hills etc which are not present in VGG16 (say) )

Now I trained this model in those datasets, when i predicted an image, it gave the correct class-name (logos, stars, hills etc..)

But when I predict an image that was already in VGG16 (say Dog, Cat), it is not giving me the right class name i.e Dog, Cat etc.

I need some help in this scenario.

Is there any way to get the get the class name?

In the original training session, there will have been the mapping between class name and numerical class ID (e.g. Cat = 0, Dog = 1, ...). The model you use will output just the IDs and you need to find the mapping that was used.

If your new data has been added in addition to the old classes, they should each have a new unique ID. If you used VGG architecture to train on your new classes from scratch, then the model will not be able to make predictions for the original VGG classes.


NLP algorithms for categorizing a list of words with specific topics Currently I am using LDA to apply topic modeling to a corpus. Since LDA is unsupervised, it returns a set of words for a given 'topic' but doesn't necessarily specify the topic itself. I was wondering if there are any suggestions for algorithms that take a list of words and sees what topics it can be categorized to?

For example [cat, dog, fish] can be categorized to animals or pets.

One output for my model:

['game', 'week', 'fantasy', 'sportsline', 'play', 'players', 'league', 'random', 'sunday', 'season', 'agent', 'elink', 'exercise', 'start', 'yards', 'free', 'injury', 'expected', 'practice', 'getbad', 'weekly', 'year', 'reports', 'starting', 'luck', 'nat', 'nfl', 'weeks', 'smith', 'fast']

Could be categorized to football or sports.

Any suggestions, specifically with Python models/packages would be much appreciated.

Extending the basic idea stated in comments by @krayyem, you are looking for an Ontology and/or Taxonomy. The short story is to (1) build them based on your data or (2) use existing ones.

Building from your data

You need lots of Is-A pairs which indicate what kind of thing the concept is e.g. Maradona is a football-player and football is a sport. For this purpose, you extract info from your text based on some patterns and update info then update pattern based on info and the loop goes on till it does not change anymore. see this answer though.

Solution you are probably looking for

Use existing knowledge bases that some of which are in Python e.g. wordnet. You can find more in the links I provided for the answer I linked above.

How to proceed kind of idea

Find the dominant keyword in the found keywords. You may simply just count and go through the concept graph (your ontology, the knowledge base) and find the parent concept. If the graph is not accessible in this form, you may create it yourself in Python with existing APIs and/or graph libraries in Python like Networkx.

If more input is needed please comment here. Good Luck!


Notebook for data science I'm searching for an online Jupiter notebook environment that allows collaborating in real-time. I've found that I've choice from two services Deepnote.com and DataLore by JetBrains. I have little experience with Deepnote and it starts to perform very poorly when you have a lot of cells in your notebook. If anyone has experience with two environments, does the DataLore suffer the same issue? What advantages and disadvantages do DeepNote and DataLore have relative over each other? Are there any other real-time collaboration environments?

Product Manager from Deepnote here.
I'm sorry it doesn't perform well when the notebook is too big. We're aware of the issue, and it's quite hard to solve, but I'll stress its importance to our engineers.
I'm not sure how Datalore handles large notebooks. When you compare them, they both have some nice features (collaboration, secure integrations, reactivity). Deepnote has a few extra ones (automatic versioning, diffing, comments...), and some great ones coming soon (scheduled reports, published applications). On the other hand, JetBrains is a company with decades of experience in the IDE business.
Some other ones you can check which have an open free tier and real-time collaboration: Nextjournal, CoCalc, Databricks (community edition). There are a few more similar products listed here.


Find the order of importance of random variables in their ability to explain a variance of Y Problem statement:
Given a set of random variables Xs, give the order of importance of their ability to explain variance in target variable Y. Assumes all Xs are normalised to have 0 mean and std dev of 1.

Can treat this as regression problem(normal or ordinal)

Solution for 2 cases:
Case 1: When the calculated covariance matrix is diagonal, I know that random variables Xs are independent wrt each other. Then I can fit a linear regression model and order Xs by their weights.

Case 2: Or If I know that Xs are independent(https://en.wikipedia.org/wiki/Independence_(probability_theory)
even though their covariance in non zero, I can fit linear regression model and order Xs by weights.

How do I solve this problem when I know that random variable Xs are not independent and have non zero covariance?

Also how can I use a non linear model such as random forest to get variable importance? scikit provides feature_importances_ api for random forest.... Is there an gotcha with this api such as results are not meaningful when variables are dependent or covariant?

Found this question elsewhere: https://stats.stackexchange.com/questions/202221/for-linear-classifiers-do-larger-coefficients-imply-more-important-features/202846

Decision trees are by nature immune to multi-collinearity. So by that principal so is Random Forest. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. - follow this link to know more.

For other models, solutions can be many, but here are the popular choices -

Regularization

This is an automatic way of handling high correlations in your Xs. It decreases the value of the co-efficients by penalizing them against the loss function and introduces bias in the system. Popular implementations are L1, L2 and Elastic Net(in case of linear and logistic models), Pruning(in case of Decision Trees), Dropout and max pooling in case of deep learning.

Almost all sklearn implementations support a regularization parameter in 1 way or the other. Depends on the algorithm you are using.

Using VIF to detect multi-collinearity

Variance Inflation Factor is a metric that you can use with all your Xs and then start dropping variables with the highest VIF one by one (usually the variable with the highest VIF is dropped and then the exercise is repeated till there is no multi-collinearity left or domain experts come in and choose what important variables to keep/drop)

Principal Component Analysis

Although the model might lose its interpretability (you won't be able to tell exactly what feature is how important), PCA has been another effective way of removing multi-collinearity. Basically it projects the data in a way that the output columns are orthogonal (independent)

Some Examples

Ridge Regression

 from sklearn.linear_model import Ridge
 import numpy as np
 n_samples, n_features = 10, 5
 np.random.seed(0)
 y = np.random.randn(n_samples)
 X = np.random.randn(n_samples, n_features)
 clf = Ridge(alpha=1.0)
 clf.fit(X, y)

Lasso Regression

 from sklearn import linear_model
 clf = linear_model.Lasso(alpha=0.1)
 clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])

Elastic Net

 from sklearn.linear_model import ElasticNet
 from sklearn.datasets import make_regression

 X, y = make_regression(n_features=2, random_state=0)
 regr = ElasticNet(random_state=0)
 regr.fit(X, y)

Random Forest
 - Almost all parameters in scikit-learn can account for regularization here

Statsmodels has a rather good implementation of VIF

PCA

import numpy as np
from sklearn.decomposition import PCA
from sklearn.svm import SVC

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
pca = PCA(n_components=2) # adjust yourself
pca.fit(X)

X_t_train = pca.transform(X)
clf = SVC()
clf.fit(X_t_train, y)


Neural network without matrices Have you ever seen a neural network without matrices?

I'm asking, because I'm currently building one for educational purposes.

Matrix multiplication is just a simplified notation for a particular set of addition and multiplication operations. You can absolutely represent a neural network without invoking matrix notation, it'll just be really tedious (and it will run slower).

Start with a single perceptron and build your way up from there.


Understanding the math behind SVM I was going through this Udacity video (part of the ML nano-degree), where the math behind maximizing the length between the data clouds of two classes is done.

So, the line closest to the +1 class cloud would be: $w^T x_1 + b = 1$

And, the line closest to the -1 class cloud would be: $w^T x_2 + b = -1$  ($x_1$ and $x_2$ being points of both the lines respectively.)

So, for getting the distance between both of them (the lines above), the professors subtract both of the equations.

And then, while finding ($x_1-x_2$), they divide both the sides with the norm of w ($||w||$), which starts at 2:24 minutes.

From then, I didn't understand the math in the video.

Why did the professors divide the sides by the norm?  What do the norm specify/imply?

Can someone please explain the math from after 2:24?

In my personal opinion the lecturers's attempt to "make things look easy" by showing "magic tricks" ("let us now divide by the norm as if we were dealing with one-dimensional vectors AND VOILA") is rather unhelpful as it does not really explain neither the methodology nor the math nor the intuition.

There are some basic, but important notions hidden behind that "voila" step, let me help you by writing them out explicitly.

One way to find the distance between two parallel lines is to:

Pick any point $x_1$ on the first line.
Pick any point $x_2$ on the second line.
Project the vector $x_1 - x_2$ on the direction, perpendicular to two lines and measure the length of the projection.

Here's an illustration, just in case: you have two lines, one point taken on each line, the vector $x_2-x_1$ connects the two points. The dotted line is a perpendicular to the two lines. The vector $v$ points in the direction of the perpendicular, and the red segment shows the distance you want to measure.

This red distance is exactly the length of the projection of $x_2 - x_1$ onto the dotted line.

At this point we shall need three facts:

(a) For any unit vector $v$ the value $x^T v$ is equal to $|x|\cos \alpha$, where $\alpha$ is the angle between the vectors. Coincidentally, this is exactly the (signed) length of the projection of $x$ on the line with direction defined by $v$.
(b) Any vector $v$ can be made a unit vector (i.e. vector of length 1) by dividing it with its actual length: $\frac{v}{|v|}$.
(c) The equation $w^T x + b = 0$ defines a line with the normal $w$. I.e. $w$ is a vector, perpendicular to the line.

We know from (c) that $w$ is perpendicular to the line, we know from (b) that $\frac{w}{|w|}$ is a unit vector, so we plug it into (a) to get that the (signed) distance between the lines must be equal to $$\frac{w^T(x_2-x_1)}{|w|}.$$

If we know that $w^T x_2 + b = -1$ and $w^Tx_1 + b = 1$ we can play a bit with the equations to get $w^T(x_2 - x_1) = -2$ and hence the (signed) distance must be $\frac{-2}{|w|}$. We can drop the sign easily. ... and voila!

This is certainly not the only way to solve the problem, so for completeness sake let me show you a similar, yet still useful alternative. Here, you need to know just one fact (which I suggest you memorize just like the facts above):

(d) (Signed) distance from point $p$ to a line defined implicitly by the equation $w^T x + b = 0$ (note the zero on the right) is equal to $\frac{w^T p + b}{|w|}$.

Now, we can compute the distance between two lines by taking a point on one of them and substituting it in (d).

Let $x_1$ be a point on the line $w^T x+ b = 1$. The second line has the form $w^T x + b = -1$ which is the same as $w^T x + b + 1 = 0$. The distance between the lines is therefore:

$$\frac{w^T x_1 + b + 1}{|w|} = \frac{1 + 1}{|w|} = \frac{2}{|w|}.$$

Yet another way to derive the necessary distance would be to note that the line $x = sw$ is perpendicular to the two lines of interest. If $x_1$ and $x_2$ are the intersection points of this perpendicular with the first and the second parallel line correspondingly, the necessary distance is just the Euclidean distance between the two points:
$$ d = |x_1 - x_2| = \sqrt{(x_1 - x_2)^T(x_1 - x_2)}. $$
I'll leave solving this to you as an exercise.


Need rules of thumb for out of core larger than ram dataset on a laptop totally new to larger than ram datasets but I have csv files that are about 100 gb each with around 300 million rows each  (just two of them).

What I am looking for is something like: don't go analyzing more than 1 terabyte of data on a laptop. Or don't analyze data that is more than 10x your ram or the wait times will prove frustrating. This is what I mean by rules of thumb.

I have a mac laptop with an i5, 8gb ram, ssd is it reasonable to process this data (100gb per file and I have 2 files) with either dask or blaze in python?

I have tried and could read in the csv but when doing simple trials like dropping one column or finding the length of the data frame it takes at least an hour For the drop (I gave up waiting) and half an hour for the length() (that did finish). Way too slow to be productive. I changed the data format to parquet and amazingly the data is down to just 10 gb. This is the only way they the length() command ever even finished.

So my question is: are these response times normal experience in general given my hardware and csv size? Are my expectations for what dask can do way too high? Any rough guidelines on how to troubleshoot or is this clearly underpowered hardware？

I am not an expert but this is how I see things: use spark. Pyspark seems to be the flavor to use since it's python and that's the language that data science has settled on.

Dataset sizes are expanding way faster than RAM is on laptops. What "they" don't tell you is that scikit learn, pandas, R, etc all require your dataset and all intermediate steps to fit inside RAM. If you have an 8gb dataset, and you pivot it, your intermediate step could easily be say 30gb. Even if it's temporary and your final cleaned dataset to be fed into. A neural network shrinks back down to say 6gb, too bad. All of your steps must fit inside RAM.

In the real world, and even the limitation that datasets be limited to RAM sizes is quite limiting. This is what I wished I had known earlier. It's like saying here is a grocery store which is all yours (scikit learn has tons of features ) but you can only leave the store with a single shopping bag full of items). I would much rather have a grocery store with the standard 100 items and let me have a buffet on just those 100.

Yes, there are some libraries here and there for scikit learn that read from files in chunks, and then you add the chunks together, etc. But you will quickly see that that is reinventing the wheel and distracts from doing data science. The fact remains that these frameworks were built for in RAM usage. It's not a bad thing at all per se, but quickly becomes limiting.

What you want is to write the code once and not have it change whether the dataset is 1gb or 100gb. This is what pyspark is for. You can just learn their API and pyspark will handle chunking and distributing the workload such that it fits in the available RAM that you have. Of course if you are RAm limited computations take longer but this is ok. At least you will get an answer.

The downside is that if you want the very much appreciated ability to not have to worry about "is this dataset too large for my ram" question then you need to stay in pyspark's API. If they don't have a feature that say scikit learn has, or keras has, then too bad. Write your own or work around it.

Thankfully the API looks very complete.

Where does it lack? Well for say neural networks you aren't going to have all kinds of special activation functions and customization that you can have with Keras.

But in my experience and it really is very little, more data and better data beats the specific algorithm much of the time. I only mean to say that I would rather be able to process tons of data and use a vanilla classifier than be limited to in RAM dataset sizes and while being able to optimize 1000  hyperparameters.

You also don't want to be switching between different aPIs all the time as a beginner.

So if there is any chance that your dataset will be larger than RAM (I think that likelihood is a near certainty in the years ahead) I would most definitely learn pyspark. Even before scikit learn.


Evaluation method for multi-class classification problem modeled as binary classification problem I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation metric.

The problem is a multiclass(16 classes) classification problem where one data point can be classified in multiple classes (not ranking based though). I plan to model it as a binary classification problem for each class but for the related evaluation metrics I was not able to find a proper application. So, first of all, should I evaluate individual performance for each class (how well class A classification is working), should I go for a general evaluation (This data point belongs A,B,C but at the end classified for A and B only), or both? Second, what kind of metrics can I have a look at? Finally, I haven't started working on the data yet but I expect an unbalanced distribution for my classes. Would it affect my results?

Unbalanced data will definitely be a problem and should be addressed. In particular, "accuracy" will not be dependable metric any more so if you decide the use unbalanced data directly, so you should use other metrics that are more reliable for such scenarios, but that also can depend on the data distribution you have. Here is a discussion of how each metric perform  for different situations.

Apart from the choice of a proper metric, There are other ways to deal with imbalanced data. Here you can find some of the methods but probably most well known way to deal with it is oversampling and undersampling, in which you equate the number of samples in each class.

Regarding the individual evaluation vs general evaluation, this probably depends on your own preference and the classes you have in the problem. But individual classification will be useful at least to fine tune your model: you would not prefer a model which performs very well on some set of samples but very badly on some other set. Such a situation might indicate that the features used might not be as useful for each class and require getting more data.

And finally, relatively small differences in the size of classes might not be a problem and ignored; but this might be more of a personal choice. What are the relative sizes of the classes?


How to best forecast simple binary data? I have a set of timeseries binary (boolean) data, with intervals of 1 day. Each day can either be 1 or 0 (true/false). What is the best way to forecast the next day/week's data based on the data I already have?

Do some feature engineering. As they told you in the comments, try
to see if the event is more likely to happen during weekends or
weekdays, during certain months, during a certain season or quarter,
etc. To read some more on this topic.

https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/
Explore different machine learning models. I suggest to start with a Random Forest.
Check your model accuracy (confusion matrix).
Iterate. Try other features, other models...

Just to get you up and running, you can take a look at the Titanic dataset in Kaggle. It will help you understand a bit more of the problem you are trying to solve.


Implement Sliding Window Algorithm to Detect Spikes How do I implement sliding window algorithm with a window size of 10 and visualize the data iteratively to see spikes/possible outliers in the dataframe, using python? Please help a beginner.

welcome to DS-SE and to Data Science in general! :)

Your problem can be solved really easily in Python. Please, take a look at Pandas DataFrame class to represent your data, it makes it really convenient because of all the pre-built methods that includes.

One of its methods is pandas.DataFrame.rolling() (see here) that does exactly what you asked for: rolling (sliding) window calculations, please take a look!

Hope it helps!


Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification] I'm following this guide to try creating both binary classifier and multi-label classifier using MeanEmbeddingVectorizer and TfidfEmbeddingVectorizer shown in the guide above as inputs.

Both embedding vectorizers are created by first, initiating w2v from documents using gensim library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.

After this step has completed, I tried using these embedding vectorizers as inputs into several model such as OneVsRestClassifier(SVC), RandomForestClassifier and ExtraTreesClassifier. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.

These are the accuracies from each models (binary classifer) :

randomF_countVect: 0.8898
extraT_countVect: 0.8855
extraT_tfidf: 0.8766
randomF_tfidf: 0.8701
svc_tfidf: 0.8646
svc_countVect: 0.8604
ExtraTrees_w2v: 0.7285
ExtraTrees_w2v_tfidf: 0.7241

Multi-label classifier also produced similar result.

I'm not sure that I've done wrong.

Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?

Word2vec initiation

model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}

Models

svc = Pipeline([("count_vectorizer", vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
svc_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
randomF = Pipeline([("count_vectorizer", vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
randomF_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
extraT = Pipeline([("count_vectorizer", vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
extraT_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
etree_w2v = Pipeline([("word2vec vectorizer", MeanEmbeddingVectorizer(w2v)), ("word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
etree_w2v_tfidf = Pipeline([("tfidf word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("tfidf word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])

all_models = [
    ("svc", svc),
    ("svc_tfidf", svc_tfidf),
    ("randomF", randomF),
    ("randomF_tfidf", randomF_tfidf),
    ("extraT", extraT),
    ("extraT_tfidf", extraT_tfidf),
    ("etree_w2v", etree_w2v),
    ("etree_w2v_tfidf", etree_w2v_tfidf)
]
scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean())
             for name, model in all_models],
            key=lambda args: -(args[1]))

My guess is that the dataset is very small. Word2Vec won't be able to capture word relationship in the embedding space with limited information. Try to train word2vec on a very large corpus to get a very good word vector before training your classifier might help.


Encoder-Decoder LSTM for Trajectory Prediction I need to use encoder-decoder structure to predict 2D trajectories. As almost all available tutorials are related to NLP -with sparse vectors-, I couldn't be sure about how to adapt the solutions to a continuous data.
In addition to my ignorance in seqence-to-sequence models, embedding process for words confused me more. I have a dataset that consists of 3,000,000 samples each having x-y coordinates (-1, 1) with 125 observations, which means the shape of each sample is (125, 2). I thought I could think of this as 125 words with 2 dimensional already embedded words, but the encoder and the decoder in this Keras Tutorial expect 3D arrays as (num_pairs, max_english_sentence_length, num_english_characters).
I doubt I need to train each sample (125, 2) separately with this model, as the way Google's search bar does with only one word written.
As far as I understood, an encoder is many-to-one type model and a decoder is one-to-many type model. I need to get a memory state c and a hiddenstate h as vectors(?). Then I should use those vectors as input to decoder and extract predictions in the shape of (x,y) as many as I determine as encoder output.
I'd be so thankful if someone could give an example of an encoder-decoder LSTM architecture over the shape of my dataset, especially in terms of dimensions required for encoder-decoder inputs and outputs, particulary on Keras model if possible.

There are multiple questions in your description:

The input to LSTMs are normally continuous representations. In NLP, you normally embed discrete elements as vectors in a continuous representation space and then you pass these vectors to an LSTM. You already have continuous representations, so you just pass your 2-dimensional vectors as input to the LSTM.
In NLP neural networks, like most neural networks, the training happens passing as input not one sample, but a minibatch of N samples. This N most of the times is chosen as the maximum number that makes the model, data and intermediate computations fit in the GPU memory.
The 3D array expected as input by the LSTM has shape [N, timesteps, feature]. In your case, this would be [N, 125, 2].
I think you don't need an encoder-decoder architecture, only the encoder. Therefore, a single LSTM would suffice. You would train it to receive any number of input elements and predict the next one. If you want more predictions ahead, you can feed the model's own predictions as input, autoregressively. To find an analogy in the NLP world, your model would be a language model, which receives words (or letters) and generate the following word.

P.S.: I may be wrong about you needing only the encoder instead of an encoder-decoder architecture, as I don't know the specific nature of the predictions you want to make.


Weighted mean with summarise_at dplyr I strictly need to use the summarise_at to compute a weighted mean, with weights based on the values of another column

    df %>% summarise_at(.vars = vars(FACTOR,tv:`smart tv/console`),
                  .funs = weighted.mean, w=INVESTMENT, na.rm=TRUE)

It always shows the error: 'INVESTMENT' is not found.

I then tried with:

df %>%summarise_at(.vars = vars(FACTOR,tv:`smart tv/console`),
               .funs = weighted.mean, w=vars(INVESTMENT), na.rm=TRUE)

But in this case :  Evaluation error: 'x' and 'w' must have the same length.

Why is this? Am I doing anything wrong? Do you have hints to solve this issue? Thanks

You can specify the weights directly within the weighted.mean() function, within the call to funs() like so:

data.frame(x=rnorm(100), y=rnorm(100), weight=runif(100)) %>%
      summarise_at(vars(x,y), funs(weighted.mean(., w=weight)))


Conceptual questions on MLP and Perceptrons I am facing some confusion regarding the terminologies assocaiated to classification and regression problems esp. using the MLP and Perceptron models.
These are the following:

1) When the data is linearly inseparable, we use MLP. Here what is meant b "data"--is it the response or the input feature that is linearly inseparable?

2) If it is linearly inseparable then does it mean that the mapping function from input to output will always be non-linear? Hence, we prefer MLP or the latest new models such as deep learning?

3) Linear regression fails in the case of linearly inseparable data or can linear regression work for inseparable data but if the function mapping is nonlinear then it fails?

When the data is linearly inseparable, we use MLP. Here what is meant by "data"--is it the response or the input feature that is linearly inseparable?

This means that a linear function of the input features is unable to separate the response.

To answer your question a bit more directly: Given only a linear function of the inputs, the response is the thing that's inseparable.

  If it is linearly inseparable then does it mean that the mapping function from input to output will always be non-linear? Hence, we prefer MLP or the latest new models such as deep learning?

Yes.  If the mapping from input to output were linear, then the output would necessarily be linearly separable by the input.

  Linear regression fails in the case of linearly inseparable data or can linear regression work for inseparable data but if the function mapping is nonlinear then it fails?

Linear regression will never be able to perfectly separate linearly inseparable data.  Consider the following example, where the input features are x1 and x2, and the output is the color:

It doesn't matter how you draw a line in the 2D space - you'll never be able to separate the colors.  The same idea applies in higher dimensions.

I hope that helps!


What is the concept of error bars in barplots in seaborn library? I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in documentation. What do these error bars represent and how are they calculated ?

It appears that even if you don't enter ci argument, it's set to 95 by default in barplot method, which gives you those error bars. If you don't want them, set ci = None while plotting. The bar is the confidence interval for the variable as the library describes, i.e. it's interval where the 95 % of your variable lies in, which is estimated by bootstrapping.


How to make it possible for a neural network to tune its own hyper parameters? I am curious about what would happen to hyperparameters when they would be set by a neural network itself or by creating a neural network that encapsulates and influences the hyperparameters of the network it encapsulates.

The goal for me here is to experiment and get some more in-depth knowledge about neural networks. But I had a hard time finding information to be able to execute such an experiment. Which point in a direction that either it has never been done before or the idea is just really dumb.

Now what I would like to know is, does anyone of you know where I would be able to find information (e.g. books, web-articles, papers, et cetera) to conduct such an experiment?

I am curious about what would happen to hyperparameters when they would be set by a neural network itself

In general this is not possible as many hyper-parameters are discrete, so they are not differentiable with respect to any objective. For example, this applies to layer sizes, number of layers, choices of transfer functions. This prevents using any form of gradient descent to tune them directly as learnable parameters.

In fact the separation between parameters and hyperparameters is exactly that hyperparameters are not learnable by the model type. This applies to other ML models, not just neural networks.

  or by creating a neural network that encapsulates and influences the hyperparameters of the network it encapsulates.

This is more feasible. You could use one neural network to try and predict the results from another. Then prefer to run tests on target networks that look like they will do well. However, using a "meta" neural network like this has some major drawbacks:

Neural networks require a lot of training data. Getting enough samples to make good predictions would require that you train your primary neural network (a time-consuming process) many times
Neural networks are bad at extrapolating to data outside of areas already experienced, so not so great at making creative predictions of new parameters to try
Neural networks have a lot of hyper-parameters to tune. Would you need a "meta meta" neural network to predict the performance of your "meta" network?

  either it has never been done before or the idea is just really dumb

This is a real issue that comes up repeatedly. In general the search for best hyper-parameters is a chore. It is an active area of research and experimentation to find efficient ways of automating it, or avoiding it by making some hyperparameters less important or not necessary.

The reason you are not finding neural networks that tune neural networks is due to the issues listed above. So the main areas of research focus on different approaches, that can work with limited data and don't have so many hyperparameters themselves. Or models that are robust to large ranges of hyperparameters, so precise tuning is not a big deal.

Here are a few pointers to help with automated searches:

You could use a variety of hyperparameter optimisation schemes, including random search, grid search, genetic algorithms, simple gradient methods etc.
Random searches, perhaps constrained by previous experience or second-hand knowledge from similar problems, can be reasonably effective.
The quality of any search is limited by the quality and amount of cross-validation data. There is not much point tuning the cv loss value to the point that you care about changes that are much less than the standard error in its estimate.
Response to hyperparameters is typically non-linear over the search space, which makes things harder.

Outside of automation, expert analysis is often a good starting point, especially if you want to assess success of regularisation. Typically you can look at learning curves for training and cross-validation data, and based on that you can make a reasonable guess as to whether to increase or reduce regularisation hyperparameters and/or learning rate, even from observing results from a single training run.

There have likely been attempts to automate some parts of reading learning curves, since sometimes it is relatively easy to detect over-fitting and under-fitting scenarios. However, I could not find any examples when searching just now.


Why we use information gain over accuracy as splitting criterion in decision tree? In decision tree classifier most of the algorithms use Information gain as spiting criterion. We select the feature with maximum information gain to split on.

I think that using accuracy instead of information gain is simpler approach. Is there any scenario where accuracy doesn't work and information gain does?

Can anyone explain what are the advantages of using Information gain over accuracy as splitting criterion?

Decision trees are generally prone to over-fitting and accuracy doesn't generalize well to unseen data. One advantage of information gain is that -- due to the factor $-p*log(p)$ in the entropy definition -- leafs with a small number of instances are assigned less weight ($lim_{p \rightarrow 0^{+} } p*log(p) = 0$) and it favors dividing data into bigger but homogeneous groups. This approach is usually more stable and also chooses the most impactful features close to the root of the tree.

EDIT: Accuracy is usually problematic with unbalanced data. Consider this toy example:

Weather Wind    Outcome
Sunny   Weak    YES
Sunny   Weak    YES
Rainy   Weak    YES
Cloudy  Medium  YES
Rainy   Medium  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO

Weather and Wind both produce only one incorrect label hence have the same accuracy of 16/17. However, given this data, we would assume that weak winds (75% YES) are more predictive for a positive outcome than sunny weather (50% YES). That is, wind teaches us more about both outcomes. Since there are only few data points for positive outcomes we favor wind over weather, because wind is more predictive on the smaller label set which we would hope to give us a rule that is more robust to new data.

The entropy of the outcome is  $ -4/17*log_2(4/17)-14/17*log_2(14/17)) =0.72$. The entropy for weather and outcome is $14/17*(-1/14*log_2(1/14)-13/14*log_2(13/14)) = 0.31$ which leads to an information gain of $0.41$. Similarly, wind gives a higher information gain of $0.6$.


Adding document vector doc2ec to seq2seq model In general we use word2vec for word embedding in seq2seq model, is it possible to add the document vector from Doc2vec with the input words , I mean using the tag of the document as a word and its vector for emending : the picture will explain mu point a view

default seq2seq

My opinion

the doc_tag is followed by its original words so will that improve my seq2seq model ( logically ) ?

The purpose of the encoder (green part) is to determine this document vector, so if you want to provide it yourself use the decoder and feed the embedding as the initial state.

I suppose you could use your embedding as a prior for the encoder output. One alternative to your suggestion, which might work, is to use the embeddings as an MSE regularizer to the encoder output.


What is the most appropriate machine learning approach for this scenario? The scenario is pretty simple, and I'm sure it's been done a million times. The problem is i don't know the terminology to find the correct resources on the web.

Scenario: I have an environment that can be described in terms of 5 parameters, including and input value A and an output  variable B. There is a dataset containing 100 rows and values for each parameter.

The output B depends on A as well as the remaining environmental variables.

The goal is to find the best value for the input A such that the output B is minimised.

What does the solution for this problem look like? Is it  Machine learning, neural networks, a mathematical optimisation problem? How is this best approached?

Extension: if I didn't have a dataset in practice,  how would I train a system to suggest different values for A until a minimum is reached? Can neural networks be applied here? Or are we talking about a looping procedure that does known maths in each operation until the output doesn't change much anymore.

I though the generalisation would make it more difficult to answer.
What I am describing is a number of temperature/humidity measure measurements for both inside my house and outside. The input that I can control is the fan speed setting on my evaporative aircon and the output is the loubge temperature which I want to minimise.

During sample gathering, I don't care much about the output. The set of 100 values was arbitrary and more (and diverse) samples can be obtained.

The range of A is a fan speed with discrete values 1-6.
Humidity is a percentage and temperature is 20-45 degrees.

This certainly looks like a basic optimization problem. However, looking from the Machine Learning perspective, what you describe can be framed as a multilabel classification. In that case, you treat the values of A and B as your input, and try to predict the values of those five parameters, which you mentioned:

A, B -> x1, x2, x3, x4, x5

The only pitfall is that 100 records can be insufficient to train a proper classification model, but if you can obtain more training data, this approach seems solid to me.


Format for time series data with non-trivial sampled data I have a data stream that I would like to share with some data scientists.

It is a regularly captured time series with some fields that are simple scalars, booleans. Each sample has a UTC time and fractional seconds since start of capture.

Also captured is a 3d spline that varies in size. These splines are also regularly sampled.

Additionally, there are some other multi dimensional fields as well ([x,y,z,pitch,yaw,roll]).

For simple data sets I would normally use CSV. However due to the nature of the more complex data I need a more appropriate format.

What are my options for formats that would allow easy loading into Matlab or other common data science tools?

Based on this article about Data Formats in Data Science, the short answer to your question is: use JSON.

This may seem natural next step after csv and no more, but it is actually quite interesting:

JSON is widely supported in many programming languages, it is a valid MIME type in Internet standards (application/json, data stream mentioned in your question) and it has several standard and non standard extensions (geoJSON, binary json). If there is not a standard structure, JSON makes it easy to invent your own, because it is easily modifiable.

According to the article JSON works as value in HADOOP. For HADOOP it is suitable, because basically JSON is plain text when you think it as a data item.

Also article mentions JSON more efficient than CSV.


How to automatically find the sentiment? I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze.

Part 1: I am just starting out in data science and wanted to know if there is such a thing as a library that will read through text and automatically classify words or sentences as being positive/negative/neutral. Is there such a thing that exists, perhaps in R or Python?

Part 2: From what I can tell, there isn't such a library, and I would have to do that myself by hand. It appears that the best way to analyze sentiment is to:

a) put the words in a corpus
b) do things like word stemming and
clean up the text
c) download a dictionary of predetermined
sentiment
d) run a program that vectorizes the text
e) compare the text from my corpus to the dictionary
f) ..... unsure what after this step.

Are these steps accurate, or am I totally off-base?

Here is a tutorial, that should help you get started.


Performing anomalie detection on a battery volatge using LSTM-RNN I am trying to detect anomalies in a battery output voltage for one month.
I have the next data frame, as it is shown the data is collected each minute for each day so I have almost 1420 sample per day.

Should I use the 'time' or the 'date' column in my time series analysis?
I could not find an example like my situation all dataframes I found looks like the next :

It is all about a unique value for each day. Please any help is appreciated!
Thank you in advance!

I will use the "date" and "time" columns to pre-process your data and to construct your neural net input.
RNN does not work well for very long-term dependancies... so, for example, creating a time series with all minutes in a month, won't probably work.
You must select:

How many samples your input data will have
What is your sampling period (minute, hour, day, week...)
What do you want to detect: If the input data contains an anomally or you might want to predict  if an anomally will occur in the future...

When you have all of this, and perhaps something else, defined you will have to use your "date" and "time" columns to create the dataset (the time-series)
Furthermore, I don't know if all batteries are checked under the same circunstances, for instance, are all of them new stock ones? If not, you can also use those columns to compute battery age or "working time" as an extra feature.


Should I prevent individuals mating with themselves in EAs? In evolutionary algorithms, should I always avoid individuals mating (crossover) with themselves? That is, should I prevent the selection algorithm from selecting a single parent twice to produce a child with itself?

I'm interested in the general case as well as for tournament selection.

When talking about Evolutionary Algorithms, you will have to decide between diversity or specialization of the possible solutions.

You should not let individuals crossover themselves because you will be loosing diversity in your algorithm.

But, there is the exception to this rule: You need to make sure you are not punishing the success of this individual if he is the best, you could make sure this individual passes to the next generation by preserving the best parents.

The answer to your question is: Prevent auto-crossover but make sure you don't loose this solution when selecting the ones passing to the next generation (Elite group, passing n of the best parents)

In tournament selection, is not common to select the same individual as a participant of the tournament twice, so this possibility should not be one of your worries.

In roulette selection, be sure you don't eliminate successful individuals.


Increasing SpaCy max NLP limit I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.

The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.

Try to raise the nlp.max_length parameter (as your error message suggests):
nlp = spacy.load('en_core_web_sm')
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:
doc = nlp("The sentences we'd like to do lemmatization on", disable = ['ner', 'parser'])

Finally you should get the results you expect with:
print([x.lemma_ for x in doc])


R: Producing multiple plots (ggplot, geom_point) from a single CSV with multiple subcategories I have a collection of bacteria data from approximately 140 monitoring locations in California. I would like to produce a scatterplot for each monitoring location with the Sampling Date on the Y-axis and the Bacteria Data on the X-axis. The Sampling Date, Bacteria Data, and Monitoring Location all reside within their own column.

I've come up with the below code:

## Create List of Files ##
filenames <- list.files(path = "C:\\Users\\...")

## Combine into one CSV ##
All_Data <- ldply(filenames, read.csv)
All_Data$SampleDate <- as.Date(All_Data$SampleDate, origin="1899-12-30")

## Save CSV for possible future use ##
write.csv(All_Data, file= "C://Users//...", row.names = FALSE)

## Construct Plots ##
ggplot(All_Data) + geom_point(mapping =aes(SampleDate, Total.Result)) + facet_wrap( ~ Identifier) +ylim(0,20000)

but this produces the below plot where every single location is crammed into the plot frame. Is there a way to print each plot individually to a folder?

I tried to incorporate the subset function like so

ggplot(All_Data) + geom_point(mapping =aes(SampleDate, Total.Result)) + facet_wrap( ~ subset(All_Data, Identifier)) +ylim(0,20000)

but received the error

Error in subset.data.frame(All_Data, Identifier) :
  'subset' must be logical

Alternatively, is it better to do this through some sort of loop through the original 15 csvs that I've combined together? I would still have the challenge of creating one plot per monitoring location. Thanks in advance for any suggestions!

something like this might work:

library(plyr)

# initialize All_Data around here

dlply(All_Data, 'Identifier', function(dataSubset) {
  g <- ggplot(dataSubset) + geom_point(mapping =aes(SampleDate, Total.Result)) + ylim(0,20000)
  file_name <- paste0("Scatter_", unique(dataSubset$Identifier), ".tiff", sep="")
  ggsave(file_name,g)
})

(I didn't test it)


How to determine if character sequence is English word or noise What kind of features you will try to extract from list of words for future predicting, is it existing word or just mess of characters ?

There is description of task that I found there.

You have to write a program that can answer whether a given word is English. This would be easy — you'd just need to look the word up in the dictionary — but there is an important restriction: your program must not be larger than 64 KiB.

So, I thought it would be possible to use logistic regression for solving the problem. I don't have a lot of experience with data-mining but the task is interesting for me.

Thanks.

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following.

ngrams

Take a random sample of words from words.txt. For each word in sample, extract every possible bi-gram of letters. For example, the word strength consists of these bi-grams: {st, tr, re, en, ng, gt, th}. Group by bi-gram and compute the frequency of each bi-gram in your corpus. Now do the same thing for tri-grams, ... all the way up to n-grams. At this point you have a rough idea of the frequency distribution of how Roman letters combine to create English words.

ngram + word boundaries

To do a proper analysis you should probably create tags to indicate n-grams at the start and end of a word, (dog -> {^d, do, og, g^}) - this would allow you to capture phonological/orthographic constraints that might otherwise be missed (e.g., the sequence ng can never occur at the beginning of a native English word, thus the sequence ^ng is not permissible - one of the reasons why Vietnamese names like Nguyễn are hard to pronounce for English speakers).

Call this collection of grams the word_set. If you reverse sort by frequency, your most frequent grams will be at the top of the list -- these will reflect the most common sequences across English words. Below I show some (ugly) code using package {ngram} to extract the letter ngrams from words then compute the gram frequencies:

#' Return orthographic n-grams for word
#' @param w character vector of length 1
#' @param n integer type of n-gram
#' @return character vector
#'
getGrams <- function(w, n = 2) {
  require(ngram)
  (w <- gsub("(^[A-Za-z])", "^\\1", w))
  (w <- gsub("([A-Za-z]$)", "\\1^", w))

  # for ngram processing must add spaces between letters
  (ww <- gsub("([A-Za-z^'])", "\\1 \\2", w))
  w <- gsub("[ ]$", "", ww)

  ng <- ngram(w, n = n)
  grams <- get.ngrams(ng)
  out_grams <- sapply(grams, function(gram){return(gsub(" ", "", gram))}) #remove spaces
  return(out_grams)
}

words <- list("dog", "log", "bog", "frog")
res <- sapply(words, FUN = getGrams)
grams <- unlist(as.vector(res))
table(grams)

## ^b ^d ^f ^l bo do fr g^ lo og ro
##  1  1  1  1  1  1  1  4  1  4  1

Your program will just take an incoming sequence of characters as input, break it into grams as previously discussed and compare to list of top grams. Obviously you will have to reduce your top n picks to fit the program size requirement.

consonants & vowels

Another possible feature or approach would be to look at consonant vowel sequences. Basically convert all words in consonant vowel strings (e.g., pancake -> CVCCVCV) and follow the same strategy previously discussed. This program could probably be much smaller but it would suffer from accuracy because it abstracts phones into high-order units.

nchar

Another useful feature will be string length, as the possibility for legitimate English words decreases as the number of characters increases.

library(dplyr)
library(ggplot2)

file_name <- "words.txt"
df <- read.csv(file_name, header = FALSE, stringsAsFactors = FALSE)
names(df) <- c("word")

df$nchar <- sapply(df$word, nchar)
grouped <- dplyr::group_by(df, nchar)
res <- dplyr::summarize(grouped, count = n())
qplot(res$nchar, res$count, geom="path",
      xlab = "Number of characters",
      ylab = "Frequency",
      main = "Distribution of English word lengths in words.txt",
      col=I("red"))

Error Analysis

The type of errors produced by this type of machine should be nonsense words - words that look like they should be English words but which aren't (e.g., ghjrtg would be correctly rejected (true negative) but barkle would incorrectly classified as an English word (false positive)).

Interestingly, zyzzyvas would be incorrectly rejected (false negative), because zyzzyvas is a real English word (at least according to words.txt), but its gram sequences are extremely rare and thus not likely to contribute much discriminatory power.


In the context of image binary classification, is it necessary to divide dataset into true positive, false positive, true negative, false negative? I am working through this course. It seems that the professor is not dividing the dataset into true positive, false positive, true negative, and false negative.

In the context of image binary classification, is it necessary to divide the dataset into true positive, false positive, true negative, and false negative?

No it's not. The numbers of true positives, false positives, true negatives, false negatives, are something you can check from a confusion matrix when you evaluate the performance of your trained model on a test set. It's useful to know where your model is making more mistakes.

It's not something you can do before training.


Is there a proper way of comparing weight coefficients of two logits? I'm currently working on web advertising topics and I'd like to know if it does make sense to, let's say, compare the resulting weights of two logits?

Let's take as an example the case where input variables are the same across models. A variable gets a weight of 0.22 as the result of the first logit, and a weight of 0.08 as the result of the second logit (all the other variables remaining the same, and their weights may change obviously)

May I conclude that this variable is of highest importance for explaining hidden patterns within the first dataset, or is such a comparison forbidden? (In fact, when thinking about the problem I couldn't find any obvious mistake...)

Of course, this makes complete sense, as long as all the other variables are the same. In fact, if you think of logistic regression as a linear regression with the log-odds of the output variable (as in http://www.wright.edu/~thaddeus.tarpey/ES714glm.pdf), then it is trivial that comparing the weights is the same as comparing them in linear regression, which makes perfect sense.