objective in policy gradient equation? I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a continuous variable X is:

But how could that be correlated? I mean I can take $r(s,a)$ as variable $x$, but why that $\pi_{\theta}(s,a)$ could be taken as $f(x)$? Thanks a lot for any hint or link or book for reference.

Assuming $\tau = (\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)$ is a sampled trajectory from $\pi_\theta(\tau)$ with return $r(\tau)=\sum_{t=1}^{T}r(\mathbf{s}_t,\mathbf{a}_t)$ then we define the expected reward (our cost function) as:

$$J(\theta)\equiv E_{\tau\sim \pi_\theta(\tau)}[r(\tau)] = \int \pi_\theta(\tau)r(\tau)d\tau$$

where $\pi_\theta(\tau)$= $\pi_\theta(\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)=p(\mathbf{s}_1)\prod_{t=1}^{T}\pi_\theta(\mathbf{a}_t|\mathbf{s}_t)p(\mathbf{s}_{t+1}|\mathbf{s}_t,\mathbf{a}_t)$.

Notice that the p.d.f $\pi_\theta(\tau)$ here  is a joint probability distribution over states and actions. Thus the expectation comes from:

$$E[X]=\int_{-\infty }^{\infty} \int_{-\infty }^{\infty} xf(x,y)dxdy$$

and this is the reason that you need to integrate over the volume of states and actions. $p(\mathbf{s}_1)$ is a distribution over initial states and independent of the integration over actions so it can get out of the 2nd integral. Taking the log of the joint distribution $\pi_\theta(\tau)$, substituting into $J(\theta)$ and differentiating both sides of the first equation,  you get an equation w.r.t the $\theta$ parameters and quantities that you can compute so you can estimate the gradient of $J(\theta)$.

I am not very familiar with your notation (i think you found it in a master thesis) but this is the general approach.


XGBOOST missing_value feature degrades my performance? I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

the cross-validated performance on the train population stays the same.
BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?

If you have more than 50% of values as missing across any columns you should impute only when you are an SME of the domain or know the reason for missing(Honestly I would drop that column rather imputing). Not only XGboost most of the algorithm would penalise for Missing values and one can increase the performance by imputation. Tip may be try Catboost


Develop classification model, given the k nearest neighbors I have a data set listing attributes of clients - a combination of ordinal, categorical and interval data. I am also given the 5 nearest neighbors within the data set for and have to essentially reverse engineer an algorithm for the nearest neighbors.

Given the complexity of the data I'm quite lost as to how I can work out and then obtain parameters of a distance function. I apologize in advance for the vagueness of the question but I'm not even sure where to begin.

If I understand you correctly, based on the data and the given nearest neighbors, you need to decide what distance calculation was used to determine the nearest neighbors.  It seems unlikely that you are being asked to create an altogether new distance measure, rather explore the different accepted distance measures.

If this is correct, it sounds like all you need to do is run the nearest neighbors recursively with different distance calculations and compare the results to the provided nearest neighbors to determine which measure created the 5 nearest neighbors you were given.


EarlyStopping based on the loss When training my CNN model, based on the random initialization of weights, i get the prediction results. In other words, with the same training and test data i get different results every time when i run the code. When tracking the loss, i can know if the result would be acceptable or not. Based on this, i want to know if there is a way to stop the training if the loss begins by a value superior to a desired one in order to re-run it. The min_delta of the EarlyStopping does not treat this case.
Thanks in advance

You can extend the base Keras implementation of callbacks with a custom on_epoch_end method which compares your metric of interest against a threshold for early stopping.
From the linked article they provide a code sample with a custom callback class + calling that class during model.fit:
import tensorflow as tf

# Implement callback function to stop training
# when accuracy reaches ACCURACY_THRESHOLD
ACCURACY_THRESHOLD = 0.95

class myCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if(logs.get('acc') > ACCURACY_THRESHOLD):
            print("\nReached %2.2f%% accuracy, so stopping training!!" %(ACCURACY_THRESHOLD*100))
            self.model.stop_training = True

# Instantiate a callback object
callbacks = myCallback()

# Load fashion mninst dataset
mnist = tf.keras.datasets.fashion_mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
# Scale data
x_train, x_test = x_train / 255.0, x_test / 255.0

# Build a conv dnn model
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])

model.compile(optimizer='adam', \
                loss='sparse_categorical_crossentropy', \
                metrics=['accuracy'])

model.fit(x_train, y_train, epochs=20, callbacks=[callbacks])

Check the linked article for more details:
https://towardsdatascience.com/neural-network-with-tensorflow-how-to-stop-training-using-callback-5c8d575c18a9


How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries Experts in my field are capable of predicting the likelyhood an event (binary spike in yellow)  30 minutes before it occurs. Frequency here is 1 sec, this view represents a few hours worth of data, i have circled in black where "malicious" pattern should be.
Interactions between the dimensions exist, therefore dimensions cannot be studied individually (or can they?)

I'm trying to build a supervised ML model using Scikit Learn which learns a normal rythm, and detects when symptoms might lead to a spike. I am lost for which direction to take. I have tried Anomaly detection, but it only works for on the spot detection, not prior.

How could I detect "malicious" patterns prior to those events (taking them as target variables) ?

I welcome any advice on which algorithms or data processing pipeline might help, thank you :)

This is a fun problem. This is a time series and from this time series you want to identify the trigger of a certain event. So it is a binary classification problem. Based on the information from the specified window will a spike occur? Yes or No.
The first step is to set up your database. What you will have is a set of instances (which can have some overlap but to avoid bias it is best for them to be independently drawn) and then for each instance a human needs to label if there was a spike or if there was not a spike.
Then you need to identify the time window you want to use for your time series analysis. You have done this and decided 30 minutes is a good start.
Now, you have 6 waveforms in a 30 minute window from which you can extract data to get information about your classification. You can use the raw data samples as your features, but this is WAY TOO many features and will lead to poor results. Thus you need some feature extraction, dimensionality reduction, techniques.
There are a million ways you can extract data from these waveforms. First, ask yourself, as a human what are the telltale signs that these other waveforms should have which would mean a spike would arise. For example, in seismic data, if you see agitation in a waveform from a neighboring town then you should expect to see agitation in your town soon.
In general, I like to extract all the basic statistics from my waveforms. Get the mean, standard deviation, fluctuation index, etc. Get whatever you think might help. Check how these statistics correlate with your labels. The more correlation the better they might be. Then there are some very good techniques for extracting time and frequency information from your time-series. Look into envelope mode decomposition and empirical mode decomposition. I have used empirical mode decomposition successfully on some time series data and obtained far better results than I expected.
Now even though you have your reduced feature space you can do better! You can apply some dimensionality reduction techniques such as PCA or LDA to get a lower dimensional space which may better represent your data. This might help, no guarantees.
Now you have a small dataset with instances that are a Frankenstein concoction which represents your 6 waveforms across the 30 minute window. Now you are all set to select your classifier. You will want a binary classification algorithm, luckily that is the most common. There are many to choose from. How to choose?

How many instances do you have?
$\# instances > 100* \#features$?
Then you are all set to use a deep learning technique such as neural networks, 1D convolutional neural networks, stacked autoencodders, etc...
Less than that!!!!
The you should stick with shallow methods. Check out kernel support vector machines, random forests, k-nearest neighbors etc..

Common misconception: A shallow method CAN and WILL perform better than a deep learning technique if you have properly selected your features. feature extraction is the most important aspect of a machine learning architecture.

I want to use anomaly detection!
This would also work and there are some good techniques that would do this. However, the nature of anomaly detection is to learn the distribution of the nominal case. So you would feed your algorithm all the instances in your dataset that did not result in a spike. Then from this your algorithm would be able to identify when a novel instance is significantly different from this nominal distribution and it will flag it as a n anomaly. This would mean that a spike will occur in your context.
Check out:

Learning Minimum Volume Sets
Anomaly Detection with Score functions based on Nearest Neighbor Graphs
New statistic in P-value estimation for anomaly detection

You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school.


Unsupervised Clustering of 6-d vectors I have 17k vectors, each having 6 points, I want to cluster the vectors based on nature of points, eg. linearly increasing into one cluster, convex in another cluster, concave in another, decreasing in another and so on. What would be a good strategy to do this? I need it to detect outlier vectors.

First of all, most clustering methods will be fine when used right. E.g. with correlation, or appropriate normalization.

Secondly, if you already have patterns in mind, use classification.

Third, in order to find outliers, clustering is not too helpful. Because it is not reliable enough. Rather use one of the clever outlier detection methods such as KNN, LOF, LOOP.

Don't forget detailed and problem oriented preprocessing to prepare your data as good as possible.


Understanding regularization I'm currently trying to understand regularization for logistic regression.

So far, I'm not quite sure whether I really got it. Basically, the problem is that when we add an additional features to a model we might overfit the training set. This leads to an algorithm which perfectly matches the training set, but fails for future values (as it too exactly fits what has been there in the past).

So, this is why we introduce regularization. With this we decrease the impact of the features, so the model may produce an algorithm which does a (seemingly) worse job on the training set, but overall works better.

Is this correct?

That's correct. Without regularization your model would fit to an irrelevant noise present in your dataset. It means that training set will fit better but the overall predictive power will decrease. This is a good article about overfitting and regularization.

Also you can get some intuition by watching these images:
Without regularization
With regularization
You can clearly see that the shape at the 1st image has problems with prediction at the upper left corner but it perfectly fits to the training points.


Calculating error from accuracy score in Sklearn I have one question thats maybe simple, but my brain overfited :)

I wrote a code for simple linear regression in Python and Scikit-Learn, and I calculated score of my model.

        accuracy_test = model.score(x_test, y_test)

I got the score of 0.97. Does that mean that my error is 0.03 (3%)?
For example, Is it correct to say, that If i predict some value with model.predict function, and I get result of 100 with that, that I can say that my result will be in range from 97 to 103 because of my 3% error (score of 0.97)?

Can you say this for accuracy_score also?

From sklearn's documentation for the score function

Returns the coefficient of determination R^2 of the prediction.

R^2 is a measure of how well the variability of the data is explained by the model. So, at 0.97, the model is able to explain that really well. And yes, it is a measure of accuracy for regression models. However, you cannot construct confidence intervals with that estimate.

accuracy_score is a measure of accuracy for classification models. Again, you cannot construct confidence intervals from the accuracy metric.


Replace a value in a column if that value appears only once I have a dataframe, I want to replace the values in one column by "other" if the value count of that value in that column is exactly 1

i        Food_group
0          Flake
1          Flake
2          Flake
3         Almond
4          Drink
5          Drink
6          Flake

I have tried,

data["food_group"] = data.food_group.apply(lambda x: "other" if x.value_counts()==1 else x)

I got error

AttributeError: 'str' object has no attribute 'value_counts'

It is easier if you store the value count separately to avoid redoing it inside the apply loop. You can do it like this:

food_count = data["food_group"].value_counts()
data["food_group"] = data["food_group"].apply(lambda x: "other" if food_count[x]==1 else x)


Fuzzy logic for clasification I am trying to implement fuzzy logic system to classifiy dataset of 12 inputs and 1 ouput. I wanna understand as first taks to fuzzify inputs how Can we set intervals or we need to segment inputs first in order to fuzzify them

below is an example of fuzzification but the chose of the intervals is not clear.

Any suggestion or explication will be appreciated

# Generate fuzzy membership functions
qual_lo = fuzz.trimf(x_qual, [0, 0, 5])
qual_md = fuzz.trimf(x_qual, [0, 5, 10])
qual_hi = fuzz.trimf(x_qual, [5, 10, 10])

I think it would be more helpful for us if you put all yours code and language which are you using. I suppose that's are you using Python. In this case try Python library scikit-fuzzy. Some examples at the link:
https://pythonhosted.org/scikit-fuzzy/auto_examples/index.html


Creating new columns based on 3 column and create new data frame Heading

Consider my data frame
rs123   T   C   0   0   1   1   0   0   1   0   0   1   0   0
rs124   T   C   0   0   1   0   0   1   0   0   1   0   0   1
rs125   A   A   1   0   0   1   0   0   1   0   0   1   0   0

Similarity, i have total 93 columns excluding first three

I want to create my data as

And then transform into new data frame as below

For first row if 1 is present in column 1 then output should be TT
For first row if 1 is present in column 2 then output should be TC
For first row if 1 is present in column 3 then output should be CC

For more detail you can refer below snip

Kindly help me to find solution using python, Its very urgent

Thanks in Advance.

The question could have been framed better. Checkout the code below, in which your final dataframe will be in output.

import pandas as pd

input_array = [["rs123", "T", "C", 0, 0, 1, 1, 0, 0, 1, 0, 0], ["rs124", "A", "G", 0, 0, 1, 0, 1, 0, 0, 0, 1]]

raw_pd = pd.DataFrame(input_array).astype(str)

def change(a):
    if list(a)[2]+list(a)[3]+list(a)[4] == "100":
        return list(a)[0] + list(a)[0]
    elif list(a)[2]+list(a)[3]+list(a)[4] == "010":
        return list(a)[0] + list(a)[1]
    else:
        return list(a)[1] + list(a)[1]

output = pd.DataFrame()

output['S1'] = raw_pd[[1, 2, 3, 4, 5]].apply(lambda x: change(x), axis = 1)

output['S2'] = raw_pd[[1, 2, 6, 7, 8]].apply(lambda x: change(x), axis = 1)

output['S3'] = raw_pd[[1, 2, 9, 10, 11]].apply(lambda x: change(x), axis = 1)

output['SNP'] = raw_pd[0]

Hope this helps ;) Mark this as the correct answer if you have no other doubts.


Matrix factorization for like/dislike/unknown data Most literature focus on either explicit rating data or implicit (like/unknown) data. Are there any good publications to handle like/dislike/unknown data? That is, in the data matrix there are three values, and I'd like to recommend from unknown entries.

And are there any good open source implementations on this?

Thanks.

This is very similar to the netflix problem, most matrix factorization methods can be adapted so that the error function is only evaluated at known points. For instance, you can take the gradient descent approach to SVD (minimizing the frobenius norm) but only evaluate the error and calculate the gradient at known points. I believe you can easily find code for this.

Another option would be exploiting the binary nature of your matrix and adapting binary matrix factorization tools in order to enforce binary factors (if you require them). I'm sure you can adapt one of the methods described here to work with unknown data using a similar trick as the one above.


Predict time to reaction (chemical engineering)? I need a hint on the problem below. This is related to predictive analysis and chemical engineering. I don't background in chemical engineering, and that's why I am looking for some hints. I want to know if there's a technical term for the variable/problem I am working on. This would help better fine my search.
Imagine I have a fluid streamline (a production line with chemicals). On the input of that line, we pour some chemicals and change some parameters (temperature, pressure, ...etc) and we want to predict the time necessary to see change at the end of that line.
We can only control the input parameters, and we want to know how long would it take to see changes at the end of the pipeline (that delay time is the target var).
I know some parts of this will depend on the velocity and length of the production line, but also there will some reactions happening on the road. Anything rings the bill here? I am trying to read some literature review about this, if you have some useful information and know some helpful keys words, please let me know.

Is the term you're looking for simply the rate law of the reaction?
Every chemical reaction has an associated kinetic rate law, consisting of its rate constant, and its reaction orders.  Consider the reaction:
$A + B  -> C$
Due to the conservation of mass, we can effectively model the rate of formation of the product C, as the rate of consumption of either reactants A or B:
$d[C]/dt = -d[A]/dt = -d[B]/dt$
The brackets in this equation represents the concentration of each respective chemical in the solution.  That is, the how much of each molecule of chemical is in a particular volume of solution.  This rate of formation is effectively the rate of the reaction itself:
$v = d[C]/dt$
with v being the reaction velocity.  As we can see, the reaction velocity is itself a function of the rate of product formation, which is itself directly related to reactant consumption.  This, in turn, also means that an instantaneous reaction velocity is also a function of the concentrations of reactant in that instant.  Typically, to determine the rate law, the initial rate is measured from initial concentrations, as they are the easiest to measure.
But how do we know HOW they are related?
In short: if we don't do an experiment, we don't.
Without doing an experiment, we can only very generally describe the rate law of a reaction.  One of the most common forms of rate laws is the following:
$v = k[A]^x[B]^y$
where k is our rate constant, and x and y are our reaction orders.  The mystery lies with these variables, and they are determined experimentally as they vary with every reaction.
I am far from an expert in kinetics, but your problem must consider the kinetics of the particular reaction being done.  Another quirk of your problem is that the rate constant k itself depends on temperature by a rather involved equation for someone who doesn't know of, or particularly care for the intricacies of chemical kinetics.
Additionally, if I'm understanding your model correctly, you'll also need to consider how the flow of the system itself may or may not impact the concentration of each chemical species.  As far as I understand, the initial addition of reactants will generate a gradient of concentration that, as a result, will impact the kinetics of the reaction as a function of its rate of diffusion through the volume of the reaction. One can reasonably conceive that the flow you're describing can affect this rate of diffusion. This is something that is far beyond my realm of expertise, and is more in the realm of fluid mechanics.  Of course, for the purpose of this problem, you can intelligently cheese these things, especially with the reaction rate issues by looking up relevant values from well-documented processes.
If you want to do some further reading on rudimentary kinetics, consider the following:
https://www.chemguide.co.uk/physical/basicratesmenu.html#top
https://opentextbc.ca/chemistry/chapter/12-3-rate-laws/


Bias variance tradeoff boosting (xgboost) vs random forest (randomized bagging) which to use when? I was looking up differences between boosting an bagging and I see this quoted everywhere

If the classifier is unstable (high variance), then we should apply
Bagging. If the classifier is stable and simple (high bias) then we
should apply Boosting.

also

Breiman [1996a] showed that Bagging is effective on ``unstable''
learning algorithms where small changes in the training set result in
large changes in predictions. Breiman [1996a] claimed that neural
networks and decision trees are examples of unstable learning
algorithms.check last lines

Aren't gbdt's the most preferred/recommended learners in xgboost ?  Which is contradictory considering how decision trees are apparently unstable learners.

The decision trees used in gradient boosting are typically shallow decision trees (with only a few nodes). Limiting the depth or number of nodes in the decision tree makes them simple. This is different from fully developed decision trees used as standalone models.


80-20 or 80-10-10 for training machine learning models? I have a very basic question.

1) When is it recommended to hold part of the data for validation and when is it unnecessary? For example, when can we say it is better to have 80% training, 10% validating and 10% testing split and when can we say it is enough to have a simple 80% training and 20% testing split?

2) Also, does using K-Cross Validation go with the simple split (training-testing)?

1) In 80-10-10 scheme, 80% is for training, 10% is for validation and 10% is for testing. Validation set required to search for the optimal hyperparameters.

For models having no hyperparameters, it doesn't do much good to use a validation set (although, it is still useful in determining when to stop the training of the model using early stop). In such situation, one might just keep 80% as training set and 20% as testing set.

2) Yes, K fold CV can be used with simple split.


how to impute missing values on numpy array created by train_test_split from pandas.DataFrame? I'm working on the dataset with lots of NA values with sklearn and pandas.DataFrame. I implemented different imputation strategies for different columns of the dataFrame based column names. For example NAs predictor 'var1' I impute with 0's and for 'var2' with mean.
When I try to cross validate my model using train_test_split it returns me a nparray which does not have column names. How can I impute missing values in this nparray?
P.S. I do not impute missing values in the original data set before splitting on purpose so I keep test and validation sets separately.

Can you just cast your np.array from train_test_split back into a pandas.DataFrame so you can carry out your same strategy. This is very common to what I do when dealing with pandas and scikit. For example,
 a = train_test_split
 new_df = pd.DataFrame(a)


What undergraduate degree should I take to get into data science? I'm going to do an undergraduate degree next year. Data Science major is there but I was wondering if a statistics major with a minor in data science would be more valuable? Which degree would be more in demand?
I am also planning on doing as many internships as possible to gain practical skills and experience. I am also contemplating the value of a masters degree, and whether it is worth the time and money, or if an undergrad+internships will suffice. I already know Python and I will learn R and SQL as well. Whether this is in university or otherwise (or both).
I will want to specialise in data analysis using statistical analysis paired with artificial intelligence.
EDIT: I’m in Australia and looking to go to the University of Melbourne (I’m pretty sure its #1 in Australia for computer science and math/statistics)

If you can manage it, I suggest focusing more on Statistics. Statistics can be a demanding field, and takes a lot of effort to attain a degree. Of course, a Data Scientist also needs several skills besides Statistics, such as Software, Data Engineering and Visualization, but Statistics knowledge/skills are the most rare to find, and therefore can be the most valuable.
I worked the last 8 years in a data science department and found that when looking to hire a data scientist, it was often difficult to find someone with a solid statistical background. Most candidates had software, engineering, data and visualization skills. However, someone with a stronger statistics background would be hired over the other applicants, and usually with a higher salary.
Internships are very valuable, they will help you get your foot in the door before you graduate. Some companies may even offer you a job during your internship if they like your work. Some of the bigger companies will pay for your continuing education as they will get a better employee. It is a good investment for both you and them.
A Master's degree will also be beneficial, but may limit your chances to get a lower paying entry level job. Larger companies will usually prefer a Master's over a Bachelor's degree.
Response to your followup question:

I strongly suggest doing the internships, they can dramatically
improve your chances to get a job when you graduate, especially with
the companies you did the internship with.
The choice between Masters with little experience vs Bachelors with
several internships would be much more difficult to make.  Overall,
the better choice would be to get the Masters degree with a few
internships along the way.
Also, I suggest going with Statistics all the way, Bachelors and
Masters. Several of the topics in Data Science will evolve over the
6+ years it takes to get the Masters, but the underlying Statistics
will change very little.
Maybe get a minor degree in Data Science or a Statistics degree with
an emphasis in Data Science.
You can also consider looking for a job in between the Bachelors
degree and Masters, and if you find a good job, you can always get
your Masters degree while you are working and earning money.


ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python I have written the following code for a neural network to perform regression on a dataset, but I am getting a ValueError. I have looked up to different answers and they suggested to use df = df.values to get a numpy array. I tried it but it still produced the same error. How to fix this?
CODE
from keras import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split

#Define Features and Label
features = ['posted_by', 'under_construction', 'rera', 'bhk_no.', 'bhk_or_rk',
            'square_ft', 'ready_to_move', 'resale', 'longitude',
            'latitude']

X=train[features].values
y=train['target(price_in_lacs)'].values

#Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state = 23, shuffle = True)

#Model
model = Sequential()
model.add(Dense(10, activation='relu', kernel_initializer='random_normal', input_dim = 10))
model.add(Dense(1, activation = 'relu', kernel_initializer='random_normal'))

#Compiling the neural network
model.compile(optimizer = Adam(learning_rate=0.1) ,loss='mean_squared_logarithmic_error', metrics =['mse'])

#Fitting the data to the training dataset
model.fit(X_train,y_train, batch_size=256, epochs=100, verbose=0)

ERROR
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int).

Before fitting the model the training data write the below 2 lines:
X_train=np.asarray(X_train).astype(np.int)

y_train=np.asarray(y_train).astype(np.int)


Error while plotting Logistic Regression Classification I was trying to plot by using the following code

# Visualising the Training set results
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, Y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('red', 'green')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Logistic Regression (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

I'm getting the following error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-26-af47df65cc77> in <module>
      2 from matplotlib.colors import ListedColormap
      3 X_set, y_set = X_train, Y_train
----> 4 X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
      5                      np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
      6 plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
   2978             if self.columns.nlevels > 1:
   2979                 return self._getitem_multilevel(key)
-> 2980             indexer = self.columns.get_loc(key)
   2981             if is_integer(indexer):
   2982                 indexer = [indexer]

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2895                 )
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:
   2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

TypeError: '(slice(None, None, None), 0)' is an invalid key

Datasets:

X_train

     Pclass  Sex        Age     Fare  Embarked
0         3    1  22.000000   7.2500         2
1         1    0  38.000000  71.2833         0
2         3    0  26.000000   7.9250         2
3         1    0  35.000000  53.1000         2
4         3    1  35.000000   8.0500         2
..      ...  ...        ...      ...       ...
886       2    1  27.000000  13.0000         2
887       1    0  19.000000  30.0000         2
888       3    0  29.699118  23.4500         2
889       1    1  26.000000  30.0000         0
890       3    1  32.000000   7.7500         1

[891 rows x 5 columns]

Y_train

     Survived
0           0
1           1
2           1
3           1
4           0
..        ...
886         0
887         1
888         0
889         1
890         0

[891 rows x 1 columns]

The error occurs because X_train and Y_train are pandas dataframes and you are trying to directly access them as arrays. There are two solutions:

Convert X_train and Y_train to arrays before using them as:

  X_set, y_set = X_train.values, Y_train.values

Whenever you are accessing their values use iloc so everywhere replace X_set by X_set.iloc


Do we need to tune same model differently for different window sizes in time series data classification? I am currently working on the time series data classification problem using deep learning. As we all know that in time series, we process the time-series data sequentially for some time steps at a time through the model which is called as a window. We slide the window and the next window is our input again. Since I have just started working in deep learning and time-series domain, my question is if we tune some deep learning model for some window size , let's say 10 then should the same tuned model be used to get the near accuracies for some different window(say 15) on the same data or the model should be tuned again for the latter window.

Currently,if I am using the same model for a different window then I get the accuracy decreased by 4-5% from the previous window. From this I believe that window size again is a hyperparameter which when changed requires the model to be tuned again.

So, what is the right thing in this case? Should the same model be used or should it be tuned again? I tried searching this online but I couldn't get any help on this. Help is appreciated as I am new to this domain.

In my opinion it should be tuned again.
Reasoning:
Using different window size is almost equivalent to using different features(as in a non time-series modeling). Tuning hyper-parameters is usually done after features were selected(I.E for a given set of features).


does storing file in hdfs parallelize it for Spark? For Spark's RDD operations, data must be in shape of RDD or be parallelized using:

ParallelizedData = sc.parallelize(data)

My question is that if I store data in HDFS, does it get parallelized automatically or I should use code above for using data in Spark? Does storing data in HDFS makes it in shape of RDD?

As you can see for the following examples presented here at the documentation: https://spark.apache.org/examples.html
you're able to read directly from HDFS without much trouble. And spark will parallelize the data for you properly.

We only use parallelize when you're coming data structures that were built in Scala themselves (like a val assignment or something like that).


Summarize and visualize a CSV in Java/Scala? I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and max of the data in a CSV file for each numeric (integer or real numbers) dimension. The standard deviation would be cool as well.

I would also like to generate some plots to visualize the data, for example 3 plots for the 3 pairs of variables that are more correlated (correlation coefficient) and 3 plots for the 3 pairs of variables that are least correlated.

R requires only a few lines to implement this. Are there any libraries (or tools) that would allow a similarly simple (and efficient if possible) implementation in Java or Scala?

PD: This is a specific use case for a previous (too broad) question.

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.

I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.


Decision Tree Induction using Information Gain and Entropy I’m trying to build a decision tree algorithm, but I think I misinterpreted how information gain works.

Let’s say we have a balanced classification problem.
So, the initial entropy should equal 1.

Let’s define information gain as follows:

info_gain = initial_entropy weighted_average(entropy(left_node)+entropy(right_node))

We gain information if we decrease the initial entropy, that is,
if info_gain > 0. If info_gain == 0
that means

weighted_average(entropy(left_node) + entropy(right_node)) == initial_entropy.

Let’s say we have 4 features such that

weighted_average(entropy(left_node) + entropy(right_node)) is in this order

  wa_of_feature_0 > wa_of_feature_1 > … > wa_of_feature_4.

The more info_gain is larger than 0, the more the feature makes order in system.

So, based on our 4 features maximum information gain will be

info_gain_max = initial_entropy - wa_of_feature_4

since that would give us bigger number than using wa_of_feature_n where 1<=n<4.

Is this the correct interpretation?

Let's us start by defining what are you trying to achieve in Decision tree.

So we want a tree which correctly classify the data?For that out of number of features available I want to select those features which gives me the best information about my classes i.e. How well a feature split my data? which shall help me classify better.

Now entropy is such techniques which helps me understand how "Pure" my subset it, i.e. if i select feature_1 to split upon how well it shall split my data that most of the class labels are from same class.

So Entropy is an impurity measure. If all instances are from same class then there is 0 impurity hence E = 0, if there are same number of instance from both classes then impurity is highest E=1.

Now, we need to select which attribute is the best to split upon first and then recursively which ones are later on ?

Here comes the Information gain.

Information gain just tells me how much useful/good an attribute is from rest of the attributes. For that we compare the entropy of the "Parent Node" before splitting to the impurity of "child Nodes" after splitting. Larger the difference , better the attribute test condition.

Higher gain = purer class

  So, the initial entropy should equal 1.

1 is the highest entropy it means if I have 4 instance, 2 says +ve and 2 says-Ve hence its highly impure. It could very well be 0 if classes are like 4+ and 0 -ve. so on as per the calculations.

  We gain information if we decrease the initial entropy

this initial entropy is the entropy of the data set before splitting or entropy of the parents node.It depends on the data.

  The more info_gain is larger than 0, the more the feature makes order
  in system.

No, Its not compare with 0 but the gains of all the attribute with each other,

  gain(w_1) = .4, g(W_2) = 1.2

Then highest gain is of W_2 hence DT will use W_2 for the split.

Imagine if you have a dataset with 3 attributes A1, A2, A3, then which attribute you should test first?

So, compute the entropy of complete set, i.e. E(complete)

same way E(A1), E(A2), E(A3),

Now

gain(A1) = E(complete)-E(A1) = .2, gain(A2) = E(complete)-E(A2) = .5,gain(A3) = E(complete)-E(A3) =1

highest gain = 1, hence we shall split on A3.

If there is same gain for attributes then we take the order in consideration.


How to use ARIMA to predict time series? Time            Volume
9/30/2019          2000
10/1/2019          1800
10/2/2019          1600
10/3/2019          1400
10/4/2019          1200
10/5/2019          1000
10/6/2019           800
10/7/2019           600
10/8/2019           400
10/9/2019          2000
10/10/2019         1800
10/11/2019         1600
10/12/2019         1400
10/13/2019         1200
10/14/2019         1000
10/15/2019          800
10/16/2019          600

  I have this dataset with Volume consumption of a tank over period of time. The volume decreases with time and on 10/9/2019 the tank has a load-up and is full tank. I have data for over a year and would like to use ARIMA to predict when would the next load-up happen based on previous consumptions.

ARIMA models try to capture autocorrelation in your time series in order to make predictions. It is composed of three main models: AR, MA and I. So, you build your model to predict future values based on a linear combination of past values, linear combination of past errors and a differencing term (I) that accounts basically for a trend. You should get more information here: Forecast: Principles and Practice
My question is: why do you believe the tank volume is predicted by its own past values? When you ask how to use ARIMA, it is straight forward, however, I am concerned that you are not think properly about your problem. Isn't there an exogenous variable you could use? Maybe try a different model? A physical one?
If you really want to use ARIMA, you need to check if your data is not a random walk, plot autocorrelation and partial autocorrelation functions to see analyze which coefficients are more appropriate for AR and MA terms. Check for stationarity. Or simply use autoarima although I don't recommend doing that for the first time.


Concept of Mutual Information I want to get mutual information in iris dataset to select best feature but i confused about mutual information.

What is concept of mutual information for selecting feature?
Can anyone explain it in simple way?

  You do not really understand something unless you can explain it to
  your grandmother.

  Albert Einstein

Mutual information (MI) quantifies the amount of information needed to express one variable with the help of another variable. If two variables are independent of each other, their MI is zero, else it will be greater than zero. MRMR, a mutual information based feature selection uses MI, which considers a feature effective if it has maximum MI with its class label(maximum relevance) and minimum MI with rest of the features(minimum redundancy). This combination of maximum relevance and minimum redundancy will ensure better performance with smaller feature dimension.


Dividing data between test, learn and predict I was posting on stats.stackexchange but perhaps I should be posting here.

Context. Subscription business that charges users a monthly fee for access to the service. Management would like to predict "churn" - subscriptions who are likely to cancel. Management would like to create an email sequence in attempt to prevent high risk accounts from churning, perhaps with a discount code of some sort. So I need to identify those accounts at risk of leaving us.

I have a dataset with say 50k records. Each line item is an account number along with some variables. One of the variables is "Churned" with a value of "yes" (they cancelled) or "No" (they are active).

The dataset I have is all data since the beginning of time for the business. About 20k records are active paying customers and about 30k are those who used to be paying customers but who have since cancelled.

My task is to build a model to predict which of the 20k active customers are currently likely to churn.

Here is where I have tied my brain in a knott. I need to run the model (Predict) on the 20k records of active customers.

How do I split my data between training, test and predict?

Does predict data have to be exclusive of train and test data?

Can I split the entire dataset of 50k into 0.8 train and 0.2 test, build a model and then predict on the 20k active accounts? That would imply I'm training and testing on data that I'm also going to predict on. Seems "wrong". Is it?

Supervised Learning:

Do you have a saved time history of the data?  For a supervised learning set you need some churned="No" cases and some churned="Yes" cases, but it sounds like you only have churned="Yes" and the unknown cases e.g. current customers who may or may not churn.  With some time history you can go back in time and definitively label the current customers as churn="No".

Then it is very easy to split up the data.  And no, you probably don't want to predict on any data that you trained on since you can only train on it if you already know the solution so it will be a waste of time and throw off any metrics you might use to assess accuracy (precision/recall/F1) in the future.

Unsupervised Learning:

If you don't have saved time history of the data then this is an unsupervised learning set for which you have churned="yes" and churned="maybe".  You could then employ anomaly or outlier detection on this set.

novelty detection:
    The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.

outlier detection:
    The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.

You can do either one but novelty is more powerful.  This is kind of a flip around as the novelty here is Churned="No" since all of your data is the confirmed Churn="Yes" cases.

Hope this helps!


Using scikit-learn FeatureHasher I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:
mail_id
DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=
UGATDXARg7jMEInKH7oXgty2nwxnwD2l0OW/8Nsa0MI=
qE9zgWiITYA97RUiN4X/t9IVWLViLz+lKijaYegyBiQ=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
4+EEK8RbNYwuFCHznY9XSRCV4Yek60bHVgnP3jtjjzk=

After doing a lot of analysis on my data, I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently?

I'd say there are pros and cons of using FeatureHasher for this purpose.
If you really striving to use it, then just instantiate it like this:
In [1]:
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=5, input_type='string')
f = h.transform(mail_id)
f.toarray()

Out[1]:
array([[ 1.,  0.,  0.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.],
      [ 1.,  0.,  0.,  0.,  0.],
      [ 0.,  0., -1.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.]])

So, after you have instantiated it, just .transform each your upcoming mail_id and use results in upstream applications ( like online learning, for instance ).
Obviously n_features is some knob to tune. But this has its flip side: the cardinality of mail ids is apriori high, so unless you have very limited amount of users you will need enormous n_features to minimize collisions.
The better would be to take logs, where your ids coappear, and learn item2vec style model. This will deliver much denser (and meaningful) representation of mail_ids than FeatureHasher would do.
Also, take a look at this.


Navigating the jungle of choices for scalable ML deployment I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model was built with Python and Keras. It takes in a CSV table of inputs and spits out the corresponding CSV table of predicted outputs.)

My naive "vision" is that I'd have the model reside on a single (master) machine, whereas the data would be equally distributed among several units (whatever unit means: nodes in a "cluster", CPUs, GPUs, ... ?) The model would be projected onto these units and the learned parameters would somehow all synchronize back to the master unit. Similarly, in the case of serving, the same model would be applied to the data that resides on the different units. Does this "vision" sound reasonable? (I have had some experience with parallel computing with MPI and I vaguely remember that's how things work.)

If I were to start from a blank slate, what architecture/infrastructure should I choose to deploy my model in a scalable fashion? Below are some of the confusingly many options I have read about. (I hardly master what each of these things do, so please forgive me if it looks like a laundry list of disparate technologies.)

Kubeflow (i.e., Kubernetes)
PySpark
Amazon SageMaker
Distributed TensorFlow
Dask
Apache Mesos
REST API
Hadoop
... and dozens more

As a pure ML guy (read: Python, Keras, pandas guy coding on a laptop), I'm out of my depth with all the infrastructure jargon that comes with the above links. It's therefore overwhelming to find a starting point, or some kind of "Hello World" example I could relate to. All I want is an architecture from which I can transpose my code and have it run in an efficient and scalable manner. Which one does the job? Despite all the hype around ML, there does not seem to be any comprehensive map or comparative review of all these solutions. I did find some comparisons, for example between Spark and SageMaker, or Spark and Dask, but given my illiteracy in these subjects, they only add to the confusion.

If you're looking for distributed training, you're probably looking for Apache Spark. It's not itself a training library (although it comes with Spark MLlib which implements most common algorithms in a distributed way), but Spark core is something other implementations often build on.

Pyspark is just the Python API for Spark. Mesos is a cluster manager, like YARN, that Spark can use. Hadoop is for our purposes here a storage system (HDFS) and YARN, so might have the data you're reading. Spark builds on that too.

Kubeflow is, AFAIK, a toolkit for running many relevant data science things on the Kubernetes resource manager. If you're using K8S this will help you get some common training and notebook tools running. I don't know how much it specifically assists distributing training.

Dask is a framework for distributing Python code but I haven't used it; I don't think it's a training library per se. Sagemaker is a hosted notebook and serving tool, mostly. (Plug: if you're interested in Sagemaker, but also using Spark, you'd probably like our Databricks platform even more.)

Many libraries have distributed implementations now. TensorFlow has its own distributed mode. Horovod from Uber also helps distributed TensorFlow in a more efficient way. There's TensorFlowOnSpark too. (Plug again, because this is near and dear to my heart: HorovodEstimator = Horovod + Spark)

MXnet, xgboost also have distributed implementations on Spark. There's BigDL and deeplearning4j, also distributed deep learning for Spark.

And there's more. You're probably going to run into Spark no matter what as its even more commonly used for ETL and other processing related to data science. Then pick your preferred platform. Then pick some distributed training tools to try on your problem.


Adjust predicted probability after smote i  have an imbalance data set and I used smote to oversample the minority class and undersample the majority class. now, I want to check the test AUC using predict_proba of the model.

I have two questions: 1. Do I have to correct the probability if I am comparing AUCs? 2. How can I correct it (a combination of undersampling and oversampling!)

No, any adjustment to the probabilities will presumably be monotonic, so the rank-ordering of the predictions will be the same, so the AUC will be the same.
See, e.g., https://datascience.stackexchange.com/a/58899/55122

See also the more complex "probability calibration" techniques.

Also, if you see better results after smote+undersampling, and can share your data and work, I'd be very interested.  I haven't yet seen an example where training on the original dataset doesn't do just as well (with appropriate thresholding).


Contemporary alternatives to SIFT for image feature extraction? I've been learning about SIFT and all the ways its descriptors can be used to do different tasks.

I am particularly interested in the way SIFT can be used for image classification. (e.g. A 2006 paper by Niester & Stewenius relies on SIFT descriptors to build a vocabulary tree).

However, as of 2017, 11 years later, Deep Learning has been replacing the classical approaches in many ways.

What are some alternatives to find feature descriptors for images that, as of 2017, have shown more promising results?

The Capsule Networks by Geoff Hinton et al. are based on a similar idea to SIFT: in fact, if you look at the 2011 paper, Transforming Auto-Encoders by Hinton et al. they explicitly cite SIFT as an inspiration for Capsules.

The main idea is that they construct a network with a new type of unit called a Capsule which outputs a vector (rather than neurons outputting scalars, as we know in traditional neural networks).

You can think of these vectors as an analogue to the SIFT keypoints and feature descriptors, but with the benefit that they can be learned via back-propagation.

There is a recent article, Dynamic Routing Between Capsules by Sabour, Frosst and Hinton elaborating on this which shows very promising results.

I have written a short summary of Capsule Networks on the AI Stack Exchange, see the question What's the main concept behind Capsule Networks?.


Most Efficient Machine Learning Algorithm for Text Analysis Looking at Understanding Convolutional Neural Networks for NLP, Convolutional Neural Networks (CNNs) seem to be suitable not only for image recognition, but also for NLP.

Are CNNs in general the best choice for NLP text analysis, e.g. for sentiment analysis?

If not, is there an overview or comparison of different algorithms with regards to recognition performance (F1 score or something similar)?

I don't have any reference right now, but you should find a lot of papers about it on Google Scholar. For all I know, CNN and RNN (mostly LSTMs) are the two best types of models for sentiment analysis. You will even find some CNN+LSTM architecture that work really well on this kind of tasks.


Extract data from json format and paste to column using python In my column with json data, I have this list I want to extract to column:
"list":[

{
    "id":"list",
    "item":[
        {
            "value":"Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon"
        },
        {
            "value":"Geruchs- und geschmacksneutral"
        },
        {
            "value":"Kältebeständig bis -60°C"
        },
        {
            "value":"Inklusive Rezeptbuch und 50 Eisstielen"
        },
        {
            "value":"spülmaschinengeeignet"
        }
    ],
    "decorators":[
    ]
},

I have extracted other data using this code but the value was string and not a dictionary:
if 'item' in item and item['item']:
                    if isinstance(item['item'], str):
                        cur_model_info[item['id']] = item['item']
                    elif isinstance(item['item'], list):
                        elements = [element['value'] for element in item['item']]
cur_model_info[item['id']] = ','.join(elements)

I tried to use this for the above format of data but I got this error:
TypeError: sequence item 0: expected str instance, dict found
What should I change in order to be able to export in a separate column the data having each element from the list in a new cell of that column?
expected output

Using the above code, the data is exported in this format

Given that the data snippet you provided is not a valid json structure I assume that the data is a list with dictionaries in it. Using this assumption, you can extract the values to a nested list using a list comprehension:
data = [
    {
    "id":"list",
    "item":[
        {
            "value":"Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon"
        },
        {
            "value":"Geruchs- und geschmacksneutral"
        },
        {
            "value":"Kältebeständig bis -60°C"
        },
        {
            "value":"Inklusive Rezeptbuch und 50 Eisstielen"
        },
        {
            "value":"spülmaschinengeeignet"
        }
    ],
    "decorators":[]
    }
]

print([[i["value"] for i in x["item"]] for x in data])

# [['Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon',
#  'Geruchs- und geschmacksneutral',
#  'Kältebeständig bis -60°C',
#  'Inklusive Rezeptbuch und 50 Eisstielen',
#  'spülmaschinengeeignet']]


shifting specific column to before/after specific column in dataframe In dataframe example :

  medcine_preg_oth medcine_preg_oth1 medcine_preg_oth2 medcine_preg_oth3
0          Berplex           Berplex              None              None
1              NaN               NaN               NaN               NaN
2              NaN               NaN               NaN               NaN
3            obmin             obmin              None              None
4              NaN               NaN               NaN               NaN

'medcine_preg_oth1' 'medcine_preg_oth2' 'medcine_preg_oth3' ,these three columns are in somewhere of dataframe with other columns.

I want to shift these three : medcine_preg_oth1 medcine_preg_oth2 medcine_preg_oth3 to the place of after 'medcine_preg_oth'.

My idea is shifting the specific columns to place after/ before specific columns in dataframe for wider purpose .
please suggest me!
Thanks

If I get the question correct, you just need to change the order of your columns. This can be simply done by reassigning the new order of your columns to the dataframe.

For example:

#['a', 'b', 'c'] <-given columns order
df = df[['c', 'b', 'a']]

You can also use the built-in function reindex to accomplish you task as follow:

cols = df.columns.tolist() #['a', 'b', 'c']
new_cols = [['c', 'b', 'a']]
df = df.reindex(columns=new_cols)

EDIT:

If you have a large number of columns, the problem will arise in how you get the new_cols list. To do this you can use list indexing and slicing. Firstly get the index of columns you wnat to replace by using:

df.columns.get_loc("b") #1

Now suppose you have 699 columns and want to place the 100th and 200th column after the 7th one, you can do this:

cols = cols[0:7] + [cols[100]] + [cols[200]] + cols[8:100] + cols[101:200] + cols[201:]

You can now use this column to change the order of your columns in the above mentioned way. The expression will vary depending on your use case.


F1_score(average='micro') is equal to calculating accuracy for multiclasification Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case?
I have tried with different values and they gave the same answer but I don't have the analytical demonstration.
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]
print(f1_score(y_true, y_pred, average='micro'))
print(accuracy_score(y_true,y_pred))

# 0.3333333
# 0.3333333

In classification tasks for which every test case is guaranteed to be assigned to exactly one class, micro-F is equivalent to accuracy.

The above answer is from:
https://stackoverflow.com/questions/37358496/is-f1-micro-the-same-as-accuracy

More detailed explanation:
https://simonhessner.de/why-are-precision-recall-and-f1-score-equal-when-using-micro-averaging-in-a-multi-class-problem/


How can I check the correlation between features and target variable? I am trying to build a Regression model and I am looking for a way to check whether there's any correlation between features and target variables?

This is my sample dataset

     Loan_ID    Gender  Married Dependents  Education Self_Employed ApplicantIncome\

0   LP001002    Male    No         0        Graduate      No            5849
1   LP001003    Male    Yes        1        Graduate      No            4583
2   LP001005    Male    Yes        0        Graduate     Yes            3000
3   LP001006    Male    Yes        0        Not Graduate  No            2583
4   LP001008    Male    No         0        Graduate      No            6000

CoapplicantIncome  LoanAmount   Loan_Amount_Term  Credit_History Area Loan_Status
      0.0               123          360.0            1.0        Urban     Y
      1508.0          128.0          360.0            1.0        Rural     N
      0.0              66.0          360.0            1.0        Urban     Y
      2358.0          120.0          360.0            1.0        Urban     Y
      0.0             141.0          360.0            1.0        Urban     Y

I am trying to predict LoanAmount column based on the features available above.

I just want to see if there's a correlation between the features and target variable. I tried LinearRegression, GradientBoostingRegressor and I'm hardly getting a accuracy of around 0.30 - 0.40%.

Any suggestions on algorithms, params etc that I should use for better prediction?

Your data can be put into a pandas DataFrame using

import pandas as pd
data = {'Loan ID': ['LP001002', 'LP001003', 'LP001005', 'LP001006', 'LP001008'],
        'Married': ['No', 'Yes', 'Yes', 'Yes', 'No'],
        'Dependents': [0, 1, 0, 0, 0],
        'Education': ['Graduate', 'Graduate', 'Graduate', 'Not Graduate', 'Graduate'],
        'Self_Employed': ['No', 'No', 'Yes', 'No', 'No'],
        'Income': [5849, 4583, 3000, 2583, 6000],
        'Coapplicant Income': [0, 1508, 0, 2358, 0],
        'LoanAmount': [123, 128, 66, 120, 141],
        'Area': ['Urban', 'Rural', 'Urban', 'Urban', 'Urban'],
        'Loan Status': ['Y', 'N', 'Y', 'Y', 'Y']}
df = pd.DataFrame(data)

Now to get a correlation we need to convert our categorical features to numerical ones. Of course the choice of order will affect the correlation but luckily all of our categories seem to be binary. If this is not the case you will need to devise a custom ordering.

df = pd.DataFrame(data)
df['Married'] =df['Married'].astype('category').cat.codes
df['Education'] =df['Education'].astype('category').cat.codes
df['Self_Employed'] =df['Self_Employed'].astype('category').cat.codes
df['Area'] =df['Area'].astype('category').cat.codes
df['Loan Status'] =df['Loan Status'].astype('category').cat.codes

Now we can get the correlation between the 'LoanAmount' and all the other features.

df[df.columns[1:]].corr()['LoanAmount'][:]

Now using some machine learning on this data is not likely to work. There just is not sufficient data to extract some relevant information between your large number of features and the loan amount.

You need at at least 10 times more instances than features in order to expect to get some good results.

To only obtain the correlation between a feature and a subset of the features you can do

df[['Income', 'Education', 'LoanAmount']].corr()['LoanAmount'][:]

This will take a subset of the DataFrame and then apply the same corr() function as above. Make sure that the subset of columns selected includes the column with which you want to calculate the correlation, in this example that's 'LoanAmount'.


Grid seach is unavailable for Keras in case of multiple outputs? I do experiments with the following Keras architecture with multiple outputs:

def create_model(conv_kernels = 32, dense_nodes = 512):
    model_input=Input(shape=(img_channels, img_rows, img_cols))
    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Dropout(0.25)(x)
    x = Flatten()(x)

    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)

    x1 = Dense(nb_classes, activation='softmax')(conv_out)
    x2 = Dense(nb_classes, activation='softmax')(conv_out)
    x3 = Dense(nb_classes, activation='softmax')(conv_out)
    x4 = Dense(nb_classes, activation='softmax')(conv_out)

    lst = [x1, x2, x3, x4]

    model = Model(inputs=model_input, outputs=lst)
    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

    return model

When I try to apply grid search this way:

model = KerasClassifier(build_fn=create_model1, epochs=nb_epoch, batch_size=batch_size, verbose=0)
param_grid = dict(conv_kernels = [16, 32, 64], dense_nodes = [128, 256, 512])
grid = GridSearchCV(estimator=model, cv=4, param_grid=param_grid, n_jobs=-1)
grid_result = grid.fit(X_train, Y_train)

I get the following error message:

  ValueError: Found input variables with inconsistent numbers of samples: [9416, 4]

9416 is the number of training examples, 4 is the model output number.

What is the problem here? Grid search is unavailable for multiple outputs? If so, what is the best way to apply parameter search (apart from pure manual method)?

It does not appear possible  as stated here.  The problem is basically how sklearn expects it's input.  Currently it only accepts a 2d array.  I think it'll be rather straight forward to go to sklearn wrapper and modify is so that it can accept multiple arrays.


Incorrectly applying random forest model? I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.

Context: I am creating a random forest model to predict employee voluntary turnover.

Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.

I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).

Possible Explanations for this that I can think of:

1) Data set doesn't seem too imbalanced:  8054 non terms vs 2158 terms

2) Overfitting? But the test set doesn't drop substantially in accuracy

3) high correlated predictor variables?

Code Attached.

Thanks!

I think you correctly identified the issue: if your model tries to classify whether an employee stays or leaves, by definition every employee "stays" as long as they are an employee in the company.

A possible direction would be to design the response variable as "does the employee leave within a year?" (or any specific period of time). This way you can use the records of past employees who have left and past employees who have stayed across time, i.e. you can have several instances corresponding to the same employee at different times. You could add features such as "had a raise or promotion in the past 2 year" for instance.


Given a single discrete data set, how should I divide it into training data and test data? I have a dataset in libSVM format consisting of 6000 entries, each with 5 indices, and each index has a binary value 1 or 2. Each of the 6000 entries has a label of 1 or 0, and I am trying to use various machine learning algorithms to determine the correct label (0 or 1) given a particular set of 5 indices/values.

For example, consider the following dataset (the real one is 6000 lines):

0 101:1 102:1 103:0 104:1 105:1
0 101:0 102:1 103:0 104:1 105:1
0 101:0 102:1 103:1 104:1 105:1
1 101:1 102:1 103:1 104:1 105:1
1 101:0 102:1 103:0 104:0 105:1
1 101:1 102:1 103:1 104:0 105:0
1 101:0 102:1 103:0 104:0 105:0

For an algorithm that predicts binary classification, like xgboost, conceptually, how do I first use my dataset to train the model, and then apply the model to the data?

I ask because xgboost asks for two files, a data training set and a data test set. It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

Any help in understanding this concept is much appreciated.

In machine learning, it is important to test out the model that you have built on your training data. This is to prevent overfitting. This is why you must split your data into testing and training. There are many different ways to split testing and training. You can randomly split the data set so that 80% of the samples are training and 20% are testing. Something else you may want to consider is using stratified sampling so that the positive labels occur in both testing and training. This is especially important if you only have a few positively labeled samples, as you could easily end up with a test set without any positive samples. In python there is an argument ‘stratify’ that you can use so that the split has balanced classes.


Difference between PCA and regularisation Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature selection. I have to admit, I am not quiet familiar with the difference between dimensional reduction and feature selection.

Lasso does feature selection in the way that a penalty is added to the OLS loss function (see figure below). So you can say that features with low "impact" will be "shrunken" by the penalty term (you "regulate" the features). Because of the L1 penalty, the $\beta_i$ can become zero (which is not the case with Ridge, L2). In the Lasso case you would "eliminate" a feature when it is "shrunken" to zero, and you could call this feature selection. Lasso can be used in "high dimensions", i.e. when you have many features ("columns") but not so many observations ("rows").

Principle components work in quite a different way. The first principle component is a normalised linear combination [of the original features] which has the largest variance. So you kind of "transform" the original features to a principle component (which is a "new feature" derived from the original ones), where you try to capture as much variance as possible in one principle component.
Principle components are uncorrelated (orthogonal). This can be very helpful when you do linear regression, in which (high) correlation between features can be a real problem. I see PCA as a tool for dimensionality reduction (not so much feature selection), since you can express many features in a (smaller) number of principle components.
So maybe a little too brief summary:

Lasso: "shrink" the estimated coefficients for features which are not too useful (but leaves the features as they are)
PCA: "combine" several features into one or more orthogonal "new" feature(s) (principle components) and use them in some type of model

For more details, refer to "Introduction to Statistical Learning" (available for free online). Ch. 6.2.2 covers the Lasso, Ch. 10.2.1 covers PCA.


How can I improve the accuracy of my neural network on a very unbalanced dataset? I have a dataset which contains data for about accidents. The dataset consists of about 15.000 entries and I can't get more. The Distribution is as follows:

88.6% of the data are class 1 accidents
10.6% of the data are class 2 accidents
0.8% of the data are class 3 accidents

As you can see, the biggest part of the training data belongs to one class. I have only very few examples for class 3 accidents (about 100 rows out of 15.000), but it would be most important to classify class 3 accidents correctly.

I trained a pretty standard deep neural network on the data and got an accuracy of ~93% on the validation set. I used a custom Tensorflow estimator with an AdamOptimizer and tuned the parameters as good as possible.
The Problem is, the network still classifies most of the accidents as class 1 accidents. So if I have for example 25 class 3 accidents in the validation set, the network misclassifies 10 of them as class 1. I want to improve that.

Are there any methods to improve the performance in this case? The obvious choice would be to get more data of class 3 accidents, but sadly that is not possible.
Does it make sense to show the existing class 3 data multiple times? So for example, train 5 Epochs with all data and then 3 additional Epochs with just the class 3 accidents?

Or could I do something during the data preprocessing? Right now I'm MinMax-Scaling the input data to get to the [0, 1] interval. Is there maybe any other way to emphasize outliers more? (If you assume outliers mostly belong to class 3)

I hope someone knows some methods to increase the accuracy in this case.

EDIT:
The Dataset has mostly categorical columns like:

Street Class (e.g. highway or country road)
Light (e.g. "good")
Weather (e.g. "rainy" or "sunny")
...

Additional it has these columns:

Accident Date (just month and day)
Age
Time of the day
Number of injured persons
Number of Vehicles

So an entry might look like this:

{
 street_class: 'highway',
 light: 'daylight',
 date: '23. Jan',
 age: 59,
 injured_persons: 2,
 vehicles: 2,
 time: 1724,
 label: 1
 ...
}

I'd try some kind of data augmentation, but from your question is not clear what type of data you have and it's impossible to suggest a solution.

Try to add an example of the data to your question.


What is difference between a Data Scientist and a Data Analyst? https://www.datacamp.com/community/tutorials/learn-data-science-infographic
https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic

These links contain almost everything but not the difference between data science and data analytics.

Is data analytics a part of the data science workflow? Is data analytics a subset of data science?

Please visit this and read the difference between Data analyst and Data scientist.
You will find the above link very interesting that fulfills your need as you will also find the words by Data scientist of LinkedIn.
Important lines from the above link:

Data analysts are masters in SQL and use regular expression to slice and dice the data. With some level of scientific curiosity data analysts can tell a story from data. A data scientist on the other hand possess all the skills of a data analysts with strong foundation in modelling, analytics, math, statistics and computer science. What differentiates a data scientist from a data analyst is the strong acumen along with the ability to communicate the findings in the form of a story to both IT leaders and business stakeholders in such a way that it can influence the manner in which a company approaches a business challenge.


Is LSTM a proper implementation for sequence classification I have 20k samples in two classes namely positive and negative. Samples are formed of 10 digits and within each sample a digit is used exactly once. For instance; 0123456789 and 9876543102 are proper samples as each digit is used once. Order of the digits are what makes samples positive or negative. A classification statement for instance, positive samples can have 2 after 5, but negatives can't. Or a more intricate one like, 90% percent of positive samples have leading 0's and none of negatives have this. For the remanining 10% percent, 512 and 315 are patterns that makes samples positive.
There is a total of 10!=3.6m possible samples. So, in order to carry out a binary classification, what should be the proper ML approach? Can LSTM's handle this? Maybe a primitive Neural Network with 10 inputs and a binary classification? Or I should look for a different ML approach? Thanks.

So, from what I gather, the task is to identify positive and negative sequence of digits, which abide by certain rules.
Depending on the rules themselves: for example if there are a small number of explicit and primitive rules (like each sample sequence must only have one of each digit), then might be better to hard-code the rules into a function and feed sequences through that function.
Machine learning is ideally used when you do not know the rules (function) for how the input is translated into the output.
 If you do not know the rules, then an LSTM would be a good idea since it will account for the sequence of digits themselves, when classifying the sequences. The input would be the sequence of digits (possibly normalised [i.e. divide by largest digit = 9] e.g. 1 would e represented as 1/9, etc.)
Then this would be a many-to-one architecture, where you feed in to the model the whole sequence of digits and then feed this output into a final softmax layer over the two classes to give you a probability distribution of the classes (i.e. how likely is this sample to be positive?)
A standard neural network would not be recommended since when you feed input into a neural network, the data is fed in parallel and does not take into account the ordering of the digits, which appears to be fundamental to your task.
If you did want to use a neural network, you could potentially use positional encodings so that you can factor for ordering of the digits (https://medium.com/nlp-trend-and-review-en/positional-embeddings-7b168da36605 in the context of Natural Language, but same would apply to your problem).


Garbage collection in Tensorflow 1.14 When a variable is created, it is added to collections to allow sharing and reusing. However, this seems like it's a potential for "memory leaks" (strictly speaking, it's not, since there still exists a reference). How should I deal with this?

If I do something like:

import tensorflow as tf
g = tf.Graph()
with g.as_default():
    tf.get_variable('myvar', shape=(2, 2))
del g

will myvar be garbage collected?

In Tensorflow 1.*, myvar will not be collected even if you lost track of the variable pointing to it, till the Python process terminates.

In Tensorfow 2.* it will be collected by the garbage collector.

For details, see the blog by Tensorflow, under the "No more globals" section.


What if MNIST dataset had another feature MNIST is a famous data set of hand written digits. Suppose we knew who wrote digits for example, female, left handed, 25 yrs old.

How would I use these information in CNN in tensorflow? or any other library.

Digits are images and that what CNN handles well but gender, dominant hand, and age are not images. How would you use that information?

Convolutional layers are useful for images because they take into consideration the neighborhood of pixels. However, for labels like gender and handedness a convolutional layer may not be particularly useful.

However, after the convolutional layers you usually tend to place some densely connected layers. It is there that you may want to add these additional features. When you reshape the 2D matrices which results from the convolutions, you can concatenate the additional features, then feed this new vector to your Dense layer.


When inputting image rgb values to MLP, should I divide by 255? I have an MLP with 3072 input nodes which are for 1024 rgb pixels. My datasets is in an array with each row representing one image and looking like this:

[red_pix1, red_pix2, ..., red_pix1024, green_pix1, green_pix2, ..., green_pix1024, blue_pix1, blue_pix2, ..., blue_pix1024]

Each array value is an integer between 0 and 255.

My question is, before training the network, should I "normalize" my dataset by dividing each element by 255? That way, each input element would have values between 0 and 1. Is this better than having values between 0 and 255?

The component values are often stored as integer numbers in the range 0 to 255, the range that a single 8-bit byte can offer,

Yes, If you divide by 255  the range can be described with a 0.0-1.0 where 0.0 means 0 (0x00) and 1.0 means 255 (0xFF).
Normalization will help you to remove distortions caused by lights and shadows in an image.

Refer to this Normalize RGB


How do I get latent features for a new row of data when doing non-negative matrix factorization? Background: The basic set-up for non-negative matrix factorization (nmf), is that we take a matrix with non-negative elements, X and find two other non-negative matrices H and T such that

  $||X - HT||$ is a minimum

$X$ in this example is a matrix that represents a data set. Rows correspond to samples of data, and columns correspond to features. I understand that nmf is used for feature reduction. My understanding is that you can use the latent features to train a model on.

However, suppose that I use nmf for feature reduction and then get a new row of data and want to find the reduced features for the new data. Do I have to run nmf again with an updated dataset and then retrain my model each time?

Given a dataset represented by a data matrix $X$, we can solve $||X-HT||$ using gradient descent, setting coefficients equal to zero if they become negative. This gives us an $H$. If we get a new row of data $x_{new}$, then we can find the corresponding reduced features $w_{new}$(weights for the latent features) by solving the equation:

$argmin_{w_{new}} = ||x_{new}-w_{new}T||$.


Find and count similar price for each store and output the max count and its corresponding rows My goal is:

for each item in each store,  find the number of items in the same store within $2 price difference (i.e. number of items with similar price within the same store).

Then list all the items which has the greatest number of items.

The dataset has about 100,000 records so efficient is also important. My snapshot is a small subset for implementing ideas.
The first step has got me. I have tried groupby, count, sum. None of these get me anywhere to get the desired output. I have used df.sort_values on store_id and price. Can anyone give me a hint?
Below is some sample of my data.
import pandas as pd
data = {'item_id':  ['6dd5392a9991','363a268ae1bc','fcd248a3fe97','20d197a04656','54c6463ffc87', '1b62f63eac43', '4ed99ff1bcdf', '6e19d5b8e99b','89c9b4655a9d', '16740613e6af'],
        'store_id': ['1d632be3f72c','1d632be3f72c','1d632be3f72c','1d632be3f72c','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e61'],
        'price':  [23.54, 20.61, 13.63, 23.69, 13.79, 14.90,  4.09, 14.30, 4.47, 4.51]
            }

df = pd.DataFrame(data)

It might not be the fastest, but it'll work:
def condition(row1, row2, threshold):
    return abs(row1['price']-row2['price'])<=threshold

def row_counter(this_df, this_row, threshold):
    return this_df.apply(lambda row: condition(this_row, row, threshold), axis=1).sum()

def counter(this_df, threshold):
    return this_df.apply(lambda row: row_counter(this_df, row, threshold), axis=1)

threshold = 2
result = df.set_index('item_id').groupby('store_id').apply(lambda store_data: counter(store_data, threshold))

The result is:
store_id      item_id
1d632be3f72c  6dd5392a9991    2
              363a268ae1bc    1
              fcd248a3fe97    1
              20d197a04656    2
b5d61bc3e61   16740613e6af    1
b5d61bc3e6d1  54c6463ffc87    3
              1b62f63eac43    3
              4ed99ff1bcdf    2
              6e19d5b8e99b    3
              89c9b4655a9d    2
dtype: int64


Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts? Is this some kind of estimation of the relative positions of words in the training texts? are they creating some kind of statistical "distribution" of words? is "cat" usually 2 or 3 words away from "milk" in English language? things have to have a meaning, havent they?  Is BERT just adding some aditional dimensions to the vector space to include info on the relative positions of words?

First, we should clarify that positional encodings are not necessarily applied to words but tokens. Nowadays, using subword tokens is the norm, and infrequent words can be divided into many subword tokens.
Regarding the role of positional encoding: according to research, positional encodings have different roles depending on where they are used:

Transformer decoders for autoregressive language modeling: positional embeddings learn about absolute positions (see What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding).

Masked language models (i.e. BERT): the positional encodings only help differentiating between tokens, and the order does not actually matter much (see Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little).

So, to answer the question:
No, positional encodings do not estimate the relative positions of words in the training corpus texts.


low error, high CV(RMSE)? I am comparing 2 neural network models. I have used the model to make predictions on unseen data.

One model returns an error of 20.9% for y1, 36.6% for y2,
4.53% for y3 on unseen data, and a CV(RMSE) of 19.3.
The other returns an error of 15.5% for y1, 33.8% for y2
and 4.83% for y3 on unseen data, and a CV(RMSE) of 31.5.

I'm struggling to interpret the results. A lower CV(RMSE) is better, yet why do I get a much higher error on unseen data?

When you calculate RMSE error on multi-output models you will normally end up with 3 RMSE's. You can average those errors into one by specifying weights. They basically reflects of bad error on y1, y2 and y3 are for your application.

Even if the outputs are averaged equally, keep in mind that RMSE is "scale sensitive". If y1 value is way higher than y2 and y3 values, it will also produce bigger RMSE.

If you want to get rid of this problem I recommend you to use RMSE% wich is a relative error metric (scale invariant) as mentionned here : https://stats.stackexchange.com/questions/190868/rmse-is-scale-dependent-is-rmse


Why not use Scaler.fit_transform on total dataframe? In sklearn I'm normalizing the data with MinMaxScaler. The example I'm following uses

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train, X_test, y_train, y_test = train_test_split(X_crime, y_crime,random_state = 0)

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Now I wonder why this is done separately on the train and test set, and not on the X_crime dataframe like:

X_crime_scaled = scaler.fit_transform(X_crime)

X_train_scaled, X_test_scaled, y_train, y_test = train_test_split(X_crime_scaled, y_crime, random_state = 0)

the R-squared score is higher and with this option I know all my values are normalized between 0 and 1.

If you would use the scaler on the full dataset you would provide the algorithm with some information about the values in the test set that it would not have otherwise.

This additional information ("data leakage") from your test set to your training set is problematic because it leads to unreliable performance estimates. It is therefore not surprising that you achieve a higher R-squared. Due to the data leakage, this R-squared might be overly optimistic because it could depend on the additional information that you introduced into the training set.

This is particularly true for the MinMaxScaler because it is by definition very sensitive to outliers. The effect will probably less problematic if you use the RobustScaler (or even the StandardScaler).

The same holds for other preprocessing steps like outlier removal, feature selection etc.

If you are worried that your training data does not adequately reflect the true distribution, you can fall back to a cross-validation approach with multiple folds so you can estimate the effect across multiple splits of the data. Again, remember to fit the scaler on all the training folds and apply it to the test fold for every iteration.


Handling nominal category features in decision tree I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation).
One of the answer states that :
Using a OneHotEncoder is the only current valid way, allowing arbitrary splits not dependent on the label ordering, but is computationally expensive.
But if we use LabelEncoder and make our tree go deep enough it will eventually isolate each category and the output remains the same.
So whats is the advantage of OneHotEncoding then?

I would not say that using One-Hot-Encoder to handle nominal features is an optimal way. Even it might affect adversely your Decision Tree model in terms of performance.
Theoretically, a Decision Tree does not need a one-hot-encoding for categorical features, besides it will increase the number of computations and will result in a relatively inefficient model. Unlike non-tree algorithms (e.g. regression algorithms), a Decision Tree cannot be biased by the label encoding of categorical features because it uses splits to reach the optimal result.
Experimental results show that one-hot-encoding gives a rise to sparsity in the trees which might drastically drop the performance of the model. Here is a wonderful analysis and explanation of the effect of one-hot-encoding on Tree-based algorithms. Also, check this useful source.


Cross validation Vs. Train Validate Test I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

Train: we train the model.
Validation: we validate and adjust model parameters.
Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets.  Until here everything is okay. Attached is a picture:

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

As an example, I found the Sci-Kit learn cross validation version as you can see in the following picture:

When doing the splitting, you can see that the algorithm that they give you, only takes care of the training part of the original dataset. So, in the end, we are not able to perform the Final evaluation process as you can see in the attached picture.

Thank you!

scikitpage

If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen. The test set has not been used so far.

Only at the very end the test set is used to test the performance of the (optimized) model.

# example: k-fold cross validation for hyperparameter optimization (k=3)

original data split into training and test set:

|---------------- train ---------------------|         |--- test ---|

cross-validation: test set is not used, error is calculated from
validation set (k-times) and averaged:

|---- train ------------------|- validation -|         |--- test ---|
|---- train ---|- validation -|---- train ---|         |--- test ---|
|- validation -|----------- train -----------|         |--- test ---|

final measure of model performance: model is trained on all training data
and the error is calculated from test set:

|---------------- train ---------------------|--- test ---|

In some cases, k-fold cross-validation is used on the entire data set if no parameter optimization is needed (this is rare, but it happens). In this case there would not be a validation set and the k parts are used as a test set one by one. The error of each of these k tests is typically averaged.

# example: k-fold cross validation

|----- test -----|------------ train --------------|
|----- train ----|----- test -----|----- train ----|
|------------ train --------------|----- test -----|


LSTM get next output with Keras So I'm learning RNN, and tried to do a prediction LSTM, but I do not understand how the output works. I have this LSTM RNN:

lstm_model = Sequential()
lstm_model.add(LSTM(100, batch_input_shape=(BATCH_SIZE, TIME_STEPS, x_t.shape[2]), dropout=0.0, recurrent_dropout=0.0, stateful=False,   kernel_initializer='random_uniform'))
lstm_model.add(Dropout(0.5))
lstm_model.add(Dense(25,activation='relu'))
lstm_model.add(Dense(1,activation='sigmoid'))
optimizer = optimizers.RMSprop(lr=0.003)
lstm_model.compile(loss='mean_squared_error', optimizer=optimizer)

and fit with:

history = lstm_model.fit(x_t, y_t, epochs=300, verbose=2, batch_size=BATCH_SIZE,
                    shuffle=False, validation_data=(trim_dataset(x_val, BATCH_SIZE),
                    trim_dataset(y_val, BATCH_SIZE)), callbacks=[csv_logger])

When I then try to predict using

gotten = lstm_model.predict(x_test_t[-500:],batch_size=BATCH_SIZE)

Will I get only one new output? I'm very confused with the way you run and get predictions

Thanks for the help

With your current model, you will indeed get a single scalar output. This is because the last layer is Dense with just 1 unit.

In other words, with this model you can map a sequence to a single value that may represent, e.g., the label of the series or its next value depending on what target you train the network on.


How much text is enough to train a good embedding model? I need to train a word2vec embedding model on Wikipedia articles using Gensim.

Eventually, I will use the entire Wikipedia for that
but for the moment, I'm doing some experimentation/optimization to improve the model quality and I was wondering how many articles would be enough to train a meaningful/good model? How many examples needed for each unique word in vocabulary?

It is not the number of many articles that matter but the total number of words.
Enough "meaningful/good" is an empirical question that depends on the dataset. One way to test the results of a newly trained model is the Google analogy test set which compares a new model's predicted word to established embedding benchmarks.
As far as minimum number examples needed for each unique token in vocabulary, the general consensus is there should be at least 40 examples per token. If there are fewer than 40 examples for a token, the vector estimates can be unstable and the token should be dropped from training.


Storing and loading bottleneck features for transfer learning on large data sets (Keras) I would like to apply transfer learning on a pretty large image data set in order to solve a classification problem. Currently I load a pre-trained net without the top layers, add my own top layers, freeze the base model and start training.
Since the data set is too big to fit into memory I load the train and validation sets using prefetched TF Datasets:
train_ds = keras.preprocessing.image_dataset_from_directory (training_data_dir,
                                                             batch_size=batch_size,
                                                             image_size=img_size,
                                                             label_mode='categorical')
val_ds = keras.preprocessing.image_dataset_from_directory (validation_data_dir,
                                                           batch_size=batch_size,
                                                           image_size=img_size,
                                                           label_mode='categorical')
train_ds = train_ds.prefetch(buffer_size=buffer_size)
val_ds = val_ds.prefetch(buffer_size=buffer_size)

Now in order to reduce the training time I found a nice approach on this page: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
Before training the model, they first run the base model for each input image and store the output of the base model, that is the bottleneck features. Training is then done on these bottleneck features. That means that during training the base network does not need to be executed, saving a lot of computation time.
Now for their approach they have a data set that fits in memory and the set of all bottleneck features over all images is represented as a single numpy array which is stored in and loaded from a single file.
Unfortunately this is not possible in my case since the original image set is so large it will not fit into memory and also converted to a bottleneck feature set it is still to large. So I am looking for a solution where the bottleneck features of a single image are stored in a single file and where those bottleneck feature vectors can be loaded in a way that is similar to how image_dataset_from_directory loads images.

You can first write the bottleneck features into a tfrecords file, and then load them as a dataset for the training phase.
In the tensorflow documentation you can find complete examples of how to do both.