Text analysis: structure and sentiment I would need to analyse the structure of texts like this:

******VIRUS ALERT****** ******VIRUS ALERT****** ******VIRUS ALERT ******

There is NEW VIRUS rapidly affecting computers on the internet. This new virus is insidious, in that it transmitted as a USENET message. Usenet is the "news group" area on the internet that users can openly discuss and exchange information on a wide variety of topics.

What makes this virus DOUBLY DANGEROUS, is that it is disguised as a common chain letter. Chain letters have been passed across usenet almost since it's beginning. Lately, a common chain letter subject is MAKE MONEY FAST.

The Make Money Fast (MMF) chain is read by thousands of people daily. It is also known as: "Easy Cash", "Make Cash Fast", "Turn 5$ into $50,000" and many others. They are all basically the same scheme, in which the reader send $1 to each of the 5 people at the bottom of the list, then moves his name onto the list.

The MMF Virus, as it has been doubed, rides along on these chain letters as a "hidden binary attachment". Since most news reader programs (computer programs used to read USENET messages) will automatically decode and store binary attachments, there is NO SAFE WAY to protect yourself from infection.

The virus attackes your system the next time you run your news reader. Though the virus is transmitted during a normal usenet session, your NEXT usenet session will probably be your last for a while. As a hidden attachment, it is automatically activated with your news reader, and very quickly destroys your partition table. Generally, this is not even noticed until the next time you try to run ANY program.

The next thing the virus does is to place your micro processor into an nth-complexity infinate binary loop, quickly destroying it. This will appear at first as a normal "lock-up" but will quickley wipe out the delicate circuitry in your system.

At this point, your ONLY hope is to NOT DOWNLOAD ANY MESSAGES that have a subject similar to above. Please, FORWARD this message to ANYONE you know that reads usenet news.

In particular, I would be interested in grammar structure of each sentence (subject, verbs, object, and so on); specific pattern at the beginning of the text and at the end; and the sentiment.

Could you please give me any tips on which Python libraries I could use to do so?

Thanks

So, I understand from this question that you want to understand the grammatical structure of these messages, as well as the sentiment.

In terms of grammatical structure, this depends on how far you want to parse the grammatical structure, whether that is as simple as Part of Speech (POS) tagging (Noun, Verb, etc.) or if you need to parse the sentences as effectively parse trees, which show how each word depends on others.

For POS tagging, you can use nltk's POS tagging methods (link: https://www.nltk.org/book/ch05.html)

To get an idea of how each words relates to each other, one way to do it would be dependency parsing (principles are shown here: https://medium.com/@5hirish/dependency-parsing-in-nlp-d7ade014186)

In terms of sentiment analysis, there are many ways you can approach this. One approach I can think of from the top of my head would be using a encoder-decoder model architecture.
In this type of architecture, we firstly encode the input text as a sequence of tokens (using something like an RNN / LSTM) into a 'hidden representation'. Then we decode this 'hidden representation' using for example a standard neural network where we have a final softmax output layer, which gives a probability distribution over sentiment classes. The sentiment is then chosen as the index with the highest probability.
Here is a really good paper, which talks about sentiment analysis using an encoder-decoder architecture, which could be of some use: https://www.aclweb.org/anthology/Q18-1002/


Cleaning data with two fields mixed in the same column? I am working on a template for a dashboard that is giving me some serious trouble. The format the data comes in is highly unstructured. In the image below, circled items are "Site Names", data is Month-level.
The issue is that the "Site Names" are mixed in with the Month, with an inconsistent number of months for each site. Some sites will display one year worth of data, others will contain up to 3 years. There is no field indicating Year, only Month.

I have managed to extract a list of unique Site Names from this column, so I have those on their own. My issue now is that I have no idea how to match them to their corresponding Months. I was thinking through how to do so with an HLOOKUP, however after planning it out briefly I'm not sure this would work. The site names I would be using as a lookup are not all present in the top row. I am absolutely stumped on how to achieve this.
Provided this isn't asking too much in one question, I also am faced with the challenge of getting Excel to understand that the first instance of a month for each site is an older instance of a month. I.E. "Jul" would be 2018 the first time it appears, 2019 the second time, 2020 the third time. Any help/tips would be greatly appreciated.

I suggest adding a company name for each corresponding month. See the attached picture.
The formula for the first column determined if it is for a month or for the company name. Assuming that all your months are in the three-letter format and there is no company named 'May' or 'Sep', the formula for the cell B2 would be
=SUMPRODUCT(--(A2={"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}))>0

The formula for C2 is
=INDEX($A$2:$A$13,MATCH(2,1 /($B$2:B2=FALSE))+1)

Please refer to this page (https://www.get-digital-help.com/index-match-last-value/) for explanations for this formula.
Finally, you can filter the third column in my example to keep months for the specific company only. They will be in the order needed.


Use deep reinforcement learning with recursive actions? Can we use recursive actions in deep reinforcement learning ? If yes how ?

For example, in a reassignment problem, for a task i the neural networks returns a node m, and then there will be some condition : If the node m is busy (If there's already a task j assigned to that node), then the task j will be reassigned to another node and so on until a vacant node will be returned (We suppose that there's always at least one vacant node).

Your description doesn't seem so much like a "recurrent action" as a "failed to progress" situation.

The simplest ways to deal with possibility of a blocked/failed action are:

Leave state and time step unchanged and pick next best action, without any new policy evaluations. This is effectively the approach used by many turn-based game playing RL, such as AlphaGo, although the rejection of non-valid actions is done using the game engine and not technically tried repeatedly in the environment, that does not make a big difference in all cases.

Or

Update the time step, receive a reward (maybe zero) and try again on next time step. The state information might usefully include blocked actions (or they can be applied programmatically as in the first option). You would typically use this approach if there was a real cost to attempting the action (i.e. a negative reward for losing time) and/or it is possible from the state information to predict which actions could be blocked, and you want to include these factors in the agent's behaviour.

The second option is the approach you might use in a maze solver, if it could only sense walls by bumping in to them.

If the blocked actions are completely unpredictable from the agent's observations, then it doesn't matter too much which approach you use. However, if there is a cost for failing to act, then the second approach would give you more accurate action values in case you are using a value-based approach or need to assess the performance of your agent numerically.


CapsNet Generative adversarial network I have a very interesting question. It's one month ago Geoffrey Hinton release his CapsNet paper. It's a complete new way for visual computing in the neural network stack. But my question is. Is it possible to use the CapsNet as a GAN.
I only understand the concept of the CapsNet but not technical limitations. So could this be possible?

This sounds for me for a nice field to research:)

It is definitely possible to use them in the Encoder (Discriminator) part of a GAN, since according to Hinton's original paper it's used for classification which means it fits in Discriminators as well even though the proper loss function is a subject to research.

I chose a topic of Capsule Nets in Discriminators of GANs for my Master thesis and I'm working on the model right now. However, I simplified the routing algorithm by not vectorizing the DigitCaps values (otherwise Keras code gets complicated, although I'm not sure if what I've done is 100% correct).


How to approach an extremely unbalanced time series dataset I need to classify a relatively small time series dataset.

Training set dimensions are 5087 rows (to classify) by 3197 columns (time samples) which are (or should be as far as I understood) the features of the model. I don't know yet if every sample is important and I will think about downsample/filtering/fourier transform later.

Unfortunately dataset is extremely unbalanced: only 37 (0.7%) out of 5087 rows are "Positive". How would you approach this? I will have to use Scikit-learn library.

Since this is my first approach with Scikit-learn I wanted to try a very simple classifier, with few hyperparameters,and build up from there.

First, choosing the classifier: logistic regression because is the easiest I can think of an this is just a test.
Second, choosing regularization parameter via tuning grid
Third, choosing the splitting cross validation strategy: I wanted to use stratified bootstrap but unfortunately it is not provided by the library so I opted for Stratified shuffle split
Fourth, choosing the metric: cohen's kappa because the dataset is so unbalanced the make accuracy too much biased

Script:

classifier = LogisticRegression(tol=1e-4, max_iter=500, random_state=1)
param_grid = {'C': list(range(3))}
splitter = StratifiedShuffleSplit(n_splits=5, random_state=1)
grid_searcher = GridSearchCV(classifier, param_grid, cv=splitter, scoring=make_scorer(cohen_kappa_score))
model = grid_searcher.fit(train_x, train_y)

First is "cv=splitter" legit? Second, what do you think of this approach? Obviously with such a trivial classifier the model predicted all Negative and I also got some warnings:

  FitFailedWarning: Estimator fit failed. The score on this train-test
  partition for these parameters will be set to nan ZeroDivisionError:
  float division by zero

Your dataset is extremely unbalanced, and most of the models would just ignore these 37 samples. After all, failing 0.7% of any test seems to be an extremely good result!

There are several ways to address the imbalanced dataset. I suggest two options: (1) Assign a very high penalty on misclassification of positive samples -- your loss function would be weighted L2, (2) Resampling -- when you draw a random row, assign a higher probability to get the positive sample than negative.

See, for example, https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets for implementation. And How to deal with class imbalance in a neural network?


Default parameters for decision trees give better results than parameters optimised using GridsearchCV I am using Gridsearch for a DecisionTreeClassifier predicting a binary outcome. When I run fit and predict with default parameters, I get the following results:

Accuracy: 0.9602242115860793
F1: 0.9581087077004674

Then I try GridsearchCV:

from sklearn.model_selection import GridSearchCV

param_grid = {"criterion": ["gini", "entropy"],
              "min_samples_split": [2, 10],
              "max_depth": [2, 5, 10]
              }

grid = GridSearchCV(dtc, param_grid, cv=3, scoring='neg_mean_squared_error')

grid.fit(X_train, y_train.values.ravel())

y_pred_class = grid.predict(X_test)

When I check the results in y_pred, they only contain one class (0) and thus I get a warning when I try to look at F1:

  site-packages\sklearn\metrics\classification.py:1135:
  UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in
  labels with no predicted samples.   'precision', 'predicted', average,
  warn_for)

Could anyone please suggest what could be the issue here and why the best parameters give the same prediction for the whole set?

Answer: With that many features in my dataset, restricting max depth and min samples split with the specified values could not give adequate results. Adding None and increasing the range fixed the issue.

My Suggestion:

The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter which is one of complexity indicators in Decision Trees. The default is None so it uses the maximum complexity it can get from max_depth but your parameter values are at most 10. To check this, you may try increasing the max_depth in your grid search (or leave it None) and see the result of grid search. If it improves then this is the point.


Calculate cosine similarity in Apache Spark I have a DataFrame with IDF of certain words computed.
For example

(10,[0,1,2,3,4,5],[0.413734499590671,0.4244680552337798,0.4761400657781007, 1.4004620708967006,0.37876590175292424,0.48374466516332])

 .... and so on

Now give a query Q, I can calculate the TF-IDF of this query.
How do I calculate the cosine similarity of the query with all documents in the dataframe (there are close to million documents)

I could do it manually in a map-reduce job by using the vector multiplication

  Cosine Similarity (Q, document) =  Dot product(Q, dodcument) / ||Q|| * ||document||

but surely Spark ML must natively support calculating cosine similarity of a text?

In other words given a search Query how do I find the closest cosines of document TF-IDF from the  DataFrame?

There's a related example to your problem in the Spark repo here.  The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method.  That will get you a matrix of all the cosine similarities.  Extract the row which corresponds to your query document and sort.  That will give the indices of the most-similar documents.

Depending on your application, all of this work can be done pre-query.


Mean Average Precision python code How do you compute MAP in python for evaluating recommender system effectiveness? Is there any library in sklearn or code in python for it?

I would like to compute the effectiveness of my Recommender System by computing Mean Average Precision.

This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see this library.


How can I find anomalies in each row of data? I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility.

-----------------------------------------------------------------------------
| Facility  | 2017  Jan Visitors | 2017 Feb Visitors | 2017  March Visitors |
-----------------------------------------------------------------------------
| Facility 1|         1234       |       1345        |  100345              |
----------------------------------------------------------------------------
| Facility 2|        56          |      567          | 34                   |
----------------------------------------------------------------------------

How can train this panda dataframe like this row-wise?

Planning to use one class SVM from sklearn. I want to get the anomalies in each facility e.g. in Facility 1 I'd mark 100345 as an anomaly. I have data spanning a couple of years. While we are here I am a super noob in ML and data science can I get a pointer to a condensed tutorial on unsupervised learning most of the ones I am coming across are for supervised.

I am assuming your goal is a time series anomaly detection, in which you want to detect the occurrence of outlier as time goes by. To achieve that, you need to set up a time series prediction model first, and then perform outlier detection base on that.

For unsupervised outlier detection, just check Andrew Ng's machine learning course on Coursera, there is a chapter about anomaly detection based on gaussian kernel. You can also read sklearn docments for OneClassSVM, LocalOutlierFactor, etc. Basically, the core idea of unsupervised anomaly detection is setup a metrics to describe the affinity between samples, and then use a user defined threshold to decide if a new sample is similar enough to other samples.


Predict the accuracy of Linear Regression How do I test if the predicted values in Linear Regression model are matching with the actuals?

I tried using - Confusion matrix, but I get this error -

#==============================================================================
# Create confusion matrix to evaluate performance of data
#==============================================================================
from sklearn.metrics import confusion_matrix
confusionMatrix = confusion_matrix (dv_test, y_pred)

print(confusionMatrix)

ValueError: Can't handle mix of multiclass and continuous

When I execute the below code -

##Performing Linear Regression
from sklearn.linear_model import LinearRegression
from sklearn import model_selection
regressor=LinearRegression()
##Fit train
regressor.fit(iv_train,dv_train)
y_pred=regressor.predict(iv_test)
print('Accuracy of LR',mean_squared_error(y_pred,dv_test))

It results - Accuracy of LR 7837176694.18

Which is incorrect.

Below is my sample data set -

longitude   latitude    housing_median_age  total_rooms total_bedrooms  population  households  median_income   ocean_proximity median_house_value
-122.23 37.88   41  880 129 322 126 8.3252  NEAR BAY    452600
-122.22 37.86   21  7099    1106    2401    1138    8.3014  NEAR BAY    358500
-122.24 37.85   52  1467    190 496 177 7.2574  NEAR BAY    352100
-122.25 37.85   52  1274    235 558 219 5.6431  NEAR BAY    341300
-122.25 37.85   52  1627    280 565 259 3.8462  NEAR BAY    342200
-122.25 37.85   52  919 213 413 193 4.0368  NEAR BAY    269700
-122.25 37.84   52  2535    489 1094    514 3.6591  NEAR BAY    299200

There are several ways to check your Linear Regression model accuracy. Usually, you may use Root mean squared error. You may train several Linear Regression models, adding or removing features to your dataset, and see which one has the lowest RMSE - the best one in your case. Also try to normalize your data before fitting into Linear Regression model.

The confusion matrix is used to check discrete results, but Linear Regression model returns predicted result as a continuous values. That is why you get the error: your dv_test data likely is integer, but y_pred is float.

You may try using classification model if it is suitable for the problem you try to solve - depends on what you try to predict. But for regression problem it would be better to use metric mentioned above.


training accuracy greater than validation accuracy The problem that I'm facing is that the training accuracy of my model is way higher than the validation accuracy, were talking about an approximate value of 0.2. And I can't understand why, yet I'm still a newbie when it comes to this so bear with me, please.
The data comes from two datasets which were created using f.data.Dataset, one for training another for validation since that's how the dataset had his folder layout.
model = keras.Sequential([
    keras.layers.Flatten(),
    keras.layers.Dense(256, activation='relu'),
    keras.layers.Dropout(.1),
    keras.layers.Dense(2, activation="softmax")
])

model.compile(optimizer='adam',
            loss="categorical_crossentropy",
            metrics=['accuracy'])

model.fit(train_ds, steps_per_epoch=STEPS_PER_EPOCH, epochs=10, validation_data=test_ds, validation_steps=VALIDATION_STEPS)

Train for 163.0 steps, validate for 20.0 steps
Epoch 1/10
163/163 [==============================] - 3s 21ms/step - loss: 3.9965 - accuracy: 0.8468 - val_loss: 0.3582 - val_accuracy: 0.8406
Epoch 2/10
163/163 [==============================] - 3s 19ms/step - loss: 0.3197 - accuracy: 0.8930 - val_loss: 0.5207 - val_accuracy: 0.7641
Epoch 3/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2009 - accuracy: 0.9191 - val_loss: 0.4350 - val_accuracy: 0.8062
Epoch 4/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1815 - accuracy: 0.9270 - val_loss: 0.5521 - val_accuracy: 0.7516
Epoch 5/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2122 - accuracy: 0.8986 - val_loss: 0.9616 - val_accuracy: 0.7156
Epoch 6/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2405 - accuracy: 0.9082 - val_loss: 1.2039 - val_accuracy: 0.7109
Epoch 7/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2013 - accuracy: 0.9183 - val_loss: 0.7242 - val_accuracy: 0.6406
Epoch 8/10
163/163 [==============================] - 3s 19ms/step - loss: 0.2055 - accuracy: 0.9176 - val_loss: 0.4830 - val_accuracy: 0.6891
Epoch 9/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1901 - accuracy: 0.9250 - val_loss: 0.3925 - val_accuracy: 0.8313
Epoch 10/10
163/163 [==============================] - 3s 19ms/step - loss: 0.1861 - accuracy: 0.9202 - val_loss: 0.5492 - val_accuracy: 0.8000

Could anyone explain to me please what could cause this big gap in between accuracy and val_accuracy, please?

Your model is clearly overfitting. You should use higher dropout value like 0.5 .For better generalization use deep models. And you can also use early stopping so that your model stops training before significantly overfitting.


How to get the formulas used by seasonal_decompose for Trend and Seasonality I'm trying to use decomposition to forecast into the future.  From my reading I understand that I can do this by adding a trend formula to a seasonality formula.  I know that I can decompose a time series with this:

import statsmodels.api as sm
result = sm.tsa.seasonal_decompose(series.values, model='additive',freq=12)

trend = result.trend
seasonal = result.seasonal
residual = result.resid

The trend, seasonal and residual variables are arrays of numbers.  I've searched google and the documentation for seasonal_decompose and haven't found a way to see/get access to the formulas used to calculate the numbers for trend and seasonal.

From my understanding, I need those formulas in order to be able to make projections.  Do you know of a way to get those formulas from seasonal_decompose?  Is there another function or method that works better for this?

I do not know how seasonal decompose works, but there is another very similar method that is well described.

Triple exponential smoothing (other name Holt-Winters method) internally makes a decomposition into trend, seasonal and residual components. But you should always check the decomposition. It always finds the seasonal component even if it does not exist.

Also the decomposition is actually sensitive to the underlying model. If the data is linear in logarithms and you do not apply a logarithm you will get a much less precise decomposition.


AttributeError: type object 'Lambda' has no attribute 'shape' I'm working on a sequence2sequence model with attention mechanism, thus I'm using an encoder-decoder architecture.

The decoder part code is :

# Now create the Decoder layers.
decoder_inputs = Input(shape=(len_target,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
               initial_state=encoder_states)
x2 = reduce_dim(decoder_lstm_out)
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([z, x2])
decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([x2,
               attn_out])
lm= Lambda(decoder_concat_input)
decoder_d2 = Dense(vocab_out_size, activation="softmax")
decoder_out = decoder_d2(lm)
model = tf.keras.Model([encoder_inputs, decoder_inputs], decoder_out)

The reduce_dim function() is:

   def reduce_dim(decoder_lstm_out):
       x = decoder_lstm_out
       x = tf.placeholder(tf.float32, shape=[None, 12, 12, 256])
       dim = tf.reduce_prod(tf.shape(x)[0:1])
       x2 = tf.reshape(x, [dim,12,256])
       return x2

An error is raised :

  AttributeError: 'Lambda' object has no attribute 'shape'

At the line:

  decoder_out = decoder_d2(lm)

If anyone can help me please?

The error happens on this line:

lm = Lambda(decoder_concat_input)

Keras is not able to infer the input shape to the layer so you need to specify the input_shape parameter explicitly. But it is also not clear why you are using a Lambda layer in the first place. The layer is supposed to wrap a function but decoder_concat_input is itself a Layer. That is probably an error.


Data Science as a Social Scientist? as I am very interested in programming and statistics, Data Science seems like a great career path to me - I like both fields and would like to combine them. Unfortunately, I have studied political science with a non-statistical sounding Master. I focused on statistics in this Master, visiting optional courses and writing a statistical thesis on a rather large dataset.

Since almost all job adds are requiring  a degree in informatics, physics or some other techy-field, I am wondering if there is a chance to become a data scientist or if I should drop that idea.

I am lacking knowledge in machine learning, sql and hadoop, while having a rather strong informatics and statistics background.

So can somebody tell me how feasible my goal of becoming a data scientist is?

The downvotes are because of the topic, but I'll attempt to answer your question as best I can since it's here.

Data science is a term that is thrown around as loosely as Big Data.  Everyone has a rough idea of what they mean by the term, but when you look at the actual work tasks, a data scientist's responsibilities will vary greatly from company to company.

Statistical analysis could encompass the entirety of the workload in one job, and not even be a consideration for another.

I wouldn't chase after a job title per se.  If you are interested in the field, network (like you are doing now) and find a good fit.  If you are perusing job ads, just look for the ones that stress statistical and informatics backgrounds.  Hadoop and SQL are both easy to become familiar with given the time and motivation, but I would stick with the areas you are strongest in and go from there.


How to aggregate face embeddings of all photos of the same person? I am classifying about 3000 thousand people's faces using FaceNet. Each person has about 100 photos.

FaceNet first calculates a face embedding ( a feature vector) for each photo. So each person has 100 face embeddings.

What I want to do is aggregate the face embedding of each person into one. What is the best way of doing this?

I have tried to use mean method. But I am not sure whether this is recommended way.

--
The reason I want this is because using a single SVM as classifier for 3000 labels is very slow. (I took 50+ hours and about 250G memory and it still won't finish training). So I need to divide the training data into subsets, and use multiple SVCs to get first level of results. Then I uses the aggregated face-embedding of each person and closest distance to get second level result.

This question is the first I've heard of FaceNet, but I don't think that the right solution to the question is to aggregate the face embeddings but to ask why you're using an SVM to classify the embeddings. Importantly, many SVM implementations of multiclass classification use a one-vs-rest method to train the classifiers -- if you're using a one-vs-rest implementation with 3000 labels, I suspect that this is the reason your training is taking so long.

You should look into how your implementation is training the classifier. Additionally, How large is your embedding size?


What is a large enough sample size? I have 600 customers per week. How many do I need to survey to get a good sample to work with?

I'm very new to stats.

It highly depends, If you want to make inferences on the total population you need to determine what margin of error and what confidence level you can allow for. These will then help you determine the needed sample size.

If we would want to estimate the population mean $\mu$. The maximum difference between the sample mean $\bar x$ and $\mu$ is given by:

$$ \epsilon = z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}  $$

Where $n$ is the sample size, $\sigma$ is the population standard deviation and $z_{\frac{\alpha}{2}}$ is commonly called the critical value which is derived from the standard normal distribution. These values can be found in tables or using r or python, e.g 1.96 corresponds to a 95% confidence level.

Knowing $\sigma$ (you usually don't) will allow you to determine the sample size needed to approximate $\mu$ within $\pm \epsilon $ with a confidence level of $1-\alpha$. You can try using $\sigma =  \frac{1}{2}$ which is usually enough. There exists methods for determining $\sigma$ as well. Anyhow, you may rearrange the above relation as follows:

$$ n = \bigg ( \frac{z_\frac{\alpha}{2}\sigma }{\epsilon}  \bigg )^2 $$

In fact, this question is quite hard to answer. There are many factors that determine the sample size needed for your study. You should definitely go through some material on basic statstics starting with
Awesome probability theory and go on studying inferential statistics.

It is worth taking some time to understand what the total population is in your case and with regard to the inferential question, does the sample include enough relevant observations? I do not feel that enough info is given in your question to provide a good answer.

You may look at similar surveys conducted and see what sample sizes they have used.

Another common approach, when you don't know much about your population is using Slovin's formula :

$$n = \frac{N}{(1+N \epsilon ^2)}$$

Where $N$ is the total population. As an example given a confidence level of 98% $\rightarrow \epsilon = 0.02$  and a total population of 600. we would get

$$n = \frac{600}{1+600\times0.02^2} = 484.$$

Note that Slovin's formula  is easily misused but can give you a good starting point.

I would highly recommend studying statistics if you are planning to continue working with DS.

Hope this provided some help at least.


How to combine nlp and numeric data for a linear regression problem I'm very new to data science (this is my hello world project), and I have a data set made up of a combination of review text and numerical data such as number of tables. There is also a column for reviews which is a float (avg of all user reviews for that restaurant). So a row of data could be like:

{
    rating: 3.765,
    review: `Food was great, staff was friendly`,
    tables: 30,
    staff: 15,
    parking: 20
    ...
}

So following tutorials, I have been able to do the following:

Created a linear regression model to predict rating with the inputs being all the numerical data columns.
Created a regression model to predict rating based on review text using sklearn.TfidfVectorizer.

But now I'd like to combine models or combine the data from both into one to create a linear regression model. So how can I utilize the vectorized text data in my linear regression model?

It sounds like you could use FeatureUnion for this. Here's an example:

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest

iris = load_iris()

X, y = iris.data, iris.target

# This dataset is way too high-dimensional. Better do PCA:
pca = PCA(n_components=2)

# Maybe some original features where good, too?
selection = SelectKBest(k=1)

# Build estimator from PCA and Univariate selection:

combined_features = FeatureUnion([("pca", pca), ("univ_select", selection)])

# Use combined features to transform dataset:
X_features = combined_features.fit(X, y).transform(X)
print("Combined space has", X_features.shape[1], "features")

svm = SVC(kernel="linear")

# Do grid search over k, n_components and C:

pipeline = Pipeline([("features", combined_features), ("svm", svm)])

param_grid = dict(features__pca__n_components=[1, 2, 3],
                  features__univ_select__k=[1, 2],
                  svm__C=[0.1, 1, 10])

grid_search = GridSearchCV(pipeline, param_grid=param_grid, cv=5, verbose=10)
grid_search.fit(X, y)
print(grid_search.best_estimator_)

Hopefully it is clear from that example how you could use this to merge your TfidfVectorizer results with your original features.


extract names in a list of names I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing.

I have  a collection of names which does not have any specific pattern and I need to separate individual names from that list.

Below is an example:

John Batista Barack Obama George W. Bush Serafim Batzoglou Vijay Pande Fei Fei Li Oussama Khatib Kwame Nkrumah John Oliver

As you can see there are two things peculiar with the paragraph.

1) space " " is the only delimiter, however, it is unusable as there are names which have middle names.

2) I have french, German, Spanish, Chinese and other ethnic sounding names written in English.

Now the challenge I am facing is to find names from this paragraph. I am very new to NLP and use Python and R for modeling work.

Any help will be appreciated.

I'm not an expert here, so here's my (brute force?) method.

SeatGeek has open-sourced a python library called fuzzywuzzy which is great at text matching.  It has a function called token_set_ratio that compares two multi-word strings and scores their distance.  It can consider just the intersection of individual words and score only that intersection. eg "Barack Obama Bill Clinton Madeleine Albright" will get a score of 100 with "Bill Clinton", but also with "Bill Barack".

You'll need to build a list of known names, judging by the list they're famous people, which a quick google search shows hundreds of sites that have that ready for you.  With that list, you can just loop your way through, computing a score with token_set_ratio.  If the famous name beats some threshold score against your string, you append it to a list of captured names.


Duplicate Question finding I have around 40K question and answers. How can I build a machine learning model so that if any new question comes it has to detect as duplicate or not?

A simple method is to compare similarity based on words. TF-IDF can be used considering the importance of each word. The similarity of words can be measured in various ways, such as cosine similarity. There are several recent algorithms:

W. E. Zhang, Q. Z. Sheng, J. H. Lau, and E. Abebe, “Detecting duplicate
posts in programming qa communities via latent semantics and association
rules,” in 26th International Conference on World Wide Web (WWW),
Geneva, Switzerland, 2017, pp. 1221–1229.
B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li, “Predicting semantically linkable knowledge in developer online forums via convolutional
neural network,” in 31st International Conference on Automated Software
Engineering (ASE), 2016, pp. 51–62.
Y. Zhang, D. Lo, X. Xia, and J.-L. Sun, “Multi-factor duplicate
question detection in Stack Overflow,” Journal of Computer Science
and Technology, vol. 30, no. 5, pp. 981–997, Sep 2015.
M. Ahasanuzzaman, M. Asaduzzaman, C. K. Roy, and K. A. Schneider,
“Mining duplicate questions in Stack Overflow,” in 13th International
Conference on Mining Software Repositories (MSR), 2016, pp. 402–412.
Y. Mizobuchi and K. Takayama, “Two improvements to detect duplicates
in Stack Overflow,” in 24th International Conference on Software Analysis,
Evolution and Reengineering (SANER), 2017, pp. 563–564.


How is DS used in the case of Payment Gateways? I know it's a general question but what type of analytics can be done in this case?
How can we apply machine learning models here?

Common use cases include:

Fraud detection
Transactions volume prediction
Next transaction date

Fraud detection

This is usually tackled with anomaly detection.

It requires information on the two transaction parties and using machine learning to figure out when a transaction is out of the norm and flagging as a potential case of fraud.

Transactions volume prediction

This is usually tackled with time-series forecasting.

The idea is to predict the amount of transaction that will be done in the next day/week/month for a large amount of users. It requires seasonal information and, in case you have a lot of C2B transactions, analysis on major business.

Next transaction date

This is also usually tackled with time-series forecasting. However, this analysis can be done on individual levels and not necessarily on a group of users.

The idea is to predict the date that the next transaction will be executed. This can be useful because there are cases where users will perform a lot of transactions in a short period of time, which could may be grouped together.


Full Doc2Vec Implementation/Repdoduction in TensorFlow? I'm looking to reproduce the doc2vec, i.e. paragraph vector approach by Le & Mikolov.
Is anyone aware of a full script using Tensorflow? In particular, I'm looking for a solution where the paragraph vectors of PV-DM and PV-DBOW are concatenated.
So far, I found:

GitHub #1: Only PD-DM, also I'm not sure how to run this script?

GitHub #2: Unclear what it is (PV-DM?)

GitHub #3: Looks like PV-DM?

There seems to be no solution that relates to the original IMDB data, and covers both PV-DM and PV-DBOW. I am aware of a few gensim scripts, but would like to know whether there's a TF solution out there.

Implementations
These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.
Bad
WangZ's implementation looks complete from a brief glance, although it only offers Adagrad and Stochastic Gradient Descent, two of the least reliable gradient descents. You really want minibatch gradient descent, which will predictably converge close to the local minimum where you initialize your system (think: optimization problems from calculus class). Beyond that, there's nothing really notable about his or her implementation.
Better
Jonathan's implementation is a bit better than WangZ's but worse than the Machine Learning Cookbook solution. He also uses Gensim instead of TensorFlow, which is not bad except from an engineering standpoint (it doesn't scale quite as well). He uses mini-batch gradient descent, but with hierarchical softmax instead of NCE. This is not going to make a huge difference except perhaps increasing training time.
Best
The Machine Learning Cookbook implementation is really just incredible. It follows all of the recommendations from Google's original papers on the topic, even improving on the gradient descent (this uses mini-batch gradient descent). Notably, this also uses NCE (Noise Contrastive Estimation), which I don't typically see, instead of hierarchical softmax to obtain the error for gradient descent, which Google showed produces slightly better results and works faster. A lot of thought clearly went into this, and whoever was behind it knew what s(he) was doing. I would go with this one if I were you.
PV-DM vs PV-DBOW
PV stands for Paragraph Vectors, the model you are talking about. It is also know as doc2vec. It comes in two flavors, CBOW (Continuous Bag of Words), sometimes also called DBOW (Distributed Bag of Words), and DM (Distributed Memory).
These are both distributed models in that each "neuron" contributes to the training task, but nothing has any meaning without the other "neurons" to give it context.
To paraphrase Google's documentation: The CBOW/DBOW model predicts the target word from a bag of context words (e.g. "mat" from "the cat sits on the") while the DM model predicts a skip-gram from a single word (e.g. "sits .. mat" from "cat").
In practice, research shows that the skip-gram/DM model works better if you have a lot of data and the CBOW/DBOW model works better with less. That said, this technique (and deep learning in general) isn't particularly effective without lots of training data.
IMDB
Any of these implementations can be trained on comment data from IMDB. For example, the Machine Learning Cookbook implementation is hard-coded to pull data from a similar dataset . To change where it gets the data from, you would modify the load_movie_data() function in text_helpers.py (same directory in the repository as your link, but a different file) to pull from your preferred source. For instance, you might want to modify it to pull data from the local file-system instead.


Number of leaves for lightgbm is smaller than categories in one feature I was looking at a notebook someone posted for a Kaggle competition.  They use lightgbm with the number of leaves set to 40.  If I understand right, that's setting a limit on the size of the weak learners (decision trees) that are used in the boosting; the trees can't have more than 40 leaves.

However, after training, we see that the feature with the greatest feature importance is a categorical variable with 1000+ categories! If a branch were ever used in a decision tree for that variable, wouldn't it necessarily have at least 1000+ leaves?

How is this situation handled? When the number of leaves on the weak learners is smaller than the number of categories within one of the variables?

There are different ways to include categorical features, and in many of them a single leaf can combine multiple categories:

With the label, target, or frequency encoding the categorical feature is effectively replaced by a numeric one, so a leaf can include multiple original categories naturally. Conversely, any numeric feature can be thought of as an ordinal categorical feature with an infinite number of possible values.
With the hash or binary encoding a single bit is activated for different categories, grouping them together.
LightGBM also supports categorical features natively. If they are marked in the configuration, LightGBM will consider various ways to partition all the categories of a given feature into two subsets.
If the one-hot encoding is used, LightGBM can choose to combine multiple categories via the Exclusive Feature Bundling (EFB) algorithm. It is discussed in a related question.

Finally, it is also possible that only a few out of these 1000+ categories are really important (in reality or only seemingly due to overfitting).


Why use deep neural networks over methods like linear regression or SVM? This is a very broad question, but I was wondering why researchers would choose a deep neural network over linear regression or SVM? As in, what are the advantages and disadvantages of both?

Linear models work best on very sparse data so for instance they would work well on a dataset with only one-hot encoded categorical variables. However they tend to perform poorly on dense and correlated datasets. Usually most dataset would be a mixed between both. Given that linear models only learn linear relationship in the data, they require some feature engineering in order to learn non-linear patterns. While neural networks can learn themselves complex patterns, they require a great deal of fine-tuning that linear model do not (selecting the right architecture, training algorithm ...). Look up Wide and Deep neural networks that are an attempt at getting the best of both worlds.

SVMs work better will medium size data so when you need to scale, they might not be the best solution. Compared to linear models SVMs are very sensitive to their hyperparameters (which are data specific).

The popularity of neural networks comes from their state of the art performance on very specific problems (usually involving unstructured data) such as image classification (https://en.wikipedia.org/wiki/ImageNet). Sometimes it is misguided as people tend to think that throwing data at a complex algorithm will give them some insight (it does not). There is some hype around neural networks: for you old good structured data, a bit of feature engineering + simple model will work just fine.

My two cents is that researchers do not really overly focus on the analysis of structured data: this we know how to do and well. The challenge is to make use of all the images, videos, text to build powerful models and this is what fuels research oriented towards neural network (finding efficient architectures, how to train them better, ...). All of the "disruptive" applications of machine learning of the recent years (autonomous vehicles, face recognition, chatbots, deep fakes) involve neural networks, not SVM or linear models.


Predicting future value with regression Model I have a set of predictor variables and another target variable .

Now I am really confused on what method to use to forecast the target variable .

For e.g my data set have customer profit(which is my target variable) and a set of predictor variable(balances of different account) for one year for each customer .

Now I need to predict profit of next 5 years .I am confused in the part that I dont have the data(predictor variables) for future .

What are my possible choices of modelling .Please assist .

You should distinct between a time series prediction, where from a known history of some attribute the future is predicted and model prediction where based on the predictor variables the target variable is calculated.

In your case you could combine both approaches, i.e. use time series prediction on the customer balances and apply the regression model to calculate the profit on the result.


Joining two dataframes on the basis of specific conditions I've added the images of both the dataframes here. Now I need to combine the two dataframes on the basis of two conditions:

Condition 1: The element in the 'arrivalTS' column in the first dataframe(flight_weather) and the element in the 'weatherTS' column element in the second dataframe(weatherdataatl) must be equal.

condition 2: The element in the 'DEST' column in the first dataframe(flight_weather) and the element in the 'place' column in the second dataframe(weatherdataatl) must be equal.

I tried the joins function but wasn't able to add both the conditions to it. it will be helpful if you could help me join them with the join/merge function. Thanks in advance.

dataframe]3]3

Follow mentioned link for your problems.

https://www.shanelynn.ie/merge-join-dataframes-python-pandas-index-1/


Padding sequences for neural sequence models (RNNs) I am padding sequences for a GRU based classifier that I am building in Keras.  I'm wondering if there's any accepted best practice for padding the leading or trailing side of the sequence.

E.g.

sequence = [1,2,3,4]
leading_pad = [0,0,0,1,2,3,4]
trailing_pad = [1,2,3,4,0,0,0]

In other projects I have generally padded the leading end of the sequence as a convention imprinted on me by various blogs, course work and other examples.

My question: is there any research that shows whether or not it matters which side of the sequence you pad?  Or am I completely over thinking this?

Any advice would be much appreciated.

Padded values are noise when they are regarded as actual values. For example, a padded temperature sequence [20, 21, 23, 0, 0] is the same as a noisy sequence where sensor has failed to report the correct temperature for the last two readings. Therefore, padded values better be cleaned (ignored) if possible.

Best practice is to use a Mask layer before other layers such as LSTM, RNN,.. to mask (ignore) the padded values. This way, it does not matter if we place them first or last.

Check out this post (my answer) that shows how to pad and mask the sequences with different length (with a sample code). You can experiment with the code to see the effect of removing the mask (treating padded values as actual values) on the deterioration of model performance.

This is the python snippet for quick reference:

model.add(Masking(mask_value=special_value, input_shape=(max_seq_len, dimension)))
model.add(LSTM(lstm_units))

where special_value is the padded value that should not have overlap with actual values.


sklearn - SimpleImputer in an empty Pipeline When building a Pipeline I'm ending up at a scenario that can be simplified like this:

FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))

Since this is one intermediary step in a larger Pipeline, I'm feeding the preceding inputs into both of these and select the corresponding dtypes within the Numerical and Categorical Pipelines.

For some datasets, however, no Categorical Columns are left leading the Pipeline to fail. I've tried returning an empty list and 'None' but both of these did not result in the Pipeline skipping the "empty" CategoricalPipeline.

After further investigation it turns out that the SimpleImputer() in the CategoricalPipeline causes the error. Depending on the order of steps the following messages are shown:

ValueError: Found array with 0 feature(s) (shape=(150, 0)) while a minimum of 1 is required.

ValueError: at least one array or dtype is required

Any ideas on how to pass the Imputer when no Column is present?

All(?) the sklearn transformers do a check on input data (check_X_y), which includes a check for an empty dataframe.  You could probably monkey-patch out that check, but that seems like overkill.

Instead, ColumnTransformer seems the way to go.  Its main purpose fits your situation.  It deals with an empty columns selector gracefully, by just not calling fit on that transformer:

  transformers_ : list
  The collection of fitted transformers... In case there were no columns selected, this will be the unfitted transformer.

Unless you're removing columns earlier in the pipeline?  In that case, please provide that additional context.


Representing the architecture of a deep CNN Suppose I am feeding a $60\times60$ RGB image as the input to deep CNN with the first layer created using the following Keras code model.add(Conv2D(64, (3, 3), input_shape=(60, 60, 3))).

Will 64 filters be created for each channel (red, green and blue) of the image?
Is the following representation of the network correct? If 64 filters are created for each channel, then should I write "3X64@58X58"?

Will 64 filters be created for each channel (red, green and blue) of
the image?

No, Rather filters also will have 3 channels. It would be (333).
Its work something like this-

Borrowed from- http://datahacker.rs/convolution-rgb-image/
More here-
https://cs231n.github.io/convolutional-networks/.


Does dataset training and test size affect algorithm? I am dealing with a dataset that has a total of 81 records. Out of them I divided it to 54 records for the training and the rest for testing. I noticed a few things:

Accuracy is low for given test data set (~ 30%).
Accuracy remains same no matter whether I use SVM or Naive Bayes.
If I test against 2-3 values accuracy shoots up to 75%.

Is this normal or I am doing something wrong?

It is completely normal in some circumstances. If you consider the learning problem from a statistical perspective, learning is done by trying to estimate the conditional estimate of your output variable given input variables. When you are doing learning you basically have to main components: sample data from training and the learning algorithm, and you use both to build an estimation of the aforementioned conditional probability. Now consider the relation of each component with the desired estimated probability.

Training sample

Training data is only a sample from all the possible data. You do not have all the possible data since if you would have it you would not need to do learning, right? So a sample is just a fraction of the data. This sample could cover more or less the whole domain of the possible data. An immediate thought is that a bigger sample is better than a smaller one. As an extreme case to illustrate the idea is to have only one observation. But the size of the sample is not the whole story. It is possible that the probability you want to estimate is very complex. The complexity of that space requires also more data than a simple probability space. To give an example consider the problem of learning the mean of the heights of males from USA. Having a small random sample often would be enough if you assume a normal distribution. One the other hand take the example of learning to recognize speech. Since pronouncing is not the same against individuals you would need a pretty big sample to unveil the useful relation needed to predict and throw away things which are not useful in the data.

Model/learning algorithm

Not any model is appropriate to express any true model of the signal you learn. For example if your true signal resembles a second degree polynomial would be impossible to fit properly with a linear model, other than some particular cases. So there is a complexity in the model too, which could be compared with the complexity of the conditional probability you want to estimate during learning. Even if you know the true model and you learning model is able to describe it, sometimes you need more data to make the model able to fit. As another example consider a random forest which estimates a simple line. Random forest is very flexible since the underlying statistical model is basically a constant given a local region in the input space. Considering that, even if the model would be capable of estimating the true model, since the model is a local approximation, you need plenty of data in every useful region of the space, to learn a good approximation.

Conclusion

We considered the individual component comparisons with probability you want to estimate. But you have both. This makes even harder to understand what is happening behind the walls of learning. Thus there are no universal ways of telling when data is enough for a given model. So this is why is often the case that one algorithm can have different accuracies for different training set size and for different algorithms the required training set size to have approximate equal accuracies.

One thing I tried often in practice is given a learning model to check how accuracy evolves as the learning sample size increase. If your training sample is large enough you have typically a region in a small sample size where error is large due to not enough data. After that you would have a region where errors stays approximately the same and this perhaps is where your model do it's best. Also is possible for even larger sample sizes to have a decrease in performance for some models, maybe do to inability of the model to describe what is there or due to overfitting or some other cases.

You can trace this kind of behavior using some sort of bootstrap validation, where the sample used for training is taken repeatedly for increasing lengths. Something like 20 samples with 5% size, 20 samples with 10% size, and so on. This kind of information can give you an idea between the relation of the sample size with your learning problem for a given algorithm.


extract classifier properties from pickled file I have *.clf file which I get from fit() of sklearn. I fit my data with SVM or KNN and want to show its properties when using it for predictions. For example I open earlier pickled classifier file and when I print it I get something like this:

SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
    decision_function_shape='ovr', degree=3, gamma='scale', kernel='rbf',
    max_iter=-1, probability=True, random_state=None, shrinking=True, tol=0.001,
    verbose=True)

How can I get the value of, for example, gamma to print out it somewhere else except for traversing it as string? Because at first I have to define either it's SVM or KNN.

clf = SVC()
clf.fit(X, y)

print(clf.get_params())


Is Cross Validation needed for regression if you already know the predictors in your model? Let's say you want to model the behavior of Y = X1 + X2 and you know that this is the model you want to make.  Whether or not that approximates the true relationship well is unknown. But since you want to be able to have coefficients that explain how Xi affects Y, you build a regression model.  You don't plan on adding/subtracting predictors (since you don't have any additional data) and you don't plan on comparing this model with another (no other model allows for interpretation).
Does it make sense to still use sample splitting or cross validation?  If you do cross validation, do you average the coefficients?  Or could you just use your entire data set to train the model.
Thanks!

You might.
(But probably not.)
The point of cross validation is to help you conduct model selection. You've already selected your model. One place where you might want to use cross validation is if you want to use some kind of regularization, but that might interfere with your ability to infer how each $X$ influences $Y$. (The regularized estimates are biased.)


Data augmentation solutions for tabular/structured data Are there any reliable libraries or methods for tabular/structured data (with numerical and categorical features) augmentation? Could you share some?

Basically I believe inventing/augmenting more data could help to improve model performance.

It's hard in general, because you don't know the joint distribution of the data, let alone joint distribution with the label -- or else you wouldn't need a classifier. Without that you can't confidently sample new, valid instances.

In the image case it's 'easy' because we know certainly that a rotation or shear or scale of an image produces another valid image, and we know its label is the same.

If you're willing to make the assumption that points in your input space between two instances with the same label are also a) valid inputs and b) have the same label, then you can apply things like SMOTE (https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html) to generate more data according to that assumption. If that assumption is wrong, it'll hurt performance.


Supervised learning for image segmentation Hello I am new to Applied ML and trying to solve a problem where I have given several images with few patches in it and masked images where these patches are classified/extracted as training data.Given that i need to use ML to train network such that It can generate those masked images for test data set.
I would like to know which technique will be best suitable in Both the cases where I want to use Python extended libraries like Scikit or Using just Numpy and Scipy.
Thanks in advance

I would suggest you use Python with Ski-Image for Image Related operation. For Machine Learning typically for Deep Learning, you should use Keras with theano/Tensorflow backend with GPU capabilities whichever suits you.

In order to generate masked images, you should use data augmentation.Data Augmentation Documentation for Keras


Prime model with mock data, good idea or not? I am currently working on text classifier with some pretty unique characteristics. The data is composed of about 2K categories but 98% of the data lives in just one of those 2K categories. However, our main problem is that we have a list of additional categories but have yet to see any transactions that fall into those categories (we know they will eventually).

Question:

1) Does it make sense to "prime" the model with these additional categories? For example, I could just add examples of these categories to my training set.

2) If priming is an acceptable are there any best practices or guides in doing so? I am wondering if I should just add the additional transactions to my training corpus, randomize them with other text, how many examples of the training set they should represent, etc..

This is not the answer you'll want to hear, but I would say, "no." Priming is not a good idea. A model is only as good as its inputs. If you're making up the inputs yourself, then your model isn't learning real patterns and you might as well just hard code a set of rules that represent how you would have created the primed data.

A fancier way might be to make a bayesian model where you create priors based on your assumptions. The bayesian model would then evolve as real data becomes available.


Should I report loss on the last batch or the entire training dataset? In the training loop below, you can see that train_loss represents the loss on the most recent batch.
Whereas the eval_loss is computed on the entire training dataset.
When I print it out, should I be recalculating the train_loss against the entire training split instead of the last batch?
for epoch in range(epochs):
    # Loop over batches.
    for i, batch in enumerate(samples_train['features']):

        train_preds = model(samples_train['features'][i])
        train_loss = loss_fn(train_preds, samples_train['labels'][i])

        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()

    # Only the training data is batched.
    eval_preds = model(samples_evaluate['features'])
    eval_loss = loss_fn(eval_preds, samples_evaluate['labels'])

    print(f"Epoch: {epoch}, Train_Loss: {float(train_loss)},  Eval_Loss: {float(eval_loss)}")

The losses should be calculated over the whole epoch (i.e. the whole dataset) instead of just the single batch. To implement this you could have a running count which adds up the losses of the individual batches and divides it by the number of batches after the epoch ends.


Sparse Representation vs Dense Representation The one of the benefits of ReLUs is sparsity. Sparsity arises when a≤0 (a = wX+b). The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.
Is it true that sparse representation is more beneficial that dense representation, especially for Neural Networks?

Sparse representations are not always better than Dense, though they do have an upper hand. In terms of various inputs being handled (varying amount of information), without a sparse prior we shall be trying to learn same size fit for all representation, that may be inadequate. Here dense representations will have lesser degrees of freedom, resulting in non-linear relationship. Overall our representation mathematically needs to mimic the topology of underlying manifold. But in how many scenarios can we have sparse representation, is the bigger question/problem, where we we have sufficient uncorrupted identities. Approaches like SLR dictionary decomposition come in handy in such scenarios.


What is the meaning of the term "pipeline" within data science? Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here

People often refer to pipelines when talking about models, data and even layers in a neural network. What can be meant by a pipeline?

A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually examined/explained.
Here is an example image (source: DataBricks)

There is actually a nifty module (class, actually) in Scikit Learn for building your own machine learning pipelines, which is literally called Pipeline. You can specify processing steps, models and other transformations, then wrap them into a pipeline, which carries out the start-to-finish process for you. This makes it much easier to work in a modular way and alter parameters, while keeping things organised.

In the documentation, they use an ANOVA (analysis of variance) model to select variables, which are then fed into an SVM (support vector machine) to perform classification.

In the context of what might be considered a single model, a pipeline may refer to the various transformations performed on data. This might include dimensionsality reduction, embeddings, encoding/decoding (GAN example), attention mechanisims, and so on.

Here is an example of what might be referred to as a pipeline: the Spatial Transformer Network:

Images are passed through a pipeline with three parts:

a localisation network
a grid generator
a sampling mechanism

these three parts might be akin to one of the parts in the MLlib Pipeline displayed above.

Another area in which pipeline is used extensively is within data management. In this case, it refers to how and where data is transferred, and perhaps by which frequency. There are large packages dedicated to building such pipelines, e.g. :

Apache Spark - can do a lot more than just pipelines these days (use case example)
Luigi - manages complex batch processes (developed by Spotify)

Note: this question was asked and removed just before I posted my answer, so am providing the content via a self-answered question


Using keras with sklearn: apply class_weight with cross_val_score I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning.

I want to avoid undersampling, so I have been looking into the class_weight hyperparameter. For sklearn's decisiontree classifier, this works really well and is easily given as a hyperparameter. However, this is not an option for sklearn's neural network (multi-layer perceptron) as far as I can tell. I have been using Keras instead and I can apply class_weight with gridsearchCV, but not with cross_val_score.

Is there a way to use class_weights in keras with cross-validation?

The sklearn MLPClassifier does not implement any option for class weights at the moment. There are at least two paths for you to follow. You could go with writing a custom loss function, which allows you to stay in the sklearn framework without reaching out to keras. Another option is to implement the cross-validation yourself, which is not difficult to do, and run your keras model for each fold. An example can be found here.


Which Technique should we use for predicting an integer output? I'm working on a problem where my target feature of type integer. i.e (n_clicks). In general, if we want to predict categorical target feature then we use classification algorithms and on the other hand for predicting a target feature of type continuous then we use regression techniques where my output will be -infinity to +infinity. But in my case if I use regression then my output will become a float value. One solution here would be convert the output float values into into integer or by use of pandas.round() to round figure the value.

Is there any alternative way to predict my output variable as an integer ?

Thanks

UPDATED: As suggested to use POISSON Regression, tried below code but still the output is same. Below is code snippet:

import statsmodels.api as sm
x_train_sm = sm.add_constant(x_train)
poisson_model = sm.GLM(y_train, x_train_sm, family=sm.families.Poisson()).fit()

If you are using a poisson regression model, this question might be better suited for cross-validated as it is a statistical model.

This appears to be related to the reason why you are getting floats instead of the expected integer value. Poisson regression returns the expected value, E[Y|X] which does not necessarily have to be an integer.

https://stackoverflow.com/questions/49416697/statsmodel-poisson-prediction-return-floats-instead-of-whole-numbers


How to properly apply CrossValidation and/or split the dataset? I have a particular problem and do not really now how to properly validate my experiments in this scenario.

There is one big data set with 100.000 samples, 99.000 y=0, 1.000 y=1
Each sample has 1.000 Features
There are 10 different subsets of feature combinations which have to be evaluated to get information about more or less expressive feature groups
Because of the different number and type of features each subset, an adequate models architecture has to be figured out manually
The performance of a few different kinds of models have to be evaluated
The performance of several different sampling techniques have to be evaluated
10-fold Cross Validation has to be applied
Random Search has to be applied for each experimental configuration (Combination of subset + model + sampling technique)

These facts lead to the following experimental routine:

train, val = splitDataset()
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, val)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

So, now I'm not sure about whether the CrossValidation is applied correctly. Because the model which has to be validated has already seen the train data (during manuallySearchGoodArchitecture and HPOptimization) and those are also used for testing purpose during CV.

So, is this routine faulty?
What would be the correct way of performing those experiments while applying proper validation technique?
How must the data set be splitted?

I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret the results accordingly. For instance in this case the results will show how different subsets of features perform on the data, but this difference in performance across subsets shouldn't be considered reliable: it's possible that a particular subset happens to be better than another by chance.

It's quite a complex setting and I assume that there are efficiency constraints to take into account. If possible the most reliable results would be obtained by doing several stages of CV (or other techniques, e.g. bagging) using different subsets of data. For instance you could run the whole process a few times, each time using a different random subset of instances: in this way you can average performance and see whether a particular subset of features is constantly better than another (for example, same idea for other parameters).

[edited] Disclaimer: I don't know if there is any standard way to proceed with a complex multi-level setting like this, my advice is based only on the experience I had with a few broadly similar cases.

Generally the idea is that every choice to make can be considered as an hyper-parameter, including the subset of features, the type of model, the architecture, the sampling technique. Therefore I think that ideally one would cross-validate at every level, i.e. put every loop level in a function and call this function k times with a different subset of data. It would look like something like this:

train1, val1 = splitDataset()
iterate each featureSubset:
    train2, val2 = splitData(train1)
    resultTrain2 = kfoldCV(processLevel2, train2)
    resultLevel2 = apply(resultTrain2, val2)
resultLevel1 = apply(resultLevel2, val1)

processLevel2:
    iterate each modelType:
        train3, val3 = splitData(train2)
        resultTrain3 = kfoldCV(processLevel3, train3)
        ...

remark: I'm not 100% sure about the algorithm, maybe I over-complicated it. I think it gives the general idea though.

But of course following this logic the computational complexity becomes way too high, so you will probably have to take a few shortcuts. One thing I've tried successfully in the past is to use genetic learning in order to optimize different parameters at the same time: that would mean having different "genes" which represent the different parameters (feature subset, model type, etc.), each with its set of values, and then run the genetic process which is supposed to converge to a set of optimal values for all the parameters (I was using CV every time a particular combination of parameters is evaluated). But again I don't know if it's a very orthodox method :)

[edit2]

After more thought I think I would try to do something like this:

innerData, valOuter = splitDataset() // keep large amount for validation set, say between 20-40%
train, valInner = splitDataset(innerData)
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, valInner)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

bestHPCombinations = select top N HPCombinations from first stage
for each HPCombination in bestHPCombinations
     model = train(innerData)
     result = apply(model, valOuter)
end

It's simpler: the idea is just to re-evaluate the results from the first stage on some fresh data in order to avoid overfiting (here HPCombination includes featuresSubset, modelType, etc.). The second stage could also include CV or bagging for more reliable results. But in general I think this option would be reasonably reliable, since it's unlikely that the best models from the first stage would also be the best models in the second stage by chance only.


Index needs sorting help! I want my index which is mixed up (i just sorted out ratings from highest to lowest) to be in ascending (sorted) order corresponding to my sorted ratings. How can I do that?

After sorting, try:

df = df.reset_index(drop = True)


Association rule mining interpretation I am trying to create a association using apriori algorithm.The data contains around 33000 records.Below is the sample of the data

id  code
1   19
1   58
1   111
2   19
2   111
2   167
3   12
3   79
3   85
4   96
5   19
6   58
7   12
7   18
7   40
7   48
7   85
7   86
7   135

In R this data is:

structure(list(id = c(1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5, 6, 7,
7, 7, 7, 7, 7, 7), code = c(19, 58, 111, 19, 111, 167, 12, 79,
85, 96, 19, 58, 12, 18, 40, 48, 85, 86, 135)), .Names = c("id",
"code"), row.names = c(NA, -19L), class = "data.frame")

Using the following code I tried to build the association

# creating a string of codes based on the id
library(dplyr)
Asso2 = Asso %>%
  group_by(patient_id) %>%
  summarise(hi = toString(hcc_id))
# converting into transactions
library(arules)
fact <- data.frame(lapply(Asso2,as.factor))
trans <- as(fact, 'transactions')
# Applying aprior
rules = apriori(trans, parameter = list(supp = 0.001, conf = 0.001,target = "rules"))
rules
inspect(rules)

I am getting totally 96 rules like below with empty lhs and I trying to understand whether we cannot make any rules from this data or am I missing anything here. Since I am novice in this, I would like to get some help.

#     lhs    rhs               support     confidence  lift
#  1  {}  => {hi=19, 96, 108}  0.001021696 0.001021696 1
#  2  {}  => {hi=176}          0.001021696 0.001021696 1
#  3  {}  => {hi=88, 108}      0.001021696 0.001021696 1
#  4  {}  => {hi=72}           0.001051746 0.001051746 1
#  5  {}  => {hi=88, 96}       0.001051746 0.001051746 1
#  6  {}  => {hi=108, 112}     0.001081796 0.001081796 1
#  7  {}  => {hi=84}           0.001081796 0.001081796 1
#  8  {}  => {hi=100, 103}     0.001111846 0.001111846 1
#  9  {}  => {hi=18, 108, 111} 0.001111846 0.001111846 1

Yes, this basically means that no sensible rules can be built with the constrains of support and confidence you impose. Note that your lift is always trivially one. The solution to this is given in the documentation of the package:

Note: The default value in APparameter for minlen is 1.
This means that rules with only one item (i.e., an empty
antecedent/LHS) like {} => {beer} will be created. These
rules mean that no matter what other items are involved
the item in the RHS will appear with the probability given
by the rule's confidence (which equals the support). If
you want to avoid these rules then use the argument
parameter=list(minlen=2).


How to display the value of activation? I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "act_layer_1". What should I do so that I can see the activation values? Thanks in advance!

W1 = tf.get_variable('W1', [n_input,n_hidden_1], initializer = tf.contrib.layers.xavier_initializer(seed = 1))
b1 = tf.Variable(tf.constant(0.0, shape=[n_hidden_1], dtype=tf.float32), trainable=True, name='b1'),
act_layer_1 = tf.nn.relu(hidden_layer(x_image, W1) + b1)
drop_layer_1 = tf.nn.dropout(act_layer_1, keep_prob)

Although you have not made it clear in your code snippet but the activation output could be inside or outside a function:

Outside Function : Just run

a = sess.run(act_layer_1)
print(a)

Inside Function: You have to return the value of this activation bunched with other values you need to print and then again use sess.run() to get all the values and print the index which contains your value.

A longer but better in debugging would be to use tf.InteractiveSession(). More details can be found here:

How to print the value of a Tensor object in TensorFlow?


Adaptive regression parameter estimation in R How would I do parameter estimation and prediction for the adaptive regression model using R, as in the 4th page of the paper linked below?

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=923635

Could anyone clarify this for me?

If you know adaptive regression models very well, share some useful link, or describe the model/parameter estimation/prediction, that would be very helpful.

Thank you so much!

Here is some notes for an R package 'earth' for adaptive regression. This may be useful.

http://cran.r-project.org/web/packages/earth/vignettes/earth-notes.pdf


Batch norm: why the initial normalization? I'm a beginner in NNs and the first thing I don't understand with batch norm is the following 2 steps:

First we normalize the batch data on a z parameter to Mu=0, sigma^2=1
Then we change z via the coefficients of Mu, sigma^2 (usu. alpha, beta) by updating them as learnable parameters.

I don't understand why the first step is necessary if we change the distribution in the second step anyway. Could someone explain please?

I hope I got your question correctly...

It’s called “Batch” Normalization because we perform this transformation and calculate the statistics only for a subpart (a batch) of the entire training set not as a whole.


Featuring scaling whole data set before spliting it. I am wondering why do we use scaling on train and test set separately. I understand that transform () on test data  μ and σ as computed from fit_transform() on  Train. But why can we compute μ and σ from all given data (before split) and then apply them on future data.

Do we do this because we don't know how the size of our future data?

You must use the same scaling factors (μ and σ) for training and test data. The reason that you must use the same scaling factors is that you should compare the data with the same pre-processing. Note that scaling factors (μ and σ) must be computed using only the training data. That is because in practice you are not provided with test data and you just have to evaluate your model on test data.


Self adjusting CNN network I am currently trying to build a self adjusting network, such that given any number of inputs, should always provide an output of shape (15,145)

The network structure is pretty simple and looks like this:

    inputs = 36
    list_of_input = [Input(shape = (45,5,3)) for i in range(inputs)]
    list_of_conv_output = []
    list_of_max_out = []
    for i in range(splits):
        list_of_conv_output.append(Conv2D(filters = 145 , kernel_size = (30,3))(list_of_input[i]))
        list_of_max_out.append((MaxPooling2D(pool_size=(3,2))(list_of_conv_output[i])))

    merge = keras.layers.concatenate(list_of_max_out)
    #reshape = Reshape((merge.shape[0],merge.shape[3]))(merge)

    dense1 = Dense(units = 1000, activation = 'relu',    name = "dense_1")(merge)
    dense2 = Dense(units = 1000, activation = 'relu',    name = "dense_2")(dense1)
    dense3 = Dense(units = 145 , activation = 'softmax', name = "dense_3")(dense2)

    model = Model(inputs = list_of_input , outputs = dense3)
    model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")

    print model.summary()

    raw_input("SDasd")
    hist_current = model.fit(x = [train_input[i] for i in range(100)],
                        y = labels_train_data,
                        shuffle=False,
                        validation_data=([test_input[i] for i in range(10)], labels_test_data),
                        validation_split=0.1,
                        epochs=150000,
                        batch_size = 15,
                        verbose=1)

It been adjusted for having 36 inputs which would given it an output shape of (15,1,145) - but how can i determine the number of filters, kernel size and pooling size that would give me the desired output size.  The network is supposed to be used for classification, and the output vector of length 15 with classes for each third entry in the first axis (45 = 15*3). the total number of classes is 145, hence output dimension (15,145)

Spatial pyramid pooling layers (https://arxiv.org/pdf/1406.4729.pdf) should solve this problem for you. These layers allow you to use input images of any dimension, instead of being restricted to 224x224 images, for example.


Predicting when component will fail having its parameters data I have a component and I need to predict when it will wear out and will need replacement. I monitor, let's say 5 parameters of this component, each one is monitored for every run cycle. So, the dataset can look like this:

No_of_runs    Para1    Para2    Para3    Para4    Para5
1              100      32        45      230       86
2              101      34        65      234       90
3              120      24        32      242       80
4              105      45        40      213       75
5              90       42        54      200       77
...            ...      ...       ...     ...       ...

In addition to the above, I have a dataset saying at which point this component needed replacement in the past. So, if the above dataset reaches up to 500 rows (run cycles of the component), the other dataset says that the component nedded replacement in cycles 50, 130, 340, 400.

Based on these data, the goal is to create a model able to predict when this component is going to fail in the future giving it the parameters' data up to this time.

I struggle to come up with a solution to the problem and as I have not much of experience in the field, I'm not sure of which approach to take.

It's a supervised problem, but I'm not sure of how to integrate the information of 'degradation' of the component (run cycles) or how to structure my features matrix.

Looks to me like a sequence labeling problem, where the class is binary indicating whether the component is still working or failed. In this option you should build a training dataset which each cycle which looks like this:

No_of_runs    Para1    Para2    Para3    Para4    Para5    status
1              100      32        45      230       86       ok
2              101      34        65      234       90       ok
3              120      24        32      242       80       ok
4              105      45        40      213       75       ok
5              90       42        54      200       77       ok
...            ...      ...       ...     ...       ...
1234           ..       ..        ..      ..        ..       fail

The order of the instances matters. After training, the model can tell you the probability of failing for an instance given its sequence of runs.

Conditional Random Fields are a standard option for such problems.


Should I transform a multiple regression with outliers into ordinal regression? I have small dataset of about 60 samples that performs poorly in regression. So I wonder how can I transform this task into predicting intervals instead of values. Is it possible to make it perform better for intervals instead of values?

The dataset is imbalanced. There are just few high values. The algorithm constantly underestimates these values. So I think if I make prediction in intervals, it'll perform better for values, say, more than 50.

But, overall, the problem is predicting for values near the limits of intervals. Is it better to mark this values as belonging to both intervals?

Your final proposal (put an observations "on the edge" of intervals into both intervals) is unconventional, but probably worth trying. Note that ordinal logistic regression need more parameters than linear regression, so be careful adding too many categories because your data is small.

You can consider transforming the prediction target to reduce the distance between the extreme high values and the rest of the data.

In general, however, you will need to understand why those extreme high values are high. Are they incorrect entries (i.e. outliers)? Or is there some feature in the data that can be added to the model in order to better predict when a data point is higher than expected?

For example:

Let's say you are trying to predict student performance on a linear algebra midterm exam. Your data set consists of last year's exam scores. Maybe your model includes these features:

Student class year (freshman, sophomore, junior, senior)
Did student take honors calculus? (binary)
Average homework grade
Recitation attendance percent

Your model generally does well. Most students get in the 75-90 range, but some do unexpectedly well and score in the high 90s, almost perfect. Your model fails to predict these students' grades. So what went wrong? One possibility is that there is some other attribute shared by the high-scoring students that the other students might not have. You look back over the exams and realize that all the high-scoring students did well on a tough problem that you had covered in an office hour session. Now you add a new feature to the model:

Attended my office hours session on October 2nd.

and now your model correctly predicts the high-scoring outliers.


Understanding Transfer Learning of Word Embeddings I can't quite visualize how transfer learning of pre-trained word embeddings is useful in an NLP task( say named entity recognition ) . I'm studying Andrew NG's Sequence Models course and he seems to say if the training set for the target task is very less, then transfer learning of word embeddings is helpful in a way that unknown words in the training set can be handled in the application .
Let's consider the task of Named Entity recognition ,
My question is , what does the very small training set for the target task contain ? Are they word embeddings or sentences labeled with entities ?
Does he seem to suggest that if the training set is of just  labeled sentences whose words have embeddings in the pre-trained model , then words which are not present in the training set but are closer to those already in the training set also get captured effectively in the application ?
Eg : Consider 'Orange' is in training set . But , 'Apple' is not .
So , in the sentences , ' I like Orange Juice ' and ' I like Apple Juice ' , Apple gets recognized as a fruit , even though it's not in the training set since it is closer to Orange .
Am I right in my assumption ? Or can someone please correct and explain to me if I am not ?

So Named Entity Recognition is a mechanism where you ask your network to learn about how to detect entities given word vectors as the input.
The theoretical aspect of word embeddings is that based on your construction of sentences, the word embeddings for Orange and Apple are very similar i.e their cosine angle is very small.
In Named entity recognition you use these word embeddings and feed them into a network where the data you are training one has tags for each of the word embeddings i.e entities or normal words. So your network is actually understanding the relationship of the word embeddings and how to tag them. This makes it incredible for us to see that Apple gets detected even though its not in the training set, precisely where word embeddings help us quite well because the word embeddings are usually trained on a large corpus of data containing the words apple, orange and other tokens. This is where transfer learning helps because you are using the word embeddings trained in an unsupervised manner and then used to learn about entities.
Hope that helps. I can elaborate if required.


Adding high p-value and low R square features in linear regression model to improve result I am working on a linear regression problem. The features for my analysis have been selected using p-values and domain knowledge.  After selecting these features, the performance of $R^2$ and the $RMSE$ improved from 0.25 to 0.85. But here is the issue, the features selected using domain knowledge have very high p-values (0.7, 0.9) and very low $R^2$ (0.002, 0.0004). Does it make sense to add such features even if your model shows improvement in performance. As far I know, according to linear regression, it is preferable to only keep features with low p-values.

Can anyone share their experience? If yes, then how can I back up my proposal of new features with high p-values.

In general, adding more features will increase the quality of model fit.
If your goal is best fitting modeling, add as many features as possible (regardless of p-value).
Sometimes people care about parsimonious models, they are will to lower the overall model fit because they also value a simpler model. Then they apply a threshold to features using p-values.


Using TF-IDF with other features in scikit-learn What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. scikit-learn's TF-IDF vectorizer transforms text data into sparse matrices. I can use these sparse matrices directly with a Naive Bayes classifier for example. But what's the way to also take into account the other features? Should I de-sparsify the tf-idf representation of the text and combine the features and the text into one DataFrame? Or can I keep the sparse matrix as a separate column for example? What's the correct way to do this?

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.


Trying to implement a "smart compose" feature I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself.

https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html

The text is super vague though - the only part that gives any direction at all is "we combined a BoW model with an RNN-LM".

I remember that BoW is an unordered representation of the input corpus where only relative magnitudes are important. But I just don't have enough experience to know how an unordered representation like this could help me predict an ordered suffix.

Does anybody have any thoughts on how this might more specifically work? Or any references for further reading? I wasn't able to find much unfortunately.

From the same blog,

  In this hybrid approach, we encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step.

The BoW part of their hybrid approach is to get the general context of the email conversation by averaging the word embeddings in the subject and the previous email. Passing this vectorized representation of the general context at each step of the RNN-LM, which takes into account word order, helps get predictions that are more tailored to the subject of the conversation.

Note that at each step, the RNN-LM is getting 3 types of input -

Embedding for the previous word - For immediate context.
State vector - For localized context
Subject and previous email averaged embeddings - For a wider, conversation-level context

Hopefully, it is clearer now, that the model incorporates order with the use of RNN-LM and the BoW part is to condition the output to the subject of the conversation.


Parameters optimization algorithms in Weka In Weka, I used the Grid and Random search parameters tuning algorithms but unfortunately, their performance (in terms of better prediction accuracy) is observed worst when we use the ML algorithms (Support Vector Regression, Linear Regression etc) without any optimization algorithms.
I wonder how it is possible? I mean one algorithm (Grid or Random search) should perform better or worst when compared with each other but they have worst performance compared to without any parameter optimization algorithms. I even tried the hybrid of both in Weka with MultiSearch option, but even the hybrid of these two have worst performance.
Kindly if someone could provide comments based on their experience in this regard.

Supposing that you refer to the performance of tuning algorithms as the improvement in algorithm performance (accuracy, error, etc.). The effectiveness of parameter optimization (or tuning) is different for different algorithms. Many papers have discussed it, and Olson et al. have shown how much performance may vary for several algorithms. SVM and LR (those that you mentioned in your question) were not improved so much after tuning.

If you have bad performance in several algorithms you tried, the problem may not be in the tuning algorithms, but in your features-target set. I suggest you take a look at the Domingos paper, it's a nice read to build a successful model.


Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros Hello Practitioners,
Being a newbie seeking help to gain experience in Data Science.
Lets take a scenario where a big company wants to forecast its sales (a specific product) across different stores in different geographic locations.
As an Analyst, a task is given to forecast few hundreds of series (sales) for next 3 months. Since, we are forecasting sales across different geographic
locations, the nature of the series would not be same for all. There would hundreds of models to check with.
What are the suggested approaches for this scenario with your experience in this field? Also, how important it is know the nature of each series in this scenario?

I can suggest auto.arima function from forecast library, if you are R user, if you are Python user then follow this link. All you need is to write simple for loop, which allows you to built best ARIMA models in different geographic locations:

   for (my_time_series in set_of_all_time_series){
        model=auto.arima(my_time_series )
         }

You can cluster your time series by correlation (make sure that your time series are stationary to avoid spurious correlation). If this reduces the number of time series (which will depend on threshold on correlation), you can take any 1 member from each class, build any model (not only ARIMA) and apply model results on each member of that class.
Contract VAR model


Which different visualizations to make for time series analysis? I hope you're doing great. I have recently started working on a time series project and I have just finished preparing my data and calculating trends (daily, weekly, yearly) and I wonder what are the different plots I should make using the the absolute values of my variables and the calculated trends?

Also, I'm seeking anyone to tell me what are the best practices to work on time series and what are the details I should pay attention to!

Thanks !

I am also new to time series forecasting. I used simple lineplot to visualize the time series data. Some of the models used for time series are :

ARIMA
SARIMA
prophet
LSTM (deep neural networks)
LBATS
You can refer to this link to know what things you should consider while building a time series model


If we are using batch normalization as the first layer, can we forego standard scaling of inputs? It is common practice to use the standard scaler on the inputs before feeding it to a deep learning architecture. I was wondering whether it is necessary if the first layer is a batch normalization layer.

Scaling is a bit different from what Batch normalization does. Performing scaling creates scale indifference amongst all the data points. Ex: Values 5 and 55 will have a higher magnitude of scale difference than the log(5)=0.698 and log(55)=1.740. This is the idea behind scaling. Similarly, We scale the images with 255 which helps in faster convergence

When we pass in the scaled data to our network, due to the operations that take place in every layer of the network, the distribution of data observed in let's say 5th layer of the network is different from the distribution in the 1st layer. This is a problem because we are kind of training the first part of the network in one form of data and the latter part of the network with another form of data. So to remove this effect, We do BatchNorm in later layers of the network