How to get the best combinations of features for a sale optimization problem? I have a database of shoes items from the same brand with many variables (features) like the size, the color or the shape. I also have the produced and sold quantity for the last years.

This is a sample of the database:

I would like to know what are the best combinations of features to have the perfect shoes, i.e the shoes which will be sold the most for sure. The Ratio (last column) is the variable which gives a "score" to an item (=an combination of features). For example, the best combination can be : construction=vulcanized + colour=brown but without width=welded.

I am using Python to read the database and using the library Pandas. This problem can be an optimization problem. What are the possibilities in data science to solve this kind of problem ?

Whenever you have an optimization problem the first question that you have to ask yourself is.

Can I make it a Linear Programming problem?

For python I normally use Gurobi. Here is a basic example to get started: https://www.gurobi.com/resources/food-manufacture-i/

You can also do Machine Learning with it, that is the hot topic nowadays, but if you can make it a linear programming problem you will achieve optimality.


Where can I find a good machine learning research team /university Lab for a PhD in CS? I am aiming to do a PhD in Machine Learning and in Germany since I have a Masters in ML already I wanted to know what are the best options to aim for ?

Thank you

University websites might be a  good way to search a suitable laboratory for you since you have a topic in mind. You may go through the current researches and research interest of the laboratories and professors. Sometimes, the university may even publish a list of available projects and/or supervisors who are looking for Ph.D. students. Try contacting the supervisors through email to discuss your ideas and the opportunities.

Another way is to search for Ph.D. opportunity through your undergraduate and master supervisor. They might have a lot of connections with other universities as well as recent researches. They may guide you based on your knowledge and interest.


Generalised data science approach to astronomical catalogue cross-match I'm interested in performing a cross-match between two source catalogues in an astronomical context. I'll try and explain the context simply.

A truth catalogue of sources (each with some position, shape and flux properties) is used to create a synthetic astronomical image, by convolving with noise and instrument artefacts.

Then 'some process' is developed (by whatever means - it could be a deep learning algorithm, or it could be a person picking sources manually) which identifies sources in the synthetic image, obviously with no knowledge of the truth catalogue.

The challenge is to map the sources in the 'estimated' catalogue with those in the truth catalogue, after which it is possible to devise a measure of the efficacy of the source recovery approach. Bear in mind that estimated positions/fluxes will differ from the truth due to noise, and not all sources may be detected.

One possible way to do this is to use a positional cross-match algorithm like this: https://arxiv.org/abs/1611.04431, and then refine the possible match candidates down by taking the 'closest' in some multi-dimensional parameter space.

However, I thought there could very well be a more generalised maximum-likelihood estimation algorithm suited to this kind of matching task. I think a nearest-neighbours type routine sounds promising, though I'm unsure how this would scale with hundreds of thousands or millions of sources. If anyone has any suggestions for a starting point, I'd be very grateful.

The generalised algorithm I was looking for was a k-dimensional tree, a space-partitioning data structure which is commonly used in nearest neighbour searches. There are implementations in scipy.spatial (KDTree and the faster cKDTree) and in sklearn.neighbours (also called KDTree).

These are the underlying engines of the Astropy and AstroML catalogue cross match methods, though for my specific use case I required the functionality of the k-d tree itself. The KDTree allows one to search in any number of dimensions, though these need to be scaled (by std dev) if they are different properties.

In some preliminary testing I found this to be over 100 times faster than the C3 algorithm I linked above for my use case. It takes around 10 secs to pick, for 85K sources, the nearest neighbour from a catalogue of 1.4M sources (when searching in 4 dimensions: 2D position, flux and size)


What's a reasonable distribution to model views over time of... this question? Sorry for the self-referential title ;)

I'd be curious to know what's a curve that could be used to model - in the scale of (say) weeks* - the "engagement lifecycle" of a single news page of an online newspaper in terms of webpage views. The same model could be applied for Stack Exchange questions views too!

What I would intuitively model is the initial spike of visits due to the fact that a new, fresh page is very visible as it's linked in the homepage (triggering also social media reshares) and then the long-term visit behavior e.g. due to that page being reached from links from other webpages, or via search engines.

One possible way would then be the sum of two Poisson processes (the first switching off after some typical characteristic time).

Another possibility would be using something like exponential decay to capture a decay over time of interest (the "freshness" of the content fades, and so does public interest).

Is anyone aware of better alternatives or literature on this?

*I wouldn't model day-night nor weekly seasonality.

One way too look at this problem, by looking at the literature, seems to frame it as an Hawkes process, as e.g. in  Rizoiu et al., https://arxiv.org/abs/1602.06033 .

The paper also cites previous work using other modeling approaches:
One set  of  approaches  describe  popularity  dynamics  [...] as being power-law shapes from either an  exogenous  shock  or  endogenous  relaxation,  a  combination of power-law and exponential decay, multiple
power-law decays with periodicity or a collection of recurrence peaks.


Use both differencing and normalization in time series modeling to make it stationary? I am working on a time series dataset. Should we use both differencing and normalizing or either of the ones to make it stationary?

Normalization does not stationarize a time series, as by definition, a non-stationary process has time-variant unconditional joint probability distributions - this implies that the mean and variance changes over time.

To see this for yourself, try computing the mean and standard deviation then normalizing the first half of the time series. Keep these estimates and transform the remaining half, and the resulting time series is still non-stationary.

Differencing $K$ times is a common method of stationarizing time series if it is integrated of order K, $I(K)$, and usually works in practice. However, note that stationarity implies $I(0)$ but the converse is not necessarily true.


What skills do I need to become a data scientist? And how to show them? I have fimiliarized myself with the recommended most important concepts (Linear Algebra, Analysis, Phython, Numpy, Pandas, a bit of Statistics, Linear regression). For the last two, I don't know how deep it should go. I know what things mean and how to get them working in python.

But the question is what now? I guess I could argue that this is a starting point and I can apply to a bad data analysis or visualisation position if I learn tableau and present myself well. But what would I do to even prove what I can do before an interview? Putting a notebook on github where I imported a dataset, cleaned it a bit, did a .desribe(), .plot() and a linear regression isn't very impressive nor interesting to anyone. So what would I do instead?

Also, this clearly isn't data science area yet. If I look at kaggle challenges, I either don't know what to do or think to myself "Clean data, LinRegression". So what should I take a look at next?

Note that I'm taking classes, but not in Data Science but in Chemistry right now.

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to.

1 - You need the basics, that is already much more than you expected it to be:

Linear Algebra: knowing the best way of inverting a matrix might be useful for a computer scientist, but you're not aiming for that. You need to understand concepts and their meaning and effects such as:

Matrix Rank (For example, this could tell you, by an Autocorrelation matrix, that your data is still not enough for things like least squares.)
Meaning of Vector Spaces and basic linear transformations such as base change
Meaning of eigenvalues and eigenvectors

Calculus: also, focus on the meaning and understanding, computers can do most of the operations, even analytically

Derivatives and Integrals
Optimization

Signals and Systems: that might be a little bit biased (since I am a signal processing researcher) of me but, learning how to model certain phenomena and how they behave may help you to solve problems, that's basically applied Linear Algebra and Calculus (most things are). Aim for really basic, Signal processing is one of the research areas most affected by Data Science/Machine Learning, to the point people were making surveys to change the name of the community in IEEE.

The above can be found on many, many books and intros and won't be difficult for you to search for them.

Statistics: Machine learning derives from statistics and this is essential. Actually, you can learn calculus from a statistic point of view instead of function optimization. Links in the subjects names are free courses to Udacity or Udemy

You will need to learn Descriptive Statistics, which will allow you to understand data.
And learning Inferential Statistics.

It is important to learn what you can do with classic statistics to avoid wasting computers in problems that could be solved easily. It is a good practice to model things in the simplest way possible and escalate to more complex model if needed.

Machine Learning: Machine learning models are your everyday tools as a Data Scientist:

You can start with a crash course on Google Developers and Intro to Machine Learning in Udacity.
Then, you can go deeper... NO! Not Deep Learning, deeper into Machine Learning, also on Udacity.
But you might want to take a more formal course such as Stanfords at some point, just to fill some voids.

Machine Learning Tasklist: You stopped at linear regression? So here are the most popular models waiting for you:

Linear and Logistic Regression: pretty basic
Decision Tree: basic but highly interpretable
Support Vector Machines: simple but powerful (and I love Kernel methods, shoot me for that)
Naive Bayes
k Nearest Neighbors
K-Means Clustering
Random Forests and Gradient Boosting (Ensembles): these are really powerful and might be interpretable if you don't let them grow without care

Also, you might need some dimensionality reduction tools such as:

Principal Component Analysis
Linear Discriminant Analysis

Then you can go for more complex Neural Networks:

Self-organizing-maps, Feed Forward, RNNs, CNN's and so on...

Note: CNN's are usual in Computer Vision applications but the only thing it asks of data is that it has been organized in a way that allows for meaningful correlations between data that is near each other. Example: a process with multiple sensors in a time series might benefit from a CNN.

Data Analysis: a Data Scientist must have a personal relationship with data. For any good relation, you will need to understand your loved one (but the relationship with data is usually toxic, hahaha). Udacity has a nice Intro to Data Analysis, also for free.
Learn to express yourself:

make a youtube channel, present small tutorials, and classes to the community.
Try to answer Stack Exchange questions and help others, that will build your respect in the community and goodwill for when it is your time to ask. Also, this is a good way to practice expressing your ideas in a text.
Write a blog, it is a good way to have a notebook and also gaining attention from the community.

You can check on hands-on books such as Data Smart and Data Science from Scratch. Data Smart is about getting Insight from information and that's mostly your job as a D-Scientist.

2 - Build Respect

Try creating packages and libraries and making them available on GitHub and sharing your relevant solutions.
Win Kaggle competitions, many companies take Kaggle seriously as .... and having a good score will get you nice positions. You don't need to win in first place by the way. Also, competitions are usually good examples of real-world problems that will get you that so needed experience that you can't get while you don't have a Data Scientist role.
Also, some competitions pay really well.
Explore Kaggle, share algorithms, read and try to improve on others and search for datasets that might be of your interest.
Making datasets is a bit exhaustive but might be a way of making money while you're not ready.

3 - Get programming skills

Not only learning frameworks but understand how things work and basics of problem-solving must be put to the test every day. Also, making everything from scratch is fun and is good for learning, but when you do your work you will need code with high maturity, checked hundreds of times all over the world.

You'll need some tools, Python is a great language for data science (since the community is active and it is free, Matlab has a lot of nice tools and marvelous documentation but it is really expensive and quite a bit slow)

Some top libraries:

NumPy is the most fundamental package, understand it well
Pandas for wrangling with data
Seaborn, Bokeh, Plotly, and Matplotlib for plotting stuff and helping you making good reports
SciKit-Learn this is usually the fastest way to test a machine learning algorithm
Theano is similar to NumPy but was constructed with Machine Learning in mind
Keras, this is a library for building Neural Networks really fast, it uses Theano or TensorFlow as backends
TensorFlow, PyTorch, and other deep learning related stuff.

Also, you might want to get some knowledge of JavaScript and libraries for acquiring data on the web.

4 - Go Deeper

You may never need deep learning, depending on the area you're going to apply but this is that good nuclear weapon you hope to never have to use but someday you may:

Understand start with Udemy
There is a nanodegree on Udacity

Also, remember DL is computationally intensive and you want to avoid needing it (since these are expensive)

5 - Finally: Career

Learning never stops and you will never stop learning new concepts, every single day.
Courses are long, take them at your own pace. Try getting the basics of learning how to use, then go back and learn it for real
Try to get a few certificates and posting them on your LinkedIn. Make a few projects and create online articles on your blog and on LinkedIn and Kaggle.
Try choosing something you can relate to while searching for work. DS covers a wide range of subjects and trying to get insight from things you understand is easier than trying to get insight from things that sound like random noise for you
Build a network of collaborators, help your colleagues and try building a vast network ranging from medicine to linguistics, they might tell you what you're doing wrong while looking at data not related to your field of expertise.
Finally, and this took a bit longer than I anticipated but: don't give up. This is a long journey but is absurdly rewarding, both financially and personally. And try not working alone, create a small group of people to work with and make some projects.
This is my longest answer to a Stack Exchange question.


When should I use Gini Impurity as opposed to Information Gain (Entropy)? Can someone practically explain the rationale behind Gini impurity vs Information gain (based on Entropy)?

Which metric is better to use in different scenarios while using decision trees?

Gini impurity and Information Gain Entropy are pretty much the same. And people do use the values interchangeably. Below are the formulae of both:

$\textit{Gini}: \mathit{Gini}(E) = 1 - \sum_{j=1}^{c}p_j^2$
$\textit{Entropy}: H(E) = -\sum_{j=1}^{c}p_j\log p_j$

Given a choice, I would use the Gini impurity, as it doesn't require me to compute logarithmic functions, which are computationally intensive. The closed form of it's solution can also be found.

  Which metric is better to use in different scenarios while using
  decision trees ?

The Gini impurity, for reasons stated above.

So, they are pretty much same when it comes to CART analytics.

Helpful reference for computational comparison of the two methods


Automated bug finder through machine learning? Assume that I have a large software application, this application is event driven i.e. when a user configures something by pressing on a button a code flow executes.

Along with this application I also have a system which can emulate a user and can perform all permutation of steps which a user might perform. Hence it is safe to say that I can bring my software application to all the possible states. I have already done the automation for this, I just need a brain which can say that the current state is buggy (undesired) or not.

Now the question is:

  Is it possible to determine if my system has gone into an unexpected state (Bug) automatically via machine learning?

At first I thought of using Rete Net to solve this. Idea was to make a knowledge base which will emulate a tester and can tell if the system is in an inconsistent state or not, but for this I need to constantly feed the knowledge base with new rules whenever a new feature is added to this software application.

Is a difficult task for a machine learning method. I would think that there are methods proper to software development that are much more appropiated for the task.

With machine learning, I could think that process analytics is what you are looking for:

I imagine you may have a dataset like this:

[step1: click in LOGIN, step2: Click in Button A, step3: Click in Button B]  -> [No Bug]
[step1: Click in LOGIN, step2: Click in Button Z]                            -> [Bug]
...

Business Process Analytics is the family of methods and tools that can be applied to these event streams in order to support decision-making in organizations. The analysis of process events can focus on the behavior of completed processes, evaluate currently running process instances, or focus on predicting the behavior of process instances in the future. Process Analytics

Is the most similar process (to your question) in Analytics I know


Text classification: accuracy I would like to understand how to compute the accuracy of cluster analysis.
I have hundred of texts. The dataset looks like as follows:

Author  Text Checked
Shakespeare ‘...’ yes
Keats ‘...’ no
...

In text there are some quotes from the author. Checked column has been filled in manually.
What I would like to do is to group texts by authors.
I already did it using kmeans and TfidfVectorizer. However I would like to understand if it is possible to test the accuracy of this approach.
Do you know how I could do it?
In case it would be not possible, could you please tell me what other approach I could take into account?
It would be good to test different approach and see the level of accuracy.
Any example it would be great.

There are different ways to address the task that you describe:

If the goal is simply to predict the author among a set of predefined authors, then this is not a clustering task (unsupervised) but a classification task (supervised). This implies that you would split the data between training and test set (or use cross-validation), train a model using the training set then apply it the test set, and finally evaluate the performance on the test set by comparing the predicted author with the true author for every text. Typically the performance is aggregated as f1-score for every author, then macro/micro f1-score.
If the goal is some kind of experiment to see whether a clustering algorithm can correctly identify the groups by author (spoiler: it certainly can't, at least not well), then you're doing unsupervised learning with an external evaluation. I'm not aware of any standard evaluation method for this context (there might be). I assume that you would need a matching method between the clusters ids and the authors (e.g. assign each author to the cluster which has the most instances by this author), then you can apply the regular classification evaluation.

Two additional points:

The general problem of distinguishing texts by their author is called stylometry. There are many variants, such as open or closed classification, author verification, author profiling, etc. It's an active area of NLP research (which means there's a lot of literature about it), and state of the art methods are quite different from regular classification/clustering. It doesn't meant that regular methods don't work at all, but usually not as well as specialized methods.
Features: TF-IDF words vectors are used for tasks where the "topic" of the text matters, for instance to classify articles about sports, politics, etc. There are better options for the features when the task is about writing style:

no focus on rare words (as IDF does), instead rare words should usually be discarded (because they cause massive overfitting)
words are not necessarily the right unit to consider. Character trigrams/4-grams have been proved to work better.
tokenizing can be tricky, it's often considered safer not to tokenize at all so that the author's punctuation habits are preserved as features.


How to compare the performance of feature selection methods? There are several feature selection / variable selection approaches (see for example Guyon & Elisseeff, 2003; Liu et al., 2010):

filter methods (e.g., correlation-based, entropy-based, random forest importance based),
wrapper methods (e.g., forward-search, hill-climbing search), and
embedded methods where the feature selection is part of the model learning.

Many published algorithms are also implemented in the machine learning tools like R, Python, etc.

What would be an appropriate method to compare different feature selection algorithms and to select the best method for a given problem / dataset?
A further question would be, whether there are any metrics known that measure the performance of feature selection algorithms?

This is a hard problem and researchers are making a lot of progress.

If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data.

Two major caveats of LASSO are that (1) the selected features will not automatically detect an interaction, so you have to craft all of your features a priori (i.e., before running them through the model) and (2) LASSO will not identify non-linear relationships (e.g., a quadratic relationship).

A way to try and get past these two caveats is to use Gradient Boosted Machines which does feature selection automatically. It's worth noting the statistical properties of GBM are a little more ambiguous than that of the LASSO.

If you're looking for unsupervised feature selection, it seems there's a similar regularization approach used by these researchers, but evaluation in this particular case becomes less obvious. People try a lot of different things like PCA/SVD or K-Means which ultimately will try to find a linear approximation to the data.

In that case, the typical measures of performance are the reconstruction error or the RMSE of the clusters.

In terms of software, R and Python both have GBM, LASSO, K-Means, SVD, and PCA. GLMNET and XGBoost in R and Sklearn for Python are the relevant libraries.


Keeping DataFrame in Memory while working - R vs Python in VS Code I am considering migrating from R-Studio to Python. So I am just starting Python with VS Code as my editor. My purpose is mainly to analyze data, build predictive models and make such models available via a web API.

When working in R-studio with a large data set, I can run a script to read in the data from say a CSV file, then I can explore the data using the normal statistical approaches. I only need to load the data set ONCE, which is great since loading a large data set often takes some time. And data exploration requires lots of repeated "probes" to view histograms, compare counts etc. So not having to reload the data each time I change a script or run a script to view a plot, is a great advantage.

However, using Python in VS Code, it seems to me that if I load the data frame say in a method at the top of a script, then have a method to draw a graph, then each time I run that script, the data is reloaded. So if I want to draw 20 histograms, each time changing only the title, I need to do the data reload each time I modify and run the script.

Am I missing a feature of Python and VS Code or is my summary correct?

This is more about the IDE than the programming language. When you use RStudio, you can import the data and the IDE saves it in its memory so you don't have to reload it when you write subsequent code in that same session.

To do this in Python, consider using a Jupyter Notebook or something similar. That way, you only need to import your dataframe once, and you can then perform exploratory analysis, visualizations, and build models as needed.


AttributeError: 'Functional' object has no attribute 'uses_learning_phase' I am trying to predict on my test set and I get the following error
AttributeError: 'Functional' object has no attribute 'uses_learning_phase'

The model is created from the Class Model imported from from keras.engine import Module
All the module I imported are from tensorflow.keras except for the package mentioned above.
I also tried Importing everything from Keras alone but I still get the same error !
I have tensroflow 2.0 and keras 2.4.3

Could you please share some snippets of your code where the error arises? Generally attribute errors are caused when documentations are updated and that specific attribute is no longer present for the version! Importing new libraries (which you have tried already) or lowering the library version can help but for a definitive answer the code snippet is required!
P.S Sorry for posting this as an answer, I can't comment yet haha.


what is difference between set() and word_tokenize()? from nltk.tokenize import sent_tokenize ,word_tokenize

sentence = 'jainmiah I love you but you are not bothering about my request,
            please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)

  Actually both word_tokenize() and set(word_tokenize()) both returns
  same answers what makes the difference?

There are two differences between word_tokenize and set.

Word_tokenize

Returns a list (try print(type(word_tok)))
Returns all the tokens, regardless if there are duplicates

Set

Returns a set (try print(type(set_all)))
Returns all the unique tokens

Try this instead

sentence = 'jainmiah jainmiah jainmiah I love you but you are not bothering about my request, please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)


How do I get similarity with autoencoders I have build an autoencoder to extract from a very high dimensional (200 dimensions) space a smaller but significant representation (16 dimensions).

Now that I have these "encoded" vectors, I would like to compute some kind of similarity score, or clustering.

I am not sure which notion of distance to apply at this point. Any ideas how I can get similarity/clusters considering that I have used autoencoders?

You can calculate the cosine similarity between two encoded vectors you would like to compare. The cosine similarity between two vectors is defined as follows:


ANN regression accuracy and loss stuck I have a data set on predicting solar power generation,the dataset has 20 independent var and 1 dependent. The accuracy of my model is stuck at 60%. I have tried several models but this accuracy is best I could get other sucks even more.
Here is my code:
data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv'
dts = pd.read_csv('solarpowergeneration.csv')
dts.head()
X = dts.iloc[:, :-1].values
y = dts.iloc[:, -1].values
print(X.shape, y.shape)
y = np.reshape(y, (-1,1))
y.shape
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
from sklearn.preprocessing import StandardScaler
sc= StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
y_train = sc.fit_transform(y_train)
y_test = sc.transform(y_test)

import keras.backend as K
def calc_accu(y_true, y_pred):
  return K.mean(K.equal(K.round(y_true), K.round(y_pred)))

def get_spfnet():
  ann = tf.keras.models.Sequential()

  ann.add(Dense(X_train.shape[1], activation='relu'))
# ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))

  ann.add(Dense(1))
  ann.compile(loss='mse',
              optimizer='adam',
              metrics=[tf.keras.metrics.RootMeanSquaredError(), calc_accu])
  return ann

spfnet = get_spfnet()
#spfnet.summary()
hist = spfnet.fit(X_train, y_train, batch_size=32, epochs=250, verbose=2)

the accuracy and loss graphs are
plt.plot(hist.history['calc_accu'])
plt.title('Model Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.show()
plt.plot(hist.history['root_mean_squared_error'])
plt.title('Model error')
plt.xlabel('Epochs')
plt.ylabel('error')
plt.show()

after 50 epochs nothing seems to improve, neither curve seems to overfit on the data
I tried other models like reducing layers and removing kernel regularizes, using
kernel_initlizers='normal' and 'he-normal'

but they perform poorly stuck at 20%.
dataset:

There are two issues here.
First, you are in a regression setting, where accuracy is meaningless - see What function defines accuracy in Keras when the loss is mean squared error (MSE)?
Although the linked thread is not directly applicable here (since you use your own custom calc_accu function), the essence of the argument remains: assessing K.equal(K.round(y_true), K.round(y_pred)) is not the correct way to measure the performance of a regression model. In such settings often the loss (MAE, MSE, RMSE) and the metric are the same; what you do here (accuracy aside), i.e having an MSE loss and an RMSE metric is also valid.
Second, there is no way to tell if an RMSE of ~ 0.5 is "good" or "bad" in itself, even more since here you scale your output y before feeding it to the model, hence the calculated RMSE is on these scaled values, and not in the true, unscaled ones. What you have to do is calculate the "true" RMSE in the original unscaled values, and then decide if it is good enough for your purpose or not. Essentially what you have to do is:

Get predictions for your (scaled) test data
Transform back your predictions to the initial scale of the original test data
Get the RMSE between these invert-transformed predictions and the original data

To do so (and a good practice in general), it is necessary to define different scalers for your x and y, i.e.:
from sklearn.preprocessing import StandardScaler

# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)

# outcome scaling:
sc_y = StandardScaler()
y_train = sc_y.fit_transform(y_train)
y_test = sc_y.transform(y_test)

Now, assuming the data have been scaled as I show above, here is how you should calculate your RMSE on the original (unscaled) test data:
from sklearn.metrics import mean_squared_error

y_pred = model.predict(X_test) # get model predictions (scaled inputs here)
y_pred_orig = sc_y.inverse_transform(y_pred) # unscale the predictions
y_test_orig = sc_y.inverse_transform(y_test) # unscale the true test outcomes

RMSE_orig = mean_squared_error(y_pred_orig, y_test_orig, squared=False)

It is this RMSE_orig that you should decide if it is satisfactory for your problem; plotting the original unscaled predictions y_pred_orig against the original (unscaled) outcomes y_test_pred might also be useful and informative.
See own answer in SO thread How to interpret MSE in Keras Regressor for a similar case.

Other than that, it is speculative to say anything more since we do not have your data; nevertheless, a general rule is that we don't use dropout by default - only when we suspect that our model might overfit the data. So, I suggest experimenting with all dropout layers removed, and possibly putting them back in only gradually, and only in the case you observe overfitting.


Why is the logistic regression decision boundary linear in X? The logistic regression model,

\begin{equation}
\operatorname{p}(X) = \frac{\operatorname{e}^{\beta_0 + \beta_1 X}}{1 + \operatorname{e}^{\beta_0 + \beta_1 X}}
\end{equation}

is said to create a decision boundary linear in $X$.

As far as I understand, only the logit is linear in $X$. Is this the reason the decision boundary is linear in X? If so, why? And if this is not the case, what is the reason for this phenomenon?

I am confused about this because the decision boundary can be expressed as:
\begin{equation}
\operatorname{p}(X) = a, a \in [0,\, 1]
\end{equation}
And $\operatorname{p}(X)$ is not linear in $X$.

The decision boundary $p(X)=a, a\in(0,1)$ is

$$\beta_0+\beta_1X=\log\frac{a}{1-a}$$

which is linear in $X$.


Orange Import and process spectra time dependent/kinetics I would like to import FTIR spectra that have been collected over time then use Orange to preprocess these as a group. I currently have the spectra as individual SPC or CSV files. How do I import the spectra then plot them as a stack? I have tried using the "Multi-file' widget but that does not work.

To accomplish this you can use the Orange Concatenate Widget under the Data tab shown below:

This widget takes a primary data set and an additional dataset and concatenates them into one dataset. Once all of your data points are contained in the same dataset, you should have no problem plotting them all together.


How to split temporal sequences to sub-sequences in a meaningful yet unsupervised manner? I have a biological process that undergoes some cellular event which I am observing.
I have a series of events, with different temporal gaps between them.
For example

30, 0, 65, 0, 100, 0, 300, 25, 5, 5, 5, 235, 0, 5, 480, 0

That is, 30 seconds between the first event and the second one, the third even occurring together with the second one (activation events can occur in different locations in the cell and I classify them as distinct events), the forth even occurring 65 seconds after the third event, and so on.

I assume that one event can trigger the next one (spatial distance also has an effect but we have decided to ignore it at this point) - forming a sub-sequence of events. Or the events can be non-related, and thus belong to different sub-sequences. I assume that this can generally determined by temporal distance.

The question is - how?
I know that the temporal distances between different sub-sequences should be larger than those between events in the same sub-sequence, but I still need to decide what the threshold is.

Seeing as I don't know what a good separation would be, I think that an unsupervised approach might be useful here.

However beyond that I do not have any idea how to approach that.

Do any of you have any methodologies suitable for this endeavor, or any other insights and tips?

Many thanks.

It seems like your problem is not a typical "time series" problem: in data science and related problems, we normally look at evenly-spaced time series (that is, a measurement every interval, with each interval being the same length, such as 1 second, 3 months, 24 microseconds, etc).

There exist methods for working with events that can take some random time to happen, see survival analysis. If the coordinate domain (e.g. cell location) is important, you probably need a much more structured statistical model. Either way, it feels like unsupervised learning won't help you much.

Unfortunately, you haven't explained your data format very well. Without a
good understanding of your $x \rightarrow y$ relationship, nobody will really be able to help you. What is being observed? What do the numbers mean? (Time until the next event?) What groups are you trying to get? (Sequences of cell events?)


How to adapt output with multiple columns for better presentation? Recently I've started using RMarkdown in RStudio, and I don't know how to adapt the output for a useful presentation. The variables in the data set being: Cia, Riesgo, Tipo.Refa, Estado.Poliza, Tipo.Endoso, Asegurado, Moneda, Suma.Asegurada, Prima.Pesos, Premio.Pesos, Puntos, and Pesos.
The only thing I want is find a way to present the data in a more understandable format, unlike the images I attached. (dimensions of the R object are 9 rows, 12 variables)

A bit of help would be much appreciated.

```{r setup, echo = FALSE}
knitr::opts_knit$set(root.dir = '/Users/Igna43/Dropbox/Nacho/Nacho 2016/Análisis R')
```

```{r echo = FALSE}
data1<-read.csv(file = "ListadoEnero2016.csv", header = TRUE, sep = ",")
opneg1<-subset(data1, data1$Pesos. < 0, )
opneg1
```

You can use kable() (see manual to create tables that are directly in markdown format.

Your code would become:

```{r,results='asis'}
 knitr::kable(opneg1)
```

See also this great manual on rmarkdown.


What's the of all values above some percentile called? How do I get it in pandas? Let's say I have a list of numbers, and I want the mean of all numbers that are greater than the 95th percentile. Is there some standard term for that value? ("Mean of a histogram bin"? "Conditional mean" on something?)
Is there a pandas standard library function to compute it?

I don't think there is a standard library for it.
You can just skip nan value as mean treat them as zero which can mess up the result.
import pandas as pd
df=pd.DataFrame(lst) #For a given list
df[df >= df.quantile(.95)].mean(skipna=True)


Predicting customers purchase Suppose we have a list of products categorized into 10 categories. We also have customers order details such as

  order_id, product_name, quantity, order_date

We want to know for a particular month what are the most probable categories from which customer might buy products. How can we approach this problem, Any suggestions.

I recommend using LSTM RNN or CNN algorithm to pick up the most popular product of on-going month based on the past purchasing history. I have created a working product ranking model for online shopping site lands'end in the United States.

In order to archieve the goal of this solution, you need to follow several steps.

collecting the dataset

I think you already gathered the dataset for this part.
Processing of the dataset for supervised learning.

Cleaning up the dataset - please remove rows that contains NaN columns
mapping the products into eyes - you need to map the categories and products into row matrix with the element of 1 and 0.
Normalization - you need to rescale all real numbers into [0, 1]
split the dataset into the training, valid and test data.

Build Deep learning model like LSTM RNN

select the deep learning framework. I recommend Tensorflow, Keras.
set the dimension of model. i.e. the number of layers, the number of neurons per layer.
optimizer, metrics

Training the model and testing
Get the predicted ranking of the category by monthly.

Protip: the most important thing to get the better result is which column would be set the output for ranking. You should sum up the total purchasing count per product or per category by month or week.


GAN to generate a custom image does not work I have been training a GAN in the cloud for some time now. I use Google's free credit. My laptop with a CPU doesn't seem to be up to the task.

The image I want to generate is this.

Even though the number of epochs is about 15000 I don't get anything close to the original.

This is the main code. I don't claim to fully understand the deep layers. It took a few days to even write this code. The rest of the code is boilerplate to train.

There is no compilation error and I look at the images using TensorBoard.

The output from the generator is (1024,1024) images. Should this be the same as my original which is a (299,299) images.

Should I calculate using formulas how each layer transforms the image to understand it better ?

How do I fix this ? I have mixed and matched API's just to create a working example assuming that doesn't create any problem.

X = tf.placeholder(tf.float32, shape=[None, 299, 299, 1], name='X')

Z = tf.placeholder(dtype=tf.float32,
                              shape=(None, 100),
                              name='Z')
is_training = tf.placeholder(dtype=tf.bool,name='is_training')

keep_prob = tf.placeholder(dtype=tf.float32, name='keep_prob')
keep_prob_value = 0.6

def generator(z,reuse=False, keep_prob=keep_prob_value,is_training=is_training):
    with tf.variable_scope('generator',reuse=reuse):
        linear = tf.layers.dense(z, 1024 * 8 * 8)
        linear  = tf.contrib.layers.batch_norm(linear, is_training=is_training,decay=0.88)
        conv = tf.reshape(linear, (-1, 128, 128, 8))
        out = tf.layers.conv2d_transpose(conv, 64,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.conv2d_transpose(out, 32,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        out = tf.layers.conv2d_transpose(out, 1,kernel_size=4,strides=2, padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.contrib.layers.batch_norm(out, is_training=is_training,decay=0.88)
        print( out.get_shape())
        out = tf.nn.leaky_relu(out)
        tf.nn.tanh(out)
        return out

def discriminator(x,reuse=False, keep_prob=keep_prob_value):
    with tf.variable_scope('discriminator',reuse=reuse):
        out = tf.layers.conv2d(x, filters=32, kernel_size=[3, 3], padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)
        out = tf.layers.conv2d(out, filters=64, kernel_size=[3, 3], padding='SAME')
        out = tf.layers.dropout(out, keep_prob)
        out = tf.nn.leaky_relu(out)
        out = tf.layers.max_pooling2d(out, pool_size=[2, 2],padding='SAME', strides=2)
        out = tf.layers.dense(out, units=256, activation=tf.nn.leaky_relu)
        out = tf.layers.dense(out, units=1, activation=tf.nn.sigmoid)
        return out

GeneratedImage = generator(Z)

DxL = discriminator(X)
DgL = discriminator(GeneratedImage, reuse=True)

Generating text as an image is extremely difficult and I have never seen a GAN applied in the image space to generate pages of text. The reason this is so hard is because of the way in which text is perceived by humans and the way a GAN works.

Humans read arbitrary symbols which are sequenced from left to right along the same line and combined into rows. Moreover, these symbols are combined into groups which represent words. This is extremely complex. The symbols must be intelligible, the words must be real ones as invented by humans. Lastly, the combination of words into sentences need to be logical and follow guidelines of human language. And even FURTHER the sequence of sentences must be coherent to transmit a message.

A GAN operating in image space will try to learn the distribution of the training set in a pixel-wise manner as that is your inputs. The distribution of the pixels will not effectively be able to group characters together in a logical manner, and the words will not be real, and the sentences will all be nonsense. You will most likely end up with blurry lines of random looking symbols, kind of like a zebra print.

Another problem is the amount of data you have. Even if this problem was possible with a GAN you would need tens of thousands of instances to train a GAN effectively.

What I suggest

I suggest to read the texts that you are training with and use this data to train a LSTM which has been shown to be very effective for generating language. But, be warned even with the best LSTMs you rarely get text that can fool humans into thinking its real.

The LSTM will provide you with your text.

Then you can train a GAN to generate the characters of the alphabet and you can use this generator to print out the text that the LSTM generates.


high accuracy on non trained class in tensorflow model I have trained a 4 multi-class (apple_nature, apple_disease, apple_blacrot, apple_healthy) image classification algorithm using TensorFlow. However, after training, we get a good accuracy model.
The problem I am facing is when I predict tomato images it gets high accuracy on apple values, so how can I fix this issue?
Actually, it gets a high accuracy rate on non trained classes.

It is normal because the model has never seen such data so the probability for such unseen data is like random, it can be high or low.
You should add all other images (like the ones of tomato) into a new dedicated class OTHER where you will put everything which is not in the 4 original classes.
That way, for an image of tomato your model will give a high probability for the OTHER class, and low probability for the apple classes.


What are the cases in which Isomap fails to do a good job? As above, what is a possible scenario/ dataset/ case in which Isomap fails to do a decent dimensionality reduction?

Here is the visualization of COIL-20 data set in t-SNE paper:

The data-set consists of images of 20 objects (clusters). In all cases provided in the paper and some cases I found on the Internet (6000 MNIST data set, slide no. 31), the quality of IsoMap (2001) is not comparable to t-SNE (2008).


How can I adjust the legend when visualizing clusters in two dimensions? How can I change the legend as we can see now the legend has some cluster numbers missing. How can I adjust the legend so that it can show all the cluster numbers (such as Cluster 1, Cluster 2 etc, no it's only 0 3 6 9)? (codes I followed this link: Perform k-means clustering over multiple columns)

kmeans = KMeans(n_clusters=10)
y2 = kmeans.fit_predict(scaled_data)

reduced_scaled_data = PCA(n_components=2).fit_transform(scaled_data)
results = pd.DataFrame(reduced_scaled_data,columns=['pca1','pca2'])
sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results)
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.show()

Edit: the legend seem not the same, the cluster 0 should be the lightest color.

Yeah this is an annoying weirdness of seaborn. Just pass legend='full' as a parameter to sns.scatterplot(). So your code becomes:
kmeans = KMeans(n_clusters=10)
y2 = kmeans.fit_predict(scaled_data)

reduced_scaled_data = PCA(n_components=2).fit_transform(scaled_data)
results = pd.DataFrame(reduced_scaled_data,columns=['pca1','pca2'])
sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results, legend='full')
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.show()

I have no idea why you'd ever want half a legend! But the parameter defaults to 'brief'. Refer to the seaborn documentation:

legend : “brief”, “full”, or False, optional
How to draw the legend. If “brief”, numeric hue and size variables will be represented with a sample of evenly spaced values. If “full”, every group will get an entry in the legend. If False, no legend data is added and no legend is drawn.

edit: To have the clusters named in the format you're after, first create a list in the right format, and then call plt.legend() and pass that list as an argument to legend():
legend = []
for i in np.unique(y2):
    legend.append('Cluster {0}'.format(i))

sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results, legend='full')
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.legend(legend)
plt.show()

To change the location of the legend, use the loc= parameter in the plt.legend() call. For example plt.legend(legend, loc='upper right') or plt.legend(legend, loc='upper left')


Strategies to encode categorical variables with many categories I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the respective count frequencies of the variables in the column. I understand why we need to transform the variables, but can someone please explain why this approach was used. What exactly is the logic behind it?

Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables (excluding the obvious one hot encoding) or point me some interesting approaches for the same. Thank you!

Generally, the logic of the categorical count transformation lies in the fact that features with similar frequencies tend to behave similarly. Have words in a corpus as an example, common words share little or no real information whereas uncommon words share more information with an algorithm.

Specifically, certain algorithms (tree-based methods) could even yield rules given an unspecified category from an event count. Say, for example, we have an unknown category whose count is 4. The algorithm may give a rule:

  If Column Count is < 5 and N is > 3 = X

That will be exactly the same as if an algorithm took a One-Hot encoded column and gave a rule:

  If One-Hot-Encoded-Column is > 0 = X

In that case, a tree-based algorithm will make several rules from many categories using the same count column. But how I said at the beginning, algorithms generalize among populations of similar counts so most likely you will find rules like:

  If Column Counts is < 10 and N is > 3 = X

Which will often contain different categories that behave similarly. Just check the models and look for parameters/ importance of the column in question to see this for yourself.

  Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables.

Feature hashing became really popular there not so long ago. Hashing has very nice properties and it's a whole topic learned at schools but the main principle is that if you have a category with high cardinality you decide a minimum number of reduced categories (hashes) that all the categories will have to share. if two categories share the same hash or bucket, that is called a hash collision. Feature hashing doesn't deal with hash collisions because according to some authors (I don't have the reference here) may improve accuracy by forcing the algorithm to pick more carefully the features.

There are many ways we can encode these categorical variables as numbers and use them in the algorithm.

1) One Hot Encoding
2) Label Encoding
3) Ordinal Encoding
4) Helmert Encoding
5) Binary Encoding
6) Frequency Encoding
7) Mean Encoding
8) Weight of Evidence Encoding
9) Probability Ratio Encoding
10) Hashing Encoding
11) Backward Difference Encoding
12) Leave One Out Encoding
13) James-Stein Encoding
14) M-estimator Encoding

Find the below cheatsheet

More Info


Communicating clearly about "samples" It is often pointed out that sample is an overloaded term in statistics and the sciences being supported by statistics.  In my field (geological sciences) as in most other sciences, the process of collecting meaningful data is critical and discussions about the traps and pitfalls in that process talk about sampling.  Not far down the road from that, particularly when lab results are back, conversations involving statisticians, data scientists, geomathematicians, GIS analysts and even normal geologists are likely to attempt to include multiple meanings of sample in the same sentence!

Q: Have any data scientists (or statisticians) found practical ways to communicate these different meanings?

One way is to always add soil, rock, statistical and so on before sample.  But I was curious if there are any other approaches to effective communication that are in use.

"Sample" as a noun usually refers to a single data point.  "Sample" as a verb is the act of extracting a subset of data points from some larger body (reality or a larger dataset).  The only way to be less ambiguous is use more specific words than "data" or "sample".

Example:

Say you collect 1MM data points from four different sensors in the field giving you four sets of 250k data points.  Say this data is too big for some demo of a model you're testing or an analysis you're running, so you pick 100k data points evenly split across the four sensors (giving four sets of 25k data points).

In this example, you're sampling twice.  First, to gather your 1MM data points sampling from reality.  Second, you sample again to decrease the size of your data set to something more manageable.  'Data' or 'sample' could refer to reality, the 1MM dataset, the 100k dataset, or any of the sensor-specific subsets.  To make it less ambiguous, establish a unique name as soon as possible for each possible definition you'll be working with.  ('reality' for the set of all possibly observed samples.  'the complete dataset', something derived from the source of the dataset, or even X for the full 1MM dataset.  'our trial dataset', or even Y for the small 100k dataset.

What you actually do comes down to context and what's appropriate for your intended audience, but the general answer is to use more specific words.


Apply cross validation on regression algorithms How can we apply K-fold cross validation on say linear regression? Regression contains weight update and iterations, and so do we integrate K-fold to it?

If done, is it like we take the first K-1 fold to train using linear regression and perform iterations to obtain weight, then take the next fold and proceed the same and so on until all folds are done and pick the best weight or fold from it?

How to use cross-validation on regression (assuming 10-fold for example purposes): separate your dataset in 10% and 90%, train on 90%, test your metric (squared error or anything you're modeling) on the remaining 10%. Do that 10 times using different 10% groups. Now you have 10 of your metrics, you can analyze its mean and range to see if the model is robust against over-fitting.

How do you pick a model: you train on the whole dataset. Cross-validation is for testing/validating purposes, you don't use the models generated by cross-validation.


When is the sum of models the model of the sum? The response variable in a regression problem, $Y$, is modeled using a data matrix $X$.

In notation, this means:

$Y$ ~ $X$

However, $Y$ can be separated out into different components that can be modeled independently.

$$Y = Y_1 + Y_2 + Y_3$$

Under what conditions would $M$, the overall prediction, have better or worse performance than $M_1 + M_2 + M_3$, a sum of individual models?

To provide more background, the model used is a GBM.  I was surprised to find that training a model for a specific $Y_i$ resulted in about equal performance than using the overall model $M$ to predict that $Y_i$.  The $Y_i$'s are highly correlated.  In hindsight, then this is not surprising because training a model for a vector correlated with the target also is correlated with the target.

For analogy, take the case with a linear model and independent response variables.

The overall model is

$Y = X\beta$

It is trivial to see that the sum of the models is the model of the sum.

$Y = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$

If the $Y$'s are independent then the $\beta$'s will be as well.  This implies that each of the model coefficients will be unchanged.  Take for example a two-dimensional case (where $X$ has two columns).

For $i \neq j$, $Y_i = X(\beta_i + \beta_j) = X\beta_j + 0$

The two models are not equivalent in general, either it might happen they provides similar results. There are multiple issues here. What you have actually in $Y$ is something like:
$$Y = \tilde{Y} + \epsilon = \tilde{Y_1} + \epsilon_1 + \tilde{Y_2} + \epsilon_2 + \tilde{Y_3} + \epsilon_3$$
where $\tilde{Y}$ and $\tilde{Y_i}$ are the true functions which you want to approximate and $\epsilon$ and $\epsilon_i$ are errors/noise.
One issue is how do you split output $Y$ into components. Supposing you have knowledge about components of $Y$ and you split it into simpler functions which are easier to fit independently. However in practice you have a sample and each observation has errors, you would have to split also the error components, which is not easy since most of the time the assumption used is that the errors are at least independent, if not identical distributed also.
The second one is the variance of the models. In the analogy with linear equations you stated
$$Y = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$$
Well, this is actually true only on expectation, aka:
$$EY = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$$
But your predictions have also variances, which can put you away from the true functions. If we assume the noise for each component is mutually independent, than the variances accumulates, giving an inflated variance for the summed model $$Var(\epsilon)\leq Var(\epsilon_1) + Var(\epsilon_2) + Var(\epsilon_3)$$
If the component errors are not independent than you can also add the covariances to the issue and you can easily go into trouble.
As a conclusion if you have three sets of observations for each $Y_i$ than it should be usually better to fit them separately rather than summed. But if you have the summed data set it is usually much harder to split data and the results have better chances to be more unstable, even if on expectation you have the same target.


How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)? I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in xgb.train(). The label (Y) of dataset itself has 4 classes and is highly imbalanced, so I would like to generate per-label PR curves for it to evaluate its performance, and would thus need to treat each class as it's own binary problem using a one-vs-rest classifier. After a lot of reading I haven't found an equivalent to sklearn's OneVsRestClassifier in the xgboost library. Could anyone provide some guidance on how to implement continuously training one-vs-rest classifiers using the XGBoost library? Thank you in in advance.

Try to give sklearn wrapper of xgboost to sklearn OneVsRestClassifier.
https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn
If it doesn't work you can create four different label arrays in which samples belong to corresponding class labeled as 1s, and others as 0. Then training with each of the label arrays you can get results for each class.


When developing machine learning models, is the size of each class in the test set important? I am thinking about the prospective application of a trained classifier in a real-world context. We know that when we do over/under-sampling to balance our dataset, we never touch the testing set as we want to keep our dataset's real behaviour. But the part that I do not understand is the role of the test set's distribution in a classifier's performance.
Let's say I have a model that can label an email as spam or non-spam. If I launch this model in my email-service, in a specific time window, all the emails that my classifier receives might be non-spam. But the trained model has a 50-50% distribution for each category. My question is, does this difference in the distribution--during the prospective application-- change the performance of the model? e.g. if my web-service receives 5 spams and 5 non-spams in that time window, should I receive a more accurate classification? Based on my understanding, the answer should be a No. Still, I see everywhere that people are talking about the importance of the test distribution and its role in the performance and accuracy of predictive models.
Thank you.

Depends if you are going to do online learning.
Lets say you will do online learning/incremental learning than test set Distribution will make difference. For example because of catastrophic forgetting of neural Networks.
If you are making Batch predictions than it makes no difference whats the test set Distribution. Model knows no difference since it does not Change ist state.


Subsequent convolution layers Note: I've read How do subsequent convolution layers work? a few times, but it's still difficult to understand because of the parameters $k_1$, $k_2$, and many proposals (1, 2.1, 2.2) in the question. This seems to be complex for other people too, I think I'm not the only one (see a few comments like "I have just struggled with this same question for a few hours"). So here it is formulated with a particular specific example with no parameters, to grasp the idea more easily.

Let's say we have a CNN with:

input: 28x28x1 grayscale images (28x28 pixels, 1 channel)
1st convolutional layer with kernel size 3x3, and 32 features
2nd convolutional layer with kernel size 3x3, and 64 features

Keras implementation:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(Conv2D(64, (3, 3), activation='relu'))

Question: how does the 2nd layer work?

More precisely:

for the 1st layer:

input size: (1, 28, 28, 1)
weights size: (3, 3, 1, 32)  (good to know: the number of weights doesn't depend on input pixel size)
output size: (1, 26, 26, 32)

for the 2nd layer:

input size: (1, 26, 26, 32)
weights size: (3, 3, 1 32, 64)
output size: (1, 24, 24, 64)

How is the latter possible? It seemed to me that, in the 2nd layer, every input 26x26 image will be convolved with the 3x3 kernels of each 64 feature maps, but this could done for all the 32 channels!

Thus I had the feeling the output of 2nd layer should be (1, 24, 24, 32*64)

How does it work here?

I think I found the reason: the correct description of the 2nd layer is:

for the 2nd layer:

input size: (1, 26, 26, 32)
weights size: (3, 3, 32, 64)
output size: (1, 24, 24, 64)

So for each one of the 64 features, the (26,26,32) input is convoled with a (3,3,32)-sized kernel, producing a (24,24) output.

Since this is for each one of the 64 features, the output will finally be (1, 24, 24, 64).

Code to display the shape of the weights:

for l in model.layers:
    if len(l.get_weights()) > 0:
        print(l.get_weights()[0].shape)        # ...[1].shape  would be for the biases


Why do we double the number in a quadratic cost function or MSE? $$ C(w,b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $$

Where y is a 10-dimensional vector, a is the output, w is the weight and b is the bias and n is the number of inputs. If this is the MSE, shouldn't it be $\frac{1}{n}$ instead?

Link

This is really just like a convention that appears in some places because we normallt want to take the derivative of the cost function (i.e. compute gradients), which means the power of 2 would be taken to the front.

If we put the $\frac{1}{2}$ at the front to begin with, it just looks nicer once we have finished. I have seen this written somewhere before in a paper, but can't find a reference right now.

Because the nominal values of cost themselves (the scale of the values) is not of importance, we can scale it as we like really. Multiplying by a constance of $0.5$ does not change the algebraic behaviour.


how to check all values in particular column has same data type or not? I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. Column is as following:

1 abc
2 def
3 ghi
4 23
5 mno
6 null
7 qwe
8 12-11-2019
...
...
...
4900 ert
5000 tyu

In above case, I can use for loop to find out rows which do not have desired dtype. I just wanted to know, is their better way to solve this issue.

Note: I am using Pandas.

You can get the type of the entries of your column with map:

df['ABC'].map(type)

So to filter on all values, which are not stored as str, you can use:

df['ABC'].map(type) != str

If however you just want to check if some of the rows contain a string, that has a special format (like a date), you can check this with a regex like:

df['ABC'].str.match('[0-9]{4}-[0-9]{2}-[0-9]{2}')

But of course, that is no exact date check. E.g. it would also return True for values like 0000-13-91, but this was only meant to give you an idea anyways.


It is because of the kind of convolution you've used. It is a valid convolution. If you want the output to be $128$, set the convolution to be same. Consider that this is also applicable to the deep layers too. They also can have either of these convolutions.


Preparing for interview - Logistic regression question So I'm doing some exercises to prepare for a interview test. However there's one of the assignments I don't understand. Maybe some of you can explain what they want me to do? It would help me to understand the underlying questionframes I may come across.
This is the assignment:
Logit Regression
Have the function LogitRegression(arr) read the input array of 4 numbers x, y, a, b, separated by space, and return an output of two numbers for updated a and b (assume the learning rate is 1). Save up to 3 digits after the decimal points for a and b. The output should be a string in the format: a, b
  def LogitRegression(arr):

  # code goes here
  return arr

  # keep this function call here
  print(LogitRegression(input()))

Logistic regression is a simple approach to do classification, and the same formula is also commonly used as the output layer in neural networks.
We assume both the input and output variables are scalars, and the logistic regression can be written as:
y = 1.0 / (1.0 + exp(-ax - b))
After observing a data example (x, y), the parameter a and b can be updated using gradient descent with a learning rate.
Examples:
Input: [1, 1, 1, 1]
Output: 0.881, 0.881
Input: [2.2, 0.0, 5.1, 5.7]
Output: 7.3, 6.7
What I don't understand is, do they only give me 4 scalars and want me to train on x, y and then predict a and b. Or is a and be supposed to be weights and bias and I need to return the trained ones? The output numbers in the second example doesn't match probabilities so it can't be that. I might be overthinking this and it's just a simple thing I need to do?
This is the code I've tried:
arr = np.array([2.2, 0.0, 5.1, 5.7])
arr2 = np.array([1.0, 1.0, 1.0,1.0])

def Logit(arr):
    learnrate = 1
    X = arr[0]
    y = arr[1]
    weights = arr[2]
    bias = arr[3]

    y_hat = 1/(1+np.exp(np.dot(X, -weights) - bias))
    new_weights = weights + learnrate * (y - y_hat) * X
    new_bias = bias + learnrate*(y - y_hat)
    print(new_weights, new_bias)

Logit(arr)

Output:
2.900000098664297 4.700000044847409

They give you the current values of the model parameters $a$ and $b$, and a new data point $(x,y)$, and they request to perform one training step with gradient descent using the data point, and returning the updated values for $a$ and $b$.
The problem with your code is that the sign of the learning rate is wrong in the parameter update. If you change it to minus, the result is as expected:
new_weights = weights - learnrate * (y - y_hat) * X
new_bias = bias - learnrate*(y - y_hat)


Fully connected layer in deep learning How to determine the best number of the fully connected layers in CNN? Can I use only one fully connected layer in CNN? How to determine the dimension of the fully connected layer output?

In CNNs, the convolutional layers are used to extract features in the input in order to reduce the cost function. These extracted convolutional features should be classified using dense layers. Consequently, the use of dense layers is for classifying convolutional extracted features. Based on the complexity of the high level features, extracted features in deep convolutional layers, you can have one or more layers.

Take a look at
How to set the number of neurons and layers in neural networks
 to investigate how to set the number of neurons in dense layers.

  Can I use only one fully connected layer in CNN?

Yes, you can. For instance, you can use only one layer for MNIST data set and get an acceptable learning.

  How to determine the dimension of the fully connected layer output?

The dimension of each fully connected layer is equal to the number of neurons in that layer. For instance, suppose you have a weight matrix $W$ which is $10\times20$. the latter number represents the number of output neurons. Consequently, the output dimension belongs to $\mathbb{R}^{20}$.


Keras: Provide One-Hot-Encoded input values to neural network I have a dataframe which has two columns of interest: A and B with string values. I am trying to build a prediction model which takes in a set of values in A as input and predicts the corresponding B values. I am trying to one-hot encode the string values before giving it to the neural network. This is what I have done:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from keras.models import Sequential
from keras.layers import Dense

def load_dataset(filename):
    # load the dataset as a pandas DataFrame
    data = pd.read_csv(filename, header=None)
    dataset = data.values
    X = dataset[:, :-1]
    y = dataset[:,-1]
    # reshape target to be a 2d array
    y = y.reshape((len(y), 1))
    return X, y

def prepare_inputs(X_train, X_test):
    ohe = OneHotEncoder()
    ohe.fit(X_train)
    X_train_enc = ohe.transform(X_train)
    X_test_enc = ohe.transform(X_test)
    return X_train_enc, X_test_enc

def prepare_targets(y_train, y_test):
    ohe = OneHotEncoder()
    ohe.fit(y_train)
    y_train_enc = ohe.transform(y_train)
    y_test_enc = ohe.transform(y_test)
    return y_train_enc, y_test_enc

# load the dataset
X, y = load_dataset('rdf.csv')

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)

# prepare input data
X_train_enc, X_test_enc = prepare_inputs(X_train, X_test)

# prepare output data
y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

# define the  model
model = Sequential()
model.add(Dense(32, input_dim=X_train_enc.shape[1], activation='relu', kernel_initializer='he_normal'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit the keras model on the dataset
model.fit(X_train_enc, y_train_enc, epochs=100, batch_size=16, verbose=2)

# evaluate the keras model
_, accuracy = model.evaluate(X_test_enc, y_test_enc, verbose=0)

print('Accuracy: %.2f' % (accuracy*100))

But I am getting this error: ValueError: Found unknown categories [...]. Is there any efficient way to solve this usecase?

what I understand from your code is you are fitting a one-hot encoder on your training set, which may not include all words that appear in your test set. So when you get a new word in your evaluation method, your transformer cannot hash it, and hence throw an error.
the easiest way to solve this would be to use the unknown_error argument in one hot encoder
def prepare_targets(y_train, y_test):
  ohe = OneHotEncoder(handle_unknown='ignore')
  ohe.fit(y_train)
  y_train_enc = ohe.transform(y_train)
  y_test_enc = ohe.transform(y_test)
  return y_train_enc, y_test_enc

now when you encounter a new word in the test set your encoder will output an array of all zeros, and if your try inverse transforms on all zeros, you will get none.
The downside is obvious here, your Neural Network does not what to do for zero vector as it has never seen such an example in training. So expect a weird response when a new word is evaluated.
The smart way of solving this problem would be to assume there are some unknown words in the training set by synthesizing few rare (unique) words as 'unknown' (replace words with the word 'unknown') and then do the one-hot encoding. while evaluating you will then first check if the word is not part of words set in training data, then you will replace that word with 'unknown'.


How to specify steps_per_epoch and validation steps on infinite dataset? I have a huge csv dataset with a size of 200 GB. I'm using CsvDataset to make dataset generator for loading data from the disk while training the model. I want all the data to be passed on each epoch. So, what should I pass in the parameters steps_per_epoch and validation steps.
Here is my Keras model using the data_set.

    training_csvs =  sorted(str(p) for p in pathlib.Path('.').glob("path-to-data/Train_DS/*/*.csv"))

    training_csvs

    training_dataset=tf.data.experimental.CsvDataset(
        training_csvs,
        record_defaults=defaults,
        compression_type=None,
        buffer_size=None,
        header=True,
        field_delim=',',
        # use_quote_delim=True,
        # na_value="",
        select_cols=selected_indices
        )

    print(type(training_dataset))
    for features in training_dataset.take(1):
        print("Training samples before mapping")
        print(features)

    validate_ds = training_dataset.map(preprocess).take(10).batch(100).repeat()
    train_ds = training_dataset.map(preprocess).skip(10).take(90).batch(100).repeat()

    model = tf.keras.Sequential([
        layers.Dense(256,activation='elu'),
        layers.Dense(128,activation='elu'),
        layers.Dense(64,activation='elu'),
        layers.Dense(1,activation='sigmoid')
        ])
    history = model.compile(optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
                            metrics=['accuracy'])

    model.fit(train_ds,
        validation_data=validate_ds,
        validation_steps=20,  # I think it's wrong, on each epoch only 20 batches would be used...
        steps_per_epoch= 20,
        epochs=20,
        verbose=1
        )

This can be explained only if you specify the total number of samples in your entire data set. The 200 Gb file size might be because of a large number of features for each sample in the dataset. The is how steps_per_epoch and  validation_steps work.
The training generator will yield steps_per_epoch batches.
When the epoch ends, the validation generator will yield validation_steps batches.
so usually the ball-park value for a decent number of batches in training is 75.  and that of validation is 25. if youre having a train to validation split of 3:1 for a 1000 sample data set. so that each of train and validation set batches will have 10 samples each.


Import Error: cannot import name 'cv2' I want to begin exploring OpenCV in Python but I'm stuck at importing the package cv2. I have installed the package through pip3 install opencv-python and it got installed at this location - C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages.

When I'm trying to import cv2 using this:

import sys
sys.path.append('C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages')
import cv2

It gives the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/Users/Kshitiz/AppData/Local/Programs/Python/Python36-32/Lib/site-packages\cv2\__init__.py", line 7, in <modul
e>
    from . import cv2
ImportError: cannot import name 'cv2'

I have searched a lot but cannot find anything relevant. Please suggest what needs to be done.

According to the official documentation, if you had previous versions of opencv-python and opencv-contrib-python installed, you should consider removing them first:

 pip uninstall opencv-python
 opencv-contrib-python

Also, can you try installing them as sudo?

sudo -H pip install opencv-python


Word2vec compact models Tell me if there are any w2v models that do not require a dictionary. So, everything that I found in torchtext first wants to know the dictionary build_vocab. But if I have a huge body of text, I would like to have a model that works at the level of phrases. But I did not find one.

Every word2vec model requires a dictionary that maps an index to a string literal. This dictionary mapping greatly speeds up training because the training is completely integer-based.

The string literal values can be any length, from individual characters to phrases.


When using GridSearchCV with regression tree how to interpret mean_test_score? I am using GridSearchCV to tune hyperparameters of regression decision tree. When I do, I get mean_test_score but I thought it would return mean MSE since it is a regressor. how to interpret mean_test_score? Is there a way to tweak GridSearchCV so it returns mean MSE?
here is my code
tree_reg = GridSearchCV(DecisionTreeRegressor(criterion="mse"), {
    "min_samples_split":[2,3,4],
    "min_samples_leaf":[1,2,3]
}, cv=5, return_train_score=False)

tree_reg.fit(X, y)

pd.DataFrame(tree_reg.cv_results_)
>>> params                   split0_test_score  .... mean_test_score
   {"min_samples_leaf":2,        0.998782             0.9989933
    "min_samples_split":3}

   {"min_samples_leaf":2,        0.998823             0.998930
    "min_samples_split":4}
            ...

what does mean_test_score mean?

By default, GridSearchCV uses the score method of its estimator; see the last paragraph of the scoring parameter on the docs:

If None, the estimator’s score method is used.

And DecisionTreeRegressor.score (indeed, all/most regressors) uses R^2.

In response to your edit: you can specify scoring='neg_mean_squared_error'.  But note too that there's a linear relationship between MSE and R^2, so optimizing either of these are equivalent.


How much data should I allocate for my training and and test sets? (in R) I have a matrix of 358.367 data.
Each row is a DNA sequence from the human genome.
I want to build a classification model in R, using XGBoost algorithm and 83 features (dinucleotides, trinucleotides, etc.).

How should I split the data for the train and test set?

For example 70% for the train set and 30% for the test set?
30% for the train set and 70% for the test set?

There is no "golden rule" here. Your data set is very handy - neither too large nor too small. Sounds like a very exciting project!

Here is how I often proceed in comparable settings.

Do all splits stratified by response or, if the rows are not independent but rather clustered by some grouping variable (e.g. the family etc), grouped sampling. Important rule: avoid any leakage across splits.
Set aside 10%-15% of rows for testing. Don't touch them until the analysis is complete. Act as you would never utilize this test set.
Select loss function and relevant performance measure.
Fit a random forest without tuning and use its OOB error as benchmark.
Choose parameters of XGB by 5-fold cross-validation iteratively by grid search, first starting with very wide parameter ranges and then making those ranges smaller and smaller. The number of boosting rounds are automatically optimized by early stopping.
Choose model and present cross-validation performance.
At the very last, reveal test performance.


Interpretation for test score , training score and validation score in machine learning? Interpretation for test score , training score and validation score ? what they actually tell us?

What's an acceptable difference between cross test score , validation score and test score?

If difference between test score and training score is small mean it is a good model/fit?

overfitting and under fitting on the basis of test score , training score and validation score?

whether any of these score or difference among them tells us , if we need more data(observations)

Interpretation for test score , training score and validation score ?
what they actually tell us?

We usually divide our data-set in 3 parts. Training-data, validation-data and test-data. Then we analyze the score:
Training Score: How the model generalized or fitted in the training data. If the model fits so well in a data with lots of variance then this causes over-fitting. This causes poor result on Test Score. Because the model curved a lot to fit the training data and generalized very poorly. So, generalization is the goal.
Validation Score This is still a experimental part. We keep exploring our model with this data-set. Our model is yet to call the final model in this phase. We keep changing our model until we are satisfied with the validation score we get.
Test Score This is when our model is ready. Before this step we have not touched this data-set. So, this represents real life scenario. Higher the score, better the model generalized.

What's an acceptable difference between cross test score , validation
score and test score?

I think the "cross test score, Validation score" there are no difference. The right naming is "cross validation score". Cross validation is used by interchanging the training and validation data in different ways multiple times. A image that describes (Copyright reserved by towardsdatascience.com):

So your question should be "What's an acceptable difference between cross validation score and test score?" There's no straightforward answer to this question. Here's a long discussion you might want to consider: Cross validation vs Test set.

If difference between test score and training score is small mean it
is a good model/fit?

Yes!! This is what we strive for. And often times to achieve this, We require various engineering techniques, calculations and parameter tuning.

overfitting and under fitting on the basis of test score , training
score and validation score?

Usually, high training score and low test score is over-fitting. Very low training score and low test score is under-fitting. First example here, in technical term is called low bias and high variance which is over-fitting. The latter example, high variance and high bias called under-fitting. In other words,  A model that is underfit will have high training and high testing error while an overfit model will have extremely low training error but a high testing error.

whether any of these score or difference among them tells us , if we
need more data(observations)

This is a very good question. you can refer here. And the best explanation of this I got from Andrew NG. This is the link. These will give you a solid understanding. but in simple words I can give example for intuition (won't be accurate): suppose I have 500 rows of data. I will first take 100 of my data. Then I will analyse my bias and variance. Next I will take 100+100=200 data from those 500. The again I will record my bias and variance. I will keep doing this by taking 300, 400 and 500 data. If I observe that my bias and variance is improving (if model is improving) it means adding more data might actually help. This is a very simplified example. But hope gives an intuition.


Number of Fully connected layers in standard CNNs I have a question targeting some basics of CNN. I came across various CNN networks like AlexNet, GoogLeNet and LeNet.
I read at a lot of places that AlexNet has 3 Fully Connected layers with 4096, 4096, 1000 layers each. The layer containing 1000 nodes is the classification layer and each neuron represents the each class.
Now I came across GoogLeNet. I read about its architecture here.
It says that GoogLeNet has 0 FC layers. However, you do need the 1000 node layer in the end with Softmax activation for the classification task. So the final layer isn't treated as FC in this?
Also then, what is the number of FC layers in LeNet-5?
A bit confused. Any help or leads would be greatly appreciated.

I think the confusion with the Inception module is the somewhat complicated structure. The point on the relevant CS231n slide (#37), saying there are no FC layers is partially correct. (Remember this is only a summary of the model to get the main points across!).  In the actual part of the model being explained on that slide, they are referring only to the Inception modules:

No FC layers!

Definitions will, however, play a big role in deciding whether or not there are FC layers in the model.

In the bigger scheme of things (beyond a single Inception module), we have first to distinguish between the train and test time architectures.

At train time there are auxilliary branches, which do indeed have a few fully connected layers. These are used to force intermediate layers (or inception modules) to be more aggressive in their quest for a final answer, or in the words of the authors, to be more discriminate.

From the paper (page 6 [Szegedy et al., 2014]):

  One interesting insight is that the strong performance
  of relatively shallower networks on this task suggests that the features produced by the layers in the
  middle of the network should be very discriminative. By adding auxiliary classifiers connected to
  these intermediate layers, we would expect to encourage discrimination in the lower stages in the
  classifier, increase the gradient signal that gets propagated back, and provide additional regularization.

The slice of the model shown below displays one of the auxilliary classifiers (branches) on the right of the inception module:

This branch clearly has a few FC layers, the first of which is likely followed by a non-linearity such as a ReLU or tanh. The second one simply squishes the 1000 input weights into whatever number of classes are to be predicted (coincidentally or not, this is a 1000 here for ImageNet).

However, at test time, these branches are not active. They were used simply to train the weights of the modules, but do not contribute to the final classification probabilities produces at the end of the entire model architecture.

This all leaves us with just the suspsicious looking block right at the end of the model:

There is clearly a big blue FC layer there!

This is where definitions come into play. It is somewhat subjective.
Is a fully connected layer one in which each $m$ weight is connected to each of $n$ nodes? Is it a layer in which representation are learned, and if so, does the layer require a non-linearity?  We know that neural networks requires the non-linearities, such as ReLU and tanh functions to be applied to the outputs of a layer (thinking in forward flow). Without these, neural networks would simply be a combinations of linear functions, and so going deeper wouldn't theoretically add any power as we essentially would just be performing a huge linear regression.

In this spirit, we can look at the final piece of the puzzle, and point out that tis final FC layer is noted to simply be linear! That is, it takes all the weights resulting from the preceding Average Pooling layer, and combines them into a linear combination of only 1000 values - ready for the softmax. This can all be understood from the tabular overview of the network architecture:

So, do you agree with the stanford guys or not? I do!


Neural Networks for Predictive typing I don't have a background in neural networks. But, various studies has been proved that neural networks (feed forward / Recurrent) outperformed n-gram language modeling for predicting words in a sequence. But, in an application to text messaging or any text-based conversation, where the language which is most likely used will be more informal or colloquial. Can still a neural networks perform well than n-gram LM? Considering the data to be fed are the text messages (colloquial phrases). If so, please enlighten me, thanks.

A neural network is in principle a good choice when you have A LOT of similar data and classification tasks. Predicting the next character (or word... which is just multiple characters) is such a szenario. I don't think it really matters which kind of language you have, as long as you have enough training data of the same kind.

See The Unreasonable Effectiveness of Recurrent Neural Networks for a nice article where a recurrent neural network (RNN) was used as a character predictor to write complete texts. They also have code on github.com/karpathy/char-rnn ready to train / go. You can feed it with a start string and ask for the next characters / words.


CNN not learning properly Before marking my question as duplicate, I would like to say that I have tried all the possible solutions mentioned in similar questions, but that doesn't seem to work.

I am currently working on blood cells classification problem where we basically have to classify blood images (4 classes). The dataset consists of 9957 images, nearly equal number of images of all the 4 classes. The accuracy always hovers around 25-27%  even after trying different optimizers and learning rates. I even tried training upto 100 epochs. Image augmentation doesn't help. Also, it is not that it is predicting same class for all images although for 1 particular batch of images, it predicts the same class. It again predicts some other class for all images in the next batch. So, I would just like to know, what am I possibly doing wrong? Is the dataset not sufficient, or the architecture should have more hidden layers, or am I not implementing optimizer or loss function correctly or is there any silly mistake I am overlooking in my code ?

My CNN architecture: (fs means filter_size, nf means number of filters, s is no. strides)

Input(80,80,1)->Conv(fs = 3, nf = 80, s = [1,1,1,1])
Activation(LeakyReLU)->Conv(fs = 3,nf=64,s=[1,1,1,1])
Activation(LReLU)->Pool(ps = [1,2,2,1],s=[1,2,2,1]
Conv(fs = 3,nf = 64,s=[1,1,1,1])->Activation(LReLU)
Dropout(prob = 0.75)->Flatten
FullyConnected(output_features = 128)->Dropout(prob = 0.5)
FullyConnected(output_features = 4)

loss_value = tf.reduce_mean(loss_fn)
optimizer = tf.train.AdamOptimizer()

loss_min_fn = optimizer.minimize(loss = loss_value)
check_prediction = tf.equal(tf.argmax(y,axis=1),y_pred)
model_accuracy = tf.reduce_mean(tf.cast(check_prediction, tf.float32)

sess.run(loss_min_fn, feed_dict = {x:X_train_batch, y:y_train_batch})
train_accuracy = train_accuracy + sess.run(model_accuracy, feed_dict={x : X_train_batch,y:y_train_batch})
train_loss = train_loss + sess.run(loss_value, feed_dict={x : X_train_batch,y:y_train_batch})

The images kind of look like this

I'll make a few observations that will hopefully help.

I would remove the dropout layer until you have evidence of overfitting on the data.  A dropout layer is generally used to make a model more generalized which is not your problem in this case and it may be hurting your model.
The images may not contain enough information to distinguish between the classes.  Can an expert assign a class label by looking at the images? If not, it may be that it is just not possible from the images.
Your accuracy scores imply that the model is just guessing since a random chance classifier would be expected to perform as well. However, it is curious that you seem to get reasonable accuracy on one class.  You may find that you need multiple models configured as one-vs-the-rest binary classifiers. Each model can be uniquely configured and trained to specialize on one class.  After the training is complete, which could be time consuming, prediction should be efficient.
Lastly, do you think that the model has stopped learning by the time you stop training?  Intuitively, I would anticipate the need for tens of thousands of epochs, and potentially millions, before seeing a really refined model.  100 epochs seems like it would be just starting to learn. Because of this cost, I would recommend you configure your model so you can continue training from a stopping point.

And by all means, keep trying different configurations of your layers.  One thing to remember though is that each time you use a convolutional layer you are losing information.  You need to consider how much information you are losing with each convolutional layer.


How to fetch text from pdf to further proceed with question answer based model from the same document? To illustrate the above title.

Suppose you have a pdf document, which is basically scanned from hardcopy, now there are set of fixed questions to answer from the document itself.
For an example a document contains a contract of land, now the set of fixed questions be "who is the seller?" "what is price of the asset? ", document has referred to this answers probably 2-3 times, as a human it's a simple task.

How to automate this?

You can use pypdf2 to extract text from pdf.

import PyPDF2

with open('sample.pdf','rb') as pdf_file, open('sample_output.txt', 'w') as text_file:
    read_pdf = PyPDF2.PdfFileReader(pdf_file)
    number_of_pages = read_pdf.getNumPages()
    for page_number in range(number_of_pages):   # use xrange in Py2
        page = read_pdf.getPage(page_number)
        print('Page No - ' + str(1 + read_pdf.getPageNumber(page)))
        page_content = page.extractText()
        text_file.write(page_content)


How to choose the best hyper-parameter when it is directly influenced by the random_state? While trying to evaluate my Ridge Regression model and using GridSearchCV to find the best parameter. I noticed that the best estimator changes every time I change the random_state in my KFold object (cv parameter). With this in mind how do I choose the most optimal hyper parameter to implement my model.

I am assuming, you are taking about the random_state of the Model, not the GridSearch (as it doesn't have a random_state) RandomizedSearchCV do need one.

A random state defines a starting point for an underlying random process.

With an ideal model, the difference should be very small. Though it will be.
If it is high, it means there is some Outlier data point/pattern due to which certain value is favoured.

Ignore - if small
 If it is large for a specific value
- Check the data
 - Increase K for the CrossVal

From another Stackexchange Question

  random state values which performed well in the validation set do not correspond to those which would perform well in a new, unseen test set.

From MachineLearning Mastery

  We can increase k and build even more models, as long as the data within each fold remains representative of the problem

Links
Stochastic Process
Randomness in ML
Is-random-state-a-parameter-to-tune