Training Deep Nets on an Ordinary Laptop Would it be possible for a an amateur who is interested in getting  some "hands-on" experience in desining and training deep neural networks, to use an ordinary laptop for that purpose (no GPU), or is it hopeless to get good results in reasonable time without a powerful computer/cluster/GPU?

To be more specific, the laptop's CPU is an Intel Core i7 5500U fith generation, with 8GB RAM.

Now, since I haven't specified what problems I would like to work on, I'll frame my questions in a different way: which deep architectures would you recommend that I try to implement with my hardware, such that the following goal is achieved: Acquiring intuition and knowledge about how and when to use techniques that were introduced in the past 10 years and were essential to the uprising of deep nets (such as understanding of initialisations, drop-out, rmsprop, just to name a few).

I read about these techniques, but of course without trying them out myself I wouldn't know exactly how and when to implement these in an effective way. On the other hand, I'm afraid that if I try using a PC which isn't strong enough, then my own learning rate will be so slow that it would be meaningless to say that I've acquired any better understanding. And if I try using these techniques on shallow nets, maybe I wouldn't be building the right intuition.

I imagine the process of (my) learning as follows: I implement a neural net, let it practice for up to several hours, see what I've got, and repeat the process. If I do this once or twice a day, I would be happy if after, say, 6 months I will have gained practical knowledge which is comparable to what a professional in the field should know.

Yes, a laptop will work just fine for getting acquainted with some deep learning projects:

You can pick a smallish deep learning problem and gain some tractable insight using a laptop so give it a try.

The Theano project has a set of tutorials on digit recognition that I've played with and moded on a laptop.

Tensorflow also has a set of tutorials.

I let some of the longer runs go overnight, but nothing was intractable.

You might also consider availing yourself of AWS or one of the other cloud services.  For 20-30 dollars you can perform some of the bigger calculations in the cloud on some sort of elastic computing node. The secondary advantage is that you can also list AWS or other cloud services as skill on your resume also :-)

Hope this helps!


Approach to labelling new data for training Overview:
Imagine an application that identifies cats and dogs from their phone camera. User's take a photo of their pet and it tells them if it is a dog or a cat. The data is then sent to the server.
I don't want to go through all the new cat and dog photos labelling whether each one is a cat or dog.
So how should I split the new data into categories to be trained on. The neural network may have identified inaccurately. What is a sure fire approach to labelling new data to be trained?

Imagine the user doesn't know whether their pet is a cat or a dog!

I don't want to go through all the new cat and dog photos labelling whether each one is a cat or dog.
This is basically the reason you do ML. Your model predicts category of new cats and dogs from a set of labeled images your model has seen before.
If you mean the distribution of new input might change over time e.g. new pose or style of photos that you did not have in your previous training set start to show up more, then there are options:

First and Best: Manually label a training-set out of them and retrain the model!
Weakly Supervised Learning: Label at least some from each class beforehand (Semi-supervised Learning) or in a querying schema (Active Learning)
Last and Least: Try clustering and if your lucky and data in each class has a friendly (!!!) distribution, you can tag your data using majority of labels in that cluster (theoretically you can see this as a KNN problem as well)

I tried to throw keywords so you can go on reading yourself. Hope it helps.


Natural Language Processing (NLP) in .net environment I'm developing a project on .net (using web API and MVC). now I have to add another module which users can make queries of Data we have in the DB and retrieve the relevant answer.

Is their a easy way to Do NLP developments in .net
If yes, It would be a great help if someone can give a brief idea on how to get started

any suggestion is greatly appreciated. Thank you.

There are plenty of NLP libraries for .Net if you Google that. Please be more specific as to what you want to accomplish when they query the DB.


Please explain CNN architecture I have this below architecture I would like to understand below question
This is my code and my image size is 50X50

 model.add(Conv2D(16, (3,3), input_shape=(image_x, image_y, 1), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Conv2D(32, (3,3), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Conv2D(64, (3,3), activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
        model.add(Flatten())
        model.add(Dense(128, activation='relu'))
        model.add(Dropout(0.2))
        model.add(Dense(num_of_classes, activation='softmax'))`
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

My question are :
1]  dense_1 :  Flatten has 1600 input and out will be 128 so how it will select 128 out put from 1600 input?
2] In which layer lost function and optimizor will be apply.

You don't select 128 from 1600. Dense layer performs a matrix-vector multiplication. First, flatten layer basically flat the output of your max_pool layer(5*5*64) to a vector (1600*1) (note that 5*5*64 =1600). It is just a re-arrangment of the element. We flatten max-pool layer because we want to use the output of max-pool layer as the input of dense layer (dense layer is also called fully connected layer or multiple layer perceptron). Dense layer is nothing but a vector-matrix multiplication. Say you have an input vector(I) whose shape is 1600*1, and you want your output (O) shape to be 128*1. Then your dense layer will perform the following operation: O=WI, where W is a 128*1600 weight matrix. That is how you got 128 output.The dimension of the W matrix is specified by your input shape and output shape and thus, after you specify the input/output shape, the W matrix is implicitly auto-generated by tensorflow. Btw, your "relu" function in the denselayer block, which is called non-activation function, performs relu function calculate on each element of your output vector (128*1) so it doesn't affect your final output shape.
Normally, all layers will be optimized. The loss function defines a loss between your true label and the output of your neural network (which is the last dense layer in your case). The gradient of the loss back propagates to each layer through the back propagation algorithm(you can check it online, but basically it is just chain rule). Thus, each layer has its own error gradients. We then use these error gradients to update coefficients of each layer to make the overall loss smaller (the loss defined by your loss function). The operation is called gradient descent algorithm. Since each layer updates itself separately, you can always freeze some layers so that these coefficients don't update, which can be done in tensorflow.


A question on realizable sample complexity I came across the following exercise, and I just can't seem to crack it:

Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. show that:
$m^{\text{stat, r}}_H (\epsilon) = O\left(m^{\text{stat, r}}_H (\epsilon/2, 1/2)\cdot \log(1/\epsilon) + \frac{\log(1/\epsilon)}{\epsilon^2}\right)$

Where $m^{\text{stat, r}}_H (\epsilon)$ is the minimal number $m$ such that for any realizable distribution over training examples $D$ we have that:
$$\mathbb{E}_{S \sim D^m}\left[ l_D(A(S)) \right]\leq \epsilon$$
And where $m^{\text{stat, r}}_H (\epsilon, \delta)$ is the minimal number $m$ such that for any realizable distribution over training examples $D$ we have that:
$$P_{S \sim D^m}\left( l_D(A(S)) \geq \epsilon \right) \leq \delta$$
Thanks a lot in advance!

We want to prove:

If H is PAC learnable, then $\forall \epsilon, \exists C, \forall m \geq m_2:=Clog(1/\epsilon)(m_1+1/\epsilon^2), E[L] \leq \epsilon \mbox{ (a)}$
where $m_1:=m(\epsilon/2,1/2)$

Since $L \leq 1$, we have $E[L] \leq 1$. So proof is trivial for $\epsilon \geq 1$. Let $\epsilon \in (0, 1)$.

Find an equivalence for $(a)$:

$\begin{align*}
E[L] &= \int_{l \geq \epsilon / 2}ldP + \int_{l< \epsilon / 2} ldP \leq \int_{l \geq \epsilon / 2}dP + \int_{l< \epsilon / 2} \frac{\epsilon}{2} dP\\
&= P(L \geq \epsilon/2) + \frac{\epsilon}{2} P(L <\epsilon/2)\\
&= (1 - \epsilon/2)P(L \geq \epsilon/2) + \epsilon/2 < \epsilon
\Leftrightarrow P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)
\end{align*}$

Therefore, if

$\forall \epsilon, \forall m \geq m_3:=m(\epsilon/2, \epsilon/(2 - \epsilon)), P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\mbox{ (b)}$ holds,

$\forall \epsilon, \forall m \geq m(\epsilon)=m_3, E[L] \leq \epsilon \mbox{ (c)}$ holds too (and vice versa)

Prove $(b) \Rightarrow (a)$:

Using The Fundamental Theorem of Statistical Learning for PAC learnable H with VC dimension $d$, we have:

$\begin{align*}
(&\epsilon/2, 1/2)\mbox{-learnable H with } m_1  \Leftrightarrow \exists C_1 > 0, m_1 \geq C_1\frac{d + log(2)}{\epsilon/2}\\
&\Leftrightarrow  log(1/\epsilon)(m_1 + 1/\epsilon^2) \geq \frac{log(1/\epsilon)(C_1d + C_1log(2) + 1/(2\epsilon))}{\epsilon/2}
\end{align*}$

which uses $1/\epsilon > 1$ and $log(1/\epsilon) > 0$.

Now we use an inequality without proof (plot the function here)

$\forall x > 1, \forall d, C_1 \geq 0, \exists C_2 > 0, f(x)=\frac{log(x)(C_1d+C_1log(2)+x/2)}{dlog(2x)+log(2x-1)} \geq C_2$

Setting $x=1/\epsilon$, we continue as:

$\begin{align*}
&... \overset{\exists C_2}{\geq} C_2\frac{dlog(2/\epsilon) + log((2-\epsilon)/\epsilon)}{\epsilon/2} \overset{\exists C_3}{\geq} \frac{1}{C_3} m_3
\Leftrightarrow (\epsilon/2, \epsilon/(2-\epsilon))\mbox{-learnable H with }m_3
\end{align*}$

By setting  $m_2 := C_3log(1/\epsilon)(m_1 + 1/\epsilon^2)$, we have $m_2 \geq m_3$, thus

$\begin{align*}
&\forall \epsilon, \forall m \geq m_3, P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\\
&\Rightarrow \forall \epsilon, \exists C, \forall m \geq m_2 := Clog(1/\epsilon)(m_1 + 1/\epsilon^2) \geq m_3, P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\\
&\Rightarrow \forall \epsilon, \exists C, \forall m \geq m_2 := Clog(1/\epsilon)(m_1 + 1/\epsilon^2), E[L] < \epsilon
\end{align*}$.

Proof is complete.


Age range prediction according to browsing history I need to predict age or age range of a user according to his/her browsing history from a news website. The only data I am able to get is USER-ID, URLs-visited, TAGs from those URLs, "which category news a person reads most" etc. My only option seems like creating this dataset somehow. But I do not know how do I decide the age range of a person. The problem is if a person is reading a politics news he could belong to an age range of 50 and above but then what if this is a high trending news and everybody is reading it. So basically how do I decide which age range people read which kind of news most?

I hope I am able to explain what my problem is. If somebody has worked on such a problem I would like to hear how you proceeded with it.
Thanks.

This problem is quite broad and I would like to add some of my suggestions. Refer to good source of Age wise browsing data, age-wise social behaviors and their preferences publicly available. Here are some datasource that you may refer and you will have to do extensive research on lot of factors to predict age based on browsing data.

Datasource 1: https://www.google.com/publicdata/directory

Datasource 2: https://www.ciser.cornell.edu/ASPs/datasource.asp

Refer to this paper : https://link.springer.com/chapter/10.1007/3-540-44934-5_3

I would definitely say that this problem doesn't seem to have any perfect solution, only thing we may do is to try our level best by various sources and implementations. Learning algorithms on public datasets on behavior, survey,  social and mental age specific activities etc can help as well. I have also attached a link to a paper on similar problem. I hope this helps to some extent.


What is the point of getting rid of overfitting? I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too sparse then your model may start to memorize data and not perform well on new data. However, are there any cases in which adding dropout, regularization, etc would increase accuracy on the validation set? For instance, if my training acc is 95% and val accuracy is 70%, would removing overfitting simply bring the training accuracy down lower to the val accuracy? Or is there a way to actually improve training accuracy? I assume there is but some intuition on this would be very much appreciated!

Basically, prediction error can be decomposed into three terms, two of which that you can control. MSE (mean square error) = Variance + Bias^2 + irreducible error. A model with high variance implies that small changes in the training dataset yield large changes in the fit of the model (in this context). A model with low bias essentially fits the training dataset extremely well. Irreducible error is the difficulty inherent to the problem itself (and therefore, not controllable) so lets ignore that.

In an ideal world, we would obviously want low variance AND low bias. However, this in reality is extremely difficult to do because intuitively, the two forces of variance and bias work against each other. The reason why is that I can arbitrarily make my model fit the data extremely well (and therefore, have low bias) by simply increasing the complexity of the model, modelling every quirk and essentially do what you refer to as "memorizing the dataset". You see, data are realizations of random variables that can potentially take on many different outcomes and therefore, there is noise in the dataset. Hence, modelling the exact pattern/trend in the dataset is not going to generalize well because the data itself is not a perfect representation of the population (which is our goal) but a sample of it.

By increasing the complexity of my model to arbitrarily improve the fit of the model on the dataset, I have effectively traded away all of my bias for large amounts of variance. I did this because clearly, by increasing the complexity of my model to the point where my model has even memorized the noise in the dataset, a relatively small change in the dataset with different observations (another random sample from the same population) will change the fitted model drastically. This is because noise is inherently random, not predictable and unique to that specific dataset (sample) only, not the general population from which the data comes from. As a consequence of my model changing a lot with my dataset that I trained it with (what many refer to as an unstable model), my models predictions will inherently be very random as well and therefore, my models performance on new held out data will be no better than say, a random guesser. This is what people mean when they say their model is "overfit". I've fit the training data set perfectly, at the cost of not modelling the actual signal from the data but a bunch of noise unique to the training set only.

So what people have figured out is that we need to make a tradeoff between fitting the training data extremely well whilst also being mindful of not modelling noise. Thus, stuff like dropout layers in NN, L1/L2 regularization, pruning in trees, learning rates, early stopping, and cost complexity penalties all work in purposely sacking some bias (fitting the training set not as well) in hopes of reducing variance (having a stable model and therefore, stable non random predictions). Striking the balance between the two is in essence what predictive modelling is about and all ML methods try to address this problem in their own way.

In your example, removing overfitting would imply moving the validation accuracy closer to the training accuracy, not the other way around. But of course, this is typically a pipe dream and you will almost always overfit somewhat. The degree to what you find acceptable, given the problem at hand, is key. If I have a training accuracy of 0.99 and a validation accuracy of 0.89 in many contexts I would still be satisfied with the model, for example.


Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group? In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that enables to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

In Hive itself? Unfortunately, the answer is simply no -- as the language definition manual shows, that statistic is simply not built in. In addition to the language manual, you can get more information on statistics in development in Hive here and here.

Having said that, there are plenty of ways to calculate Kendall's W on data that's in Hive.

You could write out the data to a file or query it into R or a statistical package such as  SAS, Stat, MATLAB, Excel, etc then run your calculation and, if necessary, write your results back to Hive.

In R, for instance, you could do something like this:

install.packages("RODBC")
require(RODBC)
db   <- odbcConnect("Hive_DB")
hql  <- "select * from table A"
data <- sqlQuery(db , hql)
kenw <- cor(x = data$a, y = data$b, method="kendall")
sqlSave(db, kenw, tablename = "new_table_of_kendall_coef")

or (if using Linux or Unix) then you could use RHive without needing to use an ODBC name.

Another way to go about it would be to take the functions that do exist in Hive (which you linked to) and calculate Kendall's coefficient yourself with a custom function. As to how to specifically implement that, well you'd probably want to post on Cross Validated (stats.stackexchange.com).


Backpropagation with log likelihood cost function and softmax activation In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$
Suppose we have 10 output neurons, when back propagating the error, only the gradient w.r.t. $y^{th}$ output neuron is non-zero and all others are zero. Is that right?

If so, how is the below equation (81) true?
$$\frac{\partial C}{\partial b_j^L} = a_j^L - y_j $$
I'm getting the expression as
$$\frac{\partial C}{\partial b_j^L} = y_j (a_j^L - 1) $$

No you actually didn't really understand how softmax functions it outputs a probability distribution hence if there are 10 output neurons you will have 10 probabilities for the 10 respective classes i.e. the neuron with the highest probability will be more activated that is none of the output neurons will give 0 as output that is what softmax is it takes exponential average of every class to produce a probability distribution over k different classes here k=10.Now as you said suppose you have 10 output neurons then while back propagating the error, only the gradient w.r.t. yth output neuron is non-zero and all others are zero,this is wrong if you go and give it a read the error or cost function is calculated as follows when there are multiple neurons:

Now as you can see the cost is calculated over all the output neurons for all the training examples in the batch that is of size 'n' here hence when you do backpropagation that is when you calculate"dc/dw" and "dc/db" it includes output from all the output neurons and your statement is wrong because if gradients of other output neurons are 0 then how will your backpropagation update there weight matrices. I know its confusing but if you read his ch-2 you should be able to understand it.i have used cross entropy to explain you this but the same method will work for any cost function that you take .

As far as the derivation is considered it is quite easy just go back and study his chapter two all four equation BP1,BP2,BP3,BP4 and understand there derivation it will take some time but it is easy once you understand the composite function nature of neural networks and how to differentiate composite functions using the chain rule.


Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold? If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be distributed across the cluster? Or do i have to use pyspark dataframe operations and pyspark ml packages for operations to be distributed?

Short answer: NO.

The moment you convert the spark dataframe into a pandas dataframe, all of the subsequent  operations (pandas, ml etc.) will be run on a single-core as those algorithms and programs are written in native-python and doesn't support multi-core distributions. In a nutshell, someone has to rewrite the whole sklearn to again to be compatible with spark.

Said that there are some progress to bring e.g. to distribute scikit-learn on a Spark cluster. Some functionalities like the expensive GridSearch are rewritten in spark and you can use it together with native sklearn in databricks, see this post, or Joblib Apache Spark Backend (previously known as spark-sklearn). This blog post is also worth reading as well.


User profiling with Mahout from categorized user behavior I'm trying to cluster and classify users with Mahout. At the moment I am at the planning phase, my mind is completely mixed with ideas, and since I'm relatively new to the area I'm stuck at the data formatting.

Let's say we have two data table (big enough). In the first table there are users and their actions. Every user has at least one action and they can have too many actions, too. About 10000 different user_actions and millions of records are in the table.

user        - user_action
u1          - a
u2          - b
u3          - a
u1          - c
u2          - c
u2          - c
u1          - b
u4          - f
u4          - e
u1          - e
u1          - d
u5          - d

In the other table, there're action categories.  Every action may have none or multiple categories. There are 60 categories.

user_action - category
a           - cat1
b           - cat2
c           - cat1
d           - NULL
e           - cat1, cat3
f           - cat4

I'm going to try to build a user classification model with Mahout but I've no idea what I should do. What type of user vectors should I create? Or do I really need user vectors?

I think I need to create something like;

u1 (a, c, b, e, d)
u2 (b, c, c)
u3 (a)
u4 (f, e)
u5 ()

Problem in here, some users performed more than 100000 actions (some of them are same actions)

So; this is more useful, I think;

u1 (cat1, cat1, cat2, cat1, cat3)
u2 (cat2, cat1, cat1)
u3 (cat1)
u4 (cat4, cat1, cat3)
u5 ()

The things I also worry about are

How should I weight categories for users? For example u1 has at least three action that related with cat1, while u3 has only 1. These one should be different?
How can I decrease the difference between active users and passive ones? Like u1 has too many actions and so categories, u3 has only 1.

Any guidance are welcome.

I think one solution for this might be using Matrix factorization, to perform some kind of collaborative Filtering. This way you will have not have to deal with the two concern points that you have, explicitly.

To do this, create a [User X Action] Matrix W, where rows are users and columns are actions. W(i,j) i.e. the Matrix entry on i row and j column would be the number of times the user i has performed action j. The matrix entries where no data is given are treated as missing data values by most collaborative filtering algorithms.

When you factorize this Matrix, you factorize it into two parts W= A*B. The A matrix will have rows are users and columns as latent dimensions. From this A, you get a feature vector for each User. You can then run clustering on these feature vectors to cluster the users. For more intuition about the factorization approach, look up "Matrix Factorization for collaborative Filtering" on google. Mahout has matrix factorization implementations that you can use.

Using the action categories is another matter altogether. It depends on what a category means. Does it mean that every action inside a category is very similar to each other or that they just have one or two aspect of similarity. A very easy approach would be to factorize a [User X (Action + Category)] Matrix. This can be solved using the same implementation mentioned earlier. You can also have more fancy schemes, like fixing part of the Matrix B to be very similar to the [Action X Category] matrix. But these schemes will require you to write optimization code yourself.


How to generate list with out for loop in python data frame

S/N Type    Number Capacity
1   Bike      2      5
2   Tempo     1      30
3   Truck-1   1      60
4   Truck-2   1      90

I would like to generate capacitylist = [5,5,30,60,90]

Is it possible to do it with out for and using map function in python. Thanks Alot.

ndf = df.loc[ np.repeat(df.index.values, df['Number']) ]
values = ndf['Capacity'].tolist()
# [5, 5, 30, 60, 90]

Repeat the rows you wish you duplicate using np.repeat, and then, takes the selected column (here, capacity) and turn it into a list using .tolist().


Learning character embeddings with GenSim I am learning deep learning, and as a first exercise to myself I am trying to build a system that learns a very simple task - capitalize the first letter of each word.  As a first step, I am tried to create "character embeddings" - a vector for each character. I am using the following code:

import gensim
model = gensim.models.Word2Vec(sentences)

where sentences is a list of lists of chars which I took from this long Wikipedia page. For example, sentences[101] is:

[' ', ' ', ' ', ' ', 'S', 'p', 'e', 'a', 'k', 'i', 'n', 'g', ' ', 'a', 't', ' ', 't', 'h', 'e', ' ', 'c', 'o', 'n', 'c', 'l', 'u', 's', 'i', 'o', 'n', ' ', 'o', 'f', ' ', 'a', ' ', 'm', 'i', 's', 's', 'i', 'l', 'e', ' ', 'e', 'x', 'e', 'r', 'c', 'i', 's', 'e', ... ]

To test the model, I did:

model.most_similar(positive=['A', 'b'], negative=['a'], topn=3)

I hoped to get 'B' at the top, since 'A'-'a'+'b'='B', but I got:

[('D', 0.5388374328613281),
 ('N', 0.5219535827636719),
 ('V', 0.5081528425216675)]

(also, my capitalization application did not work so well, but this is probably because of the embeddings).

What should I do to get embeddings that identify capitalization?

I believe that you misunderstood the word2vec concept. Basically for words, the feature vector for a word is learnt from the surrounding words.

You shall know a word by the company it keeps- Firth.J.R

In your case characters have been used, so the feature vector for each character depends upon the adjacent characters present. your example might work, if you have the following training sentences.

ABCDEFGHI
abcdefghi
AbcDEFghi
aBcdEfgHI
abcDEFgHi
ABcdEFGHi

With these training sentences, the characters 'A','a','B','b' will preserve capital features and english alphabet order feature. But whereas, when trained with wikipedia sentences, the characters will preserve the probability of being present in a meaningful word. For instance, the closest letter to 'C' would be 'a','o','e' but hardly 'x' or 'd' because there would be words like 'covenant','country','cat' as no common words would be 'cx..'


is validation and train set should be all different files? Let say I have train set and validation set
if 'A' included in the train set. 'A' should not be include in the validation set? or some is ok?

Validation set is basically to understand how your model is behaving in terms of over-fitting, and under-fitting, and also to find the best set of hyper-parameters for your algorithm. If you use some parts of the same training data on your validation set, then this hypothesis would not be justified.
Hence, it is suggested to split your data set into train/validation/test set without any overlapping of the data.


How to manage memory constraint and increase speed for 1 vs rest image similarity comparison for over 100k images for computer vision? I'm looking for ideas on how to do things in a better way, efficiently when using Machine/Deep Learning.
I am working on a search improvement problem using Computer vision where I am thinking about comparing two images and see their similarity score so theoretically there are lots of things I can try such as: Structural Similarity, Template Matching, RMSE, PSNR etc... On the second thought, I can use Siamese Network. So in both cases, I have to search every image with every other image and get top-k scores which shows that k most similar images.
PROBLEM: I have a huge and by huge I mean around 20 Million+ images data. So in theory, let's suppose even if I use 100K images, it'll take huge amount of memory for nXm dimensional image matrix and on top of that, calculating scores with every other image for each image can't work on incoming live data as it'll take too much time. Impossible for Siamese Network as we have to insert all the images one by one.
Please tell me if there is a way to achieve this? I am using MongoDB. How can this be achieved? Someone said that ElasticSearch has a functionality to store image vectors. I have no clue what Elastic is and even if I read it, I don't know how to use that specially with Python?
Can someone suggest how it is achieved in IT industries and big companies because they have huge data and there must be a simpler way to do this.

One option is locality-sensitive hashing (LSH) which hashes similar input items into the same "buckets" with high probability. LSH is frequently used for near-duplicate image detection, a variation of your problem.


Schema matching using machine learning I'm facing the following problem of integrating data from another company  (data base) to an internal one. It's about personal core data, i.e. name address etc. I would like to come up with a mapping of these keys in an automated way. I've read about HMM to use for this. However, I'm still gathering some information of some feasible and standard ways to do it. I'm looking therefore for some reference which describes a possible solution to this.

Michael Stonebraker started a company that claims to do just that, schema matching using machine learning:

https://www.tamr.com/

Their site no longer has many details on their approach but this article talks about some of the techniques they used, like :

  Perform fuzzy string comparisons over attribute names using trigram cosine    similarity.
  Treats a column of data as a document and tokenize its    values with a standard full text parser. Then, measure TF-IDF cosine
  similarity between columns. This method is suitable for text fields.
  Use minimum description length (MDL) to compare values of two    attributes.
  Compute the ratio of the size of the intersection of two    columns' data to the size of their union. This method is well suited
  for categorical fields with small number of values.
  Compute Welch's t-test for a pair of columns that contain numeric values and get the probability the columns were drawn from the same
  distribution.


Can all known ML algorithms be written as a sequence of matrix operations? I keep hearing that machine learning is just linear algebra.

Does that mean all known (and all possible?) ML algos, from random forest, to support-vector machines, to recursive neural networks, can be written as a sequence of basic matrix and vector operations, such as multiply, add, sum, etc?

Would writing the algorithm in this way be able to capture all aspects of the model, e.g. architecture, activation functions, learning rate, optimization method?

Can advanced features like convolution be expressed as basic matrix operations?

Is there any ML algorithm for which this isn't true?

Is matrix algebra Turing complete?

Linear algebra is very important for ML, but ML is not "just" linear algebra.

Any sequence of linear matrix-vector operations is linear.
Therefore any non-linear operation cannot be computed using linear algebra.

A few examples of non-linear operations that are used in ML:

The if statement: e.g. (x < 0)
The softmax activation
The rectified linear unit

  Can advanced features like convolution be expressed as basic matrix
  operations?

This statement is true actually. A single convolution can be expressed as matrix. However, any modern CNN has non-linearities following the convolution.


Add Custom Labels to NLTK Information Extractor I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to construct such a system. However, it is not clear how one would go about adding custom labels (e.g. actor, director, movie title).

Similarly, Chapter 7 of the NLTK Book discusses information extraction using a named entity recognizer, but it glosses over labeling details.

So, I have two questions:

How would I add custom labels?
If I have bare lists of relevant named entities (e.g. movies, actors, etc.), how can I include them as features? It appears that I would need to use IOB format, but I am unsure about how to do this when I only have lists of named entities.

Once you have your own lists of named entities, and you're only interested in extracting the relations, I believe there are simpler solutions (although I never tried relation extraction in NLTK, so I might be wrong):

ReVerb - a tool written in Java. Once it produces the results, you can simply keep the rows, where your labels are present as objects of the relation.
OpenIE - the successor of ReVerb (also written in Java). The authors claim better perfomance, and the output might be more informative.
IEPY - a relation extraction tool in Python. You should be able to provide your own labels/named entities using gazetees.
MITIE - this library has bindings in Python, and it offers relation extraction functionality.


How does BERT and GPT-2 encoding deal with token such as , As I understand, GPT-2 and BERT are using Byte-Pair Encoding which is a subword encoding. Since lots of start/end token is used such as <|startoftext|> and , as I image the encoder should encode the token as one single piece.

However, when I use pytorch BertTokenizer it seems the encoder also separate token into pieces. Is this correct behaviour?

from pytorch_pretrained_bert import BertTokenizer, cached_path
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
tokenizer.tokenize('<s> This is a sentence <|endoftext|>')

The results are:

['<',
 's',
 '>',
 'This',
 'is',
 'a',
 'sentence',
 '<',
 '|',
 'end',
 '##oft',
 '##ex',
 '##t',
 '|',
 '>']

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.

In the case of GPT-2, OpenAI trained it only with <|endoftext|>, but it has to be added after the tokenization. Some people mistakenly add it before tokenization, leading to problems.
<|startoftext|> is specific to the library gpt-2-simple.


Does Pooling remove spatial information of image in CNN? Pr. Geoffrey Hinton has pointed out that pooling-layers remove spatial feature information. But, essentially, does the process that last convolutional layer's features are flattened for FC layer makes the spatial information removed?

Your question text is not very clear but I try to give you what you need. Max-pooling layers give the CNNs spatial invariance ability through the layers due to the fact that they check the existence of something or not. If you stack them through multiple layers, your network will have a rough ability to be spatial invariance but this is not that much. Moreover, professor Hinton has stated somewhere that using pooling layers is a mistake and it's a disaster that they work; I didn't quote the exact words. If you want a network to be spatial invariance you should use spatial transformers which are differentiable modules and can be used in CNNs without any supervision. Take a look at here.

Finally, to answer your question, pooling layers somehow remove spatial information that is why capsnets are introduced.

About the fully connected layers, There are two main points. First the inputs, second, the outputs. In convolutional layers, the inputs of each layer is limited to the region which its is employed and for that input there is a single output. Consequently for a specific region, there is a single output which is responsible for that point. In MLPs, all the inputs go to a single neuron and for all the inputs there is just a single value. I guess this is why they don't keep the spatial information and are just used for classification tasks. Actually, they just try to classify the extracted features by the CNNs.


Sklearn LocalOutlierFactor contamination parameter Can anyone provide an intuitive explanation of the choice of contamination parameter used in sklearn's LocalOutlierFactor implementation when contamination="auto" ?

The sklearn guide suggests "as described in the paper" but I couldn't find anything obvious. Thanks.

You are specifying with a floating point number what proportion of the data you are fitting on is an outlier. If you use 'Auto' it will default to 0.1. Note that in the current documentation, there is a changed note specifying that it will default to 0.2 in a future version.


Splitting binary classification into smaller susbsets As an example. If you are tying to classify humans from dogs. Is it possible to approach this problem by classifying different kinds of animals (birds, fish, reptiles, mammals, ...) or even smaller subsets (dogs, cats, whales, lions, ...)

Then when you try to classify a new data set, anything that did not fall into one of those classes can be considered a human.

If this is possible, are there any benefits into breaking a binary class problem into several classes (or perhaps labels)?

Benefits I am looking into are: accuracy/precision of the classifier, parallel learning.

If you try to get the best accuracy, etc... for a given question you should always learn on a training set that is labeled exactly according to your questions. You shouldn't expect to get better results if you are using more granular class labels. The classifier then would then try to pick up the differences in the classes and try to separate them apart. Since in practice your variables in the training set will not perfectly explain the more granular classification question you shouldn't expect to get a better answer for your less granular classification problem.

If you are not happy with the accuracy of your model you try the following instead:

review the explanatory variables. Think about what might influence the classification problem. Maybe there us a clever way to construct new variables (from your existing ones) that helps. It's nowpossible to give a general advise on that since you have to consider the properties of your classifier
if your class distribution is very skewed you might consider over/undersampling
you might run more different classifiers and then classify based on the majority vote. Note that you will most likely sacrifice explainability of your model.

Also you seem to have some missunderstanding, when you write 'you would assign it to human if it doesn't fall into any of the granular classes'. Note that you always try to pick class labels covering the whole universe (all possible classes). This can be always defined as the complement of the other classes. Also you will have to have instances for each class in your training set.


Good books on unsupervised learning I am looking for a good book about unsupervised learning that goes beyond the typical k-means and hierarchical clustering algorithms.

Practical implementations in R or Python will be a plus.

Currently I am working in retail, so the typical use cases I am interested are customer segmentation, products segmentation.

Any ideas?

There might not be an entire book about unsupervised techniques, given the greater effectiveness of supervised learning techniques. However, there are small books and selected chapters from longer books that could be relevant:

"Unsupervised Machine Learning in Python" which includes Gaussian Mixture Models
"Python Machine Learning" which has a section on Latent Dirichlet allocation (LDA)
"Modeling Techniques in Predictive Analytics with Python and R: A Guide to Data Science" which has a chapter on Market Basket Analysis
"R for Marketing Research and Analytics" which covers Confirmatory Factor Analysis and Structural Equation Modeling


Graph layout for a network of molecules I am looking for a suitable graph representation where the nodes/vertices are molecules with 2 main variables: structural characteristic and a real number property (for example biological potency). The network is build up by molecules by matching structural properties and it can be rather complex with many clusters of high degree vertices (ie molecules that are structurally similar to each other). However the representation should give a clear picture of potencies, although high detail accuracy is not required: some sort of classification is adequate. I played with JUNG with a graph having edge weights as the change in potency and the ISOM layout thinking that it reserves (as much as possible) the edge weights but with not good results. Something like radial layout could be ideal, as it gives a clear picture of potency distribution, but this obviously would suffer from crowding of vertices that are closer to the center.  As, obviously, my knowledge is restricted, is there some layout that you would suggest and is practical for a non expert to implement?

I personally use NetworkX and igraph where the first one is for programming in Python and the second one has interfaces for Python, R and C.

If you are not into programming much, Gephi is what you are looking for!


Learning Curves and interpretations I've trained 4 classifiers on an undersampled dataset.
I plotted the learning curve for each classifier and I got the following results :

I see that for the Log Reg, both curves seem to converge and that adding more data will not help at some point.
For the SVC I have no idea (rather than adding more data seems good ! )
for Knn : adding more data will increase both accuracy
for Random Forest : I have no idea.
I would love to understand how to read these curves. Thank you very much ! :)

In general, the further away the green line is from the red line, the more the model is overfitting, however eventually enough data will cure all overfitting (there will be so much data the model can't possibly memorize all of it), and that's why the lines converge to being together (stops memorizing, red line goes down, starts generalising, green line goes up). Some models need more data to learn than others however, and so as you can see, the LogisticRegression model reaches it's best performance much faster than, for example, the SVC.
An interesting case is the KNN, who's red line doesn't go down, but rather up. I'm pretty sure the reason for this is to do with how the KNN works, it compares instances it knows to classify new instances. Thus, the KNN doesn't really memorize... New instances it can compare with will never hinder it's performance on the training set (red line). However it too will also eventually converge, the lines together.


Julia - To Learn or Not to Learn? I'm relatively new to data science, having done some machine learning projects at work. My background is in R, although I have some experience in Python. I'm considering choosing one language and getting really good at using it for data science / machine learning.

Now, I've read various articles touting Julia's strengths - namely ease of use and massive speed benefits. It sounds almost too good to be true!

So my question is, is there any particular reason(s) why I should choose Julia over R/Python?

Or alternatively, is there any reason(s) why I shouldn't choose Julia over R/Python?

As interesting as these kind of questions are; it will always be subjective and task oriented. There are a multitude of comparisons between these languages: here, here, here and many viewpoints here.

I won't list the technical differences, but just offer my perspective on each language.

Julia

If you want to learn something new and upcoming, perhaps even influence the development in the long term, Julia might be a nice candidate, because (after ca. 10 years of development) it recently made its first 1.0 release and may gain maintstream use over the coming years. Starting now would potentially make you one of relatively few experts in a few years. It is geared towards scientific computing, combining the strengths of more established languages and I agree, some features do sound too good to be true!

R

If you want to have a very well-founded language with more focus on the intracacies of statistics, proven robustness over decades and a strong community, maybe R is for you. There are packages for almost anything related to modelling, forecasting, Bayesian methods, ... the list goes on. It can be fairly painless to learn and provides a short path to results, as libraries oftentimes have a single function that does lots of work for you. R also provides hands-down the nicest plotting experience, whose importance is not to be underestimated!

Python

If you want a "Swiss Army Knife", able to analyse data, build a deep learning model and then perhaps write a webserver to stream API data, from a robot, finally pushing live results to a dynamic website (and so on). There are libraries and tutorials available on seemingly endless topics. Syntax is generally quite friendly, so the path to being productive is just as short as the others. Python isn't as well established (historically) within statistics and machine learning compared to R, but I rarley find myself lacking anything. Python is growing in popularity too, with the majority of academic research I see within Data Science appearing e.g. on GitHub in Python. So it is a mature language and is also expanding.

I think other people could offer their perspectives to give a fuller comparison, but that's my two cents. I personally switched from R to Python for almost all tasks - and haven't yet found time to take Julia for a full test-drive (but want to!).

Coming back to my starting point: all the languages are Turing complete, meaning you can achieve anything with any of the languages... but why open a can of beans with a spoon if there is also can-opener available?

For most practical cases: if you are more comfortable with one language - use that one. If a task requires a certain package from one language, or your company requires the usage of one language - your decision is made for you.

Having motivation to learn a language and get some actual work done should, in my opinion, be the main factor. If you try all three, it is likely one will feel more natural to your way of thinking.


Why predicted proababilities from this binary classifier does not sum up to 1? I have a C5.0 model that is trained to predict binary class (C1/C2) on a dataset with 20 features. The model is configured to perform boosting (10 trials) and it has a miss-classification cost function (100:1 where 100 is the cost for miss-classifying a Negative sample as Positive and 1 is the cost for miss-classifying Negative as Positive).

Looking at predicted probabilities generated by the model, I can see that it ranges from 0 to 1 for each class. i.e, I have instances where the predicted class (C1) has a probability lower than 0.5 ( for example: predicted class=C1 and predicted probability=0.1 ) - This is where the question arises: if P(C1) < 50% why is it classified as C1 (there are only two classes, C1 and C2)

Based on my understanding of decision trees, the predicted probabilities are often generated based on the percentage of test cases on the leaf node that were labeled in each class divided by the total number of instances hitting that leaf node. This method will dictate the probability for two classes must sum up to 1.

My question is why does the model classify an instance in class C1 if it has only got 0.1 confidence in it. Does a predicted probability of 0.1 on class C1 mean that there is a 0.9 confidence in it belonging to class C2? If so, why does it classify an instance in class C1 is it has less than 0.5 confidence in it?

My own theory is that this might be due to boosting and miss-classification cost and the way they are influencing the predicted class.

You are absolutely right to be skeptical about the results you are getting. I've been using boosted trees (not necessarily C5.0) for over 5 years, and I've never encountered results like what you've described here. Excluding the possibility that you simply had a typo in your code, I can think of two reasons:

Low decision threshold: As is often the case in class imbalance problems (e.g., fraud detection, marketing response, bankruptcy, etc) a low probability could still indicate the minority class if the decision threshold is also low. In your case, if the model might have automatically decided to pick a threshold of 0.05, then 0.1 would be flagged as C1.
Cost function: This is what you mentioned -- if your model has a high cost associated with missing true positives then 0.1 could be classified as C1.


A question on data distribution probability Let $D={D_k}_1^N$ denotes a set of i.i.d observations, and x denotes random variables, p() denotes probability, whether $p(D|x)$ equals to $p(D)$? Thank you for any help.

EDIT: To be more clear, I talk this problem in Bayesian inference context where we seek the posterior distribution $p(x|D)~p(x)p(D|x)$, so D is dependent on x.

You stated that D are i.i.d. so they are independent to each other.

The crucial question is whether D are all independent of x.

p(D|x) = p(D)

only if D are all independent of x


Plotting relationship between 2 data points where one data point is a boolean I am working with the titanic survivors data set.

I have the data as a DataFrame and I can create 1D visualizations such as histograms, and also see the correlations by calling data.corr().

I would like to create a scatter plot to represent the correlation between 'age' and 'survived'.  I can't figure out how to plot this data because 'survived' is effectively an integer of 0 or 1 (died or lived, respectively)

If I do something like:

titanic_data.plot(x='Age', y='Survived', style='o')

I get a plot that looks like this:

What I would like is a plot that somehow takes the average survival rate by age and created something more like this:


Dense layer: how to transform hgih dim vector to one dim? (sigmoid function) I have a dense layer who transforms a 100 dimensional vector to a 1 dimensional object using a sigmoid function. I understand how a sigmoid function transforms a number to a number between 0 and 1.

But I don't understand how it transforms the dimensionality of the vectors?

I suggest watching 3blue1brown's YouTube series on neural networks to get a better conceptual understanding of what's going on.

It's not that the sigmoid function itself transforms the dimension of a vector - the sigmoid function is just used as an activation function that affects the output of each individual node - it's the number of units/nodes you have in a layer that does what you're describing.

If you only have one unit in your output layer it will only output one number between 0 and 1. If you have 100 units in a dense layer it will output 100 numbers, each between 0 and 1.


Are there any learner-specific optimizers? In reading about machine learning (ML), and working through some basic examples, it appears to me most learning algorithms use generic optimizers. I am using the word "optimizer" to describe the technique the learner uses to minimize the loss function. Gradient decent, and it's variants, seems to be the most common. But the general idea in ML seems to be to continually iterate a learning algorithm, each time adjusting various things to try to improve the loss.

Gradient decent, and similar techniques, are generic in that they work on a wide variety of learning models. This makes intuitive sense, all you need is the predicted value and the known target value.

But, often the trade-off with generic techniques is that while they work everywhere, they are slow. In other words, in theory you could construct a learner-specific optimizer that exploits characteristics of the learning algorithm to avoid some of the work that something like SGD might have to do. So if you know you're going to use learning algorithm XYZ, then maybe you can speed up computation considerably by using a loss-minimizing technique designed explicitly for learner XYZ.

As a rough analogy, since my background is in programming, consider sorting algorithms. Quicksort is often cited as the best general sorting algorithm. But if you know in advance that your data is constrained in certain ways, other algorithms will be faster, given a specific use-case (even good old bubblesort can be fast if conditions are right!).

So, that's the question: have learner-specific optimizers been developed? (And a related question, is "optimizer" the right word to use here?)

(FWIW, I posted this same question to Kaggle Q&A, but didn't get any responses there.)

I would argue and say that different machine learning algorithms use different optimization algorithms. Linear regression has a direct solution $(X'X)^{-1}X'Y$. For logistic regression sklearn python library uses a Liblinear C++ library. The library uses "a trust region Newton method". SGD (Stochastic Gradient Descent) with Backpropagation is mainly used in neural networks and I did not hear about both of them before implementing my first neural network. LightGBM and XGBoost use their own invented heuristics to build decision trees.

Generally speaking most of ML algorithms are fast. If the training time is in seconds there is no need to optimize the time of algorithm execution. If the training time is actually in years (most recent papers on neural networks by Google where they use hundreds of GPUs and CPUs) then you definitely start thinking about speed optimization.

SGD is supposed to be inferior to other algorithms because it uses only first derivatives. The problem with the neural networks is that the gradient changes too fast and the calculation of the second derivative appears too costly (There are papers that benchmark it). SGD was specifically optimized for neural networks with momentum calculations and reducing learning rate after the networks stopped training. Constant attempts are made to improve it, for example ADAM. But ADAM is still considered to perform worse on validation sets (There was a big benchmark paper about it).

There are some papers that try to use neural networks to choose the direction of the descent instead of using SGD. Papers show some marginal improvements in speed but the algorithms are still not used as the main method of optimization.

Lots of other algorithms are tried for neural networks. For example, genetic algorithms. There are papers that use the idea of "stochastic" in SGD for genetic algorithms. According to their benchmarks they achieve 1/10th of the speed of SGD. They argue that the best neural networks are influenced a lot by the ability of SGD to find a minimum (like ResNet with residual links). But if you choose genetic algorithms then maybe it is possible to use more complicated neural networks where SGD actually fails.

In term of how to call the algorithms.

I prefere the name "optimization algorithm" or "algorithm used for training". But for some of the ML methods "optimization algorithm" sounds wrong. LightGBM and XGBoost have pretty complicated algorithms of building trees where they balance between optimization of a score (logloss or sum of squares) and using other algorithms to fight overfitting. Decision trees could be trained to zero error but then they are one of the worst predictors for the validation set. LightGBM and XGBoost do not reduce their errors to zero and thus allow to win most of machine learning competitions with structured data on Kaggle.


Test for feature dependencies in time series modelling I have time-series data that track event occurrence in 3 locations. Here's a sample:
               Count     Total
Location       A  B  C
Date
2018-06-22     0  1  1     2
2018-06-23     2  1  0     3
2018-06-24     0  0  1     1
2018-06-25     2  2  1     5
2018-06-26     0  3  1     4

I would like to use the data to predict the total number of event occurrences at a given date in the future. How do I test if an event happening in one location has an impact on events happening in another location (dependency)? I believe that if an event happening in locations B and C are dependant, I should sum the 2 columns together as 1 feature in my model.

How do I test if an event happening in one location has an impact on events happening in another location (dependency)?

Pearson correlation between the two columns would already give you a simple indication of whether there is a dependency relation.
A $\chi$-square test would tell you whether there is a significant difference between an observed variable (e.g. count in one location) and an expected variable (count in the other location). In other words, it can tell you whether the variables are independent or not.
The conditional probability $p(A|B)$ of a variable A given the other variable B tells you how likely the event A is assuming the event B happens. $A$ and $B$ are independent if $p(A|B)=p(A)$ (note that it's unlikely to be exactly equal in the case of a real sample).

I believe that if an event happening in locations B and C are dependant, I should sum the 2 columns together as 1 feature in my model.

Unless you have a specific reason to do that (e.g. you want to consider a large area which includes locations B and C), this doesn't make a lot of sense:

first dependency is not "all or nothing", two variables can have a certain degree of dependency but it doesn't mean that they follow each other exactly. Therefore you would lose some information by merging them into one feature.
this would make it impossible to predict future events for a specific location, for instance B, if the two values for B and C are combined together.


running model.evaluate many times results different accuracy and loss value tensorflow 2 I have trained a CNN network using dataset = tf.data.Dataset.from_tensor_slices((data, label)) to create the dataset. training went well but evaluating the model on test dataset results different values every time without changing anything in test dataset or the network and I am not using any Dropout or Batchnormalization:

If necessary my code:

 model = tf.keras.Sequential([
    Input((1,30,30)),
    Conv2D(filters = 8, kernel_size=(3,3), padding="same", activation="relu", name="c1", data_format="channels_first"),
    Conv2D(filters = 16, kernel_size=(3,3), padding="same", activation="relu", name="c2", data_format="channels_first"),
    MaxPool2D(pool_size=(2,2), strides=(1,1),padding="same", name="m1", data_format="channels_first"),

    Conv2D(filters = 16, kernel_size=(3,3), padding="same", activation="relu", name="c3", data_format="channels_first"),
    MaxPool2D(pool_size=(2,2), strides=(1,1),padding="same", name="m2",data_format="channels_first"),

    Flatten(),
    Dense(256, activation="relu", use_bias=True),
    Dense(5,  use_bias=True)])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"])
model.fit(train_data, verbose=1, validation_data=valid_data, epochs=20)

model.evaluate(test_data)

How I made the Dataset:

def split_dataset(dataset: tf.data.Dataset, validation_data_fraction: float):

    validation_data_percent = round(validation_data_fraction * 100)
    if not (0 <= validation_data_percent <= 100):
        raise ValueError("validation data fraction must be ∈ [0,1]")

    dataset = dataset.enumerate()
    train_dataset = dataset.filter(lambda f, data: f % 100 >= validation_data_percent)
    validation_dataset = dataset.filter(lambda f, data: f % 100 < validation_data_percent)

    # remove enumeration
    train_dataset = train_dataset.map(lambda f, data: data)
    validation_dataset = validation_dataset.map(lambda f, data: data)

    return train_dataset, validation_dataset

def load_data(path):
    data, label = data_prep(path)
    dataset = tf.data.Dataset.from_tensor_slices((data, label))
    dataset = dataset.shuffle(100000)
    train_dataset, rest = split_dataset(dataset, 0.3)
    test_dataset, valid_dataset = split_dataset(rest, 0.5)
    train_data = train_dataset.shuffle(1000).batch(10)
    valid_data = valid_dataset.batch(10)
    test_data = test_dataset.batch(10)
    return train_data, valid_data, test_data

for example running model.evaluate(test_data) gives:

885/Unknown - 2s 2ms/step - loss: 0.1039 - accuracy: 0.9663
885/Unknown - 2s 2ms/step - loss: 0.0959 - accuracy: 0.9675
885/Unknown - 2s 2ms/step - loss: 0.0999 - accuracy: 0.9661
885/Unknown - 2s 2ms/step - loss: 0.0888 - accuracy: 0.9688
885/Unknown - 2s 2ms/step - loss: 0.0799 - accuracy: 0.9715

If you train your model and then run model.evaluate N times (do not retrain the model) you should get the same answer each time PROVIDED your test data is the SAME each time. However if you train your model then run evaluate and do that combination N times results will vary due to the random weight initialization of the network.


Hand preprocessing data I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going through each data sample and preprocessing it by hand is a viable option. I think that writing an algorithm to automate the process would actually end up being more tedious and take longer than just getting my hands dirty with the data.
So, my question is: is it common for data scientists/academics/professionals to preprocess a relatively small dataset by hand if it is the best and quickest option, or is anything done without automation looked down upon?

If you have a really small dataset like a few 100 samples it's okay to do preporcessing via hand. But since you have a thousand samples it's better to automate the process. You can use the na_values attribute in pandas to fill in the "???" or "??" value with nan. Then for each column replace the Nan values using some statistical measure (in your case the most occuring word..etc). But if youre data-wrangling is unworkable via an automation process or takes too much time, then you should proceed manually. If it's about meeting a deadline a deadline the manual way works best for really messy data.


How to combine features with different temporal scale in machine learning We have various types of data features with different temporal scale. For example, some of them describe the state per second while others may describe the state per day or per month from another aspect. The former features are dense on the time scale and latter features are sparse. Simply concatenate them into one feature vector seems not proper. Is there any typical method in machine learning can handle with problem ?

Machine Learning does not have an specific method from mixing data with different sampling rates.

An extract from the paper:

We approach the problem of dealing with multiple sampling rates from an aggregation perspective. We propose Accordion, a new embedded method that constructs and selects aggregate features iteratively, in a memory-conscious fashion.

Our algorithms work on both classification and regression problems. We describe three experiments on real-world time series datasets, with satisfying results.

Mining Multivariate Time Series with Mixed Sampling Rates


How does backpropagation works through Max Pooling layer when doing a batch? Let's assume that we are using a batch size of 100 samples for learning.

So in every batch, the weight of every neuron (and bias, etc) is being updated by adding the minus of the learning rate * the average error value that we found using the 100 samples * the derivative of the error function with the respect to the current neuron weight that is being updated.

Now, when we use a Max Pool layer, how can we compute the derivative over this layer? In every sample that we feed forward, a different pixel (let's say) is chosen as the max, so when we backpropagate over 100 samples in which every time a different path was chosen, how can we do it? A solution I have in mind is to remember every pixel that was chosen as the maximum, and then maybe split the derivative over all the max-pixels. Is this what's being done?

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way

During back propagation, those activation values are used as one of the numerical sources to calculate gradients, along with gradients calculated so far working backwards and the connecting weights. Like forward propagation, back propagation is applied per example, it does not work with averaged or summed values. Only when all examples have been processed do you work with the summed or averaged gradients for the batch.

This applies equally to max pool layers. Not only do you know what the output from the pooling layer for each example in the batch was, but you can look at the preceding layer and determine which input to the pool was the maximum.

Mathematically, and avoiding the need to define indices for NN layers and neurons, the rule can be expressed like this

The forward function is $m = max(a,b)$
We know $\frac{\partial J}{\partial m}$ for some target function J (in the neural network that will be the loss function we want to minimise, and we are assuming we have backpropagated to this point already)
We want to know $\frac{\partial J}{\partial a}$ and $\frac{\partial J}{\partial b}$
If $a > b$

Locally,* $m = a$. So $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$
Locally,* $m$ does not depend on $b$. So $\frac{\partial J}{\partial b} = 0$

Therefore $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$ if $a > b$, else $\frac{\partial J}{\partial a} = 0$
and $\frac{\partial J}{\partial b} = \frac{\partial J}{\partial m}$ if $b > a$, else $\frac{\partial J}{\partial b} = 0$

When back propagation goes across a max pooling layer, the gradient is processed per example and assigned only to the input from the previous layer that was the maximum. Other inputs get zero gradient. When this is batched it is no different, it is just processed per example, maybe in parallel. Across a whole batch this can mean that more than one, maybe all, of the input activations to the max pool get some share of the gradient - each from a different subset of examples in the batch.

* Locally -> when making only infinitesimal changes to $m$.

** Technically, if $a=b$ exactly then we have a discontinuity, but in practice we can ignore that without issues when training a neural network.


What are the top schools for online part-time Masters in data science/analytics programs? Is it much better to do the program in person? I am looking to get my Masters in Science in Data Science or Analytics. I will continue working during the program so it needs to be part-time. I would prefer to do an online course, but if in-person is much better, I could be open to it. So far, I found Georgia Tech and Johns Hopkins for online, part-time programs. For in-person, I could attend Georgetown part-time. Any insight you can provide regarding which of these programs would be best, if there is another school that could be better for my situation, or if any school is essentially the same would be very helpful! Thanks!

Michigan University has a nice and good dual program that you get your Master in Data Science and also a MBA.
I'd take a look at MIT's Master of Business Analytics as well.

If you want to get a job, I don't think a Master is necessary. This Forbes article explains that pretty well https://www.forbes.com/sites/metabrown/2016/07/29/4-reasons-not-to-get-that-masters-in-data-science/#160f195a40c0


Which Optimization method to use? I have a non-function (not in closed form) that takes in a few parameters (about 20) and returns a real value. A few of these parameters are discrete while others are continuous. Some of these parameters can only be chosen from a finite space of values.

Since I don't have the function in closed form, I cannot use any gradient based methods. However, the discrete nature and the boxed constraints on a few of those parameters restrict even the number of derivative free optimization techniques at my disposal. I am wondering what are the options in terms of optimization methods that I can use.

Bayesian optimization is a principled way of sequentially finding the extremum of black-box functions. What's more, there a numerous software packages that make it easy, such as BayesOpt and MOE. Another flexible Bayesian framework that you can use for optimization is Gaussian processes: Global Optimisation with Gaussian Processes


Text Classification with mixed features in Random Forests I am working on a text classification problem on tweets. At the moment I was only considering the content of the tweets as a source of information, and I was using a simple bag of words approach using term frequencies as features, using Random Forests (this is something I cannot change).

Now my idea is to try to incorporate information present in the URLs used in tweets. Now, not all the tweets have URLs, and if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs. For this reason, I suppose that having a single set of features containing both the tweet term frequencies and the URL term frequencies could be bad. Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

Do you have any suggestions regarding this issue?

Are you using raw term frequencies, or TF-IDF?

Perhaps you could simply combine the terms in the tweet with the terms in the URL-linked pages (if any) into a single bag of words, calculate TF-IDF, and normalize to avoid bias towards longer documents (i.e., those tweets containing URL links).

  if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs

I don't understand what you mean here. Aren't your features the terms in your bag of words? So the number of features will be the size of your vocabulary, which I imagine won't change much whether you include URLs or not.

  Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

I don't understand this either. Term-document matrices are virtually always a sparse matrix, since most of the terms in your vocabulary won't appear in most of your documents. So, the vast majority of values in your TDM will be 0. I don't know where you're getting -1 from.


Keras + Tensorflow CNN with multiple image inputs I have a CNN that needs to take in 68 images that are all 59x59 pixels. The CNN should output 136 values on the output layer

My training data has shape (-1, 68, 59, 59, 1).

My current approach is to use concatenate to join multiple networks like so:

input_layer = [None] * 68
x = [None] * 68
for i in range(68):
    input_layer[i] = tf.keras.layers.Input(shape=training_data.shape[1:][1:])
    x[i] = Conv2D(64, (5,5))(input_layer[i])
    x[i] = LeakyReLU(alpha=0.3)(x[i])
    x[i] = MaxPooling2D(pool_size=(2,2))(x[i])
    x[i] = Model(inputs=input_layer[i], outputs=x[i])

combined = concatenate(x)

However, this always gives the error:

ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs

Is this approach a suitable approach or am I doing this completely wrong?

Yes it is wrong, each (68, 59, 59) input should go through one model not an array of them.

You can treat each of 68 images as a channel, for this, you need to squeeze your data axes from (-1, 68, 59, 59, 1) to (-1, 68, 59, 59) to have a 59x59 image with 68 channels corresponding to Input((68, 59, 59)), and set data_format='channels_first' in conv2D, to let the layer know that channels are in the first dimension (it expects them to be in the last dimension by default). This is similar to an RGB image that has 3 channels corresponding to Input((59, 59, 3)). The rest is the same.
If 68 images are consecutive frames from a movie, you can use conv3D to extract motion patterns across neighbor frames too; this is done by 3D kernels instead of 2D kernels. It requires  (-1, 68, 59, 59, 1) data shape corresponding to Input((68, 59, 59, 1)). Also, we should use the default data_format='channels_last' since now there is only one channel as the last dimension. Commonly, temporal axis is placed third, i.e. (-1, 59, 59, 68, 1), which can be accomplished by moving the axes.


Trying to return more than just the top result from sklearn NearestNeighbors I'm trying to compare a list of names (duplicated into a clean file and a messy file).  I then compare the files against each other.  My problem is that it returns only the top 1 result for each, which is itself (the identical record in each file).  What I am trying to capture is the second result, which would be the closest match, not being itself.

names =  pd.read_csv('C:/Temp/messynames.txt', sep='\t')
org_names = names['VariationName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams)
tf_idf_matrix = vectorizer.fit_transform(org_names)

clean_org_names = pd.read_csv('C:/Temp/cleannames.txt', sep='\t')
org_name_clean = clean_org_names['StandardName'].unique()
vectorizer = TfidfVectorizer(min_df=1, analyzer=ngrams, lowercase=False)
tfidf = vectorizer.fit_transform(org_name_clean)

nbrs = NearestNeighbors(n_neighbors=3, n_jobs=-1).fit(tfidf)
unique_org = set(names['VariationName'].values)

def getNearestN(query):
  queryTFIDF_ = vectorizer.transform(query)
  distances, indices = nbrs.kneighbors(queryTFIDF_)
  return distances, indices
distances, indices = getNearestN(unique_org)

unique_org = list(unique_org) #need to convert back to a list

matches = []
for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

matches = pd.DataFrame(matches, columns=['Match confidence (lower is better)','Matched name','Original name'])
matches.to_csv('C:/Temp/matchednames.txt', sep='\t', encoding='utf-8', index=False, quoting=3)

For a file with the following four names:

NOKIA
NOKIAA
NOKIA LMD
NOKIA LTD

The results looks like this:

Match confidence                         Matched name Original name
0                                 0.0    NOKIA LMD     NOKIA LMD
1                                 0.0       NOKIAA        NOKIAA
2                                 0.0        NOKIA         NOKIA
3                                 0.0    NOKIA LTD     NOKIA LTD

I'm trying to get to something more like:

Match confidence                         Matched name Original name
0                                 0.1    NOKIA LTD     NOKIA LMD
1                                 0.1        NOKIA        NOKIAA

My guess would be that in the following piece of code:

for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

The variable (array element):

distances[i][0]

contains the top element match for the $i^{th}$ line. Replace the second array index so it becomes:

distances[i][1]

You may want to increase one of the two zero indices in the following line as well to have a distance like value:

clean_org_names.values[j][0][0]


My Keras CNN doesn't learn I am experimenting with Keras and I managed to build a simple CNN to classify blue images (300x300 images of the same shade of blue) vs. red images (same size, just red). This is a dummy problem I assumed the NN would have solved immediately but this doesn't seem the case. In fact, even after 20+ epochs accuracy is still exactly 50%.

I assume there's a lot of stuff I could try to do differently but is there anything extraordinary wrong I am doing here that could result in such poor performance on such an easy task?

# Create a Keras model.
model = keras.Sequential()
model.add(
    keras.layers.Conv2D(
        input_shape=(300, 300, 3),
        filters=64,
        kernel_size=(3, 3),
        activation='relu',
    )
)
model.add(
    keras.layers.Conv2D(
        filters=64,
        kernel_size=(3, 3),
        activation='relu',
    )
)
model.add(
    keras.layers.MaxPooling2D(
        pool_size=(2, 2),
        strides=(2, 2),
    )
)
model.add(keras.layers.Flatten())
model.add(
    keras.layers.Dense(
        units=128,
        activation='relu'
    )
)
model.add(keras.layers.Dropout(0.5))
model.add(
    keras.layers.Dense(
        units=1,
        activation='sigmoid',
    )
)

# Train the model.
sgd = keras.optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(
    optimizer=sgd,
    loss='mean_squared_error',
    metrics=['binary_accuracy']
)
model.fit(images, labels, epochs=20, batch_size=5)

Which outputs:

20/20 [==============================] - 18s 887ms/step - loss: 0.5954 - binary_accuracy: 0.4000
Epoch 2/20
20/20 [==============================] - 16s 794ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 3/20
20/20 [==============================] - 16s 781ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 4/20
20/20 [==============================] - 17s 853ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 5/20
20/20 [==============================] - 18s 877ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 6/20
20/20 [==============================] - 18s 891ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 7/20
20/20 [==============================] - 17s 825ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 8/20
20/20 [==============================] - 17s 861ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 9/20
20/20 [==============================] - 17s 846ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 10/20
20/20 [==============================] - 17s 835ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 11/20
20/20 [==============================] - 16s 800ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 12/20
20/20 [==============================] - 16s 806ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 13/20
20/20 [==============================] - 16s 811ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 14/20
20/20 [==============================] - 17s 827ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 15/20
20/20 [==============================] - 16s 806ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 16/20
20/20 [==============================] - 16s 786ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 17/20
20/20 [==============================] - 16s 795ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 18/20
20/20 [==============================] - 16s 796ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 19/20
20/20 [==============================] - 16s 788ms/step - loss: 0.5000 - binary_accuracy: 0.5000
Epoch 20/20
20/20 [==============================] - 16s 794ms/step - loss: 0.5000 - binary_accuracy: 0.5000

If you remember the theory, CNN kernels basically work on the principle of edge detection. So now that your picture is purely blue or purely red there are no edges. I am guessing that is the main problem here, since I think ML libraries initialise kernels which are suitable for edge detection.
To be more comprehensive:

Since kernels are randomised small numbers, sum of red kernel is almost equal to sum of  blue kernel. The next layer will mainly have (sum of red kernel * 255) and (sum of blue kernel * 255) in all the channels for the next layer, which will be roughly equal. Since you are using an inappropriate loss function, the CNN becomes insensitive to small changes due to the difference in summation of kernels.

I would suggest:

Changing the padding to same might work since then edges will be introduced in the picture (black-red, black - blue).
Try by adding significant white edges to the side of the picture.
Squared error does not work that good for sigmoid activations, you need to use the cross entropy loss.
One must i believe experimenting with smaller learning rates, since CNN have the problem of exploding gradients.


Intuition behind PCA eigenvectors For undergraduate students who understand the definition of
eigenvectors and eigenvalues,
$$A v = \lambda v \;,$$
what is the intuition behind why the eigenvectors of the
covariance (or correlation) matrix correspond to the axes of maximal stretching?
Why specifically does that matrix lead to (e.g.) the largest eigenvector
corresponding to the direction of maximal spread in the data?

Since the students know about eigenvectors and eigenvalues, I will assume that they know about Lagrangian multipliers as well.

Let's start with computing the variance of data $\vec{x}$ along a given direction $\vec{e}_a$. Since we only care about the direction, $\vec{e}_a$ is a vector of unit length, i.e.
$$\vec{e}_a^T \vec{e}_a = 1$$

So first, the component of the data along $\vec{e}_a$ is given by
$$ \vec{e}_a^T \vec{x}$$
The variance $\sigma_a^2$ of this component is given by
$$ \sigma_a^2 = E[(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})^2] $$
where $\vec{\mu} = E[\vec{x}]$.
If we re-write this expression a little bit
$$ \sigma_a^2 \\
= E[(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})] \\
= E[\vec{e}_a^T(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T\vec{e}_a ] \\
= \vec{e}_a^T E[(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T ] \vec{e}_a \\
= \vec{e}_a^T \Sigma_x \vec{e}_a
$$
where in the last step we use the covariance matrix $\Sigma_x = E[(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T ]$.
So the variance along a given direction $\vec{e}_a$ is computed using $\Sigma_x$. So far, so good.

Let's now try to find the direction $\vec{e}_a^*$ where the variance is maximal. Since $\vec{e}_a^*$ is also supposed to only indicate the direction, we demand
$$\vec{e}_a^{*T}\vec{e}_a^{*} = 1 $$
With this we can formulate a constrained optimization problem using the Lagrangian multiplier $\lambda$
$$ L(\vec{e}_a) = \vec{e}_a^T \Sigma_x \vec{e}_a + \lambda (1 - \vec{e}_a^T \vec{e}_a ) $$

Taking the derivative with respect to $\vec{e}_a$ and equating it to zero, we end up with
$$ \frac{\partial L}{\partial \vec{e}_a} = \Sigma_x \vec{e}_a - \lambda \vec{e}_a = 0 $$
i.e.
$$ \Sigma_x \vec{e}_a = \lambda \vec{e}_a $$
This is a cool result. The direction of optimal (maximal) variance is a solution to the eigenvalue problem of $\Sigma_x$. Assuming that we found a normalized eigenvector $\vec{e}_b$, it is easy to interpret the eigenvalue $\lambda$

$$ \vec{e}_b^{T} \Sigma_x \vec{e}_b = \lambda = \sigma_b^{2} $$

So, to summarize:

the variance along a given direction is computed using the covariance matrix
the direction of maximal variance is a normalized eigenvector of the covariance matrix
the eigenvalues are the variances along the direction of the eigenvectors

This means, that the problem of finding the direction of maximal variance reduces to finding the eigenvector corresponding to the maximal eigenvalue. This is about as intuitive as it gets, I believe.

(I skipped the part where we need to show that all eigenvalues are non-negative and thus can be interpreted as variances, but this follows from $\Sigma_x$ being positive semi-definit.)


Extracting structure and content from invoices Lately, I have been largely inspired by this https://rossum.ai/, which is able to extract text from invoice documents.

Do you have any ideas on how this could be implemented? It's clear that they did a lot of research to reach this performance level, but in my case I am interested in the overall approach to such problems.

If I understand correctly, the first part of the pipeline is to extract different blocks from the document. In that case, is object detection the right approach to get bounding boxes around the blocks? I guess it might not be really good at extracting tabular data.

If not object detection, what is the correct way to tackle the problem?

Thanks.

I think extracting relevant details from an invoice in commercial applications certainly involves a lot of high spec algorithms. Maybe you are right that they identify relevant parts first and extract the details afterwards.

However, my first starting point would be to get all the text from an invoice (e.g. via tesseract). If you have a decent photo, tesseract will be able to OCR the content. The next step would be to identify relevant content, such as payment amount, names, and bank account numbers. This may be possible by hardcoded rules to some extent. Alternatively, one could use NLP-like models to detect certain sequences. With some effort, this should work out well since invoices are relatively structured documents.

https://pypi.org/project/pytesseract/

https://github.com/tesseract-ocr/tesseract/wiki


What does "baseline" mean in the context of machine learning? What does "baseline" mean in the context of machine learning and data science?

Someone wrote me:

  Hint: An appropriate baseline will give an RMSE of approximately 200.

I don't get this. Does he mean that if my predictive model on the training data has a RMSE below 500, it's good?

  And what could be a "baseline approach"?

A baseline is the result of a very basic model/solution. You generally create a baseline and then try to make more complex solutions in order to get a better result.
If you achieve a better score than the baseline, it is good.


Scaling sparse data for PCA Not sure how I should interpret the scaling. Is it correct to convert the sparse matrix to a dense matrix by padding with 0's and scale normally?

Assuming you are doing feature scaling: method to limit the range of variables so that they can be compared on common grounds. I'm not sure about what you are looking here. If the matrix is small, you can densify it with X.toarray(). If the matrix is large, then this will probably blow your RAM. Remember that: PCA(X) is SVD(X-mean(X)). Even If X is a sparse matrix, X-mean(X) is always a dense matrix.


My network optimises for mean_squared_error, but the predictions are useless After going through the tutorials of tensorflow I decided to do my own project. For this I did the following:

I have scraped movie reviews from a polish site called filmweb. I got a dataset o 5400 movie reviews with a rating in range 1-10
I used the “Morfeusz” python bindings to tokenize and lemmatize the text. I have stripped most basic stop words.
I mapped most used 10000 lemmas to numbers and put the data in lists.
Randomly split the data into two equal sets.
My training data is the movie review lemmas padded with zeroes (like in the tutorial). The expected result is the rating.
My model is

    vocab_size = 10000

    model = keras.Sequential()
    model.add(keras.layers.Embedding(vocab_size, 16))
    model.add(keras.layers.GlobalAveragePooling1D())
    model.add(
        keras.layers.Dense(16, activation=tf.nn.relu)
    )
    model.add(keras.layers.Dense(1))

    optimizer = tf.keras.optimizers.RMSprop(0.001)

    model.compile(optimizer=optimizer,
              loss='mean_squared_error',
              metrics=['accuracy'])

After running the optimization process i see the loss metric dropping quite fast. However the accuracy is well below 0.1 and after I check the predictions of this network, they are useless. The model predicts all the ratings to be in range of 6-7. So I understand that the network “optimized” by simply always guessing the mean value.

I wonder what should be my next step to improve. Is my model wrong for this task? Should I somehow normalize or augment my data? Or is the dataset size simply too small to get any meaningful result?

EDIT: this is the link to my dataset and the script that generated it:
https://www.dropbox.com/s/csyo934hhtbxxu9/fw-data.tar.bz2?dl=0

What you are doing right now is regression on the review score, therefore accuracy is not a suitable metric to assess performance. Your model is optimizing the mean squared error, thus it makes sense that it is predicting the (almost) average review score.

Since you are interested in the accuracy on the rating's prediction, you should set your output layer as a softmax layer with 10 possible outputs, as in:

model.add(keras.layers.Dense(10), activation='softmax')

Then you can optimize the categorical_crossentropy of your model, and report the accuracy. In this way, you'll do classification of the ratings.

model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])


np.c_ converts data type to object. Can I prevent that? Was trying my hand at the Titanic dataset, when I wanted to One Hot Encode a categorical feature, after which I wanted to combine the original data with the new one hot vectors. The datatypes are as such:

data : Pandas Dataframe

Titles_ohe : Numpy sparse matrix (float64)

I tried to merge them into a dataframe using np.c_ :

columns = (list(data))+list(Titles.values)

data = pd.DataFrame(np.c_[data.values, Titles_ohe.toarray()], columns=columns)

However on checking the data type of the resulting Dataframe, all the attributes have been changed to the object datatype. Is there any way I can prevent this while using np.c_, or is there an alternative solution? Thanks in advance for any help!

I'd use DataFrame.join() in this case:

data = data.join(pd.SparseDataFrame(Titles_ohe, index=data.index, columns=Titles))


Training data : forecasted or actual? I am working on a time series prediction problem. I am using keras models for machine learning.

For this prediction, weather variables are used as input. They can be of two types: forecasted and actual. I have acquired both these types of data of a considerable amount of time and I want to train and test my model on the data. My question is:

1) Should I use forecasted weather variables OR actual weather variables in input while training? (consider that only forecasted weather variables will be available at model inferencing time).

2) Same question 1 for testing.

Is there a rule or general practice regarding the above questions? If yes, I would like to know that.

According to me, for model training we should always use the actual data, so that your prediction is always close to realty.

  But in case if I have the prediction data as well,

  I will trained another model with the prediction data.
  Analysis the outcome of the two models.
  Identify the deviation between two outcomes.
  Will use as threshold(variation) in future prediction.


Low scale ML/statistical techniques for data poor settings I have two separate problems. One is logistic regression and other is time series prediction. But both suffer from paucity of data problems
a) For logistic regression, I have tiny dataset with 10 observations which has variables such as age, Marital_Status, income, gender and car_purchase_status (outcome flag with Yes/No values). Now I have a new 11th customer with variables such as Age, gender, Marital_Status and income. Now I would like to know whether this 11th customer will buy a car or not. Should I spend resources to influence him to buy a car? Am I spending my resources on the right customer? For ex: Is there anyway that I can find out that 11th customer has 70% or 80% pc chance of buying a car. So spending some marketing efforts such as calls can help us convince him to buy a car (100%). So, how can I do this? Any advice please? Should I just give up straight away that this problem impossible to solve with such low data or is there any simple statistical techniques that can help me gain some insights about 11th customer?
b) For time series, I have only 10 observations each spaced at 20 days gap. For ex: I have their revenue generated for day1,day21,day41,day61,day81.....day201. Now with the given 10 observations, I would like to predict the revenue generated for day500, day321, day621 etc. So, is it possible to run time series forecasting with such short series? Can you guys guide me on this please? Here, also should I give up because of low data points or is there any methods that I can use to predict the future timestamp points based on short input time series?
Can you guys please help me with list of steps/topics that can help me do this?

Generally, I'd pick a very simple, transparent/explainable model and use the results in a semi-automated way. That is, do not just derive a prediction but rather insights. You could, for example, use a (or multiple) decision tree(s) which you pre or post prune. The result could be a tree with, let's say, just 1-3 features to find simple rules like "if a customer is married and at least X years old, they have a high chance of making a purchase". With logistic regression you may use coefficients to identify features which influence the dependent variable the most. These (qualitative or semi-quantitative) rules should then be validated with domain experts.
Moreover, you need to be transparent about the accuracy and precision of your estimates. In the above example, leave purity would provide some intuition. If you report any quantitative measures (which I'd be careful with), you may want to consider confidence intervals (see here or chapter 5 in Tom Mitchell's "Machine Learning", for example). (With only 10 samples typical assumptions about normal distribution will not hold here though)
Regarding the time series I would start even simpler. Depending on the number of customers, I'd start by visualizing some or all historical data in a line plot (sales per customer over time) and check the min, max and mean per customer. This gives some intuition regarding potential trends. For example, if all observations remain constant over time for a given customer, whether there is an upward/downward trend or if the data has high variance with no clear trend. Also, there may be clusters of customers which show similar patterns. Obviously, this is neither Machine Learning nor a rigorous statistical analysis but rather a pragmatic approach supported by some basic data analytics.
What you need to be very careful with is the time horizon of any quantitative prediction: based on 10 observations at $t \in \{1,21,...,201\}$ you may derive some conclusions for, let's say, $t<=301$ (to make up a total out of the blue ballpark figure) but $t=621$  is very far in the future. Also, you need to keep seasonality in mind. For example: If your observations are all from October to April of a given year and assuming you have a winter/summer seasonal pattern. Then you cannot infer a lot for months May to September. To understand the limitations and forecast potential of your time series better, I'd speak to subject matter experts, e.g. in sales & marketing. It could also be helpful to understand their forecasting approach and cross check any insights you derive with their predictions.
But, as Erwan pointed out, be very careful to derive conclusions. Applying some "ML magic" will not find a useful pattern if there insufficient data to find any signal. And of course additional data collection would be reasonable if that is an option.