How does backpropagation differ from reverse-mode autodiff Going through this book, I am familiar with the following:

  For each training instance the backpropagation algorithm first makes a
  prediction (forward pass), measures the error, then goes through each
  layer in reverse to measure the error contribution from each
  connection (reverse pass), and finally slightly tweaks the connection
  weights to reduce the error.

However I am not sure how this differs from the reverse-mode autodiff implementation by TensorFlow. As far as I know the above algorithm first goes through the graph in the forward direction and then in the second pass computes all partial derivatives for the outputs with respect to the inputs. This is very similar to the propagation algorithm.

How does backpropagation differ from reverse-mode autodiff ?

They are more or less specify the same thing. I think the difference is that back-propagation refers to the updating of weights with respect to their gradient to minimize a function; "back-propagating the gradients" is a typical term used.

Conversely, reverse-mode diff merely means calculating the gradient of a function. Nothing here in this process implies that we aim to the weights/values of a function.


How Do I Learn Neural Networks? I'm a freshman undergraduate student (mentioning this so you may forgive my unfamiliarity) who is currently doing research using neural networks. I've coded a three-node neural network (that works) based on my professor's guidance. However, I'd like to pursue a career in AI and Data Science, and I'd like to teach myself more about these properly in-depth. Are there any books or resources that will teach me more about neural network structures, deep learning, etc. Are there any recommendations?

Note: I'm proficient in Java, Python, Bash, JavaScript, Matlab, and know a bit of C++.

If you want a good and solid start for deep learning, I would suugest to start with the appropriately named book "Deep Learning" by Ian Goodfellow et al.  After that you'll have a good base that you can expend by the many different tutorials, articles and courses available online.

However, I would also add that before doing that, you should take some basic "machine learning" class (should be available at your University). Many people these days go straight to deep learning and implementing Neural networks because it is relatively easy, but than they lack the understanding to improve it or use it to its fullest potential.


CNNs for identifying scattered object? As per my understanding, while learning the features, the Convolution part of the CNN works-on and preserves the 'spatial relationship' between the pixels.
But, I want to identify object(s) in my image whose definition might be all scattered throughout the image.
By scattered I mean - The identity of object may exist in separate disconnected islands of pixel-sets. Where each of these island in itself is a connected component of pixels.

In other words- my 'Cat' exists, not as a whole, but is broken up into pieces.

Would traditional CNN still identify my 'Cat'?
If yes, upto what level? If my Cat is broken into 4 pieces? 100 pieces?
If not, what adjustment(s) do I need to make so that I can solve this problem?

Thanks,

This should not be a problem, the convolutional kernels will have bigger and bigger receptive fields when you get deeper, and either there will be kernels at the end that will combine all the components over the whole input if the receptive field is big enough, or if you use a fully connected layer at the end it will learn that specific combinations of components combine into a specific class. The number of pieces it will be able to recognize depends on the expressiveness of your network, the amount of training samples and on the difficulty of your problem.


What kind of regression model should I do? my research question is the examine the effect of "receiving attention" from other members in an online community on "sustained participation" on the website.

I decided to measure "sustained participation" of each user by  calculating average time difference between the submissions of the user. I calculated it in the following way:

and I measured "attention" by calculating total number of the comments each user received for all the submissions he/she has posted.I also want to consider total number of votes and total number of views as well. I am not sure if it is a good idea to add those as the independent variables into model too or not?

My problem is about Dependent variable:

since some people just participated two times in two successive days and the average between submission days is 1. and some people participated 100 times and their average between submission days of them is also 1. but it is obvious that the second group that have participated 100 times had sustained participation not the first group.

So I need to consider the number of the submissions in the model too! I do not know if there is a way to consider it? how I can handle this problem?

should I group the users and do data analysis separately for them. for example users who have participated less than 10 times in one group! users with 10-20 participation in another group and etc.

I would appreciate if anyone can help me! my paper's due is so close and I need some preliminary results.

One thing you can do about your 'participation' variable is including the beginning and end of your data window. Let's say your data ranges from $start=$ 1/1/2016 to $end=$ 1/1/2017. Instead of only calculating the difference between the second and the first post, you'd calculate the difference between the first post and 1/1/2016 and then the difference between the second and the first post. (If some people join the platform after 1/1/2016 then you'd take the minimum of the join date and 1/1/2016). And you'd also calculate the difference between 1/1/2017 and last post.

So, if someone only had two posts $p_1$ and $p_2$, you would get the difference $(p_1-start, p_2-p_1,end-p_2)$. The differences $p_1-start$ and  $end-p_2$ would then be larger than the differences $p_1-start$ and  $end-p_n$ for some with $n>>2$ posts.


when can xgboost or catboost be better then Logistic regression? I need to improve the prediction result of an algorithm that is already programmed based on logistic regression ( for binary classification).

I tried to use XGBoost and CatBoost (with default parameters). but it takes a long time to train the model (LR takes about 1min and boost takes about 20 min).
and if I want to apply tuning parameters it could take more time for fitting parameters.

I want to ask if there are any suggestions to apply fastly boosting methods.
And if there are other ways to get better performance I hope to mention them, please.

Ps: My data is about 280 000 simples and 247 (numerical) features;

XgBoost often does better than Logistic Regression. I would use CatBoost when I have a lot of categorical features or if I do not have the time for tuning hyperparameters.

You should invest time in a boosting model for sure (they will always take more time than Logistic Regression) because it is worth it. If you are impatient, try CatBoost instead of XgBoost to see the improvement in accuracy. That will definitely give you the motivation to spend more time tuning your boosted trees.


Are there any tools for feature engineering? Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar to MS Excel, but that has R as the underlying language instead of VB.

Very interesting question (+1). While I am not aware of any software tools that currently offer comprehensive functionality for feature engineering, there is definitely a wide range of options in that regard. Currently, as far as I know, feature engineering is still largely a laborious and manual process (i.e., see this blog post). Speaking about the feature engineering subject domain, this excellent article by Jason Brownlee provides a rather comprehensive overview of the topic.

Ben Lorica, Chief Data Scientist and Director of Content Strategy for Data at O'Reilly Media Inc., has written a very nice article, describing the state-of-art (as of June 2014) approaches, methods, tools and startups in the area of automating (or, as he put it, streamlining) feature engineering.

I took a brief look at some startups that Ben has referenced and a product by Skytree indeed looks quite impressive, especially in regard to the subject of this question. Having said that, some of their claims sound really suspicious to me (i.e., "Skytree speeds up machine learning methods by up to 150x compared to open source options"). Continuing talking about commercial data science and machine learning offerings, I have to mention solutions by Microsoft, in particular their Azure Machine Learning Studio. This Web-based product is quite powerful and elegant and offers some feature engineering functionality (FEF). For an example of some simple FEF, see this nice video.

Returning to the question, I think that the simplest approach one can apply for automating feature engineering is to use corresponding IDEs. Since you (me, too) are interested in R language as a data science backend, I would suggest to check, in addition to RStudio, another similar open source IDE, called RKWard. One of the advantages of RKWard vs RStudio is that it supports writing plugins for the IDE, thus, enabling data scientists to automate feature engineering and streamline their R-based data analysis.

Finally, on the other side of the spectrum of feature engineering solutions we can find some research projects. The two most notable seem to be Stanford University's Columbus project, described in detail in the corresponding research paper, and Brainwash, described in this paper.


Why MA model order is from acf but not pacf For MA model in Arima, why the order references acf, but not pacf? The emphasize is why not PACF.

In https://towardsdatascience.com/significance-of-acf-and-pacf-plots-in-time-series-analysis-2fa11a5d10a8

It mentions

  Why would that be? Since our series is linear combination of the
  residuals and none of time series own lag can directly explain its
  present (since its not an AR), which is the essence of PACF plot as it
  subtracts variations already explained by earlier lags, its kind of
  PACF losing its power here

I don't still get why PACF cannot be used here, anyone can give a better intuition why it should be ACF but not PACF for giving the order for MA model?

First, I feel like crossvalidated SE might be a better fit here. Regardless, the answer comes from some mathematical facts of both AR and MA models. I don't really understand what your link is trying to suggest in its explanation.

For AR(p) models, it is a fact that the partial autocorrelation function (PACF) cuts off (abruptly hits 0) after lag p. Thus, analyzing a PACF graph can be useful since if the process follows an AR one, we can possibly identify the order if we see that the pacf graph shows significant partial autocorrelation at the first few lags and then cuts off to insignificant (i.e. not enough evidence to suggest non zero lags) lags after.

In a similar way, for MA(q) models the autocorrelation function (ACF), NOT the pacf, cuts off (abruptly hits 0) after lag p. Thus, analyzing a graph of the ACF can be useful for identifying the order of an MA(q) model for reasons very similar to the AR(p) model and the pacf graph.

For the AR(p) model, we can show the pacf for lags k > p cut to zero by solving the Yule Walker system of equations (if my memory serves correct). For the MA(q) model, we can make use of the backshift operator and/or spectral analysis (this is how I learned it, anyway) to find the autocorrelation function for lags k > q (and again, see that it is zero for all lags after q).

This methodology is part of the "Box-Jenkins" methodology, which too be honest is a bit too subjective for my tastes in the sense that we are effectively eyeballing a graph. Rarely is the case (in my own studies, anyway) do we find a time series that perfectly exhibits these cutoffs and so it becomes tricky when picking orders. I prefer instead (and I believe that this is becoming much more popular) to select orders based off information criteria, namely, AIC/AICc/BIC instead due to less subjectivity.

One final note; there are other assumptions for these models that also make up the Box-Jenkins method. Namely, ensuring the series is stationary first (no trend or seasonality, which hopefully can be addressed using differencing/seasonal differencing though not always) is an important assumption for all AR/MA/ARMA (and with differencing, ARIMA) models.


Data science after a physics degree? I'm currently pursuing a bachelor's degree in physics from a university in the UK. Most data science jobs here have a strong preference for people with PhDs in numerate degrees like physics and maths. I don't understand the point of spending 5 years of my life doing research in something I don't want to pursue a career in.

I'm confused about what master's degree I should choose if I want to apply for a job straight after graduating. A physics degree kind of leaves me hanging in the middle in terms of skills. I have done some analysis for experimental data using Python, but only very basic stuff. A bit of predictive modelling and C++, lots of maths, but barely any statistics. So I could either get a degree in CS to get better at programming or get one in statistics. Which one would give me the skills most relevant to data science?

Also, would you recommend that I get a diploma/certification in machine learning?

I think the best thing for a beginner is some practice to let the theory sink in. (If you need book suggestions, do a search.) If you are weak on computer science, contribute to an open source project like scikit-learn. If you are weak on data analysis, compete in a Kaggle competition. Write a paper and/or a few blog posts to prove your skills. Do this now while you are in school. An undergraduate physics degree should be enough to get your foot in the door if you do the above. A degree will rarely get you a job by itself; you still have to pass the job interview, and that's why I recommend getting practice.


Help me understand how word-as-vector representations are constructed Let's suppose I have a big list of words. I want to turn this list into a vector space of dimension $N$ such that each word is a vector in this vector space. But I have no idea how to go about with that. Some questions:

Is the list enough? For each element of the list, do I need $x$ example sentences also?
How does the computer deduce the dimensions of the vector space from the list/corpus?
Is there a way to figure out whether the dimensions of the vector space correspond to something in English?

I am assuming that you mean a vector representation of words, not to be confused by the vector representation produced in a bag of words approach that represent a document in vector space. Word2vec is an approach in which you train a model to represent words as a function of the provided context.

The answers that follow are:

1) No: You'll need some representation of context in which the word is used. For example: Skip-grams.
2) No: That is a user defined parameter
3) < I do not understand this question >


Is there a way to weight instances based upon the values of a feature? Is there a way to weight instances based upon the values of a feature?
For instance for making a sample better resemble a population on certain parameters.

Try reading up on Loss/ Cost functions. Samples that are closer to your model will be assigned a smaller cost while others a higher one.

https://en.wikipedia.org/wiki/Loss_function


Python - Logistic (Logit) Regression - why am I getting an Endog error? I'm running the following code:

X = dataset[['X1 transaction date', 'X2 house age', 'X3 distance to the nearest MRT station', 'X4 number of convenience stores', 'X5 latitude', 'X6 longitude', 'X7 distance to Xindian Ditsrict Office', 'X8 distance to Cardinal Tien Hospital', 'X9 distance to Shih Hsin University']]

y = dataset['Y house price of unit area']

model = sm.Logit(y, X).fit()
print(model.summary())

I'm using a CSV dataframe with information about 414 different residential properties in the Xindian District of Taiwan. My goal is to use a logistic regression to determine which attributes have the greatest effect on the properties' prices.

When running my Logistic Regression, I get an Endog Error, as follows:

  ValueError Traceback (most recent call last)
   in () 2 y = dataset['Y house
  price of unit area'] 3
  ----> 4 model = sm.Logit(y, X).fit() 5 print(model.summary())

  //anaconda/lib/python3.6/site-packages/statsmodels/discrete/discrete_model.py
  in init(self, endog, exog, **kwargs) 402 if
  (self.class.name != 'MNLogit' and 403 not np.all((self.endog>= 0) & (self.endog <= 1))):
  --> 404 raise ValueError("endog must be in the unit interval.") 405 406

  ValueError: endog must be in the unit interval.

Does anyone know why I am getting an endog error, and how to correct this issue? My outcome variable is a continuous integer.

Thank you.

The error comes from attempting to fit a classifier (logistic regression) on a regression problem. If you are trying to predict prices (continuous outcome), you should use linear regression.


How to use precomputed distance matrix and min_sample for DBSCAN clustering method? I want to perform DBSCAN on my datapoints, but I don't have access to the data, I just have the pairwise distance of datapoints. Additionally, I have no idea about the number of clusters but I do want that each cluster contains at least 40 data points. Does DBSCAN work with these conditions? For instance, can I have something like this? Or is more information needed? I want to emphasize that I have computed the pairwise distance and this is not the result of Euclidean or some other method.

from sklearn.cluster import DBSCAN

db = DBSCAN(min_samples=40, metric="precomputed")

y_db = db.fit_predict(my_pairwise_distance_matrix)

I am not sure what is eps parameter of DBSCAN(). How should I set that?

DBSCAN does not guarantee a minimum cluster size. There are known situations, c.f. Wikipedia, where a cluster can have fewer than "minPts" points. Furthermore, it has the concept of noise: points that do not have enough neighbors.

For epsilon, also see the Wikipedia article. As you don't specify the number of clusters, this parameter is what mostly controls how many clusters you get. Set it to 0, and everything will be noise. Set it to the maximum distance, and everything will be in one cluster.

Really read the article. It's about density, not about cluster sizes.


How to find impactful words affecting classification? So I know there are many methods to classify sentences into types. Like in sentiment analysis (positive, negative, neutral), spam emails (spam, not spam), etc. The thing I want to ask is how would I find the words most responsible for the categorisation. For example: sad-negative, happy-positive, the-no information, are-no information.

So how do I find the words which impact the classification?

First of all, let's just clarify that proving causation is quite difficult. And therefore, you will mostly need to show correlation. Also, for text classification, the impact of each word is not linear. Each word impacts the meaning of the sentence relative to the other words and vice versa, therefore, you need to keep that in mind.

With that said, there are a few ways you can look at the impact of words on a classification:

Frequencies

If you have multiple classes, you could look at how frequently certain words appear in each class. If the word "good" only appears in positive sentences, it is safe to assume that it has a high impact.

This method has the benefit of being model-independent.

Feature importance

If you are using a bag-of-word representation as your input, you could look at how much the presence/absence of each feature affects the results. For instance, let's say that you can classify positive sentences with 80% accuracy with all words as features. Now, try to classify the sentences again by removing certain words. If you remove the word "good" from sentences, you can look at how the performance varies.

Or, you could do the opposite and try to simply classify each word. If you classify the word "good" as positive, it means that it is probably a word that impacts sentences into being positive. If your model provides confidence, you could use that to know how much information the word brings to the sentence.

Attention mechanism

The attention mechanism will be able to tell you which part of the input influences the results the most. This works better if you use a sequential model


Best MIMO prediction algorithm for categorical variables I have researched machine learning for quite a while and would like to test out my knowledge. So I am trying to use it for lottery number prediction. The goal is not to have 100% correct prediction (which is of course impossible), but to perform better than pure random prediction.

Basically, I have the following data:

[
 [3,10,16,19,34,45],
 [7,14,15,20,28,41],
 [2,6,18,24,30,37],
...
]

As it can be seen each round six numbers will be drawn and I have made an assumption that these rounds are not independent and that each round is related to its previous round. So I am looking for a prediction algorithm that will try learn about this and output one or more predictions.

This will be a supervised learning for the algorithm. Using the above example data, it will be pre-processed so that it becomes:

[
 [[3,10,16,19,34,45], [7,14,15,20,28,41]],
 [[7,14,15,20,28,41], [2,6,18,24,30,37]],
 ...
]

Obviously, for each tuple, input is on the left while output is on the right. Also, I have another assumption that the numbers printed on the balls are really just labels, so the algorithm should consider them as categorical rather than numerical attribute. And there is no ordering for these labels, though the data appears sorted numerically in ascending order. So when the algorithm tries to learn from the training data, it should not be affected by the ball ordering in any way.

Finally, after it has been trained, if I input, for example [3,5,6,11,34,42], it should output a list of the algorithm's best guess predictions.

So what is the best prediction algorithm and the recommended programming tool for such task?

EDIT: In theory, current round should be unrelated to previous round. But I still think they do have some correlation (though very little), which is not obvious and hard to explain.

You are trying to predict random numbers here. By design the lottery numbers are unrelated and one round has no impact on the next round. For machine learning you should use input that has some kind of "predictive power". I mean data that as some information or correlation with the target variable. You can of course train a prediction algorithm on that data like that, but it does not make sense to me. In case you find something that is a solid prediction, that might mean the lottery does not draw random numbers.
...this actually should be a comment, but StackExchange does not let me write a comment.


Equation for likelihood in logistic regression I read the likelihood is defined in logistic regression as the probability
$$
L(w) = P(y|x, w) = \prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
and the log of the last equation is:

$$
log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))
$$

I understand (independent probabilities)
$$
L(w) = P(y|x, w) = \prod P(y^i| x^i,w)
$$
and I understand that from
$$
L(w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
the log is (basic log properties)
$$
log(L(w)) = \sum y^i log((\sigma(z^i)) + (1-y^i)log(1-\sigma(z^i))
$$
However, how do I get
$$
\prod P(y^i| x^i,w) = \prod (\sigma(z^i))^{y^i}(1-\sigma(z^i))^{(1-y^i)}
$$
This basically means that
$$
P(y| x,w) = (\sigma(z))^{y}(1-\sigma(z))^{(1-y)}
$$
and I just don't see that.

By the definition of the logistic regression model $\mathrm P(y = 1 | x,w) = \sigma\left(\left<w, x\right>\right)$ thus $\mathrm P(y = 0 | x,w) = 1- \sigma\left(\left<w, x\right>\right)$

You should be able to verify (by setting y=0/y=1) that this is equivalent to

$\mathrm P(y | x,w) = \sigma\left(\left<w, x\right>\right)^y \left[ 1- \sigma \left( \left<w, x \right> \right) \right]^{1-y}$

I think you understand the rest?


Combining Latitude/Longitude position into single feature I have been playing with 2 dimensional machine learning using pandas (Trying to do something like this:  https://github.com/freeman-lab/spark-ml-streaming), and I'd like to combine Lat/Long into a single numerical feature -- ideally in a linear fashion. Is there a "best practice" to do this?

A note: for those who've ended here looking for a hashing technique, geohash is likely your best choice.

Representing latitude and longitude in a single linear scale is not possible due to the fact that their domain is inherently a 3D space. Reducing that as per your needs would require a spatial flattening technique that's unheard of to me.

Reasoning

As far as lat/long merging goes, the best of best practices would be to resort to the Haversine formula, which calculates the distance between two points over a spherical surface, and receives those points' coordinates as input.

One way to incorporate that in your use case - where each point should probably have an independent lat/long combination - would be to assume the distance's origin point coordinates to be $(\varphi_1, \lambda_1) = (0, 0)$, which would render

$$d =2r \arcsin\left(\sqrt{\sin^2\left(\frac{\varphi_2 - 0}{2}\right) + \cos(0) \cos(\varphi_2)\sin^2\left(\frac{\lambda_2 - 0}{2}\right)}\right)$$

$$= 2r \arcsin\left(\sqrt{\sin^2\left(\frac{\varphi_2}{2}\right) + \cos(\varphi_2)\sin^2\left(\frac{\lambda_2}{2}\right)}\right)$$

With $r$ being Earth's radius (~6371km) and $(\varphi_2, \lambda_2)$ your point's latitude and longitude, respectively.

However, as stated before, that couldn't possibly give you a linear relation, as you can see by 3d plotting the function:

Implementation

The circumstances imply you're likely to be using pandas, or at least should be. Here's an example implementation of this relativized Haversine formula:

from math import radians, cos, sin, asin, sqrt

def single_pt_haversine(lat, lng, degrees=True):
    """
    'Single-point' Haversine: Calculates the great circle distance
    between a point on Earth and the (0, 0) lat-long coordinate
    """
    r = 6371 # Earth's radius (km). Have r = 3956 if you want miles

    # Convert decimal degrees to radians
    if degrees:
        lat, lng = map(radians, [lat, lng])

    # 'Single-point' Haversine formula
    a = sin(lat/2)**2 + cos(lat) * sin(lng/2)**2
    d = 2 * r * asin(sqrt(a))

    return d

Which could be used as in the below minimal example:

>>> import pandas as pd

>>> df = pd.DataFrame([[45.0, 120.0], [60.0, 30.0]], columns=['x', 'y'])
>>> df
      x      y
0  45.0  120.0
1  60.0   30.0

>>> df['harvesine_distance'] = [single_pt_haversine(x, y) for x, y in zip(df.x, df.y)]
>>> df
      x      y  harvesine_distance
0  45.0  120.0        12309.813344
1  60.0   30.0         7154.403197


In sklearn's classification report, is f1 the best accuracy measure? In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model?

             precision    recall  f1-score   support

          0       0.70      0.68      0.69      5007
          1       0.65      0.54      0.59      2270
          2       0.37      0.22      0.28       614
          3       0.74      0.30      0.42       252
          4       0.59      0.42      0.49       262
          5       0.35      0.11      0.17       455
          6       0.34      0.23      0.27       248
          7       0.09      0.05      0.06       133
          8       0.38      0.15      0.21       395
          9       0.43      0.31      0.36       182
         10       0.23      0.12      0.16       230

avg / total       0.60      0.52      0.55     10048

As far as I understand it:

Precision tells us the amount of samples the classifier has correctly
marked as true positive out of all positive results.
Recall tells us about the number of samples the classifier was able
to get correct out of all samples in the set.
F1-score is the harmonic mean of precision and recall.

Maybe I'm misinterpreting the classification_report, but wouldn't the f1-score give the best view on the performance of the classifier?

You got recall incorrectly. It does not mean "correct out of all the samples".
$\hspace{50mm}$

Looking at the image,

$\hspace{50mm}$ Recall = $\frac{TP}{TP+FN}$

It is explained here.

From the above documentation, you can also see that you can modify your F-beta score to suite which to weigh more, precision or recall. Therefore, f-score would give you an overall of how good your classifier is, but if you need to know at which samples your classifier fails, you need precision and recall.


Text Mining from Images I want to do the following project and I think the best way is using tensorflow/keras: I have photos as images from a list with prices of products and I need classify these text objects from the images such that i identify the product name and the price.
So I think it is different from normal Image classification since my objects are text objects?
First, I thought it is like object detection in images but I think i need more to take these texts to another neural network?

I think you are referring to a king of image auto captioning, and why not sentence alignement ! I suggest you to go over the paper Deep Visual-Semantic Alignments for Generating Image Descriptions which is one of the references today (by M. Karpathy).
The first step is about recognizing objects in an image and align sentences to them (done with R-CNN and BRNN) and the second is about generating new captions (CNN + RNN)

Even though it may be quite difficult to deeply go through, there is some code available on GitHub and it has been remade by a a lot of people with various implementations. Here is the summary.

Good luck !


How can autoencoders be used for clustering? Suppose I have a set of time-domain signals with absolutely no labels. I want to cluster them in 2 or 3 classes. Autoencoders are unsupervised networks that learn to compress the inputs. So given an input $x^{(i)}$, weights $W_1$ and $W_2$, biases $b_1$ and $b_2$, and output $\hat{x}^{(i)}$, we can find the following relationships:

$$z^{(i)} =W_1x^{(i)}+b_1$$
$$\hat{x}^{(i)} =W_2z^{(i)}+b_2$$

So $z^{(i)}$ would be a compressed form of $x^{(i)}$, and $\hat{x}^{(i)}$ the reconstruction of the latter. So far so good.

What I don't understand is how this could be used for clustering (if there is any way to do it at all). For example, in the first figure of this paper, there is a block diagram I'm not sure I understand. It uses the $z^{(i)}$ as the inputs to the feed-forward network, but there is no mention to how that network is trained. I don't know if there is something I'm ignoring or if the paper is incomplete. Also, this tutorial at the end shows the weights learned by the autoencoder, and they seem to be kernels a CNN would learn to classify images. So... I guess the autoencoder's weights can be used somehow in a feed-forward network for classification, but I'm not sure how.

My doubts are:

If $x^{(i)}$ is a time-domain signal of length $N$ (i.e. $x^{(i)}\in\mathbb{R}^{1\times N}$), can $z^{(i)}$ only be a vector as well? In other words, would it make sense for $z^{(i)}$ to be a matrix with one of its dimensions greater than $1$? I believe it would not, but I just want to check.
Which of these quantities would be the input to a classifier? For example, if I want to use a classic MLP that has as many output units as classes I want to classify the signals in, what should I put at the input of this fully-connected network ($z^{(i)}$,$\hat{x}^{(i)}$, any other thing)?
How can I use the learned weights and biases in this MLP? Remember that we assumed that absolutely no labels are available, so it is impossible to train the network. I think the learned $W_i$ and $b_i$ should be useful somehow in the fully-connected network, but I don't see how to use them.

Observation: note that I used an MLP as an example because it is the most basic architecture, but the question applies to any other neural network that could be used to classify time-domain signals.

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar.  Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space.  It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step.  You would map each input vector $x_i$ to a vector $z_i$ (not a matrix...) with a smaller dimensionality, say 2 or 3.  You'd then use some other clustering algorithm on all the $z_i$ values.

Maybe someone else can chime in on using auto-encoders for time series, because I have never done that. I would suspect that you would want one of the layers to be a 1D convolutional layer, but I am not sure.

Some people use autoencoders as a data pre-processing step for classification too.  In this case, you would first use an autoencoder to calculate the $x$-to-$z$ mapping, and then throw away the $z$-to-$\hat{x}$ part and use the $x$-to-$z$ mapping as the first layer in the MLP.


Fixed effects or random effects in RNN Lately, I have been concerned to implement fixed effects and random effects (from econometrics) in deep learning.

After reading some articles, I realized that most of them just used only the neural network based on RNN with panel data. They were not considered to panel data structure such as fixed effects or random effects.

In my knowledge, in the case of LSTM, the weights in one cell are the same for all panel subjects. There is no consideration of the group average. This seems inappropriate from the viewpoint of panel data analysis. I believe that it should be considered to fixed effects, or random effects, or multi-level model in deep learning. Although there are some articles to apply those models in deep learning(https://arxiv.org/pdf/1702.06512.pdf, http://willwolf.io/2017/06/15/random-effects-neural-networks/) , most data scientists seem to think that it is not important or not necessary.

So I would like to hear opinions from experts about using fixed or random effects in deep learning.

Thank you.

This is an interesting question. Fixed effects have been developed in a linear model world (OLS-like). So maybe the twist in a highly non-linear world (like NN) is a little different. As far as I know, NN has not yet had too much impact in econometrics. So it seems that there still is need to bridge some empirical concepts.

Let's think about fixed effects (FE). FE uses the "within variation" of each subject to which a FE is applied to estimate something. So what is the problem with deriving this within variation (the change from time t=1 to 2, e.g. x2i - x1i) and plug it into NN manually. Remember that an NN without any hidden layer is just a linear like model. If you do something like this you may probably not need a LSTM layer since all that is relevant in this setting is the change from t1 to t2. Normal dense layers will do, I guess.

Another alternative would be to use "dummy FE", so just plug in one dummy for each subject to be controlled by fixed effects. This data representation could probably work well in an LSTM setting. However, I think we cannot strictly speak of a fixed effect model in this case, since - by the nature of NN - there is no "separate constant term" (like in OLS) for each subject. But is this really a problem? I don't think so, because what you still do in the end is, to control for each subject separately.

A more fundamental question is what you gonna do with the results. FE in economics are primarily used in causal modelling (FE should make sure that certain model assumtions are met). In NN - generally a predictive modelling setup - there are way too many "coefficients" (weights) to make a meaningful causal statement. So I guess that its not so much about the strict econometric meaning of FE, but more about adequate control for FE in NN. In this case, adding dummies for each subject should be a working solution.


input_dim for Dense Layer after LSTM layers Keras Do I need to specify the input_dim (which means the number of features in one row/sample) after adding the first LSTM layer for the later Dense layers?

I was trying to create an architecture with 2 LSTM layers and 1 Feed-forwarding layer with 200 cells and 1 Feed-forwarding layer with 2 cells. First LSTM layer outputs for every timestamp second LSTM layer outputs for only the last time step so I was wondering If I created that architecture correctly

model.add(LSTM(units = 200, return_sequences = True, input_shape = (WINDOW_SIZE, 9), batch_size = 206))
model.add(LSTM(units = 200, return_sequences = False))
model.add(Dense(units = 200, input_dim = 9))
model.add(Dense(units = 2, input_dim = 9))
model.add(Dense(units = 1, input_dim = 9))

You do not need to specify the input_dim for the later layers, the model can infer the shape of those input layers from the output shape of the previous layer.

In addition, the input_dim values currently specified don't match the output dimensions of the prior layers- for example, the output dimension of the LSTM layer with return_sequences = False will be (200,). The network should fail to compile with an expected input size error with this code.

You can fix this by correcting or removing the input_dim values from the final three layers.


Qualifications for PhD Programs Yann LeCun mentioned in his AMA that he considers having a PhD very important in order to get a job at a top company.

I have a masters in statistics and my undergrad was in economics and applied math, but I am now looking into ML PhD programs. Most programs say there are no absolutely necessary CS courses; however I tend to think most accepted students have at least a very strong CS background. I am currently working as a data scientist/statistician but my company will pay for courses. Should I take some intro software engineering courses at my local University to make myself a stronger candidate? What other advice you have for someone applying to PhD programs from outside the CS field?

edit: I have taken a few MOOCs (Machine Learning, Recommender Systems, NLP) and code R/python on a daily basis. I have a lot of coding experience with statistical languages and implement ML algorithms daily. I am more concerned with things that I can put on applications.

If I were you I would take a MOOC or two (e.g., Algorithms, Part I, Algorithms, Part II, Functional Programming Principles in Scala), a good book on data structures and algorithms, then just code as much as possible. You could implement some statistics or ML algorithms, for example; that would be good practice for you and useful to the community.

For a PhD program, however, I would also make sure I were familiar with the type of maths they use. If you want to see what it's like at the deep end, browse the papers at the JMLR. That will let you calibrate yourself in regards to theory; can you sort of follow the maths?

Oh, and you don't need a PhD to work at top companies, unless you want to join research departments like his. But then you'll spend more time doing development, and you'll need good coding skills...


Linear Regression performance in R I have a problem with R's performance:
here is my Script.

The problem is that I need to use it in a base with  ~6m of clients  (with one linear model per customer) and it's taking to long to process.

Can anyone help me to improve the performance of my script? I think that the problem is in the data.frame rbind function.

There's a few problems in your script actually, and growing objects (whether using rbind() or bind_rows() ) seriously messes up your performance. So if you know how many rows you have, start by creating an object with that many rows.

Next, you rely on a data frame. If you just store the info in separate vectors and only combine them in the end, you remove again a ton of overhead. Take a look at following three mockup examples:

library(rbenchmark)
library(dplyr)
# The naive way
f1 <- function(n){
  out <- data.frame(x = numeric(), y = numeric())
  for(i in seq.int(n)){
    tmp <- data.frame(x = rnorm(1), y=rnorm(1))
    out <- bind_rows(out, tmp)
  }
  return(out)
}
# Preallocating memory
f2 <- function(n){
  out <- data.frame(x = numeric(n), y = numeric(n))
  for(i in seq.int(n)){
    out$x[i] <- rnorm(1)
out$y[i] <- rnorm(1)
  }
  return(out)
}
# Using vectors
f3 <- function(n){
  outx <- numeric(n)
  outy <- numeric(n)
  for(i in seq.int(n)){
    outx[i] <- rnorm(1)
    outy[i] <- rnorm(1)
  }
  return(data.frame(x = outx, y = outy))
}

On my machine, I get the following timings:

benchmark(
  f1(100),
  f2(100),
  f3(100),
  columns = c("test","elapsed","relative","replications")
)
##     test elapsed relative replications
## 1 f1(100)    2.16    27.00          100
## 2 f2(100)    0.34     4.25          100
## 3 f3(100)    0.08     1.00          100

Just changing this can seriously cut your running time already.


A quick glance at the docs for LogisticRegressionWithLBFGS indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's glm is returning a maximum likelihood estimate of the model while Spark's LogisticRegressionWithLBFGS is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all smaller in magnitude than those in the R model.

I'm not sure whether or not glm in R is implementing feature scaling, but this would also contribute to different model values.


Image similarity without perspective I want to determine the similarity between images based on different features. The images show the same type of object (e.g. cars). I want to order images based on their similarity (e.g. through a feature vector). There are ways to solve this, for example a convolutional neural network. However, the images may be taken from slightly different perspectives: Sometimes directly from the front, sometimes with a slight ankle. What I DONT want is images ordered by their perspective rather than by the actual object. My questions:

What can I do to avoid primary order by perspective
Do I have to expect the ankle of the picture to be a primary factor in sorting order or will it more likely be negligible?
Is there a way to extract/visualise the features the network uses (unsupervised) in order to manipulate the sorting order (e.g. by saying I want to increase the weight of THIS visual feature when ordering)?

Generative Adverserial Networks can be tolerant of this type of perturbation. In fact, if you look at figure 8 of the DCGAN paper, the authors demonstrate how the GAN learned to encode perspective directly such that the authors are able to use the GAN to rotate the camera on an image by applying a "turn vector".


Compare image similarity in Python I'm using a dataset of movies and would like to group if a movie is the same across different retailers.

Example:

Movie: Beauty and the Beast

Platforms: Google, Netflix, iTunes, Amazon.

I have access to signals like:
Studio, Movie Name, Runtime, Language, Release Year, etc. However, in this case some movies, which are not the same and signals mentioned before, are not capable of finding the right match. I need to do what a human would do: Check Movie cover. Example:

Beauty and the Beast http://www.imdb.com/title/tt2771200/
Beauty and the Beast http://www.imdb.com/title/tt6305650/?ref_=fn_al_tt_1

I have access to the art image.
I'm using Python to do this comparison.

Is there a library that can help me compare 2 images and determine if they are similar?

You can use ImageHash

The difference between the hashed images will give you a similarity score

from PIL import Image
import imagehash

hash = imagehash.average_hash(Image.open('test.png'))
otherhash = imagehash.average_hash(Image.open('other.bmp'))

print(hash - otherhash)

https://pypi.org/project/ImageHash/


First perform data augmentation or normalization? Should I first perform data augmentation or normalization in deep learning? I am mainly interested in 2D and 3D input data. In tutorials that I have seen so far the data augmentation always comes first. Is there a (mathematical) reason for that? Would it also work vice versa?

In the specific context of image/video problems, it may be okay to normalize the data before augmentation, because you already know that each feature (pixel) will have a value between 0 and 255.  As long as you normalize w.r.t. the theoretical min and max value, then the order of normalization/augmentation shouldn't matter.
But in general, data augmentation should always come first.  Otherwise, the normalized features may be incorrect after augmentation operators are applied.  After all, you can think of augmentation as gathering additional data.  The normalization computed on your original dataset may not be valid after gathering additional data.

To illustrate, suppose we have a dataset consisting of three 2x2 matrices like this:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
$$
And suppose we decide that one good way to augment the data is to flip one or more of the images horizontally.  If we do that for the third image before normalizing.  Then the dataset becomes:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
%
\begin{bmatrix}
200 & 100 \\
0 & 0
\end{bmatrix}
$$
And after normalization, the dataset will be:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.25 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 0.5 \\
0.0 & 0.0
\end{bmatrix}
$$
This is the correct encoding of our dataset, and it can be transformed back into the original feature space without errors.

Now what happens if we normalize first, then apply our augmentation operators?
Again we start with our dataset of three matrices:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
$$
This time we normalize first.  Notice that the top-left feature has a range [0, 100] in our dataset, while all other features have a range [0, 200].  So after normalization the dataset becomes:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
$$
Now let's augment the dataset like we did before, by flipping the last matrix horizontally.  The augmented dataset is:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
$$
Oh no! The last two matrices are identical in the normalized space.  But we know that they should represent two different instances.  And if we try to map these matrices back into their original form, we would get incorrect results.  The last two matrices would be identical, even though we know they should be flipped.
Usually, the errors would not be this extreme, but this example show why it is dangerous to normalize before augmentation.  After augmentation, the features may not "line up" correctly with the original normalization.


XOR Neat Python Result is incorrect? I ran the Neat Python XOR example and a neural network it found has the following properties:

  Best individual in generation 22 meets fitness threshold - complexity:
  (2, 8) Number of evaluations: 3450

  Best genome: Nodes:
          NodeGene(id=0, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=1, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=2, type=OUTPUT, bias=0.0, response=4.921960812206318, activation=sigmoid)
          NodeGene(id=3, type=HIDDEN, bias=-2.340619398413843, response=4.930091408143276, activation=sigmoid)
          NodeGene(id=4, type=HIDDEN, bias=0.4674809078722945, response=4.918937894580367, activation=sigmoid) Connections:
          ConnectionGene(in=0, out=2, weight=1.0534262789427578, enabled=True, innov=0)
          ConnectionGene(in=1, out=2, weight=-1.6479324848766828, enabled=True, innov=1)
          ConnectionGene(in=0, out=3, weight=-3.5479647367966765, enabled=True, innov=2)
          ConnectionGene(in=3, out=2, weight=5.970619032226201, enabled=True, innov=3)
          ConnectionGene(in=1, out=3, weight=4.634338591600734, enabled=True, innov=4)
          ConnectionGene(in=0, out=4, weight=0.19203601809123594, enabled=True, innov=5)
          ConnectionGene(in=4, out=2, weight=-0.6588709650159562, enabled=True, innov=6)
          ConnectionGene(in=4, out=3, weight=-3.0010419817357072, enabled=True, innov=8) Node order: [4, 3]

  Output: expected 0.00000 got 0.11990
  expected 1.00000 got 1.00000
  expected 1.00000 got 0.92939
  expected 0.00000 got 0.00395 {2} {2} {0,
  1, 3, 4} {0, 1, 2, 3, 4}

However, when trying to implement the neural network in Python, I don't get the same results:

import math

def sigmoid(x):
    f = 1 / (1 + math.exp(-x))
    return f

node0 = 1
node1 = 1
node2 = 0

node3_bias = -2.340619398413843
node4_bias = 0.4674809078722945

connection_0_2 = 1.0534262789427578
connection_1_2 = -1.6479324848766828
connection_0_3 = -3.5479647367966765
connection_3_2 = 5.970619032226201
connection_1_3 = 4.634338591600734
connection_0_4 = 0.19203601809123594
connection_4_2 = -0.6588709650159562
connection_4_3 = -3.0010419817357072

def main():
    activated_node4 = sigmoid(node4_bias * 1 +
                              node0 * connection_0_4)
    activated_node3 = sigmoid(node3_bias * 1 +
                              activated_node4 * connection_4_3 +
                              node0 * connection_0_3 +
                              node1 * connection_1_3)
    activated_node2 = sigmoid(activated_node4 * connection_4_2 +
                              activated_node3 * connection_3_2 +
                              node1 * connection_1_2 +
                              node0 * connection_0_2)

    print("The value is: ", activated_node2)

if __name__ == "__main__":

    main()

For nodes 1 and 1 I get:

The value is:  0.30957048732254944

which does not match the expected output above.

Can anyone please help me with some pointers as to what I am getting wrong?

Thank you.

In case anyone has the same issue, please follow the github thread at:

https://github.com/CodeReclaimers/neat-python/issues/79


matplotlib geostrophic wind plot I am working on an assignment that is teaching how to plot and label using matplotlib using Python. Science or math is not my background. I have been given the formula for calculating the geostrophic wind and we are to plot it (on the y-axis) versus the latitude on the x-axis.

I know how to plot give an x and a y. Beyond that, the formula is not making sense to me given my lack of background in the area.

Furthermore, there is a key to translate the symbols to the correct python code, but I don't know the English names to the symbols, making it very difficult.

For example, I am given

r'$x^{10}$'
r'$R_^{final}$'
r'$alpha^{\eta}$'

The first two are superscript and subscript. That I understand. But how this helps with the formula calculations I do not know.

I am given the values to put into the formula as well. An explanation of the order of operations would help.
If,
g0=9.81 ms-‐2;
ΔZ=60m;
Δn=2x10^5m;
and
f=2Ωsin(φ)

My question is how do I put the values into the formula and then plot them in matplotlib? is it as easy as x and y?

Your answer was incorrect because the axes were flipped and it didn't plot the entire line.

This was the correct answer I was looking for.

import matplotlib.pyplot as plt
import numpy as np

# define the parameters
g = -9.81 # m/s^2
dZ = 60 # m
dx = 2e5 # m
omega = 7.2921e-5 # rad/s
phi = np.linspace(10,90) # deg
f = 2 * omega * np.sin(np.radians(phi)) # coriolis frequency, s^-1

# compute geostrophic wind, x-component
u_g = -1. * g/f * dZ/dx

# plot phi vs V_g
fig, ax = plt.subplots()
ax.plot(phi, u_g, color='#FF33CC')
ax.set_xlabel('$\phi$')
ax.set_ylabel('$V_{g}$ (m $s^-1$)')
plt.title('Geostrophic Wind vs. Latitude')
plt.show()


Embedding a Machine Learning Model into a Web Application I'm looking  to  find some  tutorials  involving embedding a Machine Learning Model into a  Web Application eventualy with some storage and database behind( with things like slack  etc)  .

I am not sure you would get any such tutorial regarding the same. It is more like asking how to build a software.

For example, consider you have to build an ML model for doing Sentiment Analysis before a post comes onto your feed, this is how you need to build the system:

The ML model would constitute the backend of the software, as it is just another algorithm we are using for building the software.
The plots or the feed you want to display to the end user would constitute the front end, which would include the templates and the views.

The above can be achieved through an MVC framework. The data can be sent back and forth the server and the client through a protocol.

Useful link

In case, you want to integrate it with an external system, then create an API for the above system which you have built and let the other systems use your software/app by calling your api.

Now, create a simple SlackBot which can interact with your API with some certain commands, and display the relevant results in the group.

Example of such a system


Making (500,) array (500,20) I have this array (500,):
https://pastebin.pl/view/adeccfa1
Want to make it look like this one (500,20):
https://pastebin.pl/view/6650ad6b
How do I do it?

(500,) stands for a array with 500 values,
array (500,20), stands for an array which contains 500 list and each list containing 20 elements.
Examples:
array(1,2,3,4) ----------------------> (4,)

array([1,2],[3,4],[5,6])-------------> (3,2)


XGBoost (R Interface) throwing "label set cannot be empty" error I am trying to use the XGBoost model to perform a multi-class classification over 40 classes.

The code is as follows:

xgb_params = list(colsample_bytree= 0.7,
                  subsample = 0.7,
                  eta = 0.05,
                  objective= 'multi:softmax',
                  max_depth= 5,
                  min_child_weight= 1,
                  eval_metric= "mlogloss", num_class = categoryclassnos,
                  nthread=4)

fit.xgb = xgb.train(params = xgb_params,
                    data = dtrain,
                    nrounds = 500,
                    watchlist = list(train = dtrain, test=dtest),
                    print_every_n = 50)

However, I am getting the following error:

  Check failed: (info.labels.size()) != (0) label set cannot be empty

I have reproduced the dataset and the R script here.

Any help/ pointers are deeply appreciated.

In python I needed to set the label param in  dtrain = xgb.DMatrix(X_train,label=y_train,feature_names=cfg_col_X)

PS I was going to comment but missing rep points.


What is the best way to split a sentence for a keyword extraction task? I'm doing a keyword extraction using TF-IDF on a large number of documents. Currently, I'm splitting each sentence based on n-gram. More particularly, I'm using trigrams. However, this is not the best way to split each sentence into ints constituting keywords. For example a noun phrase like 'triple heart bypass' may not always get detected as one term.

The other alternative to chunk each sentence into its constituting elements look to be part of speech tagging and [chunking][1] in [Open NLP][2]. In this approach phrase like 'triple heart bypass' always gets extracted as a whole but the downside is in TF-IDF the frequency of extracted terms (phrases) dramatically drops.

Does anyone have any suggestion on either or these two approaches or have any other ideas to improve the quality of the keywords?

I think the tagging approach has some merit here. The frequency drop you're observing as a consequence of using this is to be expected, I think. After all, keywords are the words that help differentiate a document from others in a corpus. If you have access to an ontology related to the subject matter of your corpus, you could try to map your rare keyword tags to the ontology, and use parent-level information for each to get a slightly more general set of keyword tags! If this is an approach that interests you, Stanford's open source protégé system is a good framework for working on this.


Is my model to big? I am trying to predict orders for a company, and I don't know if there are typical values for macroparameters I am building a model to predict orders, from its time series (univariate), for a company. I am working with 30 layers of 400 LSTM neurons each with the activation function hyperbolic tangent of Yann Lecun, 1 layer of 400 LSTM neurons with the relu activation function, and 3 layers of 500 neurons each, with a Relu Activation Function.

These values have had the best performance yet, but I don't know if there are typical values for training, or if it is an abnormally huge model.

Theoretically, a model should be big enough to have a low bias (avoid underfitting), but not too big as to have too high of a variance (avoid overfitting), called the bias-variance tradeoff

Whether the bias is too low for you or not depends on the true distribution you are trying to model. Practically, this means whether your model is getting a low training loss or not.

Whether the bias is too high or not depends on the true distribution and how much data you have. If your model achieves low training loss but high validation loss, then it's overfitting and thus has too high of a variance. In that case you need to either lower the complexity of the model or get more data.

If you ever reach a point where

1- reducing the complexity of the model will make training loss too high, and

2- increasing the complexity of the model will make validation loss increase

Then the only way to improve is to either change your model architechture/hyperparameters in some way or get more data.


Does increasing the n_estimators parameter in decision trees always increase accuracy I'm using some ML algorithms (from sklearn lib) and on most of them there is a parameter n_estimators which is (if I understood well) the number of used trees. Intuitevely I would say that the more trees you use, the more accurate results you get. It turned out to be not exactly true, sometimes, a very few number of trees give much better results, but I can't figure out why ?

Edit

Some precisions: this is a regression problem, with a dataset containing
about 4000 samples and 500 features. I'm using GradientBoostingRegressor, ExtraTreeRegressor, AdaBoost, RandomForest.

Edit 2

Additional info: I use a cross-validation with KFold=10 to evaluate the accuracy of the algorithms. The best n_estimators value seems to be 50, which give a R2 score of ~56/57% +- 8% for all above cited algo. When I try to increase it, the score quickly decreases. I tried several values, from 100 to 500, it keeps decreasing even reaching 52%.

There are a lot of misconceptions about regression random forest. Those misconceptions about regression rf are seen also in classification rf, but are less visible. The one I will present here is that regression random forests do not overfit. Well, this is not true.

Studying the statistical properties of the random forests shows that the bootstrapping procedure decreases the variance and maintain the bias. This property should be understood under the bias-variance tradeoff framework. It is clear that the random forests approximate an expectation, which means the mean of the true structure remains the same, while the variance is reduced. From this perspective, the random forests do not overfit. There is a problem, however, that problem is the sample itself which is used for training. The expectation is taken conditional on data. And if the data is not representative of the problem, it is normal that in the limit when the number of the tree grows to infinity. In plain English, this means that the regression forest will learn too well the data and if the data is not representative, then the results are bad. In which way the data might not be representative? In many ways, one would be that you do not have enough data points in all region of interests for example. This problem is seen often with testing error, so it might not affect you so much, but is possible to see that in CV also.

Another issue with regression trees is the number of significant variables and the number of nonsignificant variables in your data set. It is known that when you have few interesting input variables and a large number of noise variables the regression forests does not behave well. Boosting procedures does not have this behavior. There is a good reason for that. Regression forests produce more uninteresting trees which have the potential to move the learned structure away from the true underlying structure. For boosting this does not happen since at each iteration only the region of interests have large weight, so the already learned regions are affected less. The remedy would be to play with the number of variables selected on learning time.

There is a drawback however even if you increase the number of variables take into account at learning time. Consider two randomly grown trees. If you would have 100 input variables and select 10 of them for learning, there are small chances that the trees look similar. If instead you would select 50 variables for learning then your trees have better chances to look similar. This is translated into the fact that if you increase the number of variables for testing candidates at learning time, then the correlations between them increases. If the correlation increases, then they will not be able to learn a more complex structure, because of their correlations. If the number of variables selected is small you have the potential to learn more due to diversity, but if the significant variables are small compared to nonsignificant, this would lead to learn noise, too much noise. This affects most of the time the CV performance.


Choosing between two object detection model checkpoints I have the mAP scores for an object detection model evaluated at different model checkpoints. I want to choose the model that has the maximum recall on the test dataset, meaning the model which missed the least number of objects. However, the model that achieves this doesn't have the highest mAP.

mAP is the most common metric for evaluating an object detection model. How do you interpret this metric?

For the best real world performance, is it better to:

Choose a model checkpoint based on the least number of false negatives, even though the mAP isn't the highest?
Choose a model checkpoint with the highest mAP, even if it means the number of false negatives is higher?

Choosing the model with the lowest loss on the validation set would be more appropriate. There are cases where the lowest loss model also exhibits the highest mAP, but that definitely is not the norm.

Regarding your question, if you are to choose the model based on accuracy-like metrics, consider what would be less harmful in your specific example. For instance, consider the case of cancer classification, predicting that a healthy patient has cancer (False Positive) might result in some frustration but it is not that big of a deal. On the other hand, predicting a cancer patient as of being healthy might cost his life.


memory problem due to large file I'm new in Python and definitely, I'm sure I do some mistakes.
Here is my problem and thanks all for your help in advance.
I have 2 files (one is Hive) and the other is CSV and merging them. I have 64GB memory and I believe the CSV file I create is around 25+ GB.
My problem is when I connect remotely, I see that memory usage hits to 100%, and then I cannot even connect my workstation remotely and it needs a hard boot.
What I'm thinking is,
when I merge these 2 tables, I like to save in CSV (let's say 100,000 rows) and clean that from the memory and continues with another 100,000 rows, append to it, and so on...
I'm not sure how to do this, I found some within Google search, most likely is about to read large files, but not sure after I read (merge or during merge in my case), to write every 100K chunks to a CSV and clean it from the memory.
Any suggestions will help.

I guess you are trying to use pandas, if so. Don't because you can't do that for what you want. I guess your operation needs all data to be loaded to the memory which is not possible. Try to use dask or a kind of sql. I'm not sure whether you want to do operations such as group by which needs all data to be loaded simultaneously or not. If you can do extra coding and your operations don't imply to load all data, such as finding the minimum of a column, you can use generators and specify the chunk_size of the read_csv method in pandas. But you have to do extra coding. You can also take a look at here. You can also take a look at here to figure out why sql and dask are more better for large operations.


Pass data to CNN with multiple outputs in keras For some reason am I getting an uexpected output dimension for my classification network.

The network has 18 inputs of shape (45,5,3) - data formatted as (samples,45,5,3)

And the output is a vector of length 15 - one class for each third of 45.  The extracted classes comes from a pool of 145 classes. - data formatted as (samples,15)

My network looks like this:

#stride = 2
#dim = 40
#window_height = 5
#splits = ((40-5)+1)/2 = 18

kernel_number = int(math.ceil(splits))
list_of_input = [Input(shape = (45,5,3)) for i in range(splits)]
list_of_conv_output = []
list_of_max_out = []
for i in range(splits):
    list_of_conv_output.append(Conv2D(filters = kernel_number , kernel_size = (int(splits-3),3))(list_of_input[i]))
    list_of_max_out.append((MaxPooling2D(pool_size=((2,2)))(list_of_conv_output[i])))

merge = keras.layers.concatenate(list_of_max_out)
print merge.shape
reshape = Reshape((15,324))(merge)

dense1 = Dense(units = 1000, activation = 'relu',    name = "dense_1")(reshape)
dense2 = Dense(units = 1000, activation = 'relu',    name = "dense_2")(dense1)
dense3 = Dense(units = 145 , activation = 'softmax', name = "dense_3")(dense2)
model = Model(inputs = list_of_input ,outputs = dense3)

But for some reason am I getting an error when I passing my output data.
It is currently stored as numpy.ndarray of shape (16828,15) and I get an value error stating:

Error when checking model target: expected dense_3 to have 3 dimensions, but got array with shape (16828, 15)

Why is it expecting 3 dim rather than 2 dim?

The model summary clearly indicates that the output dim is (15,145) as I also would expect?  15 classes from a pool of 145 classes. right?

Model summary:
https://pastebin.com/27YTQW2m

The first dimension is always the sample index. In the model summary this is written as None, so your last output (None, 15, 145) is 3D.

The error message shows that your target data has dimension (:, 15), rather than the required (:, 15, 145) so you will have to reformat it.


Neural Network Timeseries Modeling with Predictor Variables Many have shown the effectiveness of using neural networks for modeling time series data, and described the transformations required and limitations of such an approach. R's forecast package even implements one approach to this in the nnetar function.

Based on my reading, all of these approaches are for modeling a single outcome variable based on its past observations, but I'm having trouble finding a description of a neural-network-based approach that also incorporates independent predictor variables (a sort of ARIMAx analogue for neural networks). I've found references to Nonlinear autoregressive exogenous models (NARX), which seem like they should be what I'm looking for, but all the reading I've been able to find talks more about using this approach for multi-step-ahead prediction of a univariate series.

Can anyone point me in the right direction on this? For bonus points, does anyone know of an implementation of what I'm looking for in R?

I think this thesis https://dspace.mit.edu/bitstream/handle/1721.1/99565/924315586-MIT.pdf?sequence=1 is a good starting point for building a model for time series forecasting with so called leading indicators and different machine learning models. Basically you need just to training sets where your inputs are past values of the variable you wish to predict plus the additional indicators (i.e. past_value_t-6, past_value_t-5 ..., past_value_t-1, additional_variable_1, additional_variable_2, ..., additional_variable_n) and the output is either single value in future (t_1) or multiple values (t_1, t_2, ... t_n), in case you wish to predict for example the next twelve moths' sales.

It is also possible to use recursive strategy to generate future values for your prediction variable. In this case you generate next value for the prediction value based on your previous predicted value. However, at least based on my experience the first mentioned direct strategy is way more efficient.


Doesn't over(/under)sampling an imbalanced dataset cause issues? I'm reading a lot about how to use different metrics specifically for imbalanced datasets (e.g. two classes present, but 80% of the data is one class) and how to tackle the issue of imbalanced datasets.
One trick is to oversample, so to take more (or even duplicate some) data belonging to the underrepresented class. I've tried this and did achieve better results (before my models would easily just predict a single class for everything, achieving 80% accuracy lol).
However, I was wondering, will this model work well with real-life data? One of the 'laws' of data science/machine learning is that your training data has to have the same/similar attributes as your live data you're intending to use your model on. However, by oversampling, I create a dataset that's 50% one class and 50%  other, as opposed to the "natural", real-life-data having 80% of one class and 20% of the other.
So I guess the question in short is: Will oversampling my imbalanced dataset of 80/20 class distribution to 50/50 class distribution impact the usability of my model for real-life data? Why?

Yes, the classifier will expect the relative class frequencies in operation to be the same as those in the training set.  This means that if you over-sample the minority class in the training set, the classifier is likely to over-predict that class in operational use.
To see why it is best to consider probabilistic classifiers, where the decision is based on the posterior probability of class membership p(C_i|x), but this can be written using Bayes' rule as
$p(C_i|x) = \frac{p(x|C_i)p(C_i)}{p(x)}\qquad$ where $\qquad p(x) = \sum_j p(x|C_j)p(c_j)$,
so we can see that the decision depends on the prior probabilities of the classes, $p(C_i)$, so if the prior probabilities in the training set are different than those in operation, the operational performance of our classifier will be suboptimal, even if it is optimal for the training set conditions.
Some classifiers have a problem learning from imbalanced datasets, so one solution is to oversample the classes to ameliorate this bias in the classifier.  There are to approaches.  The first is to oversample by just the right amount to overcome this (usually unknown) bias and no more, but that is really difficult.  The other approach is to balance the training set and then post-process the output to compensate for the difference in training set and operational priors.  We take the output of the classifier trained on an oversampled dataset and multiply by the ratio of operational and training set prior probabilities,
$q_o(C_i|x) \propto p_t(x|C_i)p_t(C_i) \times \frac{p_o(C_i)}{p_t(C_i} = p_t(x|C_i)p_o(C_i)$
Quantities with the o subscript relate to operational conditions and those wit the t subscript relate to training set conditions.  I have written this as $q_o(C_i|x)$ as it is an un-normalised probability, but it is straight forward to renormalise them by dividing by the sum of $q_o(C_i|x)$ over all classes.  For some problems it may be better to use cross-validation to chose the correction factor, rather than the theoretical value used here, as it depends on the bias in the classifier due to the imbalance.
So in short, for imbalanced datasets, use a probabilistic classifier and oversample (or reweight) to get a balanced dataset, in order to overcome the bias a classifier may have for imbalanced datasets.  Then post-process the output of the classifier so that it doesn't over-predict the minority class in operation.


can I use public pretrained word2vec, and continue train it for domain specific text? I have a set of reviews from apparel domain, about 100K reviews (2M words).
And I want to train word2vec to do some cool NLP staff with it.

However the size is not enough for creating adequate word2vec model, it requires billions of words.

So the idea is to use public corpora (e.g. wikipedia), or even use some pretrained models (e.g. from gensim cool framework) and add my domain specific text.
I assume the model will get aware of unseen-in-public words, and could correct vectors for common word.

Does it make sense ? will the 2M words makes any effect at all ?

fastText pretrained models should give you a boost to classification task.

gensim on the other hand has possibility to load model and train it with new texts but if you need to account for new words, you need to use

  build_vocab(update=True...)

So you can take with fastText pretrained embeddings to gensim and update with your texts.


Association Rules with Python (coded dataset) I have this dataset which I really need to use association rules techniques on. The dataset has like 90 variables, many of which are ordinal. Thing is, the data is already coded using numbers instead of strings (e.g. bread = 4 instead of "bread") as well as some re-scaled numerics such as 1 = 1%-10%".
What I have so far:
from apyori import apriori
#Convert dataframe to list
val_list= []
for row in range(1,5530):
    val_list.append([str(data.values[row,column]) for column in range (0,90)])
    print('Row ', row, ' ok')

apr = apriori(val_list,min_support=0.1,min_confidence=0.2,min_lift=2)
result = list(apr)

Still, this way I don't get the feature names in the frequent "baskets" so it's not much use, since I have baskets like [33, 1, 8, 8, 1, 1] with no idea what the numbers might be referring to. What can I do and/or how do I prepare the data for association rule mining?

Create a dictionary that contains the coded variables as keys and the item names as values.
So it would look like:
dicty = {4: "bread", 7: "milk", 9: "toothpaste"}

Constructing dictionaries in python is really easy if you have them in a table or excel spreadsheet.
dicty = {i:j for i,j in zip(coded_list,normal_list)}

where coded_list is the list of variables in numbers, and normal_list is the list of variables in their categorical names.
Once you have a dictionary you can simply convert them like this:
name = dicty[9]

and it should return toothpaste for name.


Stats degree from 25 years ago: What do I need to learn now to get into Data Science? Having got a bachelors degree in math and a masters degree in statistics back in the mid 90s and then having gotten old and despondent in information technology, I have caught the data science bug, and I'd like to make a switch over the next 2 years.
I have forgotten pretty much all statistics I learned, but I'm sure it'll all come back once I start studying again. I know that I'll have to learn R and/or Python, but real my question is, what statistics do I need to learn and in what order?
I could look at some standard university curriculum, but there is a danger of going crazy and learning all kinds of things that I won't need. So, what books do I really need to read so I don't look stupid on day 1 in a new job, two years from today?

To my mind you have to start from basic understanding of what is Data Science, and what are basic concepts, before going deeper. You could almost be pretty good at data science not knowing a lot on statistic, since the global logic behind what you are doing is often way more important than the exact statistical detail.
I'd really suggest you to start by finding a global point of view, of what global concepts are, and what's the global process behind analysing data, and then you could go more specifically onto many things, like how models work, what are the precise issues you can encounter in your analysis, in your model building, what are different types of variables, how to handle them, what is "parameter tuning", ... but only in details in a second time.
I don't really have books to recommand you, since I just ended my studies, and my different teachers already provided me a lot of information to learn and work on, but to me the most important thing is to first understand global concepts and global logical behind data science, so you're not just applying technical things with making huge logical mistakes in the same time.


Identifying Waveform Segments Using Training Waveforms Problem : There are several events (eventA, eventB,....) represented by waveforms. For each event there are several csv files (eventA1.csv, eventA2.csv,...eventAn.csv) having the points(x,y) from which the corresponding waveform can be generated.

Using these waveforms as training data, waveform segments on (test data) larger data should be identified i.e. the interval during which an event occurs is needed to be labelled on the test data (See image for reference).

What would be the best strategy, tools and algorithms to address this problem?
If classification model is to be generated then,

The points are just to describe the nature of the waveform i.e. the test waveform will not be on same points but with similar nature e.g. the concavity, convexity, slope etc.
Different events in training graphs are represented using different number of points i.e. event A can be determined by 20 points, while event B has 200 points.

So,

How do I create feature vectors to include all these things for classification?
How would the test data be analysed, as it has large number of points on the large waveform i.e. how to segment it to generate features and give it to classification model?

Statistical Data Analysis

I would start with the following:

Statistically analyse your training events and see the mean and the variance of length and width and other properties to have an overall image of what they are.
For each event take the mean and the standard deviation.
Walk through the test data and compute the error between test signal and learned events and plot it.
At the end for each event type, you will have an error curve whit many local minimums.
Choose a threshold based on standard deviation of training events and detect minimums behind that threshold as events.

Pattern Recognition

More advanced option is to learn a model to classify events. At the beginning I should say the last question is theoretically wrong because you do not generate features from test data but only compute them. Features have been extracted (generated) from training data. I assume you mean how to feed the model with test data for classification according to its size. Well, I'd say you can segment it to chunk which may lead to losing some events because of cutting points but if test data is large enough that would not be a problem.

The point is that time-series are a special kind of data and there are methods specifically for them so the classic feature extraction and learning may lead you to an ocean of confusing methods. I suggest to search the term Time-series Mining instead to see more direct solutions.

Time-Series Mining

The very first approach I proposed was kind of this! There are many more algorithms for detecting similarity based on two time-series mostly based on Euclidean distance. For example look at LCSS or DTW. These methods are what you are looking for and both are advanced version of my first suggested algorithm. Please note that for all of them normalizing your training and test data plays an important role.

See this answer as well however that is about unsupervised segmentation of time-series.


BERT : text classification and feature extractionn I have tried multi-label text classification with BERT.

Here is the sample input: $15.00 hour, customer service, open to industries

One of the labels is Billing_rate and prediction score looks quite good.

Now my question is if I want to extract $15.00 hour basically feature value out of BERT. Can you please suggest what are my next step options?

You'd need to apply a tagger, either a generic NE tagger or a custom-trained one. The tagger works with each token as an instance, so that you can extract a particular sequence of tokens, e.g.:

$15.00     Begin_Billing_rate
hour       Billing_rate
customer   _
service    _
,          _
open       _
to         _
industries _

Of course in order to train a custom tagger you will also have to annotate your data token by token.


How is dimensionality reduction achieved in Deep Belief Networks with Restricted Boltzmann Machines? In neural networks and old classification methods, we usually construct an objective function to achieve dimensionality reduction. But Deep Belief Networks (DBN) with Restricted Boltzmann Machines (RBM) learn the data structure through unsupervised learning. How does it achieve dimensionality reduction without knowing the ground truth and constructing an objective function?

As you know, a deep belief network (DBN) is a stack of restricted Boltzmann machines (RBM), so let's look at the RBM: a restricted Boltzmann machines is a generative model, which means it is able to generate samples from the learned probability distribution at the visible units (the input). While training the RBM, you teach it how your input samples are distributed, and the RBM learns how it could generate such samples. It can do so by adjusting the visible and hidden biases, and the weights in between.

The choice of the number of hidden units is completely up to you: if you choose to give it less hidden than visible units, the RBM will try to recreate the probability distribution at the input with only the number of hidden units it has. An that is already the objective: $p(\mathbf{v})$, the probability distribution at the visible units, should be as close as possible to the probability distribution of your data $p(\text{data})$.

To do that, we assign an energy function (both equations taken from A Practical Guide to Training RBMs by G. Hinton)
$$E(\mathbf{v},\mathbf{h}) = -\sum_{i \in \text{visible}} a_i v_i - \sum_{j \in \text{hidden}} b_j h_j - \sum_{i,j} v_i h_j w_{ij}$$
to each configuration of visible units $\mathbf{v}$ and hidden units $\mathbf{h}$. Here, $a_i$ and $b_j$ are the biases, and $w_{ij}$ are the weights. Given this energy function, the probability of a visible vector $\mathbf{v}$ is
$$p(\mathbf{v}) = \frac 1Z \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$
With that, we know that to increase the probability of the RBM generating a training sample $\mathbf{v}^{(k)}$ (denotes the $k$-th training sample), we need to change $a_i$, $b_j$ and $w_{ij}$ so that the energy $E$ for our given $\mathbf{v}^{(k)}$ and the corresponding $\mathbf{h}$ gets lower.


How and Why to rescale image range between [0,1] and [-1,1] I am trying to implement model described in Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network in which author says in section 3.2 that

  We scaled the range of the LR input images
  to [0, 1] and for the HR images to [−1, 1]. The MSE loss
  was thus calculated on images of intensity range [−1, 1]

Now I know that when Gray-scaled images are represented in float it's value is in range [0,1] but how do you scale your colour images in range of [0,1] and [-1,1]? And why authors are using two different scales for input and High resolution images?

Any insights would be helpful.

You can use the following scalings

$$x’=\dfrac{x}{255} \qquad (1)$$
$$x’=\dfrac{x-127.5}{127.5} = \dfrac{x}{127.5}-1 \qquad (2)$$

for rescaling to $[0,1]$ or $[-1,1]$.

The rescaling of inputs tries to keep the range of weights in a small range. Theoretically it is not necessary to rescale your inputs because it can be compensated by an appropriate redefinition of your weights. Practically it is important because your weights might occupy a very large range of values.

In order to understand this I will construct a toy example with two inputs $x_1= 1$ and $x_2=1$ and a simple linear regression $y_n=w_1x_{1n}+w_2x_{2n}$. Assume the true values for the weight are $w_1=1$ and $w_2=1$. Now, assume we have the same data set but with different scales for $x_1$ and $x_2$ such that $x_1=10^k$ and $x_2=10^{-k}$. In order to obtain the same inputs we would need to have inverse weights resulting in $w_1=10^{-k}$ and $w_2=10^k$. Theoretically this is not a big deal but practically we see that for large values of $k$ we will need to use variables that store numerical values between $10^{-k}$ and $10^k$.

I don’t know the reason why the authors of the cited paper used specific transformations. Both should lead to the almost same performance as the bias should be able compensate for the additional $-1$ and the reacaling could be compensated by the weights for each input. There might be differences because of the stochasticity of many optimization techniques and because the final solution is very likely not the global optimum. I would choose one over the other if I want to compare my model with a specific model from a paper. In such a case I would use the same transformation as applied in the paper.

You could also use different scales in order to better discriminate the type of input (low resolution vs high resolution) by looking at the transformed values.


Binary classification algorithm where the input variables are arrays For a project, I'm trying to predict leaks in a network. The network consists of nodes connected by links. What I have are several 'scenarios' where each scenario has a leak present at a different node (it is also possible that there isn't a leak).
I have pressure data for each node, as well as the flow data for all the links. I also have the targets for all the scenarios (0/1, depending on if there's a leak or not). The data can be represented as below:

Scenario
Node 1
Node 2
Output

1
[69.887, 70.383, ...]
[71.875, 72.421, ...]
0

2
[60.358, 61.530, ...]
[76.840, 77.389, ...]
1

In case there is a leak present, during the time the leak exists, the pressure takes a massive dip and the flow shoots up. This is applicable to most nodes and links in the system, depending on where they're located.
Pressure when leak exists:

Flow when leak exists:

My initial idea was to represent all the data at a particular node as a single value using either standard deviation or mean, but that would result in the loss of the data's trends.
I wanted to see how I could possibly train a model while either keeping the data as is or transforming it in some way.
I was considering using a Random Forest classifier for this, but I'm open to suggestions.

A very simple approach is to build a detector with a sliding window on which you compute standard deviation. For each slide compute the mean and you can test if the next point after the window is far from mean more than a number of standard deviations. This should work because if the window is not very large it will adapt to the local mean and also if the functions are relatively constant on windows and spikes are large, as they look from your graphs


logistic like curve fitting using machine learning I have a set of points of a function k(x). I am trying to do some curve fitting to find the exact k(x) function. It seems that the data points fit to a logistic like curve only a little shifted and stressed.

So far I have tried polynomial regression, but I don't feel the fitting is correct. I have attached a snap of the fitted curve here.

So my question is, is logistic regression only used in classification tasks? Or can it be used for curve fitting?

If not what are the other available techniques to fit a logistic like curve to a set of data points?

So my question is, is logistic regression only used in classification tasks?

Name Logistic regression indicates that we could use this algorithm for regression problems.

  Or can it be used for curve fitting?

It could be used to curve fitting.

  If not what are the other available techniques to fit a logistic like curve to a set of data points?

Try numpy or scipy:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

https://lmfit.github.io/lmfit-py/model.html


Career advice for a data scientist - Am i in the right place? I need a career advice from you.

I'm 26 and working as a data scientist for less than 2 years. I have a statistics and economics background mainly. My tech skills can be summed up by a good knowledge of SQL,Python and R.

My first experience was a 6 month internship for a software house company that aims at building product for banks and financial institution. Here I worked with other 2 unexperienced data scientist, all with a statistics background like me.

I left and currently I'm working in a big company, one of the major wholesaler in Europe.

The problem I'm struggling with , it's that I'm the only data scientist in the company and in a digital marketing dept, so with non tech people.

After almost a year, I'm quite tired of working alone and i don't feel I have been growing much professionally.

A thought I had a while ago, was to ask a temporary shift to Data Wharehouse department in order to acquire more Database related knowledge.
Another idea was obviously to ask for another data scientist.

My doubt can be summed up like this:

Regarding  first idea: As a data scientist, what DB knowledge should i master? I'm quite good with SQL, mainly to perform data extraction and also create some simple table structure. It could be benificial to know more on this topic, for example data migration, ETL procedure, etc etc? I'm planning to learn more about Hadoop/Spark/Hive/Cassandra, but we don't have this tech stack at work. Just an oracle db.
For budget reason, I'm aware that if they'r gonna hire a new data scientist it couldn't be someone with much of experience. In this case it's very unclear to me if I could benefit from working with another unexperienced ds or i should prefer the first option.

Thanks for any opinion

Welcome to the site! The only thing I can really do is offer general advice:

1 - Realize that your situation is not unique. There are many, many data science departments that just contain 1 or 2 people at most and many of us work alone - it's what I call being a "one man band" in data science. I work for one of the largest corporations in the world and I work alone because (1) budgets are always hard and (2) good data scientists are very, very hard to find. Oh, you'll find resumes out there, but most people will fail even the most basic tech screenings and just be awful at data science. You should take pride in being the person that your company picked and realize that just because you leave your current company it doesn't mean that your "alone" situation will improve.

2 - So, how can you turn your situation into a positive? If they don't have the budget to get someone senior, they might still have the budget to train you better. Get them to budget you money to go to conferences, get all the books you can, sign up for online courses that your company is willing to pay for. Even if they can't give you help, they can still get you resources in other ways. That's the advantage of being alone, your "department" doesn't need a big training budget, just enough to fly you to a couple of conferences each year.

3 - The beauty of being a data scientist is that there are always skills you can work on. How's your software deployment skills? What about your skills around building data pipelines? Do you tend to over-rely on one model instead of learning news one (that happens to all of us). There's always new parts you can be adding to your toolkit. In that sense, adding some DB skills is a good idea; work on both DB design and how to create the pipelines from those DBs into your programs - that's a very valuable skill.

Good luck!


Semantic network using word2vec I have thousands of headlines and I would like to build a semantic network using word2vec, specifically google news file.
My sentences look like
    Titles
    Dogs are humans’ best friends
    A dog died because of an accident
    You can clean dogs’ paws using natural products.
    A cat was found in the kitchen

And so on.
What I would like to do is finding some specific pattern within this data, e.g. similarity in topics on dogs and cats, using semantic networks.
Could you give me some advice on how I can do it?
Code:
import pandas as pd
import gensim
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.manifold import TSNE

main_data.Titles = np.where(main_data.Titles.isnull(),'NA', main_data.Titles)

article_titles = main_data['Titles']

titles_list = [title for title in article_titles]

big_title_string = ' '.join(titles_list)

tokens = word_tokenize(big_title_string)

words = [word.lower() for word in tokens if word.isalpha()]

stop_words = set(stopwords.words('english'))

words = [word for word in words if not word in stop_words]

model = gensim.models.KeyedVectors.load_word2vec_format('path/GoogleNews-vectors-negative300.bin', binary = True)

model.vector_size

vector_list = [model[word] for word in words if word in model.vocab]

words_filtered = [word for word in words if word in model.vocab]

word_vec_zip = zip(words_filtered, vector_list)

word_vec_dict = dict(word_vec_zip)
df = pd.DataFrame.from_dict(word_vec_dict, orient='index')

tsne = TSNE(n_components = 2, init = 'random', random_state = 10, perplexity = 100)

tsne_df = tsne.fit_transform(df[:400])

sns.set()
fig, ax = plt.subplots(figsize = (11.7, 8.27))
sns.scatterplot(tsne_df[:, 0], tsne_df[:, 1], alpha = 0.5)

from adjustText import adjust_text
texts = []
words_to_plot = list(np.arange(0, 400, 10))

for word in words_to_plot:
    texts.append(plt.text(tsne_df[word, 0], tsne_df[word, 1], df.index[word], fontsize = 14))

adjust_text(texts, force_points = 0.4, force_text = 0.4,
            expand_points = (2,1), expand_text = (1,2),
            arrowprops = dict(arrowstyle = "-", color = 'black', lw = 0.5))

plt.show()

However I cannot understand how to interpret the results. I think they are wrong and probably this is not a good approach for building a semantic network. maybe I have been missing something...For instance, this code is still keeping stopwords after the part of
words = [word for word in words if not word in stop_words]

This is an example of output difficult to read and explain (at least, for me):

I would greatly appreciate if you could give me some tips and advice on how to perform a semantic network which can show semantic similarity within titles.

You can try converting your word representation into a document representation by simply taking the average over the word vectors for a document. For example, if a document has 9 words with (9, 200) dimensions, by taking an average over the words you can have a document representation with a dimension (1,200).
After you have your document-representation, you can use T-SNE to find similar documents. The documents with a similar topic or theme will cluster near each other. You can always improve the document representation by improving your word vectors.
Check this