Predicting parameters of simple configured trajectories using RNN What I'd like to do: predict orbital elements given an input observation sequence in 2D, that is

$$input = X = [position_{t_{0}}, position_{t_{1}},\ ..., position_{T}]$$
$$output = y = parameters\ of\ observed\ orbit$$

Positions are (x, y) coordinates and the parameters are a handful of values that describe an ellipse in a 2D plane.

What I've done so far: I used a very, very basic LSTM-stack model with Keras that I trained with $([X_{o_{i}}^{1},X_{o_{i}}^{2},\ ...], y_{o_{i}})$ pairs where $o_{i}$ are the 2D orbits. $[X_{o_{i}}^{1},X_{o_{i}}^{2},\ ...]$ is just a couple dozens of simulated observations of $o_{i}$. I assumed it is a regression problem so I replicated some time series prediction models I found on the internet: those usually take a two or three dimensional vector as input sequence and give a 1D value prediction of the next step. I also Googled configured trajectory LSTM or shape parameter prediction RNN.
The Issue: Results are bad, my ellipses do not have the expected shape. I have absolutely no idea if this is the RNN I should be using and if it is, what layers I should or shouldn't add. I couldn't find relevant literature on the internet and pages that tackled configured trajectories were talking about predicting the next position of a car, while in my case I'm trying to compute an estimation of the whole trajectory.
What I'm looking for: Really anything related to determining geometrical parameters based on sequential observation or relevant literature about how to design models that use LSTM for this kind of problems. Any advice from LSTM-RNN-savvy people or orbital-element-estimation-specialists would be greatly appreciated.

If you know that your trajectory has a certain parametric form then you can use methods that explore the parameter space for that form. Examples of such methods are Hough transform and custom-built moments.
Hough transform maps a point in a real space into a manifold in the parameter space, and vice-versa, it maps a point in the parameter space into a line in the real space. If you have several points in the real space, they are transformed into the same amount of manifolds in the parameter space. These manifolds intersect in a point whose coordinates are the parameters of the line that goes through your data.
The parameter space for the ellipse is five-dimensional, so a naive application of the Hough transform is inefficient. Smarter implementations are known, e.g. [1].
The method of custom-built moments described in [2] allows to find the parameters of a curve expressible as
$$G(x,y) = \sum\limits_np_nz_n(x,y)=0$$
In the case of an ellipse, we have
$$G(x,y)=p_1+p_2x+p_3y+p_4x^2+p_5y^2+p_6xy=0$$
that is $z_1=1$, $z_2=x$, $z_3=y$, $z_4=x^2$, $z_5=y^2$, $z_6=xy$. You want to determine $p_1,...,p_6$ based on your data.
Enclose the portion of the curve where you have your data in a rectangle, and make a coordinate transformation so that your rectangle is now $ S = [-1 \le x \le 1, -1 \le y \le 1]$. The rectangle is separated by the curve into two regions. Assign the value $1$ to all points from one of the regions, and $0$ to all points in the other region. For example, in this Figure, all black points are $1$, and white points are $0$.

The data points are on the boundary. They are not explicitly shown in the figure because their exact position is not important anymore. It is also not important, how many points you have (as long as they clearly determine the shape of the curve segment) and in which order you collect them.
So, now you have a function defined on the rectangle:
$$I(x,y)=\begin{cases}
      1 & (x,y) \in Region~1 \\
      0 & (x,y) \in Region~2
\end{cases}$$
The parameters $p_1,...,p_6$ are found as the solution of a system of equations
$$\sum\limits_nu_np_n=0$$
where $u_n$ are custom moments
$$u_n=\iint_S I(x,y)U_n(x,y) \,dx\,dy$$
The functions $U_n$ are defined as $U_n(x,y)=\Upsilon(F,s,z_n)$ where
$$\Upsilon(F,s,z)=\left([sz_y-zs_y]F\right)_x-\left([sz_x-zs_x]F\right)_y$$
with the $x$ and $y$ subscripts indicating partial differentiation. The functions $F(x,y)$ and $s(x,y)$ are arbitrary. They can be anything you like as long as $F$ is  continuously differentiable, $s$ is twice continuously differentiable on an open set containing the rectangle $S$, and $F=0$ on the boundaries of $S$. More details in [2]. If you don't understand that paper during the first reading, read [3] first, and if you don't understand [3], read [4], then read [3], after which [2] should become clearer.
For ellipses, the authors of [2] recommend to generate 18 arrays of functions $U_n = (U_1,...,U_6)$ from combinations of three functions $F_1 = m_0n_0$, $F_2=xF_1$, $F_3=yF_1$ with six functions $s_n=z_n,\, n=1,...,6$, where $m_0=1-x^2$, $n_0=1-y^2$. With the additional notations $m_1=1-2x^2$, $n_1=1-2y^2$, $m_2=1-3x^2$, $n_2=1-3y^2$, these 18 arrays are found to be

$F=F_1,~s=1\Rightarrow % 1
U_1=0,~
U_2=2ym_0,~
U_3=-2xn_0,~
U_4=4xym_0,~
U_5=-4xyn_0,~
U_6=2(y^2-x^2)
$
$F=F_1,~s=x\Rightarrow % 2
U_1=-2ym_0,~
U_2=0,~
U_3=2y^2(3x^2-2)+2m_1,~
U_4=2x^2ym_0,~
U_5=2y(y^2(5x^2-3)+2m_1),~
U_6=2xm_1n_0
$
$F=F_1,~s=y\Rightarrow % 3
U_1=2xn_0,~
U_2=2(y^2(2-3x^2)-m_1),~
U_3=0,~
U_4=2x(y^2(4-5x^2)+3x^2-2),~
U_5=-2xy^2n_0,~
U_6=-2ym_0n_1
$
$F=F_1,~s=x^2\Rightarrow % 4
U_1=-4xym_0,~
U_2=-2x^2ym_0,~
U_3=2x(y^2(5x^2-4)-3x^2+2),~
U_4=0,~
U_5=4xy(y^2(4x^2-3)+2-3x^2),~
U_6=2x^2(y^2(4x^2-3)+2-3x^2)
$
$F=F_1,~s=y^2\Rightarrow % 5
U_1=4xyn_0,~
U_2=2y(y^2(3-5x^2)-2m_1),~
U_3=2xy^2n_0,~
U_4=4xy(y^2(3-4x^2)+3x^2-2),~
U_5=0,~
U_6=2y^2(y^2(3-4x^2)+3x^2-2)
$
$F=F_1,~s=xy\Rightarrow % 6
U_1=2(x^2-y^2),~
U_2=-2xm_1n_0,~
U_3=2ym_0n_1,~
U_4=2x^2(y^2(3-4x^2)+3x^2-2),~
U_5=2y^2(y^2(4x^2-3)+2-3x^2),~
U_6=0
$
$F=F_2,~s=1\Rightarrow % 7
U_1=0,~
U_2=2xym_0,~
U_3=m_2n_0,~
U_4=4x^2ym_0,~
U_5=2ym_2n_0,~
U_6=x(y^2(x^2+1)+m_2)
$
$F=F_2,~s=x\Rightarrow % 8
U_1=-2xym_0,~
U_2=0,~
U_3=x(y^2(7x^2-5)-5x^2+3),~
U_4=2x^3ym_0,~
U_5=2xy(2y^2(3x^2-2)+3-5x^2),~
U_6=x^2(3-5x^2)n_0
$
$F=F_2,~s=y\Rightarrow % 9
U_1=-m_2n_0,~
U_2=x(y^2(5-7x^2)+5x^2-3),~
U_3=0,~
U_4=x^2(y^2(9-11x^2)+7x^2-5),~
U_5=y^2m_2n_0,~
U_6=-2xym_0n_1
$
$F=F_2,~s=x^2\Rightarrow % 10
U_1=-4x^2ym_0,~
U_2=-2x^3ym_0,~
U_3=x^2(y^2(11x^2-9)-7x^2+5),~
U_4=0,~
U_5=2x^2y(y^2(9x^2-7)+5-7x^2),~
U_6=x^3(y^2(9x^2-7)-7x^2+5)
$
$F=F_2,~s=y^2\Rightarrow % 11
U_1=-2ym_2n_0,~
U_2=2xy(2y^2(2-3x^2)+5x^2-3),~
U_3=-y^2m_2n_0,~
U_4=2x^2y(y^2(7-9x^2)+7x^2-5),~
U_5=0,~
U_6=xy^2(y^2(7-9x^2)+7x^2-5)
$
$F=F_2,~s=xy\Rightarrow % 12
U_1=-x(m_2+y^2(x^2+1)),~
U_2=x^2(5x^2-3)n_0,~
U_3=2xym_0n_1,~
U_4=x^3(y^2(7-9x^2)+7x^2-5),~
U_5=xy^2(y^2(9x^2-7)+5-7x^2),~
U_6=0
$
$F=F_3,~s=1\Rightarrow % 13
U_1=0,~
U_2=-m_0n_2,~
U_3=-2xyn_0,~
U_4=-2xm_0n_2,~
U_5=-4xy^2n_0,~
U_6=y(y^2(3-x^2)-x^2-1)
$
$F=F_3,~s=x\Rightarrow % 14
U_1=m_0n_2,~
U_2=0,~
U_3=y(y^2(7x^2-5)+3-5x^2),~
U_4=-x^2m_0n_2,~
U_5=y^2(y^2(11x^2-7)+5-9x^2),~
U_6=2xym_1n_0
$
$F=F_3,~s=y\Rightarrow % 15
U_1=2xyn_0,~
U_2=y((5-7x^2)y^2+5x^2-3),~
U_3=0,~
U_4=2xy(y^2(5-6x^2)+4x^2-3),~
U_5=-2xy^3n_0,~
U_6=y^2m_0(5y^2-3)
$
$F=F_3,~s=x^2\Rightarrow % 16
U_1=2xm_0n_2,~
U_2=x^2m_0n_2,~
U_3=2xy((6x^2-5)y^2+3-4x^2),~
U_4=0,~
U_5=2xy^2(y^2(9x^2-7)+5-7x^2),~
U_6=x^2y(y^2(9x^2-7)+5-7x^2)
$
$F=F_3,~s=y^2\Rightarrow % 17
U_1=4xy^2n_0,~
U_2=y^2(y^2(7-11x^2)+9x^2-5),~
U_3=2xy^3n_0,~
U_4=2xy^2((7-9x^2)y^2+7x^2-5),~
U_5=0,~
U_6=y^3(y^2(7-9x^2)+7x^2-5)
$
$F=F_3,~s=xy\Rightarrow % 18
U_1=y((x^2-3)y^2+x^2+1),~
U_2=-2xym_1n_0,~
U_3=y^2m_0(3-5y^2),~
U_4=x^2y((7-9x^2)y^2+7x^2-5),~
U_5=y^3((9x^2-7)y^2+5-7x^2),~
U_6=0
$

After taking the integrals of these $U_n$ with $I(x,y)$ you get a $18 \times 6$ matrix $U$ of the moments $u_n$. Taking the eigenvector associated with the smallest eigenvalue of $U^TU$ as described here, you will get the parameters $p_1, ... p_6$.
I tested them on the curve segment in the figure above. The segment is from an ellipse
$$\frac{((x-1)\cos(\frac{\pi}{6})+(y+1)\sin(\frac{\pi}{6}))^2}{2^2} + \frac{((x-1)\sin(\frac{\pi}{6})-(y+1)\cos(\frac{\pi}{6}))^2}{1^2} = 1$$
I got the following result
[ 1.         -1.70012217  2.53636068  0.49314339  0.91927365 -0.74693632]

which is not very accurate but I used a 2D-trapezoid rule for integration as described here and took a uniform $600 \times 600$ grid. You will definitely get better results with a more sophisticated grid and integration rule.
References

D.H. Ballard, Generalizing the Hough transform to detect arbitrary shapes, Pattern Recognition, Volume 13, Issue 2, 1981, Pages 111-122, ISSN 0031-3203,
https://doi.org/10.1016/0031-3203(81)90009-1
Popovici, I., & Withers, W. D. (2009). Curve Parametrization by Moments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 15–26. https://doi.org/10.1109/TPAMI.2008.54
Popovici, I., & Withers, W. D. (2006). Locating Thin Lines and Roof Edges by Custom-Built Moments. 2006 International Conference on Image Processing. https://doi.org/10.1109/ICIP.2006.312501
Popovici, I., & Withers, W. D. (2006). Custom-built moments for edge location. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 637–642. https://doi.org/10.1109/TPAMI.2006.75


Looking for binary class datasets with high class imbalance, that also have intra-class imbalance in the minority class Newbie question alert...
For a college project I want to compare a few variants of SMOTE in terms of how much they improve classification of the minority class, over using random oversampling.
I have a specific interest in the idea that the minority class may contain small disjuncts that may themselves exhibit imbalance within the class.
I am already looking at the credit card fraud dataset on Kaggle (https://www.kaggle.com/mlg-ulb/creditcardfraud)
Can anyone please point me towards other datasets that have the following kinds of properties:

a reasonably large number of examples (ideally at least a few thousand)
have only two class labels
are highly imbalanced, i.e. the minority class is severely under-represented
ideally the minority examples would have some intra-class imbalance too

Or even better, is there any kind of good search tool out there for finding datasets based on these kinds of characteristics?

The imblearn.datasets package (documentation is here) has a function called fetch_datasets() which is described as:

fetch_datasets allows to fetch 27 datasets which are imbalanced and binarized

I do not know them in sufficient depth to know whether they meet all four of the criteria you've listed, but these may be a good first place to start with.


Why exactly using a test set for model evaluation is a bad idea? I don't understand why using the test set for model evaluation is a bad idea.

I completely understand why you should not use your test set to train your model (because in that case, you would be memorizing and you just cannot tell whether your model will generalize well or not if you don't have a separate test set). But why is it that simply using your test set to test (not train) your model is bad? You won't be changing any parameters of the model (because you are not training).

For instance, at the end of this video, Luis says we are breaking what he calls the "Golden rule" (i.e. never use your testing data for training). However, all I can see he is doing is using the test set to verify which model performs better to then be able to make a selection on which model he will use in the end.

In the linked video, a polynomial is fitted to some data. The degree of the polynomial has to be chosen, but it cannot be trained. It is therefore a hyperparameter. In order to choose the degree, several instances of the algorithm, each one trained with a polynomial of different degree, have to be evaluated on a data set that was not used for training. However, if we used the test set for assessing the values of hyperparameters, we would also be coupling the resulting model to the data in the test set, as the chosen hyperparameter (and therefore the whole model) would depend on the test set data.

The test set should only be used for the final evaluation in order to know how your algorithm is going to behave on data that has never been seen before by it. This is why there is normally a validation set used to tune the hyperparameters.


How to cluster government census data in order to group Metropolitan statistical areas I have collected a bunch of census data from 2012 - 2018. I wanted to apply some clustering algorithms in order to compare Metropolitan statistical area (MSA's). Ideally once I run the clustering algorithm I would like to see which MSA is comparable to another.
The features that I am choosing to govern the clustering is below:
'Bachelors+',
'Estimate  Total  $10,000 to $14,999',
'Estimate  Total  $100,000 to $124,999',
'Estimate  Total  $125,000 to $149,999',
'Estimate  Total  $15,000 to $19,999',
'Estimate  Total  $150,000 to $199,999',
'Estimate  Total  $20,000 to $24,999',
'Estimate  Total  $200,000 or more',
'Estimate  Total  $25,000 to $29,999',
'Estimate  Total  $30,000 to $34,999',
'Estimate  Total  $75,000 to $99,999',
'Median Age',
'Median Gross rent as % of household inc',
'Number of educational and health service workers',
'Number of finance and real estate workers',
'Number of people in management, business, science, and arts',
'Number of service workers',
'Number of tech workers',
'Pct Asian',
'Pct Black',
'Pct Other Race',
'Pct White',
'Total Population',
'Total Population over 25'

Now a question I have is the data I have is on the tract level for every MSA in the United States from 2012 - 2018. Would I first need to aggregate the data so that I have the above features by their associated MSA then do the clustering algorithm from there?
From there how do I identify the MSAs by cluster?

If you want to identify the distance between MSAs. Then yes, I think it would be best to first aggregate your features such that each instance (row) represents an MSA. From there you will have an $n\times m$ matrix where $n$ is the number of MSA, and $m$ is the number of features you end up with.
You can then apply your clustering algorithm, there are many to choose from, among my favorites I always try are:

K-means
K-nearest neighbors
Spectral clustering
DBSCAN

Others can be found here.
Once you train the clustering algorithm then you will get an associated cluster values for each of the $n$ rows in your input matrix. With this you will know what MSAs are similar in nature given the selected set of features.


Are t-sne dimensions meaningful? Are there any meanings for the dimensions of a t-sne embedding? Like with PCA we have this sense of linearly transformed variance maximizations but for t-sne is there intuition besides just the space we define for mapping and minimization of the KL-distance?

The dimensions of the low dimensional space have no meaning. Note that the t-SNE loss function is solely based on the distances between points ($y_i$ and $y_j$) and probability distributions over those distances ($p_{ij}$ and $q_{ij}$):

$$
\frac{\delta C}{\delta y_i}=4 \sum_j(p_{ij} - q_{ij})(y_i-y_j)(1+||y_i -y_j||^2)^{-1}
$$

Thus there is no projection from the whole high-dimensional space to the low-dimensional space, t-SNE only finds a mapping from a specific set of high-dimensional points to a specific set of low dimensional points. Because there is no function from one space to the other there is also no inherent meaning of the axes.

Things you can imagine to illustrate this:

Rotating or translating the high-dimensional or low-dimensional space does not influences distances between the points. Hence, t-SNE does not care about rotation or translation in both spaces. Thus there is not absolute interpretation of the axes.
The t-Student distribution has fat tails. This causes the low-dimensional representation to be invariant to changes in points that are far away in the high-dimensional space. This also causes that points that are far away in the high-dimensional space can be either reasonably far away, far away or really far away in the low dimensional space. In this sense it stretches certain parts of the low-dimensional axes (in any arbitrary direction).

That being said, t-SNE is primarily a visualization technique and its dimension reduction effectiveness for other purpose is not obvious (probably not suitable for clustering, feature extraction or feature selection).

Also: the paper.


Prepping Data For Usage Clustering Dataset: I'm given the number of minutes individual customers use a product each day and am trying to cluster this data in order to find common usage patterns.

My question: How can I format the data so that, for example, a power user with high levels of use for a year looks the same as a different power user who has only been able to use the device for a month before I ended data collection?

So far I've turned each customer into an array where each cell is the number of minutes used that day. This array starts when the user first uses the product and ends after the user's first year of use. All entries in the cells must be double values (e.x. 200.0 minutes used) for the clustering model. I've considered either setting all cells/days after the last day of data collection to either -1.0 or NULL. Are either of these a valid approach? If not what would you suggest?

I believe your problem boils down to clustering time-series of different lengths. According to your question, you want the longer time-series of a power user to be considered similar to time-series of similar pattern but much shorter.
Therefore you should look into clustering techniques and distance metrics which allow for these properties. I don't know your language of choice but here are some of the many packages in R that you might find interesting :
- Fréchet distance - one of the packages offering this is kmlShape
- Dynamic Time Warping included in base R
- Permutation Distribution Clustering - package pdc
This would also solve your data formatting problem as to setting values to -1 or NULL would not be needed anymore. hth.


Cost function for Ordinal Regression using neural networks What is the best cost function to train a neural network to perform ordinal regression,  i.e. to predict a result whose value exists on an arbitrary scale where only the relative ordering between different values is significant (e.g: to predict which product size a customer will order: 'small' (coded as 0), 'medium'(coded as 1), 'large' (coded as 2) or 'extra-large'(coded as 3))? I'm trying to figure out if there are better alternatives than quadratic loss (modeling the problem as an 'vanilla' regression) or cross-entropy loss (modeling the problem as classification).

Another approach was suggested in this paper for face age estimation:
https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf
The authors use a number of binary classifiers predicting whether a data point is larger than a threshold, and do this for multiple thresholds. I.e. in your case the network would have three binary outputs corresponding to

larger than 0
larger than 1
larger than 2.

For example, for 'large (2)' the ground-truth would be [1 1 0]. The final cost function is a weighted sum of the individual cross-entropy cost functions for each binary classifier.
This has the advantage of inherently weighting larger errors more because more of the individual cost-entropy terms will be violated. Simply doing categorical classification of the ordered outcomes doesn't inherently have this feature.


Forward propagation in complicated neural networks In a typical neural network you have a certain number of neurons in each layer, all neurons have incoming connections from each neuron in the previous layer (and visa-versa). This I have no problem figuring out how to propagate.

However when you have a network that doesn't have the neurons in layers (I'm looking into the NEAT algorithm); where neurons are just connected based on genetic algorithm (random creation - then testing of fitness) there isn't a simple way of figuring how to forward propagate, for example;

1 input neuron - connected to neurons 2 and 3
Neuron 2 is also connected to neuron 3
Neuron 3 is connected to the output.

Now in this case, if I just go through numerically then everything works out just fine. However with this subtle change;

1 input neuron - connected to neurons 2 and 3
Neuron **3** is also connected to neuron **2**
Neuron **2** is connected to the output.

The going through; 1, 2, then 3 would mean neuron 2 doesn't include the input from neuron 3. I can see that there's a brute force way through this, but is there a better, more efficient way?

You can do topological sorting. For example, here is some pythonic pseudocode

def order(current_neuron, level):
    current_neuron.level = max(level, current_neuron.level)
    for child in sorted(current_neuron.children, key=children level, reverse=True):
        order(child, level + 1)

for input_neuron in input_neurons:
    order(input_neuron, level=0)

(Take this pseudo code with a grain of salt. This is just what came directly to my mind; it is not tested.)

Then you can calculate the activations in order of node.level.


Dwell Time computation from Real-Time user location We have a collection of real time user locations from our application. Each record contains longitude, latitude and the time at which the geo location was captured. How can I compute the dwell time at an user's stationary place from this data?

We have a collection of real time user locations from our application. Each record contains longitude, latitude and the time at which the geo location was captured. How can I compute the dwell time at an user's stationary place from this data?

Things to consider when collecting the data:

If you are using a Library such as: react-native-mauron85-background-geolocation and the Ionic Framework you would set the stationaryRadius to define an area within which the device reporting its location is considered to be stationary.

When there are obstructions and reflecting surfaces multi-path reception is likely. The signals arrive at the receiver at slightly different times and, depending on which signal the receiver chooses to believe for any given sample, the distance from the satellite also differs. Multiply this behavior by several satellites, then factor it into the overall position calculation, and the net result is that the receiver perceives abrupt shifts in position. Sometimes these can be significant - say 40 or 50 feet.

Higher level software in the receiver tries to smooth this out. Generally speaking, it does a better job of this when the tracker is moving. This is because your true change in position per unit time is much larger than the random shifts caused by multi-path reception.

You can average the reported position over a period of time to get a circle within which the device is likely to be. Using secondary information like IP address and WiFi positioning can provide additional information to narrow down the guess faster. See also: Wireless Technologies.

Here is an example that shows you where you are using Mozilla's Geolocation API, click on "Geolocation Live Example" and give permission to share your location (not with us at SE, with Mozilla).
Once you have fairly accurate data it's simply a matter of defining a radius and minimum dwell time (which would be a moving average).

If they are moving fast then a shorter stay is a dwell, while at a slower speed a dwell wouldn't occur until a longer period of time.

Advanced mathematical modeling is necessary to analyze a better result and not get confused by meandering and backtracking.


Why do Isolation Forest implementations turn it into a supervised learning problem (with random values for the target?) I am looking at various implementations of the Isolation Forest in python and R.
Both sklearn in python and solitude in R use a y variable with the ExtraTrees regressor.
Since, Isolation Forest is unsupervised, I am wondering why it is being turned into a supervised problem? Wouldnt this be an issue when scoring on previously unseen data sets?
For example sklearn (python) line 248 has this.
And in solitude line 144 as well.

Extra-random Trees needs a target variable, so Isolation Forest generates a random target (sklearn, solitude).  At prediction time, no y values are used, and the ExtraTrees doesn't actually make a prediction; instead, the samples are propagated to the leaves and the depth is extracted (sklearn).
As for the tree-building process, sklearn at least doesn't make use of the y values, because the ExtraTrees model has max_features=1 and splitter='random' (source).  I'm not so sure about solitude, since it has mtry=ncol-1 (source); maybe further down, using splitrule='extratrees' takes care of that?  Otherwise, the splits chosen will try to optimize on the random y, though since those are random it maybe doesn't matter (certainly I wouldn't call it a supervised model, anyway).


R Studio - grepl compare a column in a dataframe to a list of pattern I have a column named "MATCH" in a dataframe and a list of patterns named "PATTERN".

df1.MATCH <- c("ABC", "abc" ,"BCD")
df1 <- as.data.frame(df1.MATCH)
df2.PATTERN <- c("ABC", "abc", "ABC abc")

I want to use grepl to compare MATCH column with PATTERN, if true, I will apply my functions. The desired result would be "ABC" matches "ABC" and "ABC abc". This is the code I used:

df1 %>% filter(grepl(df1.MATCH,df2.PATTERN ))%>% ...

I get error:

"Warning message: In grepl(TXN_GROUP, parm[3]) :argument 'pattern' has length > 1 and only the first element will be used"

I understand I can't use grepl to a list of vectors. Is there any way to solve it?

TL;DR: grepl expects its first argument to be a string (length 1), not a vector. You can solve this with combinations of sapply and lapply (see below), but you are better served using a single regular expression that captures what you want to match in df1.MATCH and not use df2.PATTERN at all. This second option is much faster (if less intelligle) for a large data set. For this type of work, it is worth learning how to use regular expressions to their full potential.

df1 %>% filter(grepl(pattern = "^((ABC)( )*)+$", x = df1.MATCH, ignore.case = TRUE))

Explanation

The documentation for grepl shows the following usage:

grepl(pattern, x, ignore.case = FALSE, perl = FALSE,
  fixed = FALSE, useBytes = FALSE)

The pattern argument is first, and this argument should be a string (one element). You are providing df1.MATCH to this argument, which is a vector.

We could use sapply to apply grepl to each element of df1.MATCH.

sapply(df1.MATCH, grepl, x = df2.PATTERN)
       ABC   abc   BCD
[1,]  TRUE FALSE FALSE
[2,] FALSE  TRUE FALSE
[3,]  TRUE  TRUE FALSE

However, look at the output! You probably did not want a matrix. What happens when we run your grepl one just the first element of df1.MATCH?

grepl("ABC",df2.PATTERN)
[1]  TRUE FALSE  TRUE

We get a vector because grepl is checking ABC against each element of df2.PATTERN. To get a useful logical vector for filtering, you need to return a logical vector of the same length as df1.MATCH. I see two ways to do it.

  Method 1: Use any

Since you want to know which elements in df1.MATCH match any elements in df2.PATTERN, you can use any, which returns TRUE if any element in its arguments is TRUE. We need a little bit different syntax to make this work. We need to wrap grepl in lapply to make a list of three vectors (one for each element in df1.MATCH1) that feeds into sapply wrapped any. If we just use sapply, any will only return one value since we have a matrix input.

any(grepl("ABC", df2.PATTERN))
[1]  TRUE

sapply(
  lapply(df1.MATCH, grepl, x = df2.MATCH),
  any)
[1]  TRUE  TRUE FALSE

  Method 2: Write a better regular expression.

You want to match the contents of df1.MATCH against possible values that look like abc, ABC, ABC ABC, or ABC abc, etc. You can encompass all of this in a single regex string. The string you want is

"^((ABC)( )*)+$"
^                   # Nothing else before this
(ABC)               # Must contain ABC together as a group
( )*                # followed by any number of spaces (including 0)
((ABC)( )*)+        # Look for the ABC (space) pattern repeated one or more times
$                   # And nothing else after it

Then use grepl with ignore.case = TRUE:

grepl("^((ABC)( )*)+$", df1.MATCH, ignore.case = TRUE)
[1]  TRUE  TRUE  FALSE

  Benchmarking

In a large dataset, one of these will perform faster. Let's find out. Your benchmark results will vary by your machine's resources.

df1.MATCH <- sample(c("ABC", "abc" ,"BCD"), size = 100000, replace = TRUE)
df1 <- data.frame(df1.MATCH)
df2.PATTERN <- c("ABC", "abc", "ABC abc")

library(rbenchmark)

benchmark("any lapply" = {
              df1 %>%
                filter(sapply(lapply(df1.MATCH, grepl, x=df2.PATTERN), any) )
          },
         "better regex" = {
              df1 %>%
                filter(grepl("^((ABC)( )*)+$", df1.MATCH, ignore.case = TRUE))
          }
          )

          test replications elapsed relative user.self sys.self user.child sys.child
1   any lapply          100  149.13   70.678    147.67     0.39         NA        NA
2 better regex          100    2.11    1.000      2.10     0.02         NA        NA

It looks like the improved regex method is significantly faster. That's because it is performing only one operation per row (grepl) before filtering. The other method is performing four operations per row: lapply is performing grepl three times (one for each element of df2.PATTERN, and sapply then performs any for each list element (each row).


Issues with NLTK lemmatizer (WordNet) I want to lemmatize set of plural keywords automatically such as 'Web based technologies', 'Information systems' etc. I want to transform them to to 'Web based technology', 'Information system' respectively.

I tried NLTK as follows

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print(lemmatizer.lemmatize("Web based technologies"))

Even though this perform fine for words such as 'cats' to 'cat', for the keywords I have mentioned I get the same plural form. Any idea how to solve this? Or are there any other tools/APIs that I can make use of?

P.S. Given the keywords I only want to get the singular term of it

As far as I know the nltk lemmatizer works on words or rather ngrams. Your example is a trigram, an easier way to work through this is:

word="web based technologies"
splits=word.split()
word=" ".join(lemmatizer.lemmatize(w) for w in splits)


Test for heteroscedasticity in time series I want to test heteroscedasticity in time series. The tools in python like: statsmodels.stats.diagnostic.het_breuschpagan require residuals as input obtained by fitting model to data. Since this kind of test relies on the goodness of the model trained. I want to test the heteroscedasticity on time series without training any model, directly on the data itself.  So I use McLeod.Li test in R to test it on raw time series. I analyzed individual features have heteroscedasticity. To remove heteroscedasticity, I added 1 to all entries in data (since it has 0 entries) and computed heteroscedasticity the pvalues moved to 0. Why?

You can use the same tests on the raw time series. You are not required to use it on the residuals. That is just what these tests are usually used for (if you are interested in the quality of your model).

However, heteroscedasticity is most easily identified by visual inspection.


Why are mini-batches degrading my conv net MNIST classifier? I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 maxpool layer and a 10 node dense layer with softmax as the activation function. I am using cross entropy loss and SGD.
When I train the network on the whole training set for a single epoch with a batch size of 1, I get 95% accuracy. However, when I try with a larger batch size (16, 32, 128), the learning becomes very noisy and the end accuracy is anywhere between 47%-86%. Why is it that my network performs so much worse and noisier on mini-batches?

Your model is very small for a convnet. 1 conv layer, 1 maxpool and 1 fc is very shallow. Try adding more layers and batchnorm2d after each conv layer followed by relu. No pool layers.


Changing word inflections This might be an unusual question. I have a situation where I am creating paraphrases with a rule based system. One transformation that I'd like to implement would one that gets rid of light verbs, like so:

a) Steven made an unwilling concession to us.

b) Steven unwillingly conceded to us.

To go from (a) to (b) requires some inflectional changes such as:

unwilling -> unwillingly

concession -> conceded

Is there something out there that can take care of such inflectional changes reliably for a given word? Seems like there should be, but I don't recall seeing anything like this.

Thought I'd share what I found here.

I have not tried any of this yet, but this looks very good, the Automatically Generated Inflection Database (AGID).

There is also the SPECIALIST Lexicon.

And for those using python, there are various modules you can use, such as LemmInflect.


Clustering and confusion matrix Here is the problem
The initial four cluster partition {c1, c2, c3, c4} for the text collection is provided by this link text vs cluster. Assuming that
the ground-truth partition is given by

cacm texts belong to cluster1
cisi texts belong to cluster2
cran texts belong to cluster3
med texts belong to cluster4

build the confusion matrix (CM) for the partition provided. That is build a 4X4 matrix whose row i shows distribution of ci elements
among cluster1, cluster2, cluster3, and cluster4. For example the first row is computed as follows:
CM11=|c1∩cluster1|,  CM12=|c1∩cluster2|,  CM13=|c1∩cluster3|, and CM14=|c1∩cluster4|
I understand the problem just not sure how to code it up.
I was thinking along the lines of this

//open text file
fileID = fopen('list.txt');
C = textscan(fileID,'%s %s');
fclose(fileID);

What this does is it orders the names as $C\{1\}\{j\}$is the file name and $C\{2\}\{j\}$ is the cluster name.
By the way I am using Matlab

Assume you clustering results are store in R. You have the partition id part_id and the real clust_id, here in a small example

df <- read.table(text = "obj,  part_id, clust_id
X, 1, 1
Y, 2, 2
Z, 3, 3
U, 1, 3
V, 2, 3
W, 2, 3"

Than the table will do the work for you

> table(df[,c("part_id","clust_id")])

       clust_id
part_id 1 2 3
      1 1 0 1
      2 0 1 2
      3 0 0 1

UPDATE

For completeness you may also use SQL (as proposed in a comment), a sample query is shown below on a table identical to the data-frame above.

You will see imediately, that there is a limitation because you must define the exact column list in the query - so the query must be adjusted based on the cluster / partition list.

with clust_agg as (
select part_id, clust_id, count(*) cnt from clust
group by part_id, clust_id
)
select * from clust_agg
pivot(sum(cnt) clust  for (clust_id) in
     (1 as "1",
      2 as "2",
      3 as "3"))
order by 1;
, header = TRUE, sep=",")

.

   PART_ID    1_CLUST    2_CLUST    3_CLUST
---------- ---------- ---------- ----------
         1          1                     1
         2                     1          2
         3                                1


Sentiment Analysis of comments to understand support on a topic I have downloaded comments from a website which asked people whether they supported or opposed the implementation of a certain political policy related to immigration.

I would like to get any resources or ideas on how to extract aggregate support/opposition to this policy.

In particular, I need a method that correctly identifies that all of the following comments are "anti-immigration" in some senses such as:

We need to give Americans first priority in the job market.
We should not let Americans suffer more unemployment on account of immigrants.
Immigrants should not be allowed to take American jobs.

Similarly the method should be able to identify "pro-immigration" comments, such as:

Providing this service to immigrants will be good for the economy.
The American economy will suffer if immigrants aren't allowed to continue to work here.
I do not think that passing this law will be detrimental to American jobs.

I agree with @thebiro - you could start by classifying a sample of the comments as being either opposed to or in support of the policy.  If you only need basic for/against classification, you can do binary classification (e.g. against = 0, for = 1).  If you need specify the degree to which a statement is for or against the policy, then you could define a scale that indicates the degree to which a comment supports the policy (e.g. strongly against = -3, strongly for = 3).  Once you have chosen your scale and manually classified your sample comments, you have a dataset that you can use to train a model.

Once you have your training data, you need to come up with a numerical representation of each comment.  There are too many possible approaches to enumerate here, but some basic concepts are bag of words and word vectors(aka word embeddings).  This Kaggle tutorial may help explain those concepts.

Finally, you need to train a model that takes the numerical representation of each comment as an input and outputs a sentiment score (0 or 1 for binary or a number on your scale).  You have a lot of choices for the type of model you use.  If you are working in Python, you can use one of the supervised learning methods in Scikit-learn.  Scikit-learn also has a tutorial on working with text.  You train your model on the sample you manually classified (Something like model.fit(training_data_inputs) in Scikit-learn) and then predict the output for the rest of your dataset (model.predict(test_data_inputs) in Scikit-learn).

Some people are also using neural networks for sentiment analysis.  Keras is a great Python library for building neural networks and has sentiment analysis examples available on Github.


Degree of Profanity in a Sentence Given a comment or a sentence and a list of profane words, How do I write a program to print the degree of profanity in that sentence?

One way to approach it is to split the sentence into tokens and count the number of tokens that are profanities.

import re

def tokenize(text):
    return re.findall(r'\w+', text.lower())

profane_tokens = {"nerfherder"}

sentence = "Why you stuck-up, half-witted, scruffy-looking nerfherder!"

tokens = tokenize(sentence)

# Rate: number of occurrences normalized by total number
degree_of_profanity = sum(1 for t in tokens if t in profane) / len(tokens)

This code will not handle multiple tokens and many profanities are multiple tokens.


How to choose validation set for production environment? I am using XGBoost for a time-series regression problem.

During development, i choose my validation set on last %10 percentage of data. Using timeseries split cross validation and grid-search, I got my best model on this with corresponding xgb hyperparameters.

My question is, how to choose validation set (for early stopping) on my production environment?

1) i have chosen last %10 percentage of my data as validation set, but this set is also included on training data. therefore overfit. very sensitive to noisy data.

2) my predicted data (lets say Y) changes over time, when i choose random rows within last year (%10 amount) and dont include them in training set, it gave me worse results on production than first option.

3) when i choose last week's data as validation data, not included in training set, it gave better result on 2. option. but i am not including last week's data to training procedure.

4) Or do i need a validation set on my production environment? should i set iteration count from the experiments done in development stage? (e.g i got best result on 10k th iteration, so i should limit my production-setup iteration count with 10k without using validation set at all?)

--

So, how can i choose validation set for my production environment? Best practices, or are there any tips/tricks for this?

If you're simply re-training the XGBoost model periodically in order to account for the changing nature of your data, the best option is to hold out a recent set of data for testing (some variation of your option 3).

As you mention, option 1- that is, training on the entire available set and validating on the most recent 10 percent of the data, is very likely to overfit (and therefore overestimate its performance on future data).

Option 2 has the downfall that you're attempting to predict information that you have post-hoc (after the fact) information about- that is, you know what subsequent observation values were, which is very relevant to a time-series prediction. For example, if you're trying to predict what the value is tomorrow, and you know what the value is for the day after tomorrow already, you can do a much better job of predicting tomorrow. Unfortunately, you can never know this before the day after tomorrow begins.

Therefore, option 3 is the most valuable practice to determine the likelihood of your model to accurately predict future observations- you're using only data that would be available from the point of prediction (in the past relative to the predicted period) and holding that data out of the training set for the model.

The best practice in these circumstances is to hold yourself accountable for "not cheating"- that is, try your best to provide a fair and honest experiment on the prediction of your model on data it hasn't seen and would be available in the context of a future use situation.


SVM SMOTE fit_resample() function runs forever with no result Problem

fit_resample(X,y)

is taking too long to complete execution for 2million rows.

Dataset specifications

I have a labeled dataset about network features, where the X(features) and Y(labels) are of shape (2M, 24) and (2M,11) respectively.

i.e. there are over 2million rows in the dataset. there are 24 features, and 11 different classes/labels.

Both X and Y are numpy arrays of float dtype.

Motivation for using SVM SMOTE

Due to class imbalance, I realized SVM SMOTE is a good technique balance it, thereby, giving better classification.

Testing with smaller sub-datasets

To test the performance of my classifier, I started small. I made small subdatasets out of the big 2million row dataset.

It took the following code:-

%%time
sm = SVMSMOTE(random_state=42)
X_res, Y_res = sm.fit_resample(X, Y)

1st dataset contains only 7.5k rows. It took about 800ms to run the cell.
2nd dataset contains 115k rows. It took 20min to execute the cell.

Solution Attempts

My system crashes after running continuously for more than 48hrs, running out of memory.

I've tried some ideas, such as

1. splitting it to run on multiple CPU cores using %%px

No improvement in quicker execution

2. using NVIDIA GPU's

Same as above. Which is more understandable since the _smote.py library functions aren't built with parallel programming for CUDA in mind.

I'm pretty frustrated by the lack of results, and a warm PC. What should I do?

This is expected and is not related to SMOTE sampling.

The computational complexity of non-linear SVM is on the order of $O(n^2)$ to $O(n^3)$ where $n$ is the number of samples. This means that if it takes 0.8 seconds for 7.5K data points, it should take [3, 48] minutes for 115K, $$[(115/7.5)^{2} \times 0.8, (115/7.5)^{3} \times 0.8]s=[3,48]m,$$and from 16 hours to 175 days, 11 days for $O(n^{2.5})$, for 2M data points.

You should continue using sample sizes on the order of 100K or less. Also, it is fruitful to track the accuracy (or any other score) as a function of samples for 1K, 10K, 50K, and 100K samples. It is possible that SVM accuracy stops improving well before 100K, therefore, there will be not much to lose by limiting the samples to 100K or less.


Fuzzy rule based system: Should rules contain all inputs and outputs? I am trying to design an FRBS using Matlab fuzzy tool box. The fuzzy system will be used to predict player's type based on the inputs and a set of rules defined by experts.
I have 6 inputs and 4 outputs (types of players). The given rules do not concern all inputs. Specific inputs are used for each player type.

Is it imperative to include all inputs and outputs in a rule?
Also is there a min/max of rules that an FRBS should include or does it depend on the experts expertise?

Thank you for your help.

Not really. It is possible to create fuzzy logic model without setting whole ruleset. But there is a risk, that model will perform worse in some unexpected situations which would be handled by missing rules.

However, it is possible to figure out fuzzy rules basing on data. I personally used Wang-Mendel method and it worked fine for me: still 6 inputs and 4 outputs will be many of rules to deduce :(. Wang - Mengel method is described in this paper: http://sipi.usc.edu/~mendel/publications/Wang%26Mendel%20SMC%2011-12-92.pdf


Using L1 penalty in XGBoost I'm trying to use L1 regularization to select features in XGBoost classifier. However, I don't see any example code on how to specify the penalty of l1.

This is how I do in sklearn's LogisticRegression.

C = [10, 1, .1, .001]

for c in C:
    clf = LogisticRegression(penalty='l1', C=c)
    clf.fit(X_train, y_train)
    print('C:', c)
    print('Training accuracy:', clf.score(X_train, y_train))

How should I specify penalty and C in XGBoost?

L2 and L1 regularization are controlled via the lambda (=reg_lambda) and alpha (=reg_alpha) parameter respectively. Higher values of alpha mean more L1 regularization. See the documentation here.


Predicting probability for each tag given already chosen tags I have a set of tags (~10'000, will be extended over time) presented to a user. After he has selected 3 or more tags, I want to predict for each remaining tag what the chances are that the user will select this tag as well. I strictly need the prediction for all remaining tags. The prediction should be fast but accuracy isn't that important. The training data would be cases where users already have selected a subset of tags. What would be a good approach here?

welcome to ML and data science. This is a classic situation where a RNN would be usefull. You could either train one from Keras yourself which would help you learn and get a better model but might take more time. Or you could use a more pre-made soloution if this is your first project maybe something like TextGenRNN (will still recquire a bit of modification).

RNN theory

Examples in python

More examples

TextgenRNN Library


Regression in Predicting Tenancy Lengths I'm currently working on a project involving the prediction of tenancy lengths. I've so far managed to get to a point where I've processed the data and pruned my Random Forest model (via sklearn in Python) to the following accuracy levels (in days):

Train MAE: 131
Train R^2: 0.906
Test MAE: 259 (using cross-validation)
Test R^2: 0.651

While the model is decent for the industry, there's more performance to squeeze out of it. It currently overestimates results and has poor accuracy on the test data imo.

I'd like to further develop a Neural Network approach, as my initial implementation of an MLP Regressor seems promising:

Train MAE: 301
Train R^2: 0.582
Test MAE: 338 (using cross-validation)
Test R^2: 0.522

My question is how can I improve my results for the prediction (using Python) other than using GridSearch to play around with the MLPRegression function in sklearn? Are there any other models that could be useful in this situation? (I have tried also decision trees, gradient boosting)

In case it is relevant, my dataset contains ~5000 entries since 2008 onwards of individual tenancies, containing: tenancy dates, rent, repair costs, property information and replacements, client information, etc., currently at 41 variables.

You may have done this already, but if your target values are positive integers, it could be worth transforming your output layer in such a way that constrains it to take an appropriate range of values.

Tenancy length values presumably have some recognisable distribution that might fruitfully be modelled by a known statistical process (Poisson process, survival analysis, etc.), and so instead of using your NN to predict the tenancy length directly, you could use the NN to parametrise a relevant probability distribution, and make a point estimate of that distribution in your output layer.


BertPunc (punctuation restoration with BERT) I've found the script for punctuation restoration. And I have one question about this method.
I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), PERIOD (1), COMMA (2), QUESTION (3). Further, all words are converted to BERT tokens. Here is an example:
  2045  0
  2003  0
  2200  0
  2210  0
  3983  0
  2301  0
  2974  0
  1999  0
  2068  2

Next, we do padding. We set a segment (e.g. eight words) and for each word we take two words before a token and four words after a token. Also, we add a padding token after each word. For the very first word, there are no words before. Therefore, a word are taken from the end. Similarly, for the last word, there are no words after and therefore a word are taken from the beginning.
Here is an example of it.
[1999, 2068, 2045, 0, 2003, 2200, 2210, 3983] 0
[2068, 2045, 2003, 0, 2200, 2210, 3983, 2301] 0
[2045, 2003, 2200, 0, 2210, 3983, 2301, 2974] 0
[2003, 2200, 2210, 0, 3983, 2301, 2974, 1999] 0
[2200, 2210, 3983, 0, 2301, 2974, 1999, 2068] 0
[2210, 3983, 2301, 0, 2974, 1999, 2068, 2045] 0
[3983, 2301, 2974, 0, 1999, 2068, 2045, 2003] 0
[2301, 2974, 1999, 0, 2068, 2045, 2003, 2200] 0
[2974, 1999, 2068, 0, 2045, 2003, 2200, 2210] 2

The first column contains tokens, and the second column contains punctuation symbols. In first column, "0" corresponds to a padding. Next we do TensorDataset, and than DataLoader. In the second column, '0' corresponds to "other", and '2' corresponds to a "period". Finaly we train a model:
for inputs, labels in data_loader_train:
       inputs, labels = inputs.cuda(), labels.cuda()
       output = model(inputs)

The algorithm works well, but I do not understand the following.
What's the point of putting padding in the middle? Maybe we can do punctuation restoration with BERT in a more simple way?

I am also confused by the method of BertPunc. Actually I can't believe it really works. Beacause the method use BertLanguageModel to do the job, and LanguseModel's main capabiltiy is to predict the next word of a word. So also use your example,  BertPunc inputs 8 words, the output of bert should be 8 next words of the inputs. But BertPunc added a Linear layer to only get 1 output value, which is to predict punctuation.

But I think we should use bert the way like the graph below:


How do I validate this Kalman model for estimation of undocumeted covid cases? Tensorflow recently made a tutorial titled Estimation of undocumented SARS-CoV2 cases. It replicates 6th March 2020 paper by Li et al titled Substantial Undocumented Infection Facilitates the Rapid Dissemination of Novel Coronavirus (SARS-CoV2). It is a compartment based SEIR model where the population is represented by a state with compartments Susceptible, Exposed, Undocumented Infectious and Documented Infectious.
I am trying to test the validity of this model by training it with data for 30 days, and forecast the state for next 15 days. I did find the optimal parameters of the model and current state, but I am not sure how I can use it to predict future states. I have been attempting this for almost a week now. The programming style in the notebook is quite unfamiliar to me, hence, I am struggling to figure out how exactly to do it.
I request someone with more experience to go through the notebook once and give me suggestions about how to predict the states.
Thank you!

To summarize, the notebook describes three key components:

The state, which completely describes the system at any given time. A SEIR model by definition has four variables in the state: susceptible, exposed, infected, and reported.
The dynamics, which describe how the states evolve over time.
The measured parameters for the dynamic equations: $\beta$, $\mu$, $\theta$, $Z$, $\alpha$, and $D$. Note, these parameters are (for our purposes) constant.

The tutorial walks you through measuring the parameters. The logic is:

You know your state for 30 consecutive days
You know that one day progresses to the next through the dynamics equations
So, you can solve for the parameters that maximize the accuracy of your model

Now that you have the parameters, your goal is to make predictions for the future. This is "just" a matter of plugging your parameters into the dynamics equations and doing the math.
Now solving six coupled differential equations is rarely easy, but they've done much of the heavy lifting for you. In particular, the function transition_fn is pretty much exactly what you want: it takes the current state, current time, and some "extra" information (defined in the tutorial) and produces the prediction for the next state.
So I think what you need to do is:

Figure out how to call transition_fn. You'll have to define state_params and extra in exactly the right format. The parameters you calculated in the tutorial will become part of extra
Start by using the same time period that the tutorial uses. This is "cheating" a bit, since you used the data to derive your parameters and now you're using the parameters to predict the data. But this is a good "sanity check": your predictions should definitely be accurate here, so you'll be able to identify any problems.
Then, use a different time period to assess whether the model is useful.


How can I get prediction for only one instance in Keras? When I request Keras to apply prediction with a fitted model to a new dataset without label like this:

model1.predict_classes(X_test)

it works fine. But when I try to make prediction for only one row, it fails:

model1.predict_classes(X_test[10])

Exception: Error when checking : expected dense_input_6 to have shape (None, 784) but got array with shape (784, 1)

I wonder, why?

You can do:

q = model.predict( np.array( [single_x_test,] )  )


How to use SMOTE in Stacking in SKLearn? I have a data set X,y and split them to train and test data.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, stratify = y, random_state=10). To handle imbalanced data, I wanna use SMOTE and then use classification algorithms. However, I am going to use Stacking as my classification method. I would be thankful to know when I should use SMOTE? Should I use them in defining lower-level classifiers or in higher-level classifiers?
level0 = list()
oversample = SMOTE()
RF = RandomForestClassifier(random_state=13)
pipe1 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', RF)])
level0.append(pipe1 )

DT = DecisionTreeClassifier( random_state=0)
pipe2 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', DT)])
level0.append(pipe2)

level1 = LogisticRegression
model = StackingClassifier(estimators=level0, final_estimator=level1, cv=10, passthrough = True)
model.fit(X_train, y_train)
model.predict(X_test)

Or I should use the following code?
level0 = list()
oversample = SMOTE()
RF = RandomForestClassifier(random_state=13)
level0.append(RF)

DT = DecisionTreeClassifier( random_state=0)
level0.append(DT)

level1 = LogisticRegression
model = StackingClassifier(estimators=level0, final_estimator=level1, cv=10, passthrough = True)

pipe1 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', model)])

pipe1.fit(X_train, y_train)
pipe1.predict(X_test)

Another question, we use SMOTE in the training step to have a better model. But in pipeline, the first step is using SMOTE, and I think that in prediction on test data, at first, test data is oversampled, then classification model is applied? Is it correct? I don't know how I should use SMOTE for the final prediction. I would be thankful if someone can explain it and modify my code.

First question, whether to use SMOTE for the first or second of a stacked classifiers. Generally, SMOTE should be done before any classification since SMOTE gives the minority class an increased likelihood be being successfully learned. The first classifier should be given the most useful features. Another way to approach is looking for empirical evidence. Train models both ways and choose the ordering that performs betters.
Second question, SMOTE is only done on the training dataset. During prediction, only the data that is present is predicted. If you use imblearn's Pipeline, this automatically handled.


Machine learning without explicit training Is there any set of machine learning algorithms that do not require training and directly gives answers if a set of labeled and unlabeled data gave at once?

Is skipping explicit training of model can be skipped either in supervised or unsupervised machine learning algorithms? [I do not talk of using pre-trained model]

Nearest neighbour algorithms (kNN and variants) do not have a training phase. They work by storing all the labelled examples, and using them directly for inference on new data.

There are some caveats:

"Training" is fast as it just stores each example once, but inference is slow, as typically it involves searching for relevant examples. This can be improved by indexing routines (which are not part of the algorithm, but might be part of a specific implementation).
Although there are no parameters to train, there may still by hyper-parameters to select, which will involve running a cross-validation exercise with hold-out data and varying the choices, e.g. of number of nearest neighbours to consider, or best distance metric to use to determine closest neighbours.
Data preparation (normalisation) and feature engineering are often required to get the best results out of kNN, and also require multiple attempts and tests.

In general, approaches that store and query labelled data without a training phase are instance based learning


Is Adam's optimization susceptible to Local Minima? # Neural Network Architecture

no_hid_layers = 1
hid = 3
no_out = 1

# Xavier Ininitialization of weights w

w1 = np.random.randn(hid, n+1)*np.sqrt(2/(hid+n+1))
w2 = np.random.randn(no_out, hid+1)*np.sqrt(2/(no_out+hid+1))

# Sigmoid Activation Function
def g(x):
    sig = 1/(1+np.exp(-x))
    return sig

def frwrd_prop(X, w1, w2):
    z2 = w1 @ X.T
    z2 = norm(z2, axis=0)
    a2 = np.insert(g(z2), 0, 1, axis=0)
    h = g((w2@a2))
    return (h,a2)

# Calculating Cost and Gradient

def Cost(X, y, w1, w2, lmbda=0):
    # Initializing Cost J and Gradients dw
    J = 0
    dw1 = np.zeros(w1.shape)
    dw2 = np.zeros(w2.shape)
    # Forward Propagation to calculate the value of the output
    h, a2 = frwrd_prop(X, w1, w2)
    # Calculate the Cost Function J
    J = -(np.sum(y.T*np.log(h) + (1-y).T*np.log(1-h)) - lmbda/2*(np.sum(np.sum(w1[:,1:].T@w1[:,1:])) + np.sum(w2[:,1:].T@w2[:,1:])))/m
    # Applying Back Propagation to calculate the Gradients dw
    D3 = h-y
    D2 = (w2.T@D3)*a2*(1-a2)
    dw1[:,0] = (D2[1:]@X)[:,0]/m
    dw2[:,0] = (D3@a2.T)[:,0]/m
    dw1[:, 1:] = ((D2[1:]@X)[:,1:] + lmbda*w1[:,1:])/m
    dw2[:, 1:] = ((D3@a2.T)[:,1:] + lmbda*w2[:,1:])/m
    # Gradient clipping
    if(abs(np.linalg.norm(dw1))>4.5):
        dw1 = dw1*4.5/(np.linalg.norm(dw1))
    if(abs(np.linalg.norm(dw2))>4.5):
        dw1 = dw1*4.5/(np.linalg.norm(dw2))
    return (J, dw1, dw2)

# Adam's Optimization technique for training w

def Train(w1, w2, maxIter=50):
    # Algorithm
    a, b1, b2, e = 0.001, 0.9, 0.999, 10**(-8)
    V1 = np.zeros(w1.shape)
    V2 = np.zeros(w2.shape)
    S1 = np.zeros(w1.shape)
    S2 = np.zeros(w2.shape)
    for i in range(maxIter):
        J, dw1, dw2 = Cost(X, y, w1, w2)
        V1 = b1*V1 + (1-b1)*dw1
        S1 = b2*S1 + (1-b2)*(dw1**2)
        V2 = b1*V2 + (1-b1)*dw2
        S2 = b2*S2 + (1-b2)*(dw2**2)
        if i!=0:
            V1 = V1/(1-b1**i)
            S1 = S1/(1-b2**i)
            V2 = V2/(1-b1**i)
            S2 = S2/(1-b2**i)
        w1 = w1 - a*V1/(np.sqrt(S1)+e)*dw1
        w2 = w2 - a*V2/(np.sqrt(S2)+e)*dw2
        print("\t\t\tIteration : ", i+1, " \tCost : ", J)
    return (w1, w2)

# Training Neural Network

w1, w2 = Train(w1,w2)

I'm using Adam's Optimization to converge Gradient Descent to a global minima but the cost is becoming stagnant (not changing) after around 15 iterations(the number is not fixed). The initial cost due to random initialization of weights is changing very minutely before becoming constant. And this is giving training accuracy from 45% to 70% for different runs of the exact same code. Can you help me with the reason behind this?

All stochastic gradient descent (SGD) optimizers, including Adam, have randomization built and have no guarantees of reaching a global minima. The randomization is a result of training on a sub-sample of the data at each step. There are no guarantees of reaching a global minima because gradient descent optimizers are first-order, iterative optimization techniques.


Does out-of-bag sampling make Random Forests inherently less robust than other classifiers? All of the popular evaluation metrics (ROC-AUC, Confusion Matrices, etc.) require two lists as parameters: a list of the actual y labels associated with some arbitrary group of training examples (x's), and a parallel list of the predicted labels given to those x's by the model.

To construct such lists, you must separate a testing/validation data set from the training set you feed the model on. However, random forests automatically partition 1/3 of the training set you give it to calculate the out of bag score. I don't believe you can stop the bagging process that causes this, since I think it is vital to how the random forest functions. Because the RF model never sees 1/3 of the training set (due to bagging), do RFs create a less thorough image of the dataset then, say, a neural network would whenever a testing set is reserved for evaluation?

The out-of-bag-error is calculated from the samples that are not used anyway for the particular tree. The original set of records gets bootstrapped. So, a new set is generated that does not contain all samples anyway. The out-of-bag set can then be used to monitor the performance of the model. When the out-of-bag-error goes up (given it has a significant size) it means that the current tree is over-fitting the training sample. So, out-of-bag sampling can be used to prevent over-fitting and therefore makes the model more robust rather than less robust.

So, in fact all samples are used to train the random forest model. Though each tree only uses a subset of the dataset at a time. Don't confuse the RF model with the individual trees!

Look here for a more detailed description.


What is the benefit of using Max pooling in convnets as opposed to just using convolution layers? (from Francois Chollet's Deep Learning with Python) I am reading Francois Chollet's Deep learning with python, and I came across a section about max pooling that's really giving me trouble.

I simply don't understand what he means when he talks about "What's wrong with this setup?". How does removing the max pooling layers "reduce" the amount of the initial image that we're looking at?

Max pooling layers are useful for two things:

undersampling the dimensions of their input, to reduce model parameters and also reduce overfitting
increasing the robustness of the model to small changes in the input image (e.g. if I change the input image slightly the model should predict the same thing).

However their usefulness is limited and are starting to be replaced by convolution layers with large strides. You could read this article on the matter.


Forecasting sales and creating model In a assignment we are given macro economic indicators like GDP, Consumer price index, Producer Price index and Industrial production index. Also we are given Crude oil, Sugar prices and FM-CG Sales. We are required to forecast future quarter sales and give a model. As I'm new to this subject, I don't know where to start with it, or what to read. Can anyone provide me with some examples of what to do, or any PDFs which might be helpful.

If you are new to forecasting, I would recommend starting with a very simple model for your sales, e.g., Exponential Smoothing, which will use only the sales data and no external information, but will capture trend and seasonality. Here is an extremely good free online textbook.

Once you are comfortable with your simple model, you can look at more complex ones that do include external causal variables. For instance, you could run ARIMAX models (the X stands for eXternal or eXplanatory variables). For this, look at chapter 8 in the online textbook and check the auto.arima() function in the forecast package for R, where you can specify external variables using the xreg parameter.

Alternatively, you could look at other methods like Neural Networks, or possibly even Vector Autoregression. To learn more, you could look into standard econometrics textbooks.

Whatever you do, keep two things in mind:

If you include explanatory variables in your forecast, you will need to forecast those variables themselves. In an assignment-type situation, you may be able to work with actual future macroeconomic variables, but of course in a "real" forecasting situation, you don't know next month's sugar prices yet. Forecasting these to feed the sugar price forecast into your sales forecast model adds an additional bit of uncertainty.
You may just find out that your fancy model with lots of additional explanatory variables does not do a much better job at forecasting than your original simple model. This would not at all be surprising. Simple methods very often outperform complex ones in forecasting.


NLP - why is "not" a stop word? I am trying to remove stop words before performing topic modeling. I noticed that some negation words (not, nor, never, none etc..) are usually considered to be stop words. For example, NLTK, spacy and sklearn include "not" on their stop word lists. However, if we remove "not" from these sentences below they lose the significant meaning and that would not be accurate for topic modeling or sentiment analysis.

1). StackOverflow is helpful      => StackOverflow helpful
2). StackOverflow is not helpful  => StackOverflow helpful

Can anyone please explain why these negation words are typically considered to be stop words?

Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible.

It clearly makes sense to consider 'not' as a stop word if your task is based on word frequencies (e.g. tf–idf analysis for document classification).

If you're concerned with the context (e.g. sentiment analysis) of the text it might make sense to treat negation words differently. Negation changes the so-called valence of a text. This needs to be treated carefully and is usually not trivial. One example would be the Twitter negation corpus. An explanation of the approach is given in this paper.


Word2Vec Alternative for Smaller Datasets I am looking to employ Word2Vec to cluster documents and classify them by topic. However, it seems I need a significant data set  [1,2] to achieve such a task.

I have a data set of thousands (not millions) of documents where my total word count is somewhere in the tens of thousands or maybe low hundreds of thousands.

Is there another ML methodology that allows me to attain my goal?

I see how to use TF/IDF to generate words and phrases from a corpus  but output as expected is a list of common words and phrases along a flat dimension:

What I am looking for is something more along the lines of a high level cluster of vectors in space:
[source]

You can side step the paucity of training data, and indeed training altogether, by using pre-trained embeddings in numerous languages. After that you can calculate your document embeddings using one of these simple algorithms, which basically amount to running dimensionality reduction on the matrix of stacked word embeddings for each sentence using PCA/SVD:

A Simple but Tough-to-Beat Baseline for Sentence Embeddings (code)
Representing Sentences as Low-Rank Subspaces

Note that word embeddings themselves emerge from similar calculations:

Neural Word Embedding as Implicit Matrix Factorization


Using user defined function in groupby I am trying to use the groupby functionality in order to do the following given this example dataframe:
dates = ['2020-03-01','2020-03-01','2020-03-01','2020-03-01','2020-03-01',
        '2020-03-10','2020-03-10','2020-03-10','2020-03-10','2020-03-10']
values = [1,2,3,4,5,10,20,30,40,50]
d = {'date': dates, 'values': values}
df = pd.DataFrame(data=d)

I want to take the largest n values grouped by date and take the sum of these values. This is how I understand I should do this: I should use groupby date, then define my own function that takes the grouped dataframes and spits out the value I need:
def myfunc(df):

    a = df.nlargest(3, 'values')['values'].sum()

    return a

data_agg = df.groupby('date').agg({'relevant_sentiment':myfunc})

However, I am getting various errors, like the fact that the value keep is not set, or that it's not clearly set when I do specify it in myfunc.
I would hope to get a dataframe with the two dates 03-01 and 03-10 with respectively the values 12 and 120.
Any help/insights/remarks will be appreciated.

You could do it simple and it should work like this:
def myfunc(df):
    return df.nlargest(3, 'values')[['values']].sum()

and then:
data_agg = df.groupby('date', as_index=False).apply(myfunc)

You decide if "data_agg" is the proper name then.
Good luck!


What is a "surrogate model"? While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore:

What is a surrogate models?

Why are surrogate models good for explainability?

A surrogate model is an approximation model for a given function.
The original function is generally a black box function that we can sample from and based on the samples we can optimize our surrogate model to approximate the behaviour of the original function.
A surrogate model can be a neural network, an ensemble method, a gaussian process which can be interpreted and used later for explainability.


How should I convert Logistic Regression's coefs into action strategy? I am trying to analyze soccer's data set:

W_OVER_2_5  PREDICTED MATCH_DATE LEAGUE HOME AWAY MATCH_HOME MATCH_DRAW MATCH_AWAY  MATCH_U2_50 MATCH_O2_50
0            0        1105135200    5   260  289    2.05        3.00        4.50        1.65       2.30
0            1        1105308000    16  715  700    2.50        3.30        3.05        1.80       2.14
1            1        1105308000    11  445  479    1.36        5.25       12.00        2.15       1.78
0            1        1105308000    11  453  474    3.00        3.35        2.62        1.75       2.20....

Now, I selected 'the best estimator' -

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,
          verbose=1, warm_start=False)

with best coefs -

1. -2.40477246e-10
2. -5.57611571e-02
3. -1.32010761e-04
4.  1.51666398e-03
5.  7.54521399e-02
6.  6.38889247e-02
7. -2.25746953e-01
8. -3.79313902e-01
9.  3.70514297e-02

Now, I have a question - how should I understand coefs in terms of real strategy? As example,

If `MATCH_HOME` is min among all [`MATCH_HOME`, `MATCH_DRAW`, `MATCH_WAY`] AND `MATCH_O2_50' = 1
THEN PREDICTED := 1
ELSE PREDICTED := 0

PS. I would very appreciated for any science papers about that thematic :)

To understand the coefficients you just need to understand how the logistic regression model that you fit uses the coefficients to make predictions. No, it does not work like a decision tree. It's a linear model.

Really, predictions are based on the dot product of the coefficients and the values from some new instance to predict. This is just the sum of their products. The higher the dot product, the more positive the prediction.

So you can understand it as computing something like -2.40477246e-10 * MATCH_HOME + -5.57611571e-02 * MATCH_AWAY + ... (I don't know what coefficients go with what feature in your model.)

That generally means that inputs with bigger coefficients matter more, and inputs with positive coefficients correlate positively with a positive prediction. That's most of what you can interpret here.

The first of those conclusions is only really valid if inputs have been normalized to be on the same scale though. I'm not clear that you've done that here. You should also in general use L1 regularization if you intend to interpret the coefficients this way.


Neural Network for Multiple Output Regression I have a dataset containing 34 input columns and 8 output columns.

One way to solve the problem is to take the 34 inputs and build individual regression model for each output column.

I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used Multilayer Perceptron but that needs multiple models just like linear regression. Can Sequence to Sequence be a viable option?

I am using TensorFlow. I have code but I think it is more important to understand what I am missing out in terms of the multilayer perceptron theory.

I understand that in MLP if you have one output node it will provide one output. If you have 10 output nodes then it is a multi class problem. You pick the class with the highest probability out of the 10 outputs. But in my case it is certain there will be 8 outputs for same input.

Lets say, for a set of inputs you will get the 3D coordinate of something (X,Y,Z). Like, Inputs = {1,10,5,7} Output = {1,2,1}. So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model. So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though.

What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the nature of the problem.

Any neural network framework is able to do something like that.

The key to do that is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (34,) and the output (8,).

Update: the usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras; the network is not an MLP but it should be Ok to illustrate the idea.


How does Google categorize results from its image search? While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and creates categories.

Unfortunately, I couldn't find much about it at all. Is anyone able to shed some light on algorithms they may be using to do this, and what basis these categories are created from?

For example, if I search for "animals" I get the categories:

  "cute", "baby", "wild", "farm", "zoo", "clipart".

If I go into "wild", I then have subcategories:

  "forest", "baby", "africa", "clipart", "rainforest", "domestic".

I am not working at Google, but I think it is some sort of recommendation system based on the words which millions of users searched before. So those people who search for "animals" often search for "wild animals" for example. Like in many online stores they recommend you to buy something in addition to the product you are looking for based on the previous purchases of other users.

There are many approaches how to build such recommendation system using machine learning, no one knows for sure what google uses.


Multiple digits MNIST and transfer learning I have a sample of 50,000 images, some of which are shown below:

 $\qquad$
  $\qquad$  $\qquad$  $\qquad$

Associated to these images are labels for the digit with the largest pixel size. My goal is to build a machine learning model to predict the largest digit in an image by pixel size.

To that end, I used transfer learning on the resnext model, but only found an accuracy of 60%.

Given that this implementation uses transfer learning to train a model to predict MNIST digits, I would now like to crop each training image to retain only the largest digit and then train the model using the linked implementation.

So, my question is, how I do crop the training images to retain only the digit in each image with the largest size.

You need to manually or automatically draw bounding boxes around each digit. Then compute the area for each box and take the largest.

Given they are non-overlapping black digits on a white background, template matching would work.


How much data do you need to build a classifier? I would like to ask you what a good size of dataset would be for building a classifier.
I know that there are datasets of 1000 obs and datasets of 1m obs. But I also read papers where classifiers were built on datasets of 300 obs.
I think the size may affect the accuracy/precision of a classifier, but I am not sure 100% of that.

It's impossible to answer this question in general, because the answer strongly depends on the content of the data. More precisely it depends if the relations between the features and the target class in the training data are sufficient for the parameters of the model to be estimated as accurately as possible. In the most simple cases a few instances might be enough, for example a linear regression model with one single feature would need only two "perfect" instances. Practically the following factors have a strong impact on the number of instances needed:

the number of parameters to estimate and the complexity of the model: a more fine-grained model needs more detailed information, i.e. more instances
the number of classes, because there are more possible combinations for the model to learn and because it usually implies a higher number of parameters as well (more complex model)
the number of features, for the same reason
the amount of noise in the data, because finding the general patterns is more difficult if there are lots of inconsistencies/errors in the data, so statistically more instances are needed to distinguish the effect of chance from the real patterns.

So the only way to check how much data is needed for a particular task and dataset is to do an ablation study, i.e. a series of experiments in which a model is trained every time with a different number of instances. For example if one has 1000 instances, they could try to train a model with 100, 200, 300,...,1000 instances. Then by plotting the performance of every model one can observe whether the curve becomes stable at some point: if yes, this point shows how many instances are needed. If not (i.e. the performance keeps increasing significantly), then more than 1000 instances are needed.

I think the size may affect the accuracy/precision of a classifier, but I am not sure 100% of that.

Definitely.


Visualizing Time Series Data I hope this question isn't out of place here.

I have some time series data from a Zooz power plug that I'd like to visualize.  Excel works fine to a point, with small sets of data, but it's not easy to move the data window, to zoom in on particular moments, for example.

Are there any free data visualization apps that accept time series data and implement controls for viewing it dynamically?  I'm happy to pay for something, too, if it works well, but this is for a personal hobby project, so I'm trying to spend my money carefully :)  Looking for off-the-shelf solutions before I dive into R.  Thanks.

You can use plotly in python to get this done.

import plotly.graph_objects as go
import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([1, 3, 2, 3, 1])

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name="linear",
                    line_shape='linear'))
fig.add_trace(go.Scatter(x=x, y=y + 5, name="spline",
                    text=["tweak line smoothness<br>with 'smoothing' in line object"],
                    hoverinfo='text+name',
                    line_shape='spline'))
fig.add_trace(go.Scatter(x=x, y=y + 10, name="vhv",
                    line_shape='vhv'))
fig.add_trace(go.Scatter(x=x, y=y + 15, name="hvh",
                    line_shape='hvh'))
fig.add_trace(go.Scatter(x=x, y=y + 20, name="vh",
                    line_shape='vh'))
fig.add_trace(go.Scatter(x=x, y=y + 25, name="hv",
                    line_shape='hv'))

fig.update_traces(hoverinfo='text+name', mode='lines+markers')
fig.update_layout(legend=dict(y=0.5, traceorder='reversed', font_size=16))

fig.show()

https://plot.ly/python/line-charts/


Examining a DocumentTermMatrix in RTextTools I created a DocumentTermMatrix  for text mining using RTextTools. The rows for this DocumentTermMatrix  correspond to dataframe rows and matix columns correspond to words. My question is : How can I get the words (labels vector) for examining the DocumentTermMatrix  ? In other words, How can I get the vector of these 904 words?

require(RTextTools,quietly=TRUE)
data(USCongress)

doc_matrix <- create_matrix(USCongress$text, language="english",   removeNumbers=TRUE, stemWords=TRUE, removeSparseTerms=.998)

dim(USCongress)
[1] 4449    6

dim(doc_matrix)
[1] 4449  904

a documenttermmatrix is a simple_triplet_matrix. You can turn this into a simple matrix with the as.matrix command and then use all matrix functions.

# turn into simple matrix
mat <- as.matrix(doc_matrix)

# vector of the words
word_vector <- colnames(mat)

# Dataframe containing words and their frequency
df_words <- data.frame(words = colnames(mat), frequency = colSums(mat), row.names = NULL)


Overfit from the loss curve I have a binary classification task. I have shown the loss curve here. I have decreased the learning rate by 1/10 every 15 epochs. There is also dropout put in the model. As you can see, I am trying to figure out the optimal point for model training. My initial assumption was that the point came at around epoch 28 since the validation error almost remains constant and then increases ever so slightly. However, I still wanted to know if this is fine or the model is indeed overfitting.

Another concern I have is that the training and validation curves are very very close to each other. Is this an expected behavior?

Being a newbie I would really appreciate any help in here

I'll go through your question one by one:

  My initial assumption was that the point came at around epoch 28 since the validation error almost remains constant and then increases ever so slightly. However, I still wanted to know if this is fine or the model is indeed overfitting.

Commond knowledge from DS/ML handbooks says you're right. You should keep training until validation Loss doesn't start going up. Please keep in mind that some overfitting is inevitable (a good model is not one that eliminates overfitting, that would be impossible, but a model that is able to keep it at bay).

  Another concern I have is that the training and validation curves are very very close to each other. Is this an expected behavior?

You should keep training even after validation loss gets higher than training loss, and stop only when validation loss starts growing. Ideally, you should have a very slight overfitting, with validation set just above training, and its slope is flat. That means: I can't train my model more than that without making things worse.


why does transform from tfidf vectorizer (sklearn) not work I'm transforming a text in tf-idf from sklearn. I made the model:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = words
vectorizer = TfidfVectorizer(min_df = 15)
tf_idf_model = vectorizer.fit_transform(corpus)

And now I'm making vectors for different sets of words (documents), like:

word_set = ['dog', 'cat', 'foo']
v = vectorizer.transform(word_set)

But I want just one vector of these words, to compare to other documents. But when I use transform, the shape of v becomes:

<3x56492 sparse matrix of type '<class 'numpy.float64'>'
    with 3 stored elements in Compressed Sparse Row format>

I want a vector with shape 1x56492, and not 3x56492.. I'm certainly missing something here. Maybe you guys have some tips?

Thank you very much in advance.

As far as I can tell it's interpreting your new word_set word_set = ['dog', 'cat', 'foo'] as three separate documents containing one word each, whereas if you did word_set = ['dog cat foo'] it would interpret this as a single new document containing those words.

What behavior are you expecting from this function? Is corpus = words a list of document strings, or a list of single words? If it's the latter, this is likely not doing what you think it is doing, and you should instead make corpus a list of document strings.


Can I test my already trained model using the data it was trained on? I have a model that predicts multiple choice answers to questions. I used an 80/20 train test split of my questions and tuned it.

The questions actually form part of a game aka 10 questions in a game. The data when split was randomly shuffled and split so the questions no longer form part of a game.

Can I now use the same questions it was trained and tested on to re-test all the questions but in order to determine the percentage of games won? Or when initially training my model should the train and test data be split by games instead of questions?

It is possible to re-use the train+test dataset.

You used the train dataset to train your parameters and the test to check if there was or not overfitting in your model.

The train/test split is used to check your if your model is likely to suffer from overfitting or not, more than to obtain the "real parameters".

So, in that order of ideas, you discarded your model's overfitting and then you could use your complete dataset to recalibrate your parameters.

Remember the decision of wheter an individual belongs to train or test is arbitrary.


Python First Project: Algorithm that find a book and its author from a quote I hope this is the good site for my question.

I am new with Python and I am neither a programmer nor a data scientist. I would like to code an algorithm that given a book quote can find the book's name and author.
Do you have any suggestion/sources that can help me with my project?

I think that the simplest approach will be to include in my code the Google Books API. In fact, it is really difficult to find a large database including Books' text in order let the algorithm research the quote. Do yo agree with my first approach? Which are the limitations of using Google APIs? Can I share my code with other users or they will be not able to use it because of some kind of Google restriction?

I am sorry for the stupid questions, but I have just begun this adventure in Datascience and I think that the best way to learn is to fix some objectives and not just to do courses.

Thank you.

This is not really a data science project. It's more a programming problem as you are trying to build a program with multiple functions to gather data, sort data, etc.

Data science is about finding, researching, analyzing relationships within data itself. Data scientists don't work on APIS, neither SQL or other data sources (but it doesn't mean they don't know about it).

https://stackoverflow.com/ should be a right place for your question, specially if you have some code. I am sure they will help you there with Google API


Implementing RMSProp, but finding differences between reference versions I am researching to implement RMSProp in a neural network project I am writing.

I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a Coursera class presented by Geoffrey Hinton (lecture 6 I think). I don't think the approach has ever been formally published, despite many gradient-descent optimisation libraries having an option called "RMSProp". In addition, my searches are showing up a few variations of the original idea, and it is not clear why they differ, or whether there is a clear reason to use one version over another.

The general idea behind RMSProp is to scale learning rates by a moving average of current gradient magnitude. On each update step, the existing squared gradients are averaged into a running average (which is "decayed" by a factor) and when the network weight params are updated, the updates are divided by the square roots of these averaged squared gradients. This seems to work by stochastically "feeling out" the second order derivatives of the cost function.

Naively, I would implement this as follows:

Params:

$\gamma$ geometric rate for averaging in [0,1]
$\iota$ numerical stability/smoothing term to prevent divide-by-zero, usually small e.g 1e-6
$\epsilon$ learning rate

Terms:

$W$ network weights
$\Delta$ gradients of weights i.e. $\frac{\partial E}{\partial W}$ for a specific mini-batch
$R$ RMSProp matrix of running average squared weights

Initialise:

$R \leftarrow 1$ (i.e. all matrix cells set to 1)

For each mini-batch:

$R \leftarrow (1-\gamma)R + \gamma \Delta^2$ (element-wise square, not matrix multiply)
$W = W - \epsilon \frac{\Delta}{\sqrt{R + \iota}}$ (all element-wise)

I have implemented and used a version similar to this before, but that time around I did something different. Instead of updating $R$ with a single $\Delta^2$ from the mini-batch (i.e. gradients summed across the mini-batch, then squared), I summed up each individual example gradient squared from the mini-batch. Reading up on this again, I'm guessing that's wrong. But it worked reasonably well, better than simple momentum. Probably not a good idea though, because of all those extra element-wise squares and sums needed, it will be less efficient if not required.

So now I am discovering further variations that seem to work. They  call themselves RMSProp, and none seems to come with much rationale beyond "this works". For example, the Python climin library seems to implement what I suggest above, but then suggests a further combination with momentum with the teaser "In some cases, adding a momentum term β is beneficial", with a partial explanation about adaptable step rates - I guess I'd need to get more involved in that library before fully understanding what they are. In another example the downhill library's RMSProp implementation combines two moving averages - one is the same as above, but then another, the average of gradients without squaring is also tracked (it is squared and taken away from the average of squared weights).

I'd really like to understand more about these alternative RMSProp versions. Where have they come from, where is the theory or intuition that suggests the alternative formulations, and why do these libraries use them? Is there any evidence of better performance?

RMSProp is indeed an unpublished method, and in the lecture Geoffrey Hinton gives just the general idea behind RMSProp - to divide the gradient by a moving average of the gradient magnitude. The lecture has disappeared from YouTube but you can find the slides in the end of this PDF:

https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

When this principle is applied to Stochastic Gradient Descent, the update rule you showed is obtained. Since Hinton did not propose an exact algorithm, this principle has been applied to different optimization methods. I agree it's confusing all these methods call themselves RMSProp.

climin implements RMSProp with Nesterov momentum. Momentum methods try to avoid the oscillation that often happens with SGD by slowly changing the current direction of updates. The algorithm given in climin documentation introduces the $\beta$ parameter that controls how much of the previous update direction is retained. Nesterov momentum is implemented by first taking a step towards the previous update direction $v_t$, calculating gradient at that position, using the gradient to obtain the new update direction $v_{t+1}$, and finally updating the parameters. The climin implementation also includes the smoothing term $\iota$ inside the square root for stability (1e-8), even though it's not mentioned in the documentation.

The implementation in the Downhill library is based on the algorithm described in the paper by A. Graves. In the article (equations 38–40) the square of the average gradient is subtracted from the average square gradient. Apparently the idea is to approximate the variance of the gradient instead of its magnitude (recall that the variance of a random variable is equal to the mean of the square minus the square of the mean).


Comparing two images and showing the difference in a new image? I would like to compare two web pages images using computer vision techniques.
Show what are non-unique portions comparing both images. Which part image1 not exist in  image2 vice versa.

A simple and time efficent solution is the computation of the absoulte values of the difference image. This can be done as follows:

import cv2
img_a = cv2.imload([first_image_filename])
img_b = cv2.imload([second_image_filename])
img_diff = cv2.absdiff(img_a, img_b)
cv2.imshow('abs difference', img_diff)
cv2.waitKey(-1)


Hyper-parameter tuning of NaiveBayes Classier I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use NaiveBayes Classifier of sklearn for a task but I'm not sure about the values of the parameters that I should try.

What I want is something like this, but for GaussianNB() classifier and not SVM:

from sklearn.model_selection import GridSearchCV
C=[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1]
gamma=[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]
kernel=['rbf','linear']
hyper={'kernel':kernel,'C':C,'gamma':gamma}
gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)
gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

I've tried to search for examples for NaiveBayes, but couldn't find any. What I have right now is simply this:

model = GaussianNB()

What I want is to try different parameters and compare the scores.

from sklearn.model_selection import GridSearchCV

hyper = {'C':[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1],
         'gamma':[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0],
         'kernel':['rbf','linear']
        }

gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)

gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

Sources:

Hyperparameter Tuning of Machine Learning Model in Python

https://github.com/dataprofessor/code/blob/master/python/hyperparameter_tuning.ipynb
https://www.youtube.com/watch?v=AvWfL1Us3Kg


Random forest model in R - predictors and training data types mismatch I tried the random forest model in my research topic, but I met a problem during the validation phase.

When, I used the final model of random forest to predict on an independent dataset, I received this message:

  Type of predictors in new data do not match that of the training data

So, to detect the different categories in my factors/variables, I used:

levels(Train$Aquifer.media)           levels(Test$Aquifer.media)

For this factor "Aquifer.media", I have:

Train dataset: "Carbonates rocks"  "Crystalline rocks"  "Siliciclastic sedimentary rocks"  "Unconsolisated sediments rocks"  "Volcanic rocks"

Test Dataset: "Crystalline rocks"  "Siliciclastic sedimentary rocks"  "Unconsolisated sediments rocks"  "Volcanic rocks"

I detected that predictors  were of different categories, I would like to know, how I can solve this problem?

Is it possible  to delete some categories in the factors?

Your training set should be true representation of the entire population which is not true in your case. The levels in your train data set's media column has 4 factor levels which is 1 level less than the test data set's media column factor levels. Assuming you are using R, you can fix it with below code

levels(TrainAquifier.media) <- levels(TestAquifier.media)

You can find answer to similar question in stackoverflow here


SciKit-Learn: predict values sometimes different from top predict_proba entries? I noticed that in my text classification problem, I get significantly worse results when I use the max of predict_proba's output as compared to the straight predict value. Digging further, I found that the highest entries in each row of predict_proba don't always have index equal to the entry of predict. Example:

> [*test.target]
[61, 13, 11, 89, 11, 71, 118, 33, 52, 57, 16, 57, 100, 24, ...]
> [*text_clf.predict(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]
> [max(enumerate(prob), key=lambda p: p[1])[0] for prob in text_clf.predict_proba(test.data)]
[60, 16, 11, 16, 11, 87, 25, 32, 16, 56, 16, 115, 11, 25, ...]

You can see that predictions roughly follows test.target (not the greatest accuracy, but cest la vie), and predict_proba roughly follows predict, however for values above 25 there's an offset of 1. And for values above 87 there's an offset of 2. 115/118 and so on. Any idea why this might be? I would expect those two expressions to be identical.

My pipeline:

[
    ("vect", StemmedCountVectorizer(ngram_range=(1, 2), max_df=0.8, min_df=3),),
    ("tfidf", TfidfTransformer(use_idf=True)),
    (
        "clf",
        SGDClassifier(
            loss="modified_huber", penalty="l2", alpha=0.001, random_state=42
        ),
    ),
]

This seems to be a bug in the library.

https://github.com/scikit-learn/scikit-learn/issues/17035

The issue is predict_proba does not convert from its internal representation of categories to the dataset's representation before returning the data. Client-side we can convert using some internal APIs:

> [text_clf._final_estimator.classes_[max(enumerate(prob), key=lambda p: p[1])[0]] for prob in text_clf.predict_proba(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]
> [*text_clf.predict(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]

But the real fix needs to happen in the library.