How are the positions of the output nodes determined in the Kohonen - Self Organizing Maps algorithm? In the Cooperative stage of Kohonen's SOM, the neighborhood for a winning neuron(output node). In most cases, the neighborhood function happens to be the Gaussian Function.
For example,
$$h_j,_i = exp(-d_j,_i^2/2*\sigma^2)$$
where $h_j,_i$ is the window function ($i$ is the index of the winning neuron and $j$ is the index of the encompassing neuron), $d_j,_i$ is the lateral distance between the winning neuron $i$ and the excited neuron $j$.

Now the user who is going to implement the SOM has the input data and the randomly initialized weight vectors. How is $d_j,_i$ determined? Is it randomly initialized too? Also what is the relation between the weight vectors of the output nodes and their positions?
The figure below is just to give a visual representation of the organization of the input and the output nodes

The distance is calculated according to a distance function (euclidian, manhattan, mahalanobis and so on. As this is an unsupervised model (hence the self-organizing part in the name) there are no output neurons. Would you clarify the term?

I'll just recap the training a bit:

Inititalizing: We have a grid (let's assume 2d) of neurons $n_i = (w_i, k_i)$, where $w_i$ is a randomly initialized weight and $k_i$ the position on the grid.

We'll now pick a training sample $x_i$ randomly. For this instance $x_i$ we pick the neuron $n_m$ ($m$ for minimum) where the distance between the weight vector and the instance  is minimal given a distance function $d$, so $n_m = \text{argmin}_{n_j} \; d(x_j, W(n_j))$ where $W(\cdot)$ gives the weight for the respective neuron.
We now pick a set of neurons for which we will adapt the weight vector given a neighbourhood function (for example your gaussian, or a cone). The weights that should have their weights updated are given by $N^{+t} = \{ n_i = (w_i,k_i) \mid d_A(k_m,k_i) \leq \delta^t \}$, where $t$ is the step in time (I'll mention that later), the "reach" $\delta^t$ the neighbourhood should have and of course the position on the grid $k_m$ for the winning neuron.
We then update the weights of the neurons in the neighbourhood of the winning neuron $n_m$ according to some updating rule. This can be interpreted as moving the weights closer to the input we currently look at (because we chose the neuron with the closest weight vector). A possible update rule could be: $w_s^{t+1} = w_m^t + \epsilon^t \cdot h_{mi}^t \cdot (x_j - w_m^t)$, where $\epsilon^t$ is a time-dependent learning rate and $h_{si}^t$ weights the distance from the winning neuron to the neuron we are changing right now, also time-dependent.

In order to somewhat guarantee a topological mapping two things should be considered now:

The learning rate, which is part of the update rule, has to be decreased over time, starting with a relatively high value.
The neighbourhood "radius" has to be decreased as well, also starting from a relatively high value.

Let's visualize this on this picture real quick. The red nodes are inputs. The dark grey is the winning neuron for the input that is considered right now (the picture does not make it clear that we look at one example/input at a time!). The light grey nodes are the ones that are within the neighbourhood of the winning neuron and hence are updated slightly according to some updating rule.

Disclaimer: this was from the top of my head right now, please check any formulas or hypotheses I propose here before using it in any kind of work. (Which you should always do!)
.
See also wikipedia, they also have some explanation of the algorithm that might better suit you.

Image source
Additioanl source (German Wikipedia)


How do data scientists test the claimed accuracy of a model? For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go about verifying this claim?

I would test it on the data that I possess and see what the acc. is for detecting frauds in my data-set, and run a t-test with the sample acc., to see if it is close enough to 90% to make the call that it would detect 90% of all frauds in the entire population.

Let me know if this is the correct way to think about this question.

Thanks

There will already be a historical dataset of Fraudulent transactions (E.g.: If we take Credit Card as an example, all transactions that were disputed and dispute was accepted).

Sales team will include pre-sales developers that can help run a subset of such data through model (After various paperwork such as NDA, Legal review that a specific attribute like Gender/ Race can be used for this purpose). This will usually require :

Extracting data from source systems or warehouse or data lake
Transforming data for the model's input/output format
Quantitative + Subjective validation by Operations team that handles such transactions

Getting 90% accuracy is pretty straight-forward. Since most of the transactions are not fraudulent, any model that predicts all transactions as genuine will meet this bar.

So step #3 (validation) will include agreement on appropriate measure for validation of the model (Such as F1 score)


Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis As a part of my university project, I am researching/developing a sentiment analysis model wherein I am trying to combine NLTK Vader (SentimentIntensityAnalyzer) results with a Machine Learning trained classifier for prediction of Sentiments on Twitter data.

Detailed description -

To explain what I am trying to do is - Combine Machine Learning classifier and NLTK Vader sentiment analysis to get the better classification of tweets as Positive, Negative or Neutral.

What I have done is -

Cleaned the data (Niek Sanders twitter corpus) and pre-processed the tweets which include stop words removal, URL removal, User mention removal, remove the # symbol from Hashtags, lower case conversion, stemmer processing, etc.
Split into 80:20 training: test ratio
Used TfidfVectorizer to create a sparse matrix of features with TFIDF of words. The number of columns is equal to the number of words in clean data.
Use this vector matrix to train and test the classifiers using scikit-learn.

Classifiers used - This same vector matrix is being used to train - KNN, Random forest, Naive Bayes, SVM, Artificial Neural Network and Convolutional Neural Network.

Now the main doubt arises when trying to combine NLTK Vader (SentimentIntensityAnalyzer results).

What I am doing is - From step three above, in the tfidf vector-matrix I am adding 2 columns and I am adding the Positive and Negative polarity result for the tweet given by NLTK, so now the vector-matrix has 2 new columns n+1 (total positive polarity of tweet by NLTK), n+2 (total negative polarity of tweet by NLTK) and it looks like -

    0      1     2     3     4     5     6     7  ......... |  n+1    |   n+2
------------------------------------------------------------|---------|----------
    0.4   0.3   0.4   0.1   0.5   0.3   0.2   0.4 ......... |  0.345  |  0.345
    0.5   0.3   0.2   0.8   0.3   0.6   0.4   0.5 ......... |  0.765  |  0.523
    0.6   0.4   0.1   0.7   0.8   0.8   0.2   0.2 ......... |  0.392  |  0.664
    0.2   0.9   0.7   0.4   0.9   0.9   0.8   0.5 ......... |  0.832  |  0.658
    0.9   0.5   0.9   0.7   0.3   0.2   0.2   0.5 ......... |  0.273  |  0.283
    0.5   0.2   0.2   0.7   0.2   0.1   0.6   0.6 ......... |  0.505  |  0.194
    0.4   0.3   0.2   0.3   0.3   0.9   0.5   0.5 ......... |  0.102  |  0.927
    0.1   0.8   0.1   0.2   0.1   0.5   0.2   0.7 ......... |  0.735  |  0.455

Question - So is it correct to add it this way?

I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

Though it has increased the overall accuracy of the classifiers by small margin of 1-2%.

But am I doing it right?
If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

Interesting approach, but the whole purpose of NLTK Vader is to have a pre-trained model. After all, NLTK Vader was manually (!) labeled.

I just tested Google vs. NLTK Vader on "I did not hate this movie" (negations are notoriously hard to catch for an algorithm) and NLTK Vader did much better than Google. NLTK Vader scored it kind of positive (0.45) while Google scored it negatively (-0.6).

Interesting ... :-)

strong text

Good luck!


Xgboost interpretation: shouldn't cover, frequency, and gain be similar? I was surprised to see the results of my feature importance table from my xgboost model. Based on the tutorials that I've seen online, gain/cover/frequency seems to be somewhat similar (as I would expect because if a variable improves accuracy, shouldn't it increase in frequency as well?) but my numbers are drastically different. Am I perhaps doing something wrong or is my intuition wrong? Thank you in advance!

Feature       Gain      Cover   Frequency

1:    Var1 0.21943765 0.02821822 0.009433962
2:    Var2 0.18207910 0.05509272 0.066037736
3:    Var3 0.10746529 0.22117710 0.216981132
4:    Var4 0.10294292 0.05267401 0.018867925
5:    Var5 0.06928732 0.10185434 0.141509434
6:    Var6 0.05745753 0.05482397 0.047169811

Gain = Total gains of splits which use the feature. (In my opinion, features with high gain are usually the most important features)

Frequency = Numbers of times the feature is used in a model.

In my experience, these values are not usually correlated all of the time. I have had situations where a feature has the most gain but it was barely checked so there wasn't alot of 'frequency'.

Also, binary coded variables don't usually have high frequency because there is only 2 possible values. When it comes continuous variables, the model usually is checking for certain ranges so it needs to look at this feature multiple times usually resulting in high frequency.

For future reference, I usually just check the top 20 features by gain, and top 20 by frequency. If a feature appears in both then it is important in my opinion. Also, I wouldn't really worry about 'cover'. I don't think there is much to learn from that.


How to extract contents by topic from a document? I am trying to extract information from resumes. I tried the pdfminer for the text extraction. But I need to extract the contents from a resume with respect to its title.
For example:
I will be giving my educational details under a title EDUCATIONAL BACKGROUND, so I have to extract the content topic wise.
Is it possible to extract like that?
What will be the process behind that?
Is it possible to approach the problem in a segmentation manner.

pyresparser is useful for extracting information from resumes. I believe this should work in your case.
Check out the more details on the same here https://pypi.org/project/pyresparser/
Let me know if it works!


How to structure data and model for multiclass classification in SVM? I am trying to predict a categorial variable given a set of input variables, which are also categorical. Both the target and features variables only take the class values [below mean, mean, above mean].

I have used one-hot encoding on both the target and feature vectors, but I am not sure if this is correct.

If I have 2 feature vectors I get 2x3 columns after one-hot encoding. For example, the first instance may be:

X[0] = [0,1,0,    1,0,0]

indicating the first feature has mean value ([0,1,0]) and the second feature is below mean ([1,0,0]), with the corresponding target

y[0] = [1,0,0]

indicating the target is below mean.

With the encoded X and y, I have tried to do this:

svm_clf = SVC()
svm_clf.fit(X_train, y_train)
y_pred = svm_clf.predict(X_test)

but it gives me 'ValueError("bad input shape {0}".format(shape))'

I have given the input and output below for the real situation, where the input has 9 features (so 9x3 columns after encoding). The error message indicates there is a problem with the 'y' vector.

Should I be one-hot encoding the target?

How can I tell the classifier that the output can only take a "single value" i.e. the 3 columns in the target are not independent as they all belong to the one variable. For example, the output for a given instance cannot be [1,1,0] as this indicates it is both "below mean" and "mean" which is not possible.

I have also tried a random forest classifier which ran OK, but the results were not plausible, so I assume I am doing something wrong,

Input features:

X_train type:  and shape: (872, 27) and head(5):

[[ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 0.  1.  0.  0.  1.  0.  0.  1.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.]
 [ 1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.  1.  1.  0.  0.]
 [ 1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  1.  0.  1.  0.  0.]]

Target:

y_train type:  and shape: (872, 3) and head(5):

[[ 0.  0.  1.]
 [ 0.  0.  1.]
 [ 1.  0.  0.]
 [ 0.  0.  1.]
 [ 0.  1.  0.]]

Error message:

  File "analyse.py", line 287, in dummy_test
    svm_clf.fit(X_train, y_train)
  File "/usr/lib64/python3.6/site-packages/sklearn/svm/base.py", line 149, in fit
    X, y = check_X_y(X, y, dtype=np.float64, order='C', accept_sparse='csr')
  File "/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 578, in check_X_y
    y = column_or_1d(y, warn=True)
  File "/usr/lib64/python3.6/site-packages/sklearn/utils/validation.py", line 614, in column_or_1d
    raise ValueError("bad input shape {0}".format(shape))
  ValueError: bad input shape (872, 3)

See the svm documentation here. It does support multiclass problem. The only thing is that your y should be like this [0, 1, 2, 3]. If you do not use one-hot encoding and use the y label as it is, it might work.


CNN for image classification with two outputs Is it possible to classify my images (cars parts) by the type of cars part(door, window ...) and also by the view of the image( front, back, right, left, top and bottom). My pictures are labelled like this: View_idPart, the view is a number from 2 to 7. I want to use a CNN model, but i don't know if this is possible? I hope that I will have some answers, I will be so grateful

The view classification (front, back, ...) is the easier part as you have the correct labels in your dataset.
I would do it using transfer learning : pick a pre existing image classification model (such as VGG or Res-Net) and freeze it (parameters.require_grad = false in pytorch layer.trainable = false in keras), remove the last classification layers and replace them with your architecture with the correct number of outputs (6 classes in your case). Then train the network : it should only train the last classification part as we froze the convolutional part. And it should give good results depending on how complicated the initial CNN model is.
Transfer learning is only useful if you do not have much data in your dataset, i'd say it is not necessary to use a pre existing model if you have a lot of data (>100 000) in the dataset. If you have a big dataset, you can create your architecture from scratch and build your convolutional layers with an encoder-classifier architecture as represented below (blue + red are the convolutional layers and green is the classifier part) :

I'm not sure to understand what you mean by type of cars part(door, window ...), it seems to me you want to have another classifier that gives the car parts on the image, so the output should be something like : there is window, a door, but no trunk. This is also possible, but requires a dataset labeled with such information (which does not seem to be the case). Maybe you can find correlations between the view and the parts on the image and deduce it from there. Or if you have such a labeled dataset, just create another CNN and train it on this dataset (output may need to be slighty different if you want to predict different classes for one image).
Anyway, CNN is definitly the way to go for your problem as it is the most accurate models we have at the moment for image classification, and they are likely to perform way better than fully connected layers or other architectures.


Visualization with many lines, colors, and markers I have a bunch of plots as the one reported below. The data is from measurements performed on different times and different days.
In the plot (which is a cumulative distribution function, if that matters), the colors differentiate data relevant to different days; the markers are used to further differentiate the data within each day.
The problem is that the plot is very crowded and a bit ugly. Some markers can be barely seen.
Question: Any idea how I can better plot this data?
As you can see I tried to scatter the markers so that they would not appear in the same position.
The code I used to create the markers (with python/matplotlib):
marker_list = ['$A$', '$B$', '$C$', '$D$', '$E$', '$F$', '$G$', '$H$', '$I$', '$J$',
               '$K$', '$L$', '$M$', '$N$', '$O$', '$P$', '$Q$', '$R$', '$S$', '$T$',
               '$U$', '$V$', '$W$', '$X$', '$Y$', '$Z$', r'$\Gamma$', r'$\Lambda$',
               r'$\Omega$', r'$\Psi$',]

'set marker position'
rnd_lst = [np.random.randint(-10, 10) for i, _ in enumerate(marker_list)]
marker_pos = [[25+x, 45+x, 65+x, 87+x] for x in rnd_lst]    # fixed points + random number to scatter the markers

cc = plt.cycler(marker=marker_list, markevery=marker_pos)
plt.rc('axes', prop_cycle=cc)
plt.rc('lines', markersize=18)

However, it would be nice to "distribute the markers in the y direction". For example, it'd be nice to show the markers only where the CDF is above 0.25.
Question: how to limit the markers to a specific section of the y axis?
EDIT
I guess my best option is to use linestyles instead markers. However, using more than 4 or 5 linestyles is also bad because they will be barely distinguishable from one another. To solve I can use a combination of linestyles and markers when I have more than say 4-5 lines with same color.  The problem with this solution is that I find it not immediate to automate,  at least not by using the cycler as shown above. Any help on this would be also appreciated :)

You might want to look at these kinds of error band line plots in seaborn:

https://seaborn.pydata.org/examples/errorband_lineplots.html
You could compute this for each colour/ day of the week. The central line would be a mean value of each colour/day, the shadow being the distribution.
Then it's probably best to look at days individually to compare the distributions of functions. A box-whisker plot might be good to compare distributions, although this is typically only for 1-D data.
EDIT
You can limit the y axis with plt.ylim(0.25,1) but the previous might look more appealing


Recommendation system with active learning I have data where companies ask users to score a bunch of questions but some items may be randomly chosen while others are personalized. Users score higher in personalized questions on average. I have a user ID, question ID, corresponding score of the question by the user, and whether the question is random or personalized.

I want to build a recommendation system that incorporates the feature of a question being random or personalized.

I assume that for a personalized item to appear there must be some learning of previous random questions the company learned about the user beforehand.

But I got quite lost in terms of how to have a recommendation system that incorporates the dynamic structure.

I know basic recommendation includes matrix factorization or embedding for a user-item matrix but I don’t know how to accommodate the learning.

I would appreciate any insights/pointing to relevant literature/relevant code.

One way to approach recommendation systems that can change over time based on user input is to model them with Reinforcement Learning. There is an inherent notion of time and active choices with Reinforcement Learning.
Contextual-Bandit is a popular approach that models information about the user (context) to influence which options (arms of the bandit) are shown to a user.


What to set in steps_per_epoch in Keras' fit_generator? I am replicating, in Keras, the work of a paper where I know the values of epoch and batch_size. Since the dataset is quite large, I am using fit_generator. I would like to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way?

As mentioned in Keras' webpage about fit_generator():

steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.

You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.


Which machine learning technique for product ranking/scoring I am trying to identify a ML technique to score products based on the number of times the product was "viewed", "clicked" and knowing the "cost per click" for each product. Given the product ID and category ID, how can I proceed to score each product?

I am sure I have to coarse classify them (some have no clicks, but views, some have both, some have none)? Since there are 1000s of products... Any tip?

I guess the technique is also used in e-commerce to design recommender systems, like based on popularity of a product. Any one can shed some light?

*Edit: Though the suggestions here are interesting, still I couldn't figure out best way to do this.

So I am assuming you just want to be pushed in the right direction. There are 2 different ways you can go about this.

Netflix up until very recently did all its recommendations using classical algorithms and setups, see paper on their architecture.

For this type of light recommendation problem I would recommend using something from PredictionIO. It is very versatile and can be used to classify using a variety of inputs. It's also not very hard to learn.

You can also solve this problem using neural nets, it can be viewed as a recommendation by classification. Deep learning is all the jazz now and you can utilize these breakthroughs in the recommender space.

Youtube is the big one when it comes to deep neural nets applied to recommendations, see this paper.

They split their system into 2 separate neural net models. One for candidate generation, and then another for producing the actual recommendations.

Spotify also did some awesome stuff applying Convolutional Neural Nets to the actual audio streams with some equally interesting results:
http://benanne.github.io/2014/08/05/spotify-cnns.html

As far as implementing something like that goes I would look for examples and build in python using either tensorflow or theano and keras. It wouldn't have to be too 'deep'.

Hope that helped!


Why would a validation set wear out slower than a test set? On this page of Google's Machine Learning Crash Course, we find the following statement:

"Test sets and validation sets "wear out" with repeated use. That is, the more you use the same data to make decisions about hyperparameter settings or other model improvements, the less confidence you'll have that these results actually generalize to new, unseen data. Note that validation sets typically wear out more slowly than test sets."

I don't get the last sentence. Why would a validation set wear out slower than a test set? To me it would wear out faster as the validation set is used a lot more frequently than the test set. What am I missing?

It is difficult to say without access to the original author. However, I expect this refers to the ability of using each set to realise its purpose.

A validation set's purpose is to select hyperparameters that perform the best according to some metric. The best measurement on the validation set should always have the highest expectation of being the best in reality. If you make very many measurements, then the absolute probability of the best measurement being the real best could be low, but the chances of a generally poorly performing set of hyperparameters winning overall do not increase as fast. You can be reasonably certain that you have picked "one of the best" plus "the one with highest probability of being the best" even though that might be e.g. just a 10% chance if you have run 100s of validations.

A test set's purpose is to measure a metric without bias. If you use this for model comparison or selection, then this can be affected by maximisation bias - because there is uncertainty in the measurement, focusing on the relative values and picking a "best" almost certainly over-estimates the true value. This effect happens very quickly. If you measure metrics for two sets of hyperparameters and pick the best one, you should already expect that the value you got for the metric is an over-estimate. Note you still expect on average that you have picked the better option, but you cannot trust the measurement as much.


Which is better: GPT or RelGAN for text generation? Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN.

So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN

I am so confused about this question.

According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper:

  This research demonstrates that well-adjusted language models are a remarkably strong baseline
  and that temperature sweeping can provide a very clear characterization of model performance. A
  well-adjusted language model outperforms the considered GAN variants as evaluated on both local,
  and more surprisingly, global metrics of quality and diversity. Our temperature sweeping framework
  shares characteristics with a Receiver Operating Curve. Analogously, if one needed a single scalar to
  compare NLG models, one could compute area under the curve and seek the model with the smallest
  value (lower is better for our considered metrics).

  GAN-based generative models have been proven effective on real-valued data, however, but there
  exist many difficult pernicious issues of moving to discrete data. These issues must be overcome
  before they will improve over the strong MLE baselines. On the datasets and tasks considered,
  potential issues caused by exposure bias were less than the issues of training GANs in discrete data.
  GAN training may prove fruitful eventually, but this research lays forth clear boundaries that it must
  first surpass.

This way, OpenAI's GPT and GPT-2 may be considered superior in text generation quality to current textual GANs.


Predicting high frequency sparse time series data in python I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0's, since there is no continuous flow of cars coming to charge but rather reoccurring charging events as peaks(for example from 7-9 am seems to be a frequent charging timeframe when people are coming to the office) I have also aggregated weather and weekday/holiday data to be used as features.

I now wish to predict the energy demand for a timeframe of 6h in the future. So far I have tried SARIMA with terrible results, since the algorithm seems to be obscured by the sparse data.
I have tried different transformations (Box Cox, Normalization, Standardization), differencing, auto-arima for optimal parameters, so far no luck.
I am willing to try different machine learning as well as statistical algorithms. Does anyone have some recommendations as of what I can do to generate a moderatly accurate prediction with a sparse dataset? (Python)

Here is what you could try doing:

Get a feel of the shape of your data.
Split your time-series by day. Compare the time-series across multiple days.
Plot time in X-axis and the target variable (for forecast) on the Y-axis. Do the shapes look similar or different ? Can you eyeball N-Distinct shapes that the curve takes ?

Find out the "daily mean" of the target variable to forecast. Plot the mean over the course of the year. Do you see a trend or seasonality ? Are you able to get SARIMA or other related techniques to model the changing mean. If so that by itself may be an achievement. Repeat the same exercise for the "daily variance". Forecasting the daily mean / daily variance "may" be a simpler problem to solve - than forecasting the entire time-series.

Normalize your daily data. Subtract the daily series by the "daily mean" and divide by the "daily standard deviation". Run a clustering algorithm (perhaps K-Means) on your daily timeseries. Use the elbow to identify the best number of clusters.

Plot the centroids of your clusters. If you are lucky - you may be able to see distinct shapes that your forecast curve takes.

Use the Cluster Number to label each daily time series. Then use a classification model to predict the correct cluster. The features for the classification model could be "day of week", "is_holiday", "expected_average_temperature_for_the_day" etc etc.

Check if your classification model does a reasonably good job. If it does you are probably in luck. Your classification model assigns probabilities for each cluster. Combine the cluster centroids weighted by the predicted probabilities - to arrive at a predicted curve.

The predicted curve from the previous step is probably normalized - since the data-prep step (Step 3) normalized the data. You now have the task to rescale back to original data. From Step 2 - if you were able to construct a model that does a reasonable job at predicting the "mean" and "variance" for the target day - Then you could do something as simple as:
Final_Curve = Normalized_Curve * sqrt(Forecasted_Variance) + Forecasted_Mean

Let me know if this worked for you.


How to identify and extract patterns from emails I would like to know if it would be possible to identify some patterns in a text.
For example, looking at emails, there is some common words used at the beginning and at the end.

Dear Mr Pascal,

We regret to inform you that we will not be able to respect the deadline previously agreed for the delivery of your order. Our supplier has warned us today that they are experiencing supply problems, which will result in a delay in our production chain. We count on your understanding and thank you for your patience.

Please accept our apologies.

Best regards,

Matt

If I look at emails, they usually start with Dear/Hi/Hello/Good morning..., then the title/name of a person/company; the body; and the conclusion (I look forward to hearing from you; kind regards, best regards;....).
I would like to ask you, therefore, if there is a way to collect information about these patterns and also if it is possible to classify emails by the patterns.

It depends exactly on which kind of patterns you are talking about. Are they deterministic? That is, they are all the same, so you want to get everything after Dear, or before Att / Best Regards, you can explore regular expression patterns. In python, you can use re library:

https://docs.python.org/3/library/re.html

There are books about regular expressions, so this is a big topic and a strong start if you want to check patterns.

Other kind of tasks you could do:

Word Embeddings on raw e-mails to check word similarity.
Clustering e-mails using Bag-of-Words or Word Embeddings.
Label af set o e-mails and train a supervised algorithm to predict an outcome of interest.

Anyway, text mining is really a big discipline, so if you want a more precise answer, I would recommend to narrow down your question.

Hope it helps.


Padding in Keras with output half sized input Here is my Keras model I'm working on:

model = Sequential()
model.add(Conv2D(64, kernel_size=(7, 7), strides = 2, padding = 3,
                 input_shape=input_shape)) # (224,224,64)

model.add(MaxPooling2D(pool_size=(2, 2), strides = 2)) # (112,112,64)
model.add(Conv2D(192, kernel_size = (3,3), padding = 1)) #(112,112,192)
model.add(MaxPooling2D(pool_size = (2,2),strides = 2)) #(56,56,192)
model.add(Conv2D(128, kernel_size = (1,1))) #(56,56,128)
model.add(Conv2D(256, kernel_size = (3,3), padding = 1)) #(56,56,256)
model.add(Conv2D(256, kernel_size = (1,1))) #(56,56,256)
model.add(Conv2D(512, kernel_size = (3,3),padding = 1)) #(56,56,512)
model.add(MaxPooling2D(pool_size = (2,2), strides = 2)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(512, kernel_size = (1,1))) #(28,28,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(28,28,1024)
model.add(MaxPooling2D(pool_size = (2,2), strides = 2)) #(14,14,1024)
model.add(Conv2D(512, kernel_size = (1,1))) #(14,14,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(512, kernel_size = (1,1))) #(14,14,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(1024, kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(1024, kernel_size = (3,3), strides = 2, padding = 3)) #(7,7,1024)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(7,7,1024)
model.add(Conv2D(1024, kernel_size = (3,3), padding = 1)) #(7,7,1024)

model.add(Flatten())
model.add(Dense(4096))
model.add(Dense(7*7*30))
model.add(Reshape(7,7,30))

When I compile it, I got an error for padding because Keras knows only 'same', valid' and 'casual'. I understand these, but I really need padding somewhere to be equal to 3 because my output should be a half of input (we have strides equal to 2). I really don't know how to fix it. How to do padding if we want to half size the input with strides 2?

If you goal is simply to halve the size of your filters, you could think about using some different methods other than padding, such as dilated convolutions. Have a look at this paper for some ideas and nice explanations with pictures. Just thinking about your dimensions quickly, I am not sure you could go from 14 to 7 very easily. getting to 8 or 5 is simple enough though.

One thing that isn't always obvious if you just started learning KEras, is that you can mix in Tensorflow operations directly from the Tensorflow library in with your Keras code.

In TF there is a function called pad, which allows you to specify the padding manually on each side of a tensor. There are also options to say whether the padding is done with zeros, or if the values inside your original tensor are repeated/mirrored (using the mode argument).

You could try using this to pad the layers. I can show you how to pad the tensors to get the effect you want:

from keras.models import Sequential, Model
from keras.layers import (Input, Conv2D, MaxPooling2D,
                          Flatten, Dense, Reshape, Lambda)
import tensorflow as tf

input_shape = (448, 448, 3)
batch_shape = (None,) + input_shape
raw_input = tf.placeholder(dtype='float16', shape=batch_shape)
paddings = tf.constant([[0, 0],   # the batch size dimension
                        [3, 3],   # top and bottom of image
                        [3, 3],   # left and right
                        [0, 0]])  # the channels dimension

padded_input = tf.pad(raw_input, paddings, mode='CONSTANT',
                      constant_values=0.0)  # pads with 0 by default

padded_shape = padded_input.shape     # (454, 454, 3) because we add 2*3 padding

input_layer = Input(padded_shape, batch_shape, tensor=padded_input)
layer0 = Conv2D(192, kernel_size = (3,3), padding='valid')(input_layer)
layer1 = MaxPooling2D(pool_size=(2, 2), strides = 2)(layer0)
layer2 = Conv2D(192, kernel_size = (3,3), padding='valid')(layer1)
layer3 = MaxPooling2D(pool_size = (2,2),strides = 2)(layer2)
layer4 = Conv2D(512, kernel_size = (3,3), padding='valid')(layer3)
layer5 = MaxPooling2D(pool_size = (2,2), strides = 2)(layer4)
layer6 = Conv2D(256, kernel_size = (1,1), padding='valid')(layer5)
# layer6.shape --> [Dimension(None), Dimension(55), Dimension(55), Dimension(256)]

layer6_padded = tf.pad(layer6, paddings, mode='CONSTANT')
layer6_output = Input(input_shape=layer6_padded.shape)(layer6_padded)

# This will end up giving this error at compilation:
# RuntimeError: Graph disconnected: ...,

layer7 = Conv2D(1024, kernel_size=(3, 3), strides=2)(layer6_output)
layer8 = Flatten()(layer7)
layer9 = Dense(4096)(layer7)
layer10 = Dense(7*7*30)(layer8)
output_layer = Reshape((7, 7, 30))(layer10)

# The following both fail to get the graph as we would like it
model = Model(inputs=[input_layer], outputs=[output_layer])
#model = Model(inputs=[input_layer, layer6_output], outputs=[output_layer])

model.compile('adam', 'mse')
model.summary()

I have been unable to then bring this tensor back into the Keras model (as a layer, which is required) because the standard way of using the Input object forces it to be the entry point of the computational graph, but we want the padded tensors to form an intermediary layer.

If you don't force the padded tensors into a Keras layer, attributes will be missing:

# AttributeError: 'Tensor' object has no attribute '_keras_history'

Which you can hack by just adding the attribute from the layer before we padded:

#layer6_output._keras_history = layer6._keras_history

Unfortunately, I still ran into other errors.

Perhaps you can post a new question on StackOverflow asking how to to this, if you can find anything. I did have a quick try using the idea of creating two graphs and then joining them, but didn't succeed.


Subtract Rows of Matrix from rows of another matrix numpy I have two matrix V_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a way to do it using broadcasting?

    list=[]
for v in V_r:
    a=np.linalg.norm(vecs-v,axis=1)
    list.append(a)
M=np.vstack(list)

This is what you're looking for :  np.linalg.norm(vecs-V_r[:,np.newaxis], axis = 2)


ImageDataGenerator - trained with model.fit instead of model.fit_generator I am a beginner in using the ImageDataGenerator from Keras and I accidentally used model.fit instead of model.fit_generator.

def gen_Image_data():
   gen = ImageDataGenerator(
           width_shift_range=0.1,
           horizontal_flip=True)
   return gen

train_gen = gen_Image_data()
test_gen = ImageDataGenerator()

train_samples = train_gen.flow(X,y, batch_size=64)
test_samples = test_gen.flow(X_val, y_val, batch_size=64)

history = model.fit(train_samples, steps_per_epoch = np.ceil(len(X)/64),
              validation_data=(test_samples),
              validation_steps=np.ceil(len(X_val)/64),
              epochs=300, verbose=1, callbacks=[es])

Many thanks for every hint

The data augmentations are defined when you instantiate your data generators. An example is as such:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

Other augmentation techniques can also be used by setting the correct parameters. Refer to this link: https://keras.io/preprocessing/image/


Naive Bayes and Support Vector Machine (NBSVM) Classification I am relatively new to datascience and have a question about NBSVM. I have a two class problem and text data (headlines from the newspaper). I want to use NBSVM to predict whether a headline has the label 0 or 1.

How I understood it, how I have to proceed now:

convert the headlines to a document term matrix
calculate the log-count ratio. As I understood it, these are the probabilities of the individual documents for a class (i.e. the probability that a document is in class 0 or class 1). Please correct me if I'm wrong here.
the log-count ratios then serve as input for the SVM. It inserts the ratios and sets the boundary between the two classes. When new data comes, the SVM tells you to which class the data belongs.

Is this right? Please note that this is only a theoretical procedure, not an implementation.

you use sklearn "CountVectorizer" and "TfidfVectorizer" to covert the text data into vector

    tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5, norm='l2', encoding='latin-1', ngram_range=(1, 2), stop_words='english')
X_train, X_test, y_train, y_test = train_test_split(df['text'], df['class'], random_state = 0)

# vector representations of the text
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

# Building a SVM model
svmmodel = LinearSVC().fit(X_train_tfidf, y_train)


In Pytorch, if I have a 2D tensor, how to iterate over this tensor to get every value changed I have a 2d Tensor, whose size is 1024x1024 and the values in the tensor is 0.3333, 0.6667, and 1.0000, so I would like to change all these values to 0,1,2.

Could some one tell me how to iterate over this tensor.


How to get phrase tables from word alignments? The output of my word alignment file looks as such:

I wish to say with regard to the initiative of the Portuguese Presidency that we support the spirit and the political intention behind it . In bezug auf die Initiative der portugiesischen Präsidentschaft möchte ich zum Ausdruck bringen , daß wir den Geist und die politische Absicht , die dahinter stehen , unterstützen .   0-0 5-1 5-2 2-3 8-4 7-5 11-6 12-7 1-8 0-9 9-10 3-11 10-12 13-13 13-14 14-15 16-16 17-17 18-18 16-19 20-20 21-21 19-22 19-23 22-24 22-25 23-26 15-27 24-28
It may not be an ideal initiative in terms of its structure but we accept Mr President-in-Office , that it is rooted in idealism and for that reason we are inclined to support it .    Von der Struktur her ist es vielleicht keine ideale Initiative , aber , Herr amtierender Ratspräsident , wir akzeptieren , daß sie auf Idealismus fußt , und sind deshalb geneigt , sie mitzutragen .   0-0 11-2 8-3 0-4 3-5 1-6 2-7 5-8 6-9 12-11 17-12 15-13 16-14 16-15 17-16 13-17 14-18 17-19 18-20 19-21 21-22 23-23 21-24 26-25 24-26 29-27 27-28 30-29 31-30 33-31 32-32 34-33

How can I produce the phrase tables that are used by MOSES from this output?

This isn't my area of specialty and I'm not familiar with Moses, but I found this after some searching.

I think you are looking for GIZA++.  You'll see GIZA++ listed in the "Training" section (left menu) on the Moses home page, as the second step. GIZA++ is briefly described in tutorial fashion here. Here are a few tutorial PowerPoint slides: http://www.tc.umn.edu/~bthomson/wordalignment/GIZA.ppt


How to add more features in addition to a 100D word vector I am working on sentiment analysis using tweets text. I am able to build a word vector using Keras text_to_sequence() method with pretrained GloVe embeddings in Python.

The question is how to add extra features beside the created GloVe embeddings and shape it in right way to feed to the embedding layer and the algorithm in general? For example, assuming the embedding's length is 100 and the extra features are 3, will appending the 3 features to 100 vector resulting in 103 vector length is the right approach? If so how would you implement it.

Where:

corpusAsList is the corpus as text in a list.

maxlen is the maximum number of columns to pad the word sequence with, so that all vectors have the same length.

maxVocabulary is the maximum amount of words to be used in the sequence and ignore the rest of the words.

And tokenizer is used to represent a word as a digit in the vocabulary.

For instance, if we called texts_to_sequences() like this:

   # Create tokenized corpuss
    listTokenized = tokenizer.texts_to_sequences(corpusAsList)

    # Sequence dataset
    sequenceMatrixPrediction = sequence.pad_sequences(listTokenized, maxlen=maxlen)

And then trained a Bidirectional LSTM model like this:

def getLSTMModel(maxlen, maxVocabulary, embedSizeLSTM=100, weights=None):

    inp = Input(shape=(maxlen,))
    x = Embedding(maxVocabulary, embedSizeLSTM,
                 weights=[weights], trainable=False)(inp)
    x = Bidirectional(LSTM(80, return_sequences=True))(x)
    x = GlobalMaxPool1D()(x)
    x = Dropout(0.25)(x)
    x = Dense(70, activation="relu")(x)
    x = Dropout(0.25)(x)
    x = Dense(1, activation="sigmoid")(x)
    model = Model(inputs=inp, outputs=x)
    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

return model

How could we add three extra features to this model?

You can add a second Input layer to your architecture.

Look at this link Keras Guide Multiple Inputs for a more detailed explanation.

The output of the embedding layer can be combined with the secondary input and passed to the next layer.

auxiliary_input = Input(shape=(5,), name='aux_input')
x = keras.layers.concatenate([lstm_out, auxiliary_input])


Why does putting a 1/2 in front of the squared error make the math easier? Per wiki, the mean squared error (MSE) looks like:

$$
\operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}
$$

The professor added a $1\over2$ in front of the formula and explained it a little bit. I am a little bit confused. How does putting a $1\over2$ in front of the squared error make the math easier?

A major reason for using MSE is to optimize the parameters of a regression model. From calculus, you know how to find the minimum of a function by taking the derivative. That puts a "2" out in front, which is irritating to keep writing, so it is reasonable to put a "1/2" at the beginning so the derivative doesn't need a constant out front.

We get away with it because the minimum of f(x) and f(x)/2 is achieved at the same value(s) of x.


Looking for an advice based on an output of a LSTM model I have fitted a LSTM model first using keras.
Data:I have a time series with 560 observations. From that I trained the model using first 500 observations and then evaluated the model using last 60 observations.
I have only one predictor(X) where X is the lag price of the time series. So basically I am trying to predict the current price from the previous days price by fitting a LSTM model.
I used the following setup to fit the model:
model = Sequential()
model.add(LSTM(32,  activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.add(Dropout(0.3))

model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)

Then I evaluated the model for test data. The comparison between the actual price and the predicted price for test data looks like this:

Based on the above plot, It seems that the model is kind of capturing the shape of the actual price. But it is underestimating the price all the time.
Can anyone suggest anything to improve this model?
I am beginner to model fitting in deep learning. So any help will be highly useful.
Thank you.
Update
I tried following setup and the results improved drastically:
model = Sequential()
model.add(LSTM(50,  activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)

However it seems that the predicted price capturing the actual trend after some delay. Will it be possible to fix this?
Any advice will be really helpful.
*Edit
I am updating the question with the code that I used to obtain the fitted values and make the plot.
First I Stored the predictions based on test data:
 y_pred = model.predict(dataXtest)
    y_pred.shape

When I am fitting the model, I reshaped the data so that it is a subsequence of 2 observations. (For an example 500 training observations are reshaped as (250,2,1). Also the 60 test data reshaped as (30,2,1) )
Then I reshaped the predictions to the original form.
y_pred=y_pred.reshape(60,1)
dataYtest=dataYtest.reshape(60,1)
pred_data=pd.concat([dataYtest,y_pred],axis=1)
pred_data.head(10)

0   5933    5837.184570
1   5991    5851.091797
2   6023    5947.239258
3   5972    5939.657227
4   6028    5928.400391
5   5429    5944.728516
6   5697    5390.026367
7   5841    5619.818359
8   5980    5798.516602
9   6548    5897.926758
10  6928    6499.494141

This is how I plotted the data:
plt.figure(figsize=(14,5))
plt.plot(dataYtest, color = 'red', label = 'Actual Price')
plt.plot(y_pred, color = 'blue', label = 'Predicted Price')
plt.title('Test data')
plt.xlabel('Time')
plt.ylabel('Price')
plt.legend()
plt.show()

It looks like your indices for the predicted data are off by one. You could try to fix it with
predicted = predicted[1:]


how to create multiple plot from a panda Dataframe I want to plot multiple plots.
The data is stored in a pandas dataframe and each row should be a seperate plot.
Each row has an ID (ZRD_ID) which doenst matter and a date (TAG) and 24 values to be plotted.

import pandas as pd
import numpy as np

df = pd.read_csv('./Result_set_edited.csv')
df = df.drop("ZRD_ID", axis=1).drop("TAG", axis=1)

x = df.iloc[[0]]
print(df.head())

returns:

     W01     W02     W03    W04    W05  ...     W20    W21    W22    W23    W24
0  72616  156076  141025  72629  72631  ...       0      0      0      0      0
1  67114  171650  139920  67291  67292  ...  172924  93511  72445  72445  72445
2  66893  161919  134041  66913  66911  ...  166244  86672  67114  67120  67124
3  66603  171297  134227  66615  66631  ...  166078  86622  66871  66877  66879
4  66759  167198  133523  67126  67128  ...  163999  74525  66562  66568  66574

To start easier, since I am really new to this I thought of plotting the first line alone first.

since the columns are named 'W01', 'W02', ... , 'W24' I thought I coould use them as labels for the x-axis.
I just didn't find a way to do so, since its the header of the df I guess.
So i created a new array and tried to plot it with the first row of my Dataframe:

y = np.arange(0,24,1)
y.reshape(1,24)
print(y)
print(df.iloc[[0]].values)
plt.plot(y, x)
plt.show()

when trying to plot my values I get the following Error:

  ValueError: x and y must have same first dimension, but have shapes
  (24,) and (1, 24)

thanksfor the help on how to fix the Error for plotting the first line.

PS: I would appreciate some hints on how to improve my question, since it is my first one.

Cheers

If I correctly got what you meant: you want to plot the first row of all the columns?
I guess this could work.
x= range(0,24)
y = df.iloc[0,:].values
xticks = df.columns.tolist()
I'm assuming you want a line plot

plt.plot(x,y)
plt.xticks(xticks) # to change the x ticks of the graph.
plt.show()

The '0' signifies the first row and ':' for all the columns.


How to Define a Cost Fucntion? I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic.
Problem is I dont have any idea where to start not even sure what kind of data i should collect to make a cost function.
Please share how to proceed in this case. I understand the point after defining cost function gradient descent can be used to find , but i have no idea how to define such a cost function.

well there are two things.

You write your learning algorithm from scratch. Then you define your cost function yourself. No one usually does that unless they are asked for.
Using Machine Learning libraries in Python (Most common is SKlearn unless you want to use Neural Networks in e.g. Keras). They have their own cost functions designed. You just need to define your problem properly then it does the thing for you.

In (2) just define your problem properly. The question is fuzzy (what does "not generating traffic good traffic" mean?) but you input your data to your algorithm and it optimize the cost function according to the chosen algorithm. If you need more input, please update the question and give more details so I can help.


Why TF-IDF is working with Sentiment Analysis? Word2vec looks excellent to me as representation of corpus for sentiment analysis. It has relations between words etc. TF-IDF has only weight of the word how important it is. Results with sentiment analysis using both of these representation are quite similar ~90%

Why TF-IDF has such a good results?

One important factor to take into account is how you use the numerical representation of words / embeddings from either TF-IDF or Word2Vec to then compute sentiments. Without knowing how you do this, it is difficult to give a concrete answer. Also, which task are you working on, what does a result of 90% mean?

Regardless of how you compute TF-IDF (there are several definitions - shown below), it is essentially assigning a numerical value to a word, thus creating a mappng of sorts. Word2Vec technically created an embedding, as it maps individual words into a vector space.

Images taken from here

The final TF_IDF is simply the multiplication of the term frequency and the inverse document frequency.

I won't go into details as to how the vectors in Word2Vec are computed, but they also define a way to assign a numerical vector (an embedding) to a single word. In essence, both of these are saying how important a word is, in the context of your documents (your corpus), with Word2Vec also having the interpretability of comparison between word vectors. For example, doing this with the associated vectors of the words actually works really well:

King - Man + Woman = Queen

Perhaps the sentiments you are computing, based on either of these methods , does something similar to taking an average over many words appearing in a sentence, and you end up with a normalised and similar results.

TF-IDF takes a more intuitive approach, looking at how many times a word appears in general, in how many of the documents does it appear and how many times. Word2Vec instead looks at which words often appear together (there is a related quote that normally gets brought up here: "You can judge a person by the company they keep"). So the intuition behind each on is slightly different, but you have numerical values for each word. Perhaps a closer comparison of TF-IDF would be to look at Doc2Vec.

There is also the GLoVE embedding model, which scores well in many NLP tasks - on the same level as Word2Vec embeddings.


Differences between Test & Score results and calculated by Confusion Matrix (accuracy, sensitivity and specificity) We are using Orange and have 2 files (training and testing). We apply different Learners (kNN, AdaBoost...) and get the Evaluation Results. But we have some doubts about some options in the Test & Score window.

When we apply the option "Test on train data" does it generate for each Learner a model and then applies it to the training population?

And the same about the "Test on test data", does it generate the model with the training data and this option applies it to the testing population?

Because when we get the confusion matrices and we calculate accuracy, sensitivity and specificity the values differ from the ones in the Evaluation Results.

Test on train data uses the whole data set for training and then for testing. This method practically always gives wrong results.

So, yes, in this case, the model is learned and tested on the same data set. In most cases, the model learned this way will be overfitted and will perform poorly on unseen data (e.g., your testing dataset).

  Test on test data: the above methods use the data from Data signal only. To input another data set with testing examples (for instance from another file or some data selected in another widget), we select Separate Test Data signal in the communication channel and select Test on test data.

So, the answer to the second question is also "yes:" The model is trained on the data labeled Training in you diagram and then tested on the Testing data. The "quality" of the data and the way it was split in the training and testing set will influence the performance of the model.

Orange documentation on "Test & Score":


How does inverse of logistic function produces "linear relationship", (so we can use least-squares) I am reading about time-series analysis in "A First Course on
Time Series Analysis".  The book reviews the logistic function ($f_{log}(t)$).

Part 1.6 (PDF page 16; book page 8; screenshot below) explains how the inverse of the logistic function ($1/f_{log}(t)$) produces a linear relationship; so that "it can serve as a basis for estimating the parameters $β_1$, $β_2$, $β_3$ by an appropriate linear least squares approach"

I don't understand the last sentence:

  This means that there is a linear relationship among $1/f_{log}(t)$.

I specifically don't understand how the inverse of a logistic function, can produce a "linear relationship"; it doesn't look like a "straight line" when I graph it:

import math
import numpy as np
from matplotlib import pyplot as plt

get_logistic = lambda b1, b2, b3: lambda x: (b3/(1+b2*np.exp(-b1*x)))

x = np.linspace(-10,10)
y = get_logistic(1,1,1)(x)

plt.plot(x,y)
plt.show()
plt.plot(x,1/y)
plt.show()

It's true, the function $1/f_{log}(t)$ is not linear, in other words there is not a linear relationship between $t$ and $1/f_{log}(t)$ but that's not what the author meant; the the important word is "among";

This means that there is a linear relationship among $1/f_{log}(t)$.

So there are relationship among the y values; specifically, there is a linear relationship between $1/f_{log}(t-1)$ and $1/f_{log}(t)$.
So we can change the code to plt.plot(1/y_of_x_minus_1, 1/y), and we will see a straight line:
import math
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd

get_logistic = lambda b1, b2, b3: lambda x: (b3/(1+b2*np.exp(-b1*x)))

logistic_args = (1,1,1)
logistic_function = get_logistic(*logistic_args)

x = np.linspace(-10,10)
x_minus_1 = x-1

y = logistic_function(x)
y_of_x_minus_1 = logistic_function(x_minus_1)

plt.plot(1/y_of_x_minus_1, 1/y)
plt.show()
pd.DataFrame({'x': x, 'x_less_1': x_minus_1, 'y_for_x_less_1': 1/y_of_x_minus_1, 'y': 1/y})

Notice the straight line:

EDIT
The linear relationship among $y$ values helps us estimate the parameters for the logistic function, using a linear least squares approach:

This (linear relationship) can serve as a basis for estimating the parameters $β_1, β_2, β_3$ by an appropriate linear least squares approach, see Exercises 1.2 and 1.3.

And the exercise 1.2 asks:

Then we have the linear regression model
$z_t = a + bz_{t−1} + ε_t$ , where $ε_t$ is the error variable. Compute the least squares estimates $\hat{a}$, $\hat{b}$

And the least-squares formula for the general case of $y=β_0+β_1x+ε$ goes like this (note: beta's mean something different than the three beta's of logistic function, above) (same formulas shown here and here)

$\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n} (x_i-\bar{x})^2}$
$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$

I will try to come back and edit the answer to demonstrate how the linear relationship lets us solve for $\hat{\beta_0}$ and $\hat{\beta_1}$ (aka $\hat{a}$ and $\hat{b}$), which lets us figure out the 3 beta's of the logistic function, as suggested in the exercise 1.2:

Compute the least squares estimates $\hat{a}$, $\hat{b}$ of $a$, $b$ and motivate the estimates $\hat{β_1}:= −log(\hat{b})$, $\hat{β_3} := (1 − exp(−\hat{β}_1))/\hat{a}$ as well as $\hat{β_2} := ...$

For an exercise I tried to change the itl.nist.gov form ....

$\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})$

...to the wikipedia form...

$\sum x_i y_i - \frac{1}{n} \sum x_i \sum y_i$

Starting like this...
$\sum (x_i-\bar{x})(y_i-\bar{y})$
...then, multiply the binomials...
$\sum (x_iy_i - x_i\bar{y} - y_i\bar{x} + \bar{x}\bar{y})$
...then, distributing the sum...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + \sum \bar{x}\bar{y}$
...and because it sums over all $n$ values ($\sum_{i=1}^{n}$), the same $n$ values used for the means/expected values of $x$ and $y$ ($\hat{x}$, and $\hat{y}$) ...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + n \bar{x}\bar{y}$
... then, we transform $n\bar{x}$ into $n \frac{\sum{x_i}}{n}$ then into $\sum x_i$ (we could instead transform the $n\bar{y}$; would reach the same end result)...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + \sum x_i\bar{y}$
... then two terms will cancel....
$\sum x_iy_i  - \sum y_i\bar{x} $
... then, the wikipedia form (which is almost the covariance, aka the $
 \text{Cov}[x,y]$ (just imagine another $\frac{1}{n}$ distributed among the terms)....
$\sum x_iy_i  - \frac{1}{n}\sum y_i\sum x_i $


Policy Gradient custom loss function not working I was experimenting with my policy gradient reinforcement learning algorithm, and I was wondering if I could use a similar method to the supervised cross-entropy. So, instead of using existing labels, I would generate a label for every step in the trajectory.

Depending on the value of the action, I would shift the stochastic policy (Neural Network) output to a more efficient output and train it as a label to a cross entropy loss function.

Example of an action:
Real output: [ 0.2, 0.8 ]; Value: [ -0.5 ]; Action taken: [ 1 ] (0.8 probability).
Created label: [ 0.3, 0.7 ] (Second action not that great, reduce it's probability by a little bit)

My method didn't work, and I am really curious to know why.

There could be many reasons. Custom loss functions are difficult to get right.

One conceptual problem is that the policy of the agent should not be the label. The label should be the reward signal from the environment. If the label is the reward signal from the environment, the agent would learn which policy would maxim rewards. If the policy of the agent is the reward, the agent would "chase its own tail" and not learn environmental rewards


How to Combine tfidf with LSTM in keras? I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using keras.text_to_sequences function.
But now I want to use TfIdf with the LSTM can anyone tell me or share the code how to do it. Please also guide me if it is possible and good approach or not.
If you are wondering why I would like to do this there are two reasons:

I want to see if this improves the results.
Second, my professor asked me to perform Latent Dirichlet Allocation, and use same features for both of the tasks.

The goal of text_to_sequence + embedding in traditional LSTM is to transform text to word vectors.

If you already have the tfidf transformation, the idea is usually to get rid of the embedding layer in your LSTM when you are constructing the model, and directly connect the input (i.e., tfidf matrix) to the layer followed by the embedding layer.

Not sure if it's a good approach but that's for you to figure it out :P.


How to build a model where multiple data points contribute to a result I’m trying to figure out how to massage data and model the following scenario:

Customers at a restaurant rate the quality of the service between 1-10.

I have data on individual interactions between the servers and customers. Say - length of interaction, type of interaction (refilling beverage, ordering, cleaning, etc).

Hypothesis here is each interaction contributes to the final score. I want to build a model that tells me given an interaction, how does it move the score.

My intuition is if I arranged the data as individual interactions, with output of final score, that’ll give me what I want. Is that true?

My intuition is if I arranged the data as individual interactions, with output of final score, that’ll give me what I want. Is that true?

If I understand correctly, your goal is to predict the final score that a customer gives at the end of their dinner based on all the interactions they had during the dinner, right? If yes then I don't think you can keep individual interactions as instances, because then (1) the model would only be able to use one interaction to predict the score and (2) the model would predict a different score for each interaction.

So each instance should represent the full meal and somehow contain as features all the possible information extracted from the interactions. For example an instance could contain the number of interactions, total length, number of refills, etc.

(side note: I wouldn't like eating in a restaurant where customers and staff are monitored so closely, but maybe that's just me)


What's the purpose of padding with Maxpooling? As mentioned in the question, i've noticed that sometimes there are pooling layers with padding.

More specifically, I found this Keras tutorial, where there's a net which contains MaxPooling layers with padding.

If padding=same in convolutional layers, our output size (at least height and width, depth can change based on the number of filters) is the same as the input.

I expected the same with the MaxPooling layer, but Keras model.summary() (as shown in the article) shows that the output size after the pooling layers is half of the input.

What's the point of adding padding to the Pooling layer if we still get an output which is half of the input?

The whole purpose of pooling layers is to reduce the spatial dimensions (height and width). Therefore, padding is not used to prevent a spatial size reduction like it is often for convolutional layers. Instead padding might be required to process inputs with a shape that does not perfectly fit kernel size and stride of the pooling layer.

This is an example where it perfectly fits and your pooling layer does not require any padding:

Side note: The output dimensions are calculated using the usual formula of $O=\frac{I-K+2P}{S}+1$ with $I$ as input size, $K$ as kernel size, $P$ as padding and $S$ as stride.

However, lets take another example where it does not fit as nicely:

Here you need padding since your input size is not an integer multiple of your kernel size. Therefore, you need to add padding on one side in order make it work.

So padding="same" in Keras does not mean the spatial dimensions do not change. It just means that padding is added as required to make up for overlaps when the input size and kernel size do not perfectly fit.

Also this question for a discussion what the difference between same and valid padding is for pooling layers.


Error not decreasing in a 3 layer deep CNN using TensorFlow I'm trying to train a CNN to play an online game by feeding images of the game along with the keyboard input.

By playing the game for some time and collecting the data, I gathered 342 images with size 110x42. I'm feeding these images in the network like so:

def convolutional_neural_network(x):
    weights = {'W_conv1': tf.Variable(tf.random_normal([3, 3, 1, 16])),
               'W_conv2': tf.Variable(tf.random_normal([5, 5, 16, 32])),
               'W_conv3': tf.Variable(tf.random_normal([5, 5, 32, 64])),
               'W_conv4': tf.Variable(tf.random_normal([5, 5, 64, 64])),
               'W_fc': tf.Variable(tf.random_normal([7 * 3 * 64, 1024])),
               'out': tf.Variable(tf.random_normal([1024, n_classes]))}

    biases = {'b_conv1': tf.Variable(tf.random_normal([16])),
              'b_conv2': tf.Variable(tf.random_normal([32])),
              'b_conv3': tf.Variable(tf.random_normal([64])),
              'b_conv4': tf.Variable(tf.random_normal([64])),
              'b_fc': tf.Variable(tf.random_normal([1024])),
              'out': tf.Variable(tf.random_normal([n_classes]))}

    x = tf.reshape(x, shape=[-1, 110, 42, 1])

    conv1 = tf.nn.relu(conv2d(x, weights['W_conv1']) + biases['b_conv1'])
    conv1 = maxpool2d(conv1)

    conv2 = tf.nn.relu(conv2d(conv1, weights['W_conv2']) + biases['b_conv2'])
    conv2 = maxpool2d(conv2)

    conv3 = tf.nn.relu(conv2d(conv2, weights['W_conv3']) + biases['b_conv3'])
    conv3 = maxpool2d(conv3)

    conv4 = tf.nn.relu(conv2d(conv3, weights['W_conv4']) + biases['b_conv4'])
    conv4 = maxpool2d(conv4)

    fc = tf.reshape(conv3, [-1, 7 * 3 * 64])
    fc = tf.nn.relu(tf.matmul(fc, weights['W_fc']) + biases['b_fc'])
    fc = tf.nn.dropout(fc, keep_rate)

    output = tf.sigmoid(tf.add(tf.matmul(fc, weights['out']), biases['out'], name='pred'))

    return output

def train_neural_network(x):
    prediction = convolutional_neural_network(x)
    cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=prediction, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=1).minimize(cost)

    hm_epochs = 6
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())

        for epoch in range(hm_epochs):
            epoch_loss = 0
            for epoch_x, epoch_y, i in dataset.create_batches():
                epoch_x = epoch_x.reshape(-1,4620)
                _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})
                epoch_loss += c

            print('Epoch', epoch, 'completed out of', hm_epochs, 'loss:', epoch_loss)

        #correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
        correct = tf.equal(tf.greater(prediction, 0.5), tf.equal(y, 1.0))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy:', accuracy.eval({x: dataset.dataset['test']['x_test'], y: dataset.dataset['test']['y_test']}))

train_neural_network(x)

The errors are getting stuck in a particular value and floating up and down around this value by a small amount.

I've tried incresing/decreasing the learning rate, improving the quality of images, changing the size of the batches... and nothing seems to make the network stable.

Do you guys know what I'm doing wrong?

Two things:

You're squashing the outputs via the sigmoid function before calculating the loss via tf.nn.sigmoid_cross_entropy_with_logits. This loss function takes the class logits as inputs, meaning you should be passing in the output from the linear layer without any non-linearity activation functions applied after. NOTE that applying a sigmoid function to the logits is a regularization technique used to diminish the effect of the model outputting very large logits, but in a simple three layer CNN like you've got going on here, using this technique is going to hurt more than help.
You should be annealing the learning rate (rather than using a constant learning rate every epoch). As optimization starts to converge on a set of parameters, the parameter update magnitudes should get smaller and smaller otherwise you're very likely to jump past and around where you want to be.

Out of curiosity, what accuracy/loss are you getting?


Is it possible to connect three neural networks in Matlab? If I have 3 separate feedforward neural networks in Matlab, is it possible to connect them so that, given input data and target data the 3 work in parallel to produce output? If so, how do I do this?

If you want to combine the results from three different Neural Networks to "boost" the performance :) , you might want to look at the different Ensemble Learning Methods as I mentioned earlier.

Which method you should use, depends on how you share or divide the training data between the three NNs. For example if the NNs are trained on same data but have different parameters, you can look at simple voting ( if you are doing a classification task) or averaging ( if you are using them for regression).

The more advanced methods like AdaBoost divide the training data between the classifiers. You can read about it in Boosting Neural Networks


Automatic annotation of medical text data I have a dataset of 30000 whole-genome sequence analysis. For each sample I have a text description that does not follow any fixed format.
What I want, is an annotation of each sample indicating the disease and the particular tissue that characterize the sample.

I'm not into text mining, so I don't know which tools I can use. Any suggestions?

You could use linear regression on the genome sequence to predict the occurrence of words in the description. More specifically:

Use dummy variables to encode the genome sequence.
Use stemming to make different conjugations of the same word the same.
Use a bag-of-words representation to represent the words.
Use a scaling of the word counts $w_i$ like $\log(w_i+1)$ or the more advanced TF-IDF.
Since you have a lot of independent variables (maybe more than the number of records?) you should use some regularization of the model. Lasso would be a good choice if you want a sparse model, use ridge regression if you want to put a zero prior on the coefficients.

This method you can use to predict which words are typical for a gene sequence and, hence, characterize the sequence.

You could use the intermediate result of the linear model to see what tissue is important for the prediction. The important ones are the dummy variables that are "on" and have high coefficients. Because you have multiple outputs you could simplify this by only using the coefficients of the top-n likely words.


Metric (other than RMSE, MSE, etc.) to select the best model to better detect peaks in weather forecasting I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know.

When I looked at the results, second model (RMSE = 0.15) was better able to detect some of the peaks than the first one (RMSE = 0.1). For example look at the two results:

Best model (RMSE = 0.1):

Here is the second model's result(RMSE = 0.15):

The model which is able to detect all the peaks, as much as possible (although its RMSE is not the least among all of the models), is more preferable than a model which has less RMSE but is not able to detect peaks.
I searched through the web but didn't find what I was looking for.

Can anyone suggest me a code to evaluate the results of the models, based on their ability to detect peaks better?

Suppose the results are simply 2 arrays:

import numpy as np
np.random.seed(10)
predicted_1 = np.random.rand(10, 1)
predicted_2 = np.random.rand(10, 1)
original = np.random.rand(10, 1)

I had this problem in my work before and used weighted RMSE and assigned higher weights to the peaks.

Some ideas I took from here https://stats.stackexchange.com/questions/230517/weighted-root-mean-square-error


Modelling Unevenly Spaced Time Series I have a continuous variable, sampled over a period of a year at irregular intervals. Some days have more than one observation per hour, while other periods have nothing for days. This makes it particularly difficult to detect patterns in the time series, because some months (for instance October) are highly sampled, while others are not.

My question is what would be the best approach to model this time series?

I believe most time series analysis techniques (like ARMA) need a fixed frequency. I could aggregate the data, in order to have a constant sample or choose a sub-set of the data that is very detailed. With both options I would be missing some information from the original dataset, that could unveil distinct patterns.
Instead of decomposing the series in cycles, I could feed the model
with the entire dataset and expect it to pick up the patterns. For
instance, I transformed the hour, weekday and month in categorical
variables and tried a multiple regression with good results (R2=0.71)

I have the idea that machine learning techniques such as ANN can also pick these patterns from uneven time series, but I was wondering if anybody has tried that, and could provide me some advice about the best way of representing time patterns in a Neural network.

ARIMA, Exponential Smoothing and others indeed require evenly spaced sample points. As you write, you could bucketize your data (say into days), but as you also write, you would lose information. In addition, you may end up with missing values, so you would need to impute, since ARIMA is not very good at handling missing values.

One alternative, as you again write, is to feed time dummies into a regression framework. I personally do not really like categorical dummies, because this implies a sharp cutoff between neighboring categories. This is usually not very natural. So I would rather look at periodic splines with different periodicities. This approach has the advantage of dealing with your uneven sampling and also with missing values.

Be very careful about interpreting $R^2$. In-sample fit is notoriously misleading as a measure of out-of-sample forecast accuracy (see here). I would argue that this disconnect between in-sample fit and out-of-sample forecast accuracy also means that there is no connection between in-sample fit and how well a model "understood" the data, even if your interest lies not in forecasting, but only in modeling per se. My philosophy is that if you can't forecast a time series well, you haven't understood it in any meaningful sense.

Finally, don't overdo the modeling. Just from eyeballing your data, it is obvious that something happened in June, on one day in August and in September/October. I suggest you first find out what this something was and include this in your model, e.g., as explanatory variables (which you can include in ARIMAX if you want to). What happened there is obviously not seasonality.


How to calculate Cumulative Sum with Groupby in Python? I am trying to calculate cumulative sum with groupby using Pandas's DataFrame. However, I don't get expected output.

My Source Code:

import pandas as pd
Employee = [['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
        ['CSE', 'CSE', 'EEE', 'EEE', 'CE', 'CE', 'ME', 'ME'],
        ['Cat-1', 'Cat-2', 'Cat-1', 'Cat-2', 'Cat-1', 'Cat-2', 'Cat-1', 'Cat-2']]

index = pd.MultiIndex.from_arrays(Employee, names=['Name', 'Dept', 'Category'])

Scale = [1, 2, 2, 3, 3, 1, 2, 3]
Salary = [100, 200, 200, 300, 300, 100, 200, 300]

df = pd.DataFrame({'scale': Scale,
               'salary': Salary},
              index=index)

df1 = df.groupby(['Category', 'scale']).cumsum()
print(df1)

Expected Output:

Cat-1    1         100
         2         500
         3         800
Cat-2    1         100
         2         300
         3         900

Obtained Result:

Name Dept Category
A    CSE  Cat-1        100
B    CSE  Cat-2        200
C    EEE  Cat-1        300
D    EEE  Cat-2        500
E    CE   Cat-1        600
F    CE   Cat-2        600
G    ME   Cat-1       1000
H    ME   Cat-2       1200

Groupby doesn't work. However, if I use sum() (i.e. df1 = df.groupby(['Category', 'scale']).sum()) instead of cumsum(), groupby works perfectly.

There are multiple entries for each group so you need to aggregate the data twice, in other words, use groupby twice. Once to get the sum for each group and once to calculate the cumulative sum of these sums.

It can be done as follows:

df.groupby(['Category','scale']).sum().groupby('Category').cumsum()

Note that the cumsum should be applied on groups as partitioned by the Category column only to get the desired result.


Replace all numeric values in a pyspark dataframe by a constant value Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the dataframe by a constant numeric value (for example by the value 1)? Thanks in advance!

Example for the pyspark dataframe:
$$
\begin{array}{c|lcr}
& \text{c1} & \text{c2} & \text{c3} \\
\hline
1 & 0.04 & 1 & 1.35 \\
2 & -1 & null & -1.2 \\
3 & null & 1.2 & null
\end{array}
$$

The result should be:

$$
\begin{array}{c|lcr}
& \text{c1} & \text{c2} & \text{c3} \\
\hline
1 & 1 & 1 & 1 \\
2 & 1 & null & 1 \\
3 & null & 1 & null
\end{array}
$$

As per your problem, I think it might be easier to use lit. Try this-

from pyspark.sql.functions import lit
new_df = df.withColumn('column_name', lit(1))

Hope it helps!


Why image is more blurred through PIL? I was using PIL library to open an image and then convert it into array later on for DL operations. However I found out that image opened was blurry as opposed to the original image. Why is the image coming out the way it is? and what can be done to fix it? If a possible solution is present in OpenCV that'd be welcome as well.

I tried to use OpenCV as well but I was getting an Assertion Error i.e

error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

Code:

from PIL import Image
import numpy as np
import pandas as pd
import os
import cv2
import keras
from keras.models import Sequential
from keras.layers import Conv2D,MaxPooling2D,Dense,Flatten,Dropout
import matplotlib.pyplot as plt

image=Image.open("../content/cell_images/Parasitized/"+i)
size_image = image.resize((50, 50))
print("image")
plot(image,count)
print("resize image")
plot(size_image,count)

Here is the code for plotting:

def plot(img, count):
    w=10
    h=10
    fig=plt.figure(figsize=(8, 8))
    img = np.random.randint(10, size=(h,w))
    fig.add_subplot(rows, columns, count)
    plt.imshow(img)
    plt.show()

Some parts of the code are missing that might contain clues (e.g. you plot method would only plot random noise, as it stands). What might be happening is that you are showing the image in matplotlib in a figure that is much smaller than the image, so matplotlib will automatically scale the image to the pixel space.

The plt.imshow() method can take an argument interpolation, which can produce dramatically different effects. Here are them all compared side-by-side:

Have a look at the relevant documentation

I personally stick to using OpenCV and cv2.imread() to load images. You just have to be aware (in the context of using Tensorflow/Keras) that the channel are the first dimension. i.e. (Channels, Height, Width) - whereas e.g. plt.imread() will return (Height, Width, Channels).


How to divide MNIST training dataset into subparts? Background: I am trying to train different sub-samples of MNIST dataset(1 model for 10,000 , another model for 20,000 samples, and so on until a model for all 60,000 training samples) using Caffe.

Question: How do I sub-split the training dataset of 60,000 samples keeping the format of the training files same(intact) as originally published on http://yann.lecun.com/exdb/mnist/

I think you can just read each line of the file which gives you one data point and write it into an output file. Run a loop for the number of data points you want.


How to select variables based on the mean correlation in a correlation matrix? I have a set of independent variables and I am calculating the correlation matrix between them using the Pearson Correlation Coefficient in Python. A part of the matrix looks like this:

From this matrix, suppose I want to find out the strongly correlated components between the variable NoOfDoors and the rest(Symboling...Compression Ratio). The process I have adopted is that I have taken the mean of that column(which is calculated as 0.039604) and based on that, I have only considered those values greater than 0.039604.

Based on that, the following variables have been selected as strongly correlated:

(Make, Aspiration, Wheel Base, Length, Width, Height, Curb Weight, Engine Type, Bore, Compression Ratio)

I want to ask, is this selection correct? If yes then is there an efficient way to do this? And if no, what is the correct way? Since I am new to this field, a well explained article would be appreciated. Thanks!

There is no universal definition of "strongly correlated" and thus there is no "correct" answer to your question. One approach would be to perform a significance test to determine whether there is strong evidence that the correlations are nonzero. The mathematical details of the test are here, and the relevant Scipy API documentation is here.

However, "probably nonzero" may not be what you're looking for when you say "strongly correlated." If that's the case, you might want to consider why you're interested in finding the strongly correlated variables in the first place. Are you interested in identifying the strongly correlated features for a predictive problem? If so, you'll likely be better off skipping this exercise and starting to experiment directly with learning algorithms. Are you interested in determining what predictor variables cause changes in a certain response variable? In this case, you'll probably want to look into some kind of statistical regression model (probably starting with linear regression) and be very careful about all of the assumptions involved when drawing causal conclusions.


What is the best way to compare these small distributions? I have one distribution of size 30.
This are results (ROC-AUC for example) from training a neural network for 30 times in a row with the same hyperparameters but since they are randomly initialized the result is always a little bit different.
Then I train the same network with other hyperparameters and only want to do that for fewer runs. Lets say for 5 runs.
My null hypothesis is that the smaller runs distribution is not smaller than the distribution with 30 runs (one side test).
What kind of statistical significance test would be the best to compare these small distributions?
PS: At the moment I am using Mann Whitney U Test. Is there anything better?

To assess the effects of hyperparameters on the neural network results, you need to eliminate confounding variables. Let's say you have network $A_t$ and network $B_t$ corresponding to trial/sample $t$. The randomly initialized weights of $A_t$ should equal $B_t$. If you are using a stochastic optimizer (e.g. SGD), you need to ensure that the random instance selection is the same between training network $A_t$ and $B_t$.
Once you've eliminated confounding variables, then you effectively have paired samples, which you can compare via a Wilcoxon signed-rank test. For small sample sizes, it can be preferable over a paired t-test because 1. you can't verify that the samples are normally distributed and 2. the Wilcoxon test effectively evaluates median rather than mean. The median is more robust against outliers, which are particularly influential if you only have a few samples.


Process of solving a problem using ML newbie here. I finished a couple of online courses and read Intro to Statistical Learning, thinking of working on a personal project and would appreciate it if you can clarify some issues:

What does "cleaning" the data consists of? How do you know if your data needs cleaning anyway?
What features of the data determines which algorithm I should use? Or is it mostly trial and error?
Are there any additional things that I should be doing other than trying different algorithms and testing how well they fit the data?

Thank you!

Going to your first question,

  What does "cleaning" the data consists of? How do you know if your
  data needs cleaning anyway?

Cleaning refers to the various processes to transform the data so that we can utilize it to the fullest. Removing unwanted features, incomplete entries, NaN or null values ( if the dataset is numeric ) consist of "cleaning" the data. This process is important because directly feeding the unclean data to the model may result in its stunted performance or runtime errors.

Once you have a large dataset, you need to transform it according to the problem which you are solving. If you are training a model to classify movie reviews as positive or negative then you can easily remove columns like "user_id", "category" etc. as these do not contribute to the polarity of the review.

  What features of the data determines which algorithm I should use? Or
  is it mostly trial and error?

Well, the algorithm you choose will mostly depend on your problem. Decision trees are good for smaller datasets and Deep Neural Networks ( DNN ) would be used to complex classification and regression problems.

Text classifier systems use embedding layers, TF-IDF vectorization, n-grams model. We basically choose a model on these factors :

Size of the dataset.
The complexity of the problem.
Computational resources ( in some cases ).

We can always play around with the hyperparameters and also modify the model so that it better fulfils our need.

  Are there any additional things that I should be doing other than
  trying different algorithms and testing how well they fit the data?

We choose a model based on the problem. CNNs have been prevalent in image-related problems. Word embeddings are useful in text classification. LSTMs are used in time-series-related problems.

Tip: You can try to implement various algorithms from scratch ( without using scikit-learn or ML frameworks ). This helps you in developing an intuition regarding how the model learns from the data and makes predictions.


Python make_scorer giving incorrect outputs for Root Mean Square Logarithmic Error I want to generate a grid search for which I need the scoring parameter based on which the search will take place. I have defined the following function to provide me a Root Mean Squared Logarithmic Error. But I feel that the scorer is considering the greater value to be a better score, whereas it should consider the lower value as better score. Please let me know if I have defined a correct scorer.
Function for RMSLE
def score_func(y_true, y_pred, **kwargs):
  y_true = np.abs(y_true)
  y_pred = np.abs(y_pred)

  return np.sqrt(mean_squared_log_error(y_true, y_pred))

scorer = make_scorer(score_func)

I had to use np.abs in the above code, otherwise it was giving an error that RMSLE cannot be used when Target has negative values.

The make_scorer function takes a greater_is_better parameter (bool, default=True) which defines whether 'high is good' or 'low is good'. Changing its value to False might solve the issue you are having.


Multi-label classification for text messages (convert text to numeric vector) Given a dataset of messages which are labeled with 20 features, I want to predict the value of each feature for a new message.

Dataset example:

message      feature1 feature2 feature3 feature3 feature4 ...
'hi'         1        0        1        1        0        ...
'i am bussy' 0        0        0        0        1        ...
...          ...      ...      ...      ...      ...      ...

Split data into train & test to train the model:

from sklearn.model_selection import train_test_split

x= df.iloc[:,0:1].values
y = df.iloc[:,1:-1].values
train_x, test_x, train_y, test_y = train_test_split(x, y, random_state=42)

Now, my train_x is an array of text values (impossible to fit into a train model), how could I convert them to numeric vectors?

What you want to do is find a vector representation of those strings which are in your $X$ vector. Two such techniques are Bag-of-Words and $n$-grams.
Bag-of-Words (BoW)
This technique will build a dictionary with all the words that exist in your training set. Then we will build a vector with the count of each word in each instance. For example let's consider these three separate instances:

'hi'
'i am bussy'
'how are you doing?'

Then we can see that the following "words" in this training set are: hi, i, am, bussy, how, are, you, doing. So the vector representation of the above strings would be:

[1, 0, 0, 0, 0, 0, 0, 0]
[0, 1, 1, 1, 0, 0, 0, 0]
[0, 0, 0, 0, 1, 1, 1, 1]

There are ways to make this technique more effective by removing tenses from verbs or plurality of words. This is called stemming and should be used with BoW.
n-grams
n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...
The segmentation works as follows:
The String: Hello World
2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"
Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features. Then you can vectorize the results of the $n$-gram in the same way as BoW.
Term Frequency-Inverse Document Frequency (TF-IDF)
Both of the above techniques can be enhanced with TF-IDF. This removes words that appear too often and do not have much information about the string, for example: the, like, a, ab, is...
For a term $t$ in a string $s$, the weight for that term is given by
$W_{t,s} = TF_{t,s} log(\frac{N}{DF_t})$
where N is the number of strings in your corpus, $TF$ is the number of times the term appeared in the given instance string and $DF$ is the number of documents in which the term appears.


Filter row depending on specific object value and delete those instances I have some categorical data which also contains '?' as data in some rows. I need to filter those rows depending on '?', that which row contain that instances will be deleted.

I tried to drop those rows by applying these command but I failed.

train = train.drop[~train.str.contains('\?')]
train = train.drop[train['?']]

How could I identify those rows which contain '?' instance and drop those row ?

You can replace ? with nan and use dropna(). This will work if you don't already have rows with nan entries that you want to keep.

train = train.replace('?', np.nan).dropna()

Another option is to filter rows where any value is ?.

train = train[~(train == '?').any(axis=1)]

Update:

After looking at your data I found the problem: Your csv file has spaces after the commas, so the rows containing ? have a leading space.

If you use train = pd.read_csv('adult.data', sep=', ', engine='python') to read your data then either of the above methods will work.


Definition of weak and strong learners What is the exact definition, or at least the main idea, of  weak and strong learners? For instance, these terms are used in propositions such as "Boosting is based on weak learners".

Regarding the viewpoint that weak learners are of the type (high bias, low variance) and strong learners are (low bias,high variance). How are these terms defined without respect to specific datasets or data distributions?

For instance, is it possible to find specific datasets for which a so-called weak learner (low bias) is actually a strong learner (low variance)?
More simply, can the learning method X be a weak learner on data distribution A but a strong learner for data distribution B?

Intuitively, a weak learner is one that is somewhat better than random guessing, but not particularly good. More precisely, a weak learning algorithm is an algorithm to produce some classifier which has a high chance of being better than random guessing, depending on your training data.
There is a more formal definition given in [1]; one of the authors is Leslie Valiant, who developed the PAC learning framework. The definition given is cast in this framework – if you're unfamiliar, a good reference is chapter 2 of Mohri's Foundations of Machine Learning. If the formalism is difficult to follow, hopefully the more intuitive definition will help – in effect, the formal definition says the same thing, but in more precise terms.
It's easiest to think about binary classification. We can certainly produce a classifier that is correct with probability $1/2$, no matter what the distribution is: just always pick the more common class!
The idea of a weak learning algorithm is to ask for a bit more: can we find an algorithm $\mathcal A$ that will give us a classifier that has accuracy at least $1/2 + \varepsilon$ with high probability? Note that the classifier depends on the randomly drawn training data, so we must speak in terms of probabilities here. We might be unlucky and draw training data which makes it hard for the algorithm, for example.
I prefer the presentation of weak learning in [2], so I will paraphrase it here.

A weak learning algorithm $\mathcal A$ is an algorithm where there exist $\varepsilon > 0$, $\delta \in (0, 1)$ and $N \in \mathbb{N}$ such that given a randomly drawn training set $\{(X_i, Y_i)\}_{i = 1}^N$ from any distribution, the corresponding classifier $h_{\mathcal A}$ constructed by the algorithm satisfies
$$ \mathbb{P} \left [ h_\mathcal{A}(X) \neq Y\right ] \leq 1/2 - \varepsilon$$
with probability greater than $1 - \delta$ for randomly drawn $(X, Y)$ from the distribution.

In other words, for a weak learner as defined, given $N$ samples, the algorithm will perform strictly better than random guessing with probability at least $1 - \delta$ on any distribution. But, it might not be much better: $\varepsilon$ might be small, so we are only a little better than random guessing.
Every strong learner is also a weak learner under the formal definitions in [1] and [2], so indeed, if a strong learner exists, a weak learner could actually be a strong learner. It is certainly possible that an algorithm could produce a strong learner on one distribution while satisfying the weak learning algorithm property.
References
[1] Michael Kearns and Leslie Valiant. 1994. Cryptographic limitations on learning Boolean formulae and finite automata. J. ACM 41.1 (Jan. 1994), pp. 67–95.
[2] Leo Breiman. 1998. Arcing classifier (with discussion and a rejoinder by the author). Ann. Statist. 26.3, pp. 801–849. Open Access.