Calculate Q parameter for Deep Q-Learning applied to videogames I am working on Deep Q-learning applied to Snake, and I am confused on the methodology. Based on the DeepMind paper on the topic and other sources, the Q-value with the Bellman equation needs to be calculated as follows:

Q(s,a) = r + γ(max(Q(s',a'))

While calculating the Q-value for Q-table is straightforward, it's not clear to me the process in Deep Q-learning. According to some sources, the future possible states for the actual state need to be processed with Deep Learning and the highest Q-value needs to be chosen (input: future state, output: Q-value, or input: current state, output: Q-value for future states). Then, a table containing [state, action, reward, future_state] is stored in memory to reduce critical forgetting. I don't understand how we get the Q-values to predict in first place, if we need the states in order to compute the Q-values. Is this approach correct, or I am missing something?

Thank you!

The neural network you need to implement for Q-learning must approximate the Q function $Q(s,a)$. There are two ways to do this:

Using $(s,a)$ as input. A lot of the literature will assume this for simplicity, and you will see notations like $\hat{q}(s,a,\theta)$ to show you are approximating the function, and have parameters $\theta$ to learn for the neural network.
Using $(s)$ as input, and have the network output multiple $Q(s,a)$ values - one for each possible action. E.g. if you have four possible actions, then your output might be $[\hat{q}_0, \hat{q}_1, \hat{q}_2, \hat{q}_3]$. This can be more efficient, since in order to find the best, maximising, action, you need all the action values calculated.

Once you have a neural network set up like this, and a table of history (that grows on each action actually taken), this is how you make use of it:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run NN(future_state, a') forward for each possible action a' and find max
  Train the NN using the inputs (state, action) and desired output td_target

You would use this variant if the network output multiple Q values at once:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run the NN(future_state) forward and take the max output
  Construct the desired output:
    Run the NN(state) forward to get array e.g. [q0, q1, q2, q3]
    Substitute td_target depending on action e.g. [q0, q1, td_target, q3]
  Train the NN using the inputs (state) to learn the desired output

Although this second approach looks more complex, it avoids looping (or mini-batching) over multiple a' values to find a max, so can be more efficient.

If you are using a frozen copy of NN to help with stability (a common feature in DQN), then you should use the frozen copy exclusively to calculate TD targets, and the learning copy to calculate current outputs.

Note it is important that you don't store, but instead re-calculate Q values  at all times. That is both because the initial values will be incorrect (depending on how the NN was initialised), and also that they should improve over time as the agent learns a better policy.

Another important detail is that you should not use the NN to calculate Q value for terminal states. If future_state is terminal, then instead of using the NN, treat max_a'(Q(future_state, a')) as zero. Typically just detect that this is a terminal state and hard-code a 0 somehow for it.


Extracting and Mining PDF Data I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!

in my opinion, you have 4 possibilities:

You may treat the pdf directly using tabula
You may convert the pdf to text using pdftotext, then parse text with python
You may use an external tool, to convert your pdf file to excel or CSV, then use required python module to open the excel/CSV file.
You may also convert pdf to an image file, then use any recent OCR software (which reconstruct table automatically from the picture) to get data

This answer comes from:

https://stackoverflow.com/questions/47533875/how-to-extract-table-as-text-from-the-pdf-using-python/53050405

Your question is near similar to:

https://stackoverflow.com/questions/27927880/extracting-tables-from-a-pdf
https://stackoverflow.com/questions/17591426/extract-table-from-a-pdf
https://stackoverflow.com/questions/25125178/how-to-scrape-tables-in-thousands-of-pdf-files
https://stackoverflow.com/questions/29868541/pdf-data-and-table-scraping-to-excel
https://stackoverflow.com/questions/17217194/extracting-table-contents-from-a-collection-of-pdf-files/26110587#26110587

Regards


Checkpoints in Sklearn Is there a way to save the current state of your experiment so that you can pick up from where you left off in Sklearn similar to checkpoints in Keras?

I think the closest you can get is with either the warm_start parameter or the partial_fit call. They are available in some models and allow you to train a model several times without losing progress.

From sklearn docs:

  warm_start : bool, optional, default False

  When set to True, reuse the solution of the previous call to fit as
  initialization, otherwise, just erase the previous solution.

and:

  partial_fit(X, y, classes=None, sample_weight=None)[source]

  Perform one epoch of stochastic gradient descent on given samples.

  Internally, this method uses max_iter = 1. Therefore, it is not guaranteed that a minimum of the cost function is reached after
  calling it once. Matters such as objective convergence and early
  stopping should be handled by the user.

Which one to use depends on what model you are using and when you want to make your checkpoints. But if you for example use a RandomForestClassifier which has warm_start you could do the following:

# set warm_start so model to avoid erasing model between fits
clf = RandomForestClassifier(warm_start=True)
number_of_checkpoints = 10

for checkpoint in range(number_of_checkpoints):

    # Load only a subset of the data and train on it
    X, y = load_data_batch(batches=number_of_checkpoints, current_batch=checkpoint)
    clf.fit(X, y)

    # Save model checkpoint for each fit
    with open('path/to/models/random_forest_ckp_{}.p'.format(checkpoint), 'wb') as f:
        pickle.dump(clf, f)


Data augmentation when using flow_from_directory in CNN I would like to use a small dataset to create CNN model. So, I am using data augmentation to increase the train dataset. Should I use all augmentation techniques (arguments) that listed here?
I have noticed that adding many arguments decrease the accuracy of the model and make the training set harder than the testing set.
What is the best practices to use data augmentation when use flow_from_directory?

Augmentations often rely on the nature of your data. Imagine if an the result of an augmentation would be logical in your context.

For example let's say you have a cats vs dogs dataset. The images here can be flipped left to right. On the other hand in the MNIST dataset it makes no sense to flip the images at all (what good would it be feeding a flipped '3' to your model). Some augmentations might actually confuse the model (e.g. an upside down '9' might look like a '6').

Furthermore, if augmetation is making your model worse, try smaller augmentations (e.g. stick to affine transformations with small ranges - $[-5%, +5%]$ rotation/translation/scaling).


How do you Show a Difference between Two Groups (Clustering) I am approaching a data problem. My data set consists of observations of (X,Y) coordinates indicating a position on some grid. There are two groups based on a variable Z. Group A is all the points where Z>10 and Group B is all the points where Z<10.

Would it be a good idea to separately cluster the 2 groups of (X,Y) coordinates, then compare the location of clusters to reach some conclusion about how Z affects the location of the points?

I want to say something statistically about how differently, if at all, the group A and group B points are clustered. So I can say that the difference is because of Z.

Use MANOVA:

https://en.wikipedia.org/wiki/Multivariate_analysis_of_variance

  In statistics, multivariate analysis of variance (MANOVA) is a
  procedure for comparing multivariate sample means. As a multivariate
  procedure, it is used when there are two or more dependent
  variables, and is typically followed by significance tests
  involving individual dependent variables separately. It helps to
  answer:

  Do changes in the independent variable(s) have significant effects on the dependent variables?
  What are the relationships among the dependent variables?
  What are the relationships among the independent variables?


machine learning application in ticket pricing I want to sell a day-pass and month-pass ticket, the price is segmented based on customers' residential community or other factor. I would like to know the pricing strategy by which I can get profit or how much loss I would get.

Can someone share some ideas or reference?

This sounds a lot like a linear optimisation problem: given certain resources and constraints, maximise profit. If you are new to this, check out some of these resources:

walkthrough of an example
in depth, this book
practical example using Python #1 - and here #2

Machine learning is generally an approach to model and understand data, hopefully in such a way as to allow us to make predictions in the future, given some data that is representational of past observations. It more often than not considers a dynamic world and non-linear functions: given the state of the world, what should the ticket price be? Practically speaking, selling tickets doesn't work that way - it would be very confusing for customers if ticket prices always change! One more point that would concern me with your specific problem, is that your variables (ticket price, frequency of trains etc.) are not mutually exclusive from the number of tickets sold. If you just make each ticket cost $1000 dollars each, profit would be zero, because nobody would by a ticket. What I want to say is: we're not classifying cats and dogs here  :-)

With your problem and using machine learning, I would be more inclined to try answering questions such as:

How many passengers can we expect next Saturday?
Which are peak travel times in a certain region?
which factors should we improve to increase customer feedback scores?

Maybe you can try looking at one of these approached to help in the optimisation, i.e to help find your linear constraints.

One can of course attempt to frame it as a pure machine learning model, and there are many ways of doing this; it is optimisation, after all. I think the first step, whether using some machine learning algorithm or a linear optimisation construction, is to understand your data, the effects of each factor and their relationship to your target variable to be maximised/minimised (profit/loss, respectively).

So, as I was told back when learning about this myself: "The first step is to translate words into linear inequalities". So you could think about the factors you have and understand which ones effect price. If there are constraints/limitations, these should also be considered. For example, if you know that you are not allowed to sell more than 10,000 tickets on one day due to capacity constraints of your system.
There are some interesting points made in this thread over on Cross-Validated.

Now you understand the data a little better, you could think about which models to try out. Without knowing more about your data, I can't really offer more guidance as to which models might be worth trying, but hopefully this answer helps you along that path.


How to decide the processing power required based on the dataset? To train a machine learning model, the computer often needs more processing power. In this case, a powerful CPU is needed, since it is a large data set, it needs more memory, so rather than a CPU, GPU is the solution.

Do we need to decide which processor to use before we proceed? For example, will a 30000 sample data set need this much processing power?

Thanks in advance.

If any part of this question is not clear, please comment it.

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

Try few models with hardware (or AWS instances) you already have
Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.


Deep learning : CPU vs. GPU quality I searched for other similar questions on Google and here, but couldn't find any.
I'm consistently getting results of less quality when making deep learning with GPU, compared to CPU. Of course, it is much faster using GPU, but my results converge to lower quality on many different aspects: subjective quality of embedding, validation accuracy, loss, etc. It doesn't depend on the number of epochs, ie. results won't converge toward the same value: one is always inferior to the other. It's disconcerting when trying to achieve solid, reproducible SOTA results, where .1 or .2% is quite important (litteraly from top2 to top1).

I have a RTX2070 and a good CPU. Is it inherent to modern GPU ? Are approximations done using GPU that could explain this phenomenon ? Thanks for your inputs.

In theory, CPU and GPU should reach same accuracy.
But imo, generally people tend to use bigger batch size on GPU due to high parallelization. As you might know, in stochastic gradient descent a smaller batch size has a higher chance of reaching the global optimum. This could be a reason for the slight difference in accuracy.


Can you learn an algorithm from a trained model? Are there any papers where an algorithm was entirely based on the results of a trained model? Let me explain.

Suppose you want to come up with an algorithm that sorts three numbers $a,b,c$. I can generate several training data points $x_i = (a_i, b_i, c_i), a_i, b_i, c_i \in \mathbb{R}$ with their label $\hat y_i = (\min(x_i), mid(x_i), \max(x_i))$. That way, I can generate a lot of data points and train a model to predict them in their order.

My question is, are there any papers which were able to translate the trained model back to an algorithm that is understandable by humans (instead of just the values of the parameters ?). I'd be very interested even if the algorithms were very simple.

In this article https://arxiv.org/abs/1711.09784 the authors the fits decision tree to a neural network in order to understand what is the neural network does. Is it what you are looking for?


M1 MacBooks versus Google Colab I am just starting getting into deep learning with tf.keras. I am at the point where I have to decide where I want to develop. The thesis project will be timeseries prediction. My options are PyCharm on Macbook Air M1 2020 or a 2013 4th Gen Intel i5 Linux desktop and Google Colab (please let me know if there are others).
So the question I have now is which one is faster/better suited for my puropses. M1 got hyped a lot so I thought the M1 would savage my desktop (and acutally the hype biased my purchase decision), but well its only slightly better (like 1.2-1.5x faster in my cifar10 benchmark) and I wonder if its worth the effective 1-2 GB of RAM left on MacOS vs the ~14 GB on my Linux machine. Further there is Colab and I can't really tell which one will win the race, since Colab limits resources by demand but also allows distributed fit on cloud TPUs, which would introduce some extra coding efforts. Then again I have to say: so does ML on Apple Silicon, which comes with a handful of limitations, a peculiar MiniConda setup, a lot of issues (also severe ones, like training errors etc., problems which I would not even recognize) which are actually not really being worked on.
Is from the perspective of professional data scientists, which I hope to find here, a clear indication on which I should choose?

Better to use Collab indeed. Kaggle also provides notebooks with 38h GPU and also 30 hours of TPU per week you might want to have a look at that as well (plus Kaggle allows you to use your GCP credentials so you can link private google cloud storage buckets to your Kaggle notebook). On Kaggle you will also find plenty of public notebooks that can be of great help


How to interpret this 'Zipf' (survival, war casualties) plot? Help me interpret this chart please?

What do the X and Y axes mean here? (I asked the author directly but they didn't respond for some reason.)

Any thoughts on what 'Surivival Function' might mean here?

War Casualties from 1800 until today, n data points

As you look at the chart, you can see the vertical axis represents some kind of Survival Function and horizontal axis defines War Casualties. As you can see, the higher the magnitude of Was Casualties, the lower the Survival rate/function and vice-versa. The higher the Survival rate the lower casualty count. Each dot represents each recording. Most likely, you need to look into how Survival Function is defined to really get a good understanding.


SAS. How to write "OR" How to write "OR" in this example?

DATA a1;
set a;
if var1=1 OR 2;
run;

P.S. va1 is the categorial (with categories: 1, 2, 3)

i'm assuming this will be migrated to stack overflow, but instead of trying to do if var1=1 or 2 wouldn't it be better to use if var1 in (1, 2)?

...and somebody with enough reputation should probably create an sas tag (and an spss tag while you're at it) unless data scientists only use open source languages like r and python now...


Any one read the over 100gb csv file and successfully concatenation? I have been searching for the deal with large CSV file read method
Its over 100gb and need to know how deal with the chunk file processing
and make concatenation faster

    %%time
    import time
    filename = "../code/csv/file.csv"
    lines_number = sum(1 for line in open(filename))
    lines_in_chunk = 100# I don't know what size is better
    counter = 0
    completed = 0
    reader = pd.read_csv(filename, chunksize=lines_in_chunk)

CPU times: user 36.3 s, sys: 30.3 s, total: 1min 6s
Wall time: 1min 7s

this won't take long but the problem is concat

%%time
df = pd.concat(reader,ignore_index=True)

this part take too long and take too much memory also
is there way to make this concat process faster and efficiently ?

Its too big file to handle by standard way. You could do it by chunk
for chunk in reader:
    chunk['col1']=chunk['col1']**2 #and so on

Or dump yours csv file to database.
number of rows
num=0
for chunk in reader:
    num+=1
num_of_rows = num*lines_of_chunk

#work around in bash and python
import subprocess
subprocess.check_output(["wc","-l", "file.csv"])


Is it advisable to merge similar datasets to improve model accuracy? I'm trying to build a classifier that would help me classify whether a statement collected from Reddit is bullish, bearish or neutral.
To this end, I have hand-labelled a fairly small dataset of 2500 entries, each with max 280 characters. Unfortunately, the data is unbalanced (60% neutral, 25% bullish and 15% bearish) and my initial attempts are returning poor results for the bullish and bearish classes.
I've managed to obtain 10,000 entries of similar data from StockTwits, each labelled bullish or bearish (no neutral...) and 280 characters max.
I tested supplementing my Reddit dataset with bullish/bearish data from this dataset to balance out the classes. In other words, I added data from the StockTwits dataset until the number of bullish and bearish entries in matched the neutral. With this model, I'm getting much better results.
Is merging similar datasets in this way advisable? My gut instinct says "no" but I haven't found anything suggesting this is not a good idea.

Ideally if you mix data from different sources (different distributions), the mix should be uniform for each class.   Otherwise if for example class A gets more of source S and class B gets more of source T, there is a risk that the model will learn to associate class A with source S and B with source T and that it won't generalize across sources.
To validate such an approach, keep a separate test set with both sources and compute several accuracies, one for each (class, source).


Storing Large dataset for processing and analysis of data I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I have explored many data formats for storage . The dataset that I want to store is a heart rate pulse data generated by a sensor.

It would depend on the use cases - read vs write vs analytics etc. Nonetheless, you may want to explore Hadoop if not done already.


Are there any rules for choosing the size of a mini-batch? When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64, and 128 elements per mini batch.
Are there any rules/guidelines on how big a mini-batch should be? Or any publications which investigate the effect on the training?

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements:

  It has been observed in practice that
  when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]

  large-batch methods tend to converge to sharp minimizers of the
  training and testing functions—and as is well known, sharp minima lead to poorer
  generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

From my masters thesis: Hence the choice of the mini-batch size influences:

Training time until convergence: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.
Training time per epoch: Bigger computes faster (is efficient)
Resulting model quality: The lower the better due to better generalization (?)

It is important to note hyper-parameter interactions: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.

See also

this nice answer / related question
Efficient Mini-batch Training for Stochastic Optimization
this RNN study


When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization? I knew that Residual Network (ResNet) made He normal initialization popular. In ResNet, He normal initialization is used , while the first layer uses He uniform initialization.

I've looked through ResNet paper and "Delving Deep into Rectifiers" paper (He initialization paper), but I haven't found any mention on normal init vs uniform init.

Also:

  Batch Normalization allows us to use much higher learning rates and be less careful about initialization.

In Batch Normalization paper's abstract, it is said that Batch Normalization allows us to be less careful about initialization.

ResNet itself is still care on when to use normal init vs uniform init (rather than just go with the uniform init).

So:

When to use (He or Glorot) normal-distributed initialization over uniform initialization?
What are normal-distributed initialization effects with Batch Normalization?

Notes aside:

It rhymes to use normal init with Batch Normalization, but I haven't found any paper to back this fact.
I knew that ResNet uses He init over Glorot init because He init does better on a deep network.
I've understood about Glorot init vs He init.
My question is about Normal vs Uniform init.

The normal vs uniform init seem to be rather unclear in fact.

If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly considering the type of the distribution. As such, their theorical conclusions hold for any type of distribution of the determined variance. In fact, in the Glorot paper, a uniform distribution is used whereas in the He paper it is a gaussian one that is chosen. The only "explaination" given for this choice in the He paper is:

  Recent deep CNNs are mostly initialized by random weights drawn from Gaussian distributions

with a reference to AlexNet paper. It was indeed released a little later than Glorot's initialization but however there is no justificaion in it of the use of a normal distribution.

In fact, in a discussion on Keras issues tracker, they also seem to be a little confused and basically it could only be a matter of preference... (i.e. hypotetically Bengio would prefer uniform distribution whereas Hinton would prefer normal ones...) One the discussion, there is a small benchmark comparing Glorot initialization using a uniform and a gaussian distribution. In the end, it seems that the uniform wins but it is not really clear.

In the original ResNet paper, it only says they used a gaussian He init for all the layers, I was not able to find where it is written that they used a uniform He init for the first layer. (maybe you could share a reference to this?)

As for the use of gaussian init with Batch Normalization, well, with BN the optimization process is less sensitive to initialization thus it is just a convention I would say.


Possible to use different learning rate for different neuron in Keras/Tensorflow? The simplest example is to have faster/slower learning rates in the upper/lower layers of a network. I found this post on tensorflow.

Is there a similar trick in Keras?
Going one step further, can we set different learning rates for specific range/set of neurons/weights in a particular layer?

You can follow this thread here: https://github.com/fchollet/keras/pull/3004 Seems at least that the layer-wise tuning should come.


What are Machine learning model characteristics? This question I have received in some Machine Learning related interview and Here is the question

  What questions would you ask to learn about machine learning model characteristics?

This is what I think:

I did a bit research on the internet & found this resource, but still not very clear about How ML model characteristics are equivalent to asking trade-off between different algorithms in supervised learning settings. What I understood and framed my answer is:

First, for simplicity, I assumed that this model is used for some supervised learning task(classification/ regression) then,

I would first try to find a learning algorithm used to create this model, because this will provide me with a clue of the created model and help me to talk about different issues within it(such as,

How complex or simple your model is(feature engineering!)?
What are your training & test errors(Model performance)

Both will help me to talk about the bias vs variance trade-off.
Furthermore, I could talk about different learning algorithm trade-offs in supervised settings.

Is the model based on identifying correlations i.e output variable can be expressed in terms of the linear/non-linear combination of features?
(targeting linear regression, logistic regression, SVM or Neural network algorithms, or
it's decision tree-based algorithms.

I am not confident enough that my answer is complete or correct (since I am not able to understand what precisely characteristics of a model are?), so I need this community's help to provide feedback! please feel free to add your suggestion to it!

I am using neural networks in industrial fault diagnosis and dynamic system modeling, so I would consider:

is it a static or dynamic system? Depending on it you may choose other network structure (for example RNN for dynamic systems)

what is the quality of quantity data? Depending on it you may choose training algorithm (maybe bayesian regularization for noisy data). SVR modeling would be good for high-quality data with a lots of samples. It would be way faster learning than neural networks,

are there any delays in the system? Sometimes performance might be improved by adding delays to model. Cross-correlation might be applied to check optimal delays or diagnose some echos in signals

sometimes correlation analysis could reduce dimimensionality of a problem. Computing correlations might help selecting inputs of the model. Sometimes PCA is a nice idea to get rid of curse of dimensionality.
But depending on field of use those questions could be different.


implementing temporal difference in chess I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, mobility, pawn-structure and trapped pieces etc..... My evaluation function is derived from the

$$f(p) = w_1 \cdot \text{material} + w_2 \cdot \text{kingsafety} + w_3 \cdot \text{mobility} + w_4 \cdot \text{pawn-structure} + w_5 \cdot \text{trapped pieces}$$

where $w$ is the weight assigned to each feature. At this point i want to tune the weights of my evaluation function using temporal difference, where the agent plays against itself and in the process gather training data from its environment (which is a form of reinforcement learning). I have read some books and articles in order to have an insight on how to implement this in Java but they seem to be theoretical rather than practical. I need a detailed explanation and pseudo codes on how to automatically tune the weights of my evaluation function based on previous games.

I recommend anybody interested in the topic take a look at the paper which combines TDL and deep-learning.

Roughly, you'll need to make the engine play games against each other. Record the mini-max evaluation for each position. At the end of the game, you'll get a reward, which is {0,1,-1} for chess. Then you'll need to adjust your parameters with:

This equation tells us we should adjust the weights by temporal differences, weighted by how far you should do it. If you have a perfect evaluation, your temporal difference would always be zero, thus you wouldn't need to make any adjustment.

Next, you'll need use the new parameters to play a new game. Repeats until as many games as you can afford or when you think it's converged.

Few remarks:

The paper I quote applies a discount factor. It's done for the backproportion algorithm for neural network. You don't need it.
You'll need to experiment with the optimal learning rate (alpha in the equation). Too large will make your learning unstable, too little will take longer to converge. I've seen people using 0.70. The paper I quote used 1.0.


Fitting a model as a significance testing alternative My colleagues use somewhat unusual approach to estimate how far away performance of some of our company related processes diverge from historical levels. Their nature isn't really relevant for the purpose of this discussion. We may just assume here, that the final result can be represented with some real number which fluctuates over time and has some seasonality components.

They fit a regression model (with XGBoost) on historical data excluding the most recent period (say, the last month) and than compare the most recent values with model's forecast.

I argue, that this is basically an overkill and some sort of a crooked statistical significance test. And rather than fitting a model and making a forecast, we should actually calculate some normalised historical indicators (stratified by relevant groups and taking into account seasonality, where appropriate) and then do a standard significance test using a significance level, that would make sense from a business perspective.

Am I right or not? Are there any better ways to approach a problem of estimating performance and comparing it to historical data?
Does approach with modelling/forecasting has any legs? How it can be justified from a statistical standpoint?

It's probably a quite subjective matter, here is my opinion:

I would say that this method makes a lot of sense if the ML model uses a number of features, something which would be hard to factor properly in a standard statistical test (especially if some kind of feature selection is involved).
Also in favor of the method, statistical significance tests are usually interpreted in a binary way (typically p-value higher or lower than 0.05) even though the underlying p-value is continuous: there's very little different between a p-value of 0.049 and 0.51 but the two will be interpreted in a completely opposite way.
However I agree with you that statistical significance tests are much more grounded in statistical theory, with a clear interpretation especially about levels of confidence.

Overall I can see an interest of using such a method from a practical viewpoint, but it is indeed limited in terms of theoretical interpretation and reliability.

Disclaimer: I'm not a statistician, I might be biased towards practical ML methods ;)


In supervised learning, why is it bad to have correlated features? I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, so it is logical to remove one of them. But I can not understand why this can worsen the model.

Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:

For linear models (e.g., linear regression or logistic regression), multicolinearity can yield solutions that are wildly varying and possibly numerically unstable.
Random forests can be good at detecting interactions between different features, but highly correlated features can mask these interactions.

More generally, this can be viewed as a special case of Occam's razor. A simpler model is preferable, and, in some sense, a model with fewer features is simpler. The concept of minimum description length makes this more precise.


How to plot categorical data and encode it through colour and shape size? Given the following data set:

I am wondering what kind of plotting technique can be used to produce such a visualization:

The Skill attribute is mapped to the y-axis, while the Participant is on the x-axis. The other two attributes are encoded as colour and size features of each point in the visualization.

Although I am trying to get this done with Pandas and pdvega, I'd be happy to learn another tool that could render that. But most importantly, I would like to know the proper technical terms used to describe this type of visualization.

I have reviewed various Pandas tutorials, but the problem is that the examples provided are targeting numeric data, rather than categories. I am thinking that perhaps I could partially fake it by turning each category into a number, then somehow overriding the labels on the axis - but it sounds like a convoluted solution to a mundane problem, so there ought to be a nicer way to do it.

Here is the raw data that can be used to produce a Pandas dataframe with the data above:

import pandas as pd
raw = {"Age":{"0":"27..35","1":"18..26","2":"18..26","3":"18..26","4":"18..26"},
       "Skill":{"0":"intermediate","1":"expert","2":"novice","3":"intermediate","4":"expert"},
       "Gender":{"0":"M","1":"M","2":"F","3":"F","4":"M"}}
df = pd.DataFrame(raw)

The easiest solution I found is Altair:

import altair as alt

# assume df contains the DataFrame from the question + a numeric column `Participant`
alt.Chart(df).mark_point().encode(y='Skill', x='Participant', size='Age', color='Gender')

This is what gets rendered out of the box:


Geospatial Social Network Analysis Visualization I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.

What will be the best way to visualize this kind of data set? This visualization should also be able to use the geo coordinates to place this entities in the map to form clusters, or to superimpose them on a real map.

I am interested in an algorithm and/or a library that will help me do this. Library should be preferably written in Java, Python, Scala, or NodeJS. The record count can be as big as a thousand or hundreds of thousands.

Your dataset can be viewed as a directed graph. The party's location (latitude and longitude) can be denoted as a node and the directed edge can denote who referred whom. Once the dataset can be viewed as this, the problem boils down to joining co-ordinates with lines.


Estimating the battery capacity using current power consumption and battery percentage I want to estimate the current maximum capacity (in kWh) having the current power consumption (in kWh) and the state of charge of the battery (in %) available in a time series.

I do not have a full battery charge circle recorded but only a snippet with the state of charge going from ~95% to ~35% in a 1 second data recording interval.

At first, i cumulated the current power consumption and noticed that the progression between cumulated power consumption and state of charge was almost the same (see figure 1 and 2).

progression of state of charge vs. cumulated power consumption

So i tried to use a linear regression model to predict two values of cumulated power consumption at 0% and 100% state of charge. In my assumption the delta of these two leaves me with the current maximum capacity of the battery.

Ideally i want to monitor the capacity during the data recording. Which means i only have data of a few percent of state of charge.
In figure 3 and 4 you can see my attempt of trying to create a linear regression model for every 2% state of charge. As you can see in the right plot, the estimated capacity has a very high variance.

LEFT: regression lines (green) of every 2% state of charge; RIGHT: estimated capacity of every 2% state of charge

Since i am not an expert in neither the matter of regression nor batteries/physics, my questions are:

Is there a much simple or more exact way (or both) to estimate the current maximum capacity of the battery?
If yes, is there a good way to estimate an sufficiently exact capacity with an even smaller snippet in an ongoing process, lets say every 2% state of charge? (maybe using a characteristic curve of the discharge process)

Here is a sample of my data: sample file

Firstly, a common chemical battery discharging from 95% to 35% in 1 second is undergoing a very deep discharge and its discharge profile will be very different from a slow steady discharge, e.g. 95% to 35% within 10 hours. Secondly, inferring from a very small snippet of the charge process (2% change) may work only if you know exactly which charge profile is being exhibited by the battery during that charge cycle, so that you can identify at which point of the charge cycle is the battery right now.

For most chemical batteries, the current consumption by the load determines the discharge profile and hence the capacity at a given point of time in the future. Since this is usually non-linear, and the slope varies at different points of the discharge profile, a linear regression would not be a good fit. I would recommend trying out a polynomial regression fit on the discharge profile. Multivariate adaptive regression splines should also work well. Similarly, the rate of charging determines the charge profile/characteristic curve.

You could fit multiple models, one each for each type of charge profile (e.g. deep discharge, slow discharge, etc.) and then select the appropriate model based on the current status of the battery, to predict the future state.

To understand this process for a lead acid battery, refer to these -

http://scubaengineer.com/documents/lead_acid_battery_charging_graphs.pdf and,
http://batteryuniversity.com/learn/article/how_to_measure_state_of_charge.


How to understand incremental stochastic gradient algorithm and its implementation in logistic regression [updated]? I have a difficulty where to start the implementation of incremental stochastic gradient descent algorithm and its respective implementation in logistic regression. I don't quite understand this algorithm and there are few sources to explain it with crystal-clear interpretation and possible demo code. I am quite new with ML algorithm and I have no idea which would be efficient workaround to solve this problem.

In particular, the problem I am studying is to implement hogWild! algorithm for logistic regression, which asks me to program incremental SGD algorithm with a sequential order. Can anyone give me a general idea or possible pipeline to make this happen in python?

logistic loss function and gradient

Here is my implementation:

import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from  sklearn import datasets

X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt',
                                          n_features=X_train.shape[1])

class ISGD:
def lossFunc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y* np.dot(X,w,))))
    rslt=np.float(lossFnc)
    return rslt

def gradFnc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradF1=-y*np.nan_to_num(ss.expm1(-y))
    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
    gradF3=gradF2.resize(gradF2.shape[0],)
    return gradF3

def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
    self.w=None
    self.learnRate=learnRate
    self.verbose=verbose
    self.num_iter=num_iter

def fitt(self, X,y):
    n,d=X.shape
    self.w=np.zeros(shape=(d,))

    for i in range(self.num_iter):
        print ("\n:", "Iteration:", i)

        grd=gradFnc(self.w, X,y)
        grd.resize((grd.shape[0],1))
        self.w=self.w-grd
        print "Loss:", lossFunc(self.w,X,y)

    return self

Seems my above implementation has some problems. Can anyone help me how to correct that? Plus, I don't have a solid idea how to implement incremental SGD sequentially. How can I make this happen? Any idea?

Gradient Descent

The idea of gradient descent is to traverse a function $f_{LR}(\textbf{w})$ and find a local maximum or minimum for a set a values $\textbf{w}$. Gradient descent is an iterative process. At each iteration you will evaluate the function given your current set of values. Then you will take the derivative of that function with respect to those values to see how much each value contributes to the slope of the function. We can then change the values in a proportionate way to move towards this optimal point.

The gradient descent equation is described as

$\textbf{w}^{(k+1)} = \textbf{w}^{(k)} - \rho \frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}}$

where $\rho$ is the learning rate, usually small number. A constant which determines the speed at which we want to change the values we are optimizing.

Initializing the values

There's many ways to initialize these values. We can either set them all to zero, or set them randomly.

Then you can take a random instance in your dataset $x_i$ and $y_i$ and compute the derivative

$\frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}} = \textbf{x}_i (-y_i \frac{e^{-y_i \textbf{x}_i \textbf{w}}}{1 + e^{-y_i \textbf{x}_i \textbf{w}}})$.

Once the compute this you can put the result into the gradient descent equation and update all your weights.

You then continue to go through this process until your weights $\textbf{w}$ converge to a value, or some other condition is met. You might want to limit your algorithm with some iteration counter to avoid infinite loops caused by weights that do not converge.

Gradient Descent Code

Here is an example using gradient descent to train weights along the logistic regression loss function.

import numpy as np

def update_weights(x_i, y_i, w):
   p = 0.8                  # The learning rate
   yhat = predict(x_i, w)   # The predicted target
   error = y_i - yhat
   return w + p * (y_i - yhat) * x_i  # Update the weights

def predict(x_i, w):
   return 1/(1+np.exp(-1 * np.dot(w.T, x_i)))

def train_weights(x, y, verbose = 0):
   w = np.zeros((x.shape[1],))
   w_temp = np.zeros((x.shape[1],))

   epoch = 0
   while epoch <= 1000:
      for i, x_i in enumerate(x):
         w = update_weights(x_i, y[i], w)

      pred = 1/(1+np.exp(-1 * np.dot(x, w)))
      error = np.sum(pred - y)

      pred[pred < 0.5] = 0
      pred[pred >= 0.5] = 1
      epoch += 1

      if verbose == 1:
         print('------------------------------------------------')
         print('Targets: ', y)
         print('Predictions: ', pred)
         print('Predictions: ', pred)
         print('------------------------------------------------')

      # Check if we have reach convergence
      if np.sum(np.abs(w_temp - w)) < 0.001:
         print(epoch)
         print(error)
         return w
      else:
         w_temp = w
  return w

# Create some artificial database
x = np.zeros((4,3))
y = np.zeros((4,))
x[0] = [1, 1, 1]
x[1] = [1, 2, 2]
x[2] = [1, 10, 10]
x[3] = [1, 11, 11]
y[0] = 0
y[1] = 0
y[2] = 1
y[3] = 1

print(x)
print(y)

# Train the weights
w = train_weights(x, y, 0)
print('Trained weights: ', w)

# Get predicitons
pred = 1/(1+np.exp(-1 * np.dot(x, w)))
error = np.sum(pred - y)
pred[pred < 0.5] = 0
pred[pred >= 0.5] = 1
print('Predictions: ', pred)


Detecting and confirming sentiments in chat messages Say I have a spreadsheet of 1000 chat messages that are labeled as happy, sad or funny.

What is the best way of detecting if the labels are indeed correct or if they need improvement?

How would I go about performing an analysis on the messages? How can I build a predictive model?

What type of feature and model selection should be used?

Python is my preferred language and I wish to load them as a CSV.

You need to define best.

If best means being exactly sure of your labeling, then you'd have to manually go through all messages, use crowdsourcing and what not. This is assuming that the chat messages are in a language that you can understand.

If best means automating the process, you could use an existing sentiment analysis model to predict the sentiment of each message and see if it lines up with the existing labeling. Of course, perfect alignment is unlikely.

If best means doing it from scratch, then I would try to cluster the sentences into 3 groups and then look at the composition of the groups. Of course, this is highly dependent on the clustering and how you define the distances between the messages


Method for predicting price based on Geographical market, Product, and Company I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's price for Product A in London, but not New York. I am looking for guidance on how to impute these values given the data that I already have.

There are trends to be exploited in the data. Such as:

Geographical trends: i.e, London and New York are similar markets as measured by the median price of a given product, but markets like Cairo and Johannesburg tend to be much more expensive (though in varying degrees for different products)
Product Trends: Within a given market, for a given company, products are priced hierarchically, although the relationship is nonlinear. For example, if the products are "Good", "Better" and "Best", the price of "Better" might be 2X"Good", and the price of "Best" might be 2.5X "Good", though these relationships are not strictly held.
Company Trends: Certain companies tend to have higher prices than others in most markets, but again, these relationships are not strictly held, and may differ for certain subsets of products or markets where a company has a competitive advantage.

I describe the method that I have tried so far below, though I feel it is overly simplistic. I am looking for suggestions as to how to better capture the complexity of my data and obtain a reasonable accurate imputation for the missing values. Any advice is appreciated.

The method I have tried so far is to estimate coefficients for each category. For example, holding Company and Product constant, what is the average price multiple of Market B over Market A? Market C over Market A? Then, if I know I have a price for (Company X, product "Good") in markets B and C, I'll multiply them by those average multiples in order to obtain a "best guess based on market". Then I repeat that holding constant Company and Market, and Market and Product. At the end, I'm left with three "best guesses". Then the issue is converting my three best guesses into a single guess, because, based on some testing of this method, the true price may lie either between or outside these three best guesses. This is the histogram of errors based on a training sample:

It may be usefull to put what you are doing in a formal context, that way you can look at some standard solutions to your problem.

You have obsvations of a (random) variable X (price), together with some explanatory variables L, C, P (L="location" = city, C=company, P=price). So your data consists of cuadruples $(C_i, L_i, P_i, X_i)$.

You are assuming a $\mathbf{multiplicative}$ model:
$$ E(X|C,L,P) = \lambda \space f_C \space g_L \space h_P .$$

This model fits nicely in the context of generalized linear models, since:
$$ \ln (E(X|C,L,P)) = \ln(\lambda) + \ln( f_C) + \ln( g_L) + \ln( h_P) .$$

You are fitting the model by a variation of iterative scaling. I don't know if there is a probabilistic model that corresponds to that way of fitting the models. Both the multiplicative model and iterative scaling have long traditions, so the answer might be known.

In any case, once in the context of generalized linear models, you can fit the parameters making some assumptions on the error. Since $X$ counts the number of monetary units per item, you can start by trying a Poisson model (and a Poisson model with overdisperssion). It is also worth trying other GLM's.


Remove part of string in R I have a table in R. It just has two columns and many rows. Each element is a string that contains some characters and some numbers. I need number part of the element. How can I have number part?
For example:

    INTERACTOR_A INTERACTOR_B
1          ce7380       ce6058
2          ce7380      ce13812
3          ce7382       ce7382
4          ce7382       ce5255
5          ce7382       ce1103
6          ce7388        ce523
7          ce7388       ce8534

Thanks

You may use gsub function

> c <-  "ce7382"
> gsub("[a-zA-Z ]", "", c)
[1] "7382"

Feel free to add other characters you need to remove to the regexp and / or to cast the result to number with as.numeric.


About training and cross validation on a time series problem I am new to machine learning.

I'm having a task of predicting whether a user will churn in March, given February feature data and the churn result. However, March data is leaked and now I'm assigned to predict for April.

My strategy is to train the model with February data and perform cross validation with March data. Then I will try to predict April data using model trained with February data.

Here are my questions:

Is my strategy good?
Or should I append both February data and March data to predict April
data?

You might want to check out my answer to a related question here.

If you want to cross-validate time series data, I would suggest creating some sort of sliding window on which you train your model and predict the next month following that window.

For example, you could train your model on February data and then predict the data of the first week of March. Then slide your window to include data from the second week of February through the first week of March and then predict the data of the second week in March. The length of the windows that you use to train and test on are parameters that you will need to play around with yourself to see what gets you the best results.

Once you feel like you are getting good results from your cross-validation, I would try training a new model on all of the data you have and see how it performs on the new data coming in. Depending on the data and model you are using, this may provide better results having trained on more data. Of course, this depends on the problem you are trying to solve, the nature of your data and your choice of model.


Rank players of any given sport I've recently become interested in possibly of developing some sort of method for ranking athletes of sports such as American football and determining which players are better than others in terms of specific statistics.

My thoughts are that there are two ways to go about doing this. The first would be some sort of mathematical formula which would take in the statistics of a given player and provide some sort of standardized score which could be compared with other players to determine which is better.

My other idea would be to have some machine learning algorithm go through historical data and determine the patterns which indicate how well a certain combination of statistics would perform in the following week of play by using the patterns it recognizes over time.

I'm not sure which approach would be more effective and so I'm hoping that someone has an idea or any advice as to which would be best to look into. Thanks!

Prediction

If the main goal is predicting anything, say, the statisitcs of the player in the next game, game result, then I would not recommend to do any scoring. Better way to go is using the pure statistics data as an input to the model. Any scoring/rankning - is information loss.

Ranking

If the goal is ranking itself, than you still need to have some target variable to predict. As you may want to check real predictive value of those ranks. That could be, again, playser stats in the next game or game result itself.

References

Sport scores prediction and RFM scoring are probably the next directions for you to look at.


scikit-learn classifier reset in loop I'm trying to evaluate classifiers comparison by running the sample script that can be found here.
What I noticed is that in some cases the classifier is not reset.
In fact, duplicating some of those (with no parameter change) the score and the countour change between the two.
This can be seen simply replacing AdaBoostClassifier() in the classifier list with another MLPClassifier(alpha=1)
I guess that at every cicle of the for loop the classifier should be reset in order to make a fair comparison among the different models, and this case should behave the same I think.
In particular, differences are noticed duplicating the MPL (Neural Net) and the Random Forest, while there is no change duplicating KNN or RBF SVM.
I also tried to clone the classifier, and even del clf in the loop, but the behaviour stays the same.
How can I make the evaluation replicable and not influenced by the previous run? I want to be sure that when I use the same model, and only change the parameters the result is correct, and this will be possible only if two identical model yield the same result.

The behaviour you are seeing is not related to not properly resetting models but the stochastic nature of most of these algorithms. By setting a random seed the same random numbers will be generated every time. See:
How to seed the random number generator for scikit-learn?
However, while this will lead to a reproducible sample, this might still not be fair. If one model randomly gets a good seed and another one a bad one you will unfairly always favor the first one. What you could do is run the models multiple times with the same hyperparameters but with different seed and look for the average performance. This way you get a more fair comparison and the reproducibility. Pick the seeds up front, then you can loop over them. Something like this:
seeds = (1, 2, 3, 4, 5)
performances = []
for seed in seeds:
    performances.append(score(Model(param1=1, param2=2, random_state=seed)))


Is it possible to build a regression model for predicting movie gross using sections on their wikipedia pages? I got this as an assignment from a company recruiter and I've successfully scraped a dataset of about 650 movies with their 'Plot', 'Music' and 'Marketing' sections and gross. I've tried tfidf and count vectorizers and performed LSA/PCA to reduce the dimensions which originally are around 20k terms.

This is really boggling me, due to less instances(650) I guess the no. features should be around 100 or atleast < 600 but that is a drastic reduction of dimensions using PCA or LSA, is it the right thing to do?

Also my regression model is way off target or in some cases overfitting excessively. I've also tried normalizing and de-normalizing the target(gross) for no improvements. I know I should try NN and other models too but the results of linear regression should be atleast comparable too, right?

This makes me ponder if its even possible to model such a problem or am I doing something wrong with latter being more probable as I'm no expert in the field.

PLEASE HELP !

If your models suffers from multicollinearity problems, then yes, I'd say go for dimensionality reduction (possibly non-linear, such as t-SNE or Autoencoders). Moreover, 100 variables for 650 observations is a lot, I would apply a much more drastic reduction, but that's up to your preferences.

If your model is overfitting excessively, then I suggest you to try Neural Networks with high dropout probability. That helps a lot in preventing overfitting. Also, try other models, the more you try, the more confident you'll be about results.

Please keep in mind that normally Neural Networks require a lot of data to perform very good. If you have just 650 observations, tree-based models (Random Forest Regressions or XGBoost Regressions) could work even better than Deep Learning. They are ensembles of trees, i.e. desgined to combat overfitting, and could capture non-linear patterns of your dataset.


target in cluster analisys (PCA) i m doing dimensionaly reduction using PCA. I don't understand why some dataset already had a target ad example in Iris database or other like this (https://scikit-learn.org/stable/datasets/index.html)
had a target_names useful when plot data.

Ad example in iris database choose like color= target_names to do this

i found this code online in example.

fig = plt.figure(figsize = (8,8))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
targets = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica']
colors = ['r', 'g', 'b']
for target, color in zip(targets,colors):
    indicesToKeep = finalDf['target'] == target
    ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']
               , finalDf.loc[indicesToKeep, 'principal component 2']
               , c = color
               , s = 50)
ax.legend(targets)
ax.grid()

I m trying to do the same with my dataset, where i dont have a target

i have a table in this way

User    Movie
        0 1 2 3 4
      0 2 0 5 0 0
      1 0 1 1 0 0
      2 0 5 5 5 0

for each user i have all film and him review (0 if don't review)

When plot my graph i tried to do this

fig = plt.figure(figsize = (16,12))
ax = fig.add_subplot(111)

ax.scatter(a,b, alpha = 1)
plt.title('Method: PCA')
#plt.savefig('PCA.png', dpi = 300)
plt.show()

but i really don't know where is my target. I try to add another column to my dataset with gender for user for cluster user for gender but give an error of the shape because i just have a 2 gender but 6000 user.

I really don't know to apply a target in this way,
Same suggestion?

Iris data set comes with a label, it is a supervised learning problem. The task is to predict the species of the flowers.

Your problem need not have a label, it really depends on your task and data. If your goal is to predict the gender and you want to see if you can see a pattern in lower dimensional space, you can give it a try to treat gender as a target.

If your problem is an unsupervised one, You can also just plot out your data and see whether you see any clustering in your dataset.

Remark: You might want to be careful about the $0$ values, $0$ doesn't mean the movie is good or bad. It really means no feedback.


Term for calculated values that lose pertinence when changing scale I'm trying to find the term for a type of calculations or values that cannot be simpply added or multiplied when zooming in or out from a temporal scale. I know it's not very clear, if it were I would just google the question.

Here's an example: say you're comparing 2014 vs 2015 sales of ice cream and temperature at day's level. If you want to do an analysis by week, you can just add the sales for seven days, and compare weeks between them, the information still makes sense: there's a fact (amount of money deposited in the bank that week) that is grounded in reality that relate to the new figure. You could also calculate an average by day from the total by week.

But, adding temperatures for a total would not make any sense in reality, you need to calculate an average for that kind of data.

So is there a term for this type of values that can only make sense as average and never be added or multiplied (except as an intermediate step for calculating an average)?

Let us think of sales ($S$) as a property of a system.  You could imagine Week 1 to be a system and Week 2 to be another system.  Call the Week 1 system $W_1$ and the Week 2 system $W_2$.  Then,
$$
  S(W_1) + S(W_2) = S(W_1 + W_2)
$$
A property that has this additive character is called extensive.

Now consider the temperature ($T$) as the property with values $T(W_1)$ and $T(W_2)$ for weeks 1 and 2.  Assume that the two $T$ values are identical.  Merge the Week 1 and Week 2 systems together to get another system $W$ for the two weeks together.  Then
$$
  T(W) = T(W_1+W_2)
$$
Properties that follow this rule are called intensive and cannot be added.

These definitions were invented for physical properties and systems and one has to be careful in applying the definitions to general situations.  I'm not sure how much thought along these lines has gone into the social sciences.

For more, see https://en.wikipedia.org/wiki/Intensive_and_extensive_properties.


Evaluating performance of Generative Adverserial Network? What is the best way to evaluate performance of Generative Adverserial Network (GAN)? Perhaps measuring the distance between two distributions or maybe something else?

I think it depends on what exactly you're doing with the GANs. If you're generating images, the two most popular (to my knowledge) are the Inception Score [1] and Frechet Inception Distance [2]. GANs aren't my direct area of study, but I don't think you can miss starting with those two.

Actually, Goodfellow himself endorsed FID (a year ago) on twitter, so I'd start there.


Plotting Polynomial Regression? I'm reading through Hands-On Machine Learning with Scikit-learn and Tensorflow by Geron. I am creating a simple polynomial regression using sklearn's PolynomialFeatures.

First, I create an X and y set using numpy random numbers with quadratic shape:

m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)

Then I plot the scatterplot distribution:

plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([-3, 3, 0, 10])
plt.show()

Then I use PolynomialFeatures to add the 2nd degree:

poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)

Then I fit the LinearRegression:

lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
lin_reg.intercept_, lin_reg.coef_

Then I plot the same distribution with the quadratic regression line. My question is with the following code:

X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(X, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])
plt.show()

Why do we create X_new (np.linspace(-3,3,100).reshape(100,1) and X_new_poly? Why does this not work with the X_poly that I've already created? (I tried plotting it with the original X_poly and it definitely does not work. It's just oscillating lines up and down over and over. I'm just not sure why this is the case.)

You could use X and lin_reg.predict(X_poly), but

The plot will put those points in the order they appear in X, and since you're using a line connector it will appear to jump all over the place.  You could fix this by using a scatterplot instead.
It's vaguely disingenuous to use the training set's x-values for plotting the fitted curve; using np.linspace to get equally-spaced x-values is preferable (even if your original X was randomly generated and should fill the space reasonably well).


Is it possible that MLP has better accuracy than CNN? I am working on the epilepsy classification system which consumes EEG signals and in the result says if withing the certain period is a seizure or not. I take an advantage of Keras API for the sake of network training. I am giving a try at a few different neural network configurations and now I wonder is it possible that MLP is better than CNN in 1D classification in some cases? My question is not only related to EEG or epilepsy detection, but more if there is any example when MLP behaves better than CNN in the sense of accuracy. Or maybe it is mathematically proven that there is always some CNN which will beat MLP for the given classification problem.
For CNN I use keras.layers.Conv1D, for MLP keras.layers.Dense if that matters.

Yes, it's possible that an MLP has better accuracy than a CNN.
Here's one discussion

CNN (and RNN) models are not general improvements to the MLP design.
They are specific choices that match certain types of problem. The CNN
design works best when there is some local pattern in the data (which
may repeat in other locations), and this is often the case when the
inputs are images, audio or other similar signals.

You say:

Or maybe it is mathematically proven that there is always some CNN
which will beat MLP for the given classification problem.

which is an interesting question.  MLP's and CNN's are actually "equivalent" in the sense that if you have an MLP, I can write down a CNN that is identical to it (produces the same output), and vice versa. See discussions here and here.
All this is to say, it's not "CNN vs MLP" that matters as much as the details of the architecture / optimization / etc. The first answer I linked to explains why typical CNN architectures work well.  The phrase "some local pattern in the data (which may repeat in other locations)" is a way of saying there is some translation equivariance in the data.  If there is not translation equivariance, an MLP might work better.


Item Based Collaborative Filtering with No Ratings I am building a recommender for web pages. For each web page in our data set, we wish to generate a list of web pages that other users have also visited.

Our data only shows that a user has either visited a page, or they have not. Users do not provide any ratings of our web pages. This is a good task for item based recommendation. However, most of the algorithms (such as the one in Mahout) requires rating data.

The first solution I came up with was to use a graph database and write a query which does the following:

For each page we want recommendations for, we search for all the users who have viewed that page. Then, for each of those users, we look up all other pages they have viewed. We then count the number of users which have viewed each page in this data set, and use those with the highest count as our recommendations.

While this works pretty well, our data set has grown substantially and scaling the graph database is difficult. The queries become slower as the number of page views in our data set increases. We would like to consider a different implementation before we commit to moving to a distributed graph database.

In a more traditional item-based recommender (like Mahout's), is there a good way to 'fake' the ranking data, or is there a popular open source implementation which does not requires the ranking data?

There might be different ways to do that, like considering implicit ratings like views or clicks.

But basically, you can consider a rating of 1.0 for each user-item pair you have.

This way, your prediction will be between 0 and 1 which you can consider similar to a click prediction probability.

0 being the probability on not having a click or the item by a user
1 being the probability that a click would be made.

You can also set a threshold where you can drop a recommendation at a certain limit of your prediction level.

E.g. Let's say an user alpha has the following recommendations with a threshold of 0.7

i1 , 0.87
i2, 0,75
i3, 0,6

You can drop the i3 recommendation since you don't consider it good enough.

Nevertheless, the parameter of your recommendation engine must be determined of course with the help of your evaluation metrics.

Considering software solutions, I use Apache Spark MLlib with Scala as a base for my recommendation engine algorithms where you can compute item cosine similarity easily per example where you are using in-house implementation or an approximation with the DIMSUM algorithm.

I hope this helps!


Multi scale CNN Network Python I created a multi-scale CNN in python keras. The network architecture is similar to the diagram. Here, same image is fed to 3 CNN's with different architectures. The weights are NOT shared.

I coded the following multiscale CNN in keras which loosely resembles the architecture in the diagram. But I keep getting "Out of memory ERROR" even when the train_dir has 2 images. Would appreciate help...

#main CNN model - CNN1
main_model = Sequential()
main_model.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(32, 3, 3))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(64, 3, 3))
main_model.add(Activation('relu'))
main_model.add(MaxPooling2D(pool_size=(2, 2))) # the main_model so far outputs 3D feature maps (height, width, features)

main_model.add(Flatten())

#lower features model - CNN2
lower_model1 = Sequential()
lower_model1.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model1.add(Activation('relu'))
lower_model1.add(MaxPooling2D(pool_size=(2, 2)))
lower_model1.add(Flatten())

#lower features model - CNN3
lower_model2 = Sequential()
lower_model2.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model2.add(Activation('relu'))
lower_model2.add(MaxPooling2D(pool_size=(2, 2)))
lower_model2.add(Flatten())

#merged model
merged_model = Merge([main_model, lower_model1, lower_model2], mode='concat')

final_model = Sequential()
final_model.add(merged_model)
final_model.add(Dense(64))
final_model.add(Activation('relu'))
final_model.add(Dropout(0.5))
final_model.add(Dense(1))
final_model.add(Activation('sigmoid'))
final_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

print 'About to start training merged CNN'
train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(224, 224), batch_size=32, class_mode='binary')

test_datagen = ImageDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(args.test_images, target_size=(224, 224), batch_size=32, class_mode='binary')

final_train_generator = zip(train_generator, train_generator, train_generator)
final_test_generator  = zip(test_generator, test_generator, test_generator)
final_model.fit_generator(final_train_generator, samples_per_epoch=nb_train_samples, nb_epoch=nb_epoch, validation_data=final_test_generator, nb_val_samples=nb_validation_samples)

UPDATE 1: Providing more system info

I am using Theano-0.9.0.dev5 | Keras-1.2.1 | Python 2.7.12 | OSX Sierra 10.12.3 (16D32) | Macbook Pro 16GB RAM | CPU mode

~/.keras/keras.json contents

{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float64",
    "backend": "theano"
}

Dont have .theanorc file

Please note individual CNN models are training fine. Only the merged code above causes issues.

UPDATE 2: on 27th January, 2017. Tried the following -

Reduced the no. of parameters of the CNN from 53 million to 100K. But still no use. The network eventually fails to train due to memory issues.
Reduced the batch size of images training to 8. The network training fails with the same reason.

No working solution at the time of writing this update...

You could use the following function to determine how much memory your model requires:

def get_model_memory_usage(batch_size, model):
    import numpy as np
    from keras import backend as K

    shapes_mem_count = 0
    for l in model.layers:
        single_layer_mem = 1
        for s in l.output_shape:
            if s is None:
                continue
            single_layer_mem *= s
        shapes_mem_count += single_layer_mem

    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

    total_memory = 4.0*batch_size*(shapes_mem_count + trainable_count + non_trainable_count)
    gbytes = np.round(total_memory / (1024.0 ** 3), 3)
    return gbytes

I copied this from ZFTurbo's answer on this post.


Intuitive explanation of Convolutional layers in Deep CNN I am studying a bit about deep learning and CNNs. I understand the math. However, I am a bit stuck on a basic concept related to filters.
In Convolutional layers, the number of filters corresponding to the depth. For instance, if I have 128 filters, I'll apply each of these individually to the image. Since all these filters are being applied to the image, would this not make it Fully connected? (image is connected to each filter).
Can I have filters of different sizes in a single layer?

In each layer of a CNN you have a number of filters. Suppose for the first convolutional layer you have $C_{out} = 10$ filters. The customary dimension for each of these filters is $(C_{input}, H, W)$ where $C_{input}$ represents the number of channels of the inputs of the current convolutional layer. $H$ and $W$ represents the height and width of the window of the filter. Consider this point that you are convolving each filter with the inputs. An input is a volume with $(C_{input}, H^{'}, W^{'})$ dimension. What you do as convolution is convolving these volumes, each filter and the input. The height and the width of the input $H^{'}$ and $W^{'}$ are the same or bigger than the height and width of each filter, but the depth or the number of the channels of each filter has to be the same as the input channel. After convolving each filter you will have a two-dimensional activation map. By stacking the result of each filter you will have $C_{output}$ number of output feature maps which is equal to the number of filters.


I am doing a forecasting for corona virus cases in NY, I have two models and not sure which one i should choose I am using exponential smoothing and using tableau for forecasting. The first model I included trend and removed seasonality and it predicted the number of cases going up but the quality I got according to tables was "Ok" (Tableau has 3 ranges to describe the quality of a model -Poor, OK and Good). The second model I removed trend and added seasonality to it, This showed the number of cases going down and the quality of the model was good according to tableau. Which model should I be using here, My professor says it's flattening the curve that's why it's going down in the second model while my teammates argue the first model should be the correct one.

I will suggest, first take some training data and make both models on that and compare the result on test data e.g

If you have data from 1st Jan 2020 to 28 Apr 2020.

Give the model, data from 1st Jan 2020 to 31st March(Or you can decide another range), make the model and forecast for April month.

And compare the results for both the models on April month(1st April to 28th April).

In this way, you will be able to argue better with results.


A substitute formula for MSE I don't understand where this formula for Mean Squared Error is coming from.

How do we arrive at:

$$MSE = \frac{1}{m}||y' - y||_2^2$$

from:

$$MSE = \frac{1}{m}\cdot\sum_i(y'_{i} - y_{i})^2$$

(The source is deeplearningbook)

We have $$\|x\|_2=\sqrt{\sum_{i=1}^n x_i^2}$$

Hence $$\|x\|_2^2=\sum_{i=1}^n x_i^2$$

Now let $x=y'-y$ and you obtain your formula.


Implementing a custom hard sigmoid function I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values:

0 if x < 0
1 if x > 1
x if 0 <= x <= 1

How do I implement it with a tensorflow backend Keras?

Based on this post, hard-sigmoid in Keras is implemented as
max(0, min(1, x*0.2 + 0.5)). To obtain the graph you like you have to tweak the shift and slope parameters, i.e. leave them out in your case:

$$
max(0, min(1, x))
$$

This will generate following graph:

For Keras' TensorFlow backend you can find the implementation here.
This would be the corresponding changed "hard-sigmoid", for your case:

zero = _to_tensor(0., x.dtype.base_dtype)
one = _to_tensor(1., x.dtype.base_dtype)
x = tf.clip_by_value(x, zero, one)
return x


Understand clearly the figure: Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification I am studying the blog: Understanding Convolutional Neural Networks for NLP. It is very good blog.

One thing I can't understand clearly about this blog. As the figure Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification as following:

I want to ask:

I know the region sizes(2,3,4) is like 2-gram, 3-gram, 4-gram word, but what’s the meaning of number filters? Here is 2 filters for each region. Why in the author's code about sentence classification is the number of filters defined to 128? Could you give examples to explain the meaning of the number of filters? for example using the sentence of ‘I like this movie very much’ would be great.

2) I understand the height of region size (4) is 4, but in the figure, the height of region(2, 3) are 5 and 6 respectively, I don't know why? I think the height of region is 2 and 3.

Answering this in terms of NLP examples is quite hard, remember "All models are wrong, some models are useful." First think of this in an image classification problem context, you want to use a large number of filters to collect a large number of features out of the image, one could detect edges, the other could detect densely coloured areas, one might turn a region to b&w. Extend  a similar logic to text, by using a lot of filters, in this case 128, you are trying to capture a lot of features. For an example like , " I like movies very much", a certain filter might detect that like is a positive word and not a similarity comparison, a certain filter of size 2 might detect very much and detect that it is an expression of degeree. You can go on like that, it will be hard to come up with 128 features but the idea is to get enough features. If you think the number is unreasonable and might lead to overfitting, you can reduce the number and compare your results.
No, 1- maxpool means that you take the maximum value of the output vector after applying a filter to the input. So it has nothing to do with the longest word but rather choose an element from the output that express the extracted feature to the highest amount.


Is SVD non-linear while PCA (by eigendecompostion) is linear? I am quite confused because a colleague of mine recently told me that he preferred using SVD instead of PCA (by eigendecomposition) because, contrary to the latter, the former is non-linear so it can identify also some non-linear patterns.

However, I cannot see exactly in what way SVD is non-linear since I have the impression that it simply applies a series of linear matrix multiplications (see also this StackExchange answer).

I know that t-SNE is certainly non-linear and for this reason it is sometimes called as non-linear PCA.

Is SVD non-linear while PCA (by eigendecompostion) is linear?

To the best of my knowledge no.

SVD and PCA are both linear dimensionality reduction algorithms. Some nonlinear dimensionality reduction algorithms are e.g. LLE, Kernel-PCA, Isomap, etc.

About t-SNE I would like to add a point. It reduces the dimensionality (and does it pretty well!) but it is only for visualization and can not be used in learning process! So be careful putting all these next to each other. In other words, they are all dimensionality reduction algorithms however, PCA and SVD can be used for feature extraction but t-SNE can not. All can be used for visualization purposes (in EDA).

I certainly recommend reading this answer. Probably the fact that "the square roots of the eigenvalues of $XX^⊤$ are the singular values of $X$" confused your friend that it's a nonlinear method.

Hope it helps. Good Luck!


How to save multi-output predicted masks into two different folders after using model.predict_generator I have a multi output segmentation task, the training process went well, but when Im trying to get the prediction I found difficulties to separate the two output into two different folders, in my dataset I have one image and two relating masks, I used this test generator:

def testGenerator(test_path= "data/membrane/test/image",num_image = 1584,target_size = (224,224),flag_multi_class = False,as_gray = False):
  for i in range(num_image):
    img = io.imread(os.path.join(test_path,"%d.jpg"%i),as_gray = as_gray)
    img = img / 255.
    img = trans.resize(img,target_size)
    img = np.reshape(img,img.shape) if (not flag_multi_class) else img
    img = np.reshape(img,(1,)+img.shape)
    yield img

and this is the code for model.test_generator:

testGene = testGenerator("data/membrane/test/image")
model = new_model()
model.load_weights("new_model.hdf5")
(conv11, convc11) = model.predict_generator(testGene,10,verbose=1)

saveResult(conv11,"data/membrane/test/results/road")
saveResult(convc11,"data/membrane/test/results/cl")

for visualization I used this :

def labelVisualize(num_class,color_dict,img):
    img = img[:,:,0] if len(img.shape) == 3 else img
    img_out = np.zeros(img.shape + (3,))
    for i in range(num_class):
    img_out[img == i,:] = color_dict[i]
    return img_out / 255

def saveResult(save_path,npyfile,flag_multi_class = False,num_class = 2):
    for i,item in enumerate(npyfile):
    img = labelVisualize(num_class,COLOR_DICT,item) if flag_multi_class else item[:,:,0]
    io.imsave(os.path.join(save_path,"%d_predict.tif"%(i)),skimage.img_as_ubyte(img))

solve it, I just did this change:

from:

saveResult(conv11,"data/membrane/test/results/road")
saveResult(convc11,"data/membrane/test/results/cl")

to :

saveResult("data/membrane/test/results/road", conv11)
saveResult("data/membrane/test/results/cl",convc11)

and Ive got each output saved into different folders


Intuition behind Adagrad optimization The following paper ADADELTA: AN ADAPTIVE LEARNING RATE METHOD gives a method called Adagrad where we we have the following update rule : $$ X_{n+1} = X_n -[Lr/\sqrt{\sum_{i=0}^ng_i^2}]*g_n $$
Now I understand that this updation rule dynamically chooses the learning rate for each iteration but have the following question :

Here we see that larger gradients have smaller learning rates and smaller gradients have larger
learning rates. I don't understand why is this the property that is desired, in other words, why is this a good thing for our network

To understand the intuition behind Adagrad, let's have a look at the graphs below, representing the evolution of the loss function when model's weights are updated according to different values of the learning rate in a 1-D search space:

Graph 1 (left - Too Fast!): when updating the weights of the model, we can see that it does not converge to the global minimum. It is due to the fact that the learning rate is too high and it keeps bouncing at the bottom.

Graph 2 (center - Too Slow!): in this case, the weights' update takes place very slowly, because the learning rate is too small, thus it takes too long to converge (or even never) to the global minimum.

Graph 3 (right - Spot On!): in this case, the learning rate is adapted depending on the value of the gradient, i.e. the higher the gradient, the lower the learning rate, or, the lower the gradient, the higher the learning rate. This makes it possible to increase the chances of convergence without having the issues commented before.

Adagrad is one of several adaptive gradient descent algorithms like ADAM, ADADELTA, etc. You can check more info here.

NOTE: the pictures above have been taken from here.


Keras Sequential model returns loss 'nan' I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value.
I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
  while True:
    start_range = 0
    for batch in batch_count:
      end_range = (start_range + batch[1])
      batch_dataset = training_dataset[start_range:end_range]
      batch_labels = training_dataset_labels[start_range:end_range]
      start_range = end_range
      yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

# add input layer
mlp.add(
    keras.layers.Input(
        shape = (training_dataset.shape[1], )
    )
)
# add hidden layer
mlp.add(
    keras.layers.Dense(
        units=training_dataset.shape[1] + 10,
        input_shape = (training_dataset.shape[1] + 10,),
        kernel_initializer='random_uniform',
        bias_initializer='zeros',
        activation='relu')
    )
# add output layer
mlp.add(
    keras.layers.Dense(
        units=1,
        input_shape = (1, ),
        kernel_initializer='glorot_uniform',
        bias_initializer='zeros',
        activation='sigmoid')
    )

print('Compiling model...\n')

mlp.compile(
    optimizer='adam',
    loss=listnet_loss
)

mlp.summary() # print model settings

# Training
with tf.device('/GPU:0'):
  print('Start training')
  #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
  mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
                    steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)

How can I do?

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:

Add regularization to add l1 or l2 penalties to the weights. Otherwise, try a smaller l2 reg. i.e l2(0.001), or remove it if already exists.
Try a smaller Dropout rate.
Clip the gradients to prevent their explosion. For instance in Keras you could use clipnorm=1. or clipvalue=1. as parameters for your optimizer.
Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull().any()
Replace optimizer with Adam which is easier to handle. Sometimes also replacing sgd with rmsprop would help.
Use RMSProp with heavy regularization to prevent gradient explosion.
Try normalizing your data, or inspect your normalization process for any bad values introduced.
Verify that you are using the right activation function (e.g. using a softmax instead of sigmoid for multiple class classification).
Try to increase the batch size (e.g. 32 to 64 or 128) to increase the stability of your optimization.
Check the size of your last batch which may be different from the batch size.


Class weight ineffective in sklearn I'm dealing with an imbalanced dataset and as usual it's very easy to obtain a high accuracy, but the recall on the less frequent class is very low. I would like to improve on the false negative of the less frequent class.

Focusing for concreteness on the sklearn Random Forest, one possible strategy is to set a class_weight penalizing the errors on the less frequent class and scoring with a sklearn scoring function as ROC.

grid = GridSearchCV(pipe, cv=number_of_cross_validations, n_jobs=1, param_grid=param_grid , scoring = scorer)
classifier = grid.fit(X_train, y_train)

where pipe is pipeline containing the RandomForestClassifier and param_grid contains different class_weight to try.

My question is probably related to this question, indeed class_weight alone seems to not be enough to lower significantly the false negative. As an extreme example, if I set:

 class_weight = {0: 0.0000001, 1: 0.9999999}

(where 1 is the class with less instances, with a ratio 1:50), I would expect a final classifier predicting nearly always 1, since every time it makes an error the cost to pay is very high. But that's not what happen, the fitted classifier continue to make a significant number of false negative. Why is it so?

This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.


How do I split number string with digit pattern? I am trying to split number string to two to digit numbers

How do I get two different Numbers out that string

Example:

I want separate two numbers

x <- c("-26755.22-50150.60")

To this

-26755.22
-50150.60

I have tried stringr::str_split but I din't manage to keep the digits.

I just did this-

> library(stringr)
> x <- c("-26755.22-50150.60")
> matc = "\\-[0-9]*\\.[0-9]{2}"
> as.numeric(unlist(str_match_all(x, matc)))
[1] -26755.22 -50150.60

After this, you can select the numbers by using respective indexes.


Alterable Similarity Join for Time Series I'm currently trying to implement the MPdist (matrix profile distance) algorithm for time-series data, but I've developed a new distance metric that I'd like to use in place of the Euclidean metric. I've implemented an algorithm that techincally works, however I used brute force to compute the distance between every subsequence of every time series that I'm using.
As such, it's horribly inefficient. I want to use a similiarity join algorithm to get complexity down. Still, they're really not my area of expertise and most algorithms I've seen are not very transparent and make critical use of Euclidean distance for calculating the 1NN similarity join.
Could someone point me to an article which is perhaps a little more transparent and is such that I could more easily replace the Euclidean metric with another? Or a github repo or something. I only need 1NN

Dynamic Time Warping might be what you're looking for: it measures similarity between two time series based on the optimal alignment between the two sequences. For example point $i$ in sequence 1 might be better aligned with point $i+3$ in sequence 2 based on the evolution of the sequences (as opposed to Euclidean distance which would always compare $i$ in seq. 1 with $i$ in seq. 2). The alignment method is inspired by the Levenshtein edit distance method, which can be implemented efficiently with dynamic programming.
There are many software implementations available.


difference between model-based boosting and gradient boosting What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf
It seems to me that both terms are equivalent. However, both are used in various literature...

Gradient Boosting is fitting a base learner $f_{i}(X)$ to the gradient of the loss function of an existing model $F_{i-1}(X)$ i.e. find base learner $f_i$ which minimises $L(-g_i, f_t(x_i))$ where $g_i$ is the gradient of $L(y_i,\hat{y}_i)$ with respect to $\hat{y}=F_{i-1}(X)$ at the current iteration $i$. Effectively it's gradient descent in function space.

Component wise boosting schemes such as that used by mboost have a list of base learners of which one is selected at each step, i.e.

form2 <- y ~ bols(x1) + bols(x2) + bols(x1, by = x2, intercept = FALSE) +
               bspatial(x1, x2, knots = 12, center = TRUE, df = 1)

Specifies 4 possible base learners, bols(x1), bols(x2), bols(x1,by=x2) and bspatial(x1,x2), all of which are regression splines.

More generally gradient boosted decision trees fits a tree at each step. So the base learners are arguably more complex.

I believe the terms model based and functional are equivalent and both `mboost' and GBDT are examples.


Setting attribute weights in Weka I'm working on feature weighting techniques (chi-square, relief..) for classification tasks using Weka. Can I add these weights to the dataset's attributes? If yes, how? Do the classification algorithms in Weka make use of the feature weights?

Yes. You can add weights to the dataset's attributes.
To first add the weights:

Open the dataset in explorer
Perform any required filtering (if necessary)
Click the "Edit" button on the top panel
Right click on the respective attributes you wish to assign a weight to
Once you implemented your respective weights, click "OK"

You have now implemented weights to your attributes! However, one must keep in mind that these weights will only work for some boosting classification algorithms such as AdaBoostM1 and its variants (MultiBoostAB, etc.). Additionally, the sampling option must also be disabled and base classifier must also be a WeightedInstancesHandler. Otherwise, the algorithms likely won't use the weights correctly.


Replacing value not in list in Pandas Say I have a data frame in Python.
If I want to replace all values in the say, the Size column that are not 'M' or 'S' or 'L' with nan, how do I do so?
Thanks in advance

I don't know if it's the best way, but I'd write a function and apply it to the df['size']:
def rename_size(size):
    if size not in ['M','S','L']:
        return np.nan
    else:
        return size
df['size']= df['size'].apply(rename_size)


Feature Selection, Machine learning and Time Series analysis, for large financial timeseries I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

I have m( around O(millions) ) of rows of type

timestamp | val | ind1 | ind2 | ind3 | .... k entries

My task is to predict the value of "val" for any future given time. How should I model this as a machine learning problem. Note that we want the value of "val" given the past data without knowing the current value of

ind1 | ind2 | ind3 | .... etc.

of the same row. This is different in that sense.

Formally, assume that I have first i rows(each entry is known) of the table. And I want to predict the value of $val_j$ (j>i, i.e. near future). How will I do that? I have not done these kind of predictions using ML or TensorFlow earlier . If we were given $ind1_j,ind2_j,...$ etc but not $val_j$. This kind of prediction would have been easy.

Even if I model my val as a function of ind1, ind2, etc of previous indices. How will I get it for different future times ?

About data I do not know much about data. All I know is that it is related to stocks and exchanges. Maybe "val" is the entity which is dependent on ind1, ind2, etc. You assume whatever is suitable apart from that.

And please give any idea, How will system learn iteratively as new data rows are added. I do not want to do the whole training again and again.

Your model should be based on what your data represents.  If you want meaningful results, you should get some idea of what your data is representing before trying to answer this question.

For instance, you could use your ind# columns to predict the values of your ind#${_j}$ rows, then use the ind#${_j}$s to predict the val${_j}$.  But this would only make sense if there is a predictive relationship between the ind#${_j}$s.  And you would definite want to have some idea of what that relationship might be in order to choose an appropriate algorithm for each particular column.

Furthermore, the same would apply for $j_{th}$ row val${_j}$ prediction.  At the very least you would want to  make an educated guess about the relationship between your ind#${_j}$s and your val${_j}$ column so that an appropriate algorithm could be chosen.

As another example, you may be able to use the previous row as a predictor for the next value in the time series. i.e. ind#${_{j-1}}$s  => val${_j}$.  But again, this only makes sense if you are able to make reasonable assumptions about the relationship of the data, because you ultimately need to choose the algorithm you use to make the predictions.