Statistical Commute Analysis in Java I have a rather large commute every day - it ranges between about an hour and about an hour and half of driving.

I have been tracking my driving times, and want to continue to do so. I am capturing the date, my time of departure, my time of arrival, the route I took (there are two or three possible ones), weather conditions (wet/dry and clear/hazy/foggy), and whether I stopped (and if so, for what reason - fuel/toilet break/food break, and for how long) for every journey to and from work.

I would like to create a system to analyse this data and suggest an optimal departure time (for the next journey) based on day of the week, weather conditions, and whether i need to stop.

Anecdotally, I can see that Tuesday mornings are worse than other mornings, the earlier I leave the more likely I am to take a toilet break or a food break, and obviously that the journey takes longer on rainy or foggy days than on clear and dry days - but I would like the system to empirically tell me that!

I assume this is a machine-learning and statistical analysis problem.

However, I have absolutely no knowledge of machine-learning, or statistical methods.

What statistical methods should I use to do this kind of analysis to the point where the data will lead to suggestions like "tomorrow is Tuesday and it is going to rain, so you must leave home between 7.50 and 8.00, and take route XYZ, to get the optimal driving time. Oh and chances are you will need a toilet break - and I have factored that in"? (assume that I manually enter tomorrow’s weather forecast - I’ll look into integrating with a weather service later)

Note that this is life-hacking for me, trying to optimise the hell out of a tedious process, and it is very personal - specific to me and my habits, specific to this route, and specific to the morning/evening commute times. Google Maps with Traffic, TomTom with IQ, and Waze do very well in the more open-ended situations of ad-hoc driving-time prediction. Even Apple is happy to tell me on my iPhone notification screen how long it will take me to get home if I leave right now.

Also note, it appears to me that traffic is not a consideration - that is to say, I do not think I need to know the actual traffic conditions - traffic is a function of day of the week and weather. For example, there are more people on the roads on Monday and Tuesday mornings, and people drive more slowly, and more people are in cars (opting to drive instead of cycle or take public transport) when it rains.

To what extent can I let the data do all the talking? I have a somewhat ambiguous hidden agenda which may not be apparent from the data;

I should be at work at 9.30 (i.,e. 9.15 +/- 15 minutes) every day, but the occasional 10am arrival is OK
I want to leave home as late as possible, and yet arrive at work as early as possible
I want to leave work as early as possible, and yet have done at least 8 hours’ work
it is OK for me to, say, leave half an hour early on one day but stay late on another to compensate

I think I can come up with a procedural formula that can encompass all of these rules, but my gut feeling is that statistical analysis can make it a lot smarter.

Apart from the methods of analysis, the technology stack is not an issue. Java is my language of choice - I am quite familiar with programming in it, and in creating web applications.

Assuming that it is possible, are there Java libraries that can provide the requisite methods?

What limitations are there? I want to keep capturing more and more data every day, making the data set bigger, hopefully, making the prediction more accurate.

What other ways are there to do it? Can I push this data into, say, Wolfram Programming Cloud, or maybe something Google provides to get the desired results?

You do want to model the traffic, at least over a work day, otherwise it wouldn't matter what time you traveled! Absent any data, I'd assume there isn't much variance over the working week, but that's one thing the data will quickly confirm or refute. If it is varying, you can use a different model for each day.

You have two variables; the departure times from home and work, respectively. Let's call them t_h and t_w. Let's call the commute time T_c(t), where t is the time of day. You can estimate this function from the data, so I'll assume it is given.

You want to maximize c t_h - (1-c) t_w subject to the constraints t_h + T_c(t_h) < 9.5 and t_w > t_h + T_c(t_h) + 8

where c is a constant you can set to adjust the relative importance of leaving home early relative to leaving work early. You should be able to solve this numerical optimization problem with Mathematica, MATLAB, or something similar. I would not recommend Java; it's not meant for this. The only tricky part is estimating T_c. You know that it's a non-negative function, so you could use the standard trick of estimating it's logarithm (say, with kernels) and exponentiating. For implementation with Mathematica see Smoothing Data, Filling Missing Data, and Nonparametric Fitting and Constrained Optimization.


Facing a difficult regular expression issue in cleaning text data I am trying to substitute a sequence of words with some symbols from a long string appearing in multiple documents. As an example, suppose I want to remove:

Decision and analysis and comments

from a long string. Let the string be:

s = Management's decision and analysis and comments is to be removed.

I want to remove Decision and analysis and comments from s. The catch is, between Decision, and, analysis, and, comments, in s there could be 0, 1 or multiple spaces and newline characters (\n) showing up with no pattern in different documents, for example, one document shows:

Management's decision  \n \n and analysis\n and \n comments is to be removed

while another has a different pattern. How do I account for this and still remove it from the string?

I tried the following, of course unsuccessfully:

st = 'Management's decision  \n \n and analysis\n and  \n comments is to be removed'
re.sub(r'Decision[\s\n]and[\s\n]analysis[\s\n]and[\s\n]comments','',s)

To remove multiple white space matches, you will need [\s\n]+, note the inclusion of the + (match one or more).

Code:

Here is a function which will build the regex automatically from a text snippet:

def remove_words(to_clean, words, flags=re.IGNORECASE):
    regex = r'[\s\n]+'.join([''] + words.split() + [''])
    return re.sub(regex, ' ', to_clean, flags)

Test Code:

st = "Management's decision  \n \n and analysis\n " \
     "and  \n comments is to be removed"
print(remove_words(st, 'decision and analysis and comments'))

Results:

Management's is to be removed


Event prediction sequence I need to create an app to predict events.

I'll explain my requirements: I've a set of intermediate events (eg.: 1, 2, 3...9) and a set of final event (eg.: A for OK and B for KO). This events bring a correlation id to relate each other.

What ML algorithm can I use to predict if a sequence of events could cause a future KO event (B in my example) or not?

I need an unsupervised algorithm.

Any idea about the algorithm?

Maybe sequential association rules could be applied here. Please check spmf java library, section 'Sequential Rule Mining'.


Algorithm/Analysis that utilises incremental information? I am looking for techniques which utilise information in an incremental manner.

Example: A day with inclement climate is likely to be followed by another day with inclement climate. Or when an entire dataset sorted by date is available, an algorithm able to identify that a person is likely to call in sick when the previous day had bad weather.

Is there any effective analysis which utilises this prior information where not all events are completely independent?

It looks like you are looking for time series based Machine Learning (or Sequential Machine Learning as Emre pointed out).

This means that you have an entire dataset available to you beforehand. You train on this dataset and then use it to predict next day's weather given the weather of the last n days. Please note that you do not learn incrementally here, i.e. you do not learn from the new data that is rolling in every day. The learning is limited to the dataset that you had beforehand.

Widely used algorithms for this kind of learning are Conditional Random Fields(CRF), Hidden Markov Models etc. Statistical methods like Autoregressive Models can also be used sometimes. A good starting point would be CRFs.

If on the other hand you want to learn incrementally, read about online learning.


Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint We have followed the following steps:

Trained 5 TensorFlow models in local machine using 5 different training sets.
Saved those in .h5 format.
Converted those into tar.gz (Model1.tar.gz,...Model5.tar.gz) and uploaded it in the S3 bucket.
Successfully deployed a single model in an endpoint using the following code:

from sagemaker.tensorflow import TensorFlowModel
sagemaker_model = TensorFlowModel(model_data = tarS3Path + 'model{}.tar.gz'.format(1),
                                  role = role, framework_version='1.13',
                                  sagemaker_session = sagemaker_session)
predictor = sagemaker_model.deploy(initial_instance_count=1,
                                   instance_type='ml.m4.xlarge')
predictor.predict(data.values[:,0:])

The output was:
{'predictions': [[153.55], [79.8196], [45.2843]]}
Now the problem is that we cannot use 5 different deploy statements and create 5 different endpoints for 5 models. For this we followed two approaches:
i) Used MultiDataModal of Sagemaker
from sagemaker.multidatamodel import MultiDataModel
sagemaker_model1 = MultiDataModel(name = "laneMultiModels", model_data_prefix = tarS3Path,
                                 model=sagemaker_model, #This is the same sagemaker_model which is trained above
                                  #role = role, #framework_version='1.13',
                                  sagemaker_session = sagemaker_session)
predictor = sagemaker_model1.deploy(initial_instance_count=1,
                                   instance_type='ml.m4.xlarge')
predictor.predict(data.values[:,0:], target_model='model{}.tar.gz'.format(1))

Here we got an error at deploy stage which is as follows:
An error occurred (ValidationException) when calling the CreateModel operation: Your Ecr Image 763104351884.dkr.ecr.us-east-2.amazonaws.com/tensorflow-inference:1.13-cpu does not contain required com.amazonaws.sagemaker.capabilities.multi-models=true Docker label(s).
ii) Created endpoint manually
import boto3
import botocore
import sagemaker
sm_client = boto3.client('sagemaker')
image = sagemaker.image_uris.retrieve('knn','us-east-2')
container = {
    "Image": image,
    "ModelDataUrl": tarS3Path,
    "Mode": "MultiModel"
}
# Note if I replace "knn" by tensorflow it gives an error at this stage itself
response = sm_client.create_model(
              ModelName        = 'multiple-tar-models',
              ExecutionRoleArn = role,
              Containers       = [container])
response = sm_client.create_endpoint_config(
    EndpointConfigName = 'multiple-tar-models-endpointconfig',
    ProductionVariants=[{
        'InstanceType':        'ml.t2.medium',
        'InitialInstanceCount': 1,
        'InitialVariantWeight': 1,
        'ModelName':            'multiple-tar-models',
        'VariantName':          'AllTraffic'}])
response = sm_client.create_endpoint(
              EndpointName       = 'tarmodels-endpoint',
              EndpointConfigName = 'multiple-tar-models-endpointconfig')

Endpoint couldn't be created in this approach as well.

I also have been looking for answers regarding this before, and after several days of trying with my friend, we manage to do it. I attach some code snippet that we use, you may modify it according to your use case
image = '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.2.0-cpu'
container = {
    'Image': image,
    'ModelDataUrl': model_data_location,
    'Mode': 'MultiModel'
}

sagemaker_client = boto3.client('sagemaker')

# Create Model
response = sagemaker_client.create_model(
              ModelName = model_name,
              ExecutionRoleArn = role,
              Containers = [container])

# Create Endpoint Configuration
response = sagemaker_client.create_endpoint_config(
    EndpointConfigName = endpoint_configuration_name,
    ProductionVariants=[{
        'InstanceType': 'ml.t2.medium',
        'InitialInstanceCount': 1,
        'InitialVariantWeight': 1,
        'ModelName': model_name,
        'VariantName': 'AllTraffic'}])

# Create Endpoint
response = sagemaker_client.create_endpoint(
              EndpointName = endpoint_name,
              EndpointConfigName = endpoint_configuration_name)

# Invoke Endpoint
sagemaker_runtime_client = boto3.client('sagemaker-runtime')

content_type = "application/json" # The MIME type of the input data in the request body.
accept = "application/json" # The desired MIME type of the inference in the response.
payload = json.dumps({"instances": [1.0, 2.0, 5.0]}) # Payload for inference.
target_model = 'model1.tar.gz'

response = sagemaker_runtime_client.invoke_endpoint(
    EndpointName=endpoint_name,
    ContentType=content_type,
    Accept=accept,
    Body=payload,
    TargetModel=target_model,
)

response

also, make sure your model tar.gz files have this structure
└── model1.tar.gz
     └── <version number>
         ├── saved_model.pb
         └── variables
            └── ...

more info regarding this


How to approach Multilingual Text Classification? The company I'm working for runs social network sites, and we're classifying messages sent one-to-one as spam or not spam. The issue is that it's been trained exclusively on German training data, since most of the networks we operate are local to Germany. Soon we will need to support multiple languages for this, since the one trained on only German will often think messages in other languages are spam as well.

I'd prefer not to have to maintain different training data for different languages if possible, so my question is:

What's a reasonable approach to doing text classification for multiple languages?

A spam message in our cases usually contains obfuscated links and subtle references to other websites. Example of a message that should be classified as spam:

  hi funnyguy kennst du mich noch ? konnte hier kein bilder hoch laden komm mal bitte zu ( somethingelse.com) mein nik ist ( 19theusername91 ) mal sehen ob du mich noch kennst :)

  Somegirl

The current approach is the normal stemming, TF-IDF and LSA for pre-processing, then a two-level classifier: an ensemble of normal classifiers that's used as input for a linear classifier that will make the final decision.

In a similar situation, -after trying some alternatives- I had to build a language classifier in front of all learning and classification steps.

That is, for learning:

Detect the language of the input (say, an enumeration like "DE", "EN", etc.)
Apply language specific stemming to the words of the input.
Prepend words in the learning phase with the language identifier (i.e. "de_du", "en_you")
Use these words in a single training model.

In classification phase, use the same steps.


Machine Learning with intended missing values I have a dataset relating to humans completing reviews, the target variable is whether the review decision is correct / incorrect and one of my features is a trailing 4 week accuracy score for the reviewer.
These accuracy scores are not always available however. My question is around how to model this data - the fact that there is no available accuracy score might be a signal. From my research into this - everything I see tells me that the missing values must be imputed or removed. I am wondering whether there are techniques to incorporate the fact that the data is missing into the dataset.
Perhaps I could convert the score into a categorical variable {low, medium, high, not available] - would this be common practice? I am open to suggestions and would love to hear what is commonly done in these scenarios

The common case of missing values for which data is imputed or removed assumes that missing values appear randomly in the data, so the absence of value has no relevance to the task.
From your description, in your data the fact that a value is missing is significant by itself. So I'd say that yes, it makes sense in this case to represent this information as a categorical variable. Note that it can be represented as a special value for the score feature indeed, but it doesn't have to be the same variable.


Modeling and Predicting Co-Occuring Values I have data for a 100,000 people with different personality traits. Here would be sample data:

Person  Trait-1   Trait-2   Trait-3 .....   Trait-N
John     1          0         1                1

I need a model where for a new user when I see Trait-X, I need a prediction of the likelihood of all other Traits i.e. how likely is he to have any of the other traits. Can someone point me to a possible model I could use? I am a novice so don't know much about this space.

I would recommend a Restricted Boltzmann machine, which is a type of neural network that can model probability distributions. What you are trying to do is to estimate the marginal distribution of the missing features. In a probabilistic model, this is done through marginalization. The beauty of this method is that it provides the full distribution over the missing features rather than mere point estimates.

Here are two tutorials.


Feature selection, is it possible to combine wrapper and embedded methods? I'm using neural network to predict PM10 concentration (a regression problem). Since the wrapper method is dependent on the model, so passing the neural network model that's optimized for all the features in the dataset won't work correctly (since every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned).

So my question is, can I use the embedded method (Lasso) as an estimator in the wrapper method?

This is the feature selection part that I' asking about:

from mlxtend.feature_selection import SequentialFeatureSelector

feature_selector = SequentialFeatureSelector(LassoCV(),
           k_features=3,
           forward=True,
           verbose=0,
           scoring='neg_mean_absolute_error',
           cv=0)
features = feature_selector.fit(X, y)
filtered_features= cols[list(features.k_feature_idx_)]
print(filtered_features)

I'm new to data science and machine learning, so I want to know if this is right or wrong.

Can I use Lasso as the estimator in a "wrapper" feature selection method?

It will certainly work, but the embedded feature selection of Lasso is unlikely to actually come into play in your example.  The forward sequential feature selection builds the model for each candidate feature to be added to the selected feature set, starting from no features.  When you only build up to three features then, the only way Lasso's feature selection will have an effect is when it determines that fewer than three variables should be kept, or if it somehow decides that one of the already-selected features should have zero coefficient.  The latter would be very surprising, and the former only makes sense if your data is very simple (only two variables contribute anything significant to the predictions).

So, here Lasso is probably only really contributing as regularization, shrinking coefficients but probably not zeroing any of them out.

How should I do feature selection for a neural network?

Since Lasso is a linear model assuming feature independence, you may be handicapping the net's ability to find and use nonlinear relationships or feature interactions.

I doubt there's a "right" answer here (no free lunch sort of thing).  Here are some ideas though.

Don't do feature selection.  Count on the neural network, perhaps with heavy regularization methods (dropouts, L1 penalty especially on the first layer, etc.), to sort out what's important.
Do hyperparameter tuning inside the wrapper.  This is very computationally expensive, but may be the most performant approach, and might be doable for smallish nets.
Fix the architecture inside the wrapper.  You say "every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned," but this might not be a problem.  Often enough, if a net has more neurons than it needs, several may learn nearly the same feature.  (You might try to turn this into a compromised version of (2), by applying some network pruning instead of full hyperparameter tuning.)
Use another model inside the wrapper: one that can capture nonlinearities and feature interactions unlike a linear model, say a tree-based model.  This still may fail to attribute importance to features that a network would pick up on, but it should be closer than a linear model.


Estimating Titan X graphics card impact on performance I'm currently training CNNs using Tensorflow (Python) on my GTX 970 (specs here). I recently took a look at the new pascal based Titan Xs and I'm wondering what an estimated performance/speed gain would be if I upgraded?

The memory increase is an obvious benefit for larger models, but I'm mostly wondering about speed.

Does the doubling of core count and bump in memory speed offer that much of a performance gain? Anyone have any experience with the new (or old) Titan X cards?

There are a lot of parameters which matter when using GPU's for machine learning, some of them are:

CUDA core count
Memory bandwidth (GB/s)
Memory per core (MB)
Raw Speed (MHz)
Total Memory available (GB)
Performance on 16-bit, 32-bit floating ops/sec

Tim Dettmers has an excellent (frequently updated) blog where he's compared different cards, near the end he's also given a simple speed-up comparison. Using that as a guide, it can be estimated that Titan X pascal would be upto 5 times faster than a GTX 970 for Deep Learning.


Are euclidian vectors and unit vectors same thing? Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any vector in R3 is a linear combination of e1, e2 and e3.

A unit vector in a normed vector space is a vector of length 1. A unit vector is an euclidian vector of length 1. Not every euclidean vetor has length 1.


What is the effect of highly correlated data on a Convolutional Neural Network? A speech audio sample can be converted to MFCC coefficients for further analysis. I wanted to know the effect of correlated data on a CNN. I know the process of computing the MFCC coefficient, which applies Discrete Cosine Transform (DCT) to decorrelate the filter bank coefficients and yield a compressed representation of the filter banks.

I wanted to argue on the usage of filter banks vs MFCC when we want to further analyze using a CNN.

Filter banks result in highly correlated data.
MFCC result in decorrelated data.

Which of the two should i choose if i plan to train a CNN model for speech classification?

I actually don't believe it really matters.  When applying the same idea to visual problems, you'll find that the raw RGB vs a DCT compressed version lead to similar results as seen here.  If there exist a negative, one may argue that the correlated data may have a slower learning rate.  Nevertheless, the best way to figure out the effect is to give both a try.


How to obtain original coefficients after performing linear regression on normalized data? I am reading data from a file using pandas which looks like this:

data.head()

   ldr1  ldr2  servo
0   971   956     -2
1   691   825   -105
2   841   963    -26
3   970   731     44
4   755   939    -69

I proceed to normalize this data to perform gradient descent:

my_data = (my_data - my_data.mean())/my_data.std()
my_data.head()

       ldr1      ldr2     servo
0  1.419949  1.289668  0.366482
1 -0.242834  0.591311 -1.580420
2  0.647943  1.326984 -0.087165
3  1.414011  0.090200  1.235972
4  0.137231  1.199041 -0.899949

I perform multivariate regression and end up with fitted parameters on the normalized data:

Thetas:  [[-3.86865143e-17,  8.47885685e-01, -5.39083511e-01]]

I would like to plot the plane of best fit on the original data and not the normalized data using the normalized thetas.

I used scipy.optimize.curve_fit to perform multivariate linear regression and come up with the optimal fitted parameters. I know that the original thetas should be close to the following:

[   0.26654135   -0.15218007 -107.79915373]

How can I get the 'original' thetas for the original data-set in order to plot, without using Scikit-Learn?

Any suggestions will be appreciated.

Coefficients of the linear regression for unnormalized features

If parameters in the normalized space are denoted as $(\theta_0', \theta_1', \theta_2')$, parameters in the original space $(\theta_0, \theta_1, \theta_2)$ can be derived as follows
$$\begin{align*}
y' &= \theta_2'x_2'+\theta_1'x_1'+\theta_0'\\
\frac{y-\mu_Y}{\sigma_Y} &= \theta_2'\frac{x_2 - \mu_{X_2}}{\sigma_{X_2}} + \theta_1'\frac{x_1 - \mu_{X_1}}{\sigma_{X_1}} +\theta_0' \\
 y &= \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_2}}\theta_2'\right)}^{\theta_2}x_2+ \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_1}}\theta_1'\right)}^{\theta_1}x_1
 + \overbrace{\sigma_Y\left(-\theta_2'\frac{\mu_{X_2}}{\sigma_{X_2}}-\theta_1'\frac{\mu_{X_1}}{\sigma_{X_1}} + \theta_0'\right) + \mu_Y}^{\theta_0}
\end{align*}$$

Generalization to D features

$$\begin{align*}
\theta_d = \left\{\begin{matrix}
 \sigma_Y \left(\theta_0' - \sum_{i=1}^{D}\theta_i'\frac{\mu_{X_i}}{\sigma_{X_i}} \right) + \mu_Y& d=0\\
\frac{\sigma_{Y}}{\sigma_{X_d}}\theta_d' & d > 0
\end{matrix}\right.
\end{align*}$$

A trick

For visualization, we can plot the plane in original, un-normalized space without changing the parameters (Thetas). We only need to re-label (re-scale) the plot axes.

For example, a point $(x_1', x_2', y')$ in the normalized space corresponds to point $$(x_1, x_2, y)=(\sigma_{X_1}x_1'+\mu_{X_1}, \sigma_{X_2}x_2'+\mu_{X_2}, \sigma_{Y}y'+\mu_{Y})$$

in the original space. So you just need to rename the plot axes from $(x_1', x_2', y')$ to $(x_1, x_2, y)$.

Note that $y'=\theta_2'x_2'+\theta_1'x_1'+\theta_0'$ is still calculated using normalized features.


After choosing top models in classification? Can I apply it on the rest of my dataset I am working with a corpus that has 5 datasets in product reviews (A, B, C, D and E), mine is a text classification problem and I need to find the best 5 top models in terms of classification performance (F1).

I started with collection A: the mp3 reviews,  Because it has the largest numbers of documents (900: yes, 750: No).

I trained the data using 10-fCv using different algorithms and pre-processing tasks, got the weighted results for all experiments.

I chose the top 5 models and I want to apply them to the rest of the corpus: B, C, D and E (other products' reviews).

My plan is to run 10-fCv and get the results for all the collections and compute the Micro-average for precision, recall and F1.

Is this the right way to choose a model for a large collection?

This is an interesting question.

In general the split of data is about the underlying distribution. That means you split a dataset into train-test sets in a way that random split of train-test does not dramatically affect the distribution. But Splitting based on topics is not random!

Specially in your case you are talking about text in which the distribution is super sensitive to the domain i.e. if you collect the commentary of 1000 football games and the narrations of 1000 documentary movies about wild life, you will see that they are literally two different things. The conceptual difference between products most likely affects the distribution of words/terms/phrases therefore the model trained on reviews of mp3s MUST NOT be validated on reviews of football shoes!

In your case, I would say the train-test split (CV folds) should be done on whole data together so that you maintain the original topology of word distribution (topology here is not a Math term but I simply mean the shape of distribution).

In this case if you do Topic Modeling on the whole training data you simply see 5 different product topics. Or if you use word2vec or doc2vec you hopefully see 5 different clusters. Then you can run your models in this setting for evaluation.

NOTE:

If the size of classes are very different you need to come up with some solution for small classes. If it was the case, just drop me a line in comments and we can discuss solutions.

Good Luck :)


how does splitting occur at a node in a decision-tree with non-categorical data? According to a website (:http://dataaspirant.com/2017/01/30/how-decision-tree-algorithm-works/) , these values are chosen randomly:

I don't think this is the case with any optimized way of creating a decision tree. In this image(different example) the value is 2.45 for the root node:

Was this value chosen randomly like explained in the website? If not and the value is not chosen randomly then how is it calculated?

No, I don't think values according to which the branches are seperated are chosen at random. Instead, weighted average is calculated for each category and the category with the highest weighted average is chosen as the root node. This is also referred as Information gain
Consider this dataset

.

Consider the above picture, Here the outlook is chosen as Root node, And how is outlook chosen as root node?

First, We calculate the total entropy of the data. Lets say its 0.95. Now inorder the pick the right root node, We will find weighted averages of all the subcategories. There are 4 four categories here, So we will obtain 4 weighted entropy averages. Lets say they are 0.3, 0.2, 0.4, 0.8. Now we will subtract the induvidual weighted entropy averages from the total entropy. So we will get (0.95-0.3), (0.95-0.2), (0.95-0.4), (0.95-0.8). Among all the three which ever category has the highest value that category will be chosen as the root node. These 4 values are the information gain of each of the categories i.e Whichever category has the highest information gain, we will pick it as the root node. In our case, its the outlook category/feature .Hope it helps

Check this for more clarity


Explaining machine learning models Is there a way to get an explanation of the model prediction for a specific example?

You can use LIME. It gives you the most relevant features to a certain prediction the model gave by approximating the model locally as a simpler model. Here is a good lecture on it: https://www.youtube.com/watch?v=HcaAKI1tVGM


Difference between "reducing batch_size" and "increasing epochs" to decrease loss amount? In my experience, both reducing batch_size and increasing epochs can decrease loss amount. But I like to know is there any difference to using which one? Has decreased loss amount same meaning and it's not important you reached it with what way(I mean has no effects on results)?

For ex, I got same loss amount 2.5e-4 with both the following case:

1. batch_size = 1 , epochs = 100
2. batch_size = 60 , epochs = 1000

Are they same result?

The concept of overfitting does not generalize over a specific combination of batch size and epochs. It depends on your data and the architecture of your model

A friend of mine ran into these scenarios with a CPU based image classifier:

1) If I use more epochs ,it may take me a lot of time to come to a desirable
   outcome.

2) If I prefer small batch-sizes over small epochs , It might take less time
   time to compute , but not reach the desirable outcome by that epoch limit.

I used a GPU and my results were different. Using low epochs , and better convolutional architecture , I reached a better accuracy with not so small batch sizes.

I increased epochs, and my accuracy improved until I felt I reached overfitting.
I increased batch-sizes and my accuracy was not increasing at a decent rate.

I had to come to balance at which my model was acceptable.

Its a balance , yes. A balance which I am afraid, the designer needs to take care of. Not always they are inversely proportional. The data set and the architecture makes all the difference in this debate I am afraid.


Do the benefits of ridge regression diminish with larger datasets? I have a question about ridge regression and about its benefits (relative to OLS) when the datasets are big. Do the benefits of ridge regression disappear when the datasets are larger (e.g. 50,000 vs 1000)? When the dataset is large enough, wouldn't the normal OLS model be able to determine which parameters are more important, thus reducing the need for the penalty term? Ridge regression makes sense when the data sets are small and there is scope for high variance, but do we expect its intended benefits (relative to OLS) to disappear for large datasets?

One thing that you might be overlooking with your reasoning is the fact that increased predictors does not necessarily lead to a better model. In this case, the more predictors you have the higher the risk of collinearity that there is between them - thus increasing the utility of ridge regression.
In fact, a second point:
The larger the number of predictors that you have in your model the more useful techniques like ridge regression are. This is because with so many predictors is is very difficult to determine if collinearity or other relationships exist between predictors. When you have a smaller model with say 5 predictors, you could verify this fairly easily. With 1,000 predictors this would be much more difficult.
In the case of the Lasso regression the benefits are even more obvious. Since predictor coefficients values can be shrunk fully to zero this acts as a form of feature selection. Here, you are potentially removing predictors that capture redundant information.
Also - in answer to the second part of your question - OLS does not perform any form of feature selection. Therefore, it will not be able to pick out which parameters are the most important by adding more. The more parameters you add to your model the higher the variance gets. Why is this? You have more estimations to make and therefore collectively your model is less reliable. Consequently, your model will likely inflate the coefficients on the predictors giving undue "weight" or importance to certain predictors.


How to see which transformation is the best In R I have data and I want to make a regression analysis, finding a function that can fit the data.
So head(data) gives

promotion   new_users
39.5              100
36.1               79
 0.0               18

To find the optimal regression function that fitted data I plot the residuals of the regression model to see if the residuals are systematic close the zero line. But I do not know which transformation is the best one to use. Here I tried to do linear transformation, sqrttransformation and finally log-transformation.

lm.linear = lm(formula= data$new_users ~ data$promotion )
plot(resid(lm.linear), col="blue")

lm.sqrt = lm(formula= data$new_users ~ sqrt(data$promotion) )
plot(resid(lm.sqrt), col="blue")

lm.log = lm(formula= data$new_users ~ log(0.1+data$promotion ))
plot(resid(lm.log), col="blue")

If I simply just plot the data and the fitted regression function I can't see which regression function fittest the data best because they are very similar.
Which transformation is the best one and is there another way to find out what is the best transformation ?

Thanks.

To see if I can use poission regression model I type

model=glm(data$new_users ~data$promotion, familiy="poisson", data=data)

I use goodnes of fit to see if the model fits.

with(m1, cbind(res.deviance = deviance, df = df.residual,
  p = pchisq(deviance, df.residual, lower.tail=FALSE)))

I get a low p-value meaning that the model is a good fit?

Furthermore say we want to compare two poison regression models (one poission regr. model from data and another one from another dataset say data_new) and see if there is a significantly difference between the two, how would one do that? I assume one could use anova test to solve this.

There are two main reasons people typically transform their data.  Either to help meet assumptions or to help boost predictive performance.

Help meet assumptions
In your case of a linear regression,  you have several assumptions that you have to meet including things like linearity and residuals have a constant variance and appear independent of each other.  Transforming your response can help make your relationship more linear and help with your residual assumptions, while a transformation on your predictors primarily helps with those residuals.  Typically if you are doing inference, you need normal residuals (or large sample size for CLT to kick in) which you can assess with a qq-plot or a test for normality, but if you are doing predictions you don't have to worry about this.  Trying a few popular transformations (sqrt, square, log) is a good idea, as is the Box-Cox transformation (on predictors) which basically finds a transformation that maximizes the normal likelihood function of the residuals.  The box-cox goes through several power transformations in a systematic manner, and it primarily used to make the residuals normal, though it can help the other issues too.
Improve predictive power
This is more of an open game, and I would be wary of over fitting.  See what works best and go with it (can be tough, might have to go with cross-validation if you're data isn't too big).
Your Case
Usually your x-axis for those plots is the fitted values (you can plot a glm object, not sure about lm, so run through most of your basic diagnostic plots). just with those plots its also tough to assess normality, which is typically done with a qq-plot or a hisotgram of the residuals.  It looks like your response variable new_users is a count; maybe a poisson model would be another option.  "best" transformation is the one that helps you meet your assumptions as noted in (1), its more of finding the transformation that makes you feel most confident that your assumptions are met.  If you are looking to increase performance, $R^2$ will help indicate a helpful transformation, but again be wary of overfitting.  The power transformations e.g.$(x^{1/3}), log(x+0.1), x^{2}$ are usually a good route to go.


How to measure variance in a classification dataset? I have a dataset that contains 20 predictor variables (both categorical and numeric) and one target variable (2 classes - Good and Bad). But, there are only 23 observations in the dataset. While I wait to receive significantly more observations, what tests / models can I perform on the available dataset to understand the variance between the good and bad cases, and to understand the variance within the cases classified as 'good'?

Ideally, for the data to make sense, I would want the variance within the good cases to be low, and the variance between the good and bad cases to be high.

Would multivariate analysis of variance (MANOVA) work in this case?

I would start with logistic regression, and get a measure of importance for each of your 20 predictor variables. It's a little tough to understand what you mean by 'variance' in your two classes of good and bad. If you mean variance in your predictor variables that lead to your response, you can calculate the variance of a linear combination of predictor variables (in your logistic regression model) using an appropriate theorem from Wikipedia


Classify text as logical/ not logical Can some one advise me direction where to look in.Or some resources. Here is a task:

User leaves feed back-text with min 50 characters.
I need to check if it's normal human sentences/ word combination OR just bag of words and characters.

For ex ( 1-normal, 0-not normal):
"I wrote question.hope for answer" - 1(class)
"Bla bla goog goog goog gooo" - 0(class)
Maybe some dataset available.or some approach?
Thanks in advance!

What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer is incorrect according to the two other answers).
Applying the language model to a sentence gives you a probability (or a perplexity score,  which works the opposite way), so you have to define a threshold in order to classify as real language or not.


User identification using machine learning I am quite new to machine learning, and the task I was given was to find attributes that will allow me to identify users.

I have a set of data for training that contains roughly 48,000 instances, where every instance is labeled with user id.
The data are TCP sessions with data on destination, source, ttl, start time, endtime, destination IP addresses, and more.

My task is to find attributes so that in given a difference TCP flows in the same format form the same group of users, I will be able to labeled them with the appropriate user Id.

There are 10 users in the data set.

My question is what the best way to do it, I tried using weka to run number of machine learning algorithm, but I couldn't configure them right, for example I tried to use K-means but the results didn't fit with the labeled user id.

From your question I'm interpreting that you've extracted data from a log which has TCP session related data, some of the fields you've mentioned are: source/dest IP address, TTL, start time, end time.
You're trying to segregate this network traffic to isolate the traffic belonging to 10 unique client machines.

There are several ways to approach this client traffic segregation problem. But first you need to identify which variables are going to be most meaningful under different circumstances, since the network setup is not described, I'll list the different possible scenarios:

Clients reaching the server without intervening firewall/proxy: In this case each client will have a unique source ip-address which should help segregate the traffic without further analysis.
Clients behind a proxy which filters traffic and changes the ip-address to set a unique ip-address of the gateway for all the traffic coming to the server. In this case, you could use a combination of the following:

a) Segregate clients by TCP/IP packet attributes: Attributes such as TTL, SYN packet size, TCP window size, etc, can identify client OS. For example - Linux and Mac-OSX have default starting TTL values of 64, whereas windows has a default of 128, others like Cisco/Solaris 2.x have 255. So you can roughly segregate the traffic by OS type.

b) Since the TTL is decremented by 1 for each router placed between the client and the server, so even if you have 2 windows machines, but placed in a network segment separated by one more router, then the TTL number will be one lesser than the other windows clients.

c) Further identify unique sessions by the sequence number and acknowledgement number in the TCP headers. Each acknowledgement number of the destination server can be correlated by the seq number of subsequent packets from the client for that session. This will let you segregate users connecting to the same server at the same time e.g. 10 different sessions to stackexchange.com.

d) In case you can get further web-server log details, then attributes such as user-agent strings and HTTP cookies can help you identify a client.

e) In some cases, the random source port no. used by clients for the TCP session might overlap for different clients accessing different dest. ip-addresses at the same time, you can then safely assume these are traffic form two different clients.

This still leaves you with the problem of segregating traffic for different servers at different points of time. For this you could analyze the data in the following manner:

Identify a client using their ISN: There is a certain logic in generating random initial sequence numbers by each OS, and if you have a large sample of ISN nos available, then you could identify the attributes of the random number generator to pinpoint the client. This ISN generation process has been exploited for DoS spoofing attacks, and many OS's have strengthened it since then, but this may still provide some clues to identify machines uniquely.
Analyse SSL handshakes to identify clients via their attributes such as supported cipher suites information exchanged during the SSL setup handshake, etc. See this paper for details about this technique.
The source port nos. are usually assigned in increasing order, and it is unlikely that every machine started at the same time and had the same TCP traffic volume; this helps identify clients uniquely.
TCP/IP timestamps: Most OS's will increment the timestamps in TCP headers and this keeps increasing as long as the system is up. Since systems may be switched on at different times, it helps segregate clients even if they have exactly the same OS and browser versions. See this RFC for mode details.
Some proxies my use extra HTTP headers (Via, X-Forwarded-For) to indicate the exact client hostname or address requesting the data. This provides a  straightforward way to identify the client, but only if you can access the HTTP header information from the packet.

After segregating traffic using all these means, then you may use network traffic time-series analysis to identify distinct users from the periodicity in their usage. For example, the repetitive pattern of a person who logs into work everyday at 8:15 AM and checks on some specific websites for 5 minutes can be identified and that client uniquely segregated from the rest.

More hypothesis and directions to take can only be identified after exploring the dataset in detail.

Once you're satisfied with this preparatory work, you could then calculate and add all these features along with the other direct indicators to train a multinomial classifier for predicting these 10 user ids. You could use a random forest, LogitBoost or Neural Networks as starters. If a variable is not identified as important for the classification, then discard it to reduce the total number of features.

For new traffic, you would need to generate all these features and then use the final trained model to classify the traffic by user.


Should I create a separate column for each Id value in a feature column or can I use the feature column as it is? I am working on developing a model for predicting, revenue that a movie will make. One of the features in the training set contains id of the series that a movie belongs to.Say, Star Wars series has Id 1, then the corresponding value in that column for "Return Of Jedi" movie will be 1.

Should I create a separate column for each of the series and add value 1 if the film belongs to that series, or will just keeping the existing column containing ids will be fine?

You should not leave the column as it is since similar id values are not semantically related. Maybe a deep neural network could handle this, but linear models definitely do not.

There are two options, depending on the cardinality (the number of unique entries/categories) of the column.

If the cardinality is low you should one-hot-encode the series, one column for each series, like you propose. However, in your dataset, there are probably many series, and your data will end up having too many features. See this image (source), you need one column/feature for each category if you one-hot encode.

If the cardinality is high, you should find an encoding for the id column. One way would be using target statistics: Calculate the mean target (in your case revenue) a movie of a series makes for each series-category of the training set. Then replace the id column with a column of the target-means for the respective series. If there is a series in the test data, which is not in the training data, just use the overall target mean of the training set.
It is important for this approach that you never use target-statistics calculated from the test set, otherwise, your error estimate will be highly optimistic!
The rationale behind this approach is, that the model is allowed to learn things like "continuations of successful movies are often successful".


What does the Orange widget "Image Embedding" do exactly? I would need to know what this widget does. What architecture does it use? What dataset has it been trained on? And so on.

The source reveals it's likely Google's Inception V3 model.


How to Find the Average of the Input Vectors I want to find the average of input vectors. I tried to use tf.math.reduce_mean, but it went error.

If I have to use keras.layers.Average, I have to make a list of the hidden states.

Does anyone know how to solve this?

Using tf.math.reduce_mean

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = tf.math.reduce_mean(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)

Error:

ValueError: Input 0 is incompatible with layer Output: expected min_ndim=2, found ndim=0

Using Average layer

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = Average()(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)

Error:

ValueError: A merge layer should be called on a list of inputs.

Does this work for you? After the Bi-Directional LSTM layer, I sum the tensor over the batch axis and then divide by the batch size.

from tensorflow.keras import layers, Sequential
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
from tensorflow.keras.utils import plot_model

inputs = layers.Input(batch_shape=(4, 500), name="Input")
embed = layers.Embedding(input_dim=500, output_dim=30, name="Embedding")(inputs)
encoder = layers.Bidirectional(
                  layers.LSTM(units=20,return_sequences=True,dropout=0.3,name="LSTM")
                  ,name="Bi-LSTM")

hidden_states = encoder(embed)
total = layers.Lambda(lambda x: K.sum(x, axis=0)/K.cast(K.shape(x)[0],"float32"))(hidden_states)

model = Model(inputs, total)
model.summary()


Are Hadoop and Python SciPy used for the same? Right now

I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.

Specifically, I've read that:

Hadoop: allows you to perform any task you want, if it fits the Map-Reduce paradigm. It can add concurrence using a cluster, etc.

SciPy: python based ecosystem for (in fact, a lot of things)...

So my question is

If I've data collected from an environment, and want to correlate the data, calculate means, extract conclusions, etc...

Is SciPy as suitable as Hadoop (or more) to do so?

I'm basically trying to choose between them in order to build a little example, so I wonder which one fits better, or may I be misunderstanding the uses of each one.

I think you're quite confused.

Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster.  You can access file stores as you would in a local file store (with some modification) and modify things via Java API.  Furthermore, ON TOP OF the file system there exist a MapReduce engine that allows for distributive workflow.

Python on the other hand is a generic programming language that can be made to do a myriad of task such as build a web applciation, to generating reports and even peforming analytics.

SciPy is a package that can be used in conjunction with Python (and often numpy) as a way to perform common scientific task.

Truthfully, they focus on different paradigms.  If you have LARGE DATA (ie terabytes worth of it), it might be worth wild to setup a hadoop cluster (ie multiple servers and racks) and use Java MapReduce, Hive, Pig or Spark (of which there is a python version) to do analytics.

If your data is small or you only have one computer, then it probably makes sense to just use python instead of adding the overhead of setting up hadoop.

Edit:  Made correction via comment.


MLPClassifier threshold factor to eliminate test samples that are not in match with train data I am using MLPClassifer example from
scikit-learn

The code for training:

from sklearn.neural_network import MLPClassifier
X = [[0., 0.], [1., 1.]]
y = [0, 1]
clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(5, 2), random_state=1)

clf.fit(X, y)

At the predict step, we use test data [2., 2.], [-1., -2.] in
clf.predict([[2., 2.], [-1., -2.]]). The output of this function is
array([1, 0])

As we observe, the test data [2.,2.] is not in the train dataset we passed. Still, we got the closest match as label 1.

What i am trying to find is if the test data i supplied is not in the train dataset, i should print a message to user that data is not valid instead of telling him the wrong label as 1.

For instance, in knn classification, i have kneighbours function which tells the distance of my closest neighbours to the test data i supplied in a 0 to 1 scale. So, i could easily eliminate the test data samples which are highly distant from my train data samples by keeping threshold at 0.6 or 0.7.

Is there any criteria/threshold like this i could do with MLPClassifier or with any one of Incremental Classifiers mentioned here which can restrict my test samples if not present in train dataset ?

Question migrated from SO

SGDClassifier has desicion_function which tells the distance to the hyperplane, where the values are compared to.

This value could imply too big and too low values.


Extracting text into columns I have a set of customer reports, each is in ms word file. they are all in a similar pattern, for example they start with Name: --, Age: --, Date: --, etc...

is there a way to extract particular strings from each file to form a data set.

In orange, I was able to compile the word documents into corpus which I can display as one column (each report is in one cell). Does orange have a way to extract strings into columns (for example if between "age:" and "gender")?

Maybe you could use Orange3-Text add-on, widget Preprocess Text, Tokenization > Regexp. The source code indicates it's a Python regex, so you might be able to use a regular expression pattern such as:

(?ix)        # ignore case, ignore comments and whitespace in this RE
(?<=age:\s)  # preceded by 'age: '
.+           # characters you wish to match
(?=gender:)  # followed by 'gender:'


Comparing training and validation data set Root MSE for a best subset regression? I’ve a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables).
Here is the code that I used,

library(MASS)
set.seed(1)
train=sample(seq(678),452,replace=FALSE)
train

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

val.errors=rep(NA,14)

x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])
for(i in 1:14){
  coefi=coef(regfit.exh,id=i)
  pred=x.test[,names(coefi)]%*%coefi
  val.errors[i]=mean((Mydata$HPV[-train]-pred)^2)
}

plot(sqrt(val.errors),ylab="Root MSE",ylim=c(3,12), pch=11, type="b")
points(sqrt(regfit.exh$rss[-1]/452),col="blue",pch=11,type="b")
legend("topright",legend=c("Training","Validation"),col=c("blue","black"),pch=11)

How come the validation root MSE could always beat the training?. Any feedback would be appreciated.

Im a little late, but better late than never.  It looks like your line where you find the coefficients:

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

should have data=Mydata[train,] instead of data=Mydata.

Your model had already seen your test samples, so the validation error is not an accurate assessment.


What are the key differences between FPGA and GPUs for Deep Learning? I'm trying to investigate the ways in which FPGAs differ to GPUs for the purpose of deep learning. I understand this is a complex question and not necessarily easy to answer in one go, however what I'm looking for are the key differences between the two technologies for this domain.

Ideally, it would be nice to see the benefits and cons of both technologies with regards to Deep Learning.

In addition, from my understanding, and please correct me if I'm wrong, is that with FPGAs, those undertaking a DL project would need someone who is able to configure the FPGA according to the type of project they want through the use of languages such as Verilog or VDHL. Furthermore, if they want to change the type of DL project they want to do, they have to reconfigure the FPGA to follow suit. Have I got this correct, or is something like Intel's Deep Learning Inference Accelerator (https://www-ssl.intel.com/content/www/us/en/servers/accelerators/deep-learning-inference-accelerator-product-detail.html) a lot more similar to GPUs than I thought in terms of configuration?

I'll address the example you linked to first.

The link in your answer redirects, it seems that an equivalent product from the same company is Myrtle. The only product on that webpage is Nervana. All of Intel's AI products (including their Xenon processors, containing no AI acceleration) are supported by their libraries. That permits running your project on a range (most, but not all) of their hardware.

Attempting to use Wayback or search for the old webpage turns up nothing useful.

  Myrtle has significantly accelerated the time-to-market for FPGA designs by automating the translation of AI network designs using compiler technology targeting the Intel® Arria® 10 FPGA. Trained, high level algorithms, specified in a wide range of network descriptor formats - Torch, Caffe etc. are compiled and mathematically optimised, directly to VHDL to produce efficient, low latency designs. In cases where proprietary non-standard, deep learning layers are used these first need to be implemented.

  Myrtle.ai sets new deep learning standard; trains ResNet network 4.5x faster, using 8x fewer GPUs and at a quarter of the cost than previous benchmark winner.

Currently it's being implemented on the Intel Arria 10 GX FPGA:

  The Intel Nervana NNP is a purpose built architecture, an application specific integrated circuit (ASIC) that is custom-designed and optimized for deep learning.

  There are two different versions:

  The Intel® Nervana™ Neural Network Processor-T 1000 put a large amount of HBM memory and local SRAM much closer to where compute actually happens. This means more of the model parameters can be stored on-die to save significant power for an increase in performance. The Intel® Nervana™ Neural Network Processor features high-speed on- and off-chip interconnects enabling multiple processors to connect card to card and chassis to chassis, acting almost as one efficient chip and scale to accommodate larger models for deeper insights.
  The Intel® Nervana™ Neural Network Processor-I 1000 is a discrete accelerator designed specifically for the growing complexity and scale of inference applications. It is built on Intel’s 10nm process technology with Ice Lake cores to support general operations as well as neural network acceleration.

This is the Nervana:

The FPGA has optimized libraries available that compile direct to VHDL, if you want your own libraries you'll be writing your own code. It is somewhat analogous to wanting a custom BIOS for a GPU, you can wait for the manufacturer, look for a third party offering, or write your own - not simple, not extremely difficult. The FPGA is highly configurable, though that one isn't particularly powerful.

The ASIC also has optimized libraries, the same ones (with a compiler switch) so there's no difficulty switching from one to another; or running your code slower on a GPU or CPU. The ASIC isn't configurable, after years of development on the FPGA they decided to implement the results in dedicated hardware; gone is the reconfiguration but there's an increase in speed and a decrease in price (and TDP).

  ... would need someone who is able to configure the FPGA according to the type of project they want through the use of languages such as Verilog or VDHL.

You could, much as you could write your own BIOS for your GPU, like some cryptominers do. It's not something you have to do, but there are easy tools and brain surgery tools. The manufacturer offers downloadable updates from time to time.

  Furthermore, if they want to change the type of DL project they want to do, they have to reconfigure the FPGA to follow suit.

Maybe, probably not. Just change the program and recompile is probably good enough, unless you want to squeeze another few percent more performance out of it; and you know more than the manufacturer about the device and optimized AI architecture.

Now to your main question, which seems to be the first thing that you asked:

  I'm trying to investigate the ways in which FPGAs differ to GPUs for the purpose of deep learning. I understand this is a complex question and ...

First, a similarity: They are a means to solve AI problems, they have a certain amount of potential (both capability and processing speed); better ones cost more but because GPUs are mass produced (comparatively, FPGAs are mass produced also) they will be cheaper and there will be more free tech support (Forums).

A cheap FPGA (or ASIC) won't be fast, and a fast one won't be cheap - if you think GPUs are expensive then that will be your cheaper route, unless you have a new algorithm, a breakthrough, then make your own ASIC and sell your own card for the greatest profit. Assuming that you're the average person an off the shelf solution is likely better priced.

The "biggest challenge" is Bayesian optimization, recurrent network and reinforcement learning or genetic algorithms for parameter tweaking; something that GPUs don't do well.

Intel has its Loihi neuromorphic research chip to represent the next direction in AI.

While training is done the conventional way the inference is ran on that tiny device, much faster than previous generation devices.

Neuromorphic computing is one of the newest directions for AI, an FPGA or GPU is a stone-age tool. Here's what BrainChip says:

  BrainChip is the leading provider of neuromorphic computing solutions, a type of artificial intelligence that is inspired by the biology of the human neuron - spiking neural networks. Our spiking neural network technology can learn autonomously, evolve and associate information just like the human brain. We have recently unveiled the Akida Neuromorphic System-on-Chip.

  The Akida™ NSoC is both low cost and low power, making it ideal for edge applications such as advanced driver assistance systems (ADAS), autonomous vehicles, drones, vision-guided robotics, surveillance and machine vision systems. Its scalability allows users to connect many Akida devices together to perform complex neural network training and inference for cybersecurity and financial technology (FinTech) applications.

Notice that it also comes full circle back to FPGA. If you could write VHDL and have considerable experience you could have developed that but the FPGA probably isn't an Arria 10. A GPU is the least expensive and quite a flexible solution for the average person, it's also not a one trick pony. Something a few years old, like the Intel Deep Learning Inference Accelerator shown below, costs U$10K.


Tensorflow oscillating Test and Train Accuracy? I have implemented a CNN with images as input and 101 classes as output.
I have applied mean subtraction and normalization to the input before giving it as input to the network.
I have also experimented with multiple optimizers (ADAM & SGD) and also changed the learning rate (0.1,0.01,0,001) and also batch sizes (19,32,64,128,200).
But with all of these cases I had observed a similar pattern.
I dont think the network is learning much and more importantly the accuracy oscillating between a few values.
I had initially thought it was because of the batch size but I was wrong.
I have built and trained the exact same network with same inputs with caffe.  And achieved a 55% accuracy(check for working of  network model).
My guess is the implementation went wrong somewhere but not sure where.
Mycode
Output
This particular output is with lr=0.001 batch_size=200 and ADAM optimizer
Iter 2000, Minibatch Loss= 12560.197266, Training Accuracy= 0.01000
Testing Accuracy: 0.025
Iter 4000, Minibatch Loss= 4587.021484, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 6000, Minibatch Loss= 2196.316406, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 8000, Minibatch Loss= 918.784119, Training Accuracy= 0.01500
Testing Accuracy: 0.0
Iter 10000, Minibatch Loss= 315.327911, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 12000, Minibatch Loss= 102.744026, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 14000, Minibatch Loss= 59.417763, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 16000, Minibatch Loss= 33.026432, Training Accuracy= 0.00500
Testing Accuracy: 0.02
Iter 18000, Minibatch Loss= 30.361868, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 20000, Minibatch Loss= 33.944790, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 22000, Minibatch Loss= 15.029477, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 24000, Minibatch Loss= 13.751340, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 26000, Minibatch Loss= 9.085027, Training Accuracy= 0.01500
Testing Accuracy: 0.015
Iter 28000, Minibatch Loss= 14.407433, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 30000, Minibatch Loss= 11.090684, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 32000, Minibatch Loss= 9.385777, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 34000, Minibatch Loss= 12.902111, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 36000, Minibatch Loss= 5.580408, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 38000, Minibatch Loss= 11.641799, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 40000, Minibatch Loss= 6.941562, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 42000, Minibatch Loss= 6.781301, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 44000, Minibatch Loss= 6.840708, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 46000, Minibatch Loss= 5.501908, Training Accuracy= 0.01000
Testing Accuracy: 0.03
Iter 48000, Minibatch Loss= 4.667253, Training Accuracy= 0.03000
Testing Accuracy: 0.015
Iter 50000, Minibatch Loss= 4.613351, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 52000, Minibatch Loss= 7.234328, Training Accuracy= 0.02000
Testing Accuracy: 0.01
Iter 54000, Minibatch Loss= 8.120478, Training Accuracy= 0.02500
Testing Accuracy: 0.01
Iter 56000, Minibatch Loss= 6.804164, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 58000, Minibatch Loss= 5.060484, Training Accuracy= 0.03000
Testing Accuracy: 0.02
Iter 60000, Minibatch Loss= 4.937673, Training Accuracy= 0.02000
Testing Accuracy: 0.015
Iter 62000, Minibatch Loss= 8.758166, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 64000, Minibatch Loss= 4.616434, Training Accuracy= 0.01500
Testing Accuracy: 0.025

From your results I see that the network is working good: the loss function is reducing, that means that the calculated results are approaching the expected ones.
But in facts the accuracy is not, so there should be some error in the way you calculate it.

The fact that the loss function start to oscillate at some point (~after 20000 iterations) is expected, and that is the reason why usually one lower the learning rate after some epochs, or use some other technique of annealing.


Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization or normalization should be done after splitting the data into train and test subsets, but I am not still sure about the imputation process. For the classification task, I am planning to use Random Forest, Logistic Regression, and XGBOOST (which are not distance-based).

Could someone please explain which should come first? Split > imputation or imputation>split? In case that split>imputation is correct, should I follow imputation>standardization or standardization>imputation?

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak.

Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially seen data.

I don't think the order of scaling/imputing is as strict. I would impute first if the method might throw of the scaling/centering.

Your steps should be:

Splitting
Imputing
Scaling

Here are some related questions to support this:

Imputation before or after splitting into train and test?

Imputation of missing data before or after centering and scaling?


Choosing the right parameters to train a Tf-Idf vectoriser I'm very new to the DS world, so please bear with my ignorance.

I'm trying to analyse user comments in Spanish. I have a somewhat small dataset (in the 100k's -- is that small?), and when I run the algorithm in a, let's say, naïve way (scikit-learn's default options +remove accents and no vocabulary / stop words) I get very high values for very common and low value words (such as spanish equivalents of "to", "at", etc.).

What would be the most effective way to train the vectoriser on a 100k long corpus of ~200 char long docs?

I was thinking of using a larger corpus of Spanish text or looking for a stop word removing implementation for Spanish, but would love to have some expert advise before jumping into it.

Thanks!

There are no silver bullets. But here are some suggests:

Use a better stopwords vocabulary. If you still have words like "to" and "at", then you are either not removing stopwords or using a lousy vocabulary. Try using the Spanish stopwords from nltk:

from nltk.corpus import stopwords
stopwords.words('spanish')

Use max_df < 1. This will truncate words that appear in more than that percentage number of documents.
The TF-IDF part that punishes common words (transversal to all documents) is the IDF part of TF-IDF, which means inverse document transform. Several functions may be used as your IDF function. Usually, IDF=$\log\frac{\text{#documents}}{\text{#documents where word appears}}$ is used. You could try a more punitive IDF function. sklearn does not seem to allow to specify it, but you can use nltk or gensim or easily implement your own TF-IDF vectorization. It needs no more than five lines of code.

I would try each of these suggestions in this order, and stop when it is good enough. It sounds like using a better stopwords vocabulary will be good enough for you.

EDIT: I forgot to mention, but of course you can add more uninteresting words to the stopwords list, if you have the need to do so.

You may also want to first evaluate the impact TF-IDF is having in your counts. I think this will work:

m = TfidfVectorizer().fit(docs)
c = [m.transform([word]) for word in m.vocabulary_]
for i in np.argsort(c)[::-1][:20]:
    print(m.vocabulary_[i], c[i])

You may have to adjust the code. Try to use with IDF disabled and enabled, and other tweaks.


Dealing with historic data drift I'm trying to predict a continuous target in an industrial context.
The problem I'm facing is that the some of the predictors have changed over time, for example the pressure in the machine was increased. This influenced some of the other predictors, but hasn't influenced my target.
As an example (in R formula notation):
$ Y \sim U_1$ The target depends on some unobservable variable
$ X_j ~ \sim U_1 + X_i$   One of my observed variables depends on the unobservable varible and another observed variable. Therefore $X_j$ is helpful for predicting $Y$
Now $X_i$ has changed a couple of times. This clearly hasn't influenced my target. But I'm also not really able to learn the relation $Y \sim X_j$ because $X_j$ has changed with $X_i$.
I know some of these dependencies for fact from physics. But there's no way I can fix this by hand because there are about 1000 variables.
When reading about data drift, it's always about how to adjust your existing model to a sudden change, but for me the changes already happened in the past.
The time periods where nothing has changed are too small to just use the latest batch, but it seems like just using the whole dataset without any adjustments doesn't work either.
Can anyone give advice on how to address this?
(Right now I'm using XGBoost but I'm open to other models)

Based on your post (my emphasis):

The problem I'm facing is that the some of the predictors have changed over time, for example the pressure in the machine was increased. This influenced some of the other predictors, but hasn't influenced my target.

So to be clear, the input parameters changed but made no impact on the output parameter?
There are two possibilities:

The data is part of a single population, then the change in the input parameters (over that range) has little or no impact on the output parameter.  This can be addressed by creating a new feature, which is a delta from some baseline and see if that new feature is a better fit for your model.

Pressure
Piston Position
Leakage Flow (Out)

10
20
30

11
22
30

12
23
33

The inputs into row 2 have changed relative to row 1 but no change in output. Therefore a small change in parameter 1 and 2 has no impact on output (physical examples include say overcoming hysteresis or localised energy storage or data buffer or a bucket that has to be filled with water before spilling over).
In this case the output isn't sensitive to the small changes and perhaps they're a function of the original value. So new features could be:

Pressure
Piston Position
Leakage Flow (Out)
F1
F2

10
20
30
0
0

11
22
30
0
0

12
23
33
1
1

So F1 only increases if the difference between row 1 and the other rows is greater than 1 otherwise it's 0 etc.

The data is part of more than one population. As an example, say you're taking measurements of a product. Over time the measurement device loses calibration and you see a drift in the measurement. The only real fix is to determine the drift due to a loss of accuracy and apply a corrective factor as function of time or base value.


R: I need to understand the mechanism behind reading a subset of a file in R (e.g., via sqldf) or other data-science-centered programming language By reading only a subset of the records of a file into R (or Python), one can speed up performance tremendously. I am curious about the mechanism behind this kind of file input. To read only a small part of a file, my guess is that I have to somehow access the file, load it in memory and filter out that small part that I am interested in (Otherwise how else would I be able to query the subset of the records that I want?). In otherwards that should then mean I have anyway had to read the whole file into memory. So if it was a 20GB file, it means I have had to read 20GB so as to filter off the 1GB that I want. Where do the performance benefits come from then?

There are multiple aspects to performance when it comes to reading a portion of the file. I am going to discuss the two most common one based on functions that are available in the base R.

What is R doing at the time of reading?

Establish a connection with the file.
Open the file
Read the data sequentially
Load the data in memory
Close the connection
dump the object into a variable if assigned, else render the output in some gui.

While it is doing all these steps, it is also trying to:

Understand how big the data is so that it can allocate appropriate memory
What are the data types or classes so that it can (again) allocate appropriate memory.

Sub bullet 2 is important here because unless R reads the entire dataset, it cannot determine the datatype. Providing this information alone at the time of the read makes processing much faster.

By reading a subset of the data, R makes it easier to interpret the datatype and then assigns it to the subsequent sets without worrying about what the actual datatype is.

Does R really loads all the data every time?

Functions that allow filter at the time of read, really target the third step of reading data sequentially.

Each row is compared against the filter criteria and the result stored as a binary logical index. This index is then used to determine, what remains when the connection closes. I could not find specific reference in the documentation if the discard happens immediately or post entire read, but I will assume that the R collects the garbage automatically as it reads with a filter criteria and makes the memory available to a subsequent line of data.

May be there are other experts who can throw more light on the real inner workings here and either validate or correct my assumptions, but hope my context helps.


Replace missing values of numerical features with unique numbers Let's suppose that I have a dataset with 5 numerical features of which each of them has some missing values and all of them have only non negative values.

Some suggested ways to deal with missing data are:

Remove the rows which have even one missing value
Impute the missing values

I do not prefer (1) because then you miss some valuable information from the rest of the features for these rows.

I do not prefer (2) because in general (it depends on the application) it introduces quite a lot of noise in the data.

What I am thinking is to:
 3. Replace the missing values with a unique value (eg -1 or -999)

As I said, in my example the features have only non-negative numbers so values such as -1 or -999 will be only encountered by the algorithm for missing data.

What are your thoughts on (3)?

What are the advantages and disadvantages of this approach?

This depends on the exact learning algorithm which is used, but most of the time numerical values are treated... numerically.

It's easy to see with the case of linear regression: let's say a feature has values between 0 and 100, but with a few missing values. Now replace missing values with -999: the coefficient learned for this feature will be completely different and based on  meaningless indications, so the performance will decrease.

This idea might work in the case where the numerical values are 'binned' into several classes, i.e. transformed into ordinal/categorical variables.

But the best way to use such values is to use an algorithm and framework which can deal with missing values in the first place.


Embedding Values in word2vec Are the embedding values for a particular word using word2vec Skipgram model the weights of the first layer or the softmax output of the function?
Does the embedding value change according to the training corpus?

The word embeddings are the weights of the first layer i.e. the embedding layer and not the softmax output of the function. The embedding values represent a vector which gives the location of the word with respect to other words in a high dimensional vector space. And yes, the embedding values change according to the training corpus. However, if you are using a given language (for example English) and have a large amount of training data the final values of the vectors will turn out to be pretty close even with training corpus of different contexts.


How to feed data to multi-output Keras model from a single TFRecords file I know how to feed data to a multi-output Keras model using numpy arrays for the training data. However, I have all my data in a single TFRecords file comprising several feature columns: an image, which is used as input to the Keras model, plus a sequence of outputs corresponding to different classification tasks: eg. an output encodes the age of the person in the image, another output encodes the gende, and so on.

From what I have seen in examples, when the output of the model is made of various heads, the model should be fed with multiple data sources, one for the input, and one for each of the ouputs.

Is there an easy way to do that when the data is all in a single TFRecords? I mean, without having to create separate TFRecords for the input and each of the ouputs?

After playing around with tf.data.map operations I found the answer was easier than expected, I simply had to preprocess the data and put all the labels for each output of the model as a different key of a dictionary.

First I create a dataset from the tfrecords file

dataset = tf.data.TFRecordDataset(tfrecords_file)

Next, I parse data from the file

feature = {'image/encoded': tf.io.FixedLenFeature((), tf.string),
           'image/shape': tf.io.FixedLenFeature((3), tf.int64),
           'age': tf.io.FixedLenFeature((), tf.int64),
           'gender': tf.io.FixedLenFeature((), tf.int64),
           'ethnicity': tf.io.FixedLenFeature((), tf.int64),
 }

return tf_util.parse_pb_message(protobuff_message, feature)

dataset = dataset.map(parser).map(process_example)

At this point, we have a standard dataset we can operate with doing batching, shuffling, augmentation or whatever we wan. Finally, before feeding the data into the model, we have to transform it to fit the requirements of the model. The code below shows an example of both input and label preprocessing. Previoulsy, I concatenated all the labels, now I create a dictionary witht the names of the outputs in the model as keys.

def preprocess_input_fn():
    def _preprocess_input(image,image_shape, age, gender, ethnicity):
        image = self.preprocess_image(image)
        labels = self.preprocess_labels(age, gender, ethnicity)
        return image, labels

    return _preprocess_input

def preprocess_image(image):
    image = tf.cast(image)
    image = tf.image.resize(image)
    image = (image / 127.5) - 1.0
    return image

def preprocess_labels(age,gender,ethnicity):
    gender = tf.one_hot(gender, 2)
    ethnicity = tf.one_hot(ethnicity, self.ethnic_groups)
    age = tf.one_hot(age, self.age_groups)
    return {'Gender': gender, 'Ethnicity': ethnicity, 'Age': age}

In my model, Gender, Ethnicity and Age are the names of the last layers of the model, so my model is defined as having three outputs:

model = Model(inputs=inputs,
              outputs=[gender, ethnic_group, age_group])

Now I can use a dataset to fit the model by applying the preprocessing function first:

data = dataset.map(preprocess_input_fn())

model.fit(data, epochs=...)


What kinds of data other than geographical are topologically spherical? I'm trying to think of a data set that is essentially topologically spherical. It's easier to think of cylindrical datasets (two dimensions, one periodic) or toroidal datasets (two dimensions, both periodic).

Obvious candidates are geographical and astronomical, ground and sky; but I think the only thing spherical about the sky is its projection onto the ground, so it really just comes back to Earth.

I find it helpful to think about in terms of a circle's fundamental polygon:

Any data consisting of a three element vector of numbers  $v =\{x,y,z\}$ could be viewed as topologically spherical if, after converting to polar coordinates, there is only one datum for every value of the 3D angle $\theta$. Thus, every point in the data represents an exterior of a topological sphere. This is taking the term "topological" literally and formally.

There are also graph structures (acyclic) that have neighborhood structures that allows the graph to be mapped topologically to a sphere. For example, all graphs that can be mapped to Platonic Solids (i.e. vertexes to vertexes, edges to edges, 1:1) are topological spheres. This can include irregular neighborhood structures, too.

As far as practical examples of these, none come to mind immediately.


When should train and test data be merged? I am very new to data science and machine learning. I am following courses on datacamp, and then trying to solve problems on kaggle/drivendata.

Very often, I try to use the sklearn.model_selection train_test_split()-method, but because my training (X) and test (y) data is not same shape, I get the error:

ValueError: Found input variables with inconsistent numbers of samples: [913000, 45000]

When I look at other people's solution, it looks like they very often combine training and test data (datasets in this case: train & test), like this:

all_data = train.append(test, sort = False)

Then later, they split this up again into variables X and y, which they can use on the train_test_split method. (You can probably hear I am very new and don't quite understand what's happening yet).

Can anyone explain me, why I need to combine my training and test data, when I already have them in separate .csv files? Maybe share some resources that explain it? Thanks a lot

The answer to this is simple.

When we perform the cleaning of the dataset we'll need to do the whole cleaning process for training data first then we'll do the same data cleaning process for the test dataset too. So to avoid doing the same data cleaning process twice, we merge the training and testing data then we perform the data cleaning process and after that we separate both dataset.


Feature importance with high-cardinality categorical features for regression (numerical depdendent variable) I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many levels (on the order of 100-1000). Given that one-hot encoding creates a dummy variable for each level, the feature importances are for each level and not each feature (column). What is a good way to aggregate these feature importances?

I thought about summing or getting the average importance for all levels of a feature (probably the former will be biased towards those features with more levels). Are there any references on this issue?

What else can one do to decrease the number of features? I am aware of group lasso, could not find anything easy to use for scikit-learn.

It depends on how you're one-hot encoding them. Many automated solutions for that will name all the converted booleans with a pattern so that a categorical variable called "letter" with values A-Z would end up like:

letter_A, letter_B, letter_C, letter_D,....

If after you've figured out feature importance you've got an array of feature and the associated weight/importance, I would analyze the array and perhaps sum up the feature importance weights for anything starting with "letter%".


Understanding input of LSTM I am a little confused with the input of LSTM. Basicaly my train input data is of shape (53394, 3). I reshaped my 2D data into 3D data in order to set it according to the input of LSTM. I have two configurations:

trainX = (53394, 3, 1)
model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))

trainX = (53394,1, 3)
model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))

I want to understand how input is feed into the LSTM in both the cases. like one neuron is taking one column value or one complete row with three columns is going as input to one neuron in input layer.

LSTM layers work on 3D data with the following structure (nb_sequence, nb_timestep, nb_feature).

nb_sequence corresponds to the total number of sequences in your dataset (or to the batch size if you are using mini-batch learning).
nb_timestep corresponds to the size of your sequences.
nb_feature corresponds to number of features describing each of your timesteps.

Thus an LSTM layer will work like this :
Let $t_i$ be the $i^{th}$ timestep of sequence $seq_j$, with $i \in [0, nb\_timestep], \ j \in [0, nb\_sequence]$.
An LSTM layer will make a prediction $p_i$ according to the $nb\_feature$ descriptors of $t_i$ with respect to its hidden state which is a representation of the timesteps $t_0$ to $t_{i-1}$.

Now, let's see what this means for your two configurations. For the sake of the explanation, I will suppose that we have sequences of words

For (53394, 3, 1) the LSTM will work on 53394 different sequences. Each sequence is 3 words long and each word is described through one feature only. For the first word of each sequence, the LSTM will make a prediction on its sole descriptor. For the second word, the prediction will be done from the unique descriptor with respect to what the first word was. Finally, for the third and last word in the sequence, the LSTM will emit a prediction from the unique descriptor with respect to what the two previous words were. Then, the LSTM begins the process anew for the following sequence.
For (53394, 1, 3), your sequences contain only one word which is described through 3 features. Sequences of one word are not really sequences, so the LSTM layer will not be useful in this case.

Hope it clears up how data are fed to an LSTM !

NB. Not related to the question but it may help : from your original shape, it seems your dataset contains 53394 words described with 3 features. If I am right, you would need a 3D shape like (53394, nb_timestep, 3) but with $nb\_timestep \neq 1 $. What you need then, is to define some window instead of reshaping your data.


How discriminator knows if the image is real or fake at the initial phase? If the discriminator and generator in GAN learn together, How the discriminator knows whether the image is real or fake in the initial phase of training? Does the discriminator need to get trained earlier?

This a good question. I'll post my own personal considerations that should not be taken as rigorous answers, I'd like to broaden the question a little bit
[Disclaimer: personal considerations, not rigorous answer]
The generator and discriminator are not strictly learning together, they are learning one against other. Indeed, when the discriminator is training, the generator is frozen and vice versa.
At the very beginning of the training phase, the generated outputs of the generator are expected to be very far away from the real samples. It is also true that the discriminator is expected to not have a good feature space for the classification, but I think it gets to tell them apart faster than the generator begins to produce closer fake samples. Then, the minmax game evolves...
I thougth once to use a pre-trained classifier to force the generator produce real good fake images, but I also got to the feeling that, in that way, the battle will not be fair and will be biased to the discriminator side, regarding this:

Which input data should should be fed to the discriminator in its pre-training phase? I mean, to pre-train the discriminator, some pre-fake images are needed. One approach is to pre-train for a few steps the generator and produce pre-fake images which are far away from real images

Since the discriminator is still learning, are they chances it becomes even better?

The discriminator is so good it won't learn, is the game closer to nash equlibirum and so, closer to its end?
Taking into account the enormous problems when training GANs, I've never tried it out.

It would be fantastic if more experienced GANners offer more insight into these considerations


Multivariate prediction model using R Imagine having a large database coming from a super market chain with panel data, whose cross-section variables are the different products sold in each super-market. The frequency is daily, that is you have one value for each product sold in each super-market per day.
Beyond their own history, you have the time-series of other variables e.g. dummy variables that signal the existence of offers.

Your objective is to predict the number of goods sold for each product in each super-market.

Which functions would you use in R to make a week-ahead multivariate forecast?
Which approach would you suggest?

You could try also  R  h2o

You  have implementations  for   random forest,   gbm , linear models and also   deep  learning .
THe  advantage is  that    h2o is pretty   fast  compared  to   other  implementations


Can dropout and batch normalization be applied to convolution layers Can dropout be applied to convolution layers or just dense layers. If so, should it be used after pooling or before pooling and after applying activation?
Also I want to know whether batch normalization can be used in convolution layers or not.

I've seen here but I couldn't find valuable answers because of lacking reference.

In short, yes.

Batch Normalization Batch Normalization layer can be used in between two convolution layers, or between two dense layers, or even between a convolution and a dense layer. The important question is Does it help? Well, it is recommended to use BN layer as it shows improvement generally but the amount of improvement you will get is more problem dependent.
Dropout: Convolution layers, in general, are not prone to overfitting but it doesn't mean that you shouldn't use dropout. You can, but again this is problem dependent. For example, I was trying to build a network where I used Dropout in between conv blocks and my model got better with it. It is better if you apply dropout after pooling layer.


What supervised machine learning model can be used to generate a scorecard-like result? A scorecard is typically used in Credit Application. One very common model for developing a credit scorecard is logistic regression since it has well-defined probabilities.

Apart from logistic regression, is there any model that can be used in the scorecard?

For example, I don't know whether Support Vector Machine can be used since it only outputs a decision boundary.

More on the scorecard:

Features are assigned with weightings
All features are categorical
The sum of weightings of all features with value True is the total score (like a checklist)
There will be a cutoff point to classify good/bad (label, +1,-1)
How far from the cutoff point represents probabilities.

It depends what you mean by "can be used": any regression algorithm can be used, the question is how reliably it would perform. You can compare different algorithms experimentally (if you have a dataset).

[Updated after question edited]

In general the way to use ML with this kind of setting is to train a classification model based only on the categorical features. Depending on the type of algorithm, the combination of features might not always be a weighted sum, and the result label may or may not be based on a cutoff point. In order to have a cutoff point (thus a numerical prediction), the method must be a soft classification method. Alternatively a regression model could be trained for predicting the numerical value.

So that leaves you with many options:

soft classification: linear/logistic regression, Naive Bayes, ...
regression: linear/logistic regression, SVM, decision trees, ...

Note: technically the probability doesn't represent "how far from the cutoff point", it represents the probability of the instance being positive (p=1).