Clustering with groups in data related to cluster label I want to predict which device got used in which room. Therefore I've got device and sensor data.

My idea was to create a feature vector lie this:

             ----------------------------------------------------------
Data-Vector: | u_1 u_2 u_3 | x_1 ... x_7 | y_1 ... y_12 | z_1 ... z_4 |
             ----------------------------------------------------------
 Categories: | device_data | room 1 data | room 2 data  | room 3 data |
             ----------------------------------------------------------

My device data contains amongst other things:
+ timestamps when the device got turned on/off
+ average power consumption and divergences

My room data contains for example:
+ sensor data of motion detector and timestamps
+ sensor data of lamps (turned on/off) and timestamps
+ weather data

In the feature vector I've got the room data closest to the turn on/off timestamp.

All data points itself are floats.

My idea was to use k-means for clustering.
My problems are:
1. When using k-means, how can I tell which cluster correlates to which room label (room1, room2 or room3)?
2. I think it could be beneficial if I add somehow the information: which sensor is which room.

Can I manipulate the data in k-means algorithm so it will only consider:
the device data and room 1 data for the first cluster and sets everything else to zero
the device data and room 2 data for the second cluster
and so on...

This way I could tell that cluster x correlates to room x.

Or will this somehow break the k-means algorithm?

If you have apriori known labels, then use the labels, not clustering.

Clustering is quite difficulty and fragile, so always prefer a simpler solution if you can.

It's not very clear what you are trying to solve. Select columns to split the data into rooms? Just select the columns then.


Multi target classification for different types of target variables I am new to machine learning and I got this task in my university. I have a dataset with over 100 columns and two target variables: $target1$ is categorical i.e. $0$ or $1$ and $target2$ is continuous i.e. values in range $0 \space to \space 100$.

How can I predict this type of problem?

I tried using multi-output classification from sklearn using the Random forest as an ensembler and it is predicting nicely for continuous target variable but not for categorical target variable.

You have one classification task and one regression task, but sklearn's multioutput meta-estimators only support two tasks of the same type.

The best solution here is to train two models:

A binary classifier to predict $target1$
A regressor to predict $target2$

For example:

from sklearn.cross_validation import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

# X, y = load training data

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = RandomForestClassifier()
clf.fit(X_train, y_train[:,0])
print("classifier accuracy:", clf.score(X_test, y_test[:,0]))

regr = RandomForestRegressor()
regr.fit(X_train, y_train[:,1)
print("regressor R^2:", regr.score(X_test, y_test[:,1]))


data science python data cleaning I am preparing a dataset for a model, but somehow the code just doesn't run well.

The major error is:

File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
  'income2' = pd.to_numeric(Adult.income, errors='coerce')
                                                        ^
SyntaxError: can't assign to literal

Code:

# import statement
import numpy as np
import pandas as pd
import csv
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
# loading dataset, it is a combination of categorical and numerical data
hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range']
# dataset basics
hp.head()
hp.shape
hp.dtypes

# account for all value '?'
hp.replace('?','na')
hp.isnull().sum()

#Remove onsolete data point in income
hp('income').dropna()

# replace all aberrant values
hp.replace('nan', 0)
hp.replace('NULL', 0)

# change data type of certain data point to numerical
number = LabelEncoder()
hp['income'] = number.fit_transform(hp['income'.astype('str')])
hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

# Choose the datapoint 'income' to perform the data cleaning and remove outliers
LimitHi=np.mean('income') + 2*np.std('income')
LimitLo=np.mean('income') + 2*np.std('income')
BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

# Replace outliars
RightIncome = ~BadIncome
x[BadIncome] = np.mean(x[RightIncome])

# normalize the Income Column using numpy
#'income2' = pd.to_numeric(Adult.income, errors='coerce')
minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

# bin age data into several ranges
hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

# construct new categorical data point with existing data point
hp['EvalonInvestment'] = 'zzz'
hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

print(hp)

It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string


Typing error handling n-gram character index and vector space model Suppose I apply tri-gram indexing for my document collection, and is implementing a vector-space model to help retrieving the document. In the text it is mentioned implementing a trigram will introduce a new step in filtering the result. However, what are the problems that I need to be aware of if I implement tfidf/vector-space model? The reason I am exploring this option is to try handling basic spelling error handling, does it really work in practice?

Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help.  You need some form of fuzzy matching for that.

For example the string, "I like dosg too" would fool a unigram model because "dosg" is likely "dogs" misspelled, and it will encode it as "dosg" : 1.  But you have the same problem in a trigram model.  It will encode "I like dosg" : 1, "like dosg too" : 1.  Which is not really better, as it will still not match any trigrams with the word "dogs" in it.


General question on the approach to optimise numbers I have a huge huge model in SQL that nobody knows what it is doing. This model spits out some numbers and those numbers should be optimised to match another batch of 'correct' numbers as much as possible. So I get one set of numbers form my black box model that change according to how different parameters change inside that model - there are around 400 parameters that can be changed to get different numbers out of it (no, I don't want to even touch it, less trying to understand how it works) and I have another 'correct' set of numbers that I get for those parameters from a person in charge of it. What would be the best approach to 'optimise' my black box numbers to match the correct ones? The black box numbers will relatively match the correct numbers depending on how parameters are set up but I need to make them more correct. I was thinking maybe Bayesian optimisation in Python or GridSearch but not sure if that is the best approach. Any ideas would be appreciated.

For this, you can treat output provided by "person in charge" as ground truth. I assume that you have historical records of :

Numbers generated by this person
Parameters that were fed into the model to generate numbers

treat these 400 parameters are train_x and numbers generated by the person as train_y.

Once you have this data in CSV, you can explore various Algo to find a model that can match train_y with minimal error.

Since no details are available on "how" either of these outputs are produced, you need to pick a tool that enables quick experimentation.

I will suggest Weka since this is a UI tool and does not require any coding. Once you identify an Algo, you can implement it in Python.

Some tutorials :

https://www.youtube.com/watch?v=6tDnNyNZDF0

https://www.youtube.com/watch?v=kFKxUQDnLV8

https://www.youtube.com/watch?v=COpXNK0O8As


Do categorical features always need to be encoded? I'm using Spark's Machine Learning Library, and features are categorical. The features are strings, and Spark's MLlib (like many other machine learning libraries) does not accept Strings as inputs.

The normal procedure for overcoming this is to convert Strings to integers, and then encode these integers (using a onehotencoder for example), because converting to integers implies that there is an ordering between features.

My question is - do categorical features always need to be encoded? In what situation could integers be used instead of encoding?

I'm using Logistic Regression and Naive Bayes.
When using integers as features I get an 84% accuracy, when these integers are encoded I get an 82% accuracy.

Is it necessary to encode?

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features").

I will just clarify the terminology a bit more.

Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat)
Ordinal data: information has categories with natural ordering defined between them (annual income scale defined in terms of categories \$40000,   \$40000 - $80000).

If the variable is of type ordinal, you can replace it with integers and proceed with the algorithm. If it is categorical, it should be converted as well as encoded.

Hope this helps.


Trying to predict extreme values corresponding to rare events I need some advice on methodology.

I need to predict a numeric value (claim amount) being as good as possible on high values corresponding to rare events (corporal damage, technological disaster...).

Any suggestion on methodology would be appreciated. Thanks.

My suggestion:

There is a technique called Quantile Regression, which allows you to determine the highest (or lowest) values of a variable. This is valid only if you have regressors (exogenous variables).


How to collect tweets by geo-location? I am extracting tweets on brands for sentiment analysis. I am using twitteR package on R. Is there a specific way by which I can only collect tweets from a location ex NY or Paris or Canada etc. I have tried Geocode but here I have to give lat and long data and the vicinity range.

To get location based tweets, you have to specify a location circle with center (lat and long) and radius using reverse_geocode.

There is no way to find tweets by setting a polygon or drawing a border.


Dimension problem in keras neural networks So every time I try to write a CNN or RNN I have problems understanding how the dimensions work.

I don't understand why does the model compile, which in my opinion means that the output dimension of the previous layer is the input dimension of the next one and so on(the layers are compatible with each other). So how is it possible that when I fit my model I get an error in the 3rd or 4th layer(or any other layer except the first one) saying that for ex it expects 3 dimensions but got 2.
On a more precise example, lets say I have the following one-shot model:

vocab_size =10000
src_txt_length =10
sum_txt_length = 100

inputs = Input(shape=(src_txt_length,))
encoder1 = Embedding(vocab_size, 128)(inputs)
encoder2 = LSTM(128)(encoder1)
encoder3 = RepeatVector(sum_txt_length)(encoder2)

decoder1 = LSTM(128, return_sequences=True)(encoder3)
outputs = TimeDistributed(Dense(vocab_size, activation='softmax'))(decoder1)
model = Model(inputs=inputs, outputs=outputs)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(X,y)

I get an error in the TimeDistributed layer saying that it expects 3 dimensions but got 2.

X.shape == (20000,10)
len(y) == 20000  # y is a list of labels 0s and 1s

So if I want to calculate the dimension of each layer when I fit my data on paper, how will I do that ?

Ok I was able to recreate the error. The TimeDistributed layer applies the dense layer to each of the time slices. In this model you will have 100 different time slices as defined by sum_txt_length and the RepeatVector layer. These time slices will be propagated through the network all the way to the TimeDistributed layer. Thus, your $y$ vector must have dimensions $(None, 100, 10000)$.

You can use model.summary() to get the expected dimensions after each layer.

You will need to fix the output layer to match the label dimensions. I believe you will need to add a Flatten() layer before the densely connected layer, and do not use the TimeDistributed layer at the output. That way your output will be of size $vocab_size$, thus predicting the next vocabulary word given an input.


Do 3D bar charts have advantages over 2D bar charts? I vaguely remember that there was a study / blog post which made a strong point against 3D bar charts. Do you have a source at hand which compares the two - 2D bar charts and 3D bar charts?

A relevant source seems to be

  A Psychophysical analysis of chart readability, 2001

Sadly, I don't have access to it.

Anecdotes

If you have an area chart, giving it a 3D-effect and rotating slightly can help readability. But I don't remember a context where you could not have used a line chart in the first place.

Bad sources

PRO: Looks good (targetdashboard.com)
CON: Values are harder to read from the axes only - you need to add values to
the bars themselves (targetdashboard.com)


Why isn't dimension sampling used with gradient boosting machines (GBM)? GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used with dimension sampling at every split of the tree like is common practice with random forests.

Are there some tests that show that dimensional sampling with GBM would decrease its accuracy because of which this is avoided, either in literature form or in practical experience?

sklearn's GradientBoostingClassifier / GradientBoostingRegressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split.


How can I create a classifier using the feature map of a CNN? I intend to make a classifier using the feature map obtained from a CNN. Can someone suggest how I can do this?

Would it work if I first train the CNN using +ve and -ve samples (and hence obtain the weights), and then every time I need to classify an image, I apply the conv and pooling layers to obtain the feature map? The problem I find in this, is that the image I want to classify, may not have a similar feature map, and hence I wouldn't be able to find the distance correctly. As the order of the features may by different in the layer.

You can use restricted boltman machine to obtain feature map, and then build a CNN with this architecture : conv layer -> pooling->conv layer pooling-> ....->fully connected layer.   Add conv layer and pooling layer until the matrix size reduced into $1\times 1$, then treat them as input for your fully connected layer, then train the fully connected layer


Predict when a customer would buy from previous buying history (Mining Buying patterns) I'm trying to predict when a customer would buy a product(which month of the year) based on the historical transactions. I have 3 years transactions data and good amount of customers are frequent ones.
Or basically want to mine the historical transaction data, and see for a set of customers (in a Geo/type of customer etc.) which months are likely period of buying a product X.

Can someone please advice which Machine learning technique or statistical approach would be best in this case.

I am trying
- Decision tree classification/Logistic regression model taking month as predictor variable.
- Statistical analysis to see which month is more significant for a set of customers.

I would try using xgboost classifier. In my case, xgboost outperformed Random Forest.

import xgboost as xgb
xgb_model = xgb.XGBClassifier(
    learning_rate=0.1, n_estimators=1000, max_depth=3, min_child_weight=1,
    gamma=0, subsample=0.8, colsample_bytree=0.8, objective='binary:logistic',
    nthread=4, scale_pos_weight=1, seed=27)

You can use the below snippet to avoid over fitting problem.

xgb_model = xgb_model.fit(X_train, y_train.astype(int), early_stopping_rounds=50,
                          eval_set=[(X_test, y_test['target'])])


Calling function on a list object I have a list (sub_indices) containing dataframe index values. Using these index values, I need to make changes in another dataframe (main_df). Changes must be done on these index values that are in the list.

What should be the best way to solve this task? I'v tried lambda and map functions but am getting an error: "AttributeError: ("'list' object has no attribute 'apply'", 'occurred at index 0')".

Seems that lambda function cannot be called on a list.

def function_1(new_value):
    sub_indices = sub_df.index.values.tolist()
    sub_indices.apply(lambda x: function_2(x))

def function_2(df_indices):
    # Replacement that needs to be done. 'indes_value' is the next element from 'sub_indices' list.
    main_df.at[indes_value, 'column_name'] = "new value"

You can simply use df.loc to achieve what you want. Assuming indice is the list of indices where you want to make the changes, do:

main_df.loc[indice, 'column_name'] = new_value


ML on dataset with heavy dependency on prefix I have a target function, which heavily depends on vector prefix:

(1, 2, 0, 0, 1, 2, 4) -> A
(1, 2, 0, 55, 1, 99, 1) -> A
(1, 2, 124, 55, 1, 99, 71) -> A
(1, 3, -5, 0, 1, 2, 4) -> B
(1, 3, 12, 55, 1, 99, 1) -> B
(1, 3, 124, 55, 1, 99, 71) -> C
(1, 3, 124, 21, 10, 195, 11) -> C
...

Prefix length may vary.
pre-condition: a training set of vectors and their target values
target: to be able to evaluate target value on arbitrary vector
I have tried HMM, but it doesn't seem to work well in this case. What algorithm should I use instead?

Consider LSTMs. They're a type of recurrent neural network that are generally very good at sequence prediction tasks and have the ability to learn very long range dependencies like you describe.

Andrej Karpathy gives a tremendously great intro to RNN/LSTM on his blog.

Although going as far as a neural network opens up a whole new range of problems, such as long training time and finding the right architecture and gradient descent parameters. If your problem is very dependent on only a handful of terms then you may be able to get adequate results with a tabular multinomial regression approach and some smart encoding of your sequence. Something like:

y prefix1 prefix2 prefix3 has1 has2 has3  ...
A       1       2       0    1    1    0  ...
B       1       3      -5    1    0    1  ...


Sklearn randomforest online learning Can I do online learning with random forests? I have a few million datapoints and the classifier fails to finish on the cross validation step.

Can i break it up in chunks sequentially?
Current code:

X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()

   print('Planting trees...')
   clf = RandomForestClassifier(
       n_estimators=50,
       max_depth=None,
       min_samples_split=1,
       random_state=0
   )

   print('Growing trees...')
   classifier = clf.fit(X_train, y_train)

   # see how we did
   print('Testing trees...')
   scores = cross_val_score(classifier, X_test, y_test)
   print(scores)
   print('accuracy: %d' % (scores.mean()))

Can I change it to something like:

for chunk in df:
     clf.fit(...)
     cross_validate...

There's nothing out of the box that will do true online learning.  In order for a scikit-learn algorithm to support online learning it must provide the partial_fit function, which RandomForestClassifier does not.  I think the code you have given will just refit the entire forest on the subset of data it is currently looking at.

One alternative you could try is to initialize RandomForestClassifier with warm_start set to True.  Then on each subsequent call to fit, the random forest will add a new set of trees to the ensemble.  You may have to reduce the number of estimators in the ensemble because the number of estimators you end up with will be the number you initialized with times the number of chunks you loop over.

This won't behave exactly the same way as if you were to train the entire ensemble in a truly online way, but it may be close enough for your purposes.


resampling image takes a large amount of time I am working on medical image exactly CT scan images, there is a method for reading these type of images, also another method for resampling, the code for two methods shown below:

def load_itk_image(filename):
    # read infos from route
    itkimage = sitk.ReadImage(filename)
    # read imagearray
    numpyImage = sitk.GetArrayFromImage(itkimage)
    # read coords
    # the given coords and spacing comes: x, y, z and we should transfer it to z, y, x
    numpyOrigin = np.array(list(reversed(itkimage.GetOrigin())))
    numpySpacing = np.array(list(reversed(itkimage.GetSpacing())))

    return numpyImage, numpyOrigin, numpySpacing

def resample(image,oldspacing,newspacing):
    start_time = time.time()
    resize_factor = np.array(oldspacing).astype(np.float)/np.array(newspacing).astype(np.float)
    new_real_shape = image.shape * resize_factor
    new_shape = np.round(new_real_shape)
    real_resize_factor = new_shape/image.shape
    image = scipy.ndimage.interpolation.zoom(image, real_resize_factor, mode = "nearest")
    print("%s time takes in seconds" % (time.time() - start_time))
    return np.array(image)

my problem is, resample function takes a large amount of time to resample only one 2d image approximatly 30 seconds, why it takes these amount of time? is there any way to reduce resampling time?

Although my money would be on the interpolation function taking a fair while, here are a few other ideas:

1. Is that resize_factor matrix going to be different for every single image? You could otherwise pre-compute it one for all images of a given size and then just use it, instead of re-computing it for every image.

2. One aspect, which may answer your other related question on DS), is memory usage, that can take time if the arrays are large enough. Those matrix divisions that you perform will return a new matrix, that means a new chunk of memory. If it turns out (from the profiling idea below), that those division operations are costly, you could instead try pre-allocating an array to hold all your results (assuming you know the relaevant shapes), then filling the array with the output.

3. ``scipy.interpolation.zoomreturns anndarray, so you don't need to callnp.arrayon it again in your return statement of theresample` function.

I would suggest profiling your script. This will allow you to see which function calls are called most often and which take the most time. You can then focus on making making changes that will have the biggest effect on the runtime. Here is a nice example tutorial if you're new to profiling in Python.

You can use cProfile, a built-in python module. You can do this in the terminal quite simply (if your script is designed to be run that way) by running:

python -m cProfile -o profiling_results.prof your_script.py

In words (flags shown in bold): python run the module cProfile, to run your_script.py, producing the output file profiling_results.prof.

The generic version of this command it:

python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py)

You can open and read the output file in a normal text editor. The suffix is a common convention, because you can pass it to a tool like SnakeViz, which will allow you to visualise the results and introspect the method tree of your script; the order in which functions call each other.

There is a notebook on Kaggle, which you may find interesting, as it performs similar tasks on the same LUNA16 dataset.


Advice on the learning resources for deep learning Which is better for beginner in machine learning: The deep learning book written by Yoshua Bengio or the videos and notes in CS231n from Stanford?

http://neuralnetworksanddeeplearning.com/
http://www.deeplearningbook.org/

They are the two very popular online free books. The first link even has working deep learning code.


Using standard deviation as a metric for model selection I'm really getting stuck with overfitting and I'm trying all I can to reduce it.

I want't to write a metric to help score models in a cv loop. I'm using 10x5 folds and still getting out of sample accuracy scores +/- 5-8% from the mean of my CV score. (i.e when i upload to kaggle)

Does the validation score's standard deviation have a correlation with overfitting / error and can this be used in my scoring? currently the stdev is around 5% that seems ALOT.

I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

Does any one have any experience with this?

Any help welcome.

Does the validation score's standard deviation have a correlation with overfitting / error

Yes definitely: a high variance shows that the model is not stable across different training sets, which indicates a high risk of overfitting.

  and can this be used in my scoring?

Using the std dev directly in the scoring itself, I'm not sure. I would consider this more like a kind of qualitative indication. However if two different models have  similar performance but one with higher variance than the other, it's usually a good idea to select the latter.

  currently the stdev is around 5% that seems ALOT.

It always depends on the specifics of the data, but yes I'd say that 5% is quite high.

  I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

I don't understand what you mean here.

In general the standard way to reduce overfitting is to increase the ratio between number of instances and number/complexity of the features:

Add more instances in the training set if possible (but it's rarely possible)
Apply feature selection to reduce the number of features
Simplify the most complex features; that depends on the task, but typically removing values which appear very rarely can help.


how to remove columns in Transformer function in Pipeline I already use a custom transformation function in a sklearn's pipeline. In this function I only add features to my data frame. It works greatly.
Below is a working example.

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_blobs

x, y = make_blobs(n_samples=300, n_features=2, centers=1)
x_train = pd.DataFrame(x[:150,:], columns=['x1','x2'])
x_test = pd.DataFrame(x[150:,:], columns=['x1','x2'])

class myTransformation(object) :
  def __init__(self, colname):
    self.colname = colname

  def transform(self, x) :
    dat = x.copy()
    squared = dat.loc[:,self.colname]**2
    squared.name = "%s_sqre"%self.colname
    dat.loc[:,squared.name] = squared
    dat.loc[:, self.colname+'_2'] = dat[self.colname]
    return dat

  def fit(self, dat, y=None) :
    return self

makePipe = Pipeline([('makeTransfo', myTransformation(colname="x2"))])
fittedPipe = makePipe.fit(x_train)
x_1 = fittedPipe.transform(x_train)
x_2 = fittedPipe.transform(x_test)

Now I want to be able to add the possibility to remove the equal columns in the data frames.
For now, I have the following function :

def delSameCols(df) :
  cols = []
  for i in range(df.shape[1]) :
    for j in range(i+1, df.shape[1]) :
      if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :
        if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :
          cols.append(df.columns[j])
  cols = list(set(cols))
  print( u'      -%s features removed'%len(cols) )
  return df.drop(cols, axis=1), cols

I have no idea how to deal with this : add a new function in the pipeline or directly in the existing function ?
Does anyone have any idea please ?

I succeeded to get a satisfying solution. I post an entire working script. What do you think about it ? Especially the creation of an attribute (self.lstRemCols) not initialized in the init function ?

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_blobs

class myTransfo(object) :
  def __init__(self, colname):
    self.colname = colname

  def transform(self, x) :
    dat = x.copy()
    squared = dat.loc[:,self.colname]**2
    squared.name = "%s_sqre"%self.colname
    dat.loc[:,squared.name] = squared
    dat.loc[:, self.colname+'_2'] = dat[self.colname]
    return dat

  def fit(self, dat, y=None) :
    return self

class removeSameCols(object) :
  def __init__(self) :
    pass

  def _delSameCols(self, df) :
    cols = []
    for i in range(df.shape[1]) :
      for j in range(i+1, df.shape[1]) :
        if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :
          if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :
            cols.append(df.columns[j])
    cols = list(set(cols))
    print( u'      - %s features to be removed'%len(cols) )
    return cols

  def transform(self, x) :
    dat = x.copy()
    lstcols = list(set(dat.columns) - set(self.lstRemCols))
    return dat.loc[:, lstcols]

  def fit(self, x, y=None) :
    dat = x.copy()
    self.lstRemCols = self._delSameCols(dat)
    return self

x, y = make_blobs(n_samples=300, n_features=5)
x_train = pd.DataFrame(x[:150,:], columns=['x1','x2','x3','x4','x5'])
x_test = pd.DataFrame(x[150:,:], columns=['x1','x2','x3','x4','x5'])

makePipe2 = Pipeline([('makeCols', myTransfo(colname="x2")),
                      ('remCols', removeSameCols())])
makePipe2.fit(x_train)
x_1 = makePipe2.transform(x_train)
# test if only same columns in x_train are removed.
x_test.x4 = x_test.x5
x_2 = makePipe2.transform(x_test)


Keras inconsistent training results I have a CNN network in keras. I do the training on a cloud GPU. I get completely different accuracy and loss graphs when I run the training twice. I set random seeds as below, still no luck. Anything missing? Or is this normal behaviour on external GPU? I read that sometimes that they induce randomness because of certain libraries they might be using? I see %2-4 difference in accuracy everytime I run. So makes it difficult to judge my hyperparameter tuning.

import numpy as np
np.random.seed(3)
import tensorflow as tf
tf.set_random_seed(4)
import keras
keras.backend.clear_session()

from keras.layers import LeakyReLU
from keras.models import Sequential
from keras.layers import Activation
from keras.layers import Convolution2D, MaxPooling2D, Dropout, Reshape
from keras.layers.pooling import GlobalAveragePooling2D
from keras.optimizers import Adamax, Adadelta, Adagrad, Adam, RMSprop
from keras.callbacks import EarlyStopping, ModelCheckpoint,
 ReduceLROnPlateau
from keras.regularizers import l2,l1,l1_l2
from sklearn.metrics import precision_recall_fscore_support,
 roc_auc_score
from keras.models import model_from_json
from keras.layers.normalization import BatchNormalization
from sklearn.metrics import confusion_matrix, f1_score,
 precision_score, recall_score
import keras.backend as K
from keras.layers import Dense, Dropout, Flatten
from sklearn.preprocessing import Normalizer

The Dropout layer induces randomness (noise) in the training, because random neurons get disabled in every epoch. This leads to slightly different results per training, but the overall performance should be similar, especially for a large amount of epochs.

In order to set the seed for the Droupout layer, you should modify the seed=None argument when you initialize the layer as keras.layers.Dropout(rate, noise_shape=None, seed=None), as explained in the official webpage.


I have few basic questions on R Stem and Leaf Plot This data when I plot - 60 85 72 59 37 75 93 7 98 63 41 90 5 17 97

stem(data)

 0 | 577
 2 | 7
 4 | 19
 6 | 0325
 8 | 50378

Now If I reconstruct data it is 5, 7, 7, 27, 60, 63, 62, 65, 85, 80, 83, 87, 88

This does not produce the input in reverse.

With the default scale, you see that numbers left of the bar go up by two - hence anything after the 4 is forty-something or fifty-something:

> stem(d,scale=1)

  The decimal point is 1 digit(s) to the right of the |

  0 | 577
  2 | 7
  4 | 19
  6 | 0325
  8 | 50378

Using scale=2, you'll see numbers left of the bar go up by one, so now you can get exact reconstruction of your input, since you input integers:

> stem(d,scale=2)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 | 7
  2 |
  3 | 7
  4 | 1
  5 | 9
  6 | 03
  7 | 25
  8 | 5
  9 | 0378

Going further you can even split it by first and second five within each decade:

> stem(d,scale=4)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 |
  1 | 7
  2 |
  2 |
  3 |
  3 | 7
  4 | 1
  4 |
  5 |
  5 | 9
  6 | 03
  6 |
  7 | 2
  7 | 5
  8 |
  8 | 5
  9 | 03
  9 | 78

Stem plots do not always guarantee you can reproduce the data by reversing the process, and that's not what they're for.


Can somebody explain me the the following parameters of Keras LSTM layer keras.layers.LSTM(units,stateful=False,unroll=False)

What units,stateful and unroll represents here??

units : According to the official docs, it defines the output dimensionality. In simple words, the number of LSTM units which will be used.

  units: Positive integer, dimensionality of the output space.

Units are nothing but the LSTM cells which will be used to process the inputs.

stateful :

According to the docs :

  stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.

If this argument is set to False then, the states or the memory of the LSTM cell will get reset at every sample which is passed through it. It follows this manner:

Take a sample from the batch.
The sample will pass through the LSTM and produce a state vector ( conditional memory ).
The output is given
If stateful=True then, the state vector will be used as the initial state for the 2nd sample.
If stateful=False then, a new state vector will be used for the 2nd sample.

Hence, the memory should migrate from one sample to another is decided by the stateful argument.

unroll :

According to the docs,

  unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences.

It can thought as :

  RNNs, once unfolded in time, can be seen as very deep feedforward networks in which all the layers share the same weights. ( Deep Learning, Nature, 2015 )

You can read an excellent article here.

The unrolling will the make the LSTM, a deep feed forward network which has shared weights. You can see and understand through this figure.


Can a neural network be of variable depth? It is very common for neural networks to be asymmetric about the x axis, that is, to have many more nuerons in the first few layers than in the last few layers. Common example:

But can neural networks be structured in such a way that they're asymmetric about the y axis, that is, such that some parts of the network are 'deeper' than others?
An example of where this could be useful is where a problem involves some unsophisticated features (that require only a small number of layers), and some much more sophisticated features (which may benefit greatly from having more layers - i.e. greater 'depth').

"Wide and Deep network"  was used in a 2016 paper.

$\hspace{5cm}$Wide & Deep Learning for Recommender Systems
You may create using keras Concatenate layer
concat = keras.layers.Concatenate()([input, hidden2])

References to read -
The paper
Google blog
Another paper


Try:

from sklearn.linear_model import LinearRegression
import numpy as np
regression_model = LinearRegression()
Year = np.array([[1970], [1971], [1972]]) #your data
Income = np.array([3399.299037, 3768.297935, 4251.175484]) #your data
regression_model.fit(Year, Income)
regression_model.predict([[1973]]) #make predictions


Does PMML support probability calibration? As I need to port a decision tree model from Python to Java, I would like to know whether PMML (Predictive Model Markup Language) supports probability calibration.

I'm not too familiar with PMML, but probability calibration (or at least, the most well-known methods, Platt scaling and isotonic regression) can be viewed as a stacked ensemble, with the output of your model being fed into a univariate regression model.  PMML appears to support ensembling; see the fourth example at http://dmg.org/pmml/v4-3/MultipleModels.html


Which one is better performer on wrangling big data, R or Python? I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and libraries performs the best on wrangling and cleaning Big datasets?

I think this may help you:
Is the R language suitable for Big Data

I think it really depends on what you are comfortable with and what your objectives are.  In Python, I can execute SQL queries directly into Pandas data frames.  From there, data cleaning and visualizing (I like seaborn) is fairly straightforward.  I can easily perform matrix manipulations using Numpy, and the sklearn is a well documented ML package.

That being said, I still like R and there are definitely times when I will use R.  Learn both!


Data preprocessing: Should we normalise images pixel-wise? Let me present you with a toy example and a reasoning on image normalisation I had:

Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (black) to 255 (white).

Class 0:
Images that contain a filled circle in the centre of the image. The pixel values within the circle are high, close to 255. Pixels outside the circle are low, close to 0. Below is an example of a class 0 picture.

Class 1:
Images that are empty, containing pixels close to 0. An example is below, basically a dark image.

Note: The images show basically binary values: white/black, however, assume that black values could be in the range of [0, 30] and white ones [230-255] with some random distribution.

Prior to feeding the network with the images, we pre-process and normalize the data. This basically means (i) centring the data and (ii) scaling the data. A typical and simple approach could be to subtract 127.5 to all pixels and scale by 255 so as to have pixel values ranging [-1, 1]. Alternatively, another approach consists of computing the "mean image" and the "dynamic-range image" and centre and scale the images pixel-wise. Let us focus on the latter.

Computing the "mean image" mainly means summing all images pixel-wise and taking the average pixel-wise. As a result, we obtain a NxN average image. This can sometimes shed some light on our task and help us understand the data we are working with. Likewise, the "dynamic range image" is obtained by computing the maximum and minimum values pixel-wise. The "dynamic range image" shows us which range of values each pixel can take.

Hypothesis 1

During training, the kernels of the different layers in the CNN will be trying to detect the circle. Since the task is rather simple, the kernels basically would have to learn that there is a strong pixel-value-transition from "empty region" to "circle region" when a circle is present.

Hypothesis 2

After normalising the images pixel-wise, we have that all pixels have the same range and same mean, namely [-1, 1] and 0, respectively.

Final Hypothesis

Consider a class 0 image (i.e. image has a circle). Before normalizing, pixel (i, j) has value 30 (dark), which happens to be the maximum that pixel (i, j) ever takes. Pixel (i, j+1) has a value 255 (white), also the maximum value that pixel (i, j+1) ever takes. So we have a strong transition that a kernel from first layers should be able to learn.

But, if we normalise the data with "mean image" and "dynamic range image"
isn't this strong transition going to be softened? In the particular, the transition would be from value 1 to value 1, since both were the maximums at their respective positions. From a clear and hard transition we have ended with a transitioned from similarly valued pixels.

Conclusion
Using this pixel-wise normalization we end up losing global pixel information, which might make the job of the kernels more difficult. I believe same conclusions might be drawn if "variance image" is used instead of "dynamic range image".

Question
Am I missing some point or is this normalisation no longer used?

Bonus
I also thought of Batch Normalisation, which sometimes is used at the first layer so that normalisation is done within the network. I thought that maybe the parameters "beta" and "gamma" used in BN might be able to fix this issue if it ever appears while training.

We first center our data by subtracting the mean of the batch.  We also divide by the standard deviation, so our formula becomes:

$ z = \frac{x - \mu}{\sigma} $

where:

$ x $ is the pixel value

$ \mu $ is the arithmetic mean of the channel distribution

$ \sigma $ is the standard deviation of the channel, calculated as such:

$ \sigma = \sqrt{\frac{\sum^N_{i=1} |x_i-\mu|^2}{N}} $

By standardizing the data in this fashion, we obtain a mean ($\mu$) = 0 and a standard deviation ($\sigma$) = 1.  This allows SGD to converge faster.

It is easier to see how feature scaling can benefit simpler models such as SVMs, but CNNs benefit similarly.  (Mini-)Batch normalization makes every pixel have a similar distribution, which helps SGD converge faster.  This is because networks generally have shared hyperparameters.  For instance, the learning rate is a multiplier for all weights in a network, rather than having a learning rate for every weight; therefore, we want a single learning rate to have proportional influence in all regions.  Otherwise, the learning rate could make some weights over-correct and other weights under-correct.

This article does a great job of explaining CNN preprocessing.  I'll try to highlight the key points (the following images are pulled from it)

Let's consider a face recognition challenge:

Taking the mean (left) and standard deviation (right) of the batch, we get the following:

Our centered data resembles a face, but more importantly our standard deviation image shows great variance along the borders and everywhere except the face.  From our z-score formula, we can predict that the standardized values near the border and irrelevant areas will be relatively squashed more than values around the face.  The resultant normalization appears as such:

Normalizing the input images is widely used and often considered an essential preprocessing step.  However, making an absolute statement such as, "normalization is always beneficial" or "normalization is never beneficial" would be misguided.

In the scenario you presented, normalizing the data would not benefit the CNN, but the network would still be able to learn the data well.  However, this is a very primitive situation, and most real-world problems have significantly more noise and variance for which to account.  In general, normalizing the input image will improve SGD convergence time.


regression network that will be optimised on a subset of the data I am trying to optimise my network that is trying to perform regression. Currently, the dataset is dominated by values in a certain range. It is very good at predicting these values. I want to try to find a way to make it focus on the values outside of this bulk, these values outside the bulk are much higher. We thought about splitting the dataset into parts, and training multiple networks, but we really want the regression to be done by a single network.
An idea that we had was to adapt how the loss function calculated loss as we are training. So, after a certain number of epochs passed, or after a certain accuracy was achieved, we would only pass the loss function values that were being very poorly predicted. The motivation was that this may improve the performance of the network as training would have to focus on these points.
I have not found much literature, or examples, trying something like this out. Although, it seems like it could work. Naively, it seems like it's in the same vein as weighting a loss function, or when you perform transfer learning and retrain just the final few layers of the network to make it specific to your problem.
My questions:

Generally, how does this sound to people? Does anyone have any opinions on it or ideas on how to improve it?
Is there any work been done doing something similar that people know of?

Your reasoning sounds similar to curriculum learning: the idea is to pass training data to the network not randomly, but based on some scoring function $f$. It was proven that a network learns faster and better, even though the process itself is not well understood. I think it is a better approach than the one you suggested: I am afraid that, if you pass only poorly predicted data after a while, the network will forget what learned before.
What you can also do is to perform data augmentation to compensate your unbalanced data (if I understood well, higher ranges in your dataset are not well represented compared to a certain range).


scikit-learn pipeline strategy with both numeric and categorical columns We are implementing an sklearn pipeline as follows (pseudo code):

feature_num = numeric_columns
feature_cat = categorical_columns

num_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_num))
                        ,('feature_num_prep', custom_num_prep)
                       ])
cat_pipeline = Pipeline([('feature_select', DataFrameSelector(feature_cat))
                        ,('feature_cat_prep', custom_cat_prep)
                       ])
feature_pipeline = FeatureUnion(transformer_list=[
                               ('num_pipeline', num_pipeline)
                               ,('cat_pipeline', cat_pipeline)
])

Since we have in the custom_num_prep transformer the possibility to remove outliers - when we get to the feature_pipeline there is an error:

  ValueError: [blocks[0,:] has incompatible row dimensions....

The reason is obvious since the cat_pipeline has observations that the num_pipeline has removed.
The question is:
1. is there a pythonic way to sync the two dataframes within the pipelines?
2. if not what strategy is there for such situations?

One option is scikit-learn's ColumnTransformer applied to mixed types. ColumnTransformer is designed for the purpose of applying different preprocessing and feature extraction pipelines to different subsets of the features.


How to normalize just one feature by scikit-learn? Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact.

the dataset format is something like:
[
[1, 0.2, 1000],
[2, 0.1, 2400],
[3, 0.9, 7620]
]

I need to transform only one column, the third in this example.

I don't want to use pandas.

As mentioned, the easiest way is to apply the StandardScaler to only the subset of features that need to be scaled, and then concatenate the result with the remaining features.

Alternatively, scikit-learn also offers (a still experimental, i.e. subject to change) ColumnTransformer API. It works similar to a pipline:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import  StandarScaler
column_trans = ColumnTransformer(
    [('scaler', StandardScaler(),2],
    remainder='passthrough')
column_trans.fit_transform(X)

The third entry of the tuple (here : 2) can be a single or a list of column indices (or column names when working with a pandas DataFrame).


ML methods for prediction, using categorical variables and time Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate time series data, where the fields are all categorical variables.

Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the type of alert, the location of the problem, etc. are stored in the alert. These fields are all categorical variables.

My data is very similar to the one described in another question, so I used a similar description. However my question is a bit more general.

Given a chain of alerts, what is a good approach for using ML to predict the location and type of the next alert (at t(1) ), having knowledge of common alert patterns and the values for t(0), t(-1), t(-2), ...t(-n) ?

One way to frame the problem of predicting patterns of complex discrete states is probabilistic graphical models (PGMs). PGMs model joint (multivariate) distributions over large numbers of random variables that interact with each other.
PGM do not have a notion of linear time. They model state. Given the current state and previous states, how likely are other states in the next time step? In your problems, state is the type and location of alert.
Common examples of PGMs are Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs).


When classifying documents with naive Bayes, which probabilities do you multiply in case of repeated terms? In short: For naive Bayes and text classification, do you multiply a probability for each instance of a word in a document or once if the word occurs?

In more detail: The question is how to calculate naive Bayes for a document. We have a corpus of text from which we can calculate the frequency of documents with class $C$ having a word $w$. We then calculate the probability $P(w_1|C)...P(w_n|C)$ and choose the $C$ that maximizes this number. But I'm not sure whether we multiply a probability for each word whenever it occurs or just once if it occurs in the document? Like, if the text is "$w_1 \ w_2 \ w_1$", is the estimated probability $P(w_1|C)^2 P(w_2|C)$ or is it $P(w_1|C)P(w_2|C)$?

You could use either of the two approaches:

Just count whether your documents in your corpus contain or not a word $w$ and then estimate the likelihoods of $P(w_1|C)...P(w_n|C)$, as you describe
You could count how many instances of any word has each of the documents of your corpus, but then the estimation of the likelihoods becomes a bit more complicated than what you describe; in essence you need to use a multinomial distribution: $P(w_1|C=k) = \sum_{i=1}^{N}x_{it}z_{ik}/\sum_{s=1}^{|V|}\sum_{i=1}^{N}x_{is}z_{ik}$, where $N$ is the number of documents, $x_{it}$ in the numerator is the counts of word $w_1$ per document, $s$ in the denominator iterates the number of terms $|V|$ in your corpus and $x_{is}$ is the number times each term appears in each document, and finally $z_{ik}$ is 1 if the document belongs to class k, 0 otherwise, in both numerator in the denominator

Hopefully the short explanation makes sense, anyway you have a very complete and useful reference here (the same notation in the formula above).


What is the proper way to bin variables for calculating WoE during credit scoring? I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that:
1. Each bin should have at least 5% of the observations
2. Each bin should be non-zero for both good and bad loans
3. The WOE should be distinct for each category. Similar groups should be aggregated or binned together. It is because the bins with similar WoE have almost the same proportion of good or bad loans, implying the same predictive power
4. The WOE should be monotonic, i.e., either growing or decreasing with the bins
5. Missing values are binned separately

This seems like a lot of work to accomplish manually (each column needs to be divided in bins, each of these five conditions need to be checked manually, bins should then be adjusted and the conditions need to be checked again). Is there a faster way to do it? Or is there any algorithm/function that bins continuous variables in most practical way.

Or is there any algorithm/function that bins continuous variables in most practical way.

Sure there is, but that's the wrong question: the standard method for discretizing a continuous variable consists in splitting the values into equal intervals, that's it. Of course it doesn't guarantee any of the 5 conditions, since these conditions are about additional constraints almost exclusively related to expert knowledge and the specifics of the data.
Note that these conditions can certainly be automated, there's no need for manual verification. There might be some domain-specific packages which do this for you, but there's no reason a standard ML/statistics library would provide methods for every specific problem like this one.


How to compare the output of Self-organizing maps? I am trying to simultaneously cluster and visualize text documents using Self-organizing maps. Since text documents can be represented in various ways (vector space model, GloVe etc), I am trying to figure out how to tell which representation generates the best map. Measures like Quantization error etc., determine the goodness of the map given a dataset. However, they are not useful for quantitatively telling which representation gives a better output.

Is there a quantitative measure to compare the maps generated using different representations (for example, Tf-idf and GloVe) and tell for which representation the output is better?

From Wikipedia:

  A self-organizing map (SOM) or self-organizing feature map (SOFM) is a type of artificial neural network (ANN) that is trained using unsupervised learning to produce a low-dimensional (typically two-dimensional), discretized representation of the input space of the training samples, called a map, and is therefore a method to do dimensionality reduction.

So you only have the original data itself; no additional data (like labels in a supervised setting). If you are also say the result has to have two dimensions, you basically look at functions

$$f: X \rightarrow \mathbb{R}^2$$

where $X \subsetneq \mathbb{R}^n$ in most cases. You already mentioned quantization error.

Up to my knowledge there is nothing better measure which does not include getting more knowledge about the data itself by human inspection / using other datasets.

With human inspection you can, of course, tell for a given dataset and a given human if one mapping seems to make more sense.

You might also consider other dimensionality reduction techniques:

t-SNE
Auto-Encoders


Why multi-class SVM can't detect some classes? I use PCA and multi-class SVM for classifying 4-class problem in the Python environment. But in results, I see some differences in detection rate (Unweighted Accuracy in this problem). For example:

FV  | cl.1  % |   cl.2  %  |   cl.3  % |    cl.4 % |

fv1  | 5.4 % | 47.4 % |  2.6 % |   78.6 % |

fv2 | 2.6 % |  41.6 % |  8.3 % |   78.4 % |

fv3 | 5.9 % |  49.3 % |  1.8 % |   76.5 % |

you can see that SVM couldn't detect classes 1 and 3 as well as classes 2 and 4. I would be appreciated if anybody helps me to solve this issue.

Well, based on how PCA works, and the fact that you're using multi-class SVM, which is a pretty solid family of algorithm, the only possibility i can think of is that the problem comes from your data. It can still depends on the implementation you're using, but i think it come from your data.


How to continuously train a model with a stream of new incoming data I have limited experience with machine learning, I trained a few networks, but nothing out of the ordinary. I have the following problem but I am not quite sure how to approach it and I'm hoping to get some advice here.
I have an series of time dependent data (timestamp + value) which I want to use to predict the next couple of outcomes. Usually this problem could be solved by using an LSTM. Every few minutes I get a new datapoint which I want to store and also take into consideration for future predictions.
The problem I think I have is that I'd need to train the model every few minutes with changing input size (I want to train the model with the entire history at once, I don't know if that makes sense) and also I don't want to overfit the model over time.
I'm not sure which architecture or concept is suited best for such a problem, so I hope to find some advice here. Thanks!

This is a form of continual learning / life-long learning, which tends to be very challenging because retraining on new data make traditional models "forget" what they had previously learned. Depending on the frequency by which you receive new data and the time it takes to train the model, it is often infeasible to retrain on all data every time you receive a new sample.
w.r.t. LSTMs, you don't need to change your input size because you can partition your time series data according to a preset and constant input size. For example, if you have 100 ordered samples, you could have the LSTM receive input of 10 samples and predict/regress the following 11th sample. So you'd have 90 partitioned samples of constant size. If you had another ordered sample (101), then you'd have 91 partitioned samples of size 10.
Continual learning is an active research field, so you can find many recent works such as this one from a couple months ago on adversarial CL.
For a review of CL in robotics, you can see this paper.


What fits in a Data Description Report/ Data Exploration Report? So I am trying to get familiar with Crisp-DM and found the terms "Data Description Report" and "Data Exploration Report", which seem oddly vague in their definition. So far I only found this right here:
https://www.ibm.com/support/knowledgecenter/en/SS3RA7_15.0.0/com.ibm.spss.crispdm.help/crisp_data_description_report.htm

But this seems to be on the shorter end in my opinion. Is there any example of a Data Description Report anywhere? If not, is there any systematic methodology you personally use to record your findings while trying to understand data?

Generally speaking each data set may have different structure and may relate to different business aspects.
Because of that I think its hard generalizing the description/exploration steps.

Yet, these are my two cents:

Data description:
  * Data sources - how was data created, generated, collected.
  * Data shape - number of rows, columns.
  * Data types(per each column) - numeric, strings, other..
  * Missing values - how much, where, why?
  * Time frames - if relevant.
  * Entities(geographical, markets, population segments, devices) - what identifies a row in the data(if applicable).

Data exploration:
* Counts per entity(grouping keys).
* Categorical variables breakdown.
* Values ranges distributions.
* Simple correlations.
* Story telling(visualizations that depict key aspects in the data<>business realm).
* Missing data deep dive.


Decision tree where identical set of features results in different outcomes I am following the example described in this page to test my decision tree program.

The initial data set is

age            astigmatism, tp-rate, contact-lenses
---------------------------------------------------
young,          no,  normal,  soft
young,          yes, reduced, none
young,          yes, normal,  hard
pre-presbyopic, no,  reduced, none
pre-presbyopic, no,  normal,  soft
pre-presbyopic, yes, normal,  hard
pre-presbyopic, yes, normal,  none
pre-presbyopic, yes, normal,  none
presbyopic,     no,  reduced, none
presbyopic,     no,  normal,  none
presbyopic,     yes, reduced, none
presbyopic,     yes, normal,  hard

age, astigmatism, tp-rate are features and the type of contact lenses is the outcome.

When I trace the example (where features with lowest entropies are used as the decision nodes), we reach the following situation a little further down the page:

The blue circles are obviously mine to make tracing easier. As we see in the constructed tree up to this point, the path

[Astigmatism = yes] -> [TP-Rate = normal] -> [Age = PP (pre-presbyopic)]

will lead to the outcome of [Contact-lenses = None].

But when you look at the data, we see that this path can lead to either a hard or a none type of contact lens.

So has the example made the right decision?

And if not, what should we do in situations where the split data-set ends up, under an identical set of features, with more than one outcome?

The task in hand can be modeled as a classification problem.
Meaning based on some given features we aim to predict the 'right' class of the outcome.
Once trained, most classification models, decisions tree included, will predict the most probable class given a set of input features.
Learning what is the most probable class, is usually done in the 'fitting' step of model training.
In your example the set of features results in 2 possible classes(hard, none) meaning that there is a 50% chance being either.
The classifier is just as wrong as its right in predicting the 'none' label.

** Some decisions tree implementations include an option to predict the probability for each class, and predict a class only if a specific confidence threshold is defined.


create sequence of non dictionary words I have a few word vectors-
recvfrom,sendto,epoll_pwait,recvfrom,sendto,epoll_pwait

getuid,recvfrom,writev,getuid,epoll_pwait,getuid

Now i want to tokenized them and then make them into sequences to feed into the model-
For a standard word vector I would do something like this-
### Create sequence
vocabulary_size = 20000
tokenizer = Tokenizer(num_words= vocabulary_size)
tokenizer.fit_on_texts(df['text'])
sequences = tokenizer.texts_to_sequences(df['text'])
data = pad_sequences(sequences, maxlen=50)

But in my data I have non dictionary words and also I have some repeating words. How do I convert this data into sequences?

this can be done in multiple ways i am not one hundred percent understand the question because it is badly worded but there are 3 ways!
if you are doing this for a neural network you can use keras embedding layer
and to create the sequence to feed to this embedding layer you can use one hot and padding from the preprocessing packages of course the sequence can be used regardless of if it is a nn the second thing you can do is use gensim to create word to vector the create a list of words or if you want the most control you can do something like
for i in input_texts:
    for char in i:
        if char not in input_characters:
            input_characters.add(char)

for i in target_texts:
    for char in i:
        if char not in target_characters:
            target_characters.add(char)
            ```
and then

input_token_index = dict(
    [(char, i/num_encoder_tokens) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i/num_decoder_tokens) for i, char in enumerate(target_characters)])

in your scenario you can just iterate through the sentence with a for loop and then create a dictionary after which you can feed it through a one hot and padding preprocessing from Keras.preprocessing and finally create an embedding


Reduce size of a network graph for bipartite projection I have a graph that I created from a pandas data frame. The length of the graph is ~450k edges. When I try to run the weighted_projected_graph function, it runs for a long time (I have not seen it finish), presumably because of the size of this data set. What is a good method for reducing the size of this data set before creating the bipartite graph?
I have tried narrowing it down by using the most connected components:
trim1 = len([c for c in net.connected_component_subgraphs(g) if len(c) > 10])
C = net.Graph()
gg = sorted(net.connected_component_subgraphs(g), key=len, reverse=True)[trim1]

but I don't think this is giving me the results I want and, further, I'm not confident this is an analytically sound strategy. Does anybody have any other recommendations for reducing the size of this set?
EDIT: full code, without reducing. What I would be trying to do with the above would swap out g below for gg from above in the call to bi.projected_graph
reviews = pd.read_csv(r"\BX-Book-Ratings.csv", sep=";", encoding = "ISO-8859-1")
users = reviews['User-ID'].values.tolist()
books = reviews['ISBN'].values.tolist()

g=net.from_pandas_edgelist(reviews,'User-ID','ISBN',['Book-Rating'])

print(len(g))
>>> 445839

p = bi.projected_graph(g, users)
```

I think the problem is not with the data size, but with the presence of large degree nodes (the degree of a node is its number of neighbours).
Indeed all neighbours of a node in the bipartite graph are linked together in the projection; they form a clique (complete sub-graph) in the projection. This is terrible, as the number of links in a clique of $d$ nodes is $\frac{d\cdot(d-1)}{2} \sim d^2$.
In concrete words, a node of degree 2,000 in the bipartite graph will induce almost two million links in the projection... A node of degree 20,000 will induce almost two hundred million links...
For more discussion of the problems with projections of bipartite graphs, you may have a look for instance at this paper: Basic Notions for the Analysis of Large Affiliation Networks / Bipartite Graphs


How to build a machine translation system for a new language I am trying to figure out what are the options for building a natural language translation model for a language that is not yet supported by existing machine translations.
The project is to build a system for translating a very limited subset of a small east african language into English (only one-way needed).
The language is to my knowledge not yet supported by any machine translation systems, but it is related to several other big African languages (putting sentences into Google Translate, the language is mostly auto detected as Kiswahili or Shona, and the English translation is sometimes usable).

I know building a translator for a new language is by no means trivial, but the problem domain is very small and I think it should be doable. Are there any features of the big cloud providers that support this, ML frameworks, or vendors that build these models as a service?

If this is not the right stack exchange for this question, kindly guide me to a better place.

I think your case would benefit from tunning a existing language to the new one but that is only a good approach if you plan to use it commercially. Also Google accepts help to improve their translation algorithm and you could petition for them to assemble a team for this and donate data.

Google uses a Neural Network that they call Google's NMT (Neural Machine Translation) for translation and it works as a encoder-decoder pair. You can read more on their paper.

Also, Google's NMT is available in tensorflow and can be trainned and improved. It is license under Apache License 2.0 and has a nice tutorial and explanation available on their GitHub:

Check on this GitHub folder here


Learning Algorithm that decide which model gives better results for each testing instance Is their any existing Ensemble technique which uses subset of training data to predict which algorithm is better for predicting each instance of testing data?

Let's say we have N sized training set and K sized testing set in which a particular attribute needed to be predicted using the training set. But there are hundreds of algorithms and ways we can use. We can divide training set into two parts and train each model with first half and decide test on second half. Based on characteristics, we can decide which algorithm to use for real test cases (K sized set). As an example lets say dataset have an attribute named "temperature". Particular algorithm may work well when temperature is higher than 100 Celsius. We can then classify all the 100 degree or above instances to particular class. Then final prediction will be done based on that by with that model class trained with all N sized data.

What I am asking is that is their any existing method similar to that?

Applying any non-linear model in a model stacking approach should do what you want. In brief the approach is to take predictions from other models as new features, plus the original data and labels, then use them to train a meta-model. Read the link, it offers practical advice on how to do this within a k-fold validation framework, which will give it a much better chance of doing well.

Non-linear models that combine simpler units - e.g. neural networks and models that use multiple decision trees (e.g. xgboost) - already perform this kind of internal split during training. So if you are already using those, you might not gain such a big improvement over simpler ensemble techniques, such as taking a mean or weighted mean over models.


Student answer evaluation Given a question, with some model answers and their grades, and possibly other 'learning material' content.

What tools or technologies can be used, to implement a system that evaluates a student answer?

I am new to machine learning.

Thanks.

One idea that could be leveraged - use all the model answers and learning material, find some sort of semantic similarity between all that text and the student answer. The higher the similarity with a model answer with a specific grade tag, the higher the probability of assigning that answer that particular grade. You can couple this with a metric of similarity between the student answer and learning material, more similarity between the two, higher probability of a better grade.

This seems like a very hard problem considering it is still a very active area of research in NLP, Some tools or concepts that might be interesting to you - Latent Semantic Analysis, word embeddings, text/document similarity, semantic coherence.


Is a Neural Network with 20 times the number of input neurons (on hidden layers) guaranteed to overfit? When is this not so? I'm aware of the problem of over-fitting.  One way to describe it is your Neural Network learning your training data to a high accuracy and performing poorly (generalizing) on new data.

Was wondering if there are situations where having 20 times the number of input neurons on the first hidden layer wouldn't necessarily produce overfitting always.

Here is a screen shot of my work flow.  As you can see I'm using test data.  The way I've done it is split the data myself using a "Past/Future" column with a P or an F.  The future is the last 5% of the sequential time data.

I don't use randomization since I don't think it makes much sense to unsort the sequential time datapoints.

Thanks.

Figuring out the correct size for hidden layers in NN is a bit of an art, that said I have never used hidden layers with more than twice the number of neurones as the input layer.

As an example the MNIST problem has an input size of 784, I cannot imagine how the network will not overfit if the hidden layer is of size 784*20...

So answering to your question, I think overfit will pretty much occur on those occasions, if unsure, just try it in the MNIST dataser


Newton-Raphson or EM Algorithm in Python Is there any implementation of Newton-Raphson or EM Algorithm? Can I get the source code of it?

I tried googling, but didn't come across any. So asking here.

Thanks!

scikit learn has the EM algorithm here.

Source code is available.

And if you are an R fan the mclust package is available here.


Classification problem with many images per instance I am working in the following kind of classification problem: I have to classify every instance as class A or class B using many images of the instance. That is, every training example has not one image (which is the usual thing in image classification), but many images, and the number of images for every training instance is not fixed. That is, instance 1 can have 3 images, and according to these images we have to classify it as A or B, and instance 2 can have instead 5 images.

As any machine learning problem, I am provided with many labelled images and I have to build a classifier.

Although ideas are also welcome, I am looking for a documented way to attack this kind of problem (Kaggles, papers or books, mainly).

My main idea was the following: train a model $f$ that given one image gives a probability of that image being of class A. Then, for every training instance, evaluate $f$ in every image of the instance and compute statistics (aggregate) of the distribution of these probabilities, as the mean, median, maximum and minimum. Then, train a model $g$ that has as inputs these aggregates and use the composition of $f$, aggregates and $g$ as the final model. This idea is a bit simple so I am looking for something better.

This might be a hack, but, have you considered repeating some images to make every instance contain a fix number of images? Let's say you have images A, B, C, D, E, F, G, H, and I. Your dataset might look like this:

A , B
C
D, E, F
G
H, I

Make a function that will randomly select one of the image in an instance and then use that as placeholder. From our example, we now modify our dataset into:

A, B, A
C, C, C
D, E, F
G, G, G
H, I, I

Then, from here, you can use a convolutional neural network with 3 inputs. Here is an example of a CNN with multiple inputs: Multi-Input Convolutional Neural Network for Flower Grading - images of same flowers taken at different angles.

Also, you can shuffle the images in each instance to produce more training samples.


Fully-Configured Deep Learning Virtual Machines in Python (VirtualBox or VMware) Fully-Configured environment setup for Deep Learning Virtual Machines in Python (VirtualBox or VMware)

Often when we start working on any new technology, the most common challenge that we face is "How can I not waste time setting up my environment but directly working on the technology?"

If you aren’t a Linux user, it can be really hard to figure out how to get a system fully configured with all the required machine learning libraries and tools like TensorFlow, Theano, Keras, OpenCV, and dlib. A lot of people get stuck while installing everything and give up before ever getting to play around with any code.

Earlier I used to work on Fully-Configured Deep Learning VM in Python VMware shared by Adam G with the below configuration.

  Link for VMware Image

  Link for VitualBox Image

    The virtual machine image has Ubuntu Linux Desktop 16.04 LTS 64-bit pre-installed with the following machine learning tools:

Python 3.5
OpenCV 3.2 with Python 3 bindings
dlib 19.4 with Python 3 bindings
TensorFlow 1.0 for Python 3
Keras 2.0 for Python 3
Theano
face_recognition for Python 3 (for playing around with face recognition)
PyCharm Community Edition already set up and ready to go for all these libraries
Convenient code examples ready to run, right on the desktop!
Even the webcam is preconfigured to work inside the Linux VM for OpenCV /
face_recognition examples
(as long as you set up your webcam to be accessible in the VMware settings).

Currently looking for the latest version of Ubuntu with updated Deep Learning packages.

I've been searching around the Internet for a while but I have not been able to find Fully pre-configured latest VM with the updated versions.

I would really appreciate if anyone here can share his/her method if he/she had used a Fully-Configured Deep Learning VM in Python (VirtualBox or VMware)

downnload here.

Python 3.5
OpenCV 3.2 with Python 3 bindings
dlib 19.9 with Python 3 bindings
TensorFlow 1.5 for Python 3
Keras 2 for Python 3
face_recognition for Python 3 (for playing around with face recognition)
PyCharm Community Edition already set up and ready to go for all these libraries


Identify "steady-state" time series window I'm new with the time-series analysis.
I have several time-series (noisy of course) part of the same set of measurements (sampled simultaneously).
The time series are the results of a stochastic process (dynamic system) where part of the time-series are inputs to the process and other time-series are the outputs.
I need to find a time window corresponding to a steady operation, i.e. where both inputs and outputs don't change over time (for a deterministic signal would say derivative over time almost zero).
Do you know any way to accomplish this task? The measurements are quite long, I need a rather computationally efficient method ideally without involving any visual inspection.
Thanks!

This could be defined as a classification problem over your time series.
One approach I can think of is to build an LSTM based classifier with a couple of dense layers for binary classification (whether the window is steady or not) and define your own loss function.
So in theory, you're training the LSTM network to sample batches of your time series and classify them as steady state or not.


Gradient Descent in logistic regression Logistic and Linear Regression have different cost functions. But I don't get how the gradient descent in logistic regression is the same as Linear Regression.

We get the Gradient Descent formula by deriving the Squared Error cost function. However in Logistic Regression we use a Logarithmic Cost function instead. I think I am lost here.

Gradient Descent is an universal method, you can us it with basically every loss function you can find in known ML algorithms.

In your case, you have only to derive the logarithmic cost function. You can find a detailed calculation at

https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression


Is there a replacement for small p-values in big data? If small p-values are plentiful in big data, what is a comparable replacement for p-values in data with million of samples?

There is no replacement in the strict sense of the word.  Instead you should look at other measures.

The other measures you look at depend on what you type of problem you are solving.  In general, if you have a small p-value, also consider the magnitude of the effect size.  It may be highly statistically significant but in practice meaningless.  It is also helpful to report the confidence interval of the effect size.

I would consider this paper as mentoned in DanC's answer to this question.


Machine learning solution approach to match loan repayments I'm relatively new to the AI/ML space, but come from a programming background.

The problem: I have a dataset of users transactions who have taken short-term loans from a single loan provider and I need to match the loan repayments to a particular loan receipt.
The sample dataset is below:

transactionDate | amount | creditDebit

2017-09-01      | 500 | Credit
2017-09-08 | 250 | Debit
2017-09-17 | 294 | Debit
2017-12-11 | 300 | Credit
2017-12-15 | 150 | Debit
2018-01-07 | 200 | Credit
2018-01-12 | 398.8 | Debit
2018-01-19 | 200 | Debit
2018-02-02 | 150 | Debit

Here the 'Credit' values for creditDebit column indicate a received loan and 'Debit' indicates a repayment of some loan received previously.

As you can see the first 3 rows are pretty easy to match because there is 1 Credit (amount of 500) followed by 2 Debits (amounts 250 and 294).

From 4th row onward is where i cannot figure out how best to match.

2017-12-11 there is a Credit transaction for a loan
2017-12-15 is a partial repayment of this loan
Then is another Credit for a new loan
2018-01-12 is a partial repayment for the 2 loans previously
The final 2 Debits are partial repayments also

I have been able to manually determine which repayment makes most sense for which loan, but trying to write it out into an if-else function would be very hard in my opinion and not the best.
For example the hardest part is to determine the Debit on 2018-01-12 which is a combined repayment transaction for both of the previously received loans. I have manually calculated a reasonable split of this Debit for the loans at 170 for the loan received on 2017-12-11 and 228 for loan on 2018-01-11.

Is there some model/algorithm that I could use to match which debit transactions are most likely to be repayments of which credit transactions and if the repayment transaction looks like a compound repayment of multiple previous loans - then what would be a reasonable split of the debit to satisfy the credit transactions?

I don't know if this makes sense to you..

I tried thinking about how I could use the clustering algorithms, but i don't I could since the clusters would not indicate matching of credit and debit transactions, especially the compound repayment debits.

Any ideas on how to tackles this are much appreciated!

I don't think machine learning is the right tool.  It's difficult (at least for me) to formulate this as a learning problem.

You could consider genetic algorithms as an alternative approach.  Based on your description of how you do the matching by hand, it seems like the following assumptions hold:

You're looking for an assignment of payments to loans such that the difference between (principle + interest) and $\sum debits$ is minimized
You favor solutions where the time between loan and repayment is minimized.

If the above is true, you may want to consider genetic algorithms as an approach.   For GAs, you need to specify a representation, an objective function, and mutation/crossover operations.  I think the following would work:

Individuals in the population (i.e. candidate solutions) are represented as vectors of length $N$, where $N$ is the total number of debits.  Each entry in this vector assigns the debit to a credit.  For example, the individual $[1, 1, 2, 3, 2, 3]$ means that the first two debits are associated with the first credit, the third debit is associated with the second credit, the fourth debit is associated with the third credit, and so on.

The objective function that you wish to minimize is described by the two bullet points above.  I think it should be pretty easy to programmatically compute this function.  For each credit, compute the magnitude of the difference between (principle + interest) and $\sum debits$.  Sum these differences, then add a penalty for temporal separation between credit and repayment.  One easy-ish way to implement the penalty is to add the edit distance between a solution $V$ and its sorted counterpart $sort(V)$.  You'll want to play with the weight of the time penalty such that some delay between credit and debits is allowed, but you still favor quick repayments.

You'll need to pick suitable mutation and crossover operations.  For this representation, the traditional two-point crossover would likely work well, and uniform mutation would probably work fine.

As an aside, I sure hope this isn't a real-world approach to loan repayment matching.  You will wind up with some angry customers when a debit is inevitably assigned to the incorrect credit.


How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library Sorry for a long story, but it is a long story. :)
I am using h2o library for python to build a decision tree and to extract a decision rules out of it.
I use some data for training where labels get TRUE and FALSE values.
My final goal is to extract the significant path (leaf) of the tree where the number of TRUE cases significantly exceeds that of FALSE ones.

treemodel=H2OGradientBoostingEstimator(ntrees = 3, max_depth = maxDepth, distribution="bernoulli")
treemodel.train(x=somedata.names[1:],y=somelabel.names[0], training_frame=somedata)
dtree = H2OTree(model = treemodel, tree_number = 0, tree_class = False)

def predict_leaf_node_assignment(self, test_data, type="Path"):
    if not isinstance(test_data, h2o.H2OFrame): raise ValueError("test_data
    must be an instance of H2OFrame")
       assert_is_type(type, None, Enum("Path", "Node_ID"))
    j = h2o.api("POST /3/Predictions/models/%s/frames/%s" % (self.model_id,
    test_data.frame_id),
        data={"leaf_node_assignment": True, "leaf_node_assignment_type":
    type})
    return h2o.get_frame(j["predictions_frame"]["name"])
dfResLabH2O.leafs = predict_leaf_node_assignment( dtree,test_data=dfResLabH2O , type="Path")

In sklearn there is an option to explore the leaves by using tree.values
There is not such an option for h2o as I understand.
Instead of that there is an option in h2o to return predictions on leaves.

When I run dtree.predictions
I am getting pretty weird results

dtree.predictions
Out[32]: [0.0, -0.020934915, 0.0832189, -0.0151052615, -0.13453846, -0.0039859135, 0.2931017, 0.0836743, -0.008562919, -0.12405087, -0.02181114, 0.06444048, -0.01736593, 0.13912177, 0.10727943]***

My questions (and somebody has already asked it, but no clear answer was provided so far)

What's the meaning of negative predictions? I expect to get a proportions p of TRUE to ALL or FALSE to ALL, where 0<=p<=1. Anything wrong with my model.
I ran it in skitlearn and I can point out the certain significant paths and extract rules.
For positive values : Is it TRUE to ALL or False to ALL proportion? I am guessing it so FALSE as I mentioned Class=False, but I am not sure.
Is there any method or solution for h20 trees to reveal the sample size of the certain leaf and the [n1,n2] for TRUE and FALSE cases respectively in  a similar way that sklearn provides?
I found in some forums a function def predict_leaf_node_assignment that aims to predict on a dataset and to return the leaf node assignment (only for tree-based models), but it returns no output and I cannot find any example how to implement it.
The bottom line : I'd like to be able to extract the sample size values of the leaf and to extract the specific path to it, implementing [n1,n2] or valid proportions.

I'll appreciate any kind of help and suggestions.
Thank you.

So far I'm not seeing a way to extract training information from the model.  The H2OTree.predictions can/should give you proportion information, but won't give you leaf sample sizes.  For that, you should be able to use predict_leaf_node_assignment, passing your training set in (to wastefully get passed through the model, *shrug*).

predict_leaf_node_assignment should return a dataframe with the leaf assignment for each of your training points.  (The R version appears to support returning either the path or the node id, but the python one doesn't seem to have it.)  You could take this, join to the original frame, and use group and aggregation functions to produce the desired [n1,n2].*

Regarding the output of predictions, see  https://stackoverflow.com/questions/44735518/how-to-reproduce-the-h2o-gbm-class-probability-calculation .  In particular, the default learning rate in H2O's GBM is 0.1, which helps explain your muted results.

Finally, for a little more fun with the the model's tree objects, see https://www.pavel.cool/machine%20learning/h2o-3/h2o-3-tree-api/
and
https://novyden.blogspot.com/2018/12/finally-you-can-plot-h2o-decision-trees.html

*EDIT: For doing the grouping and aggregation:
(I'm more used to pandas than H2O frames, so I'll convert first.  And given that H2O thinks your FALSE class is the main class, maybe those are strings not boolean?)

predicted_leaves_frame = treemodel.predict_leaf_node_assignment(data).as_data_frame()
df = data.as_data_frame()
df['binary_dep_var'] = df['dep_var'].apply(lambda x: 1 if x=='TRUE' else 0)
df['T1'] = predicted_leaves_frame['T1.C1']
print(df.groupby('T1')['binary_dep_var'].agg(['sum','count','mean'])

This should give for each leaf the number of TRUE samples and the total number of samples and the ratio.  If you really need the number of FALSE samples, you could define your own aggregation function or just post-process this new dataframe.


Unequal length observation sequences when training Hidden Markov model I want to train a sequence classifier with Hidden Markov Model. The length of observation sequences is not fixed. I tried some HMM packages such as Matlab's HMM toolbox and Kevin Murphy's library. All of them seem to require the user to specify the size of transition probability matrix and emission probability matrix.

I understand that for a Hidden Markov Model (HMM), the sizes of the transition probability matrix and emission probability matrix are dependent on the number of hidden states and the length of observation sequence.

For example if:

states = ('Rainy','Sunny')
observations = ('walk', 'shop', 'clean')

The number of states is 2, the length of observation is 3. Then,
transition probability would be a 2x2 matrix. Whereas,
emission probability would be a 2x3 matrix.

What if the length of observation sequence is not fixed?

For example:

observation 1 = ('walk', 'shop', 'clean')
observation 2 = ('walk', 'shop', 'clean','eat pizza')
observation 3 = ('walk', 'shop', 'clean','drink beer','eat pizza')
...so on

What's the size of emission probability matrix in this case? Or can I just make the observation sequence the same length by padding with zeros?

No worries, length of observation would help in training of model. longer the length, training for Forward-backward Algo/baum Welch algo would be better. But length of observation is nothing to do with forecasting. read my blog below-

http://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html

transition probabilities would give next hidden state. and from next hidden state you can get next observation.So number of observation would not matter as long as u have trained model( emission and transition probabilities). Also if are intrested to know in detail that how next state would be calculated, follow below article.

http://machinelearningstories.blogspot.com/2017_03_01_archive.html


How do AI's learn to act when the problem space is too big I learn best through experimentation and example. I'm learning about neural networks and have (what I think) is a pretty good understanding of classification and regression and also supervised and unsupervised learning, but I've stumbled upon something I can't quiet figure out;

If I wanted to train an AI to play a complicated game; I'm thinking something like a RTS (eg. Age of Empires, Empire Earth etc.). In these types of games there is typically a number of entities controlled by the player (units, buildings) each with different capabilities. It seem like the problem of that the AI does would be classification (eg. choose that unit, and that action), however since the number of units is a variable how does one handle a classification problem in this way?

The only thing I can think of is multiple networks that do different stages (one for overall strategy, one for controlling this type of unit, one for that type of building etc.); but this seems like I'm making the problem to complicated.

Are there any good example of machine learning/neural networks learning complex games (not specifically RTS, but more complicated the Mario)?

That is a good question and many scientists around the world are asking the same. Well, first a game like Age of Empires is not considered to have a really big solution space, there are not so many things you can do. It's the same in games like Mario Bros. The problem of learning in easy games like Atari games was solve by the guys of DeepMind (here the paper), that was acquired by Google. They used an implementation of Reinforcement Learning with Deep Learning.

Going back to your question. A really big problem is how to imitate the amount of decisions a human being takes every day. Wake up, have breakfast, take a shower, leave your house... All these action need a really high level of intelligence and many actions to develop.

There are many people working on this problem, I'm one of them. I don't know the solution but I can tell you in which way I'm looking. I follow the theories of Marvin Minsky, he is one of the fathers of AI. This book, the Emotion Machine, tells a very good view of the problem. He suggested that the way to create a machine that imitates the human behavior is not by constructing an unified compact theory of artificial intelligence. On the contrary, he argues that our brain contains resources that compete
between each other to satisfy different goals at the same moment. They called this Ways to Think.


Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please? First of sorry for asking a possibly beginner question, but i don't understand pca seems to be the same as feature selection, but when i search online they seems to be talked differently. What people usually say is PCA for reducing dimensionality and feature selection for selecting feature. Didn't those two do the same thing? Reducing number of attributes/features/dimension? Please kindly help me understand. Thank you

PCA reduces dimensionality by generating orthogonal (i.e. uncorrelated) "new" features from the original features. This can be very useful, e.g. when you do linear regression. So principle components "transform" the original features to orthogonal ones. You can see this as some kind of feature engineering. Usually PCA will reduce the number of features which can be very helpful. See Ch. 10.2 in ISL.
Feature selection basically means to choose features which perform well in some model (e.g. have high explanatory power) or exclude features which are noisy. This can be any kind of features (transformed or original features). So feature selection is a very broad term and may be linked to various different methods.