<jupyter_start><jupyter_text>Conditions of Language Dynamics, week 5 Syntactic constraints lab  This simulation features a replication ofCulbertson & Smolensky (2012)'s model of learning biases for Greenberg’s Universal 18.|.       | N-Adj | Adj-N ||---------|-------|-------||**Num-N**|  17%  | 27%   ||**N-Num**|  52%  | 4%    |The learning biases were previously experimentally explored in Culbertson, Smolensky & Legendre (2012). In this paper, the authors tested three hypotheses:1. Learning involves tracking input statistics2. Learners regularise variation3. Learners regularise but only orders that are easier to learnThe model in Culbertson & Smolensky (2012) reformulates these hypotheses in terms of [Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference) (which takes into account prior biases as well as the input likelihood) to find the best fit for the experimental data:1. Input likelihood given a flat/uninformative prior (no bias)2. Input likelihood given a regularisation prior (regularisation bias)3. Input likelihood given a regularisation prior and a harmonic prior (regularisation and harmony biases)  OverviewThe model uses Bayesian inference to predict the types of grammars learners will infer given (1) a set of counts of Adj-N, N-Adj, Num-N and N-Num utterances, and (2) their prior expectations in terms of variation (i.e., regularity) and ordering combinations (harmonicity). The first part of code imports all the libraries taht you will need to operate with the binomial and beta distributions, to do logs and exponentials, and to generate random numbers of various kinds.<jupyter_code>import random
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg', 'pdf')

from scipy.stats import beta, binom
from math import log, exp
from scipy.special import logsumexp
from numpy.random import choice, binomial<jupyter_output><empty_output><jupyter_text>The input dataThe input consists of counts of [Adj-N, N-Adj, Num-N, N-Num] with 40 total per modifier type. Whether the input is skewed toward pre- or post-nominal modifiers for each phrase type depends on the condition, as shown in the table.|condition   | training counts||------------|---------------||1 | [28,12,28,12] ||2 | [12,28,12,28] ||3 | [12,28,28,12] ||4 | [28,12,12,28] |We’ll use a grid of probabilities to describe the space of possible generating grammars (i.e. hypotheses). We’ll need to use the grid twice–once for the probability of Adj-N (vs N-Adj), and again for the probability of Num-N (vs. N-Num).<jupyter_code># Training data is a list of lists of counts for each condition:
# (The list slots for each pattern correspond to [Adj-N, N-Adj, Num-N, N-Num]) 
#
#                     pattern 1        pattern 2          pattern 3        pattern 4
#                  (Adj-N,Num-N)     (N-Adj,N-Num)     (N-Adj,Num-N)     (Adj-N,N-Num)
training_data = [[28, 12, 28, 12], [12, 28, 12, 28], [12, 28, 28, 12], [28, 12, 12, 28]]

grid_granularity = 100 
possible_p = []
for i in range(1, grid_granularity):
    possible_p.append(i / (grid_granularity + 0.))<jupyter_output><empty_output><jupyter_text>*Check out `training_data`. How would you access just the counts for condition 2? Also take a look at the grid. What would a hypothesis of 0.99 mean?*  LikelihoodThe likelihood here is calculated using the binomial distribution. We are interested in the likelihhod of the individual's inferred grammar (i.e., NP ordering rules) given the proportion of Adj-N in Adj trials, and the proportion of Num-N out in Num trials. ***Wait a second? Binomial distribution?- Coin toss example: How many head out of total tosses? Is it a fair coin or a biased coin?        binomial(k = 5 heads | p = 0.5, n = 10 tosses) = 0.246                binomial(k = 5 heads | p = 0.9, n = 10 tosses) = 0.001  Take a second to check the above [here](https://www.socscistatistics.com/tests/binomial/default2.aspx) and [here](https://shiny.rit.albany.edu/stat/binomial/).- Adj, N ordering example: How many Adj-N out of the total of Adj utterances? Does the grammar tend to use Adj-N?        binomial(k = 28 Adj-N | p = 0.5, n = 40 Adj) = 0.005         binomial(k = 28 Adj-N | p = 0.7, n = 40 Adj) = 0.137***The function below calculates the log likelihood of the data, (where the data are counts representing number of Adj-N out of total Adj instances and number of Num-N out of total Num instances) given point probability of Adj-N, and Num-N.<jupyter_code>def U18_likelihood(data, p_AdjN, p_NumN):
    loglikelihood = []
    loglikelihood_AdjN = binom.logpmf(data[0], data[0] + data[1], p_AdjN)
    loglikelihood_NumN = binom.logpmf(data[2], data[2] + data[3], p_NumN)
    loglikelihood = loglikelihood_AdjN + loglikelihood_NumN #summing logs = multiplying non-log
    return loglikelihood<jupyter_output><empty_output><jupyter_text>*Check out U18_likelihood function. It returns log probabilities. Find the line of code in the function that would allow you to calculate the log likelihood of getting 28 Adj-N counts out of a total of 40 when the underlying probability of Adj-N according to the grammar is 0.7 (versus 0.3 of N-Adj). Calculate it! Why is the result different from the one above? How would you obtain the same result?*  PriorThe prior in this model has two parts. One part is regularisation as encoded by the parameters of the beta distribution. The two parameters of the beta distribution are (annoyingly) called alpha and beta. Here we want two separate asymmetrical beta distributions favoring either probabilities close to one or close to zero, but not both (i.e., we only expect one-way regularisation). The second part of the prior is harmony. This is the part pf the prior which favors particular combinations of orders (i.e., Adj-N with Num-N); in order to get it we need to promote or penalize particular parts of the two dimensional grammar space. We can do this by defining four “components" using different combinations of alpha and beta. The graph below shows beta distributions with (alpha=10, beta=10), (alpha=15, beta=3), (alpha=15, beta=0.1) for Adj-N, and the reverse for Num-N:![alt text](https://www.morphdomains.uzh.ch/CarmenSaldana/CLD/labs/fig2.jpeg "grammar space")![alt text](https://www.morphdomains.uzh.ch/CarmenSaldana/CLD/labs/fig3.jpeg "beta distributions")Given these four components, we calculate the prior probability of any grammar. This will actually be the sum of its probability given the beta distributions governing p(Adj-N) and p(Num-N) for each component, weighted by the probability of that component. The latter is determined by gamma (g). If the probability of a given component (e.g., g[4], which corresponds to component 4) is low, then pattern 4 grammars very likely to be generated by it will have a low prior probability.The function below calculates the log prior probability of a given p_AdjN and p_NumN given a set of parameters. This is a sum over the probabilities given by each mixture component. Parameters are: g (set of four mixture weights), a, b (Beta shape parameters which can function as alpha or beta).<jupyter_code>def U18_prior(g, a, b, p_AdjN, p_NumN):
    pattern1_component = [a, b, a, b] # higher prob for Adj-N, Num-N
    pattern2_component = [b, a, b, a] # higher prob for N-Adj, N-Num
    pattern3_component = [b, a, a, b] # higher prob for N-Adj, Num-N
    pattern4_component = [a, b, b, a] # higher prob for Adj-N, N-Num
    components = [pattern1_component, pattern2_component, 
                  pattern3_component, pattern4_component]
    logprior = []
    for i in range(0,4): # loop over all four components
        logprior_i_Adj = beta.logpdf(p_AdjN, components[i][0], components[i][1])
        logprior_i_Num = beta.logpdf(p_NumN, components[i][2], components[i][3])
        logprior_i = logprior_i_Adj + logprior_i_Num
        logprior.append(logprior_i)
    # a+b+... in log space = log(exp(a)+exp(b)+...)
    logprior = log((g[0] * exp(logprior[0])) + 
                   (g[1] * exp(logprior[1])) + 
                   (g[2] * exp(logprior[2])) + 
                   (g[3] * exp(logprior[3])))
    return logprior<jupyter_output><empty_output><jupyter_text>*Take a look at the four components in the prior. Can you see why they are defined by those combinations of alpha and beta?**Each component - a combination of two beta distributions - can in principle generate any grammar, that is any pair [p(Adj-N), p(Num-N)]. Which component assigns the highest probability to the pair [0.8,0.8]? How about [0.4,0.9]?*  The optimal prior parametersThe parameters of the prior are alpha, beta, gamma. The first two (called a, and b in the code) encode the regularization bias. The higher a is relative to b, the more regularization. The third, g is a set of weights for the four components, summing to 1. Culbertson & Smolensky (2012) searched through these parameters to find the ones that fit their Culbertson, Smolensky & Legendre (2012) experimental data (shown in the plot below) best.![alt text](https://www.morphdomains.uzh.ch/CarmenSaldana/CLD/labs/fig4.jpeg "experimental data")The best fit parameters were:<jupyter_code>#                  gamma                         a      b
fit_parameters = [[0.6293, 0.3706, 0.0001, 0], 16.5,  0.001]<jupyter_output><empty_output><jupyter_text>Prior probability of patterns given the fit parameters:![](https://www.morphdomains.uzh.ch/CarmenSaldana/CLD/labs/priors_CS12.png)  PosteriorThe posterior distribution over grammars can now be calculated using the likelihood and the prior with the fit parameters. This function calculates the log posterior probability of a set of counts for all possible p_AdjN, p_NumN combinations, given prior parameters g, a, b.<jupyter_code>def U18_posterior(g, a, b, data):
    posterior = [] 
    for p_a in range(len(possible_p)):
        for p_n in range(len(possible_p)):
            lik_i = U18_likelihood(data, possible_p[p_a], possible_p[p_n])
            prior_i = U18_prior(g, a, b, possible_p[p_a], possible_p[p_n]) 
            posterior.append(lik_i + prior_i)  
    return posterior<jupyter_output><empty_output><jupyter_text>*Given the fit parameters, will the model predict a weak or strong regularization bias? Will a set of counts generated from a grammar like [0.8,0.2] (for [p(Adj-N), p(Num-N)])  have a high, medium or low posterior probability?*  Plot the resultsThe model can now be used to generate predicted learning outcomes; given some input data, and the prior parameters, we can generate sampled grammars and plot them in the two-dimensional grammar space we’ve be using. The figure belows shows this as reported in the original Culbertson & Smolensky (2012) paper. ![alt text](https://www.morphdomains.uzh.ch/CarmenSaldana/CLD/labs/fig5.jpeg "culbertson & smolensky results")You can use the `U18_roulette_wheel` function to reproduce this. This function samples pairs of p(Adj-N), p(Num-N) probabilities, and it lets you specify how many such samples you want. You’ll notice there’s also a normalisation function `normalize_log_distribution` which takes log probabilities output by the posterior and makes them a proper (non-log) probability distribution for plotting.<jupyter_code>def normalize_log_distribution(distribution):
    exp_dist = []
    for logp in distribution:
        exp_dist.append(exp(logp))
    norm_dist = []
    for p in exp_dist:
        norm_dist.append(p / sum(exp_dist))
    return norm_dist


def U18_roulette_wheel(g, a, b, data, num_samps):
    post = U18_posterior(g, a, b, data) # calculate posterior given training data and prior parameters
    post = normalize_log_distribution(post)
    # make a grid of all possible p_AdjN, p_NumN combinations at the granularity specified
    grid = []
    for p_a in range(len(possible_p)):
        for p_n in range(len(possible_p)):
            grid.append([possible_p[p_a], possible_p[p_n]])
    # samples some grammars!
    grammars = []
    for i in range(num_samps):
        r = choice(a=range(len(grid)), p=post)   # choose an index from the grid according to it's posterior probability  
        grammars.append(grid[r])
    return grammars<jupyter_output><empty_output><jupyter_text>Now we have a function to plot the results as a scatter plot.<jupyter_code>def plot_grammars(g_1, g_2, g_3, g_4):
    
    plt.figure(figsize=(6, 6))
    for g in g_1:
        plt.scatter([g[0]], [g[1]], color='red', alpha=0.5)

    for g in g_2:
        plt.scatter([g[0]], [g[1]], color='blue', alpha=0.5)

    for g in g_3:
        plt.scatter([g[0]], [g[1]], color='orange', alpha=0.5)

    for g in g_4:
        plt.scatter([g[0]], [g[1]], color='purple', alpha=0.5)

    plt.xlabel("P(Adj-N)")
    plt.ylabel("P(Num-N)")
    plt.xlim(0, 1)
    plt.ylim(0, 1)
    plt.grid()<jupyter_output><empty_output><jupyter_text>Questions1. Generate samples `g_1, g_2, g_3, g_4` using function calls like:```pythong_1 = U18_roulette_wheel(fit_parameters[0], fit_parameters[1], fit_parameters[2], training_data[0], 100)```Then call the plotting function to plot them. Does your plot look like figure above from the paper? (N.B. This might take a while to run.)<jupyter_code>g_1 =  U18_roulette_wheel(fit_parameters[0], fit_parameters[1], fit_parameters[2], training_data[0], 100)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup