<jupyter_start><jupyter_text>KNN Density Estimation  Q1: implement density estimation using the K-NN method.   Data Preparation Obtain an iid sample of N ≥ 1 points from a univariate normal (Gaussian) distribution (let us call the random variable X) centered at 1 and with variance 2  Density Estimation Now, empirically obtain an estimate of the density from the sample points using the K-NN method, for any value of K, where 1 ≤ K ≤ N. Produce one plot for each of the following cases (each plot should show the following three items: - the N data points (instances or realizations of X) and - the true and estimated densities versus x for a large number – e.g., 1000, 10000– of discrete, linearly-spaced x values): (i) K = N = 1, (ii) K = 2, N = 10, (iii) K = 10, N=10,(iv)K=10,N=1000,(v)K=100,N=1000,(vi)K=N=50,000. Please provide appropriate axis labels and legends. Thus there should be a total of six figures(plots), each figure similar to Fig. 8 on p. 20 of the pdf notes.<jupyter_code>import numpy as np
import math
import matplotlib.pyplot as plt
def density_estimation_knn(N, K, 
                          mean_value=1,variance_val=2, seed_value=12,
                          n_linearly_spaced_discrete_values=1000):
    
    np.random.seed(seed_value)
    # variance = square(standard_deviation)
    # scale is standard deviation
    # hence sqrt(variance_value) = scale/standard_deviation
    gaussian = norm(loc=mean_value, scale=math.sqrt(variance_val))
    
    # input data is X
    input_data = gaussian.rvs(N)
    #input_data = np.random.normal(loc=mean_value, scale=math.sqrt(variance_val), size=N)
    print("Distribution Stats:")
    print("Size: ", len(input_data),
         "X-Min: ", input_data.min(), "X-Max: ", input_data.max())
    
    linearly_spaced_discrete_values = np.linspace(start=-8, 
                                                  stop=8, 
                                                  num=n_linearly_spaced_discrete_values)
    
    print("n_linearly_spaced_discrete_values: ", len(linearly_spaced_discrete_values))
    
    N = len(input_data)
    probabilities = np.zeros_like(linearly_spaced_discrete_values)
    
    fig, axes = plt.subplots(1,1, figsize=(8, 5))
    for index, x_discrete_val in enumerate(linearly_spaced_discrete_values):
        distances = np.abs(input_data - x_discrete_val)
        neighbor_indices_of_sorted_distances = distances.argsort()
        kth_neighbor_distance = neighbor_indices_of_sorted_distances[K-1]
        """
        dV = πd/2/Γ(d2 +1)rd 
        f(x) = K/N * 1/dV
        
        where Γ(·) indicates the gamma function. (Recap: For a positive integer n, 
        Γ(n) = (n − 1)! and Γ(1) = √π, Γ(3) = 1√π, etc.) (Verify: the volume of a
        sphere (d = 3) is 4πr3; the area
         22
         
        When d=1, dV = 2 * rd
        """
        dV = (2 * distances[kth_neighbor_distance])
        probabilities[index] =  (K/N) * 1/dV
        # (K/n) * 1/(2 * dists[neighbour_K])
        
    line1, = axes.plot(linearly_spaced_discrete_values, gaussian.pdf(linearly_spaced_discrete_values))
    line2, = axes.plot(linearly_spaced_discrete_values, probabilities)
    points3, = axes.plot(input_data, np.zeros_like(input_data), 'bo', color='black')
    axes.set_title(f"K-NN density estimation K={K},N={N}")
    axes.set_xlabel("Discrete linearly spaced x values")
    axes.legend([line1, line2, points3], ['True (unknown) density', 'Estimated density', 'Training points'])<jupyter_output><empty_output><jupyter_text>case-1<jupyter_code>density_estimation_knn(N=1,K=1,
                       mean_value=1,variance_val=2, seed_value=123,
                       n_linearly_spaced_discrete_values=1000)

# Note that variance will be 0 for N=1<jupyter_output>Distribution Stats:
Size:  1 X-Min:  -0.535313522914939 X-Max:  -0.535313522914939
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-2:<jupyter_code>density_estimation_knn(N=10,K=2,
                       mean_value=1,variance_val=2, seed_value=144,
                       n_linearly_spaced_discrete_values=1000)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  -0.8364567882597602 X-Max:  3.623439963332321
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-3<jupyter_code>density_estimation_knn(N=10,K=10,
                       mean_value=1,variance_val=2, seed_value=144,
                       n_linearly_spaced_discrete_values=1000)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  -0.8364567882597602 X-Max:  3.623439963332321
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-4<jupyter_code>density_estimation_knn(N=1000,K=10,
                       mean_value=1,variance_val=2, seed_value=144,
                       n_linearly_spaced_discrete_values=1000)<jupyter_output>Distribution Stats:
Size:  1000 X-Min:  -3.627026934137941 X-Max:  5.6367665290810764
n_linearly_spaced_discrete_values:  1000<jupyter_text>case 5<jupyter_code>density_estimation_knn(N=1000,K=100,
                       mean_value=1,variance_val=2, seed_value=144,
                       n_linearly_spaced_discrete_values=1000)<jupyter_output>Distribution Stats:
Size:  1000 X-Min:  -3.627026934137941 X-Max:  5.6367665290810764
n_linearly_spaced_discrete_values:  1000<jupyter_text>case 6<jupyter_code>density_estimation_knn(N=50000,K=50000,
                       mean_value=1,variance_val=2, seed_value=144,
                       n_linearly_spaced_discrete_values=1000)<jupyter_output>Distribution Stats:
Size:  50000 X-Min:  -4.515991233558055 X-Max:  7.109242012764549
n_linearly_spaced_discrete_values:  1000<jupyter_text>Q2: Repeat Q1 (i.e., produce six more plots) by replacing the Gaussian density with the following mixture of two uniform densities: f(x) = 0.4f1(x) + 0.6f2(x)f1(x) = 1 if2≤x≤3;0 otherwise.f2(x)0.2 if15≤x≤20;0 otherwise.<jupyter_code>from scipy import stats
from scipy.integrate import quad

    

class f1x_distribution(stats.rv_continuous):        
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # integrate area of the PDF in range a..b
        #self.scale, _ = quad(f1x_pdf, self.a, self.b)

    def _pdf(self, x):
        #return f1x_pdf(x) / self.scale  # scale PDF so that it integrates to 1 in range a..b 
        #print("x for f1: ", x)
        return  np.where((x <= 3) & (x >=2), 1, 0)

class f2x_distribution(stats.rv_continuous):        
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # integrate area of the PDF in range a..b
        #self.scale, _ = quad(f2x_pdf, self.a, self.b)

    def _pdf(self, x):
        #return f2x_pdf(x) / self.scale  # scale PDF so that it integrates to 1 in range a..b 
        #print("x: ", x)
        return  np.where((x <= 20) & (x >=15), 0.2, 0)



f1x_distribution_object = f1x_distribution(name="f1x_distribution", a=-8,b=8)
f2x_distribution_object = f2x_distribution(name="f2x_distribution",a=-20,b=25)

def get_data_from_mixture_distribution(n_samples):
    """
        Mixture of two uniform densities
        
        Two Step Process, Binomial Distribution
        
        A single sample can be drawn from the mixture density fM in a two-step algorithm (this
            process can be repeated until the desired number of samples have been obtained):
            • Sample an instance of the discrete random variable Z, following Z’s multinomial (binomial)
            distribution. Call the instance z. Then z ∈{1, ··· , c}.
            • Sample an instance of Xz, call it x0, from the density fXz . Return this x0 as a sample
            drawn from the density fM .
        Ref: https://docs.scipy.org/doc/scipy/tutorial/stats.html#building-specific-distributions
        
        Quad integration is required to avoid the runtime errors
    """
    two_distributions = ['f1x', 'f2x']
    output_distribution = []
    for _ in range(n_samples):
        # step -1: sampling from binomial distribution can be achieved by this
        selected_distribution = np.random.choice(two_distributions, p=[0.4,0.6])

        # step -2: Sample an instance of Xz, call it x0, from the density fXz . Return this x0 as a sample
        #drawn from the density fM
        distribution_class_obj = None
        if selected_distribution == "f1x":
            distribution_class_obj = f1x_distribution_object
        else:
            distribution_class_obj = f2x_distribution_object
        # Return this x0 as a sample
        #drawn from the density fM
        curr_selected_sample_from_mixture = distribution_class_obj.rvs(size=1)
        output_distribution.append(curr_selected_sample_from_mixture)
    return np.array(output_distribution)

def get_pdf_from_mixture_distribution(ip_dataset):
    """
        Mixture of two uniform densities
        
        Two Step Process, Binomial Distribution
        
        A single sample can be drawn from the mixture density fM in a two-step algorithm (this
            process can be repeated until the desired number of samples have been obtained):
            • Sample an instance of the discrete random variable Z, following Z’s multinomial (binomial)
            distribution. Call the instance z. Then z ∈{1, ··· , c}.
            • Sample an instance of Xz, call it x0, from the density fXz . Return this x0 as a sample
            drawn from the density fM .
        Ref: https://docs.scipy.org/doc/scipy/tutorial/stats.html#building-specific-distributions
        
        Quad integration is required to avoid the runtime errors
    """
    two_distributions = ['f1x', 'f2x']
    output_pdf = []
    for ip in ip_dataset:
        # step -1: sampling from binomial distribution can be achieved by this
        selected_distribution = np.random.choice(two_distributions, p=[0.4,0.6])

        # step -2: Sample an instance of Xz, call it x0, from the density fXz . Return this x0 as a sample
        #drawn from the density fM
        distribution_class_obj = None
        #print("selected_distribution: ", ip, selected_distribution)
        if selected_distribution == "f1x":
            distribution_class_obj = f1x_distribution_object
        else:
            distribution_class_obj = f2x_distribution_object
        # Return this x0 as a sample
        #drawn from the density fM
        curr_pdf_val_from_mixture = distribution_class_obj.pdf(np.array([ip]))
        #print("selected_distribution: ", ip, selected_distribution, curr_pdf_val_from_mixture)
        output_pdf.append(curr_pdf_val_from_mixture)
    return np.array(output_pdf)
def density_estimation_knn_mixture_density(N, K,seed_value=12,
                          n_linearly_spaced_discrete_values=1000, fig_size=(8, 5)):
    
    np.random.seed(seed_value)
    
    # input data is X
    input_data = get_data_from_mixture_distribution(N)
    print("Distribution Stats:")
    print("Size: ", len(input_data),
         "X-Min: ", input_data.min(), "X-Max: ", input_data.max())
    
    linearly_spaced_discrete_values = np.linspace(start=-20, 
                                                  stop=20, 
                                                  num=n_linearly_spaced_discrete_values)
    
    print("n_linearly_spaced_discrete_values: ", len(linearly_spaced_discrete_values))
    
    N = len(input_data)
    probabilities = np.zeros_like(linearly_spaced_discrete_values)
    
    fig, axes = plt.subplots(1,1, figsize=fig_size)
    for index, x_discrete_val in enumerate(linearly_spaced_discrete_values):
        distances = np.abs(input_data - x_discrete_val)
        neighbor_indices_of_sorted_distances = distances.argsort()
        kth_neighbor_distance = neighbor_indices_of_sorted_distances[K-1]
        """
        dV = πd/2/Γ(d2 +1)rd 
        f(x) = K/N * 1/dV
        
        where Γ(·) indicates the gamma function. (Recap: For a positive integer n, 
        Γ(n) = (n − 1)! and Γ(1) = √π, Γ(3) = 1√π, etc.) (Verify: the volume of a
        sphere (d = 3) is 4πr3; the area
         22
         
        When d=1, dV = 2 * rd
        """
        dV = (2 * distances[kth_neighbor_distance])
        probabilities[index] =  (K/N) * 1/dV
        # (K/n) * 1/(2 * dists[neighbour_K])
        
    true_densites = get_pdf_from_mixture_distribution(linearly_spaced_discrete_values)
    
    line1, = axes.plot(linearly_spaced_discrete_values, true_densites)
    line2, = axes.plot(linearly_spaced_discrete_values, probabilities)
    points3, = axes.plot(input_data, np.zeros_like(input_data), 'bo', color='black')
    axes.set_title(f"K-NN density estimation K={K},N={N}")
    axes.set_xlabel("Discrete linearly spaced x values")
    axes.legend([line1, line2, points3], ['True (unknown) density', 'Estimated density', 'Training points'])<jupyter_output><empty_output><jupyter_text>case-1<jupyter_code>density_estimation_knn_mixture_density(N=1,K=1)<jupyter_output>Distribution Stats:
Size:  1 X-Min:  2.7400496962126786 X-Max:  2.7400496962126786
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-2<jupyter_code>density_estimation_knn_mixture_density(N=10,K=2,fig_size=(10,15))<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.521226026908451 X-Max:  19.263677705691684
n_linearly_spaced_discrete_values:  1000<jupyter_text>Note:         From the above figure, first set of rectangles between 0 and 5 belong to f1(x) whose height is 1.0 and second set of blue rectangles belong to f2(x) whose height is 0.2  case-3<jupyter_code>density_estimation_knn_mixture_density(N=10,K=10, fig_size=(10,15))<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.521226026908451 X-Max:  19.263677705691684
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-4<jupyter_code>density_estimation_knn_mixture_density(N=1000,K=10)<jupyter_output>/Users/reshmapriyamannemuddu/opt/anaconda3/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:1801: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return integrate.quad(self._pdf, _a, x, args=args)[0]<jupyter_text>case-5<jupyter_code>density_estimation_knn_mixture_density(N=1000,K=100)<jupyter_output>/Users/reshmapriyamannemuddu/opt/anaconda3/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:1801: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return integrate.quad(self._pdf, _a, x, args=args)[0]<jupyter_text>case-6<jupyter_code>density_estimation_knn_mixture_density(N=50000,K=50000)<jupyter_output><empty_output><jupyter_text>Q3:Estimate the (mixture-)density f(x) (that was used in Q2) using the kernel method (use Gaussian kernel).  That is, first obtain an iid sample of N ≥ 1 points from the true density f(x), and then obtain, by Gaussian KDE, an estimate of f(x) using those N sample points. Produce plots (each plot showing the data points and the true and the estimated densities), one plot for each of the following sample sizes (note: sample size= N): 1, 2, 5, 10, 50, 100, 1000. A total of seven plots are to be produced for this part.<jupyter_code>from sklearn.neighbors import KernelDensity
def density_estimation_guassian_kernel_mixture_density(N, seed_value=12,
                          n_linearly_spaced_discrete_values=1000):
    
    np.random.seed(seed_value)
    
    # input data is X
    input_data = get_data_from_mixture_distribution(N)
    print("Distribution Stats:")
    print("Size: ", len(input_data),
         "X-Min: ", input_data.min(), "X-Max: ", input_data.max())
    
    linearly_spaced_discrete_values = np.linspace(start=-20, 
                                                  stop=20, 
                                                  num=n_linearly_spaced_discrete_values)
    
    print("n_linearly_spaced_discrete_values: ", len(linearly_spaced_discrete_values))
    
    N = len(input_data)
    
    input_data = input_data.reshape(-1,1)
    fig, axes = plt.subplots(1,1, figsize=(8, 5))
    
    kde = KernelDensity(kernel='gaussian').fit(input_data)
    
    linearly_spaced_discrete_values_reshaped = linearly_spaced_discrete_values.reshape(-1,1)
    # log densites are returned hence np.exp
    density_estimates = np.exp(kde.score_samples(linearly_spaced_discrete_values_reshaped))
    
    true_densities = get_pdf_from_mixture_distribution(linearly_spaced_discrete_values)
    print("density_estimates : Min and Max", density_estimates.min(), density_estimates.max())
        
    line1, = axes.plot(linearly_spaced_discrete_values, true_densities)
    line2, = axes.plot(linearly_spaced_discrete_values, density_estimates)
    points3, = axes.plot(input_data, np.zeros_like(input_data), 'bo', color='black')
    axes.set_title(f"Guassian KDE density estimation N={N}")
    axes.set_xlabel("Discrete linearly spaced x values")
    axes.legend([line1, line2, points3], ['True (unknown) density', 'Estimated density', 'Training points'])<jupyter_output><empty_output><jupyter_text>case-1<jupyter_code>density_estimation_guassian_kernel_mixture_density(1)<jupyter_output>Distribution Stats:
Size:  1 X-Min:  2.7400496962126786 X-Max:  2.7400496962126786
n_linearly_spaced_discrete_values:  1000
density_estimates : Min and Max 2.0508347020524997e-113 0.3989408337396886<jupyter_text>case 2<jupyter_code>density_estimation_guassian_kernel_mixture_density(2)<jupyter_output>Distribution Stats:
Size:  2 X-Min:  2.5337393932921186 X-Max:  2.7400496962126786
n_linearly_spaced_discrete_values:  1000
density_estimates : Min and Max 1.1045970673321401e-111 0.3967853653210849<jupyter_text>case-3<jupyter_code>density_estimation_guassian_kernel_mixture_density(5)<jupyter_output>Distribution Stats:
Size:  5 X-Min:  2.5337393932921186 X-Max:  15.68604660790301
n_linearly_spaced_discrete_values:  1000
density_estimates : Min and Max 4.4190820683119685e-112 0.23641499446583444<jupyter_text>case-4<jupyter_code>density_estimation_guassian_kernel_mixture_density(10)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.521226026908451 X-Max:  19.263677705691684
n_linearly_spaced_discrete_values:  1000
density_estimates : Min and Max 5.538580691619089e-112 0.197273943456541<jupyter_text>case-5<jupyter_code>density_estimation_guassian_kernel_mixture_density(50)<jupyter_output>Distribution Stats:
Size:  50 X-Min:  2.116273021110876 X-Max:  19.975690787099733
n_linearly_spaced_discrete_values:  1000
density_estimates : Min and Max 5.604482417481562e-109 0.13966077989395928<jupyter_text>case-6<jupyter_code>density_estimation_guassian_kernel_mixture_density(100)<jupyter_output>/Users/reshmapriyamannemuddu/opt/anaconda3/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:1801: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return integrate.quad(self._pdf, _a, x, args=args)[0]<jupyter_text>case-7<jupyter_code>density_estimation_guassian_kernel_mixture_density(1000)<jupyter_output>/Users/reshmapriyamannemuddu/opt/anaconda3/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:1801: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  return integrate.quad(self._pdf, _a, x, args=args)[0]<jupyter_text>Q.4 Histogram Method Repeat Q3 by replacing the KDE method by the histogram method. Show a plot of the histogram, separately for three sample sizes 5, 10, 500 with 2, 5, and 20 bins used for each sample size. Thus a total of nine histograms will be produced. A major problem with histograms, however, is that the choice of binning can have a disproportionate effect on the resulting visualization<jupyter_code>def plot_histogram(sample_size:int, n_bins:int, seed_value=123,
                  n_linearly_spaced_discrete_values=1000):
    np.random.seed(seed_value)
    
    # input data is X
    input_data = get_data_from_mixture_distribution(sample_size)
    print("Distribution Stats:")
    print("Size: ", len(input_data),
         "X-Min: ", input_data.min(), "X-Max: ", input_data.max())
    
    bins = np.linspace(-30, 30, n_bins)
    
    #print(bins)
    linearly_spaced_discrete_values = np.linspace(start=-20, 
                                                  stop=20, 
                                                  num=n_linearly_spaced_discrete_values)
    
    print("n_linearly_spaced_discrete_values: ", len(linearly_spaced_discrete_values))
    
    N = len(input_data)
    #print(input_data)
    fig, ax = plt.subplots(1, 1, sharex=True, sharey=True)
    fig.subplots_adjust(hspace=0.05, wspace=0.05)
    ax.hist(input_data[:, 0], bins=bins, fc="#AAAAFF")
    ax.set_title(f"Histogram plot: N={sample_size}, N_bins={n_bins}")
    #ax.text(-3.5, 0.31, "Histogram")<jupyter_output><empty_output><jupyter_text>case-1: N = 5, bins=2<jupyter_code>plot_histogram(5,2)<jupyter_output>Distribution Stats:
Size:  5 X-Min:  2.5513147648848786 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-2: N = 5, bins=5<jupyter_code>plot_histogram(5,5)<jupyter_output>Distribution Stats:
Size:  5 X-Min:  2.5513147648848786 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-3: N = 5, bins=20<jupyter_code>plot_histogram(5,20)<jupyter_output>Distribution Stats:
Size:  5 X-Min:  2.5513147648848786 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-4: N = 10, bins=2<jupyter_code>plot_histogram(10,2)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.1754517611573134 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-5: N = 10, bins=5<jupyter_code>plot_histogram(10,5)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.1754517611573134 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-6: N = 10, bins=20<jupyter_code>plot_histogram(10,20)<jupyter_output>Distribution Stats:
Size:  10 X-Min:  2.1754517611573134 X-Max:  18.424148694796827
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-7: N = 500, bins=2<jupyter_code>plot_histogram(500,2)<jupyter_output>Distribution Stats:
Size:  500 X-Min:  2.021761438059864 X-Max:  19.980431620429787
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-8: N = 500, bins=5<jupyter_code>plot_histogram(500,5)<jupyter_output>Distribution Stats:
Size:  500 X-Min:  2.021761438059864 X-Max:  19.980431620429787
n_linearly_spaced_discrete_values:  1000<jupyter_text>case-9: N = 500, bins=20<jupyter_code>plot_histogram(500,20)<jupyter_output>Distribution Stats:
Size:  500 X-Min:  2.021761438059864 X-Max:  19.980431620429787
n_linearly_spaced_discrete_values:  1000
StarcoderdataJupyterStructuredCleanDedup