<jupyter_start><jupyter_text>run NOPTMAX=0 to get residuals, then load and reweight like we were doing for Monte Carlo. But, also set up standard deviation column for noise on observations for the ensembles  NOTE: Make sure  `run_ensemble` is set appropriately - If `run_ensemble` is set to `True`, local runs are performed. If `run_ensemble` set to `False`results from the journal article are used.<jupyter_code>run_ensemble=False
if run_ensemble==True:
    input_dir = '../noptmax0_testing/' # read the pst file from here and get residuals
else:
    input_dir = '../output/noptmax0/' # read the pst file from here and get residuals
output_dir = '../run_data'  # write out the new run file here
pst_root = 'never_iES_noptmax0' # the PST root we are reading in from

ies_noise_obs_file = 'never_ies_0.noise_lt_obs.pst' # Name of the PST file to write out as the start of iES<jupyter_output><empty_output><jupyter_text>read in the PEST control file<jupyter_code>pst = pyemu.Pst(os.path.join(input_dir,'{}.pst'.format(pst_root)))<jupyter_output><empty_output><jupyter_text>make a quick pie chart showing the current distribution of the observation groups in the objective function<jupyter_code>pst.plot(kind='phi_pie')<jupyter_output><empty_output><jupyter_text>we can use `pyemu` functionality to assign new weights that adjust and honor whatever balance we seek<jupyter_code>new_proportions = pst.phi_components.copy()
new_proportions<jupyter_output><empty_output><jupyter_text>here we assign proportions (that sum to 1.0) to the various groups. We want to retain the same total Phi so we multiply our proportions by the total Phi<jupyter_code>new_proportions['flux'] = 0.3*pst.phi
new_proportions['head'] = 0.2*pst.phi
new_proportions['land_surface'] = 0.5*pst.phi
new_proportions['budget'] = 0
new_proportions<jupyter_output><empty_output><jupyter_text>using the `phi_components` dictionary, making a copy of it, and reassigning values, we can update the PST object using the `adjust_weights` function<jupyter_code>pst.adjust_weights(obsgrp_dict=new_proportions)
pst.observation_data
pst.plot(kind='phi_pie')
# set some values for pestpp-ies
if run_ensemble == True:
    pst.pestpp_options["ies_num_reals"] = 20  
else:
    pst.pestpp_options["ies_num_reals"] = 500 
    
pst.pestpp_options["ies_bad_phi_sigma"] = 2.5
pst.pestpp_options["overdue_giveup_fac"] = 4
pst.pestpp_options["ies_save_rescov"] = True
pst.pestpp_options["ies_no_noise"] = True
pst.pestpp_options["ies_drop_conflicts"] = False
pst.pestpp_options["ies_pdc_sigma_distance"] = 3.0
pst.pestpp_options['ies_autoadaloc']=True
pst.pestpp_options['ies_num_threads']=4
pst.pestpp_options['ies_lambda_mults']=(0.1,1.0,10.0,100.0)
pst.pestpp_options['lambda_scale_fac'] = (0.75,0.9,1.0,1.1)
pst.pestpp_options['ies_subset_size']=10

# set SVD for some regularization
pst.svd_data.maxsing = 250
# check number of realizations
pst.pestpp_options["ies_num_reals"]<jupyter_output><empty_output><jupyter_text>we want to use the observations in the `land_surface` group as inequality observations (e.g. unweighted unless needed to enforce "less than" condition). By naming the groups starting with "less_than" PESTPP-IES will interpret them as such<jupyter_code>pst.observation_data.loc[pst.observation_data.obgnme=='land_surface', 'obgnme'] = 'less_than_land_surface' # rename for less than inequality
pst.observation_data<jupyter_output><empty_output><jupyter_text>Add standard deviation column for observations to control noise realizations  make a "standard_deviation" column in the pst observation_data dataframe. This will inform PESTPP-IES to use these value for sampling in generating the observation noise on the ensembles. Start out with a value of -9999 that we can use as a test to make sure we set a value for every observation group.<jupyter_code>obs = pst.observation_data
obs['standard_deviation'] = -9999<jupyter_output><empty_output><jupyter_text>now we set some subjective standard deviation values that we assert are more appropriate than the straight up weights.<jupyter_code>obs.loc[obs.obsnme=='q_1436500'].obsval/(3.33*3)
obs.loc[obs.obsnme=='q_1366650'].obsval/30
obs.loc[obs.obsnme=='q_1436500', 'standard_deviation'] = obs.loc[obs.obsnme=='q_1436500'].obsval/(3.33*3)
obs.loc[obs.obsnme=='q_1366650', 'standard_deviation'] = obs.loc[obs.obsnme=='q_1366650'].obsval/(10.*3)
obs.loc[obs.obgnme=='head', 'standard_deviation'] =  5/3  #  5 is the range, div by 3 for the range
obs.loc[obs.obgnme=='less_than_land_surface', 'standard_deviation'] = 10/3
obs.loc[obs.obgnme=='budget', 'standard_deviation'] = 9999<jupyter_output><empty_output><jupyter_text>use an `assert` statment to make sure we set obs to have a standard deviation not -9999<jupyter_code>assert pst.observation_data.standard_deviation.min()>0
pst.observation_data<jupyter_output><empty_output><jupyter_text>Write out an updated parameter summary XLS file<jupyter_code>parsum = pst.write_par_summary_table('../report_materials/initial_iES_parsum.xlsx', report_in_linear_space=True)
parsum<jupyter_output><empty_output><jupyter_text>Now set `NOPTMAX=10` and write out the PEST control file. Note - to use the standard deviation column requires writing out in the new `version=2` format. In this format, the parameter and observation data sections are written to external files. This is not backward compatible with older version of PEST, but enables this new capability. Filenames for the external files will be identified in the PST file.<jupyter_code>pst.control_data.noptmax = 10
pst.pestpp_options["ies_no_noise"] = False
pst.write(os.path.join(output_dir, ies_noise_obs_file), version=2)<jupyter_output><empty_output><jupyter_text>If `run_ensemble=True` the cell below will run a local `never_ies_0.noise_lt_obs.pst` iES history-matching run* simillar process as in notebooks 1.0, 1.3, and 2.0* using the number of realizations per iteration specified by `pst.pestpp_options["ies_num_reals"]`* will run in parallel locally using the number of cores specified below by `num_workers`* creates a new directory called `"../master_ies/"` which is a copy of run_data * while running function generates worker directories that are removed when run is complete* results moved to `"../run_data/"`<jupyter_code>if run_ensemble==True:
    #  set some variables for starting a group of PEST++ workers on the local machine
    # MAKE SURE THAT PESTPP-IES and MF6 executables are in your system path or are in '../run_data'
    
    num_workers = 5  # number of local workers -- VERY IMPORTANT, DO NOT MAKE TOO BIG
    if sys.platform == 'win32':
        pst_exe = 'pestpp-ies.exe'
    else:
        pst_exe = 'pestpp-ies'
        
    template_ws =  '../run_data'  #  template_directory
    m_d = '../master_ies'
    
    pyemu.os_utils.start_workers(worker_dir=template_ws, 
                             exe_rel_path=pst_exe,
                             pst_rel_path=ies_noise_obs_file,
                             num_workers=num_workers, 
                             master_dir=m_d
                            )
if run_ensemble==True:
    #  move results into run_data and clean up 
    
    move_result_files = glob.glob(os.path.join(m_d, 'prior_mc_wide*'))
    move_result_files = [f for f in move_result_files if 'pst' not in f]
    
    [shutil.copy(os.path.join(m_d, file), output_dir) for file in move_result_files]
    
    #  Remove master dir.
    shutil.rmtree(m_d)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup