3. Experiment settings and modifications
After creating a case (see Experiments) and running the case.setup script the case folder is created including a number of different files and folders containing information and settings for your case. The case folder contains:
- README.case
File detailing your create_newcase usage. This is a good place for you to keep track of run-time problems and changes. The file contains information about e.g. how the case was created, compset details, grid information and which branch, git commit and model code were used for case creation.
- CaseStatus
File containing a list of operations done in the current case.
- Buildconf
Directory containing scripts to generate component namelists and component and utility libraries (e.g., PIO, MCT). You should never have to edit the contents of this directory
- SourceMods
Directory where you can place modified source code. In SourceMods there are sub-folders for the different models; cam, clm, cice, blom, mosart and so on. If you want to change the code or add subroutines, you place copies of the fortran files here.
- user namelists
namelists for the different models where you can change parameters and model settings, add output variables. If you use usermods when creating your case, any pre-defined user namelists from the usermods will appear immediately when running the create_newcase script. Otherwise, empty user namelists will be created with running the case.setup script. There is one namelist for each model component (i.e. user_nl_cam, user_nl_cice, user_nl_clm, user_nl_blom, user_nl_cpl). Use the user namelists to change the contents of the full namelists in the CaseDocs folder (see below).
- CaseDocs
here you find the namelists containing all the inputfiles and parameters used. These files will be modified after rebuild. The details of parameter values and input files are listed in the
<component>_in
files. You should never have to edit the contents of this directory. If you wish to make changes to the<component>_in
files, you change theuser_nl_<component>
and rebuild.- LockedFiles
Directory that holds copies of files that should not be changed. You should never edit the contents of this directory
- Tools
Directory containing support utility scripts.
In addition, the case folder contains different environment files (env_*.xml files) that contain environment settings such as the length of your run, the run type, what restart files to use, etc. Modifications to these files can be made directly using your favorite editor, or using the xmlchange script. Note not all files can be edited at any time, for instance build environment variables should not be edited after a build is invoked (or you have to re-build the model). Check the header of the xml files for guidelines.
Below we review the purpose and some of the most important settings for the environment files you will modify most often:
env_run.xml
: Used to set different configuration details for the experiment such as run type and length.env_batch.xml
: Used to set the arguments for the batch job commands.env_mach_pes.xml
: Used to set the machine-specific processor layout for the different model components
We will not discuss the following files as modifications to these files are not usually required:
env_mach_specific.xml
: Used to set a number of machine-specific environment variables for building and/or running set in <noresm-base>/cime/config/cesm/machines/config_machines.xml.env_build.xml
: Used to set various variables needed when building the model. Should not be modified once the model has been build.env_case.xml
: Used to set case-specific variables (e.g. model components, model and case root directories) and cannot be modified after a case has been created. To make changes, your need to re-run ./create_newcase.sh in <noresm-base>/cime/scripts/ with different options.env_archive.xml
: Sets variables needed for the short-term archiving job.
3.1. Run environment
The file
env_run.xml
sets the configuration details for the experiment such as the run type, the length of the run, how often restart files should be produced, output of coupler diagnostics, and short-term and long-term restart file archiving.
3.1.1. Some common configuration settings
RUN_TYPE
:startup: In a startup run (the default), all components are initialized using baseline states. These baseline states are set independently by each component and can include the use of restart files, initial files, external observed data files, or internal initialization (i.e., a cold start). If you want to initialize the model using non-default files, you can do this in the user_nl_<component> files.
branch: In a branch run, all components are initialized using a consistent set of restart files from a previous run (determined by the RUN_REFCASE and RUN_REFDATE variables in env_run.xml). The case name is generally changed for a branch run, although it does not have to be. In a branch run, setting RUN_STARTDATE is ignored because the model components obtain the start date from their restart data sets. Therefore, the start date cannot be changed for a branch run. RUN_REFCASE and RUN_REFDATE are required for branch runs. To set up a branch run, locate the restart tar file or restart directory for RUN_REFCASE and RUN_REFDATE from a previous run, then place those files in the RUNDIR directory.
hybrid: Not as strict as branch. In a hybrid run the model is initialized as a startup, BUT uses initialization data sets from a previous case. This is somewhat analogous to a branch run with relaxed restart constraints. A hybrid run allows users to bring together combinations of initial/restart files from a previous case (specified by RUN_REFCASE) at a given model output date (specified by RUN_REFDATE). Unlike a branch run, the starting date of a hybrid run (specified by RUN_STARTDATE) can be modified relative to the reference case.
RUN_REFCASE
:Reference data used for hybrid or branch runs. The name of the reference simulation the model components are initialized from. The restart and rpointer files should be copied to the run directory before the job is submitted.
RUN_REFDATE
:The reference date for branch or hybrid runs. The reference date must match the date of the restart files from the simulation set in
RUN_REFCASE
RUN_STARTDATE
:Set the date (of your own wish) for the beginning of the simulation. Only used for hybrid runs.
STOP_OPTION
:Sets the run length along with
STOP_N
. Can choose between e.g.: none, never, nstep, nhours, ndays, nday, nmonths, nyears, date.
STOP_N
:Provides a numerical count for
STOP_OPTION
. E.g. ifSTOP_OPTION
is set to years andSTOP_N
set to 20, the model will run for 20 years.
REST_OPTION
:Sets the frequency of model restart files output (same options as
STOP_OPTION
)
REST_N
:Provides a numerical count for
REST_OPTION
. E.g. ifREST_OPTION
is set to years andSTOP_N
set to 5, the model will produce restart files every 5 years.
CONTINUE_RUN
:Needs to be
FALSE
when you first begin the run. When you successfully run and get restart files (if the model crashes after the restart files are produced you can setCONTINUE_RUN
toTRUE
as well), you will need to changeCONTINUE_RUN
toTRUE
for the remainder of your simulation.
RESUBMIT
:If
RESUBMIT
is greater than 0, then case will automatically resubmit. Enables the model to automatically resubmit a new run. This is very useful for long simulations. E.g.RESUBMIT
is set to 2 and the simulation length is set to 20 years (STOP_OPTION
is years andSTOP_N
is 20), the total length of the simulation will be 60 years.
RESUBMIT_SETS_CONTINUE_RUN
:Needs to be
TRUE
(default) for theRESUBMIT
flag to causes a resubmisson of the case
DOUT_S_SAVE_INTERIM_RESTART_FILES
:Set to
TRUE
to archive all restart files that are produced or toFALSE
to only archive restart files that are produced at the end of the run. Default isTRUE
.
3.1.2. NorESM2-specific configuration settings
OCN_FLUX_SCHEME=1
In the coupled NorESM2 simulations, the flux parameterization used for the transfer of heat, moisture and momentum between the ocean and atmosphere is the so-called COARE flux parameterization. This choice is activated by OCN_FLUX_SCHEME=1 in env_run.xml, and ends up in the driver_in namelist as flux_scheme=1. This parameterization is different from the standard flux-parameterization used in CESM, which is activated by OCN_FLUX_SCHEME=0. In env_run.xml:
<entry id="OCN_FLUX_SCHEME" value="1"> <type>integer</type> <valid_values>0,1,2</valid_values> <desc> Default is false and true for N* cpmpsets, controls value of FLUX_SCHEME </desc> </entry> </group>
COSZ_AVG=.true.
A feature of the coupled NorESM2 simulations, i.e., taking into account the fact that the solar zenith angle used for the calculation of the surface albedo changes over the atmospheric model time step of 30 minutes. In env_run.xml:
<entry id="COSZ_AVG" value=".true."> <type>char</type> <valid_values>.true.,.false.</valid_values> <desc> Default is false and true for N* cpmpsets, controls value of ALB_COSZ_AVG </desc> </entry>
3.1.3. Setting up a hybrid simulation
Step by step guide for setting up an hybrid (restart) simulation.
When the case is created and compiled, edit env_run.xml
. Below is an example for restart with CMIP6 historical initial conditions:
<entry id="RUN_TYPE" value="hybrid">
<entry id="RUN_REFDIR" value="<full-path-to-restart-file-directory>"> # path to restarts
<entry id="RUN_REFCASE" value="NHISTfrc2_f09_tn14_20191025"> # experiment name for restart files
<entry id="RUN_REFDATE" value="2015-01-01"> # date of restart files
<entry id="RUN_STARTDATE" value="2015-01-01"> # date in simulation
<entry id="GET_REFCASE" value="TRUE"> # get refcase from outside rundir
If it is not possible to link directly to restarts, copy the restart files and rpointer files to the run directory. In this case it is not necessary to set RUN_REFDIR
and GET_REFCASE
can be set to FALSE
, e.g.:
<entry id="RUN_TYPE" value="hybrid">
<entry id="RUN_REFCASE" value="NHISTfrc2_f09_tn14_20191025"> # Experiment name for restart files
<entry id="RUN_REFDATE" value="2015-01-01"> # date of restart files
<entry id="RUN_STARTDATE" value="2015-01-01"> # date in simulation
<entry id="GET_REFCASE" value="FALSE"> # get refcase from outside rundir
3.2. Batch job environment
The file
env_batch.xml
sets the arguments to the batch job commands. There are two jobs; one for running the model (case.run
) and one for moving the files from the RUNDIR
to the archive directory (case.st_archive
). The archiving is usually very fast (less than one hour), but for very large jobs (high resolution or large output) it can take several hours.
Some of the most commonly modified configuration settings are those related to the walltime for the two jobs.
3.3. Machine-specific environment
The file
env_mach_pes.xml
sets the component machine-specific processor layout. The settings are critical to a well-load-balanced simulation. Here you set the number of cpus or tasks (NTASKS
) used for each model component and the coupler (usually land + ice + rof (river run off) = atm = coupler). An example of the NTASKS-settings from an env_mach_pes.xml
file could be:
<entry id="NTASKS">
<type>integer</type>
<values>
<value compclass="ATM">768</value>
<value compclass="CPL">768</value>
<value compclass="OCN">186</value>
<value compclass="WAV">300</value>
<value compclass="GLC">768</value>
<value compclass="ICE">504</value>
<value compclass="ROF">8</value>
<value compclass="LND">256</value>
<value compclass="ESP">1</value>
</values>
<desc>number of tasks for each component</desc>
</entry>
Note that positive values are used for setting the number of tasks whereas negative values can be used to set the number of nodes instead.
The env_mach_pes.xml
file will usually be created with default values based on the machine you specify (with the –mach option) when you create the case with the create_newcase script.
For more information, see description in the header for the env_mach_pex.xml
file.
3.4. User namelists
3.4.1. Output frequency
In NorESM a number of monthly output variables are automatically written to the output/history files (see Standard output). Output variables and/or output frequencies (daily, 6-hourly, etc) that are not outputted by default can be added using the user namelists.
For instance, if you edit user_nl_cam and add the following lines at the end of the file:
avgflag_pertape = ’A’,’I’
nhtfrq = 0 ,-6
mfilt = 1 , 30
ndens = 2 , 2
fincl1 = ’FSN200’,’FSN200C’,’FLN200’,
’FLN200C’,’QFLX’,’PRECTMX:X’,’TREFMXAV:X’,’TREFMNAV:M’,
’TSMN:M’,’TSMX:X’
fincl2 = ’T’,’Z3’,’U’,’V’,’PSL’,’PS’,’TS’,’PHIS’
avgflag_pertape
Sets the averaging flag for all variables in a particular history file series. Options are: Average (A), Instantaneous (I), Maximum (X), and Minimum (M). The default behavior is the use the same averaging flag for all variables in each particular history file series, but this can be overridden (more information below).
nhtfrq
Array of write frequencies for each history files series.
nhtfrq = 0, the file will be a monthly average. Only the first file series may be a monthly average.
nhtfrq > 0, frequency is input as number of time steps.
nhtfrq < 0, frequency is input as number of hours.
mfilt
Array of number of time samples to write to each history file series (a time sample is the history output from a given time step)
nden
Array specifying output format for each history file series. Valid values are 1 or 2. ‘1’ implies output real values are 8-byte and ‘2’ implies output real values are 4-byte. Default: 2
fincl1
List of fields to add to the primary history file. Note in the above example the default averaging behavior for the file is overridden for some variables by adding a “:” followed by an averaging option.
fincl2
List of fields to add to the auxiliary history file.
For a detailed description of NorESM2 output, please see Output and standard results
3.4.2. Parameter settings
If you need to change some variable values or activate/deactivate flags, that can also be done in user_nl_<component>
. The syntax is:
&namelist_group
namelist_var = new_namelist_value
E.g for a quadrupling of the atmospheric CO2 concentration
&chem_surfvals_nl
co2vmr = 1137.28e-6
Note that BLOM uses a different syntax than the rest. In user_nl_blom:
set BDMC2 = .15
set NIWGF = .5
set SRF_ANO3 = 0, 2, 2
you need to include set before the name of the variable and it does not matter what namelist group the variable belong. Input entries in user_nl_blom
are case-insensitive (for model developers: variable names defined in the BLOM buildnml
file should be capitalized to allow replacement values from user_nl_blom
).
3.4.3. Input data
All active model components and data components use input data sets. Wherever you are running the model, you need to store the input locally under DIN_LOC_ROOT
in order to run NorESM. If you want to use non-default input data, you can set the path to the file you want to use in the relevant user_nl_<components>
.
Input data is handled by the build process as follows:
The
buildnml
scripts in Buildconf/ create listings of required component input data sets in theBuildconf/<component>.input_data_list
filesThe
case.build
scripts checks for the presence of the required input data files in the root directoryDIN_LOC_ROOT
. If all required data sets are found on local disk, then the build can proceed.If any of the required input data sets are not found, the build script will abort and the files that are missing will be listed. At this point, you must obtain the required data from the input data server using
check_input_data
with the-export
option.
3.5. Aerosol diagnostics
The model can be set up to output AeroCom-specific variables, effective forcing estimates, and other additional aerosol output. See Aerosol diagnostics and output for details.
3.6. COSP
NorESM2 can be run with the CFMIP Observation Simulator Package (COSP) to calculates model cloud diagnostics that can be directly compared with satellite observations from ISCCP, CloudSat, CALIOP, MISR, and MODIS. Please see COSP output for details.
3.7. Code modifications
Sometimes you will want to make changes that go beyond what is possible from just changing the user namelists, and you will need to modify the source code itself (i.e. the fortran files). One way of doing this is to use the SourceMods folder in the case directory. The SourceMods folder contains sub-directories for all model component. Make a copy of the fortran file(s) you want to modify in the relevant sub-folder and modify the file(s) as needed before building the model. When compiling, the model will prioritize the modified file located under the SourceMods folder over the default version of the file located in the model source code under <noresm-base>.
Another option is to make a new branch for your code modifications following the procedure outlined in Git recipes and best practices. This has several advantages to using SourceMods, including that your changes are more easily visible for others (in your NorESM fork on GitHub), making them easy to share, and that the changes can more easily be considered for inclusion in the main NorESM repository on GitHub.
In either case, make sure that you use the source code from the same commit as you used to create the case (for commit details see README.case in the case folder).