- The Earth System Model eValuation Tool (ESMValTool)
is a community-development that aims at improving diagnosing and understanding of the causes and effects of model biases and inter-model spread. The ESMValTool can process multiple model outputs, including NorESM, but these model output need to be post-processed to conform with the CMIP data standard (i.e., CMOR). ESMValTool support all the published model data output on the ESGF
The esmvaltool-on-nird discussion group can be a useful resource if you plan to run ESMValTool on Nird.
1.1. Run ESMValTool on NIRD service node¶
The ESMValTool is currently installed under the dedicated IPCC service node (<ipcc.nird.sigma2.no>_) for data post-processing. The following NIRD project areas are mounted under /projects of the ``IPCC` node: NS2345K, NS9034K, NS9039K, NS9252K, NS9560K, NS9588K.
1.1.1. Activate and run ESMValTool¶
An example of steps to run the ESMValTool on ipcc.nird.sigma2.no
login in the ipcc node:
ssh -l username ipcc.nird.sigma2.no
load ESMValTool on nird:
conda activate /conda/esmvaltool/2.0.0/
(note, the esmvaltool may be upgraded in the future. Therefore, use
ls /conda/esmvaltool/ to check the currently installed vesion if 2.0.0 does not exist.)
esmvaltool config get_config_user
copy the following config file to ~/.esmvaltool/:
cp /projects/NS9252K/share/esmvaltool/config/config-ipcc.yml ~/.esmvaltool/
You may copy and modify the config-ipcc.yml file to store some intermediate data files and final plots to your own directory.
copy a recipe to ~/your-recipes/:
mkdir ~/your-recipes cp /projects/NS9252K/share/esmvaltool/recipes/recipe_validation_CMIP6.yml ~/your-recipes/
esmvaltool run --config_file .esmvaltool/config-ipcc.yml ./your-recipes/recipe_validation_CMIP6.yml
Check for results, sample results are under:
1.1.2. Download data automatically with Synda¶
is a tool to download and manage model data form the ESGF, it can be called by the
esmvaltoolas a command line option
--syndaso that it can automatically download necessary model data as specified in the receipe of ESMValTool. For example,
esmvaltool -c config_heyc.yml ./recipe_seaice.yml --synda
Note, it is OPTIONAL to use synda to download data automatically during the runtime of ESMValTool. One can download the data by some other tools or scripts before running the ESMValTool.
If you do want to use
synda, there are two major steps you need to do:
Activate the tool
It is install with
conda, so you can add
synda exectable to your search path, e.g.,
Then it should be availabe in your command windown by the
Set up your credentials
register (one of) the ESGF node and Globus transfer (not toally sure if account on globus is mandatory if this option will not be used, but seem yes according to my experience, you can firstly try without it)
paste your username and password to in
~/.synda/conf/sdt.conf, use my setting as template.
maybe you need get globus token. Run the following command:
synda token -p globus renew
it will give out something like:
Native App Authorization URL: https://auth.globus.org/v2/oauth2/authorize?code_challenge=BrmiBhFVVuHVNyGDj6hn5N8M1-EKJNnNgptobIsbTqI&state=_default&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code&response_type=code&client_id=83ec00c1-e67a-4356-9f1f-f7e31177e31a&scope=openid+email+profile+urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall&code_challenge_method=S256&access_type=offline Enter the auth code:
Paste the above https address to browser, and you will find a authen code, and past back to the command line.
1.2. Run ESMValTool on NIRD toolkit service¶
The ESMValTool is also available from NIRD Toolkit applications (Jupyter and JupyterHub), provided by Docker images.
Access to the NIRD Toolkit service is provided according to a 3 tier ranking
Owner : a project leader/executive officer of a NSxxxxK project
Admin : a person authorized by an owner to manage NIRD Toolkit applications
Member : a person authorized by the owner or admin to access a NIRD Toolkit application
In order to use NIRD Toolkit, an owner of of a NSxxxxK project should create a group in dataporten, and invite members to join, who may be given admin status by the owner or existing admin people. See Sigma2 information page about deploing service for details about the procedure.
1.2.1. Install and run an application¶
Applications in NIRD Toolkit are available from:
An owner/admin can launch a new application by installing it, and request resources to be made available to the application via the Kybernetes management system. A member can launch an existing application that has previously been set up by an owner/admin. The application setup allows access to storage areas under NSxxxxK storage volumes (read-only by default) and a user storage area under a specific NSxxxxK/subfolder with write access, but only applications pre-defined in the docker image provided to the Kybernetes system can be used.
1.2.2. Install ESMValTool docker image¶
ESMValTool is not included in the default docker images provided by Sigma2, but has been installed in modified docker images. These are created by building ESMValTool on top of an official Sigma2 docker image, and package in a new docker container. Such modified docker images are available for jupyter and jupyterhub applications.
To install the docker image you need access to the Nird Toolkit service through a project, and install a new instance of e.g. “jupyter”. On the installation page you select whatever standard settings you like (application name, projectspace, persistent storage, machine type), and then select “Show advances configuration…” and replace the standard dockerimage with:
One can activate the “JupyterLab” option in the configuration (but it is may not necessary to do so to run ESMValTool). To test if the image is installed correctly (provided the jupyter notebook environment builds without errors) you can open a terminal in the environment and type
esmvaltool -h, which should give you the help page for the tool in the terminal window.
As the development of ESMValTool and the updating of the NIRD Toolkit base system are not syncronous, there may sometimes be conflits in the package dependencies of what ESMValTool wants and what NIRD Toolkit provides. Hence, it is sometimes necessary to build
esmvaltool in a separate
conda environment (not “base”). If
esmvaltool -h fails to give the expected output, check if there exist any alternative environments by doing
conda env --list
If you find an
esmvaltool environment, this can be activated by
source activate esmvaltool
At the time of writing (10 June 2020), the preferred activation method
conda activate esmvaltool is not recognized inside a NIRD Toolkit application.