`NEMO-Cmd` API

This section documents the NEMO command processor Application Programming Interface (API). The API provides Python function interfaces to command processor sub-commands for use in other sub-command processor modules, and by other software.

nemo_cmd.api.combine(run_desc_file)[source]

Run the NEMO rebuild_nemo tool for each set of per-processor results files.

The output of rebuild_nemo for each file set is logged at the INFO level.

Parameters: run_desc_file – File path/name of the run description YAML file.

nemo_cmd.api.deflate(filepaths, max_concurrent_jobs)[source]

Deflate variables in each of the netCDF files in filepaths using Lempel-Ziv compression.

Converts files to netCDF-4 format. The deflated file replaces the original file.

Parameters

filepaths (sequence) – Paths/names of files to be deflated.
max_concurrent_jobs (int) – Maximum number of concurrent deflation processes allowed.

nemo_cmd.api.gather(results_dir)[source]

Move all of the files and directories from the present working directory into results_dir.

If results_dir doesn’t exist, create it.

Delete any symbolic links so that the present working directory is empty.

Parameters: results_dir (pathlib.Path) – Path of the directory into which to store the run results.

nemo_cmd.api.prepare(run_desc_file, nocheck_init=False)[source]

Prepare a NEMO run.

A UUID named temporary run directory is created and symbolic links are created in the directory to the files and directories specifed to run NEMO. The output of hg parents is recorded in the directory for the NEMO-code and NEMO-forcing repos that the symlinks point to. The path to the run directory is returned.

Parameters

run_desc_file (pathlib.Path) – File path/name of the YAML run description file.
nocheck_init (boolean) – Suppress initial condition link check the default is to check

Returns

Path of the temporary run directory

Return type

pathlib.Path

nemo_cmd.api.run_description(config_name='SalishSea', run_id=None, walltime=None, mpi_decomposition='8x18', NEMO_code=None, XIOS_code=None, forcing_path=None, runs_dir=None, forcing=None, init_conditions=None, namelists=None)[source]

Return a NEMO run description dict template.

Value may be passed for the keyword arguments to set the value of the corresponding items. Otherwise, the returned run description dict that must be updated by assignment statements to provide those values.

Note

The value of the ['forcing']['atmospheric'] item is set to /results/forcing/atmospheric/GEM2.5/operational/ which is appropriate for runs on salish, but needs to be changed for runs on WestGrid.

Parameters

config_name (str) – NEMO configuration name to use for the run.
run_id (str) – Job identifier that appears in the qstat listing.
walltime (str) – Wall-clock time requested for the run.
mpi_decomposition (str) – MPI decomposition to use for the run.
NEMO_code (str) – Path to the NEMO-code/ directory where the NEMO executable, etc. for the run are to be found. If a relative path is used it will start from the current directory.
XIOS_code (str) – Path to the XIOS/ directory where the XIOS executable for the run are to be found. If a relative path is used it will start from the current directory.
forcing_path (str) – Path to the NEMO-forcing/ directory where the netCDF files for the grid coordinates, bathymetry, initial conditions, open boundary conditions, etc. are found. If a relative path is used it will start from the current directory.
runs_dir (str) – Path to the directory where run directories will be created. If a relative path is used it will start from the current directory.
forcing (dict) – Forcing link data structure. The default of None produces “sensible defaults” for NEMO-3.4, but None for NEMO-3.6. See the Run Description File Structure docs for the version of NEMO that you are using for details of the data structure.
init_conditions (str) – Name of sub-directory in NEMO-forcing/ where initial conditions files are to be found, or the path to and name of a restart file. If a relative path is used for a restart file it will start from the current directory.
namelists (dict) – Namelists data structure. The default of None produces “sensible defaults” for a physics-only run for both NEMO-3.4, annd NEMO-3.6. See the Run Description File Structure docs for the version of NEMO that you are using for details of the data structure.

nemo_cmd.api.run_in_subprocess(run_id, run_desc, results_dir)[source]

Execute nemo run in a subprocess.

Parameters

run_id (str) – Job identifier that appears in the qstat listing. A temporary run description YAML file is created with the name run_id_subprocess_run.yaml.
run_desc (dict) – Run description data structure that will be written to the temporary YAML file.
results_dir (str) – Directory to store results into.

nemo_cmd.api.pbs_common(run_description, n_processors, email, results_dir, pmem='2000mb')[source]

Return the common PBS directives used to run NEMO in a TORQUE/PBS multiple processor context.

The string that is returned is intended for inclusion in a bash script that will be to the TORQUE/PBS queue manager via the qsub command.

Parameters

run_description (dict) – Run description data structure.
n_processors (int) – Number of processors that the run will be executed on. For NEMO-3.6 runs this is the sum of NMEO and XIOS processors.
email (str) – Email address to send job begin, end & abort notifications to.
results_dir (str or pathlib.Path) – Directory to store results into.
pmem (str) – Memory per processor.

Returns

PBS directives for run script.

Return type

Unicode str

Functions for Preparing NEMO Runs

The functions are probably most useful for packages that extend the NEMO-Cmd package to create command processors for particular NEMO configurations. The SalishSeaCast NEMO Command Processor package SalishSeaCmd is an example of such an extension package.

nemo_cmd.prepare.add_agrif_files(run_desc, desc_file, run_set_dir, run_dir, nocheck_init)[source]

Add file copies and symlinks to temporary run directory for AGRIF runs.

Parameters

run_desc (dict) – Run description dictionary.
desc_file (pathlib.Path) – File path/name of the YAML run description file.
run_set_dir (pathlib.Path) – Directory containing the run description file, from which relative paths for the namelist section files start.
run_dir (pathlib.Path) – Path of the temporary run directory.
nocheck_init (boolean) – Suppress restart file existence check; the default is to check

Raises

SystemExit with exit code 2 if mismatching number of sub-grids is detected

nemo_cmd.prepare.check_nemo_exec(run_desc)[source]

Calculate absolute path of the NEMO executable’s directory.

Confirm that the NEMO executable exists, raising a SystemExit exception if it does not.

Parameters: run_desc (dict) – Run description dictionary.
Returns: Absolute path of NEMO executable’s directory.
Return type: pathlib.Path
Raises: SystemExit with exit code 2

nemo_cmd.prepare.check_xios_exec(run_desc)[source]

Calculate absolute path of the XIOS executable’s directory.

Confirm that the XIOS executable exists, raising a SystemExit exception if it does not.

Parameters: run_desc (dict) – Run description dictionary.
Returns: Absolute path of XIOS executable’s directory.
Return type: pathlib.Path
Raises: SystemExit with exit code 2

nemo_cmd.prepare.copy_run_set_files(run_desc, desc_file, run_set_dir, run_dir, agrif_n=None)[source]

Copy the run-set files given into run_dir.

The YAML run description file (from the command-line) is copied. The IO defs file is also copied. The file path/name of the IO defs file is taken from the output stanza of the YAML run description file. The IO defs file is copied as iodef.xml because that is the name that XIOS expects.

The domain defs and field defs files used by XIOS are also copied. Those file paths/names of those file are taken from the output stanza of the YAML run description file. They are copied to domain_def.xml and field_def.xml, respectively, because those are the file names that XIOS expects. Optionally, the file defs file used by XIOS-2 is also copied. Its file path/name is also taken from the output stanza. It is copied to file_def.xml because that is the file name that XIOS-2 expects.

Parameters

run_desc (dict) – Run description dictionary.
desc_file (pathlib.Path) – File path/name of the YAML run description file.
run_set_dir (pathlib.Path) – Directory containing the run description file, from which relative paths for the namelist section files start.
run_dir (pathlib.Path) – Path of the temporary run directory.
agrif_n (int) – AGRIF sub-grid number.

nemo_cmd.prepare.get_hg_revision(hg_repo, run_dir)[source]

Gather revision and status information from a Mercurial repo.

Effectively record the output of hg parents -v and hg status -mardC.

Files named CONFIG/cfg.txt and TOOLS/COMPILE/full_key_list.txt are ignored because they change frequently but the changes generally of no consequence; see https://github.com/SalishSeaCast/NEMO-Cmd/issues/18.

Parameters

hg_repo (pathlib.Path) – Path of Mercurial repository to get revision and status information from.
run_dir (pathlib.Path) – Path of the temporary run directory.

Returns

Mercurial repository revision and status information strings.

Return type

list

nemo_cmd.prepare.get_n_processors(run_desc, run_dir)[source]

Return the total number of processors required for the run as specified by the MPI decomposition key in the run description.

Parameters

run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.

Returns

Number of processors required for the run.

Return type

int

nemo_cmd.prepare.make_executable_links(nemo_bin_dir, run_dir, xios_bin_dir)[source]

Create symlinks in run_dir to the NEMO and XIOS executables.

Parameters

nemo_bin_dir (pathlib.Path) – Absolute path of directory containing NEMO executable.
run_dir (pathlib.Path) – Path of the temporary run directory.
xios_bin_dir (pathlib.Path) – Absolute path of directory containing XIOS executable.

nemo_cmd.prepare.make_forcing_links(run_desc, run_dir)[source]

Create symlinks in run_dir to the forcing directory/file names given in the run description forcing section.

Parameters

run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.

Raises

SystemExit with exit code 2 if a symlink target does not exist

nemo_cmd.prepare.make_grid_links(run_desc, run_dir, agrif_n=None)[source]

Create symlinks in run_dir to the file names that NEMO expects to the bathymetry and coordinates files given in the run_desc dict.

For AGRIF sub-grids, the symlink names are prefixed with the agrif_n; e.g. 1_coordinates.nc.

Parameters

run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.
agrif_n (int) – AGRIF sub-grid number.

Raises

SystemExit with exit code 2

nemo_cmd.prepare.make_namelists(run_set_dir, run_desc, run_dir, agrif_n=None)[source]

Build the namelist files for the NEMO run in run_dir by concatenating the lists of namelist section files provided in run_desc.

If any of the required namelist section files are missing, delete the run directory and raise a SystemExit exception.

Parameters

run_set_dir (pathlib.Path) – Directory containing the run description file, from which relative paths for the namelist section files start.
run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.
agrif_n (int) – AGRIF sub-grid number.

Raises

SystemExit with exit code 2

nemo_cmd.prepare.make_restart_links(run_desc, run_dir, nocheck_init, agrif_n=None)[source]

Create symlinks in run_dir to the restart files given in the run description restart section.

Parameters

run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.
nocheck_init (boolean) – Suppress restart file existence check; the default is to check
agrif_n (int) – AGRIF sub-grid number.

Raises

SystemExit with exit code 2 if a symlink target does not exist

nemo_cmd.prepare.make_run_dir(run_desc)[source]

Create the temporary directory from which NEMO will be run.

The location is in the runs directory from the run description, and its name is the run id combined with an ISO-format date/time stamp.

Parameters: run_desc (dict) – Run description dictionary.
Returns: Path of the temporary run directory
Return type: pathlib.Path

nemo_cmd.prepare.record_vcs_revisions(run_desc, run_dir)[source]

Record revision and status information from version control system repositories in files in the temporary run directory.

Parameters

run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.

nemo_cmd.prepare.remove_run_dir(run_dir)[source]

Remove all files from run_dir, then remove run_dir.

Intended to be used as a clean-up operation when some other part of the prepare process fails.

Parameters: run_dir (pathlib.Path) – Path of the temporary run directory.

nemo_cmd.prepare.set_mpi_decomposition(namelist_filename, run_desc, run_dir)[source]

Update the &nammpp namelist jpni & jpnj values with the MPI decomposition values from the run description.

A SystemExit exception is raise if there is no MPI decomposition specified in the run description.

Parameters

namelist_filename (str) – The name of the namelist file.
run_desc (dict) – Run description dictionary.
run_dir (pathlib.Path) – Path of the temporary run directory.

Raises

SystemExit with exit code 2

nemo_cmd.prepare.write_repo_rev_file(repo, run_dir, vcs_func)[source]

Write revision and status information from a version control system repository to a file in the temporary run directory.

The file name is the repository directory name with _rev.txt appended.

Parameters

repo (pathlib.Path) – Path of Mercurial repository to get revision and status information from.
run_dir (pathlib.Path) – Path of the temporary run directory.
vcs_func – Function to call to gather revision and status information from repo.

Functions for Working with File System Paths

nemo_cmd.fspath(path)[source]

Return the string representation of a file system path.

This function provides a uniform interface for working with pathlib.Path objects in Python versions both before and after Python 3.6 in which the PEP 519 file system protocol appeared.

Parameters: path (pathlib.Path or str) – Path to get string representation of.
Returns: String representation of path.
Return type: str

nemo_cmd.expanded_path(path)[source]

Expand shell and user variables in path and produce a pathlib.Path object.

Parameters: path (pathlib.Path or str) – Path to expand variables in.
Returns: Path with shell and user variables expanded.
Return type: pathlib.Path

nemo_cmd.resolved_path(path)[source]

Expand shell and user variables in path and resolve symbolic links to produce an absolute pathlib.Path object.

Parameters: path (pathlib.Path or str) – Path to expand variables in and resolve.
Returns: Absolute path with shell and user variables expanded and symlinks resolved.
Return type: pathlib.Path

Utility Functions

nemo_cmd.prepare.get_run_desc_value(run_desc, keys, expand_path=False, resolve_path=False, run_dir=None, fatal=True)[source]

Get the run description value defined by the sequence of keys.

Parameters

run_desc (dict) – Run description dictionary.
keys (sequence) – Keys that lead to the value to be returned.
expand_path (boolean) – When True, return the value as a pathlib.Path object with shell and user variables expanded via nemo_cmd.expanded_path().
resolve_path (boolean) – When True, return the value as an absolute pathlib.Path object with shell and user variables expanded and symbolic links resolved via nemo_cmd.resolved_path(). Also confirm that the path exists, otherwise, raise a SystemExit exception.
run_dir (pathlib.Path) – Path of the temporary run directory.
fatal (boolean) – When True, delete the under construction temporary run directory, and raise a SystemExit exception. Otherwise, raise a KeyError exception.

Raises

SystemExit or KeyError

Returns

Run description value defined by the sequence of keys.

nemo_cmd.prepare.load_run_desc(desc_file)[source]

Load the run description file contents into a data structure.

Parameters: desc_file (pathlib.Path) – File path/name of the YAML run description file.
Returns: Contents of run description file parsed from YAML into a dict.
Return type: dict

NEMO-Cmd API

Functions for Preparing NEMO Runs

Functions for Working with File System Paths

Utility Functions

`NEMO-Cmd` API