2. CCPP-Compliant Physics Parameterizations

The rules for a scheme to be considered CCPP-compliant are summarized in this section. It should be noted that making a scheme CCPP-compliant is a necessary but not guaranteed step for the acceptance of the scheme in the pool of supported CCPP Physics. Acceptance is dependent on scientific innovation, demonstrated value, and compliance with the rules described below. The criteria for acceptance of a scheme into the CCPP is under development.

It is recommended that parameterizations be comprised of the smallest units that will be used independently. For example, if a given pair of deep and shallow convection schemes will always be called together and in a pre-established order, it is acceptable to group them within a single scheme. However, if one envisions that the deep and shallow convection schemes may someday operate independently, it is recommended to code two separate schemes to allow more flexibility.

Some schemes in the CCPP have been implemented using a driver as an entry point. In this context, a driver is defined as a wrapper of code around the actual scheme, providing the CCPP entry points. In order to minimize the layers of code in the CCPP, the implementation of a driver is discouraged, that is, it is preferable that the CCPP be composed of atomic parameterizations. One example is the implementation of the MG microphysics, in which a simple entry point leads to two versions of the scheme, MG2 and MG3. A cleaner implementation would be to retire MG2 in favor of MG3, to turn MG2 and MG3 into separate schemes, or to create a single scheme that can behave as MG2 and MG3 depending on namelist options.

The implementation of a driver is reasonable under the following circumstances:

  • To preserve schemes that are also distributed outside of the CCPP. For example, the Thompson microphysics scheme is distributed both with the Weather Research and Forecasting (WRF) model and with the CCPP. Having a driver with CCPP directives allows the Thompson scheme to remain intact so that it can be synchronized between the WRF model and the CCPP distributions. See more in mp_thompson.F90 in the ccpp-physics/physics directory.

  • To perform unit conversions or array transformations, such as flipping the vertical direction and rearranging the index order, for example, cu_gf_driver.F90 or gfdl_cloud_microphys.F90 in the ccpp-physics/physics directory.

Schemes in the CCPP are classified into two categories: primary schemes and interstitial schemes. A primary scheme is one that updates the state variables and tracers or that produces tendencies for updating state variables and tracers based on the representation of major physical processes, such as radiation, convection, microphysics, etc. This does not include:

  • Schemes that compute tendencies exclusively for diagnostic purposes.

  • Schemes that adjust tendencies for different timesteps (e.g., create radiation tendencies based on a radiation scheme called at coarser intervals).

  • Schemes that update the model state based on tendencies generated in primary schemes.

Interstitial schemes are modularized pieces of code that perform data preparation, diagnostics, or other “glue” functions, and allow primary schemes to work together as a suite. They can be categorized as “scheme-specific” or “suite-level”. Scheme-specific interstitial schemes augment a specific primary scheme (to provide additional functionality). Suite-level interstitial schemes provide additional functionality on top of a class of primary schemes, connect two or more schemes together, or provide code for conversions, initializing sums, or applying tendencies, for example. The rules and guidelines provided in the following sections apply both to primary and interstitial schemes.

CCPP-compliant physics parameterizations are broken down into one or more of the following five phases:

  • The init phase, which performs actions needed to set up the scheme before the model integration begins. Examples of actions needed in this phase include the reading/computation of lookup tables, setting of constants (as described in Section 2.6), etc.

  • The timestep_init phase, which performs actions needed at the start of each physics timestep. Examples of actions needed in this phase include updating of time-based settings (e.g. solar angle), reading lookup table values, etc.

  • The run phase, which is the main body of the scheme. Here is where the physics is integrated forward to the next timestep.

  • The timestep_finalize phase, which performs post-integration calculations such as computing statistics or diagnostic tendencies. Not currently used by any scheme.

  • The finalize phase, which performs cleanup and finalizing actions at the end of model integration. Examples of actions needed in this phase include deallocating variables, closing files, etc.

The various phases have different rules when it comes to parallelization, especially with regards to how data is blocked among parallel processes; see Section 2.7 for more information.

2.1. General Rules

A CCPP-compliant scheme is written in the form of Fortran modules. Each scheme must be in its own module, and must include at least one of the following subroutines (entry points): _init, _timestep_init, _run, _timestep_finalize, and _finalize. Each subroutine corresponds to one of the five phases of the CCPP Framework as described above. The module name and the subroutine names must be consistent with the scheme name; for example, the scheme “schemename” can have the entry points schemename_init, schemename_run, etc. The _run subroutine contains the code to execute the scheme. If subroutines _timestep_init or _timestep_finalize are present, they will be executed at the beginning and at the end of the host model physics timestep, respectively. Further, if present, the _init and _finalize subroutines associated with a scheme are run at the beginning and at the end of the model run. The _init and _finalize subroutines may be called more than once depending on the host model’s parallelization strategy, and as such must be idempotent (the answer must be the same when the subroutine is called multiple times). This can be achieved by using a module variable is_initialized that keeps track whether a scheme has been initialized or not.

Listing 2.1 contains a template for a CCPP-compliant scheme, which includes the _run subroutine for an example scheme_template scheme. Each .F or .F90 file that contains an entry point(s) for CCPP scheme(s) must be accompanied by a .meta file in the same directory as described in Section 2.2

    module scheme_template

      contains

!> \section arg_table_scheme_template_run Argument Table
!! \htmlinclude scheme_template_run.html
!!
      subroutine scheme_template_run (errmsg, errflg)

         implicit none

         !--- arguments
         ! add your arguments here
         character(len=*), intent(out)   :: errmsg
         integer,          intent(out)   :: errflg

         !--- local variables
         ! add your local variables here

         continue

         !--- initialize CCPP error handling variables
         errmsg = ''
         errflg = 0

         !--- initialize intent(out) variables
         ! initialize all intent(out) variables here

         !--- actual code
         ! add your code here

         ! in case of errors, set errflg to a value != 0,
         ! assign a meaningful message to errmsg and return

         return

      end subroutine scheme_template_run

    end module scheme_template

Listing 2.1: Fortran template for a CCPP-compliant scheme showing the _run subroutine. The structure for the other phases (_timestep_init, _init, _finalize, and _timestep_finalize) is identical.

The three lines in the example template beginning !> \section are required. They begin with ! and so will be treated as comments by the Fortran compiler, but are interpreted by Doxygen as part of the process to create scientific documentation. Those lines specifically insert an external file containing metadata information (in this case, scheme_template_run.html) in the documentation. See more on this topic in Section 2.8.

All external information required by the scheme must be passed in via the argument list, including physical constants. Statements such as use EXTERNAL_MODULE should not be used for passing in any data. See Section 2.6 for more information on how to use physical constants.

Note that standard names, variable names, module names, scheme names and subroutine names are all case insensitive.

Interstitial modules (schemename_pre and schemename_post) can be included if any part of the physics scheme must be executed sequentially before (_pre) or after (_post) the scheme, but can not be included in the scheme itself (e.g., for including host-specific code).

2.2. Metadata Table Rules

Each CCPP-compliant physics scheme (.F or .F90 file) must have a corresponding metadata file (.meta) that contains information about CCPP entry point schemes and their dependencies. These files contain two types of metadata tables: ccpp-table-properties and ccpp-arg-table, both of which are mandatory. The contents of these tables are described in the sections below.

Metadata files (.meta) are in a relaxed config file format and contain metadata for one or more CCPP entry points.

2.2.1. ccpp-table-properties

The [ccpp-table-properties] section is required in every metadata file and has four valid entries:

  1. type: In the CCPP Physics, type can be scheme, module, or ddt (derived data type) and must match the type in the associated [ccpp-arg-table] section(s).

  2. name: This depends on the type. For types ddt and module (for variable/type/kind definitions), name must match the name of the single associated [ccpp-arg-table] section. For type scheme, the name must match the root names of the [ccpp-arg-table] sections for that scheme, without the suffixes _timestep_init, _init, _run, _finalize, or _timestep_finalize.

  3. dependencies: type/kind/variable definitions and physics schemes often depend on code in other files (e.g. “use machine” –> depends on machine.F). These dependencies must be provided as a comma-separated list. Relative path(s) to those file(s) must be specified here or using the relative_path entry described below. Dependency attributes are additive; multiple lines containing dependencies can be used. With the exception of specific files, such as machine.F, which provides the kind_phys Fortran kind definition, shared dependencies between schemes are discouraged.

  4. relative_path: If specified, the relative path is added to every file listed in the dependencies.

The information in this section table allows the CCPP to compile only the schemes and dependencies needed by the selected CCPP suite(s).

An example for type and variable definitions from the file ccpp-physics/physics/radlw_param.meta is shown in Listing 2.2.

Note

A single metadata file may require multiple instances of the [ccpp-table-properties] section. For example, if a scheme requires multiple derived data types, each should have its own [ccpp-table-properties] entry. Subsequent [ccpp-table-properties] sections should be preceded by a separating line of # characters, as shown in the examples on this page.

[ccpp-table-properties]
  name = topflw_type
  type = ddt
  dependencies =

[ccpp-arg-table]
  name = topflw_type
  type = ddt

########################################################################
[ccpp-table-properties]
  name = sfcflw_type
  type = ddt
  dependencies =

[ccpp-arg-table]
  name = sfcflw_type
  type = ddt

########################################################################
[ccpp-table-properties]
  name = proflw_type
  type = ddt
  dependencies =

[ccpp-arg-table]
  name = proflw_type
  type = ddt

########################################################################
[ccpp-table-properties]
  name = module_radlw_parameters
  type = module
  dependencies =

[ccpp-arg-table]
  name = module_radlw_parameters
  type = module
[topflw_type]
  standard_name = topflw_type
  long_name = definition of type topflw_type
  units = DDT
  dimensions = ()
  type = topflw_type
[sfcflw_type]
  standard_name = sfcflw_type
  long_name = definition of type sfcflw_type
  units = DDT
  dimensions = ()
  type = sfcflw_type
[proflw_type]
  standard_name = proflw_type
  long_name = definition of type proflw_type
  units = DDT
  dimensions = ()
  type = proflw_type

Listing 2.2: Example of a CCPP-compliant metadata file showing the use of the [ccpp-table-properties] section and how it relates to [ccpp-arg-table].

An example metadata file for the CCPP scheme mp_thompson.meta (with many sections omitted as indicated by ...) is shown in Listing 2.3.

[ccpp-table-properties]
  name = mp_thompson
  type = scheme
  dependencies = machine.F,module_mp_radar.F90,module_mp_thompson.F90,module_mp_thompson_make_number_concentrations.F90

########################################################################
[ccpp-arg-table]
  name = mp_thompson_init
  type = scheme
[ncol]
  standard_name = horizontal_dimension
  long_name = horizontal dimension
  units = count
  dimensions = ()
  type = integer
  intent = in
...

########################################################################
[ccpp-arg-table]
  name = mp_thompson_run
  type = scheme
[ncol]
  standard_name = horizontal_loop_extent
  long_name = horizontal loop extent
  units = count
  dimensions = ()
  type = integer
  intent = in
...

########################################################################
[ccpp-arg-table]
  name = mp_thompson_finalize
  type = scheme
[errmsg]
  standard_name = ccpp_error_message
  long_name = error message for error handling in CCPP
  units = none
  dimensions = ()
  type = character
  kind = len=*
  intent = out
...

Listing 2.3: Example metadata file for a CCPP-compliant physics scheme using a single [ccpp-table-properties] entry and how it defines dependencies for multiple [ccpp-arg-table] entries. In this example the timestep_init and timestep_finalize phases are not used.

2.2.2. ccpp-arg-table

For each CCPP compliant scheme, the ccpp-arg-table for a scheme, module or derived data type starts with this set of lines

[ccpp-arg-table]
 name = <name>
 type = <type>

  • ccpp-arg-table indicates the start of a new metadata section for a given scheme.

  • <name> is name of the corresponding subroutine/module.

  • <type> can be scheme, module, or DDT.

  • The metadata must describe all input and output arguments to the scheme using the following format:

[varname]
 standard_name = <standard_name>
 long_name = <long_name>
 units = <units>
 rank = <rank>
 dimensions = <dimensions>
 type = <type>
 kind = <kind>
 intent = <intent>

  • The intent argument is only valid in scheme metadata tables, as it is not applicable to the other types.

  • The following attributes are optional: long_name, kind.

  • Lines can be combined using | as a separator, e.g.,

type = real | kind = kind_phys

  • [varname] is the local name of the variable in the subroutine.

  • The dimensions attribute should be empty parentheses for scalars or contain the standard_name for the start and end for each dimension of an array. ccpp_constant_one is the assumed start for any dimension which only has a single value. For example:

dimensions = ()
dimensions = (ccpp_constant_one:horizontal_loop_extent, vertical_level_dimension)
dimensions = (horizontal_dimension,vertical_dimension)
dimensions = (horizontal_dimension,vertical_dimension_of_ozone_forcing_data,number_of_coefficients_in_ozone_forcing_data)

  • The order of arguments in the entry point subroutines must match the order of entries in the metadata file.

Listing 2.4 contains the template .meta file for an example CCPP-compliant scheme (scheme_template.meta)

[ccpp-table-properties]
  name = ozphys
  type = scheme
  dependencies = machine.F

[ccpp-arg-table]
  name = ozphys_run
  type = scheme
[errmsg]
  standard_name = ccpp_error_message
  long_name = error message for error handling in CCPP
  units = none
  dimensions = ()
  type = character
  kind = len=*
  intent = out
  optional = F
[errflg]
  standard_name = ccpp_error_code
  long_name = error code for error handling in CCPP
  units = 1
  dimensions = ()
  type = integer
  intent = out
  optional = F

Listing 2.4: Fortran template for a metadata file accompanying a CCPP-compliant scheme.

2.2.3. horizontal_dimension vs. horizontal_loop_extent

It is important to understand the difference between these metadata dimension names.

  • horizontal_dimension refers to all (horizontal) grid columns that an MPI process owns/is responsible for, and that are passed to the physics in the init, timestep_init, timestep_finalize, and finalize phases.

  • horizontal_loop_extent or, equivalent, ccpp_constant_one:horizontal_loop_extent stands for a subset of grid columns that are passed to the physics during the time integration, i.e. in the run phase.

  • Note that horizontal_loop_extent is identical to horizontal_dimension for host models that pass all columns to the physics during the time integration.

Since physics developers cannot know whether a host model is passing all columns to the physics during the time integration or just a subset of it, the following rules apply to all schemes:

  • Variables that depend on the horizontal decomposition must use

    • horizontal_dimension in the metadata tables for the following phases: init, timestep_init, timestep_finalize, finalize.

    • horizontal_loop_extent or ccpp_constant_one:horizontal_loop_extent in the run phase.

2.3. Standard names

Variables available for CCPP physics schemes are identified by their unique standard name. This policy is in place to ensure that variables will always be unique and unambiguous when communicating between different schemes and different host models. Schemes are free to use their own variable names within their individual codes, but these variables must be assigned to a standard name within the scheme’s metadata table as described in Section 2.4.

Standard names are listed and defined in a GitHub repository (https://github.com/ESCOMP/CCPPStandardNames), along with rules for adding new standard names as needed. While an effort is made to comply with existing standard name definitions of the Climate and Forecast (CF) conventions (http://cfconventions.org), additional names are used in the CCPP to cover the wide range of use cases the CCPP intends to include. Each hash of the CCPP Physics repository contains information in the top-level README.md file indicating which version of the CCPPStandardNames repository corresponds to that version of CCPP code.

An up-to-date list of available standard names for a given host model can be found by running the CCPP prebuild script (described in Chapter 8), which will generate a LaTeX source file that can be compiled to produce a PDF file with all variables defined by the host model and requested by the physics schemes.

2.4. Input/Output Variable (argument) Rules

  • A standard_name cannot be assigned to more than one local variable (local_name). The local_name of a variable can be chosen freely and does not have to match the local_name in the host model.

  • All variable information (standard_name, units, dimensions) must match the specifications on the host model side, but sub-slices can be used/added in the host model. For example, when using the UFS Atmosphere as the host model, tendencies are split in GFS_typedefs.meta so they can be used in the necessary physics scheme:

    [dt3dt(:,:,1)]
  standard_name = cumulative_change_in_temperature_due_to_longwave_radiation
  long_name = cumulative change in temperature due to longwave radiation
  units = K
  dimensions = (horizontal_dimension,vertical_dimension)
  type = real
  kind = kind_phys
[dt3dt(:,:,2)]
  standard_name = cumulative_change_in_temperature_due_to_shortwave_radiation
  long_name = cumulative change in temperature due to shortwave radiation
  units = K
  dimensions = (horizontal_dimension,vertical_dimension)
  type = real
  kind = kind_phys
[dt3dt(:,:,3)]
  standard_name = cumulative_change_in_temperature_due_to_PBL
  long_name = cumulative change in temperature due to PBL
  units = K
  dimensions = (horizontal_dimension,vertical_dimension)
  type = real
  kind = kind_phys

    For performance reasons, slices of arrays should be contiguous in memory, which, in Fortran, implies that the dimension that is split is the rightmost (outermost) dimension as in the example above.

  • The two mandatory variables that any scheme-related subroutine must accept as intent(out) arguments are errmsg and errflg (see also coding rules in Section 2.5).

  • At present, only two types of variable definitions are supported by the CCPP Framework:

    • Standard intrinsic Fortran variables are preferred (character, integer, logical, real, complex). For character variables, the length should be specified as * in order to allow the host model to specify the corresponding variable with a length of its own choice. All others can have a kind attribute of a kind type defined by the host model.

    • Derived data types (DDTs). While the use of DDTs is discouraged, some use cases may justify their application (e.g. DDTs for chemistry that contain tracer arrays or information on whether tracers are advected). These DDTs must be defined by the scheme itself, not by the host model. It should be understood that use of DDTs within schemes forces their use in host models and potentially limits a scheme’s portability. Where possible, DDTs should be broken into components that could be usable for another scheme of the same type.

  • It is preferable to have separate variables for physically-distinct quantities. For example, an array containing various cloud properties should be split into its individual physically-distinct components to facilitate generality. An exception to this rule is if there is a need to perform the same operation on an array of otherwise physically-distinct variables. For example, tracers that undergo vertical diffusion can be combined into one array where necessary. This tactic should be avoided wherever possible, and is not acceptable merely as a convenience.

  • If a scheme is to make use of CCPP’s subcycling capability, the current loop counter and the loop extent can be obtained from CCPP as intent(in) variables (see a mandatory list of variables that are provided by the CCPP Framework and/or the host model for this and other purposes).

  • It is preferable to use assumed-size array declarations for input/output variables for CCPP schemes, i.e. instead of

    real(kind=kind_phys), dimension(is:ie,ks:ke), intent(inout) :: foo

    one should use

    real(kind=kind_phys), dimension(:,:), intent(inout) :: foo

    This allows the compiler to perform bounds checking and detect errors that otherwise may go unnoticed.

    Warning

    Fortran assumes that the lower bound of assumed-size arrays is 1. If foo has lower bounds is and ks that are different from 1, then these must be specified explicitly:

    real(kind=kind_phys), dimension(is:,ks:), intent(inout) :: foo

2.5. Coding Rules

  • Code must comply to modern Fortran standards (Fortran 90 or newer), where possible.

  • Uppercase file endings (.F, .F90) are preferred to enable preprocessing by default.

  • Labeled end statements should be used for modules, subroutines, functions, and type definitions; for example, module scheme_template → end module scheme_template.

  • Implicit variable declarations are not allowed. The implicit none statement is mandatory and is preferable at the module-level so that it applies to all the subroutines in the module.

  • All intent(out) variables must be set inside the subroutine, including the mandatory variables errflg and errmsg.

  • Decomposition-dependent host model data inside the module cannot be permanent, i.e. variables that contain domain-dependent data cannot be kept using the save attribute.

  • The use of goto statements is discouraged.

  • common blocks are not allowed.

  • Schemes are not allowed to abort/stop execution.

  • Errors are handled by the host model using the two mandatory arguments errmsg and errflg. In the event of an error, a meaningful error message should be assigned to errmsg and errflg set to a value other than 0. For example:

errmsg = ‘Logic error in scheme xyz: ...’
errflg = 1
return

  • Schemes are not allowed to perform I/O operations except for reading lookup tables or other information needed to initialize the scheme, including stdout and stderr. Diagnostic messages are tolerated, but should be minimal.

  • Line lengths of no more than 120 characters are suggested for better readability.

Additional coding rules are listed under the Coding Standards section of the NOAA NGGPS Overarching System team document on Code, Data, and Documentation Management for NOAA Environmental Modeling System (NEMS) Modeling Applications and Suites (available at https://docs.google.com/document/u/1/d/1bjnyJpJ7T3XeW3zCnhRLTL5a3m4_3XIAUeThUPWD9Tg/edit).

2.6. Using Constants

There are two principles that must be followed when using physical constants within CCPP-compliant physics schemes:

  1. All schemes should use a single, consistent set of constants.

  2. The host model must control (define and use) that single set, to provide consistency between a host model and the physics.

As long as a host application provides metadata describing its physical constants so that the CCPP framework can pass them to the physics schemes, these two principles are realized, and the CCPP physics schemes are model-agnostic. Since CCPP-compliant hosts provide metadata about the available physical constants, they can be passed into schemes like any other data.

For simple schemes that consist of one or two files and only a few “helper” subroutines, passing in physical constants via the argument list and propagating those constants down to any subroutines that need them is the most direct approach. The following example shows how the constant karman can be passed into a physics scheme:

subroutine my_physics_run(im,km,ux,vx,tx,karman)
  ...
real(kind=kind_phys),intent(in)   ::  karman

Where the following has been added to the my_physics.meta file:

[karman]
  standard_name = von_karman_constant
  long_name = von karman constant
  units = none
  dimensions = ()
  type = real
  intent = in

This allows the von Karman constant to be defined by the host model and be passed in through the CCPP scheme subroutine interface.

For pre-existing complex schemes that contain many software layers and/or many “helper” subroutines that require physical constants, another method is accepted to ensure that the two principles are met while eliminating the need to modify many subroutine interfaces. This method passes the physical constants once through the argument list for the top-level _init subroutine for the scheme. This top-level _init subroutine also imports scheme-specific constants from a user-defined module. For example, constants can be set in a module as:

module my_scheme_common
  use machine,     only : kind_phys
  implicit none
  real(kind=kind_phys)   ::  pi, omega1, omega2
end module my_scheme_common

Within the _init subroutine body, the constants in the my_scheme_common module can be set to the ones that are passed in via the argument list, including any derived ones. For example:

module my_scheme
  use machine, only: kind_phys
  implicit none
  private
  public my_scheme_init, my_scheme_run, my_scheme_finalize
  logical :: is_initialized = .false.
contains
  subroutine my_scheme_init (a, b, con_pi, con_omega)
    use my_scheme_common, only: pi, omega1, omega2
      ...
    pi = con_pi
    omega1 = con_omega
    omega2 = 2.*omega1
      ...
    is_initialized = .true.
  end subroutine my_scheme_init

  subroutine my_scheme_run (a, b)
    use my_scheme_common, only: pi, omega1, omega2
      ...
  end subroutine my_scheme_run

  subroutine my_scheme_finalize
      ...
    is_initialized = .false.
    pi = -999.
    omega1 = -999.
    omega2 = -999.
      ...
  end subroutine my_scheme_finalize
end module my_scheme

After this point, physical constants can be imported from my_scheme_common wherever they are needed. Although there may be some duplication in memory, constants within the scheme will be guaranteed to be consistent with the rest of physics and will only be set/derived once during the initialization phase. Of course, this will require that any constants in my_scheme_common that are coming from the host model cannot use the Fortran parameter keyword. To guard against inadvertently using constants in my_scheme_common without setting them from the host, they should be initially set to some invalid value. The above example also demonstrates the use of is_initialized to guarantee idempotence of the _init routine. To clean up during the finalize phase of the scheme, the is_initialized flag can be set back to false and the constants can be set back to an invalid value.

In summary, there are two ways to pass constants to a physics scheme. The first is to directly pass constants via the subroutine interface and continue passing them down to all subroutines as needed. The second is to have a user-specified scheme constants module within the scheme and to sync it once with the physical constants from the host model at initialization time. The approach to use is somewhat up to the developer.

Note

Use of the physcons module (ccpp-physics/physics/physcons.F90) is not recommended, since it is specific to FV3 and will be removed in the future.

2.7. Parallel Programming Rules

Most often, shared memory (OpenMP: Open Multi-Processing) and distributed memory (MPI: Message Passing Interface) communication is done outside the physics, in which case the loops and arrays already take into account the sizes of the threaded tasks through their input indices and array dimensions.

The following rules should be observed when including OpenMP or MPI communication in a physics scheme:

  • CCPP standards require that in every phase but the run phase, blocked data structures must be combined so that their entire contents are available to a given MPI task (i.e. the data structures can not be further subdivided, or “chunked”, within those phases). The run phase may be called by multiple threads in parallel, so data structures may be divided into blocks for that phase.

  • Shared-memory (OpenMP) parallelization inside a scheme is allowed with the restriction that the number of OpenMP threads to use is obtained from the host model as an intent(in) argument in the argument list (Listing 6.2).

  • MPI communication is allowed in the init, timestep_init, timestep_finalize, and finalize, phases for the purpose of computing, reading or writing scheme-specific data that is independent of the host model’s data decomposition.

  • If MPI is used, it is restricted to global communications: barrier, broadcast, gather, scatter, reduction. Point-to-point communication is not allowed. The MPI communicator must be passed to the physics scheme by the host model, the use of MPI_COMM_WORLD is not allowed (see list of mandatory variables).

  • An example of a valid use of MPI is the initial read of a lookup table of aerosol properties by one or more MPI processes, and its subsequent broadcast to all processes.

  • The implementation of reading and writing of data must be scalable to perform efficiently from a few to thousands of tasks.

  • Calls to MPI and OpenMP functions, and the import of the MPI and OpenMP libraries, must be guarded by C preprocessor directives as illustrated in the following listing. OpenMP pragmas can be inserted without C preprocessor guards, since they are ignored by the compiler if the OpenMP compiler flag is omitted.

#ifdef MPI
  use mpi
#endif
#ifdef OPENMP
  use omp_lib
#endif
...
#ifdef MPI
  call MPI_BARRIER(mpicomm, ierr)
#endif

#ifdef OPENMP
  me = OMP_GET_THREAD_NUM()
#else
  me = 0
#endif

2.8. Scientific Documentation Rules

Scientific and technical documents are important for code maintenance and for fostering understanding among stakeholders. As such, physics schemes are required to include scientific documentation in order to be included in the CCPP. This section describes the process used for documenting parameterizations in the CCPP.

Doxygen was chosen as a tool for generating human-readable output due to its built-in functionality with Fortran, its high level of configurability, and its ability to parse inline comments within the source code. Keeping documentation with the source itself increases the likelihood that the documentation will be updated along with the underlying code. Additionally, inline documentation is amenable to version control.

The purpose of this section is to provide an understanding of how to properly document a physics scheme using doxygen inline comments in the Fortran code and metadata information contained in the .meta files. It covers what kind of information should be in the documentation, how to mark up the inline comments so that doxygen will parse them correctly, where to put various comments within the code, how to include information from the .meta files, and how to configure and run doxygen to generate HTML output. For an example of the HTML rendering of the CCPP Scientific Documentation, see https://dtcenter.ucar.edu/GMTB/v6.0.0/sci_doc/index.html Part of this documentation, namely metadata about subroutine arguments, has functional significance as part of the CCPP infrastructure. The metadata must be in a particular format to be parsed by Python scripts that “automatically” generate a software cap for a given physics scheme. Although the procedure outlined herein is not unique, following it will provide a level of continuity with previous documented schemes.

Reviewing the documentation for CCPP parameterizations is a good way of getting started in writing documentation for a new scheme.

2.8.1. Doxygen Comments and Commands

All doxygen commands start with a backslash (”\”) or an at-sign (”@”). The doxygen inline comment blocks begin with “!>”, and subsequent lines begin with “!!”, which means that regular Fortran comments using “!” are not parsed by doxygen.

In the first line of each Fortran file, a brief one-sentence overview of the file’s purpose should be included, using the doxygen command \file:

!> \file cires_ugwp.F90
!! This file contains the Unified Gravity Wave Physics (UGWP) scheme by Valery Yudin (University of Colorado, CIRES)

A parameter definition begins with “!<”, where the “<” sign tells Doxygen that documentation follows. Example:

integer, parameter, public :: NF_VGAS = 10   !< number of gas species
integer, parameter         :: IMXCO2  = 24   !< input CO2 data longitude points
integer, parameter         :: JMXCO2  = 12   !< input CO2 data latitude points
integer, parameter         :: MINYEAR = 1957 !< earlist year 2D CO2 data available

2.8.2. Doxygen Documentation Style

To document a physics suite, a broad array of information should be included in order to serve both software engineering and scientific purposes. The documentation style could be divided into four categories:

  • Doxygen Files

  • Doxygen Pages (overview page and scheme pages)

  • Doxygen Modules

  • Bibliography

2.8.2.1. Doxygen files

Doxygen provides the \file tag as a way to provide documentation on the level of Fortran source code files. That is, in the generated documentation, one may navigate by source code filenames (if desired) rather than through a “functional” navigation. The most important documentation organization is through the “module” concept mentioned below, because the division of a scheme into multiple source files is often functionally irrelevant. Nevertheless, using a \file tag provides an alternate path to navigate the documentation and it should be included in every source file. Therefore, it is prudent to include a small documentation block to describe what code is in each file using the \file tag, e.g.:

!>\file cu_gf_deep.F90
!! This file is the Grell-Freitas deep convection scheme.

The brief description for each file is displayed next to the source filename on the doxygen-generated “File List” page:

_images/DoxygenFileList.png

2.8.2.2. Doxygen Overview Page

Pages in Doxygen can be used for documentation that is not directly attached to a source code entity such as a file or module. In the context of CCPP they are used for external text files that generate pages with a high-level scientific overview, typically containing a longer description of a project or suite. You can refer to any source code entity from within a page.

The DTC maintains a main page, created by the Doxygen command \mainpage, which contains an overall description and background of the CCPP. Physics developers do not have to edit the file with the mainpage (mainpage.txt), which is formatted like this:

/**
\mainpage Introduction
...
*/

All other pages listed under the main page are created using the Doxygen tag \page described in the next section. In any Doxygen page, you can refer to any entity of source code by using Doxygen tag \ref or @ref. Example from suite_FV3_GFS_v16.txt:

/**
\page GFS_v16_page GFS_v16 Suite

\section gfsv16_suite_overview Overview

Version 16 of the Global Forecast System (GFS) was implemented operationally by the NOAA
National Centers for Environmental Prediction (NCEP) in 2021. This suite is available for
use with the UFS SRW App and with the CCPP SCM.

The GFS_v16 suite uses the parameterizations in the following order:
 - \ref GFS_RRTMG
 - \ref GFS_SFCLYR
 - \ref GFS_NSST
 - \ref GFS_OCEAN
 - \ref GFS_NOAH
 - \ref GFS_SFCSICE
 - \ref GFS_SATMEDMFVDIFQ
 - \ref GFS_UGWP_v0
 - \ref GFS_OZPHYS
 - \ref GFS_H2OPHYS
 - \ref GFS_SAMFdeep
 - \ref GFS_SAMFshal
 - \ref GFDL_cloud

\section sdf_gfsv16b Suite Definition File
\include  suite_FV3_GFS_v16.xml
...
*/

The HTML result of this Doxygen code can be viewed here. You can see that the - symbols at the start of a line generate a list with bullets, and the \ref commands generate links to the appropriately labeled pages. The \section comands indicate section breaks, and the \include commands will include the contents of another file.

Other valid Doxygen commands for style, markup, and other functionality can be found in the Doxygen documentation.

2.8.2.3. Physics Scheme Pages

Each major scheme in CCPP should have its own scheme page containing an overview of the parameterization. These pages are not tied to the Fortran code directly; instead, they are created with a separate text file that starts with the command \page. For CCPP, the stand-alone Doxygen pages, including the main page and the scheme pages, are contained in the ccpp-physics repository, under the ccpp-physics/physics/docs/pdftxt/ directory. Each page (aside from the main page) has a label (e.g., “GFS_SAMFdeep” in the following example) and a user-visible title (“GFS Scale-Aware Simplified Arakawa-Schubert (sa-SAS) Deep Convection Scheme” in the following example). It is noted that labels must be unique across the entire doxygen project so that the \ref command can be used to create an unambiguous link to the structuring element. It therefore makes sense to choose label names that refer to their context.

/**
\page GFS_SAMFdeep GFS Scale-Aware Simplified Arakawa-Schubert (sa-SAS) Deep Convection Scheme
\section des_deep Description
 The scale-aware mass-flux (SAMF) deep convection scheme is an
 updated version of the previous Simplified Arakawa-Schubert (SAS) scheme
 with scale and aerosol awareness and parameterizes the effect of deep
 convection on the environment (represented by the model state variables)
 in the following way...
...
\section intra_deep  Intraphysics Communication
\ref arg_table_samfdeepcnv_run

\section gen_al_deep General Algorithm
\ref general_samfdeep

*/

The physics scheme page will often describe the following:

  1. A “Description” section, which usually includes:

    • Scientific origin and scheme history

      • External sources and citations can be referenced with \cite tags

    • Key features and differentiating points compared to other schemes

    • Tables, schematics, other images inserted using the \image tag

      • To insert images into doxygen documentation, you’ll need to prepare your images in a graphical format, such as Portable Network Graphic (png), depending on which type of doxygen output you are planning to generate. For example, for LaTeX output, the images must be provided in Encapsulated PostScript (.eps), while for HTML output the images can be provided in the png format. Images are stored in ccpp-physics/physics/docs/img directory. Example of including an image for HTML output:

        \image  html  gfdl_cloud_mp_diagram.png "Figure 1: GFDL MP at a glance (Courtesy of S.J. Lin at GFDL)" width=10cm

  2. An “Intraphysics Communication” section

    The argument table for CCPP entry point subroutine {scheme}_run will be in this section. It is created by inserting a reference link (\ref) to the corresponding Doxygen label in the Fortran code for the scheme. In the above example, the \ref arg_table_samfdeepcnv_run tag references the section of Doxygen-annotated source code in ccpp-physics/physics/samfdeepcnv.f that contains the scheme’s argument table as an included html document, as described in the following section.

  3. A “General Algorithm” section

    The general description of the algorithm will be in this section. It is created by inserting a reference link (\ref) pointing to the corresponding Doxygen-annotated source code for the scheme, as described in the following section.

As can be seen in the above examples, symbols /\*\* and */ need to be the first and last entries of the page.

Note that separate pages can also be created to document something that is not a scheme. For example, a page could be created to describe a suite, or how a set of schemes work together. Doxygen automatically generates an index of all pages that is visible at the top-level of the documentation, thus allowing the user to quickly find, and navigate between, the available pages.

2.8.2.4. Doxygen Modules

The CCPP documentation is based on doxygen modules (note this is not the same as Fortran modules). Each doxygen module pertains to a particular parameterization and is used to aggregate all code related to that scheme, even when it is in separate files. Since doxygen cannot know which files or subroutines belong to each physics scheme, each relevant subroutine must be tagged with the module name. This allows doxygen to understand your modularized design and generate the documentation accordingly. Here is a list of modules defined in CCPP.

A module is defined using:

!>\defgroup group_name group_title

Where group_name is the identifier and the group_title is what the group is referred to in the output. In the example below, we’re defining a parent module “GFS radsw Main”:

!> \defgroup module_radsw_main GFS radsw Main
!! This module includes NCEP's modifications of the RRTMG-SW radiation
!! code from AER.
!! ...
!!\author   Eli J. Mlawer, emlawer@aer.com
!!\author   Jennifer S. Delamere, jdelamer@aer.com
!!\author   Michael J. Iacono, miacono@aer.com
!!\author   Shepard A. Clough
!!\version NCEP SW v5.1  Nov 2012 -RRTMG-SW v3.8
!!

One or more contact persons should be listed with author. If you make significant modifications or additions to a file, consider adding an author and a version line for yourself. The above example generates the Author, Version sections on the page. All email addresses are converted to mailto hypertext links automatically:

Author

Eli J. Mlawer, emlawer@aer.com

Jennifer S. Delamere, jdelamer@aer.com

Michael J. Iacono, miacono@aer.com

Shepard A. Clough

Version

NCEP SW v5.1 Nov 2012 -RRTMG-SW v3.8

In order to include other pieces of code in the same module, the following tag must be used at the beginning of a comment block:

\ingroup group_name

For example:

!>\ingroup module_radsw_main
!> The subroutine computes the optical depth in band 16:  2600-3250
!! cm-1 (low - h2o,ch4; high - ch4)
!-----------------------------------
      subroutine taumol16
!...................................

In the same comment block where a group is defined for a physics scheme, there should be some additional documentation. First, using the \brief command, a brief one or two sentence description of the scheme should be included. After a blank doxygen comment line, begin the scheme origin and history using \version, \author and \date.

Each subroutine that is a CCPP entry point to a parameterization should be further documented with a documentation block immediately preceding its definition in the source. The documentation block should include at least the following components:

  • A brief one- or two-sentence description with the \brief tag

  • A more detailed one or two paragraph description of the function of the subroutine

  • A comment indicating that metadata information about the subroutine arguments follows (in this example, the subroutine is called SUBROUTINE_NAME. Note that this line is also functional documentation used during the CCPP prebuild step.

!! \section arg_table_SUBROUTINE_NAME Argument Table

  • A second comment indicating that a table of metadata to describe the subroutine arguments will be included from a separate file in HTML format (in this case, file SUBROUTINE_NAME.html). Please refer to the section below for information on how to generate the HTML files with metadata information from the .meta files.

    The argument table should be immediately followed by a blank doxygen line “!!”.

!! \htmlinclude SUBROUTINE_NAME.html
!!

  • A section called “General Algorithm” with a bullet or numbered list of the tasks completed in the subroutine algorithm

  • At the end of initial subroutine documentation block, a “Detailed algorithm” section is started and the entirety of the code is encompassed with the !> @{ and !> @} delimiters. This way, any comments explaining detailed aspects of the code are automatically included in the “Detailed Algorithm” section.

For subroutines that are not a CCPP entry point to a scheme, no inclusion of metadata information is required. But it is suggested that following \ingroup and \brief, use \param to define each argument with local name, a short description and unit, i.e.,

!>  \ingroup HEDMF
!!  \brief This subroutine is used for calculating the mass flux and updraft properties.
!!  ...
!!
!!  \param[in] im      integer, number of used points
!!  \param[in] ix      integer, horizontal dimension
!!  \param[in] km      integer, vertical layer dimension
!!  \param[in] ntrac   integer, number of tracers
!!  \param[in] delt    real, physics time step
!!  ...
!!  \section general_mfpbl mfpbl General Algorithm
!!  -# Determine an updraft parcel's entrainment rate, buoyancy, and vertical velocity.
!!  -# Recalculate the PBL height ...
!!  -# Calculate the mass flux profile and updraft properties.
!!  \section detailed_mfpbl mfpbl Detailed Algorithm
!>  @{
       subroutine mfpbl(im,ix,km,ntrac,delt,cnvflg,                       &
       &   zl,zm,thvx,q1,t1,u1,v1,hpbl,kpbl,                              &
       &   sflx,ustar,wstar,xmf,tcko,qcko,ucko,vcko)
         …
       end subroutine mfpbl
!>  @}

2.8.2.5. Bibliography

Doxygen can handle in-line paper citations and link to an automatically created bibliography page. The bibliographic data for any papers that are cited need to be put in BibTeX format and saved in a .bib file. The .bib file for CCPP is included in the CCPP Physics repository (ccpp-physics/physics/docs/library.bib), and the doxygen configuration option cite_bib_files points to the included file.

Citations are invoked with the following tag:

\cite bibtex_key_to_paper

2.8.2.6. Equations

See the Doxygen documentation for information about including equations. For the best rendering, the following option should be set in the Doxygen configuration file:

USE_MATHJAX            = YES
MATHJAX_RELPATH        = https://cdn.jsdelivr.net/npm/mathjax@2

There are many great online resources to use the LaTeX math typesetting used in doxygen.

2.8.3. Doxygen Configuration

2.8.3.1. Configuration File

The CCPP is distributed with a doxygen configuration file ccpp-physics/physics/physics/docs/ccpp_doxyfile, such that you don’t need to create an additional one.

Doxygen files for layout (ccpp_dox_layout.xml), HTML style (doxygen-awesome-ccpp.css), and the bibliography (library.bib) are provided with the CCPP. Additionally, a configuration file is supplied, with the following variables modified from the default:

2.8.3.2. Diagrams

On its own, Doxygen is capable of creating simple text-based class diagrams. With the help of the additional software GraphViz, Doxygen can generate additional graphics-based diagrams, optionally in Unified Modeling Language (UML) style. To enable GraphViz support, the configure file parameter “HAVE_DOT” must be set to “YES”.

You can use doxygen to create call graphs of all the physics schemes in CCPP. In order to create the call graphs you will need to set the following options in your doxygen config file:

HAVE_DOT             = YES
EXTRACT_ALL          = YES
EXTRACT_PRIVATE      = YES
EXTRACT_STATIC       = YES
CALL_GRAPH           = YES

Note that you will need the DOT (graph description language) utility to be installed when starting doxygen. Doxygen will call it to generate the graphs. On most distributions the DOT utility can be found in the GraphViz package. Here is the call graph for subroutine mpdrv in GFDL cloud microphysics generated by doxygen:

_images/DoxygenCallGraph.png

2.8.4. Including metadata information

As described above, a table of metadata information should be included in the documentation for every CCPP entrypoint scheme. Before doxygen is run, the table for each scheme must be manually created in separate files in HTML format, with one file per scheme. The HTML files are included in the Fortran files using the doygen markup below.

!! \htmlinclude SUBROUTINE_NAME.html
!!

The tables should be created using a Python script distributed with the CCPP Framework, ccpp-framework/scripts/metadata2html.py.

Note

You will need to set the environment variable PYTHONPATH to include the directories ccpp/framework/scripts and ccpp/framework/scripts/parse_tools. As an example for bash-like shells:

export PYTHONPATH=`pwd`/ccpp/framework/scripts:`pwd`/ccpp/framework/scripts/parse_tools

For the example of the SCM, where both scripts need to be called from the host model top-level directory:

./ccpp/framework/scripts/metadata2html.py -m ccpp/physics/physics/file.meta -o ccpp/physics/physics/docs

where -m is used to specify a file with metadata information and -o is used to specify the directory for output. Note that a single input file (.meta) may have more than one CCPP entrypoint scheme, and therefore can be used to generate more than one HTML file.

Note that the .meta files are supplied in the CCPP Physics repository, and that there is a .meta file for each Fortran file that contains one or more CCPP entrypoint scheme. The .meta files are located in the same directory as the scheme Fortran files (ccpp-physics/physics).

To generate the complete Scientific Documentation, the script ./ccpp/framework/scripts/metadata2html.py must be run separately for each .meta file available in ccpp-physics/physics. Alternatively, a batch mode exists that converts all metadata files associated with schemes and variable definitions in the CCPP prebuild config; again using the SCM as an example:

./ccpp/framework/scripts/metadata2html.py -c ccpp/config/ccpp_prebuild_config.py

Note that the options -c and -m are mutually exclusive, but that one of them is required. The option -m also requires the user to specify -o, while the option -c will ignore -o. For more information, use

./ccpp/framework/scripts/metadata2html.py --help

2.8.5. Using Doxygen

In order to generate the doxygen-based documentation, you will need to follow five steps:

  1. Have the executables doxygen (https://doxygen.nl/), graphviz (https://graphviz.org/), and bibtex (http://www.bibtex.org/) installed on your machine and in your PATH. These utilities can be installed on MacOS via Homebrew, or installed manually via the instructions on each utility’s page linked above.

  2. Document your code, including the doxygen main page, scheme pages, and inline comments within the source code as described above.

  3. Run metadata2html.py to create files in HTML format containing metadata information for each CCPP entrypoint scheme.

  4. Prepare a Bibliography file in BibTex format for papers or other references cited in the physics suites.

  5. Create or edit a doxygen configuration file to control which doxygen pages, source files, and bibliography get parsed, in addition to how the source files get parsed, and to customize the output.

  6. Run doxygen from the directory ccpp/physics/physics/docs using the command line to specify the doxygen configuration file as an argument. For the CCPP Scientific documentation, this file is called ccpp_doxyfile:

doxygen ccpp_doxyfile

Running this command may generate warnings or errors that need to be fixed in order to produce proper output. The location and type of output (HTML, LaTeX, etc.) are specified in the configuration file. The generated HTML documentation can be viewed by pointing an HTML browser to the index.html file in the ./docs/doc/html/ directory.

For precise instructions or other help creating the scientific documentation, visit the CCPP GitHub discussions page at https://github.com/NCAR/ccpp-physics/discussions