scope package

Submodules

scope.cli module

Console script for scope.

scope.cli.yaml_file_to_dict(filepath: str) → dict

Given a scope configuration yaml file, returns a corresponding dictionary.

If you do not give an extension, tries again after appending one:

• .yml
• .yaml
• .YML
• .YAML

Note that this function also uses ~jinja2 to replace any templated variables found in the under the top-level key template_replacements. This key is then deleted from the remainder of the dictionary.

Parameters:filepath (str) – Where to get the YAML file from Returns:A dictionary representation of the yaml file. Return type:dict Raises:OSError if the file cannot be found.

scope.models module

Not sure what to do with this stuff yet…

class scope.models.Component

Bases: scope.models.SimObj

NAME = 'Generic Component Object'

class scope.models.Model

Bases: scope.models.SimObj

NAME = 'Generic Model Object'

class scope.models.SimObj

Bases: object

after_run()

before_recieve()

before_send()

recieve()

send()

NAME = 'Generic Sim Object'

scope.scope module

Here, the scope library is described. This allows you to use specific parts of scope from other programs.

scope consists of several main classes. Note that most of them provide Python access to cdo calls via Python’s built-in subprocess module. Without a correctly installed cdo, many of these functions/classes will not work.

We provide a quick summary, but please look at the documentation for each function and class for more complete information. The following functions are defined:

• determine_cdo_openMP – using cdo --version, determines if you have openMP support.

The following classes are defined here:

• Scope – an abstract base class useful for starting other classes from. This provides a way to determine if cdo has openMP support or not by parsing cdo --version. Additionally, it has a nested class which gives you decorators to put around methods for enabling arbitrary shell calls before and after the method is executed, which can be configured via the Scope.config dictionary.
• Send – a class to extract and combine various NetCDF files for further processing.
• Recieve – a class to easily regrid from one model to another, depending on the specifications in the scope_config.yaml

class scope.scope.Recieve(config: dict, whos_turn: str)

Bases: scope.scope.Scope

Parameters:

• config (dict) – A dictionary (normally recieved from a YAML file) describing the scope configuration. An example dictionary is included in the root directory under examples/scope_config.yaml
• whos_turn (str) – An explicit model name telling you which model is currently interfacing with scope e.g. echam or pism.

Warning

This function has a filesystem side-effect: it generates the couple folder defined in config["scope"]["couple_dir"]. If you don’t have permissions to create this folder, the object initialization will fail…

Some design features are listed below:

• ``pre`` and ``post`` hooks

Any appropriately decorated method of a scope object has a hook to call a script with specific arguments and flags before and after the main scope method call. Best explained by an example. Assume your Scope subclass has a method “send”. Here is the order the program will execute in, given the following configuration:

pre_send:
    program: /some/path/to/an/executable
    args:
        - list
        - of
        - arguments
    flags:
        - "--flag value1"
        - "--different_flag value2"

post_send:
    program: /some/other/path
    args:
        - A
        - B
        - C
    flags:
        - "--different_flag value3"

Given this configuration, an idealized system call would look like the example shown below. Note however that the Python program calls the shell and immediately destroys it again, so any variables exported to the environment (probably) don’t survive:

$ ./pre_send['program'] list of arguments --flag value1 --different_flag value2
$ <... python call to send method ...>
$ ./post_send['program'] A B C --different_flag value 3

_calculate_weights(model, type_, interp)

_combine_tmp_variable_files(target_file, source_files)

recieve()

regrid_one_var(model, type_, interp, variable, target_file)

regrid_recieve_from(model, type_, interp, variable, target_file, recv_from)

run_cdos(model, type_, variable, target_file, cdo_commands)

class scope.scope.Scope(config: dict, whos_turn: str)

Bases: object

Base class for various Scope objects. Other classes should extend this one.

Parameters:

• config (dict) – A dictionary (normally recieved from a YAML file) describing the scope configuration. An example dictionary is included in the root directory under examples/scope_config.yaml
• whos_turn (str) – An explicit model name telling you which model is currently interfacing with scope e.g. echam or pism.

Warning

This function has a filesystem side-effect: it generates the couple folder defined in config["scope"]["couple_dir"]. If you don’t have permissions to create this folder, the object initialization will fail…

Some design features are listed below:

• ``pre`` and ``post`` hooks

Any appropriately decorated method of a scope object has a hook to call a script with specific arguments and flags before and after the main scope method call. Best explained by an example. Assume your Scope subclass has a method “send”. Here is the order the program will execute in, given the following configuration:

pre_send:
    program: /some/path/to/an/executable
    args:
        - list
        - of
        - arguments
    flags:
        - "--flag value1"
        - "--different_flag value2"

post_send:
    program: /some/other/path
    args:
        - A
        - B
        - C
    flags:
        - "--different_flag value3"

Given this configuration, an idealized system call would look like the example shown below. Note however that the Python program calls the shell and immediately destroys it again, so any variables exported to the environment (probably) don’t survive:

$ ./pre_send['program'] list of arguments --flag value1 --different_flag value2
$ <... python call to send method ...>
$ ./post_send['program'] A B C --different_flag value 3

class ScopeDecorators

Bases: object

Contains decorators you can use on class methods

static _wrap_hook(self, meth, pre_or_post)

classmethod post_hook(meth)

classmethod pre_hook(meth)

Based upon the self.config, runs a specific system command

Using the method name, you can define

get_cdo_prefix(has_openMP: bool = False)

Return a string with an appropriate cdo prefix for using OpenMP with the -P flag.

Parameters:has_openMP (bool) – Default is False. You can explicitly override the ability of cdo to use the -P flag. If set to True, the config must have an entry under config[scope][number openMP processes] defining how many openMP processes to use (should be an int) Returns:A string which should be used for the cdo call, either with or without -P X, where X is the number of openMP processes to use. Return type:str

class scope.scope.Send(config: dict, whos_turn: str)

Bases: scope.scope.Scope

Subclass of Scope which enables sending of models via cdo. Use the send method after building a Precprocess object.

Parameters:

• config (dict) – A dictionary (normally recieved from a YAML file) describing the scope configuration. An example dictionary is included in the root directory under examples/scope_config.yaml
• whos_turn (str) – An explicit model name telling you which model is currently interfacing with scope e.g. echam or pism.

Warning

This function has a filesystem side-effect: it generates the couple folder defined in config["scope"]["couple_dir"]. If you don’t have permissions to create this folder, the object initialization will fail…

Some design features are listed below:

• ``pre`` and ``post`` hooks

Any appropriately decorated method of a scope object has a hook to call a script with specific arguments and flags before and after the main scope method call. Best explained by an example. Assume your Scope subclass has a method “send”. Here is the order the program will execute in, given the following configuration:

pre_send:
    program: /some/path/to/an/executable
    args:
        - list
        - of
        - arguments
    flags:
        - "--flag value1"
        - "--different_flag value2"

post_send:
    program: /some/other/path
    args:
        - A
        - B
        - C
    flags:
        - "--different_flag value3"

Given this configuration, an idealized system call would look like the example shown below. Note however that the Python program calls the shell and immediately destroys it again, so any variables exported to the environment (probably) don’t survive:

$ ./pre_send['program'] list of arguments --flag value1 --different_flag value2
$ <... python call to send method ...>
$ ./post_send['program'] A B C --different_flag value 3

_all_senders()

A generator giving tuples of the reciever_type (e.g. ice, atmosphere, ocean, solid earth), and the configuration for the reciever type, including variables and corresponding specifications for which files to use and how to process them.

Example

Here is an example for the reciever specification dictionary. See the documentation regarding scope configuration for further information:

temp2:
    files:
        pattern: "{{ EXP_ID }}_echam6_echam_{{ DATE_PATTERN }}.grb"
        take:
            newest: 12
    code_table: "echam6"
aprl:
    files:
        dir: "/work/ollie/pgierz/scope_tests/outdata/echam/"
        pattern: "{{ EXP_ID }}_echam6_echam_{{ DATE_PATTERN }}.grb"
        take:
            newest: 12
    code_table: "/work/ollie/pgierz/scope_tests/outdata/echam/PI_1x10_185001.01_echam.codes"

Yields:

tuple of (str, dict) – The first element of the tuple, reciever_type, is a string describing what sort of model should get this data; e.g. “ice”, “atmosphere”

The second element, reciever_spec, is a dictionary describing which files should be used.

_combine_tmp_variable_files(reciever_type, files_to_combine)

Combines all files in the couple directory for a particular reciever type.

Depending on the configuration, this method combines all files found in the couple_dir which may have been further processed by scope to a file <sender_type>_file_for_<reciever_type>.nc

Parameters:reciever_type (str) – Which reciever the model is sending to, e.g. ice, ocean, atmosphere Returns: Return type:None

Notes

This executes a cdo merge command to concatenate all files found which should be sent to particular model.

_construct_filelist(var_dict)

Constructs a file list to use for further processing based on user specifications.

Parameters:var_dict (dict) – Configuration dictionary for how to handle one specific variable. Returns:A list of files for further processing. Return type:file_list

Example

The variable configuration dictionary can have the following top-level keys:

• files may contain:

  • a filepattern in regex to look for
  • take which files or timesteps to take, either specific, or newest/latest followed by an integer.
  • dir a directory where to look for the files. Note that if this is not provided, the default is to fall back to the top level outdata_dir for the currently sending model.

_make_tmp_files_for_variable(varname, var_dict)

Generates temporary files for further processing with scope.

Given a variable name and a description dictionary of how it should be extracted and processed, this method makes a temporary file, <sender_name>_<varname>_file_for_scope.dat, e.g. echam_temp2_file_for_scope.dat in the couple_dir.

Parameters:

• varname (str) – Variable name as that should be selected from the files
• var_dict (dict) – A configuration dictionary describing how the variable should be extracted. An example is given in _construct_filelist.

Notes

In addition to the dictionary description of files, further information may be added with the following top-level keys:

• code_table describing which GRIB code numbers correspond to which variables. If not given, the fallback value is the value of code_table in the sender configuration.

Converts any input file to nc via cdo. Runs both select and settable.

Returns: Return type:None

_rename_send_as_variables(variable_name, variable_dict)

_run_cdo_for_variable(variable_name, variable_dict)

send()

Selects and combines variables from various file into one single file for futher processing.

• <sender_type>_file_for_<reciever_type> (e.g. atmosphere_file_for_ice.nc)

Parameters:None – Returns: Return type:None

scope.scope.determine_cdo_openMP() → bool

Checks if the cdo version being used supports OpenMP; useful to check if you need a -P flag or not.

Parameters:None – Returns:True if OpenMP is listed in the Features of cdo, otherwise False Return type:bool

scope.scope.determine_fileextension(f: str) → str

scope.scope.get_newest_n_timesteps(f: str, take: int) → str

Given a file, takes the newest n timesteps for further processing.

Parameters:

• f (str) – The file to use.
• take (int) – Number of timesteps to take (newest will be taken, i.e. from the end of the file). Please use a positive value!

Returns:

A string with the path to the new file

Return type:

str

scope.scope.get_oldest_n_timesteps(f: str, take: int) → str

Given a file, takes the oldest n timesteps for further processing.

Parameters:

• f (str) – The file to use.
• take (int) – Number of timesteps to take (oldest will be taken, i.e. from the beginning of the file).

Returns:

A string with the path to the new file

Return type:

str

scope.scope.rename_with_suggested_fileext(f: str) → None

Renames a file with the suggested file extension

scope.scope.suggest_fileext(f: str) → str

Given a file, uses CDO to determine which file extension is should have, and gives back an appropriate string that can be used for renaming.

Module contents

Top-level package for SCOPE.