dartwrf package

Subpackages

Submodules

dartwrf.assimilate module

dartwrf.assimilate.archive_adapt_inflation(cfg, time)
dartwrf.assimilate.archive_filter_diagnostics(cfg, time, f_out_pattern)

Copy the filter output txt to the archive

dartwrf.assimilate.archive_filteroutput(cfg, time)

Archive filter output files (filter_restart, preassim, postassim, output_mean, output_sd)

dartwrf.assimilate.evaluate(cfg, assim_time, obs_seq_out=False, prior_is_filter_output=False, f_out_pattern: str = './obs_seq.final')

Calculates either prior or posterior obs space values.

Note: Depends on a prepared input_list.txt, which defines the ensemble (prior or posterior).

Output file: Saves obs_seq.final to /archivedir/obs_seq_final/

Parameters:

  • assim_time (datetime) – time of assimilation

  • obs_seq_out (str, optional) – use the argument as obs_seq.out file, defaults to use the existing obs_seq.out file at these observations, the posterior will be evaluated

  • f_out_pattern (str, mandatory) – output filename

  • prior_is_filter_output (bool, optional) – if True, use the filter output as prior, else use already linked ensemble files

Returns

None (writes file)

dartwrf.assimilate.filter(cfg)

Calls DART ./filter program

Parameters:

nproc (int) – number of cores for use in ./filter call

Returns:

None (writes to file)

dartwrf.assimilate.get_obsseq_out(cfg, time, prior_path_exp, prior_init_time, prior_valid_time)

Prepares an obs_seq.out file in the run_DART folder

Options: 1) Use existing obs_seq.out file 2) Use precomputed FO (e.g. cloud fraction) 3) Generate new observations from nature run with new noise

Parameters:

  • time (datetime) – time of assimilation

  • prior_path_exp (str) – path to the experiment folder

  • prior_init_time (datetime) – time of the prior forecast init

  • prior_valid_time (datetime) – time of the prior

Returns:

obsseq.ObsSeq

dartwrf.assimilate.get_parametrized_error(obscfg, osf_prior) → numpy.ndarray

Calculate the parametrized error for an ObsConfig (one obs type)

Parameters:

  • obscfg (object) – Configuration of observations

  • osf_prior (obsseq.ObsRecord) – Contains truth and prior values from obs_seq.final (output of ./final in evaluate-mode (no posterior))

Returns:

observation error std-dev for assimilation

Return type:

np.array

dartwrf.assimilate.main(cfg: Config)

Assimilate observations as defined in config/cfg.py for a certain timestamp (argument) of the nature run (defined in config/clusters.py)

Workflow: 1) prepare nature run & prior ensemble for DART 2) create obs_seq.in 3) create obs from nature (obs_seq.out) or use existing one optional: quality-control 4) Assimilate with assigned errors 5) Evaluate posterior (after DART clamping, e.g. setting negative cloud water to zero)

Parameters:

  • assim_time (dt.datetime) – time of output

  • prior_init_time (dt.datetime) – forecast start of prior

  • prior_valid_time (dt.datetime) – valid time of prior (may be different to assim_time)

  • prior_path_exp (str) – path to prior experiment

Returns:

None

dartwrf.assimilate.prepare_DART_grid_template(cfg)

Prepare DART grid template wrfinput_d01 file from a prior file

DART needs a wrfinput file as a template for the grid information. No data except grid info will be read from this file. The grid information must match exactly with the prior file “wrfout_d01”

dartwrf.assimilate.prepare_adapt_inflation(cfg, time, prior_init_time)

Prepare inflation files (spatially varying)

Recycles inflation files from previous assimilations or takes default files from archive.

Parameters:

  • time (datetime) – time of assimilation

  • prior_init_time (datetime) – time of prior assimilation

dartwrf.assimilate.prepare_prior_ensemble(cfg, assim_time, prior_init_time, prior_valid_time, prior_path_exp)

Prepares DART files for running filter i.e. - links first guess state to DART first guess filenames - creates wrfinput_d01 files (grid file, not a real wrfinput file) - adds geo-reference (xlat,xlon) coords so that DART can deal with the files - writes txt files so DART knows what input and output is - removes probably pre-existing files which could lead to problems

dartwrf.assimilate.prepare_run_DART_folder(cfg: Config)
dartwrf.assimilate.reject_small_FGD(cfg, time, oso)

Quality control of observations We assume that the prior values have been evaluated and are in run_DART/obs_seq.final

Parameters:

  • time (datetime) – time of the assimilation

  • oso (ObsSeq) – python representation of obs_seq.out file, will be modified and written to file

Returns:

None (writes to file) The pre-existing obs_seq.out will be archived. The new obs_seq.out will be written to the DART run directory.

dartwrf.assimilate.set_obserr_assimilate_in_obsseqout(cfg, oso, outfile='./obs_seq.out')

“Overwrite existing variance values in obs_seq.out files

Parameters:

oso (ObsSeq) – python representation of obs_seq.out file, will be modified and written to file

Returns:

None (writes to file)

Variables:
osf_prior (ObsSeq): python representation of obs_seq.final (output of filter in evaluate-mode without posterior)

contains prior values; used for parameterized errors

For reproducibility, write the path of the prior to a txt file

dartwrf.assimilate.use_filter_output_as_prior(cfg)

Use the last posterior as input for DART, e.g. to evaluate the analysis in observation space

dartwrf.assimilate.use_linked_files_as_prior(cfg)

Instruct DART to use the prior ensemble as input

dartwrf.assimilate.write_list_of_outputfiles(cfg)

dartwrf.create_wbubble_wrfinput module

dartwrf.create_wbubble_wrfinput.main(cfg)

dartwrf.dart_nml module

dartwrf.dart_nml.read_namelist(filepath: str) → dict

Read the DART namelist file into a dictionary.

Parameters:

filepath (str) – Path to namelist file

Returns:

namelist[section][parameter] = [[arg1, arg2,], [arg3, arg4]]

Return type:

dict

dartwrf.dart_nml.write_namelist(cfg: Config, just_prior_values=False) → dict

Write a DART namelist file (‘input.nml’)

  1. Uses the default namelist (from the DART source code)

  2. Calculates localization parameters from the experiment configuration

  3. Overwrites other parameters as defined in the experiment configuration

  4. Writes the namelist to the DART run directory

Note

Vertical localization in pressure or levels is not implemented.

Parameters:

just_prior_values (bool, optional) – If True, only compute prior values, not posterior. Defaults to False.

Raises:

ValueError – If both height and scale-height localization are requested

Returns:

None

dartwrf.dart_nml.write_namelist_from_dict(d, filepath)

Write a DART namelist dictionary to a file.

Parameters:

  • d (dict) – keys are namelist sections, values are dictionaries. these dictionaries contain keys (=parameters) and values (list type) every item in values is a line (list type) every line contains possibly multiple entries

  • filepath (str) – Path to namelist file

Note

The namelist dictionary should have the following structure:

>>> d = {'&section1': {'parameter1': [['arg1', 'arg2'], ['arg3', 'arg4']],},}
Returns:

None (writes to file)

dartwrf.namelist_handler module

class dartwrf.namelist_handler.WRF_namelist

Bases: object

Class to read and write WRF namelist files

Example

n = WRF_namelist(‘/path_to_existing/namelist.input’) n.read()

n.namelist # is a dictionary n.namelist[‘time_control’][‘run_days’] = 1

n.write(‘/path_to_new/namelist.input’)

read(fname: str) → None

Read a WRF namelist file and store it in a dictionary

write(fname: str) → None

Write a WRF namelist file may overwrite existing files, not guaranteed to overwrite

dartwrf.prep_IC_prior module

dartwrf.prep_IC_prior.create_updated_wrfinput_from_wrfout(cfg, time: datetime, prior_init_time: datetime, prior_path_exp: str, new_start_time: datetime) → None

Create a new wrfinput file from wrfout file Output is created inside the WRF run directory

Parameters:

  • time – time of the wrfout file

  • prior_init_time – initial time of the prior run

  • prior_path_exp – path to the prior run

  • new_start_time – time of the new wrfinput file If provided, overwrites the valid time of the initial conditions; This hack allows you to use a prior of a different time than your forecast start time. Usually, you don’t want to do this.

dartwrf.prep_IC_prior.create_wrfrst_in_WRF_rundir(cfg, time: datetime, prior_init_time: datetime, prior_path_exp: str) → None

Copy WRF restart files to run_WRF directory These files will be used as initial conditions for the next WRF run

dartwrf.prep_IC_prior.main(cfg)

dartwrf.prepare_namelist module

Create namelist.input files

Usage: prepare_namelist.py <config>

dartwrf.prepare_namelist.run(cfg: Config) → None

Prepare WRF namelist files.

Note

Important parameters are: restart, restart_interval, hist_interval_s, WRF_start, WRF_end, WRF_namelist_template

dartwrf.prepare_wrfrundir module

Prepare WRF run directories, to use wrf.exe later Creates a directory where WRF will run (temporary); Creates softlinks from WRF’s run/ directory; Links executables set in cfg.ideal and cfg.wrfexe; If exp.input_profile is set, links the input profiles.

param init_time:

Initialization time in format YYYY-MM-DD_HH:MM

type init_time:

str

returns:

None

dartwrf.prepare_wrfrundir.run(cfg: Config)

Prepare WRF run directories, to use ideal.exe or wrf.exe later

dartwrf.print_config module

Script to pretty-pring a config file (pickle format)”

Usage:

python print_config.py <config_file>

dartwrf.update_IC module

dartwrf.update_IC.update_initials_in_WRF_rundir(cfg: Config) → None

Updates wrfrst-files in /run_WRF/ directory with posterior state from ./filter output, e.g. filter_restart_d01.0001

dartwrf.utils module

Utility functions for DART-WRF

Caution: You can not use the configuration files in here, because loading this would lead to a circular import.

class dartwrf.utils.Config(name: str, model_dx: int, ensemble_size: int, max_nproc: int = 20, max_nproc_for_each_ensemble_member: int = 9, dir_archive: str = './sim_archive/<exp>/', dir_wrf_run: str = './run_WRF/<exp>/<ens>/', dir_dart_run: str = './run_DART/<exp>/', use_slurm: bool = False, pattern_obs_seq_out: str = './diagnostics/%Y-%m-%d_%H:%M_obs_seq.out', pattern_obs_seq_final: str = './diagnostics/%Y-%m-%d_%H:%M_obs_seq.final', update_vars: list = [], dart_nml: dict = {}, assimilate_existing_obsseq: bool | str = False, nature_wrfout_pattern: bool | str = False, **kwargs)

Bases: object

Collection of variables which define the experiment

Parameters:

  • expname (str) – Name of the experiment

  • model_dx (int) – WRF grid spacing in meters

  • ensemble_size (int) – Ensemble size

  • nature_wrfout_pattern (str) – Path to the nature run WRF files, where can be generated from; the path can contain wildcards (,?), e.g. ‘/jetfs/exp1//1/wrfout_d01_%Y-%m-%d_%H:%M:%S’

  • input_profile (str) – Path to sounding profiles as initial condition (see WRF ideal guide) e.g. ‘/data/initial_profiles/wrf/ens/raso.fc.<iens>.wrfprof’; <iens> is replaced by 001-040 for a 40-member ensemble

  • update_vars (list of str) – Variables which will be updated after assimilation (update_IC.py) e.g. [‘U’, ‘V’, ‘W’, ‘THM’, ‘PH’, ‘MU’, ‘QVAPOR’,]

  • observations (list of dict) – Each dictionary defines one observation type The keys should be (optional): kind: Identifier of the observation type as defined in DART; error_generate: measurement error standard-deviation; error_assimilate: assigned observation error std-dev; heights: list of integers at which observations are taken; loc_horiz_km: float of horizontal localization half-width in km; loc_vert_km: float of vertical localization half-width in km;

  • assimilate_existing_obsseq (str, False) – Path to existing obs_seq.out file (False: generate new one); time string is replaced by actual time: /path/%Y-%m-%d_%H:%M_obs_seq.out

  • dart_nml (dict) – Updates to the default input.nml of DART. Keys are namelist section headers (e.g. &filter_nml). Values are dictionaries of parameters and values (e.g. dict(ens_size=10,))

  • dir_archive (str) – E.g. ‘/jetfs/home/lkugler/data/sim_archive/<exp>/’

  • dir_wrf_run (str) – E.g. ‘/jetfs/home/lkugler/data/run_WRF/<exp>/<ens>/’

  • dir_dart_run (str) – E.g. ‘/jetfs/home/lkugler/data/run_DART/<exp>/’

  • geo_em_forecast (str) – file path to geo_em containing coordinates of the forecast model

  • geo_em_nature (str) – file path to geo_em containing coordinates of the nature simulation

static from_file(fname: str) → Config

Read a configuration from a file

generate_name()
to_file(filename: str)

Write itself to a python file

update(**kwargs)

Update the configuration with new values

class dartwrf.utils.Timer(message='Code block')

Bases: object

A context manager for measuring the execution time of a code block. Prints a message before and after the timed block.

dartwrf.utils.append_file(f_main, f_gets_appended)

Append the contents of one file to another

Parameters:

  • f_main (str) – Path to file that will be appended

  • f_gets_appended (str) – Path to file that will be appended to f_main

Returns:

None

dartwrf.utils.copy(src, dst, remove_if_exists=True)
dartwrf.utils.copy_contents(src, dst)
dartwrf.utils.display_config(filename: str) → None
dartwrf.utils.import_from_path(module_name, file_path)

Create symbolic links for all files in src to dst

Parameters:

  • src (str) – Path to source directory

  • dst (str) – Path to destination directory

Returns:

None

dartwrf.utils.load_dict(fpath)
dartwrf.utils.mkdir(path)
dartwrf.utils.obskind_read(dart_srcdir: str) → dict

Read dictionary of observation types + ID numbers (“kind”) from DART f90 script and return it as python dictionary

dartwrf.utils.print(*args)
dartwrf.utils.read_pickle(filename: str) → dict

Read a dictionary from a python file, return as Config object

dartwrf.utils.read_yaml(filename: str) → dict
dartwrf.utils.run_bash_command_in_directory(command, directory)

Runs a Bash command in the specified directory.

Parameters:

  • command (str) – The Bash command to execute.

  • directory (str) – The path to the directory where the command should be run.

Returns:

An object containing information about the executed command.

Return type:

subprocess.CompletedProcess

dartwrf.utils.save_dict(dictionary, fpath)
dartwrf.utils.script_to_str(path)
dartwrf.utils.sed_inplace(filename, pattern, repl)

Perform the pure-Python equivalent of in-place sed substitution Like sed -i -e ‘s/’${pattern}’/’${repl}’ “${filename}”.

Parameters:

  • filename (str) – path to file

  • pattern (str) – string that will be replaced

  • repl (str) – what shall be written instead of pattern?

Example

sed_inplace(‘namelist.input’, ‘<dx>’, str(int(exp.model_dx)))

Returns:

None

dartwrf.utils.shell(args, pythonpath=None)

Create a symbolic link from src to dst Creates the folder if it does not exist

dartwrf.utils.try_remove(f)
dartwrf.utils.write_txt(lines, fpath)

Write a list of strings to a text file

Parameters:

  • lines (list) – List of strings

  • fpath (str) – Path to file

Returns:

None

dartwrf.workflows module

These functions call python scripts via the shell, e.g. assimilate() calls dartwrf/assimilate.py through the shell.

This would not be necessary, but some users might want to use queueing systems (e.g. SLURM) which must call scripts.

class dartwrf.workflows.WorkFlows(cfg: Config)

Bases: object

assimilate(cfg, depends_on=None)

Creates observations from a nature run and assimilates them.

Returns:

job ID of the submitted job

Return type:

str

evaluate_obs_posterior_after_analysis(cfg, depends_on=None)
generate_obsseq_out(cfg, depends_on=None)

Creates observations from a nature run for a list of times

Parameters:

times (list) – list of datetime objects

Returns:

job ID of the submitted job

Return type:

str

prepare_IC_from_prior(cfg: Config, depends_on=None)

Create initial conditions from prior wrfrst files

Parameters:

  • prior_path_exp (str) – Path to experiment which provides the prior

  • prior_init_time (dt.datetime) – Timestamp of the prior’s initialization (directory of prior wrfrst files)

  • prior_valid_time (dt.datetime) – Timestamp of prior wrfrst files

  • new_start_time (dt.datetime, optional) – If provided, overwrites the valid time of the initial conditions; This hack allows you to use a prior of a different time than your forecast start time. Usually, you don’t want to do this.

  • depends_on (str, optional) – job ID of a previous job after which to run this job

Returns:

job ID of the submitted job

Return type:

str

prepare_WRFrundir(cfg)

Prepare WRF run directories for all ensemble members

Note

Optionally copy input sounding profiles to WRF run directories if defined in cfg.py

Parameters:

init_time (datetime) – WRF initialization time

Returns:

None

run_RTTOV(cfg, depends_on=None)

Run a job array, one job per ensemble member, to create satellite images

run_WRF(cfg, depends_on=None)

Run the forecast ensemble

Note

Set the following parameters in cfg: WRF_start: start time of the WRF simulation; WRF_end: end time of the WRF simulation; hist_interval_s (int): history output frequency in seconds; restart: whether it uses a restart file; restart_interval: interval in minutes to write restart files;

Returns:

job ID of the submitted job

Return type:

str

run_ideal(cfg, depends_on=None)

Run WRF’s ideal.exe for every ensemble member

Parameters:

depends_on (str, optional) – job ID of a previous job after which to run this job

Returns:

job ID of the submitted job

Return type:

str

run_job(cmd, cfg, depends_on=None, **kwargs)

Run scripts in a shell

If not using SLURM: calls scripts through shell if using SLURM: uses slurmpy to submit jobs, keep certain default kwargs and only update some with kwarg overwrite_these_configurations

Parameters:

  • cmd (str) – Bash command(s) to run

  • jobname (str, optional) – Name of SLURM job

  • cfg_update (dict) – The config keywords will be overwritten with values

  • depends_on (int or None) – SLURM job id of dependency, job will start after this id finished.

Returns

None

update_IC_from_DA(cfg, depends_on=None)

Update existing initial conditions with the output from the assimilation

Parameters:

  • assim_time (dt.datetime) – Timestamp of the assimilation

  • depends_on (str, optional) – job ID of a previous job after which to run this job

Returns:

job ID of the submitted job

Return type:

str

verify(cfg: Config, init=False, depends_on=None)

Not included in DART-WRF

wrfinput_insert_wbubble(cfg, depends_on=None)

Inserts warm-bubble temperature perturbations into wrfinput files

Note

Assumes that WRF run directories with wrfinput files exist.

Parameters:

  • perturb (bool, optional) – if True, perturb the location of the warm-bubble (False: nature run)

  • depends_on (str, optional) – job ID of a previous job after which to run this job

Returns:

job ID of the submitted job

Return type:

str

dartwrf.wrfinput_add_geo module

Add geogrid data to wrfinput. DART needs a georeference, but ideal.exe does not provide it

Takes LAT,LON, mapfac from geogrid, so that they are consistent. Does not change E, F, HGT_M as they would alter the dynamics and have no impact on assimilation

example call:

./wrfinput_add_geo.py geo_em.d01.nc wrfinput_d01

dartwrf.wrfinput_add_geo.run(cfg: Config) → None

dartwrf.wrfout_add_geo module

dartwrf.wrfout_add_geo.run(geo_data_file: str, wrfout_file: str, path_ncks: str) → None

Add geogrid data to a wrfout file DART needs a georeference, but ideal.exe does not provide it

Takes LAT,LON, mapfac from geogrid, so that they are consistent. Does not change E, F, HGT_M as they would alter the dynamics and have no impact on assimilation

Parameters:

  • cfg – geo_em_forecast (str), ncks (str)

  • attributes (needs) – geo_em_forecast (str), ncks (str)

  • wrfout_file (str) – Path to WRF history (wrfout) file

Returns:

None

Module contents