clowder.util package¶
Submodules¶
clowder.util.clowder_utils module¶
Clowder command line utilities
-
clowder.util.clowder_utils.
existing_branch_groups
(groups, branch, is_remote)¶ Checks if given branch exists in any project
Parameters: - groups (list[Group]) – Groups to check
- branch (str) – Branch to check for
- is_remote (bool) – Check for remote branch
Returns: True, if at least one branch exists
Return type: bool
-
clowder.util.clowder_utils.
existing_branch_projects
(projects, branch, is_remote)¶ Checks if given branch exists in any project
Parameters: - projects (list[Project]) – Projects to check
- branch (str) – Branch to check for
- is_remote (bool) – Check for remote branch
Returns: True, if at least one branch exists
Return type: bool
-
clowder.util.clowder_utils.
filter_groups
(groups, names)¶ Filter groups based on given group names
Parameters: - groups (list[Group]) – Groups to filter
- names (list[str]) – Group names to match against
Returns: List of groups in groups matching given names
Return type: list[Group]
-
clowder.util.clowder_utils.
filter_projects
(groups, group_names=None, project_names=None)¶ Filter projects based on given project or group names
Parameters: - groups (list[Group]) – Groups to filter
- group_names (list[str]) – Group names to match against
- project_names (list[str]) – Project names to match against
Returns: List of projects in groups matching given names
Return type: list[Project]
-
clowder.util.clowder_utils.
get_clowder_yaml_import_path
(import_name)¶ Return path to imported clowder.yaml file
Parameters: import_name (str) – Name of imported clowder.yaml Returns: Path to imported clowder.yaml file Return type: str
-
clowder.util.clowder_utils.
get_saved_version_names
()¶ Return list of all saved versions
Returns: List of all saved version names Return type: list[str]
-
clowder.util.clowder_utils.
link_clowder_yaml
(version=None)¶ Create symlink pointing to clowder.yaml file
Parameters: version (Optional[str]) – Version name of clowder.yaml to link Raises: ClowderExit –
-
clowder.util.clowder_utils.
options_help_message
(options, message)¶ Help message for groups option
Parameters: - options (list[str]) – List of options
- message (str) – Help message
Returns: Formatted options help message
Return type: str
-
clowder.util.clowder_utils.
print_parallel_groups_output
(groups, skip)¶ Print output for parallel group command
Parameters: - groups (list[Group]) – Groups to print output for
- skip (list[str]) – Project names to skip
-
clowder.util.clowder_utils.
print_parallel_projects_output
(projects, skip)¶ Print output for parallel project command
Parameters: - projects (list[Project]) – Projects to print output for
- skip (list[str]) – Project names to skip
-
clowder.util.clowder_utils.
run_group_command
(group, skip, command, *args, **kwargs)¶ Run group command and print output
Parameters: - group (Group) – Group to run command for
- skip (list[str]) – Project names to skip
- command (str) – Name of method to invoke
- args – List of arguments to pass to method invocation
- kwargs – Dict of arguments to pass to method invocation
-
clowder.util.clowder_utils.
run_project_command
(project, skip, command, *args, **kwargs)¶ Run project command and print output
Parameters: - project (Praject) – Project to run command for
- skip (list[str]) – Project names to skip
- command (str) – Name of method to invoke
- args – List of arguments to pass to method invocation
- kwargs – Dict of arguments to pass to method invocation
-
clowder.util.clowder_utils.
validate_groups
(groups)¶ Validate status of all projects for specified groups
Parameters: groups (list[Group]) – Groups to validate Raises: ClowderExit –
-
clowder.util.clowder_utils.
validate_print_output
(clowder, group_names, **kwargs)¶ Validate projects/groups and print output
-
clowder.util.clowder_utils.
validate_print_output
(clowder, group_names, project_names=None, skip=[])
Parameters: - clowder (ClowderController) – ClowderController instance
- group_names (list[str]) – Group names to validate/print
Keyword Arguments: - project_names (list[str]) – Project names to validate/print
- skip (list[str]) – Project names to skip
-
-
clowder.util.clowder_utils.
validate_projects
(projects)¶ Validate status of all projects
Parameters: projects (list[Project]) – Projects to validate Raises: ClowderExit –
-
clowder.util.clowder_utils.
validate_projects_exist
(clowder)¶ Validate existence status of all projects for specified groups
Parameters: clowder (ClowderController) – ClowderController instance Raises: ClowderExit –
clowder.util.connectivity module¶
Network connectivity
-
clowder.util.connectivity.
is_offline
(host='8.8.8.8', port=53, timeout=3)¶ Returns True if offline, False otherwise
Service: domain (DNS/TCP)
Note
Implementation source https://stackoverflow.com/a/33117579
Parameters: - host (str) – Host to check. Default is 8.8.8.8 (google-public-dns-a.google.com)
- port (int) – Port number. Default is 53/tcp
- timeout (int) – Seconds to wait until timeout
Returns: True, if offline
Return type: bool
Raises:
-
clowder.util.connectivity.
network_connection_required
(func)¶ If no network connection, print offline message and exit
clowder.util.decorators module¶
clowder.yaml validation
-
clowder.util.decorators.
clowder_required
(func)¶ If no clowder repo, print clowder not found message and exit
-
clowder.util.decorators.
valid_clowder_yaml_required
(func)¶ If clowder.yaml is invalid, print invalid yaml message and exit
clowder.util.execute module¶
Subprocess execution utilities
-
clowder.util.execute.
execute_command
(command, path, **kwargs)¶ Execute command via thread
-
clowder.util.execute.
execute_command
(command, path, shell=True, env=None, print_output=True)
Parameters: - command (str or list[str]) – Command to run
- path (str) – Path to set as
cwd
Keyword Arguments: - shell (bool) – Whether to execute subprocess as
shell
- env (dict) – Enviroment to set as
env
- print_output (bool) – Whether to print output
Returns: Command return code
Return type: int
Raises: -
-
clowder.util.execute.
execute_forall_command
(command, path, forall_env, print_output)¶ Execute forall command with additional environment variables and display continuous output
Parameters: - command (str or list[str]) – Command to run
- path (str) – Path to set as
cwd
- forall_env (dict) – Enviroment to set as
env
- print_output (bool) – Whether to print output
Returns: Command return code
Return type: int
-
clowder.util.execute.
execute_subprocess_command
(command, path, **kwargs)¶ Execute subprocess command
-
clowder.util.execute.
execute_subprocess_command
(command, path, shell=True, env=None, stdout=None, stderr=None)
Parameters: - command (str or list[str]) – Command to run
- path (str) – Path to set as
cwd
Keyword Arguments: - shell (bool) – Whether to execute subprocess as
shell
- env (dict) – Enviroment to set as
env
- stdout (int) – Value to set as
stdout
- stderr (int) – Value to set as
stderr
Returns: Subprocess return code
Return type: int
Raises: -
-
clowder.util.execute.
subprocess_exit_handler
(process)¶ terminate subprocess
Parameters: process (Popen) – Popen subprocess instance
clowder.util.file_system module¶
File system utilities
-
clowder.util.file_system.
force_symlink
(file1, file2)¶ Force symlink creation
Parameters: - file1 (str) – File to create symlink pointing to
- file2 (str) – Symlink location
Raises:
-
clowder.util.file_system.
remove_directory
(path)¶ Remove directory at path
Parameters: path (str) – Path to remove Raises: ClowderExit –
-
clowder.util.file_system.
symlink_target
(path)¶ Returns target path if input is a symlink, otherwise returns original path
Parameters: path (str) – Path of file or symlink Returns: Target path if input is a symlink, otherwise original path Return type: str
clowder.util.formatting module¶
String formatting utilities
-
clowder.util.formatting.
clowder_command
(cmd)¶ Return formatted clowder command name
Parameters: cmd (str) – Clowder command name Returns: Formatted clowder command name Return type: str
-
clowder.util.formatting.
command
(cmd)¶ Return formatted command name
Parameters: cmd (str or list[str]) – Clowder command name Returns: Formatted clowder command name Return type: str
-
clowder.util.formatting.
command_failed_error
(cmd)¶ Format error message for failed command
Parameters: cmd (str or list[str]) – Clowder command name Returns: Formatted clowder command name Return type: str
-
clowder.util.formatting.
depth_error
(depth, yml)¶ Return formatted error string for invalid depth
Parameters: - depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
- yml (str) – Path to yaml file
Returns: Formatted depth error
Return type: str
-
clowder.util.formatting.
empty_yaml_error
(yml)¶ Return formatted error string for empty clowder.yaml
Parameters: yml (str) – Path to yaml file Returns: Formatted empty yaml error Return type: str
-
clowder.util.formatting.
error
(err)¶ Return error message for generic error
Parameters: err (Exception) – Generic error Returns: Formatted generic error Return type: str
-
clowder.util.formatting.
file_exists_error
(path)¶ Format error message for already existing file
Parameters: path (str) – File path name Returns: Formatted file exists error Return type: str
-
clowder.util.formatting.
fork_string
(name)¶ Return formatted fork name
Parameters: name (str) – Fork name Returns: Formatted fork name Return type: str
-
clowder.util.formatting.
get_path
(path)¶ Return formatted path
Parameters: path (str) – Path name Returns: Formatted path name Return type: str
-
clowder.util.formatting.
group_name
(name)¶ Print formatted group name
Parameters: name (str) – Group name Returns: Formatted group name Return type: str
-
clowder.util.formatting.
invalid_protocol_error
(protocol, yml)¶ Return formatted error string for incorrect protocol
Parameters: - protocol (str) – Git protocol
- yml (str) – Path to yaml file
Returns: Formatted invalid ref error
-
clowder.util.formatting.
invalid_ref_error
(ref, yml)¶ Return formatted error string for incorrect ref
Parameters: - ref (str) – Git reference
- yml (str) – Path to yaml file
Returns: Formatted invalid ref error
-
clowder.util.formatting.
invalid_yaml_error
()¶ Return error message for invalid clowder.yaml
Returns: Formatted yaml error Return type: str
-
clowder.util.formatting.
missing_entries_error
(name, yml)¶ Return formatted error string for invalid entry in collection
Parameters: - name (str) – Entry name
- yml (str) – Path to yaml file
Returns: Formatted missing entries error
Return type: str
-
clowder.util.formatting.
missing_entry_error
(entry, name, yml)¶ Return formatted error string for missing entry in dictionary
Parameters: - entry (str) – Name of entry to check
- name (str) – Name of entry to print if missing
- yml (str) – Path to yaml file
Returns: Formatted missing entry in dictionary error
Return type: str
-
clowder.util.formatting.
missing_imported_yaml_error
(path, yml)¶ Return formatted error string for missing imported clowder.yaml
Parameters: - path (str) – File path
- yml (str) – Path to yaml file
Returns: Formatted missing YAML error
Return type: str
-
clowder.util.formatting.
missing_yaml_error
()¶ Format error message for missing clowder.yaml
Returns: Formatted missing YAML error Return type: str
-
clowder.util.formatting.
offline_error
()¶ Return error message for no internet connection
Returns: Offline error message Return type: str
-
clowder.util.formatting.
open_file_error
(path)¶ Format error message for failing to open file
Parameters: path (str) – File path Returns: Formatted file error Return type: str
-
clowder.util.formatting.
parallel_exception_error
(path, *args)¶ Return formatted error string for parallel error
Parameters: - path (str) – Clowder file path
- args – Method arguments
Returns: Formatted parallel exception error
Return type: str
-
clowder.util.formatting.
recursive_import_error
(depth)¶ Format error message for too many recursive imports
Parameters: depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer Returns: Formatted too many recursive imports error Return type: str
-
clowder.util.formatting.
ref_string
(ref)¶ Return formatted ref name
Parameters: ref – Git reference Returns: Formatted ref name Return type: str
-
clowder.util.formatting.
remote_already_exists_error
(remote_name, remote_url, actual_url)¶ Format error message when remote already exists with different url
Parameters: - remote_name (str) – Remote name
- remote_url (str) – Remote URL
- actual_url (str) – Actual URL
Returns: Formatted remote exists error
Return type: str
-
clowder.util.formatting.
remote_name_error
(fork, project, remote)¶ Return formatted error string for fork with same remote as project
Parameters: - fork (str) – Fork name
- project (str) – Project name
- remote (str) – Remote name
Returns: Formatted duplicate remote fork name error
Return type: str
-
clowder.util.formatting.
remote_string
(remote)¶ Return formatted remote name
Parameters: remote (str) – Remote branch name Returns: Formmatted remote branch name Return type: str
-
clowder.util.formatting.
remove_prefix
(text, prefix)¶ Remove prefix from a string
Parameters: - text (str) – String to remove prefix from
- prefix (str) – Prefix to remove
Returns: Text with prefix removed
Return type: str
-
clowder.util.formatting.
save_default_error
(name)¶ Format error message for trying to save ‘default’ version
Parameters: name (str) – Version name Returns: Formatted default version error Return type: str
-
clowder.util.formatting.
save_file_error
(path)¶ Format error message for failing to save file
Parameters: path (str) – File path Returns: Formatted save failure error Return type: str
-
clowder.util.formatting.
save_version
(version_name, yml)¶ Format message for saving version
Parameters: - version_name (str) – Clowder version name
- yml (str) – Path to yaml file
Returns: Formatted version name
Return type: str
-
clowder.util.formatting.
save_version_exists_error
(version_name, yml)¶ Format error message previous existing saved version
Parameters: - version_name (str) – Version name
- yml (str) – Path to yaml file
Returns: Formatted version exists error
Return type: str
-
clowder.util.formatting.
skip_project_message
()¶ Return skip project message
Returns: Skip project message Return type: str
-
clowder.util.formatting.
type_error
(name, yml, type_name)¶ Return formatted error string for value with wrong type
Parameters: - name (str) – Value name
- yml (str) – Path to yaml file
- type_name (str) – Type name
Returns: Formatted incorrect type error
Return type: str
-
clowder.util.formatting.
unknown_entry_error
(name, collection, yml)¶ Return formatted error string for unknown entry in collection
Parameters: - name (str) – Entry name
- collection (dict) – Entries collection
- yml (str) – Path to yaml file
Returns: Formatted unknown entry error
Return type: str
-
clowder.util.formatting.
version
(version_name)¶ Return formatted string for clowder.yaml version
Parameters: version_name (str) – Clowder version name Returns: Formatted clowder version name Return type: str
-
clowder.util.formatting.
yaml_file
(yml)¶ Return formatted string for clowder.yaml file
Parameters: yml (str) – Path to yaml file Returns: Formatted YAML string Return type: str
-
clowder.util.formatting.
yaml_path
(yml)¶ Returns formatted yaml path
Parameters: yml (str) – Path to yaml file Returns: Formatted YAML path Return type: str
-
clowder.util.formatting.
yaml_string
(yaml_output)¶ Return yaml string from python data structures
Parameters: yaml_output (dict) – YAML python object Returns: YAML as a string Return type: str Raises: ClowderExit –
clowder.util.parallel_commands module¶
Clowder parallel commands
-
clowder.util.parallel_commands.
async_callback
(val)¶ Increment async progress bar
Parameters: val – Dummy parameter to satisfy callback interface
-
clowder.util.parallel_commands.
forall_parallel
(commands, skip, ignore_errors, projects)¶ Runs command or script for projects in parallel
Parameters: - commands (list[str]) – Command to run
- skip (list[str]) – Project names to skip
- ignore_errors (bool) – Whether to exit if command returns a non-zero exit code
- projects (list[Project]) – Projects to run command for
-
clowder.util.parallel_commands.
herd_parallel
(clowder, group_names, **kwargs)¶ Clone projects or update latest from upstream in parallel
-
clowder.util.parallel_commands.
herd_parallel
(clowder, group_names, branch=None, tag=None, depth=0, rebase=False, project_names=None, skip=[], protocol=None)
Parameters: - clowder (ClowderController) – ClowderController instance
- group_names (list[str]) – Group names to herd
Keyword Arguments: - branch (str) – Branch to attempt to herd
- tag (str) – Tag to attempt to herd
- depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
- protocol (str) – Git protocol (‘ssh’ or ‘https’)
- rebase (bool) – Whether to use rebase instead of pulling latest changes
- project_names (list[str]) – Project names to herd
- skip (list[str]) – Project names to skip
-
-
clowder.util.parallel_commands.
herd_project
(project, branch, tag, depth, rebase, protocol)¶ Herd command wrapper function for multiprocessing Pool execution
Parameters: - project (Project) – Project instance
- branch (str) – Branch to attempt to herd
- tag (str) – Tag to attempt to herd
- depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
- rebase (bool) – Whether to use rebase instead of pulling latest changes
- protocol (str) – Git protocol (‘ssh’ or ‘https’)
-
clowder.util.parallel_commands.
pool_handler
(count)¶ Pool handler for finishing parallel jobs
Parameters: count (int) – Total count of projects in progress bar Raises: ClowderExit –
-
clowder.util.parallel_commands.
reset_parallel
(clowder, group_names, **kwargs)¶ Reset project branches to upstream or checkout tag/sha as detached HEAD in parallel
-
clowder.util.parallel_commands.
reset_parallel
(clowder, group_names, timestamp_project=None, project_names=None, skip=[])
Parameters: - clowder (ClowderController) – ClowderController instance
- group_names (list[str]) – Group names to reset
Keyword Arguments: - timestamp_project (str) – Reference project to checkout commit timestamps of other projects relative to
- project_names (list[str]) – Project names to reset
- skip (list[str]) – Project names to skip
-
-
clowder.util.parallel_commands.
reset_project
(project, timestamp)¶ Reset command wrapper function for multiprocessing Pool execution
Parameters: - project (Project) – Project instance
- timestamp (str) – If not None, reset to commit at timestamp, or closest previous commit
-
clowder.util.parallel_commands.
run_project
(project, commands, ignore_errors)¶ Run command wrapper function for multiprocessing Pool execution
Parameters: - project (Project) – Project instance
- commands (list[str]) – Commands to run
- ignore_errors (bool) – Whether to exit if command returns a non-zero exit code
-
clowder.util.parallel_commands.
sync_parallel
(projects, protocol, rebase=False)¶ Sync projects in parallel
Parameters: - projects (list[Project]) – Projects to sync
- protocol (str) – Git protocol, ‘ssh’ or ‘https’
- rebase (Optional[bool]) – Whether to use rebase instead of pulling latest changes
-
clowder.util.parallel_commands.
sync_project
(project, protocol, rebase)¶ Sync command wrapper function for multiprocessing Pool execution
Parameters: - project (Project) – Project instance
- protocol (str) – Git protocol, ‘ssh’ or ‘https’
- rebase (bool) – Whether to use rebase instead of pulling latest changes
-
clowder.util.parallel_commands.
worker_init
()¶ Process pool terminator
Note
Implementation source https://stackoverflow.com/a/45259908
clowder.util.progress module¶
Progress bar wrapper class