conductor.lib.common module

class conductor.lib.common.Config

static add_api_settings(settings_dict)

static create_default_config(path)

default_config = {'api_key_path': '/tmp/tmp.CENU7DBH6S/auth/conductor_api_key', 'api_url': 'https://api.conductortech.com', 'auth_url': 'https://dashboard.conductortech.com', 'base_url': 'atomic-light-001.appspot.com', 'instance_cores': 16, 'instance_flavor': 'standard', 'local_upload': True, 'log_level': 'INFO', 'md5_caching': True, 'priority': 5, 'thread_count': 4, 'url': 'https://atomic-light-001.appspot.com'}

default_config_locations = {'darwin': '/root/Application Support/Conductor/config.yml', 'linux2': '/root/.conductor/config.yml', 'win32': 'Conductor Technologies/Conductor/config.yml'}

get_config_file_paths(config_file=None)

get_environment_config()

Look for any environment settings that start with CONDUCTOR_.

Cast any variables to bool if necessary.

get_user_config()

required_keys = []

static validate_api_key(config)

Load the API Key (if it exists)

Parameters:config – client configuration object

verify_required_params(config)

class conductor.lib.common.DecRetry(retry_exceptions=<type 'exceptions.Exception'>, skip_exceptions=(), tries=8, static_sleep=None)

Bases: object

Decorator that retries the decorated function using an exponential backoff sleep.

retry_exceptions

An Exception class (or a tuple of Exception classes) that this decorator will catch/retry. All other exceptions that occur will NOT be retried. By default, all exceptions are caught (due to the default arguemnt of Exception)

Type:Class or tuple[Class]

skip_exceptions

An Exception class (or a tuple of Exception classes) that this decorator will NOT catch/retry. This will take precedence over the retry_exceptions.

Type:Class or tuple[Class]

tries

Number of times to try (not retry) before raising

Type:int

static_sleep

The amount of seconds to sleep before retrying. When set to None, the sleep time will use exponential backoff. See below.

This retry function not only incorporates exponential backoff, but also “jitter”. see http://www.awsarchitectureblog.com/2015/03/backoff.html. Instead of merely increasing the backoff time exponentially (determininstically), there is a randomness added that will set the sleeptime anywhere between 0 and the full exponential backoff time length.

sleep(seconds)

class conductor.lib.common.ExceptionAction(raise_=True, omitted_exceptions=(), disable_var='')

Bases: object

This is a base class to be used for constructing decorators that take a specific action when the decorated method/function raises an exception.

For example, it can send a message or record data to a database before the exception is raised, and then (optionally) raise the exception. Optionally specify particular exceptions (classes) to be omitted from taking action on (though still raise the exception)

disable_var

An environment variable name, which if found in the runtime environment will disable the action from being taken. This can be useful when a developer is actively developing and does not want the decorator to take action.

Type:str

take_action(e)

Overide this method to do something useful before raising the exception.

e.g.:

print "sending error message to mom: %s" % traceback.format_exc()

class conductor.lib.common.ExceptionLogger(message='', log_traceback=True, log_level=30, raise_=False, omitted_exceptions=(), disable_var='')

Bases: conductor.lib.common.ExceptionAction

DECORATOR If the decorated function raises an exception, this decorator can log the exception and continue (suppressing the actual exception. A message may be prepended to the exception message.

example output:

>>> broken_function()
# Warning: conductor.lib.common : My prependend message
Traceback (most recent call last):
  File "/usr/local/lschlosser/code/conductor_client/conductor/lib/common.py", line 85, in decorater_function
    return function(*args, **kwargs)
  File "<maya console>", line 4, in broken_function
ZeroDivisionError: integer division or modulo by zero

take_action(error)

Log out the message

conductor.lib.common.base_dir()

Return the top level directory for the local Conductor repository.

This is derived by traversing up directories from this current script.

Note that due to symkinks, we can’t use os.path.realpath on __file__ because __file__ may be a symlinked path and would return the directory for the “real” file (as opposed to the directory of the symlinked file (__file__))

conductor.lib.common.dec_catch_exception(raise_=False)

DECORATOR Wraps the decorated function/method so that if the function raises an exception, the exception will be caught, it’s message will be printed, and optionally the function will return (suppressing the exception) .

conductor.lib.common.dec_timer_exit(log_level=20)

conductor.lib.common.generate_md5(filepath, base_64=False, blocksize=65536, poll_seconds=None, callback=None, log_level=20)

Generate and return md5 hash (base64) for the given filepath

Parameters:

• filepath (str) – The file path to generate an md5 hash for.
• base_64 (bool) – whether or not to return a base64 string.
• poll_seconds (int) – The number of seconds to wait between logging out to the console when md5 hashing (particularly a large file which may take a while)
• log_level (logging.level) – The log level that should be used when logging messages.
• callback (callable) – A callable that is called during the md5 hashing process. It’s called every time a block of data has been hashed (see blocksize arg). The callable receives the following arguments: - filepath: see above - file_size: the total size of the file (in bytes) - bytes_processed: the amount of bytes that has currently been hashed - log_level: see above

conductor.lib.common.get_base64_md5(*args, **kwargs)

conductor.lib.common.get_conductor_gpu_configs()

Get the list of available GPU configs from the resources.yml file

conductor.lib.common.get_conductor_instance_types()

Get the list of available instances types from the resources.yml file

conductor.lib.common.get_human_bytes(bytes_)

For the given bytes (integer), convert and return a “human friendly: representation of that data size.

conductor.lib.common.get_human_duration(seconds)

convert the given seconds (float) into a human friendly unit

conductor.lib.common.get_human_timestamp(seconds_since_epoch)

convert the given seconds since epoch (float)

conductor.lib.common.get_md5(file_path, blocksize=65536)

conductor.lib.common.get_package_ids()

Get the list of available instances types from the resources.yml file

conductor.lib.common.get_progress_percentage(current, total)

Return a string percentage, e.g. “80%” given current bytes (int) and total bytes (int)

conductor.lib.common.load_resources_file()

Return the resource yaml file as a dict.

If the $CONDUCTOR_RESOURCES_PATH environment variable is set, then use it to find load the resource file from. Otherwise look for it in the default location.

To Do:

(lws) the resource filepath should also be able to be dictated in the config But can’t check that here because this module (common.py) should not have any imports from the conductor package, i.e. this module creates the config, and therefore cannot be reliant on the config.

conductor.lib.common.load_yaml(filepath, safe=True, omit_tags=False)

Helper class that loads the given yaml filepath into a python object.

This function is somewhat complex(and ugly) because it must create it’s own functions and classes.

The reason it must create it’s own Loader class is because when adding a custom constructor, that constructor remains loaded (and imbedded in the PyYaml’s native Loader class) for the duration of the python session.

This could lead to disastrous behavior for customers that may be using the same session to do additional python work (think inside of Maya).

So instead of using PyYaml’s Loader classes directly, we subclass our own so that we can dispose of it when we’re done.

This all could have been avoided if PyYaml also provided a “remove_multi_constructor” function.

Parameters:

• safe (bool) – When True, will use yaml’s safe_loader.
• omit_tags (bool) – When True, will skip any yaml tags in the file. This can be useful when you want to read a yaml file but don’t have all of the necessary yaml tag constructors to do so. All tags in the file will be given a value of u'<TAG OMITTED>'

conductor.lib.common.on_windows()

Return True if the current system is a Windows platform

conductor.lib.common.register_sigint_signal_handler(signal_handler=<function signal_handler>)

conductor.lib.common.run(cmd)

conductor.lib.common.signal_handler(sig_number, stack_frame)