conductor.lib.conductor_submit module¶
Command Line General Submitter for sending jobs and uploads to Conductor
-
exception
conductor.lib.conductor_submit.
BadArgumentError
¶ Bases:
exceptions.ValueError
-
class
conductor.lib.conductor_submit.
Submit
(args)¶ Bases:
object
Conductor Submission
Command Line Usage:
$ python conductor_submit.py --upload_file /path/to/upload_file.txt --frames 1001-1100 --cmd
Help:
$ python conductor_submit.py -h
-
classmethod
cast_metadata
(metadata, strict=False)¶ Ensure that the data types in the given metadata are of the proper type (str or unicode). If strict is False, automatically cast (and warn) any values which do not conform. If strict is True, do not cast values, simply raise an exception.
-
classmethod
cast_metadata_value
(value)¶ Attempt to cast the given value to a unicode string
-
consume_args
(args)¶ “Consume” the give arguments (which have been parsed by argparse). For arguments that have not been specified by the user via the command line, default to reasonable values. If inherit_config is True, then default unspecified arguments to those values found in the config.yml file.
- Use the argument provided by the user via the command line. Note that this may be a non-True value, such as False or 0. So the only way to know whether the argument was provided by the user is to check whether is value is not None (which implies that argparse arguments should not be configured to give a default value when the user has not specified them).
- If the user has not provided the argument, and if inherit_config is True, then use the value found in the config.
- If inherit_config is False or the the value in the config doesn’t exist, then default to an empty or non-True value.
-
get_upload_files
()¶ Resolve the “upload_paths” and “upload_file” arguments to return a single list of paths.
-
main
()¶ Submitting a job happens in a few stages:
- Gather depedencies and parameters for the job
- Upload dependencies to cloud storage (requires md5s of dependencies)
- Submit job to conductor (listing the dependencies and their corresponding md5s)
In order to give flexibility to customers (because a customer may consist of a single user or a large team of users), there are two options avaiable that can dicate how these job submission stages are executed. In a simple single-user case, a user’s machine can doe all three of the job submission stages. Simple. We call this local_upload=True.
However, when there are multiple users, there may be a desire to funnel the “heavier” job submission duties (such as md5 checking and dependency uploading) onto one dedicated machine (so as not to bog down the artist’s machine). This is called local_upload=False. This results in stage 1 being performed on the artist’s machine (dependency gathering), while handing over stage 2 and 3 to an uploader daemon. This is achieved by the artist submitting a “partial” job, which only lists the dependencies that it requires (omitting the md5 hashes). In turn, the uploader daemon listens for these partial jobs, and acts upon them, by reading each job’s listed dependencies (the filepaths that were recorded during the “partial” job submission). The uploader then md5 checks each dependency file from it’s local disk, then uploads the file to cloud storage (if necesseary), and finally “completes” the partial job submission by providing the full mapping dictionary of each dependency filepath and it’s corresponing md5 hash. Once the job submission is completed, conductor can start acting on the job (executing tasks, etc).
-
metadata_types
= (<type 'str'>, <type 'unicode'>)¶
-
static
parse_upload_file
(upload_filepath)¶ Parse the given filepath for all listed paths (one per line)
-
classmethod
resolve_arg
(args, arg_name, default, combine_config=False)¶ Helper function to resolve the value of an argument.
The general premise is as follows:
- If an argument value was provided then use it.
- However, if the
combine_config
bool isTrue
, then we want to combine that value with the value found in the config.
This only works with argument types which can be combined, such as lists or dictionaries.
In terms of which value trumps the other, the argument value will always trump the config value.
For example, if the config declared a dictionary value, and the argument also provided a dictionary value, they would both be combined, however, the argument value will replace any keys that clash with keys from the config value.
If the argument nor the config have a value, then use the provided default value
The order of resolution is:
Check whether the user explicitly specified the argument when calling/instantiating the class. This is indicated by whether the argument’s value is set to
None
or not.If the value is not
None
, then use it (combining it with the config value if appropriate)If the specified argument does not have value (i.e. it’s
None
), then use the value found in the config.If the config does not define the expected argument then use the default argument.
Parameters: combine_config (bool) – Indicates whether to combine the argument value with the config value. This will only ever occur if both values are present.
-
send_job
(upload_files, upload_size)¶ Construct args for two different cases:
- upload_only
- running an actual command (cmd)
Parameters: upload_files (dict) – Where they key is the filepath, and the value is the md5. e.g.
{"/batman/v06/batman_v006_high.abc": "oFUgxKUUOFIHJ9HEvaRwgg==", "/batman/v06/batman_v006_high.png": "s9y36AyAR5cYoMg0Vx1tzw=="}
-
validate_args
()¶ Ensure that the combination of arguments don’t result in an invalid job/request
# TODO: (lws) Clean this shit up. Too complex. indicates poor design IMO.
-
classmethod
-
conductor.lib.conductor_submit.
run_submit
(args)¶
-
conductor.lib.conductor_submit.
set_logging
(level=None, log_dirpath=None)¶