conductor.submitter module

class conductor.submitter.ConductorSubmitter(parent=None)

Bases: sphinx.ext.autodoc.importer._MockObject

Base class for PySide front-end for submitting jobs to Conductor.

Intended to be subclassed for each software context that may need a Conductor front end e.g. for Maya or Nuke.

The self.getExtendedWidget method acts as an opportunity for a developer to extend the UI to suit his/her needs. See the getExtendedWidgetthe docstring

static addLoggingMenu(menu)

For the given menu object, dynamically generate a action item for each log level (from the loggeria module).

addResetPreferencesMenu(menu)

For the given menu object, dynamically generate a action item for each log level (from the loggeria module).

applyDefaultSettings()

Set the UI to default settings.

autoDetectPackageIds()

Of the given packages, return the packages that match.

1. host software

2. plugins

   • renderer
   • others

autoPopulateJobPackages()

checkJobSize()

Make sure the user is aware they are about submit a large job.

A large job has more tasks than a threshold. We first test the number of frames, because calculating the number of tasks is slow. If frames length is below the threshold, then there can’t possibly be too many tasks. If there are too many tasks, then show the confirmation dialog.

closeEvent(**kwds)

When the Conductor UI is closed, save the user settings.

company_name = 'Conductor'

createJobPackageTreeWidgetItem(package)

For the given software package, create a QTreeWidgetItem for it.

with a text/name that describes it

static createPackageTreeWidgetItem(package)

For the given software package, create a QTreeWidgetItem for it.

with a text/name that describes it

createUI()

Create UI widgets and make.

default_instance_type = 16

filterPackages(show_all_versions=False)

generateConductorArgs(data)

Return a dictionary which contains the necessary conductor argument names and their values. This dictionary will ultimately be passed directly into a conductor Submit object when instantiating/calling it.

Generally, the majority of the values that one would populate this dictionary with would be found by quering this UI, e.g.from the “frames” section of ui.

getAutoretryCount()

Get the UI’s Auto Retry spinbox value.

getAutoretryPolicy()

Construct/return an autoretry policy if the user as selected to use preemptible instances.

getBestPackage(software_info, packages)

getChunkSize()

Return UI’s Frame Chunk Size spinbox value.

getCustomFrameString()

Return the UI’s custom frame field.

getDefaultScoutFrames()

Return the default scout frames for the open scene file.

This should attempt to return the first, middle, and last frame.

Logic order:

1. If the “start” and “end” fields are active/populated in the submitter UI then use that range to determine first, middle, last scout frames
2. If the use has specified a custom frame range in the submitter UI, Then return None

getDockerImage()

Return the Docker image name to use on Conductor.

If there is a docker image in the config.yml file, then use it.

getEndFrame()

Return UI’s end frame field.

getEnforcedMd5s()

Note that this is only relevant when local_upload=False.

Return a dictionary of any filepaths and their corresponding md5 hashes that should be verified before uploading to cloud storage.

When local_upload is False, uploading files to cloud storage is handled by an uploader daemon. Because there can be a time delay between when a user has pressed “submit” and when the daemon actually goes to upload the files to cloud storage, there is the possibility that files have actually changes on disk. For example, a user may submit 3 jobs to conductor within 7 seconds of one another, by quickly rotating the camera, saving the maya scene on top of itself, and pressing “Submit” (and doing this three time). Though this is probably not a great workflow, it’s something that needs to be guarded against.

This method returns a dictionary of filepaths and and their corresponding md5 hashes, which is used by the uploader daemon when uploading the files to conductor. The daemon will do its own md5 hash of all files in this dictionary, and match it against the md5s that the dictionary provides. If the uploader finds a mismatch then it fails the upload process (and fails the job).

Generally there is gray area as to what is considered “acceptable” for files being changed from underneath the feet of the artist. Because a source file (such as a maya scene or katana file, for example), has many external dependencies (such as texture files, alembic, etc), those depencies could possibly be changed on disk by the time the uploader get a chance to upload them to Conductor. As a compromise on attempting to create an exact snapshot of the state of ALL files on disk (i.e. md5 hashes)that a job requires at the moment the artist presses “submit”, Conductor only guarantees exact file integrity on the source file (maya/katana file). That source file’s dependencies are NOT guaranteed to match md5s. In otherwords, a texture file or alembic cache file is not md5 checked to ensure it matches the md5 of when the user pressed “submit”. Only the maya file is.

getEnvironment()

Return a dictionary of environment variables to use for the Job’s environment.

Merge any environment settings defined in the config with the environment of the packages that the user has selected.

getExtendedAdvancedWidget()

This method extends the Conductor UI by providing a single PySide widget that will be added to the UI. The widget may be any class object derived from QWidget.

In order to do so, subclass this class and override this method to return the desired widget object. This widget will be inserted at the bottom of the “Advanced” tab. See illustration below:

 ____________________
|   Advanced <tab>   |
|--------------------|
|                    |
|                    |
|                    |
|--------------------|
|                    |
|  <EXTENDED WIDGET> |   <-- your extended advanced widget goes here.
|____________________|

getExtendedWidget()

This method extends the Conductor UI by providing a single PySide widget that will be added to the UI. The widget may be any class object derived from QWidget.

In order to do so, subclass this class and override this method to return the desired widget object. This widget will be inserted between the Frame Range area and the Submit button. See illustration below:

 ____________________
|     Conductor      |
|--------------------|
|  Frame Range Area  |
|--------------------|
|                    |
|  <EXTENDED WIDGET> |   <-- your extended widget goes here.
|                    |
|--------------------|
|   submit button    |
|____________________|

getForceUploadBool()

Return whether the “Force Upload” checkbox is checked on or off.

getFrameRangeString()

Construct the frameRange string for the frames argument.

getGpuConfig()

Return the number of GPUs and GPU type that the user has selected from the “GPU Acceleration” combobox.

getHostPackageTreeItems(host_product_name=None)

getHostProductInfo()

getInstanceType()

Return the number of cores that the user has selected from the “Instance Type” combobox.

getJobPackages()

Return the package dictionaries for ALL QTreeItems.

static getLocalUpload()

Return a bool indicating whether the uploading process should occur on this machine or whether it should get offloaded to the uploader daemon. If False, uploading will occur on the daemon.

Simply return the value set in the config.yml. In the future this option may be exposed in the UI

getMatchingPackage(software_info, packages, strict=False)

getNotifications()

Query the UI’s Notification field and split string on commas, returning a list of email addresses

getOutputDir()

Return the UI’s Output Directory field.

classmethod getParentWindow()

Return a QtWidget object that should act as the parent for the submitter window.

This is oftentime the case when launching the submitter within another application such as nuke or maya. In these cases, return the main QtWindow object for those applications.

getPluginsProductInfo()

getPreemptibleCheckbox()

Return whether or not the “Preemptible” checkbox is checked.

getProject()

Return the UI’s Projectj field.

getScoutFrames()

If the “Scout Job” checkbox is checked, promput the user to ender desired scout frames.

If the user has submitted scout frames in the past, then prepopulate those frames in the dialog box. Otherwise generate default scout frames

getScoutJobCheckbox()

Return the checkbox value for the “Scout Job” checkbox.

getSelectedAvailablePackages()

Return the package dictionaries for each QTreeItem that is selected by the user.

getSelectedJobPackages()

Return the package dictionaries for each QTreeItem that is selected by the user.

getSoftwarePackageIds()

Return the software packages that the submitted job will have access to when running.

These packages are referred to by their ids. Merge the any packages that are defined in the config with those that are selected by the user.

getSourceFilepath()

Return the filepath for the currently open file.

This is the currently opened maya/katana/nuke file, etc

getStartFrame()

Return UI’s start frame field.

getStepFrame()

Return UI’s step frame spinbox value.

getTreeItemPackages()

getUserPrefsFileWidgets()

Return a list of widget objects that are appropriate for restoring their state (values) from a user’s preference file.

Specifically, these widgets are those which a user’s FILE-SPECIFIC should be recorded/loaded for. These widgets are identified by a dynamic property that has been set on them via Qt Designer.

getUserPrefsGlobalWidgets()

Return a list of widget objects that are appropriate for restoring their state (values) from a user’s preference file.

Specifically, these widgets are those which a user’s GLOBAL preferences should be recorded/loaded for. These widgets are identified by a dynamic property that has been set on them via Qt Designer.

get_package_by_id(package_id)

introspectSoftwareInfo()

launchScoutFramesDialog(default_scout_frames)

Launch a dialog box to prompt the user to enter their desired scout frames.

Prepopulate the lineedit field with any scout frames that have been submitted previous for the current file (read from preferences).

launch_result_dialog(response_code, response)

link_color = 'rgb(200,100,100)'

loadUserSettings()

Read and apply user preferences (widget values, etc)

on_ui_add_to_job_pbtn_clicked()

on_ui_auto_detect_pbtn_clicked()

on_ui_choose_output_path_pbtn_clicked()

on_ui_find_pbtn_clicked()

on_ui_refresh_tbtn_clicked(**kwds)

on_ui_remove_selected_pbtn_clicked()

on_ui_show_all_versions_chkbx_clicked()

on_ui_start_end_rdbtn_toggled(on)

on_ui_submit_pbtn_clicked()

This gets called when the user pressed the “Submit” button in the UI.

Submission Control Flow:

Below is the method calling order of when a user presses the “submit” button in the UI:

1. self.runPreSubmission(). Run any pre-submission processes.
2. self.runConductorSubmission(). Run the submission process.
3. self.runPostSubmission(). Run any post-submission processes.

Each one of these methods has the opportunity to return data, which in turn will be available to the next method that is called. If that mechanism does not meet all pre/post submission needs, then overriding those methods is also an available/appropriate methodology.

openAvailableTreeMenu(position)

openJobTreeMenu(position)

populateGpuCmbx()

Populate the GPU combobox with all of the available GPU types.

populateInstanceTypeCmbx()

Populate the Instance combobox with all of the available instance types.

populateJobPackages(auto=False)

1. query the prefs for package ids
2. Query the config for package ids and add
3. if none, then auto detect

populateJobSoftwareTrwgt(packages)

populateProjectCmbx()

Populate the project combobox with project names.

If any projects are specified in the config file, then use them. Otherwise query Conductor for all projects

populateSoftwareVersionsTrWgt()

Populate the QTreeWidget with all of the available software packages.

This is a little tricky because packages can be plugins of other packages, and we want to represent that relations (via nesting). We also want to group packages that are of the same type (and have them be able to expand/collpase that group.

Here is an example hierarchy:

maya  # The group for all maya version packages
    |
    |_maya 2014 SP1  # The actual package
            |
            |_vray  # The group for all vray packages (for maya 2014 SP1)
                    |
                    |_vray 1.2.4 # The actual package
            |
            |_arnold # The group for all arnold packages (for maya 2014 sp1)
                    |
                    |_arnold 2.73.1 # The actual package
    |
    |_maya 2014 SP2  # The actual package
            |
            |_vray  # The group for all vray packages (for maya 2014 SP2)
                    |
                    |_vray 1.2.4   # The actual package
            |
            |_arnold # The group for all Arnold packages (for maya 2014 SP2)
                    |
                    |_arnold 2.73.1  # The actual package

A package’s data structure may look like this:

{"package_id": "6b24ca0ea1085ebad536c18307192dce"
 "product": "maya",
 "major_version": "2015",
 "minor_version": "SP3",
 "release_version": "",
 "build_version": "",
 "plugin_host_product": "",
 "plugin_host_version": "",
 "plugin_hosts": [],
 "plugins": [  "8ac98ef9362171a889c5ac8dca8a2a47",
               "03be394b4f84ec9a0d86e4b6f5f3fe26",
               "86779e759adcbbd38e1d058125ace737"]}

populateUi()

Populate the UI with data.

This data should be global information that is not specific to the open file

product = None

runConductorSubmission(data)

Instantiate a Conductor Submit object with the given conductor_args (dict), and execute it.

runPostSubmission(data)

Run any post submission processes.

The “data” argument contains the results of the main runConductorSubmission method, so that any results can be “inspected” and acted upon if desired.

runPreSubmission()

Run any pre submission processes, returning optional data that can be passed into the main runConductorSubmission method.

classmethod runUi(**kwds)

Launch the submitter UI.

This is intended to be run within a software context such as Maya or Nuke (as opposed to a shell). By default, this will show any existing submitter UI that had been closed prior by the user (as opposed to creating a new instance of the UI). Use the force_new flag to force a new instance of it.

classmethod runUiStandalone(**kwds)

Note that this UI class is to be run directly from a python shell, e.g. not within another software’s context (such as maya/nuke).

saveJobPackagesToPrefs()

savePreemptiblePref(**kwargs)

Save the “Preemptible” checkbox user preference for the open file.

saveScoutJobPref(**kwargs)

Save the “Scout Job” checkbox user preference for the open file.

saveUserSettings()

Save current widget settings to the user’s preference file.

These settings are recorded per source file (maya/katana/nuke file, etc).

setAutoretryCount(count)

Set the UI’s Auto Retry spinbox value.

setChunkSize(chunk_size)

Set the UI’s Frame Chunk Size spinbox value.

setCustomFrameString(custom_frame_str)

Set the UI’s custom frame field.

setEndFrame(end_frame)

Set the UI’s end frame field.

setFrameRange(start, end)

Set the UI’s start/end frame fields.

setInstanceType(core_count)

Set the UI’s “Instance Type” combobox.

This is done by specifying the core count int.

setNotifications(value)

Set the UI’s Notification field to the given value.

setOutputDir(dirpath)

Set the UI’s Output Directory field.

setPreemptibleCheckbox(bool_)

Set the checkbox value for the “Preemptible” checkbox.

setProject(project_str, strict=True)

Set the UI’s Project field.

setScoutJobCheckbox(bool_)

Set the checkbox value for the “Scout Job” checkbox.

setStartFrame(start_frame)

Set the UI’s start frame field.

setStepFrame(step_frame)

Set the UI’s step frame spinbox value.

setupMenuBar()

Setup the gui’s menu bar (The top menu bar)

software_packages = None

validateJobPackages()

Validate that all packages that have been added to the job are…valid.

Return True if so, otherwise launch a dialog box to the user.

class conductor.submitter.SubmitterPrefs(company_name, application_name, file_widgets=(), global_widgets=())

Bases: conductor.lib.pyside_utils.UiFilePrefs

A class for interfacing with User preferences for the submitter UI. This subclasses adds functionality that is specific to the Conductor submitter.

Preference names:

scoutframes_str # str. The frames that the user set for scout frames

Things to consider:

1. Because preferences are optional/”superflous” they should NEVER break the submitter UI. Therefore all pref reading/writing needs to be safegaurded against(i.e. try/except) so that it doesn’t halt usage.
2. Because the getting/setting of a preference may fail, be sure to model preferences in a way so that they will not result in dangerous behavior if they fail. i.e. don’t allow the absense of a preference value cause the user to default to doing something dangerous (such as submit a high frame/corecount job)

PREF_JOB_PACKAGES_IDS = 'job_packages_ids'

PREF_SCOUTFRAMES_STR = 'scoutframes_str'

getFileScoutFrames(**kwargs)

Return the user settings for:

the frame/range to use for scout frames

getJobPackagesIds(**kwargs)

Return the user settings for:

the frame/range to use for scout frames

loadSubmitterUserPrefs(**kwargs)

Load both the global and file-specific (e.g.nuke/maya file) user preferences for the UI widgets. This will reinstate any values on the widgets that that were recorded on them from the last time.

First load the global preferences, then load the file-specific prefs which will override and of the global prefs

saveSubmitterUserPrefs(**kwargs)

Save current widget settings to the user’s preference file.

These settings are recorded per source file (maya/katana/nuke file, etc).

setFileScoutFrames(**kwargs)

Set the user settings for:

the frame/range to use for scout frames

setJobPackagesIds(**kwargs)

Set the user settings for:

the frame/range to use for scout frames

class conductor.submitter.TaskFramesGenerator(frames, chunk_size=1, uniform_chunk_step=True)

Bases: object

This base class provides functionality to Job’s command to be suitable for execution for a single Task. The general idea is that the submitted job command has subsitutable characters (args) that should be populated differently for each task (e.g. so that each task renders different frames, etc).

Because every rendering software has it’s own rendering command with differing arguments/syntax, this class is intended to be subclassed for each product.

Specific problems that this class addresses:

1. Converting a job’s “frame_range” argument into individual frames so

   that each task is allocated unique frame(s) to render.

2. Reading and applying those frames to the Task’s render command

   arguments. The render command syntax may be different for each product.

3. Taking into consideration the job’s “chunk_size” argument so that

   a single Task can work on multiple frames. This also includes interpreting any “steps” that have been indicated in the job’s “frame_range” argument.

Note that this class is an iterator so that it can be iterated upon until a command for each task has been dispensed. It provides two items upon each iteration:

• A command that has been fully resolved for a task
• a list of frames (ints) that the task will be rendering

Example usage (using MayaTaskCommand child class):

# Instantiate the class with proper args
>>> cmd_generator = base_utils.MayaTaskCommand(command="Render <frame_args> deadpool.ma",
...                                            frame_range ="1-10x2",
...                                            frame_padding = 4,
...                                            chunk_size= 2)

# Iterate over the class object, dispensing command and frame data for each task
>>> for command, frames in cmd_generator:
...     print "command:", command
...     print "frames:", frames
...
command: Render -s 1 -e 3 -b 2 deadpool.ma
frames: (1, 3)
command: Render -s 5 -e 7 -b 2 deadpool.ma
frames: (5, 7)
command: Render -s 9 -e 9 -b 1 deadpool.ma
frames: (9,)

classmethod derive_steps(frames)

Inspect the list of frames and derive what the “step” is between them. If more than one step is found, return the lowest and highest steps.

e.g.

from FRAMES	to STEPS	Notes
[1,2,3,4]	[1]	one step count between frames (1)
[-1,-3,-5,-7]	[2]	one step count between frames (2)
[1,3,5,17]	[2, 12]	multiple step counts between frames (2, 12)
[4]	[1]	if there isn’t an actual step count, default to 1

The main functionality taken from: http://stackoverflow.com/questions/3428769/finding-the-largest-delta-between-two-integers-in-a-list-in-python

get_next_frames_chunk(chunk_size, uniform_chunk_step=True)

Return then “next” chunk of frames (that the task will render). This may be a list of one frame or several.

e.g.

[1] # single frame
[1, 2 ,3]  # multiple frames
[1,2,3,40,100,101] # multiple frames (No common step. This can be problematic depending on the render command)

Parameters:uniform_chunk_step (bool) – If True, will only return a chunk of frames that have a uniform step size between them.

more complicated example::

frame_range = “1-5x2,20-25,10-30x5,200,1000” chunk_size = 4 step_size=5

resulting task frames:

• task 0: [1]
• task 1: [3]
• task 2: [5, 10, 15, 20], range conforms to step size (5)
• task 3: [21]
• task 4: [22]
• task 5: [23]
• task 6: [24]
• task 7: [25,30], range conforms to step size (5)
• task 8: [200]
• task 9: [1000]

classmethod get_padded_frame(frame_int, padding_int)

Return the given (frame) number as string with the given padding applied to it.

classmethod group_contiguous_frames(frames, ends_only=False)

Separate the given list of frames into groups(sublists) of contiguous frames/

e.g. [1,2,3,15,16,17,21,85] to [(1,2,3),(15,16,17), (21), (85)]

if ends_only==True, return only first and last frames of each group (i.e. “bookends only”) e.g.

[(1,3),(15,17), (21), (85)]

taken from: http://stackoverflow.com/questions/2154249/identify-groups-of-continuous-numbers-in-a-list

next()

Everytime the object is iterated on, return a tuple that contains two items:

• a new command (str) that is appropriate to be executed by a task. (unique from the prior ones)
• a list of corresponding frames (ints) that the command will render

start_frame, end_frame, step, task_frames

task_frames = None