conductor.lib.worker module

class conductor.lib.worker.JobManager(job_description, reporter_description=None)

add_task(task)

drain_queues()

error_handler_target()

join()

Block until all work is complete

kill_reporters()

kill_workers()

mark_all_tasks_complete()

start()

start_error_handler()

stop_work()

worker_queue_status_text()

class conductor.lib.worker.MetricStore

This provides a thread-safe integer store that can be used by workers to share atomic counters.

Note: writes are eventually consistent

append(list_name, value)

do_append(*args)

do_increment(*args)

do_set_dict(*args)

get(variable)

get_dict(dict_name, key=None)

get_list(list_name)

increment(variable, step_size=1, filename='')

join()

set(key, value)

set_dict(dict_name, key, value)

start()

needs to be single-threaded for atomic updates

target()

class conductor.lib.worker.Reporter(metric_store=None)

kill(block=False)

start()

target()

static working()

class conductor.lib.worker.ThreadWorker(**kwargs)

Bases: object

Abstract worker class.

The class defines the basic function and data structures that all workers need.

To Do:

Move this into it’s own lib

static PoisonPill()

check_for_poison_pill(job)

do_work(job)

This needs to be implemented for each worker type. The work task from the in_queue is passed as the job argument.

Returns the result to be passed to the out_queue

join()

kill(block=False)

mark_done()

put_job(job)

start()

Start number_of_threads threads.

target(thread_int)

conductor.lib.worker.logger = <logging.Logger object>

This is used to signal to workers if work should continue or not