PyCondor API

The main functionality of PyCondor is contained within the Job and Dagman objects.

Job

class pycondor.Job(name, executable, error=None, log=None, output=None, submit=None, request_memory=None, request_disk=None, request_cpus=None, getenv=None, universe=None, initialdir=None, notification=None, requirements=None, queue=None, extra_lines=None, dag=None, arguments=None, retry=None, verbose=0)[source]

Job object consisting of an executable to be run, potentially with a series of different command-line arguments.

Note that the submit, error, log, and output parameters can be explicitly given or configured by setting PYCONDOR_SUBMIT_DIR, PYCONDOR_ERROR_DIR, PYCONDOR_LOG_DIR, and PYCONDOR_OUTPUT_DIR environment variables. An explicitly given value will be used over an environment variable, while an environment variable will be used over a default value.

Parameters

namestr: Name of the Job instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Job.
executablestr: Path to corresponding executable for Job.
errorstr or None, optional: Path to directory where condor Job error files will be written (default is None, will not be included in Job submit file).
logstr or None, optional: Path to directory where condor Job log files will be written (default is None, will not be included in Job submit file).
outputstr or None, optional: Path to directory where condor Job output files will be written (default is None, will not be included in Job submit file).
submitstr, optional: Path to directory where condor Job submit files will be written (defaults to the directory was the Job was submitted from).
request_memorystr or None, optional: Memory request to be included in submit file.
request_diskstr or None, optional: Disk request to be included in submit file.
request_cpusint or None, optional: Number of CPUs to request in submit file.

New in version 0.1.0.
getenvbool or None, optional: Whether or not to use the current environment settings when running the job (default is None).
universestr or None, optional: Universe execution environment to be specified in submit file (default is None).
initialdirstr or None, optional: Initial directory for relative paths (defaults to the directory was the job was submitted from).
notificationstr or None, optional: E-mail notification preference (default is None).
requirementsstr or None, optional: Additional requirements to be included in ClassAd.
queueint or None, optional: Integer specifying how many times you would like this job to run.
extra_lineslist or None, optional: List of additional lines to be added to submit file.
dagDagman, optional: If specified, Job will be added to dag (default is None).
argumentsstr or iterable, optional: Arguments with which to initialize the Job list of arguments (default is None).
retryint or None, optional: Option to specify the number of retries for all Job arguments. This can be superseded for arguments added via the add_arg() method. Note: this feature is only available to Jobs that are submitted via a Dagman (default is None; no retries).
verboseint, optional: Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build_submit()

Attributes

argslist: List of arguments for this Job instance.
parentslist: Only set when included in a Dagman. List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Job is submitted to HTCondor.
childrenlist: Only set when included in a Dagman. List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Job has completed.

Methods

`add_arg`(arg[, name, retry])	Add argument to Job
`add_args`(args)	Adds multiple arguments to Job
`add_child`(node)	Adds node to children list
`add_children`(nodes)	Adds nodes to the children list
`add_parent`(node)	Adds node to parents list
`add_parents`(nodes)	Adds nodes to the parents list
`build`([makedirs, fancyname])	Build and saves the submit file for Job
`build_submit`([makedirs, fancyname, ...])	Calls build and submit sequentially
`haschildren`()	Checks for any children nodes
`hasparents`()	Checks for any parent nodes
`submit_job`([submit_options])	Submits Job to condor

add_arg(arg, name=None, retry=None)[source]

Add argument to Job

Parameters

argstr: Argument to append to Job args list.
namestr or None, optional: Option to specify a name related to this argument. If a name is specified, then a separate set of log, output, and error files will be generated for this particular argument (default is None).

New in version 0.1.2.
retryint or None, optional: Option to specify the number of times to retry this node. Default number of retries is 0. Note: this feature is only available to Jobs that are submitted via a Dagman.

New in version 0.1.2.

Returns

selfobject: Returns self.

add_args(args)[source]

Adds multiple arguments to Job

Parameters

argsiterable: Iterable of arguments to append to the arguments list

Returns

selfobject: Returns self.

add_child(node)

Adds node to children list

Parameters

nodeBaseNode: Job or Dagman to append to the children list.

Returns

selfobject: Returns self.

add_children(nodes)

Adds nodes to the children list

Parameters

nodeslist or tuple: List of nodes to append to the children list

Returns

selfobject: Returns self.

add_parent(node)

Adds node to parents list

Parameters

nodeBaseNode: Job or Dagman to append to the parents list.

Returns

selfobject: Returns self.

add_parents(nodes)

Adds nodes to the parents list

Parameters

nodeslist or tuple: List of nodes to append to the parents list

Returns

selfobject: Returns self.

build(makedirs=True, fancyname=True)[source]

Build and saves the submit file for Job

Parameters

makedirsbool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancynamebool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).

Returns

selfobject: Returns self.

build_submit(makedirs=True, fancyname=True, submit_options=None)[source]

Calls build and submit sequentially

Parameters

makedirsbool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancynamebool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).
submit_optionsstr, optional: Options to be passed to condor_submit for this Job (see the condor_submit documentation for possible options).

Returns

selfobject: Returns self.

haschildren()

Checks for any children nodes

Returns

bool: Returns whether or not this node has any child nodes.

hasparents()

Checks for any parent nodes

Returns

bool: Returns whether or not this node has any parent nodes.

submit_job(submit_options=None)[source]

Submits Job to condor

Parameters

submit_optionsstr, optional: Options to be passed to condor_submit for this Job (see the condor_submit documentation for possible options).

Returns

selfobject: Returns self.

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build()
>>> job.submit_job(submit_options='-maxjobs 1000 -interactive')

Dagman

class pycondor.Dagman(name, submit=None, extra_lines=None, dag=None, verbose=0)[source]

Dagman object consisting of a series of Jobs and sub-Dagmans to manage.

Note that the submit parameter can be explicitly given or configured by setting the PYCONDOR_SUBMIT_DIR environment variable. An explicitly given value for submit will be used over the environment variable, while the environment variable will be used over a default value.

Parameters

namestr: Name of the Dagman instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Dagman.
submitstr: Path to directory where condor dagman submit files will be written (defaults to the directory was the Dagman was submitted from).
extra_lineslist or None, optional: List of additional lines to be added to submit file.

New in version 0.1.1.
dagDagman, optional: If specified, Dagman will be added to dag as a subdag (default is None).
verboseint, optional: Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Attributes

jobslist: The list of jobs for this Dagman instance to manage.
parentslist: List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Dagman is submitted to HTCondor.
childrenlist: List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Dagman has completed.

Methods

`add_child`(node)	Adds node to children list
`add_children`(nodes)	Adds nodes to the children list
`add_job`(job)	Add job to Dagman
`add_parent`(node)	Adds node to parents list
`add_parents`(nodes)	Adds nodes to the parents list
`add_subdag`(dag)	Add dag to Dagman
`build`([makedirs, fancyname])	Build and saves the submit file for Dagman
`build_submit`([makedirs, fancyname, ...])	Calls build and submit sequentially
`haschildren`()	Checks for any children nodes
`hasparents`()	Checks for any parent nodes
`submit_dag`([submit_options])	Submits Dagman to condor
`visualize`([filename])	Visualize Dagman graph

add_child(node)

Adds node to children list

Parameters

nodeBaseNode: Job or Dagman to append to the children list.

Returns

selfobject: Returns self.

add_children(nodes)

Adds nodes to the children list

Parameters

nodeslist or tuple: List of nodes to append to the children list

Returns

selfobject: Returns self.

add_job(job)[source]

Add job to Dagman

Parameters

jobJob: Job to append to Dagman jobs list.

Returns

selfobject: Returns self.

add_parent(node)

Adds node to parents list

Parameters

nodeBaseNode: Job or Dagman to append to the parents list.

Returns

selfobject: Returns self.

add_parents(nodes)

Adds nodes to the parents list

Parameters

nodeslist or tuple: List of nodes to append to the parents list

Returns

selfobject: Returns self.

add_subdag(dag)[source]

Add dag to Dagman

Parameters

dagDagman: Subdag to append to Dagman jobs list.

Returns

selfobject: Returns self.

build(makedirs=True, fancyname=True)[source]

Build and saves the submit file for Dagman

Parameters

makedirsbool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancynamebool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).

Returns

selfobject: Returns self.

build_submit(makedirs=True, fancyname=True, submit_options=None)[source]

Calls build and submit sequentially

Parameters

makedirsbool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancynamebool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).
submit_optionsstr, optional: Options to be passed to condor_submit_dag for this Dagman (see the condor_submit_dag documentation for possible options).

Returns

selfobject: Returns self.

haschildren()

Checks for any children nodes

Returns

bool: Returns whether or not this node has any child nodes.

hasparents()

Checks for any parent nodes

Returns

bool: Returns whether or not this node has any parent nodes.

submit_dag(submit_options=None)[source]

Submits Dagman to condor

Parameters

submit_optionsstr, optional: Options to be passed to condor_submit_dag for this Dagman (see the condor_submit_dag documentation for possible options).

Returns

selfobject: Returns self.

visualize(filename=None)[source]

Visualize Dagman graph

Parameters

filenamestr or None, optional: File to save graph diagram to. If None then no file is saved. Valid file extensions are ‘png’, ‘pdf’, ‘dot’, ‘svg’, ‘jpeg’, ‘jpg’.