Visualizing Workflows

An image of the directed acyclic graph for a Dagman object can be generated by calling the Dagman visualize method or using the pycondor.visualize function.

pycondor.visualize.visualize(dag, filename=None)

Visualize Dagman graph

Parameters

dagpycondor.Dagman

Dagman to visualize.

filenamestr or None, optional

File to save graph image to. If None then no file is saved. Valid file extensions are ‘png’, ‘pdf’, ‘dot’, ‘svg’, ‘jpeg’, ‘jpg’.

Below is an example workflow where a processing script is run on a list of files, then a script to merge the processed outputs is run, and finally a sub-Dagman which does some final processing is run.

from pycondor import Job, Dagman

# Load files
files = ...

dagman = Dagman(name='example_dagman')

merge = Job(name='merge',
            executable='merge.py',
            dag=dagman)

for idx, f in enumerate(files):
    processing = Job(name='processing_{}'.format(idx),
                     executable='process.py',
                     dag=dagman)
    merge.add_parent(processing)

final_processing = Dagman(name='final_processing',
                          dag=dagman)
final_processing.add_parent(merge)

We can see what the Dagman graph looks like by calling the visualize method.

dagman.visualize('workflow.png')

Jobs are shown by circles and sub-Dagman objects are squares in the diagram. Parent/child relationships are indicated by arrows where parent tasks point to their child tasks.

Note that visualizing Dagman graphs requires both the Graphviz system library and the graphviz Python library to be installed. If you use the Conda package manager, you can install python-graphviz from the conda-forge channel which will install both the system Graphviz as well as the Python library.