Running Workflows

Parsl workflows are a Python “main” program that defines Apps, how the Apps are invoked, and how results are passed between different Apps.

The core concept of workflows is that Parsl Apps produce Futures with all features from those in Python’s concurrent.futures module and more.

The Parsl programs typically contain a parsl.load statement to start the services which manage scheduling computations.

from parsl.configs.htex_local import config  # See the "Configuration" documentation
from parsl import python_app  # See the "Apps" documentation
import parsl

@python_app
def f(x):
    return x + 1

with parsl.load(config):
    future = f(1)
    print(f'1 + 1 = {future.result()}')

• Futures
  • AppFutures
  • DataFutures
• Example parallel patterns
  • Bag of Tasks
  • Sequential workflows
  • Parallel workflows
  • Parallel workflows with loops
  • MapReduce
  • Caching expensive initialisation between tasks
• Error handling
  • Exceptions
  • Retries
  • Lazy fail
  • Retry handlers
• Lifted operators
  • Lifted [] operator
  • Lifted . operator
• Memoization and checkpointing
  • App caching
  • Checkpointing
• ParslPoolExecutor Interface
  • Creating a ParslPoolExecutor
  • Using a ParslPoolExecutor
  • Differences