parsl.dataflow.memoization.BasicMemoizer

class parsl.dataflow.memoization.BasicMemoizer(*, checkpoint_files: Sequence[str] | None = None, checkpoint_period: str | None = None, checkpoint_mode: Literal['task_exit', 'periodic', 'dfk_exit', 'manual'] | None = None, memoize: bool = True)

Memoizer is responsible for ensuring that identical work is not repeated.

When a task is repeated, i.e., the same function is called with the same exact arguments, the result from a previous execution is reused. wiki

The memoizer implementation here does not collapse duplicate calls at call time, but works only when the result of a previous call is available at the time the duplicate call is made.

For instance:

No advantage from                 Memoization helps
memoization here:                 here:

 TaskA                            TaskB
   |   TaskA                        |
   |     |   TaskA                done  (TaskB)
   |     |     |                                (TaskB)
 done    |     |
       done    |
             done

The memoizer creates a lookup table by hashing the function name and its inputs, and storing the results of the function.

When a task is ready for launch, i.e., all of its arguments have resolved, we add its hash to the task datastructure.

__init__(*, checkpoint_files: Sequence[str] | None = None, checkpoint_period: str | None = None, checkpoint_mode: Literal['task_exit', 'periodic', 'dfk_exit', 'manual'] | None = None, memoize: bool = True)

Initialize the memoizer.

KWargs:

• checkpoint_filessequence of str, optional

  List of paths to checkpoint files to load. By default, all checkpoints from the run directory will be restored. This is usually the right behaviour, but this parameter allows that behaviour to be overridden. See parsl.utils.get_all_checkpoints() and parsl.utils.get_last_checkpoint() for helpers.

• checkpoint_periodstr, optional

  Time interval (in “HH:MM:SS”) at which to checkpoint completed tasks. Only has an effect if checkpoint_mode='periodic'.

• checkpoint_modestr, optional

  Checkpoint mode to use, can be 'dfk_exit', 'task_exit', 'periodic' or 'manual'. If set to None, checkpointing will be disabled. Default is None.

• memoize : str, enable memoization or not.

Methods

__init__(*[, checkpoint_files, ...])	Initialize the memoizer.
check_memo(task)	Create a hash of the task and its inputs and check the lookup table for this hash.
checkpoint()	This is the user-facing interface to manual checkpointing.
checkpoint_one(cc)	Checkpoint a single task to a checkpoint file.
checkpoint_queue()	Checkpoint all tasks registered in self.checkpointable_tasks.
close()	Called at DFK shutdown.
load_checkpoints(checkpointDirs)	Load checkpoints from the checkpoint files into a dictionary.
start(*, run_dir, config_run_dir)	Called by the DFK when it starts up.
update_memo_exception(task, e)	Called by the DFK when a task completes with an exception.
update_memo_result(task, r)	Called by the DFK when a task completes with a successful result.

Attributes

run_dir

check_memo(task: TaskRecord) → Future[Any] | None

Create a hash of the task and its inputs and check the lookup table for this hash.

If present, the results are returned.

Parameters:

task (-) – task from the dfk.tasks table

Returns:

• Result of the function if present in table, wrapped in a Future

This call will also set task[‘hashsum’] to the unique hashsum for the func+inputs.

checkpoint() → None

This is the user-facing interface to manual checkpointing.

checkpoint_one(cc: CheckpointCommand) → None

Checkpoint a single task to a checkpoint file.

By default the checkpoints are written to the RUNDIR of the current run under RUNDIR/checkpoints/tasks.pkl

Kwargs:

• task : A task to checkpoint.

Note

Checkpointing only works if memoization is enabled

checkpoint_queue() → None

Checkpoint all tasks registered in self.checkpointable_tasks.

By default the checkpoints are written to the RUNDIR of the current run under RUNDIR/checkpoints/tasks.pkl

Note

Checkpointing only works if memoization is enabled

close() → None

Called at DFK shutdown. This gives the checkpoint system an opportunity for graceful shutdown.

load_checkpoints(checkpointDirs: Sequence[str] | None) → Dict[str, Future]

Load checkpoints from the checkpoint files into a dictionary.

The results are used to pre-populate the memoizer’s lookup_table

Kwargs:

• checkpointDirs (list) : List of run folder to use as checkpoints Eg. [‘runinfo/001’, ‘runinfo/002’]

Returns:

• dict containing, hashed -> future mappings

run_dir: str

start(*, run_dir: str, config_run_dir: str) → None

Called by the DFK when it starts up.

This is an opportunity for the memoization/checkpoint system to initialize itself.

The path to the per-run run directory and the base run directory are passed as parameters.

update_memo_exception(task: TaskRecord, e: BaseException) → None

Called by the DFK when a task completes with an exception.

On every task completion, either this method or update_memo_result will be called, but not both.

This is an opportunity for the memoization/checkpoint system to record the outcome of a task for later discovery by a call to check_memo.

update_memo_result(task: TaskRecord, r: Any) → None

Called by the DFK when a task completes with a successful result.

On every task completion, either this method or update_memo_exception will be called, but not both.

This is an opportunity for the memoization/checkpoint system to record the outcome of a task for later discovery by a call to check_memo.