Measuring performance with parsl-perf
parsl-perf is tool for making basic performance measurements of Parsl
configurations.
It runs repeated batches of no-op apps, giving a measurement of tasks per
second. By default, parsl-perf will attempt to estimate a batch size
that will take 2 minutes to execute.
This can give a basic measurement of some of the overheads in task execution.
parsl-perf must be invoked with a configuration file, which is a Python
file containing a variable config which contains a Config object, or
a function fresh_config which returns a Config object. The
fresh_config format is the same as used with the pytest test suite.
To specify a parsl_resource_specification for tasks, add a --resources
argument.
To change the target runtime from the default of 120 seconds, add a
--time parameter.
To change the iteration algorithm used to pick the next batch size and to
decide when to stop, use the --iterate parameter:
--iterate=estimate(the default) will run a small batch, then repeatedly re-estimate a new batch size until it finds one that takes at least 75% of the target run time. This will rapidly find a batch size of around the target size.
--iterate=exponentialwill start with a small batch, then double the size in each subsequent batch until reaching the target run time. This is useful for understanding how a configuration scales down as batch sizes increase, which is sometimes sub-linear, and to get more consistent batch sizes across runs ofparsl-perf.
--iterate=10,10000,10000(or any other sequence of batch sizes) will run a fixed sequence of batches: in this example, first a batch of 10 tasks, then two batches of 10000 tasks. This is useful for making tests that compare fixed batch sizes along with warmup steps between different configurations, without the additional batches ofestimateandexponentialto discover time-sized (rather than count-sized) batch sizes.
For example:
$ python -m parsl.benchmark.perf --config parsl/tests/configs/workqueue_ex.py --resources '{"cores":1, "memory":0, "disk":0}'
==== Iteration 1 ====
Will run 10 tasks to target 120 seconds runtime
Submitting tasks / invoking apps
warning: using plain-text when communicating with workers.
warning: use encryption with a key and cert when creating the manager.
All 10 tasks submitted ... waiting for completion
Submission took 0.008 seconds = 1248.676 tasks/second
Runtime: actual 3.668s vs target 120s
Tasks per second: 2.726
[...]
==== Iteration 4 ====
Will run 57640 tasks to target 120 seconds runtime
Submitting tasks / invoking apps
All 57640 tasks submitted ... waiting for completion
Submission took 34.839 seconds = 1654.487 tasks/second
Runtime: actual 364.387s vs target 120s
Tasks per second: 158.184
Cleaning up DFK
The end