File: D:/HostingSpaces/SBogers10/kommabasic.nl/vendor/phpbench/phpbench/docs/writing-benchmarks.rst
Writing Benchmarks
==================
Benchmark classes have the following characteristics:
- The class and filename must be the same.
- Class methods that start with ``bench`` will be executed by the benchrunner
and timed.
PHPBench does not require that the benchmark class be aware of PHPBench
library - it does not need to extend a parent class or implement an interface.
The following is a simple benchmark class:
.. code-block:: php
<?php
// HashBench.php
class HashBench
{
public function benchMd5()
{
hash('md5', 'Hello World!');
}
public function benchSha1()
{
hash('sha1', 'Hello World!');
}
}
And it can be executed as follows:
.. code-block:: bash
$ phpbench run examples/HashBench.php --progress=dots
Running benchmarks.
...
3 subjects, 30 iterations, 30000 revs, 0 rejects
⅀T: 30543μs μSD/r 0.05μs μRSD/r: 4.83%
min mean max: 0.78 1.02 1.47 (μs/r)
.. note::
The above command does not generate a report, add ``--report=default`` to
view something useful.
PHPBench reads docblock annotations in the benchmark class. Annotations can be
placed in the class docblock, or on individual methods docblocks.
.. note::
Instead of prefixing a method with ``bench`` you can use the
``@Subject`` annotation or specify a :ref:`custom pattern <configuration_subject_pattern>`.
.. _revolutions:
Revolutions
-----------
When testing units of code where microsecond accuracy is important, it is
necessary to increase the number of *revolutions* performed by the
benchmark runner. The term "revolutions" (invented here) refers to the number
of times the benchmark is executed consecutively within a single time
measurement.
We can arrive at a more accurate measurement by determining the mean time
from multiple revolutions (i.e. ``time / revolutions``) than we could with a
single revolution. In other words, more revolutions means more precision.
Revolutions can be specified using the ``@Revs`` annotation:
.. code-block:: php
<?php
/**
* @Revs(1000)
*/
class HashBench
{
// ...
}
You may also specify an array:
.. code-block:: php
<?php
/**
* @Revs({1, 8, 64, 4096})
*/
class HashBench
{
// ...
}
Revolutions can also be overridden from the :ref:`command line
<overriding_iterations_and_revolutions>`.
.. _iterations:
Iterations
----------
Iterations specify how many samples should be taken - i.e. how many times we
run the :ref:`revolutions <revolutions>` and capture time and memory information (for example).
By looking at the separate time measurement of each iteration we can determine
how *stable* the readings are. The less the measurements differ from each
other, the more stable the benchmark.
.. note::
In a *perfect* environment the readings would all be *exactly* the same -
but such an environment is unlikely to exist
Iterations can be specified using the ``@Iterations`` annotation:
.. code-block:: php
<?php
/**
* @Iterations(5)
*/
class HashBench
{
// ...
}
As with :ref:`revolutions <revolutions>`, you may also specify an array.
Iterations can also be overridden from the :ref:`command line
<overriding_iterations_and_revolutions>`.
You can instruct PHPBench to continuously run the iterations until the
deviation of each iteration fits within a given margin of error by using the
``--retry-threshold``. See :ref:`retry_threshold` for more information.
Benchmark Hooks
---------------
Method hooks
~~~~~~~~~~~~
Any number of methods can be executed both before and after each benchmark
**subject** using the ``@BeforeMethods`` and
``@AfterMethods`` annotations. Before methods are useful for bootstrapping
your environment, for example:
.. code-block:: php
<?php
/**
* @BeforeMethods({"init"})
*/
class HashBench
{
private $hasher;
public function init()
{
$this->hasher = new Hasher();
}
public function benchMd5()
{
$this->hasher->md5('Hello World!');
}
}
Multiple before and after methods can be specified.
.. note::
If before and after methods are used when the ``@ParamProviders``
annotations are used, then they will also be passed the parameters.
Class Hooks
~~~~~~~~~~~
Sometimes you will want to perform actions which establish an *external*
state. For example, creating or populating a database, creating files, etc.
This can be achieved by creating **static** methods within your benchmark
class and adding the ``@BeforeClassMethods`` and ``@AfterClassMethods``:
These methods will be executed by the runner once per benchmark class.
.. code-block:: php
<?php
/**
* @BeforeClassMethods({"initDatabase"})
*/
class DatabaseBench
{
public static function initDatabase()
{
// init database here.
}
// ...
}
.. note::
These methods are static and are executed in a process that is separate
from that from which your iterations will be executed. Therefore **state
will not be carried over to your iterations!**.
.. _parameters:
Parameterized Benchmarks
------------------------
Parameter sets can be provided to benchmark subjects. For example:
.. code-block:: php
<?php
class HashBench
{
public function provideStrings()
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
/**
* @ParamProviders({"provideStrings"})
*/
public function benchMd5($params)
{
hash('md5', $params['string']);
}
}
The ``benchMd5`` subject will now be benchmarked with each parameter set.
The param provider can return a set of parameters using any `iterable`.
For example the above could also be retuned as an array:
.. code-block:: php
<?php
class HashBench
{
public function provideStrings()
{
return [
'hello' => [ 'string' => 'Hello World!' ],
'goodbye' => [ 'string' => 'Goodbye Cruel World!' ]
];
}
}
.. warning::
It should be noted that Generators are consumed completely before the
subject is executed. If you have a very large data set, it will be read
completely into memory.
Multiple parameter providers can be used, in which case the data sets will be
combined into a `cartesian product`_ - all possible combinations of the
parameters will be generated, for example:
.. code-block:: php
<?php
class HashBench
{
public function provideStrings()
{
yield 'hello' => [ 'string' => 'Hello World!' ];
yield 'goodbye' => [ 'string' => 'Goodbye Cruel World!' ];
}
public function provideNumbers()
{
yield 'md5' => [ 'algorithm' => 'md5' ];
yield 'sha1' => [ 'algorithm' => 'sha1' ];
}
/**
* @ParamProviders({"provideStrings", "provideNumbers"})
*/
public function benchHash($params)
{
hash($params['algorithm'], $params['string']);
}
}
Will result in the following parameter benchmark scenarios:
.. code-block:: php
<?php
// #0
['string' => 'Hello World!', 'algorithm' => 'md5'];
// #1
['string' => 'Goodbye Cruel World!', 'algorithm' => 'md5'[;
// #2
['string' => 'Hello World!', 'algorithm' => 'sha1'];
// #3
['string' => 'Goodbye Cruel World!', 'algorithm' => 'sha1'];
.. _groups:
Groups
------
You can assign benchmark subjects to groups using the ``@Groups`` annotation.
.. code-block:: php
<?php
/**
* @Groups({"hash"})
*/
class HashBench
{
// ...
}
The group can then be targeted using the command line interface.
Skipping Subjects
-----------------
You can skip subjects by using the ``@Skip`` annotation:
.. code-block:: php
<?php
class HashBench extends Foobar
{
/**
* @Skip()
*/
public function testFoobar()
{
}
}
Extending Values
----------------
When working with annotations which accept an array value, you may wish to
extend the values of the same annotation from ancestor classes. This can be
accomplished using the ``extend`` option.
.. code-block:: php
<?php
abstract class AbstractHash
{
/**
* @Groups({"md5"})
*/
abstract public function benchMd5();
}
/**
* @Groups({"my_hash_implementation"}, extend=true)
*/
class HashBench extends AbstractHash
{
public function benchMd5()
{
// ...
}
}
The ``benchHash`` subject will now be in both the ``md5`` and
``my_hash_implementation`` groups.
This option is available on all annotations featuring a list of values.
Sleeping
--------
Sometimes it may be necessary to pause between iterations in order to let
the system recover. Use the ``@Sleep`` annotation, specifying the number of
**microseconds** required:
.. code-block:: php
<?php
class HashBench
{
/**
* @Iterations(10)
* @Sleep(1000000)
*/
public function benchMd5()
{
md5('Hello World');
}
}
The above example will pause (sleep) for 1 second *after* each iteration.
.. note::
This can be overridden using the ``--sleep`` option from the CLI.
.. _time_unit:
Time Units
----------
Specify *output* time units using the ``@OutputTimeUnit`` annotation
(`precision` is optional):
.. code-block:: php
<?php
class HashBench
{
/**
* @Iterations(10)
@OutputTimeUnit("seconds", precision=3)
*/
public function benchSleep()
{
sleep(2);
}
}
The following time units are available:
- ``microseconds``
- ``milliseconds``
- ``seconds``
- ``minutes``
- ``hours``
- ``days``
.. _throughput:
.. _mode:
Throughput Representation
-------------------------
The output mode determines how the measurements are presented, either `time`
or `throughput`. `time` mode is the default and shows the average execution
time of a single :ref:`revolution <revolutions>`. `throughput` shows how many *operations*
are executed within a single time unit:
.. code-block:: php
<?php
class HashBench
{
/**
* @OutputTimeUnit("seconds")
* @OutputMode("throughput")
*/
public function benchMd5()
{
hash('md5', 'Hello World!');
}
}
PHPBench will then render all measurements for `benchMd5` similar to
`363,874.536ops/s`.
Warm Up
-------
Use the ``@Warmup`` annotation to execute any number of revolutions before
actually measuring the revolutions time.
.. code-block:: php
<?php
// ...
class ReportBench
{
// ...
/**
* @Warmup(2)
* @Revs(10)
*/
public function benchGenerateReport()
{
$this->generator->generateMyComplexReport();
}
}
As with :ref:`revolutions <revolutions>`, you may also specify an array.
.. _timeouts:
Timeout
-------
Use the ``@Timeout`` annotation to specify the maximum number of seconds
before an iteration timesout and fails. The following example will fail after
0.1 seconds:
.. code-block:: php
<?php
// ...
class ReportBench
{
/**
* @Timeout(0.1)
*/
public function benchGenerateReport()
{
sleep(1);
}
}
.. _assertions:
Assertions
----------
You can annotate your benchmarks with *assertions* which will cause PHPBench
to report failures and exit with a non-zero exit code if they fail.
For example, assert that the mode is less than 1234 microseconds:
.. code-block:: php
<?php
/**
* @Assert("variant.mode < 100 microseconds")
*/
public function benchFoobar()
{
// ...
}
Assert a throughput greater than 0.25ops/µs:
.. code-block:: php
<?php
/**
* @Assert("variant.mode > 0.25 ops/microsecond")
*/
public function benchFoobar()
{
// ...
}
You can also specify assertions from the command line:
.. code-block:: bash
$ phpbench run --assert='variant.mode > 0.25 ops/microsecond'
See :doc:`assertions` for more information.
.. _cartesian product: https://en.wikipedia.org/wiki/Cartesian_product
.. _Relative standard deviation: https://en.wikipedia.org/wiki/Coefficient_of_variation