Commit 9666220d authored by Simon Glass's avatar Simon Glass
Browse files

doc: test: Explain how to run pytests in parallel



Add documentation for this so people can try it out. At present it does
not fully work.

Series-to: u-boot
Series-cc: heinrich, stephen, trini
Series-links: 254822
Series-prefix: RESEND
Series-version: 2
Series-changes: 2
- Add documentation for how to run parallel tests

Cover-letter:
test: Try to deal with some co-dependent tests
Tests are supposed to be independent. With driver model tests, the
environment is reset before each test, which ensures that.

With Python tests there is no reset of the board between tests, since we
want to run all the tests as quickly as possible and without needing the
external scripts running constantly.

In principle the Python tests can be independent if they each put the
world back the way they found it, but it turns out that some are not.
This means that some tests cannot be run unless another test is run
first. It also means that tests cannot be run in parallel, e.g. on
sandbox.

This series fixes some of them. Those that remain:

   test_gpt_swap_partitions - not sure?
   test_pinmux_status - not sure?
   test_sqfs_load - cannot be run more than once!
   test_bind_unbind_with_uclass - relies on previous test

The last one would be much better done as a C test, so it doesn't have
to deal with the changing driver tree. There isn't a lot of value in
running the test on a real board, since sandbox should find any bugs
in driver model or the 'bind' command.

If the above can be resolved we can enable parallel tests. On my test
machine (32 threads) it reduces the time from 38 seconds to 7.5s

To use this feature, see the documentaiton added, or:

   pip3 install pytest-xdist

   test/py/test.py -B sandbox --build-dir /tmp/xx -q -k 'not slow' -n32
END
Signed-off-by: Simon Glass's avatarSimon Glass <sjg@chromium.org>
Change-Id: I4b868f615036d4b8bc4d9f6a8555db49be9f6009
parent 6b908422
Pipeline #9174 canceled with stages
......@@ -103,6 +103,36 @@ will be written to `${build_dir}/test-log.html`. This is best viewed in a web
browser, but may be read directly as plain text, perhaps with the aid of the
`html2text` utility.
Running tests in parallel
~~~~~~~~~~~~~~~~~~~~~~~~~
Note: This does not fully work yet and is documented only so you can try to
fix the problems.
First install support for parallel tests::
pip3 install pytest-xdist
Then build sandbox in a suitable build directory. It is not possible to use
the --build flag with xdist.
Finally, run the tests in parallel using the -n flag::
# build sandbox first, in a suitable build directory. It is not possible
# to use the --build flag with -n
test/py/test.py -B sandbox --build-dir /tmp/b/sandbox -q -k 'not slow' -n32
At least the following non-slow tests are known to fail:
- test_fit_ecdsa
- test_bind_unbind_with_uclass
- ut_dm_spi_flash
- test_gpt_rename_partition
- test_gpt_swap_partitions
- test_pinmux_status
- test_sqfs_load
Testing under a debugger
~~~~~~~~~~~~~~~~~~~~~~~~
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment