1. 19 Sep, 2021 1 commit
    • Simon Glass's avatar
      doc: test: Explain how to run pytests in parallel · 9666220d
      Simon Glass authored
      
      
      Add documentation for this so people can try it out. At present it does
      not fully work.
      
      Series-to: u-boot
      Series-cc: heinrich, stephen, trini
      Series-links: 254822
      Series-prefix: RESEND
      Series-version: 2
      Series-changes: 2
      - Add documentation for how to run parallel tests
      
      Cover-letter:
      test: Try to deal with some co-dependent tests
      Tests are supposed to be independent. With driver model tests, the
      environment is reset before each test, which ensures that.
      
      With Python tests there is no reset of the board between tests, since we
      want to run all the tests as quickly as possible and without needing the
      external scripts running constantly.
      
      In principle the Python tests can be independent if they each put the
      world back the way they found it, but it turns out that some are not.
      This means that some tests cannot be run unless another test is run
      first. It also means that tests cannot be run in parallel, e.g. on
      sandbox.
      
      This series fixes some of them. Those that remain:
      
         test_gpt_swap_partitions - not sure?
         test_pinmux_status - not sure?
         test_sqfs_load - cannot be run more than once!
         test_bind_unbind_with_uclass - relies on previous test
      
      The last one would be much better done as a C test, so it doesn't have
      to deal with the changing driver tree. There isn't a lot of value in
      running the test on a real board, since sandbox should find any bugs
      in driver model or the 'bind' command.
      
      If the above can be resolved we can enable parallel tests. On my test
      machine (32 threads) it reduces the time from 38 seconds to 7.5s
      
      To use this feature, see the documentaiton added, or:
      
         pip3 install pytest-xdist
      
         test/py/test.py -B sandbox --build-dir /tmp/xx -q -k 'not slow' -n32
      END
      Signed-off-by: Simon Glass's avatarSimon Glass <sjg@chromium.org>
      Change-Id: I4b868f615036d4b8bc4d9f6a8555db49be9f6009
      9666220d
  2. 12 Mar, 2021 1 commit
  3. 28 Feb, 2021 1 commit
  4. 23 Jan, 2021 1 commit
  5. 16 Apr, 2020 1 commit
  6. 11 Apr, 2020 1 commit
    • Simon Glass's avatar
      test/py: Allow using buildman to build U-Boot · f5ec7eeb
      Simon Glass authored and Tom Rini's avatar Tom Rini committed
      
      
      It is a pain to have to set the CROSS_COMPILE environment variable when
      using test.py's --build option. It is possible to get this using the -A
      option from buildman. But it seems better to just use buildman to do the
      build when it is available.
      
      However using buildman adds a new dependency to the test system which we
      want to avoid. So leave the default as is and add a flag to make it use
      buildman.
      
      Note that most of these changes relate to test.py and the parts of the
      travis/gitlab/azure scripts which relate to running test and building a
      suitable U-Boot to run the tests on.
      Signed-off-by: Simon Glass's avatarSimon Glass <sjg@chromium.org>
      Reviewed-by: Tom Rini's avatarTom Rini <trini@konsulko.com>
      f5ec7eeb
  7. 30 Oct, 2019 1 commit
  8. 22 Apr, 2019 1 commit
  9. 08 Oct, 2018 1 commit
    • Simon Glass's avatar
      binman: Run tests concurrently · 11ae93ee
      Simon Glass authored
      
      
      At present the tests run one after the other using a single CPU. This is
      not very efficient. Bring in the concurrencytest module and run the tests
      concurrently, using one process for each CPU by default. A -P option
      allows this to be overridden, which is necessary for code-coverage to
      function correctly.
      
      This requires fixing a few tests which are currently not fully
      independent.
      
      At some point we might consider doing this across all pytests in U-Boot.
      There is a pytest version that supports specifying the number of processes
      to use, but it did not work for me.
      Signed-off-by: Simon Glass's avatarSimon Glass <sjg@chromium.org>
      11ae93ee
  10. 22 Mar, 2018 2 commits
  11. 23 Oct, 2017 1 commit
  12. 29 Sep, 2017 1 commit
  13. 12 Apr, 2016 1 commit
  14. 15 Feb, 2016 1 commit
  15. 21 Jan, 2016 1 commit
    • Stephen Warren's avatar
      test/py: Implement pytest infrastructure · d201506c
      Stephen Warren authored and Simon Glass's avatar Simon Glass committed
      This tool aims to test U-Boot by executing U-Boot shell commands using the
      console interface. A single top-level script exists to execute or attach
      to the U-Boot console, run the entire script of tests against it, and
      summarize the results. Advantages of this approach are:
      
      - Testing is performed in the same way a user or script would interact
        with U-Boot; there can be no disconnect.
      - There is no need to write or embed test-related code into U-Boot itself.
        It is asserted that writing test-related code in Python is simpler and
        more flexible that writing it all in C.
      - It is reasonably simple to interact with U-Boot in this way.
      
      A few simple tests are provided as examples. Soon, we should convert as
      many as possible of the other tests in test/* and test/cmd_ut.c too.
      
      The hook scripts, relay control utilities, and udev rules I use for my
      own HW setup are published at https://github.com/swarren/uboot-test-hooks
      
      .
      
      See README.md for more details!
      Signed-off-by: Stephen Warren's avatarStephen Warren <swarren@wwwdotorg.org>
      Signed-off-by: Stephen Warren's avatarStephen Warren <swarren@nvidia.com>
      Tested-by: default avatarMichal Simek <michal.simek@xilinx.com>
      Tested-by: Simon Glass's avatarSimon Glass <sjg@chromium.org>
      Acked-by: Simon Glass <sjg@chromium.org> #v3
      d201506c