py_testing.rst 17.5 KB
Newer Older
1
2
U-Boot pytest suite
===================
3

4
5
Introduction
------------
6
7
8
9
10
11
12
13
14
15

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach to the
U-Boot console, run the entire script of tests against it, and summarize the
results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact with
  U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
  It is asserted that writing test-related code in Python is simpler and more
16
17
  flexible than writing it all in C. But see :doc:`tests_writing` for caveats
  and more discussion / analysis.
18
19
- It is reasonably simple to interact with U-Boot in this way.

20
21
Requirements
------------
22
23
24
25
26
27

The test suite is implemented using pytest. Interaction with the U-Boot console
involves executing some binary and interacting with its stdin/stdout. You will
need to implement various "hook" scripts that are called by the test suite at
the appropriate time.

28
29
30
31
32
33
34
35
In order to run the test suite at a minimum we require that both Python 3 and
pip for Python 3 are installed. All of the required python modules are
described in the requirements.txt file in the /test/py/ directory and can be
installed via the command

.. code-block:: bash

   pip install -r requirements.txt
36
37

In order to execute certain tests on their supported platforms other tools
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
will be required. The following is an incomplete list:

* gdisk
* dfu-util
* dtc
* openssl
* sudo OR guestmount
* e2fsprogs
* util-linux
* coreutils
* dosfstools
* efitools
* mount
* mtools
* sbsigntool
* udisks2

Please use the appropriate commands for your distribution to match these tools
56
up with the package that provides them.
57
58
59
60
61
62
63
64
65

The test script supports either:

- Executing a sandbox port of U-Boot on the local machine as a sub-process,
  and interacting with it over stdin/stdout.
- Executing an external "hook" scripts to flash a U-Boot binary onto a
  physical board, attach to the board's console stream, and reset the board.
  Further details are described later.

66
67
Using `virtualenv` to provide requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
68

69
70
71
The recommended way to run the test suite, in order to ensure reproducibility
is to use `virtualenv` to set up the necessary environment.  This can be done
via the following commands:
72
73


74
75
76
77
78
79
80
.. code-block:: console

    $ cd /path/to/u-boot
    $ sudo apt-get install python3 python3-virtualenv
    $ virtualenv -p /usr/bin/python3 venv
    $ . ./venv/bin/activate
    $ pip install -r test/py/requirements.txt
81

82
83
84
85
Testing sandbox
---------------

To run the test suite on the sandbox port (U-Boot built as a native user-space
86
87
application), simply execute:

88
89
90
.. code-block:: bash

    ./test/py/test.py --bd sandbox --build
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105

The `--bd` option tells the test suite which board type is being tested. This
lets the test suite know which features the board has, and hence exactly what
can be tested.

The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
omit this option and build U-Boot yourself, in whatever way you choose, before
running the test script.

The test script will attach to U-Boot, execute all valid tests for the board,
then print a summary of the test process. A complete log of the test session
will be written to `${build_dir}/test-log.html`. This is best viewed in a web
browser, but may be read directly as plain text, perhaps with the aid of the
`html2text` utility.

106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
Running tests in parallel
~~~~~~~~~~~~~~~~~~~~~~~~~

Note: This does not fully work yet and is documented only so you can try to
fix the problems.

First install support for parallel tests::

    pip3 install pytest-xdist

Then build sandbox in a suitable build directory. It is not possible to use
the --build flag with xdist.

Finally, run the tests in parallel using the -n flag::

    # build sandbox first, in a suitable build directory. It is not possible
    # to use the --build flag with -n
    test/py/test.py -B sandbox --build-dir /tmp/b/sandbox -q -k 'not slow' -n32

At least the following non-slow tests are known to fail:

- test_fit_ecdsa
- test_bind_unbind_with_uclass
- ut_dm_spi_flash
- test_gpt_rename_partition
- test_gpt_swap_partitions
- test_pinmux_status
- test_sqfs_load


136
137
Testing under a debugger
~~~~~~~~~~~~~~~~~~~~~~~~
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152

If you need to run sandbox under a debugger, you may pass the command-line
option `--gdbserver COMM`. This causes two things to happens:

- Instead of running U-Boot directly, it will be run under gdbserver, with
  debug communication via the channel `COMM`. You can attach a debugger to the
  sandbox process in order to debug it. See `man gdbserver` and the example
  below for details of valid values for `COMM`.
- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
  time to execute commands. This is useful if U-Boot is stopped at a breakpoint
  during debugging.

A usage example is:

Window 1:
153
154
155
156

.. code-block:: bash

    ./test/py/test.py --bd sandbox --gdbserver localhost:1234
157
158

Window 2:
159
160
161
162

.. code-block:: bash

    gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
163
164
165
166

Alternatively, you could leave off the `-ex` option and type the command
manually into gdb once it starts.

167
You can use any debugger you wish, as long as it speaks the gdb remote
168
169
170
171
172
173
174
175
protocol, or any graphical wrapper around gdb.

Some tests deliberately cause the sandbox process to exit, e.g. to test the
reset command, or sandbox's CTRL-C handling. When this happens, you will need
to attach the debugger to the new sandbox instance. If these tests are not
relevant to your debugging session, you can skip them using pytest's -k
command-line option; see the next section.

176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
Command-line options
--------------------

--board-type, --bd, -B
  set the type of the board to be tested. For example, `sandbox` or `seaboard`.

--board-identity`, --id
  sets the identity of the board to be tested. This allows differentiation
  between multiple instances of the same type of physical board that are
  attached to the same host machine. This parameter is not interpreted by th
  test script in any way, but rather is simply passed to the hook scripts
  described below, and may be used in any site-specific way deemed necessary.

--build
  indicates that the test script should compile U-Boot itself before running
  the tests. If using this option, make sure that any environment variables
  required by the build process are already set, such as `$CROSS_COMPILE`.

--buildman
  indicates that `--build` should use buildman to build U-Boot. There is no need
  to set $CROSS_COMPILE` in this case since buildman handles it.

--build-dir
  sets the directory containing the compiled U-Boot binaries. If omitted, this
  is `${source_dir}/build-${board_type}`.

--result-dir
  sets the directory to write results, such as log files, into.
  If omitted, the build directory is used.

--persistent-data-dir
  sets the directory used to store persistent test data. This is test data that
  may be re-used across test runs, such as file-system images.
209

210
211
212
213
214
215
`pytest` also implements a number of its own command-line options. Commonly used
options are mentioned below. Please see `pytest` documentation for complete
details. Execute `py.test --version` for a brief summary. Note that U-Boot's
test.py script passes all command-line arguments directly to `pytest` for
processing.

216
217
-k
  selects which tests to run. The default is to run all known tests. This
218
219
  option takes a single argument which is used to filter test names. Simple
  logical operators are supported. For example:
220
221
222

  - `'-k ums'` runs only tests with "ums" in their name.
  - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
223
224
    case, "ut_dm" is a parameter to a test rather than the test name. The full
    test name is e.g. "test_ut[ut_dm_leak]".
225
226
227
  - `'-k not reset'` runs everything except tests with "reset" in their name.
  - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
  - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
228
    their name.
229
230
231

-s
  prevents pytest from hiding a test's stdout. This allows you to see
232
  U-Boot's console log in real time on pytest's stdout.
233

234
235
Testing real hardware
---------------------
236
237
238
239
240
241
242
243
244

The tools and techniques used to interact with real hardware will vary
radically between different host and target systems, and the whims of the user.
For this reason, the test suite does not attempt to directly interact with real
hardware in any way. Rather, it executes a standardized set of "hook" scripts
via `$PATH`. These scripts implement certain actions on behalf of the test
suite. This keeps the test suite simple and isolated from system variances
unrelated to U-Boot features.

245
246
Hook scripts
~~~~~~~~~~~~
247

248
249
Environment variables
'''''''''''''''''''''
250
251
252
253
254
255
256
257
258
259

The following environment variables are set when running hook scripts:

- `UBOOT_BOARD_TYPE` the board type being tested.
- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
  specified.
- `UBOOT_SOURCE_DIR` the U-Boot source directory.
- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
- `UBOOT_BUILD_DIR` the U-Boot build directory.
- `UBOOT_RESULT_DIR` the test result directory.
260
- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
261

262
263
u-boot-test-console
'''''''''''''''''''
264
265
266
267
268
269

This script provides access to the U-Boot console. The script's stdin/stdout
should be connected to the board's console. This process should continue to run
indefinitely, until killed. The test suite will run this script in parallel
with all other hooks.

270
271
This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
via exec().
272

273
If you are able to run U-Boot under a hardware simulator such as QEMU, then
274
275
276
277
278
279
280
you would likely spawn that simulator from this script. However, note that
`u-boot-test-reset` may be called multiple times per test script run, and must
cause U-Boot to start execution from scratch each time. Hopefully your
simulator includes a virtual reset button! If not, you can launch the
simulator from `u-boot-test-reset` instead, while arranging for this console
process to always communicate with the current simulator instance.

281
282
u-boot-test-flash
'''''''''''''''''
283
284

Prior to running the test suite against a board, some arrangement must be made
285
so that the board executes the particular U-Boot binary to be tested. Often
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
this involves writing the U-Boot binary to the board's flash ROM. The test
suite calls this hook script for that purpose.

This script should perform the entire flashing process synchronously; the
script should only exit once flashing is complete, and a board reset will
cause the newly flashed U-Boot binary to be executed.

It is conceivable that this script will do nothing. This might be useful in
the following cases:

- Some other process has already written the desired U-Boot binary into the
  board's flash prior to running the test suite.
- The board allows U-Boot to be downloaded directly into RAM, and executed
  from there. Use of this feature will reduce wear on the board's flash, so
  may be preferable if available, and if cold boot testing of U-Boot is not
  required. If this feature is used, the `u-boot-test-reset` script should
302
  perform this download, since the board could conceivably be reset multiple
303
304
305
306
307
308
309
310
  times in a single test run.

It is up to the user to determine if those situations exist, and to code this
hook script appropriately.

This script will typically be implemented by calling out to some SoC- or
board-specific vendor flashing utility.

311
312
u-boot-test-reset
'''''''''''''''''
313
314
315
316
317
318
319
320
321
322
323
324

Whenever the test suite needs to reset the target board, this script is
executed. This is guaranteed to happen at least once, prior to executing the
first test function. If any test fails, the test infra-structure will execute
this script again to restore U-Boot to an operational state before running the
next test function.

This script will likely be implemented by communicating with some form of
relay or electronic switch attached to the board's reset signal.

The semantics of this script require that when it is executed, U-Boot will
start running from scratch. If the U-Boot binary to be tested has been written
325
326
to flash, pulsing the board's reset signal is likely all this script needs to
do. However, in some scenarios, this script may perform other actions. For
327
328
329
330
331
example, it may call out to some SoC- or board-specific vendor utility in order
to download the U-Boot binary directly into RAM and execute it. This would
avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
saving wear on the flash chip(s).

332
333
Examples
''''''''
334

335
https://source.denx.de/u-boot/u-boot-test-hooks contains some working example hook
336
337
338
scripts, and may be useful as a reference when implementing hook scripts for
your platform. These scripts are not considered part of U-Boot itself.

339
340
Board-type-specific configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
341
342
343
344
345
346
347
348
349
350
351

Each board has a different configuration and behaviour. Many of these
differences can be automatically detected by parsing the `.config` file in the
build directory. However, some differences can't yet be handled automatically.

For each board, an optional Python module `u_boot_board_${board_type}` may exist
to provide board-specific information to the test script. Any global value
defined in these modules is available for use by any test function. The data
contained in these scripts must be purely derived from U-Boot source code.
Hence, these configuration files are part of the U-Boot source tree too.

352
353
Execution environment configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370

Each user's hardware setup may enable testing different subsets of the features
implemented by a particular board's configuration of U-Boot. For example, a
U-Boot configuration may support USB device mode and USB Mass Storage, but this
can only be tested if a USB cable is connected between the board and the host
machine running the test script.

For each board, optional Python modules `u_boot_boardenv_${board_type}` and
`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
board-specific and board-identity-specific information to the test script. Any
global value defined in these modules is available for use by any test
function. The data contained in these is specific to a particular user's
hardware configuration. Hence, these configuration files are not part of the
U-Boot source tree, and should be installed outside of the source tree. Users
should set `$PYTHONPATH` prior to running the test script to allow these
modules to be loaded.

371
372
Board module parameter usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
373
374
375
376

The test scripts rely on the following variables being defined by the board
module:

377
- none at present
378

379
380
U-Boot `.config` feature usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
381
382
383
384
385
386
387
388
389
390
391
392
393
394

The test scripts rely on various U-Boot `.config` features, either directly in
order to test those features, or indirectly in order to query information from
the running U-Boot instance in order to test other features.

One example is that testing of the `md` command requires knowledge of a RAM
address to use for the test. This data is parsed from the output of the
`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.

For a complete list of dependencies, please search the test scripts for
instances of:

- `buildconfig.get(...`
- `@pytest.mark.buildconfigspec(...`
395
- `@pytest.mark.notbuildconfigspec(...`
396

397
398
Complete invocation example
~~~~~~~~~~~~~~~~~~~~~~~~~~~
399
400
401
402
403
404
405

Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
any required environment configuration Python modules into $HOME/ubtest/py,
then you would likely invoke the test script as follows:

If U-Boot has already been built:

406
407
408
.. code-block:: bash

    PATH=$HOME/ubtest/bin:$PATH \
409
    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
410
411
412
413
    ./test/py/test.py --bd seaboard

If you want the test script to compile U-Boot for you too, then you likely
need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
414
follows:
415

416
417
418
.. code-block:: bash

    CROSS_COMPILE=arm-none-eabi- \
419
    PATH=$HOME/ubtest/bin:$PATH \
420
    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
421
422
    ./test/py/test.py --bd seaboard --build

423
424
or, using buildman to handle it:

425
426
.. code-block:: bash

427
428
429
430
    PATH=$HOME/ubtest/bin:$PATH \
    PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
    ./test/py/test.py --bd seaboard --build --buildman

431
432
Writing tests
-------------
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457

Please refer to the pytest documentation for details of writing pytest tests.
Details specific to the U-Boot test suite are described below.

A test fixture named `u_boot_console` should be used by each test function. This
provides the means to interact with the U-Boot console, and retrieve board and
environment configuration information.

The function `u_boot_console.run_command()` executes a shell command on the
U-Boot console, and returns all output from that command. This allows
validation or interpretation of the command output. This function validates
that certain strings are not seen on the U-Boot console. These include shell
error messages and the U-Boot sign-on message (in order to detect unexpected
board resets). See the source of `u_boot_console_base.py` for a complete list of
"bad" strings. Some test scenarios are expected to trigger these strings. Use
`u_boot_console.disable_check()` to temporarily disable checking for specific
strings. See `test_unknown_cmd.py` for an example.

Board- and board-environment configuration values may be accessed as sub-fields
of the `u_boot_console.config` object, for example
`u_boot_console.config.ram_base`.

Build configuration values (from `.config`) may be accessed via the dictionary
`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
names.