Commit c5ee8e88 authored by Stefano Babic's avatar Stefano Babic
Browse files

Sync with 2021.11



Signed-off-by: Stefano Babic's avatarStefano Babic <sbabic@denx.de>
parent 29122638
.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
.. SPDX-License-Identifier: GPL-2.0-only
=============================================
Language Bindings
=============================================
Overview
--------
In general, SWUpdate is agnostic to a particular language it is operated from,
thanks to :doc:`SWUpdate's socket-based control <swupdate-ipc>` and
:doc:`progress APIs <progress>` for external programs. As long as the language
of choice has proper socket (library) support, SWUpdate can be operated with it.
However, for convenience, a Lua language binding in terms of a shared library,
currently ``lua_swupdate.so.0.1``, is provided.
Lua Language Binding
--------------------
The Lua language binding is realized in terms of the ``swupdate`` module that
defines three bindings, namely for the control interface, the progress
interface, and a convenience function yielding a table holding all local
network interfaces including their IP addresses and submasks.
Note that, depending on the filesystem location of the Lua binding's shared
library, Lua's ``package.cpath`` may have to be adapted by setting the
environment variable ``LUA_CPATH``, modifying ``package.cpath`` prior to
a ``require('lua_swupdate')``, or , as last resort, using ``package.loadlib()``
instead of ``require('lua_swupdate')``.
Control Interface
.................
The ``swupdate`` module's control interface binding conveniently makes
:doc:`SWUpdate's socket-based control API <swupdate-ipc>` available to pure Lua.
The binding is captured in the ``swupdate_control`` object that is returned
by a call to ``swupdate.control()``. This object offers the three methods
``connect()``, ``write(<chunkdata>)``, and ``close()``:
The ``connect()`` method initializes the connection to SWUpdate's control
socket, sends ``REQ_INSTALL``, and waits for ``ACK`` or ``NACK``, returning the
socket connection file descriptor, mostly for information purposes, or, in case
of an error, ``nil`` plus an error message.
The artifact's data can then be sent to SWUpdate via the ``write(<chunkdata>)``
method, returning ``true``, or, in case of errors, ``nil`` plus an error message.
Finally, the ``close()`` method closes the connection to SWUpdate's control
socket after which it waits for SWUpdate to complete the update transaction and
executes the post-install command, if given.
The following example snippet illustrates how to use the control interface binding:
::
local artifact = io.open("/some/path/to/artifact.swu", "rb" )
swupdate = require('lua_swupdate')
local ctrl = swupdate.control()
if not ctrl:connect() then
-- Deliberately neglecting error message.
io.stderr:write("Error connecting to SWUpdate control socket.\n")
return
end
while true do
local chunk = artifact:read(1024)
if not chunk then break end
if not ctrl:write(chunk) then
-- Deliberately neglecting error message.
io.stderr:write("Error writing to SWUpdate control socket.\n")
break
end
end
local res, msg = ctrl:close()
if not res then
io.stderr:write(string.format("Error finalizing update: %s\n", msg))
end
Progress Interface
..................
The ``swupdate`` module's progress interface binding conveniently makes
:doc:`SWUpdate's socket-based progress API <progress>` available to pure Lua.
The binding is captured in the ``swupdate_progress`` object that is returned
by a call to ``swupdate.progress()``. This object offers the three methods
``connect()``, ``receive()``, and ``close()``:
The ``connect()`` method connects to SWUpdate's progress socket, waiting until
the connection has been established. Note that it is only really required to
explicitly call ``connect()`` to reestablish a broken connection as the
``swupdate_progress`` object's instantiation already initiates the connection.
The ``receive()`` method returns a table representation of the ``struct
progress_msg`` described in the :doc:`progress interface's API description
<progress>`.
The ``close()`` method deliberately closes the connection to SWUpdate's progress
socket.
IPv4 Interface
..............
For convenience, the ``swupdate`` module provides the ``ipv4()`` method
returning a table holding the local network interfaces as the table's keys and
their space-separated IP addresses plus subnet masks as respective values.
.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
.. SPDX-License-Identifier: GPL-2.0-only
==================================
meta-swupdate: building with Yocto
==================================
Overview
========
The Yocto-Project_ is a community project under the umbrella of the Linux
Foundation that provides tools and template to create the own custom Linux-based
software for embedded systems.
.. _Yocto-Project: http://www.yoctoproject.org
.. _meta-SWUpdate: https://github.com/sbabic/meta-swupdate.git
Add-on features can be added using *layers*. meta-swupdate_ is the layer to
cross-compile the SWUpdate application and to generate the compound SWU images
containing the release of the product. It contains the required changes
for mtd-utils and for generating Lua. Using meta-SWUpdate is a
straightforward process. As described in Yocto's documentation
about layers, you should include it in your *bblayers.conf* file to use it.
Add meta-SWUpdate as usual to your bblayers.conf. You have also
to add meta-oe to the list.
In meta-SWUpdate there is a recipe to generate an initrd with a
rescue system with SWUpdate. Use:
::
MACHINE=<your machine> bitbake swupdate-image
You will find the result in your tmp/deploy/<your machine> directory.
How to install and start an initrd is very target specific - please
check in the documentation of your bootloader.
What about libubootenv ?
========================
This is a common issue when SWUpdate is built. SWUpdate depends on this library,
that is generated from the U-Boot's sources. This library allows one to safe modify
the U-Boot environment. It is not required if U-Boot is not used as bootloader.
If SWUpdate cannot be linked, you are using an old version of U-Boot (you need
at least 2016.05). If this is the case, you can add your own recipe for
the package u-boot-fw-utils, adding the code for the library.
It is important that the package u-boot-fw-utils is built with the same
sources of the bootloader and for the same machine. In fact, the target
can have a default environment linked together with U-Boot's code,
and it is not (yet) stored into a storage. SWUpdate should be aware of
it, because it cannot read it: the default environment must be linked
as well to SWUpdate's code. This is done inside the libubootenv.
If you build for a different machine, SWUpdate will destroy the
environment when it tries to change it the first time. In fact,
a wrong default environment is taken, and your board won't boot again.
To avoid possible mismatch, a new library was developed to be hardware independent.
A strict match with the bootloader is not required anymore. The meta-swupdate layer
contains recipes to build the new library (`libubootenv`) and adjust SWUpdate to be linked
against it. To use it as replacement for u-boot-fw-utils:
- set PREFERRED_PROVIDER_u-boot-fw-utils = "libubootenv"
- add to SWUpdate config:
::
CONFIG_UBOOT=y
CONFIG_UBOOT_NEWAPI=y
With this library, you can simply pass the default environment as file (u-boot-initial-env).
It is recommended for new project to switch to the new library to become independent from
the bootloader.
The swupdate class
==================
meta-swupdate contains a class specific for SWUpdate. It helps to generate the
SWU image starting from images built inside the Yocto. It requires that all
components, that means the artifacts that are part of the SWU image, are present
in the Yocto's deploy directory. This class should be inherited by recipes
generating the SWU. The class defines new variables, all of them have the prefix
*SWUPDATE_* in the name.
- **SWUPDATE_IMAGES** : this is a list of the artifacts to be packaged together.
The list contains the name of images without any extension for MACHINE or
filetype, that are added automatically.
Example :
::
SWUPDATE_IMAGES = "core-image-full-cmdline uImage"
- **SWUPDATE_IMAGES_FSTYPES** : extension of the artifact. Each artifact can
have multiple extension according to the IMAGE_FSTYPES variable.
For example, an image can be generated as tarball and as UBIFS for target.
Setting the variable for each artifact tells the class which file must
be packed into the SWU image.
::
SWUPDATE_IMAGES_FSTYPES[core-image-full-cmdline] = ".ubifs"
- **SWUPDATE_IMAGES_NOAPPEND_MACHINE** : flag to use drop the machine name from the
artifact file. Most images in *deploy* have the name of the Yocto's machine in the
filename. The class adds automatically the name of the MACHINE to the file, but some
artifacts can be deployed without it.
::
SWUPDATE_IMAGES_NOAPPEND_MACHINE[my-image] = "1"
- **SWUPDATE_SIGNING** : if set, the SWU is signed. There are 3 allowed values:
RSA, CMS, CUSTOM. This value determines used signing mechanism.
- **SWUPDATE_SIGN_TOOL** : instead of using openssl, use SWUPDATE_SIGN_TOOL to sign
the image. A typical use case is together with a hardware key. It is
available if SWUPDATE_SIGNING is set to CUSTOM
- **SWUPDATE_PRIVATE_KEY** : this is the file with the private key used to sign the
image using RSA mechanism. Is available if SWUPDATE_SIGNING is set to RSA.
- **SWUPDATE_PASSWORD_FILE** : an optional file containing the password for the private
key. It is available if SWUPDATE_SIGNING is set to RSA.
- **SWUPDATE_CMS_KEY** : this is the file with the private key used in signing
process using CMS mechanism. It is available if SWUPDATE_SIGNING is set to
CMS.
- **SWUPDATE_CMS_CERT** : this is the file with the certificate used in signing
process using using CMS method. It is available if SWUPDATE_SIGNING is
set to CMS.
- **SWUPDATE_AES_FILE** : this is the file with the AES password to encrypt artifact. A new `fstype` is
supported by the class (type: `enc`). SWUPDATE_AES_FILE is generated as output from openssl to create
a new key with
::
openssl enc -aes-256-cbc -k <PASSPHRASE> -P -md sha1 > $SWUPDATE_AES_FILE
To use it, it is enough to add IMAGE_FSTYPES += "enc" to the artifact. SWUpdate supports decryption of
compressed artifact, such as
::
IMAGE_FSTYPES += ".ext4.gz.enc"
Automatic sha256 in sw-description
----------------------------------
The swupdate class takes care of computing and inserting sha256 hashes in the
sw-description file. The attribute *sha256* **must** be set in case the image
is signed. Each artifact must have the attribute:
::
sha256 = "@artifact-file-name"
For example, to add sha256 to the standard Yocto core-image-full-cmdline:
::
sha256 = "@core-image-full-cmdline-machine.ubifs";
The name of the file must be the same as in deploy directory.
BitBake variable expansion in sw-description
--------------------------------------------
To insert the value of a BitBake variable into the update file, pre- and
postfix the variable name with "@@".
For example, to automatically set the version tag:
::
version = "@@DISTRO_VERSION@@";
Automatic versions in sw-description
------------------------------------
By setting the version tag in the update file to `@SWU_AUTO_VERSION` it is
automatically replaced with `PV` from BitBake's package-data-file for the package
matching the name of the provided filename tag.
For example, to set the version tag to `PV` of package `u-boot`:
::
filename = "u-boot";
...
version = "@SWU_AUTO_VERSION";
Since the filename can differ from package name (deployed with another name or
the file is a container for the real package) you can append the correct package
name to the tag: `@SWU_AUTO_VERSION:<package-name>`.
For example, to set the version tag of the file `packed-bootloader` to `PV` of
package `u-boot`:
::
filename = "packed-bootloader";
...
version = "@SWU_AUTO_VERSION:u-boot";
To automatically insert the value of a variable from BitBake's package-data-file
different to `PV` (e.g. `PKGV`) you can append the variable name to the tag:
`@SWU_AUTO_VERSION@<package-data-variable>`.
For example, to set the version tag to `PKGV` of package `u-boot`:
::
filename = "u-boot";
...
version = "@SWU_AUTO_VERSION@PKGV";
Or combined with a different package name:
::
filename = "packed-bootloader";
...
version = "@SWU_AUTO_VERSION:u-boot@PKGV";
Using checksum for version
--------------------------
It is possible to use the hash of an artifact as the version in order to use
"install-if-different". This allows versionless artifacts to be skipped if the
artifact in the update matches the currently installed artifact.
In order to use the hash as the version, the sha256 hash file placeholder
described above in Automatic sha256 in sw-description must be used for version.
Each artifact must have the attribute:
::
version = "@artifact-file-name"
The name of the file must be the same as in deploy directory.
Template for recipe using the class
-----------------------------------
::
DESCRIPTION = "Example recipe generating SWU image"
SECTION = ""
LICENSE = ""
# Add all local files to be added to the SWU
# sw-description must always be in the list.
# You can extend with scripts or wahtever you need
SRC_URI = " \
file://sw-description \
"
# images to build before building swupdate image
IMAGE_DEPENDS = "core-image-full-cmdline virtual/kernel"
# images and files that will be included in the .swu image
SWUPDATE_IMAGES = "core-image-full-cmdline uImage"
# a deployable image can have multiple format, choose one
SWUPDATE_IMAGES_FSTYPES[core-image-full-cmdline] = ".ubifs"
SWUPDATE_IMAGES_FSTYPES[uImage] = ".bin"
inherit swupdate
Simplified version for just image
---------------------------------
In many cases there is a single image in the SWU. This is for example when
just rootfs is updated. The generic case described above required an additional
recipe that must be written and maintained. For this reason, a simplified version
of the class is introduced that allowed to build the SWU from the image recipe.
Users just need to import the `swupdate-image` class. This already sets some variables.
A sw-description must still be added into a `files` directory, that is automatically searched by the class.
User still needs to set SWUPDATE_IMAGE_FSTYPES[`your image`] to the fstype that should be packed
into the SWU - an error is raised if the flag is not set.
In the simple way, your recipe looks like
::
<your original recipe code>
SWUPDATE_IMAGES_FSTYPES[<name of your image>] = <fstype to be put into SWU>
inherit swupdate-image
.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
.. SPDX-License-Identifier: GPL-2.0-only
Contributing to SWUpdate
========================
Contributions are welcome ! Please follow the following guideline for contributions.
Contribution Checklist
----------------------
These are mostly general recommendations and are common practice in a lot of
FOSS projects.
- use git to manage your changes [*recomended*]
- follow as much as possible kernel codestyle [*recomended*]
Nevertheless, some rules are not so strict as in kernel. The maximum line length
can be extended over 80 chars if this increase code readability.
- add the required copyright header to each new file introduced [**required**]
- add signed-off to all patches [**required**]
- to certify the "Developer's Certificate of Origin", see below
- check with your employer when not working on your own!
- add version number for your patches if follow-up versions are requested [*recomended*]
- Add a "Change from Vx" description under the commit message to take track
of the history of the patch.
- It is suggested to use excellent "patman" tool to manage patches series.
This is part of U-Boot's project (tools/patman), but it can be used in other projects, too.
- check that your patches do not break build [**required**]
- There is a set of configuration files in the `configs/` directory.
Please run a build for all files in the directory to ensure that SWUpdate is
still buildable from configurations different as yours.
- post patches to mailing list [**required**]
- use `git format-patch` to generate your patches.
- use `git send-email` if possible. This avoid corruptions due
to the mailers
- add a prefix [meta-swupdate] if patches are intended to the Yocto's meta layer.
- send patches inline, do not append them
- no HTML emails!
- do not use github Pull Request. github facilities are not used for this project.
The review is done in a single place : the Mailing List. PR from github are ignored.
Patches are tracked by patchwork (see http://jk.ozlabs.org/projects/patchwork/).
You can see the status of your patches at http://patchwork.ozlabs.org/project/swupdate/list.
Developer's Certificate of Origin 1.1
-------------------------------------
When signing-off a patch for this project like this
Signed-off-by: Random J Developer <random@developer.example.org>
using your real name (no pseudonyms or anonymous contributions), you declare the
following:
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
..
SPDX-FileCopyrightText: 2021 Stefano Babic <sbabic@denx.de>
SPDX-License-Identifier: GPL-2.0-only
==========================
Delta Update with SWUpdate
==========================
Overview
--------
The size of update packages is steadily increasing. While once the whole software was just
a bunch of megabytes, it is not unusual now that OS and application on devices running
Linux as OS reach huge size of Gigabytes.
Several mechanisms can be used to reduce the size of downloaded data. The resulting images
can be compressed. However, this is not enough when bandwidth is important and not cheap.
It is very common that a device will be upgraded to a version that is similar to the
running one but add new features and solves some bugs. Specially in case of just fixes,
the new version is pretty much equal as the original one. This asks to find methods to
download just the differences with the current software without downloading a full image.
In case an update is performed from a known base, we talk about *delta updates*. In the following
chapter some well known algorithms are considered and verified if they can be integrated
into SWUpdate. The following criteria are important to find a suitable algorithm:
- license must be compatible with GPLv2
- good performance for smaller downloads, but not necessarily the best one.
- SWUpdate remains with the concept to deliver one package (SWU), the same
independently from the source where the SWU is stored (USB, OTA, etc.)
- It must comply to SWUpdate's security requirements (signed images, privilege separation, etc.)
Specific ad-hoc delta updates mechanisms can be realized when the nature of the updated files is
the same. It is always possible with SWUpdate to install single files, but coherency and compatibility
with the runningsoftware must be guaranteed by the integratot / manufacturer. This is not covered here:
the scope is to get an efficient and content unaware *delta* mechanism, that can upgrade in differential
mode two arbitrary images, without any previous knowledge about what they content.
FOSS projects for delta encoding
--------------------------------
There are several algorithms for *delta encoding*, that is to find the difference between files,
generally in binary format. Only algorithms available under a compatible FOSS license (GPLv2)
are considered for SWUpdate.
One of the goals in SWUpdate is that it should work independently which is the format of the
artifacts. Very specialized algorithm and libraries like Google's Courgette used in Chromium will give
much better results, but it works on programs (ELF files) and take advantages of the structure of compiled
code. In case of OTA update, not only software, but any kind of artifact can be delivered, and this includes
configuration data, databases, videos, docs, etc.
librsync_
.........
librsync_ is an independent implementation for rsync and does not use the rsync protocol. It is well
suited to generate offline differential update and it is already integrated into SWUpdate.
However, librsync takes the whole artifact and generates a differential image that is applied
on the whole image. It gives the best results in terms of reduced size when differences are
very small, but the differential output tends to be very large as soon as the differences
are meaningful. Differential images created for SWUpdate show that, as soon as the difference larger is,
the resulting delta image can even become larger as the original one.
SWUpdate supports `librsync` as delta encoder via the rdiff handler.
xdelta_
.......
xdelta_ uses the VCDIFF algorithm to compute differences between binaries. It is often used
to deliver smaller images for CD and DVD. The resulting images are created from an installed
image that should be loaded entirely in main memory. For this reason, it does not scale well
when the images are becoming larger and it is unsuitable for embedded systems and SWUpdate.
casync_
.......
casync_ is, according to his author. a tool for distributing images. It has several interesting
aspects that can be helpful with OTA update.
Files itself are grouped together in chunks and casync creates a "Chunk storage" where each chunk
is stored on a separate file. The chunk storage is part of the delivery, and it must be stored on
a server. casync checks if the chunk is already present on the target, and if not
download it. If this seems to be what is required, there are some drawbacks if casync
should be integrated in SWUpdate:
- because of the nature of casync, each chunk is a separate file. This cause a huge
number of new connections, because each file is a separate GET on the server.
The overhead caused to re-instantiate connection is high on small devices,
where SSL connections are also increasing CPU load. There are downloads of
hundreds or thousands of small files just to recreate the original metadata
file.
- casync has no authentication and verification and the index (.caidx or .caibx)
are not signed. This is known, but casync goals and scopes are outside the ones
on embedded devices.
- it is difficult to deliver a whole chunk storage. The common usage for OTA is to deliver
artifacts, and they should be just a few. Thousands of files to be delivered to let
casync to compute the new image is not practical for companies: they have a new "firmware"
or "software" and they need an easy way to deliver this file (the output from their build system)
to the devices. In some cases, they are even not responsible for that, and the firmware is given to
another authority that groups all packages from vendors and realizes a sort of OTA service.
- casync is quite a huge project - even if it was stated that it will be converted into
a library, this never happened. This makes difficult to interface to SWUpdate,
and using it as external process is a no way in SWUpdate for security reason.
It breaks privilege separation, and adds a lot of code that is difficult
to maintain.
For all these reasons, even if the idea of a chunk storage is good for an OTA updater, casync
is not a candidate for SWUpdate. A out-of-the-box solution cannot be found, and it is required to
implement an own solution that better suits for SWUpdate.
Zchunk_ - compression format
............................
zchunk_ seems to combine the usage of a chunk storage without having to deliver it on a server.
zchunk is a FOSS project released under BSD by its author_. The goal of this project is something else:
zchunk creates a new compression format that adds the ability to download the differences between
new and old file. This matches very well with SWUpdate. A zchunk file contains a header that
has metadata for all chunks, and according to the header, it is known which chunks must be
downloaded and which ones can be reused. zchunk has utilities to download itself the missing chunks,
but it could be just used to find which part of an artifact must be downloading,
and SWUpdate can go on with its own way to do this.
One big advantage on this approach is that metadata and compressed chunks are still bound into a single file,
that can be built by the buildsystem and delivered as it is used to. The updater needs first the metadata, that is
the header in zchunk file, and processes it to detect which chunks need to be downloaded. Each chunk has
its own hash, and the chunks already available on the device are verified against the hash to be sure
they are not corrupted.