Sync with 2021.11

Signed-off-by: Stefano Babic <sbabic@denx.de>

_sources/bindings.rst.txt

+.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
+.. SPDX-License-Identifier: GPL-2.0-only
+
+=============================================
+Language Bindings
+=============================================
+
+Overview
+--------
+
+In general, SWUpdate is agnostic to a particular language it is operated from,
+thanks to :doc:`SWUpdate's socket-based control <swupdate-ipc>` and
+:doc:`progress APIs <progress>` for external programs. As long as the language
+of choice has proper socket (library) support, SWUpdate can be operated with it.
+
+However, for convenience, a Lua language binding in terms of a shared library,
+currently ``lua_swupdate.so.0.1``, is provided.
+
+
+Lua Language Binding
+--------------------
+
+The Lua language binding is realized in terms of the ``swupdate`` module that
+defines three bindings, namely for the control interface, the progress
+interface, and a convenience function yielding a table holding all local
+network interfaces including their IP addresses and submasks.
+
+Note that, depending on the filesystem location of the Lua binding's shared
+library, Lua's ``package.cpath`` may have to be adapted by setting the
+environment variable ``LUA_CPATH``, modifying ``package.cpath`` prior to
+a ``require('lua_swupdate')``, or , as last resort, using ``package.loadlib()``
+instead of ``require('lua_swupdate')``.
+
+
+Control Interface
+.................
+
+The ``swupdate`` module's control interface binding conveniently makes
+:doc:`SWUpdate's socket-based control API <swupdate-ipc>` available to pure Lua.
+
+The binding is captured in the ``swupdate_control`` object that is returned
+by a call to ``swupdate.control()``. This object offers the three methods
+``connect()``, ``write(<chunkdata>)``, and ``close()``:
+
+The ``connect()`` method initializes the connection to SWUpdate's control
+socket, sends ``REQ_INSTALL``, and waits for ``ACK`` or ``NACK``, returning the
+socket connection file descriptor, mostly for information purposes, or, in case
+of an error, ``nil`` plus an error message.
+
+The artifact's data can then be sent to SWUpdate via the ``write(<chunkdata>)``
+method, returning ``true``, or, in case of errors, ``nil`` plus an error message.
+
+Finally, the ``close()`` method closes the connection to SWUpdate's control
+socket after which it waits for SWUpdate to complete the update transaction and
+executes the post-install command, if given.
+
+The following example snippet illustrates how to use the control interface binding:
+
+::
+
+	local artifact = io.open("/some/path/to/artifact.swu", "rb" )
+	swupdate = require('lua_swupdate')
+	local ctrl = swupdate.control()
+	if not ctrl:connect() then
+		-- Deliberately neglecting error message.
+		io.stderr:write("Error connecting to SWUpdate control socket.\n")
+		return
+	end
+
+	while true do
+		local chunk = artifact:read(1024)
+		if not chunk then break end
+		if not ctrl:write(chunk) then
+			-- Deliberately neglecting error message.
+			io.stderr:write("Error writing to SWUpdate control socket.\n")
+			break
+		end
+	end
+
+	local res, msg = ctrl:close()
+	if not res then
+		io.stderr:write(string.format("Error finalizing update: %s\n", msg))
+	end
+
+
+Progress Interface
+..................
+
+The ``swupdate`` module's progress interface binding conveniently makes
+:doc:`SWUpdate's socket-based progress API <progress>` available to pure Lua.
+
+The binding is captured in the ``swupdate_progress`` object that is returned
+by a call to ``swupdate.progress()``. This object offers the three methods
+``connect()``, ``receive()``, and ``close()``:
+
+The ``connect()`` method connects to SWUpdate's progress socket, waiting until
+the connection has been established. Note that it is only really required to
+explicitly call ``connect()`` to reestablish a broken connection as the
+``swupdate_progress`` object's instantiation already initiates the connection.
+
+The ``receive()`` method returns a table representation of the ``struct
+progress_msg`` described in the :doc:`progress interface's API description
+<progress>`.
+
+The ``close()`` method deliberately closes the connection to SWUpdate's progress
+socket.
+
+
+IPv4 Interface
+..............
+
+For convenience, the ``swupdate`` module provides the ``ipv4()`` method
+returning a table holding the local network interfaces as the table's keys and
+their space-separated IP addresses plus subnet masks as respective values.

_sources/building-with-yocto.rst.txt

+.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
+.. SPDX-License-Identifier: GPL-2.0-only
+
+==================================
+meta-swupdate: building with Yocto
+==================================
+
+Overview
+========
+
+The Yocto-Project_ is a community project under the umbrella of the Linux
+Foundation that provides tools and template to create the own custom Linux-based
+software for embedded systems.
+
+.. _Yocto-Project: http://www.yoctoproject.org
+.. _meta-SWUpdate:  https://github.com/sbabic/meta-swupdate.git
+
+Add-on features can be added using *layers*. meta-swupdate_ is the layer to
+cross-compile the SWUpdate application and to generate the compound SWU images
+containing the release of the product.  It contains the required changes
+for mtd-utils and for generating Lua. Using meta-SWUpdate is a
+straightforward process. As described in Yocto's documentation
+about layers, you should include it in your *bblayers.conf* file to use it.
+
+Add meta-SWUpdate as usual to your bblayers.conf. You have also
+to add meta-oe to the list.
+
+In meta-SWUpdate there is a recipe to generate an initrd with a
+rescue system with SWUpdate. Use:
+
+::
+
+	MACHINE=<your machine> bitbake swupdate-image
+
+You will find the result in your tmp/deploy/<your machine> directory.
+How to install and start an initrd is very target specific - please
+check in the documentation of your bootloader.
+
+What about libubootenv ?
+========================
+
+This is a common issue when SWUpdate is built. SWUpdate depends on this library,
+that is generated from the U-Boot's sources. This library allows one to safe modify
+the U-Boot environment. It is not required if U-Boot is not used as bootloader.
+If SWUpdate cannot be linked, you are using an old version of U-Boot (you need
+at least 2016.05). If this is the case, you can add your own recipe for
+the package u-boot-fw-utils, adding the code for the library.
+
+It is important that the package u-boot-fw-utils is built with the same
+sources of the bootloader and for the same machine. In fact, the target
+can have a default environment linked together with U-Boot's code,
+and it is not (yet) stored into a storage. SWUpdate should be aware of
+it, because it cannot read it: the default environment must be linked
+as well to SWUpdate's code. This is done inside the libubootenv.
+
+If you build for a different machine, SWUpdate will destroy the
+environment when it tries to change it the first time. In fact,
+a wrong default environment is taken, and your board won't boot again.
+
+To avoid possible mismatch, a new library was developed to be hardware independent.
+A strict match with the bootloader is not required anymore. The meta-swupdate layer
+contains recipes to build the new library (`libubootenv`) and adjust SWUpdate to be linked
+against it. To use it as replacement for u-boot-fw-utils:
+
+        - set PREFERRED_PROVIDER_u-boot-fw-utils = "libubootenv"
+        - add to SWUpdate config:
+
+::
+
+                CONFIG_UBOOT=y
+                CONFIG_UBOOT_NEWAPI=y
+
+With this library, you can simply pass the default environment as file (u-boot-initial-env).
+It is recommended for new project to switch to the new library to become independent from
+the bootloader.
+
+The swupdate class
+==================
+
+meta-swupdate contains a class specific for SWUpdate. It helps to generate the
+SWU image starting from images built inside the Yocto. It requires that all
+components, that means the artifacts that are part of the SWU image, are present
+in the Yocto's deploy directory.  This class should be inherited by recipes
+generating the SWU. The class defines new variables, all of them have the prefix
+*SWUPDATE_* in the name.
+
+- **SWUPDATE_IMAGES** : this is a list of the artifacts to be packaged together.
+  The list contains the name of images without any extension for MACHINE or
+  filetype, that are added automatically.
+  Example :
+
+::
+
+        SWUPDATE_IMAGES = "core-image-full-cmdline uImage"
+
+- **SWUPDATE_IMAGES_FSTYPES** : extension of the artifact. Each artifact can
+  have multiple extension according to the IMAGE_FSTYPES variable.
+  For example, an image can be generated as tarball and as UBIFS for target.
+  Setting the variable for each artifact tells the class which file must
+  be packed into the SWU image.
+
+
+::
+
+        SWUPDATE_IMAGES_FSTYPES[core-image-full-cmdline] = ".ubifs"
+
+- **SWUPDATE_IMAGES_NOAPPEND_MACHINE** : flag to use drop the machine name from the
+  artifact file. Most images in *deploy* have the name of the Yocto's machine in the
+  filename. The class adds automatically the name of the MACHINE to the file, but some
+  artifacts can be deployed without it.
+
+::
+
+        SWUPDATE_IMAGES_NOAPPEND_MACHINE[my-image] = "1"
+
+- **SWUPDATE_SIGNING** : if set, the SWU is signed. There are 3 allowed values:
+  RSA, CMS, CUSTOM. This value determines used signing mechanism.
+- **SWUPDATE_SIGN_TOOL** : instead of using openssl, use SWUPDATE_SIGN_TOOL to sign
+  the image. A typical use case is together with a hardware key. It is
+  available if SWUPDATE_SIGNING is set to CUSTOM
+- **SWUPDATE_PRIVATE_KEY** : this is the file with the private key used to sign the
+  image using RSA mechanism. Is available if SWUPDATE_SIGNING is set to RSA.
+- **SWUPDATE_PASSWORD_FILE** : an optional file containing the password for the private
+  key. It is available if SWUPDATE_SIGNING is set to RSA.
+- **SWUPDATE_CMS_KEY** : this is the file with the private key used in signing
+  process using CMS mechanism. It is available if SWUPDATE_SIGNING is set to
+  CMS.
+- **SWUPDATE_CMS_CERT** : this is the file with the certificate used in signing
+  process using using CMS method. It is available if SWUPDATE_SIGNING is
+  set to CMS.
+
+- **SWUPDATE_AES_FILE** : this is the file with the AES password to encrypt artifact. A new `fstype` is
+  supported by the class (type: `enc`). SWUPDATE_AES_FILE is generated as output from openssl to create
+  a new key with
+
+  ::
+
+                openssl enc -aes-256-cbc -k <PASSPHRASE> -P -md sha1 > $SWUPDATE_AES_FILE
+
+  To use it, it is enough to add IMAGE_FSTYPES += "enc" to the  artifact. SWUpdate supports decryption of
+  compressed artifact, such as
+
+  ::
+
+        IMAGE_FSTYPES += ".ext4.gz.enc"
+
+
+Automatic sha256 in sw-description
+----------------------------------
+
+The swupdate class takes care of computing and inserting sha256 hashes in the
+sw-description file. The attribute *sha256* **must** be set in case the image
+is signed. Each artifact must have the attribute:
+
+::
+
+        sha256 = "@artifact-file-name"
+
+For example, to add sha256 to the standard Yocto core-image-full-cmdline:
+
+::
+
+        sha256 = "@core-image-full-cmdline-machine.ubifs";
+
+
+The name of the file must be the same as in deploy directory.
+
+BitBake variable expansion in sw-description
+--------------------------------------------
+
+To insert the value of a BitBake variable into the update file, pre- and
+postfix the variable name with "@@".
+For example, to automatically set the version tag:
+
+::
+
+        version = "@@DISTRO_VERSION@@";
+
+Automatic versions in sw-description
+------------------------------------
+
+By setting the version tag in the update file to `@SWU_AUTO_VERSION` it is
+automatically replaced with `PV` from BitBake's package-data-file for the package
+matching the name of the provided filename tag.
+For example, to set the version tag to `PV` of package `u-boot`:
+
+::
+
+        filename = "u-boot";
+        ...
+        version = "@SWU_AUTO_VERSION";
+
+Since the filename can differ from package name (deployed with another name or
+the file is a container for the real package) you can append the correct package
+name to the tag: `@SWU_AUTO_VERSION:<package-name>`.
+For example, to set the version tag of the file `packed-bootloader` to `PV` of
+package `u-boot`:
+
+::
+
+        filename = "packed-bootloader";
+        ...
+        version = "@SWU_AUTO_VERSION:u-boot";
+
+To automatically insert the value of a variable from BitBake's package-data-file
+different to `PV` (e.g. `PKGV`) you can append the variable name to the tag:
+`@SWU_AUTO_VERSION@<package-data-variable>`.
+For example, to set the version tag to `PKGV` of package `u-boot`:
+
+::
+
+        filename = "u-boot";
+        ...
+        version = "@SWU_AUTO_VERSION@PKGV";
+
+Or combined with a different package name:
+
+::
+
+        filename = "packed-bootloader";
+        ...
+        version = "@SWU_AUTO_VERSION:u-boot@PKGV";
+
+Using checksum for version
+--------------------------
+
+It is possible to use the hash of an artifact as the version in order to use
+"install-if-different".  This allows versionless artifacts to be skipped if the
+artifact in the update matches the currently installed artifact.
+
+In order to use the hash as the version, the sha256 hash file placeholder
+described above in Automatic sha256 in sw-description must be used for version.
+
+Each artifact must have the attribute:
+
+::
+
+        version = "@artifact-file-name"
+
+The name of the file must be the same as in deploy directory.
+
+Template for recipe using the class
+-----------------------------------
+
+::
+
+        DESCRIPTION = "Example recipe generating SWU image"
+        SECTION = ""
+
+        LICENSE = ""
+
+        # Add all local files to be added to the SWU
+        # sw-description must always be in the list.
+        # You can extend with scripts or wahtever you need
+        SRC_URI = " \
+            file://sw-description \
+            "
+
+        # images to build before building swupdate image
+        IMAGE_DEPENDS = "core-image-full-cmdline virtual/kernel"
+
+        # images and files that will be included in the .swu image
+        SWUPDATE_IMAGES = "core-image-full-cmdline uImage"
+
+        # a deployable image can have multiple format, choose one
+        SWUPDATE_IMAGES_FSTYPES[core-image-full-cmdline] = ".ubifs"
+        SWUPDATE_IMAGES_FSTYPES[uImage] = ".bin"
+
+        inherit swupdate
+
+Simplified version for just image
+---------------------------------
+
+In many cases there is a single image in the SWU. This is for example when
+just rootfs is updated. The generic case described above required an additional
+recipe that must be written and maintained. For this reason, a simplified version
+of the class is introduced that allowed to build the SWU from the image recipe.
+
+Users just need to import the `swupdate-image` class. This already sets some variables.
+A sw-description must still be added into a `files` directory, that is automatically searched by the class.
+User still needs to set SWUPDATE_IMAGE_FSTYPES[`your image`] to the fstype that should be packed
+into the SWU - an error is raised if the flag is not set.
+
+In the simple way, your recipe looks like
+
+::
+        <your original recipe code>
+
+        SWUPDATE_IMAGES_FSTYPES[<name of your image>] = <fstype to be put into SWU>
+        inherit swupdate-image

_sources/contributing.rst.txt

+.. SPDX-FileCopyrightText: 2013-2021 Stefano Babic <sbabic@denx.de>
+.. SPDX-License-Identifier: GPL-2.0-only
+
+Contributing to SWUpdate
+========================
+
+Contributions are welcome ! Please follow the following guideline for contributions.
+
+Contribution Checklist
+----------------------
+
+These are mostly general recommendations and are common practice in a lot of
+FOSS projects.
+
+- use git to manage your changes [*recomended*]
+
+- follow as much as possible kernel codestyle [*recomended*]
+  Nevertheless, some rules are not so strict as in kernel. The maximum line length
+  can be extended over 80 chars if this increase code readability.
+
+- add the required copyright header to each new file introduced [**required**]
+
+- add signed-off to all patches [**required**]
+    - to certify the "Developer's Certificate of Origin", see below
+    - check with your employer when not working on your own!
+
+- add version number for your patches if follow-up versions are requested [*recomended*]
+    - Add a "Change from Vx" description under the commit message to take track
+      of the history of the patch.
+    - It is suggested to use excellent "patman" tool to manage patches series.
+      This is part of U-Boot's project (tools/patman), but it can be used in other projects, too.
+
+- check that your patches do not break build [**required**]
+
+  - There is a set of configuration files in the `configs/` directory.
+    Please run a build for all files in the directory to ensure that SWUpdate is
+    still buildable from configurations different as yours.
+
+- post patches to mailing list [**required**]
+    - use `git format-patch` to generate your patches.
+    - use `git send-email` if possible. This avoid corruptions due
+      to the mailers
+    - add a prefix [meta-swupdate] if patches are intended to the Yocto's meta layer.
+    - send patches inline, do not append them
+    - no HTML emails!
+
+- do not use github Pull Request. github facilities are not used for this project.
+  The review is done in a single place : the Mailing List. PR from github are ignored.
+
+Patches are tracked by patchwork (see http://jk.ozlabs.org/projects/patchwork/).
+You can see the status of your patches at http://patchwork.ozlabs.org/project/swupdate/list.
+
+Developer's Certificate of Origin 1.1
+-------------------------------------
+
+When signing-off a patch for this project like this
+
+    Signed-off-by: Random J Developer <random@developer.example.org>
+
+using your real name (no pseudonyms or anonymous contributions), you declare the
+following:
+
+    By making a contribution to this project, I certify that:
+
+        (a) The contribution was created in whole or in part by me and I
+            have the right to submit it under the open source license
+            indicated in the file; or
+
+        (b) The contribution is based upon previous work that, to the best
+            of my knowledge, is covered under an appropriate open source
+            license and I have the right under that license to submit that
+            work with modifications, whether created in whole or in part
+            by me, under the same open source license (unless I am
+            permitted to submit under a different license), as indicated
+            in the file; or
+
+        (c) The contribution was provided directly to me by some other
+            person who certified (a), (b) or (c) and I have not modified
+            it.
+
+        (d) I understand and agree that this project and the contribution
+            are public and that a record of the contribution (including all
+            personal information I submit with it, including my sign-off) is
+            maintained indefinitely and may be redistributed consistent with
+            this project or the open source license(s) involved.
+

_sources/delta-update.rst.txt

+..
+        SPDX-FileCopyrightText: 2021 Stefano Babic <sbabic@denx.de>
+        SPDX-License-Identifier: GPL-2.0-only
+
+==========================
+Delta Update with SWUpdate
+==========================
+
+Overview
+--------
+
+The size of update packages is steadily increasing. While once the whole software was just
+a bunch of megabytes, it is not unusual now that OS and application on devices running
+Linux as OS reach huge size of Gigabytes.
+
+Several mechanisms can be used to reduce the size of downloaded data. The resulting images
+can be compressed. However, this is not enough when bandwidth is important and not cheap.
+It is very common that a device will be upgraded to a version that is similar to the
+running one but add new features and solves some bugs. Specially in case of just fixes,
+the new version is pretty much equal as the original one. This asks to find methods to
+download just the differences with the current software without downloading a full image.
+In case an update is performed from a known base, we talk about *delta updates*. In the following
+chapter some well known algorithms are considered and verified if they can be integrated
+into SWUpdate. The following criteria are important to find a suitable algorithm:
+
+        - license must be compatible with GPLv2
+        - good performance for smaller downloads, but not necessarily the best one.
+        - SWUpdate remains with the concept to deliver one package (SWU), the same
+          independently from the source where the SWU is stored (USB, OTA, etc.)
+        - It must comply to SWUpdate's security requirements (signed images, privilege separation, etc.)
+
+Specific ad-hoc delta updates mechanisms can be realized when the nature of the updated files is
+the same. It is always possible with SWUpdate to install single files, but coherency and compatibility
+with the runningsoftware must be guaranteed by the integratot / manufacturer. This is not covered here:
+the scope is to get an efficient and content unaware *delta* mechanism, that can upgrade in differential
+mode two arbitrary images, without any previous knowledge about what they content.
+
+FOSS projects for delta encoding
+--------------------------------
+
+There are several algorithms for *delta encoding*, that is to find the difference between files,
+generally in binary format. Only algorithms available under a compatible FOSS license (GPLv2) 
+are considered for SWUpdate.
+One of the goals in SWUpdate is that it should work independently which is the format of the
+artifacts. Very specialized algorithm and libraries like Google's Courgette used in Chromium will give
+much better results, but it works on programs (ELF files) and take advantages of the structure of compiled
+code. In case of OTA update, not only software, but any kind of artifact can be delivered, and this includes
+configuration data, databases, videos, docs, etc.
+
+librsync_
+.........
+
+librsync_ is an independent implementation for rsync and does not use the rsync protocol. It is well
+suited to generate offline differential update and it is already integrated into SWUpdate.
+However, librsync takes the whole artifact and generates a differential image that is applied
+on the whole image. It gives the best results in terms of reduced size when differences are
+very small, but the differential output tends to be very large as soon as the differences
+are meaningful. Differential images created for SWUpdate show that, as soon as the difference larger is,
+the resulting delta image can even become larger as the original one.
+
+SWUpdate supports `librsync` as delta encoder via the rdiff handler.
+
+xdelta_
+.......
+
+xdelta_ uses the VCDIFF algorithm to compute differences between binaries. It is often used
+to deliver smaller images for CD and DVD. The resulting images are created from an installed
+image that should be loaded entirely in main memory. For this reason, it does not scale well
+when the images are becoming larger and it is unsuitable for embedded systems and SWUpdate.
+
+casync_
+.......
+
+casync_ is, according to his author. a tool for distributing images. It has several interesting
+aspects that can be helpful with OTA update.
+Files itself are grouped together in chunks and casync creates a "Chunk storage" where each chunk
+is stored on a separate file. The chunk storage is part of the delivery, and it must be stored on
+a server. casync checks if the chunk is already present on the target, and if not
+download it. If this seems to be what is required, there are some drawbacks if casync
+should be integrated in SWUpdate:
+
+        - because of the nature of casync, each chunk is a separate file. This cause a huge
+          number of new connections, because each file is a separate GET on the server.
+          The overhead caused to re-instantiate connection is high on small devices,
+          where SSL connections are also increasing CPU load. There are downloads of
+          hundreds or thousands of small files just to recreate the original metadata
+          file.
+        - casync has no authentication and verification and the index (.caidx or .caibx)
+          are not signed. This is known, but casync goals and scopes are outside the ones
+          on embedded devices.
+        - it is difficult to deliver a whole chunk storage. The common usage for OTA is to deliver
+          artifacts, and they should be just a few. Thousands of files to be delivered to let
+          casync to compute the new image is not practical for companies: they have a new "firmware"
+          or "software" and they need an easy way to deliver this file (the output from their build system)
+          to the devices. In some cases, they are even not responsible for that, and the firmware is given to
+          another authority that groups all packages from vendors and realizes a sort of OTA service.
+        - casync is quite a huge project - even if it was stated that it will be converted into
+          a library, this never happened. This makes difficult to interface to SWUpdate,
+          and using it as external process is a no way in SWUpdate for security reason.
+          It breaks privilege separation, and adds a lot of code that is difficult
+          to maintain.
+
+For all these reasons, even if the idea of a chunk storage is good for an OTA updater, casync
+is not a candidate for SWUpdate. A out-of-the-box solution cannot be found, and it is required to
+implement an own solution that better suits for SWUpdate.
+
+Zchunk_ - compression format
+............................
+
+zchunk_ seems to combine the usage of a chunk storage without having to deliver it on a server.
+zchunk is a FOSS project released under BSD by its author_. The goal of this project is something else:
+zchunk creates a new compression format that adds the ability to download the differences between
+new and old file. This matches very well with SWUpdate. A zchunk file contains a header that
+has metadata for all chunks, and according to the header, it is known which chunks must be
+downloaded and which ones can be reused. zchunk has utilities to download itself the missing chunks,
+but it could be just used to find which part of an artifact must be downloading,
+and SWUpdate can go on with its own way to do this.
+
+One big advantage on this approach is that metadata and compressed chunks are still bound into a single file,
+that can be built by the buildsystem and delivered as it is used to. The updater needs first the metadata, that is
+the header in zchunk file, and processes it to detect which chunks need to be downloaded. Each chunk has
+its own hash, and the chunks already available on the device are verified against the hash to be sure
+they are not corrupted.
+