1. 07 Mar, 2021 40 commits
    • Gilles Chanteperdrix's avatar
      gpio: pl061: ipipe: enable interrupt pipelining · 853d99e4
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the pl061 GPIO driver in order to channel interrupts through
      the interrupt pipeline.
      853d99e4
    • Gilles Chanteperdrix's avatar
      gpio: mxc: ipipe: enable interrupt pipelining · 91ea9550
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the MXC GPIO driver (i.MX series) in order to channel
      interrupts through the interrupt pipeline.
      91ea9550
    • Gilles Chanteperdrix's avatar
      gpio: mvebu: ipipe: enable interrupt pipelining · d5f63baf
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the Marvell EBU GPIO controller driver in order to channel
      interrupts through the interrupt pipeline.
      d5f63baf
    • Gilles Chanteperdrix's avatar
      gpio: davinci: ipipe: enable interrupt pipelining · 0ce4bbbc
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the Davinci GPIO driver in order to channel interrupts through
      the interrupt pipeline.
      0ce4bbbc
    • Gilles Chanteperdrix's avatar
      gpio: omap: ipipe: enable interrupt pipelining · c01d12bd
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the omap GPIO driver in order to channel interrupts through the
      interrupt pipeline.
      c01d12bd
    • Dongjiu Geng's avatar
      clocksource: arm_arch_timer: ipipe: correct the wrong ipipe_timer setup · a1918140
      Dongjiu Geng authored and Jan Kiszka's avatar Jan Kiszka committed
      
      
      The arch_timer_mem_use_virtual variable is used to judge the type
      of memory-mapped timer, it can not be used to judge the type of CP15
      timer, so fix this issue, otherwise it will lead to kernel panic when
      running as a VM.
      Signed-off-by: default avatarDongjiu Geng <gengdongjiu@huawei.com>
      a1918140
    • Gilles Chanteperdrix's avatar
      clocksource: arch_timer: ipipe: enable interrupt pipelining · f9d1e1e5
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Fix up the ARM architected timer driver in order to channel interrupts
      through the interrupt pipeline.
      
      FIXME: split.
      f9d1e1e5
    • Greg Gallagher's avatar
      clocksource: ipipe: add support for BCM2835 chips · 6777f391
      Greg Gallagher authored and Jan Kiszka's avatar Jan Kiszka committed
      This change will add support for bcm2835 chips.  This includes the raspberry pi
      0 and 1 family of boards.
      
      FIXME: split.
      6777f391
    • Gilles Chanteperdrix's avatar
      clocksource: dw_apb: ipipe: add support for user-visible TSC · ecc4456b
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface if enabled.
      ecc4456b
    • Gilles Chanteperdrix's avatar
      clocksource: arm/gt: ipipe: add support for user-visible TSC · edaa464c
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
      edaa464c
    • Gilles Chanteperdrix's avatar
      clocksource: imx/gpt: ipipe: add support for user-visible TSC · d6086420
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
      d6086420
    • Gilles Chanteperdrix's avatar
      clocksource: sp804: ipipe: add support for user-visible TSC · 9aeb6354
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
      9aeb6354
    • Philippe Gerum's avatar
      ipipe: add cpuidle control interface · 5cb4c552
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Add a kernel interface for sharing CPU idling control between the host
      kernel and a co-kernel. The former invokes ipipe_cpuidle_control()
      which the latter should implement, for determining whether entering a
      sleep state is ok. This hook should return boolean true if so.
      
      The co-kernel may veto such entry if need be, in order to prevent
      latency spikes, as exiting sleep states might be costly depending on
      the CPU idling operation being used.
      5cb4c552
    • Gilles Chanteperdrix's avatar
      pinctrl: ipipe: enable interrupt pipelining · 7b17e0db
      Gilles Chanteperdrix authored and Jan Kiszka's avatar Jan Kiszka committed
      Do the fixups in the generic pin controller driver required to channel
      the incoming interrupts through the interrupt pipeline.
      7b17e0db
    • Philippe Gerum's avatar
      gpio: ipipe: convert to hard locking · fc5da725
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      The generic GPIO chip needs to be hard locked as its handlers may be
      called from out-of-band code running in the head domain. To this end,
      convert the regular spinlock protecting the gpio_chip descriptor to a
      hard lock.
      fc5da725
    • Philippe Gerum's avatar
      ftrace: ipipe: enable tracing from the head domain · f3de3200
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Enabling ftrace for a co-kernel running in the head domain of a
      pipelined interrupt context means to:
      
      - make sure that ftrace's live kernel code patching still runs
        unpreempted by any head domain activity (so that the latter can't
        tread on invalid or half-baked changes in the .text section).
      
      - allow the co-kernel code running in the head domain to traverse
        ftrace's tracepoints safely.
      
      The changes introduced by this commit ensure this by fixing up some
      key critical sections so that interrupts are still disabled in the
      CPU, undoing the interrupt flag virtualization in those particular
      cases.
      f3de3200
    • Philippe Gerum's avatar
      mm: ipipe: disable ondemand memory · c2baf136
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Co-kernels cannot bear with the extra latency caused by memory access
      faults involved in COW or overcommit. __ipipe_disable_ondemand_mappings()
      force commits all common memory mappings with physical RAM.
      
      In addition, the architecture code is given a chance to pre-load page
      table entries for ioremap and vmalloc memory, for preventing further
      minor faults accessing such memory due to PTE misses (if that ever
      makes sense for them).
      c2baf136
    • Philippe Gerum's avatar
      fork: ipipe: announce mm dismantling · 923b881f
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      IPIPE_KEVT_CLEANUP is emitted before a process memory context is
      entirely dropped, after all the mappings have been exited. Per-process
      resources which might be maintained by the co-kernel could be released
      there, as all tasks have exited.
      923b881f
    • Philippe Gerum's avatar
      sched: ipipe: announce CPU affinity change · 873e7f18
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Emit IPIPE_KEVT_SETAFFINITY to the co-kernel when the target task is
      about to move to another CPU.
      
      CPU migration can only take place from the root domain, the pipeline
      does not provide any support for migrating tasks from the head domain,
      and derives several key assumptions based on this invariant.
      873e7f18
    • Philippe Gerum's avatar
      sched: ipipe: announce signal receipt · 2296ab89
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Emit IPIPE_KEVT_SIGWAKE when the target task is about to receive a
      (regular) signal. The co-kernel may decide to schedule a transition of
      the recipient to the root domain in order to have it handle that
      signal asap, which is commonly required for keeping the kernel sane.
      
      This notification is always sent from the context of the issuer.
      2296ab89
    • Philippe Gerum's avatar
      sched: ipipe: announce task exit · 323ff853
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Emit IPIPE_KEVT_EXIT from do_exit() to the co-kernel before the
      current task has dropped the files and mappings it owns.
      323ff853
    • Philippe Gerum's avatar
      KVM: ipipe: keep hypervisor state consistent across domain preemption · 7b099da6
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      In order for the hypervisor to operate properly in presence of a
      co-kernel, we need:
      
      - the virtualization core to know when the hypervisor stalls due
        to a preemption by the co-kernel.
      
      - to know when the VM enters and leaves guest mode.
      7b099da6
    • Philippe Gerum's avatar
      sched: ipipe: add domain debug checks to common scheduling paths · fa5a1f30
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Catch invalid calls of root-only code from the head domain from common
      paths which may lead to blocking the current task linux-wise. Checks
      are enabled by CONFIG_IPIPE_DEBUG_CONTEXT.
      fa5a1f30
    • Philippe Gerum's avatar
      sched: ipipe: enable task migration between domains · db8d4efb
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      This is the basic code enabling alternate control of tasks between the
      regular kernel and an embedded co-kernel. The changes cover the
      following aspects:
      
      - extend the per-thread information block with a private area usable
        by the co-kernel for storing additional state information
      
      - provide the API enabling a scheduler exchange mechanism, so that
        tasks can run under the control of either kernel alternatively. This
        includes a service to move the current task to the head domain under
        the control of the co-kernel, and the converse service to re-enter
        the root domain once the co-kernel has released such task.
      
      - ensure the generic context switching code can be used from any
        domain, serializing execution as required.
      
      These changes have to be paired with arch-specific code further
      enabling context switching from the head domain.
      db8d4efb
    • Philippe Gerum's avatar
      clockevents: ipipe: connect clock chips to abstract tick device · bf6640d8
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Announce all clock event chips as they are registered to the
      out-of-band tick device infrastructure, so that we can interpose on
      key handlers in their descriptors.
      bf6640d8
    • Philippe Gerum's avatar
      ipipe: add kernel event notifiers · 48858891
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Add the core API for enabling (regular) kernel event notifications to
      a co-kernel running over the head domain. For instance, such a
      co-kernel may need to know when a task is about to be resumed upon
      signal receipt, or when it gets an access fault trap.
      
      This commit adds the client-side API for enabling such notification
      for class of events, but does not provide the notification points per
      se, which comes later.
      48858891
    • Philippe Gerum's avatar
      printk: ipipe: add raw console channel · b61bde48
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      A raw output handler (.write_raw) is added to the console descriptor
      for writing (short) text output unmodified, without any logging,
      header or preparation whatsoever, usable from any pipeline domain.
      
      The dedicated raw_printk() variant formats the output message then
      passes it on to the handler holding a hard spinlock, irqs off.
      
      This is a very basic debug channel for situations when resorting to
      the fairly complex printk() handling is not an option. Unlike early
      consoles, regular consoles can provide a raw output service past the
      boot sequence. Raw output handlers are typically provided by serial
      console devices.
      b61bde48
    • Philippe Gerum's avatar
      dump_stack: ipipe: make dump_stack() domain-aware · 7a0e3ce8
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      When dumping a stack backtrace, we neither need nor want to disable
      root stage IRQs over the head stage, where CPU migration can't
      happen.
      
      Conversely, we neither need nor want to disable hard IRQs from the
      head stage, so that latency won't skyrocket either.
      7a0e3ce8
    • Philippe Gerum's avatar
      driver core: ipipe: defer dev_printk() from head domain · 374cff39
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Just like printk(), dev_printk() cannot run from the head domain
      and/or with hard IRQs disabled. In such a case, log the output
      directly into the staging buffer we use for printk().
      
      NOTE: when redirected to the buffer, the output does not include the
      dev_printk() header text but is merely sent as-is to the log.
      374cff39
    • Philippe Gerum's avatar
      printk: ipipe: defer printk() from head domain · 799c0d98
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      The printk() machinery cannot immediately invoke the console driver(s)
      when called from the head domain, since such driver code belongs to
      the root domain and cannot be shared between domains.
      
      Output issued from the head domain is formatted then logged into a
      staging buffer, and a dedicated virtual IRQ is posted to the root
      domain for notification. When the virtual IRQ handler runs, the
      contents of the staging buffer is flushed to the printk() interface
      anew, which may eventually pass the output on to the console drivers
      from such a context.
      799c0d98
    • Philippe Gerum's avatar
      PM / hibernate: ipipe: protect against out-of-band interrupts · d1b692b1
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      We must not allow out-of-band activity to resume while we are busy
      suspending the devices in the system, until the PM sleep state has
      been fully entered.
      
      Pair existing virtual IRQ disabling calls which only apply to the root
      domain with hard ones.
      d1b692b1
    • Philippe Gerum's avatar
      module: ipipe: enable try_module_get() from hard atomic context · 60fcca2f
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      We might have out-of-band code calling try_module_get() from the head
      domain, or from a section covered by a hard spinlock where the root
      domain must not reschedule. This requires the preemption management
      calls in try_module_get() (and the converse module_put()) to be
      converted to their hard variant.
      
      REVISIT: using try_module_get() from such contexts is questionable,
      client domains should be fixed.
      60fcca2f
    • Philippe Gerum's avatar
      KGDB: ipipe: enable debugging over the head domain · 6ede0ff0
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Make the KGDB stub runnable over the head domain since we may take
      traps and interrupts from that context too, by converting the locks to
      hard spinlocks.
      6ede0ff0
    • Philippe Gerum's avatar
      context_tracking: ipipe: do not track over the head domain · 3f0532dd
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Context tracking is a prerequisite for FULL_NOHZ, so that the RCU
      subsystem can detect CPU idleness without relying on the (regular)
      timer tick.
      
      Out-of-band activity running over the head domain should by definition
      not be involved in such detection logic, as the root domain has no
      knowledge of what happens - and when - on the head domain whatsoever.
      3f0532dd
    • Philippe Gerum's avatar
      lib/smp_processor_id: ipipe: exclude head domain from preemption check · cf2f0f40
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      There can be no CPU migration from the head stage, however the
      out-of-band code currently running smp_processor_id() might have
      preempted the regular kernel code from within a preemptible section,
      which might cause false positive in the end.
      
      These are the two reasons why we certainly neither need nor want to do
      the preemption check in that case.
      cf2f0f40
    • Philippe Gerum's avatar
      atomic: ipipe: keep atomic when pipelining IRQs · b61ed2ed
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Because of the virtualization of interrupt masking for the regular
      kernel code when the pipeline is enabled, atomic helpers relying on
      common interrupt disabling helpers such as local_irq_save/restore
      pairs would not be atomic anymore, leading to data corruption.
      
      This commit restores true atomicity for the atomic helpers that would
      be otherwise affected by interrupt virtualization.
      b61ed2ed
    • Philippe Gerum's avatar
      preempt: ipipe: : add preemption-safe hard_preempt_{enable, disable}() ops · 15c424da
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      Some inner code of the interrupt pipeline may have to traverse regular
      kernel code which manipulates the preemption count, expecting full
      serialization including with out-of-band contexts.
      
      The hard_preempt_*() variants are substituted to the original
      preempt_{enable, disable}() calls in these cases.
      15c424da
    • Philippe Gerum's avatar
      genirq: ipipe: protect generic chip against domain preemption · aadb1279
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      As described in Documentation/ipipe.rst, irq_chip drivers need to be
      specifically adapted for dealing with interrupt pipelining safely.
      
      The basic issue to address is proper serialization between some
      irq_chip handlers which may be called from out-of-band context
      immediately upon receipt of an IRQ, and the rest of the driver which
      may access the same data / IO registers from the regular - in-band -
      context on the same CPU.
      
      This commit converts the generic irq_chip lock to a hard spinlock,
      which ensures such serialization.
      aadb1279
    • Philippe Gerum's avatar
      ipipe: add latency tracer · 9fb9e230
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      The latency tracer is a variant of ftrace's 'function' tracer
      providing detailed information about the current interrupt state at
      each function entry (i.e. virtual interrupt flag and CPU interrupt
      disable bit). This commit introduces the generic tracer code, which
      builds upon the regular ftrace API.
      
      The arch-specific code should provide for ipipe_read_tsc(), a helper
      routine returning a 64bit monotonic time value for timestamping
      purpose. HAVE_IPIPE_TRACER_SUPPORT should be selected by the
      arch-specific code for enabling the tracer, which in turn makes
      CONFIG_IPIPE_TRACE available from the Kconfig interface.
      9fb9e230
    • Philippe Gerum's avatar
      ipipe: add out-of-band tick device · e0cecb8d
      Philippe Gerum authored and Jan Kiszka's avatar Jan Kiszka committed
      
      
      The out-of-band tick device manages the timer hardware by interposing
      on selected clockevent handlers transparently, so that a client domain
      (e.g. a co-kernel) eventually controls such hardware for scheduling
      the high-precision timer events it needs to. Those events are
      delivered to out-of-hand activities running on the head stage,
      unimpeded by (only virtually) interrupt-free sections of the regular
      kernel code.
      
      This commit introduces the generic API for controlling the out-of-band
      tick device from a co-kernel. It also provides for the internal API
      clock event chip drivers should use for enabling high-precision
      timing for their hardware.
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      e0cecb8d