1. 18 Jun, 2021 1 commit
  2. 15 Jun, 2021 22 commits
    • Philippe Gerum's avatar
      ARM: irq_pipeline: preserve lockdep irq state across kernel entries · 4c475643
      Philippe Gerum authored
      
      
      When lockdep is enabled, per_cpu(hardirqs_enabled) may not be in sync
      with the in-band stall bit on kernel entry upon preemption by an
      IRQ. This may happen since the relevant local_irq_* calls do not
      manipulate the stall bit and the lockdep irq state atomically (for a
      good reason). In addition, the raw_local_irq API may be used directly,
      without lockdep tracking whatsoever (e.g. when manipulating raw
      spinlocks).
      
      As a result, the kernel may observe a stalled in-band stage, with
      per_cpu(hardirqs_enabled) not mirroring the interrupt state, e.g.:
      
      /* in-band, irqs_disabled=0, percpu(hardirqs_enabled)=1 */
      raw_local_irq_irqsave /* e.g. raw_spin_lock */
      /* irqs_disabled=1, percpu(hardirqs_enabled)=1 */
      	<IRQ>
      	trace_hardirqs_off_pipelined /* on entry */
      		trace_hardirqs_off
      		/* irqs_disabled=1, percpu(hardirqs_enabled)=0 */
      		....
      	trace_hardirqs_on_pipelined /* irqs_disabled on exit -> skips trace_hardirqs_on */
      	</IRQ> /* percpu(hardirqs_enabled) still 0 */
      raw_local_irq_restore
      WARN_ON(lockdep_assert_irqs_enabled()); /* TRIGGERS! */
      
      kentry_enter_pipelined and kentry_exit_pipelined are introduced to
      preserve the full irq state for the in-band stage across a kernel
      entry (IRQ and fault), which is comprised of the stall bit and the
      lockdep irq state (per_cpu(hardirqs_enabled)) now tracked
      independently.
      
      These helpers are normally called from the kernel entry/exit code in
      the asm section by architectures which do not use the generic kernel
      entry code, in order to save the interrupt and lockdep states for the
      in-band stage on entry, restoring them when leaving the kernel.
      
      At this chance, the pipelined fault entry/exit routines are simplified
      by relying on these helpers for preserving the virtual interrupt state
      across the fault handling code.
      
      This fixes random kernel splats with CONFIG_PROVE_LOCKING enabled such
      as:
      
      [   25.735750] WARNING: CPU: 0 PID: 65 at kernel/softirq.c:175 __local_bh_enable_ip+0x1e4/0x264
      [   25.747380] Modules linked in:
      [   25.750529] CPU: 0 PID: 65 Comm: kworker/u3:1 Not tainted 5.10.42-00593-g5753d0a33341-dirty #5
      [   25.759307] Hardware name: Generic AM33XX (Flattened Device Tree)
      [   25.765463] IRQ stage: Linux
      [   25.768473] Workqueue: xprtiod xs_stream_data_receive_workfn
      [   25.774237] [<c030fe14>] (unwind_backtrace) from [<c030c3f8>] (show_stack+0x10/0x14)
      [   25.782129] [<c030c3f8>] (show_stack) from [<c033cf30>] (__warn+0x118/0x11c)
      [   25.789317] [<c033cf30>] (__warn) from [<c033cfe4>] (warn_slowpath_fmt+0xb0/0xb8)
      [   25.796944] [<c033cfe4>] (warn_slowpath_fmt) from [<c0342cc0>] (__local_bh_enable_ip+0x1e4/0x264)
      [   25.805904] [<c0342cc0>] (__local_bh_enable_ip) from [<c0f778b0>] (tcp_recvmsg+0x31c/0xa54)
      [   25.814402] [<c0f778b0>] (tcp_recvmsg) from [<c0fb17a8>] (inet_recvmsg+0x48/0x70)
      [   25.822024] [<c0fb17a8>] (inet_recvmsg) from [<c1072b90>] (xs_sock_recvmsg.constprop.9+0x24/0x40)
      [   25.831042] [<c1072b90>] (xs_sock_recvmsg.constprop.9) from [<c1073e34>] (xs_stream_data_receive_workfn+0xe0/0x630)
      [   25.841652] [<c1073e34>] (xs_stream_data_receive_workfn) from [<c035b008>] (process_one_work+0x2f8/0x7b4)
      [   25.851367] [<c035b008>] (process_one_work) from [<c035b508>] (worker_thread+0x44/0x594)
      [   25.859605] [<c035b508>] (worker_thread) from [<c0361c6c>] (kthread+0x16c/0x184)
      [   25.867142] [<c0361c6c>] (kthread) from [<c0300184>] (ret_from_fork+0x14/0x30)
      [   25.874431] Exception stack(0xc406ffb0 to 0xc406fff8)
      [   25.879610] ffa0:                                     00000000 00000000 00000000 00000000
      [   25.887946] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
      [   25.896197] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
      [   25.902936] irq event stamp: 81142
      [   25.906460] hardirqs last  enabled at (81152): [<c0394420>] console_unlock+0x374/0x5cc
      [   25.914451] hardirqs last disabled at (81159): [<c03943f0>] console_unlock+0x344/0x5cc
      [   25.922516] softirqs last  enabled at (80912): [<c0ec18a4>] lock_sock_nested+0x30/0x84
      [   25.930572] softirqs last disabled at (80915): [<c0ec494c>] release_sock+0x18/0x98
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      4c475643
    • Philippe Gerum's avatar
      lockdep: irq_pipeline: fix ambiguous naming · 14d1ae2f
      Philippe Gerum authored
      lockdep_save/restore_irqs_state might convey the wrong information:
      this is not about saving+disabling then conditionally re-enabling the
      tracked state, but merely to read/write such state
      unconditionally. Let's change this to non-equivocal names.
      14d1ae2f
    • Philippe Gerum's avatar
      genirq: irq_pipeline: synchronize log on irq exit to kernel · 8e9eadfe
      Philippe Gerum authored
      
      
      We must make sure to play any IRQ which might be pending in the
      in-band log before leaving an interrupt frame for a preempted kernel
      context.
      
      This completes "irq_pipeline: Account for stage migration across
      faults", so that we synchronize the log once the in-band stage is
      unstalled. In addition, we also care to do this before
      preempt_schedule_irq() runs, so that we won't miss any rescheduling
      request which might have been triggered by some IRQ we just played.
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      Suggested-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      8e9eadfe
    • Philippe Gerum's avatar
      cd093a83
    • Philippe Gerum's avatar
      irq_pipeline: genirq: Mark inband stalled also on exceptions · d0c1c621
      Philippe Gerum authored
      
      
      Make sure that inband is marked stalled when entering from user mode,
      taking an exception.
      
      This affects x86 which is currently the only arch using generic
      irqentry_enter_from_user_mode on exceptions. It fixes this lockdep
      warning:
      
      DEBUG_LOCKS_WARN_ON(!lockdep_stage_disabled())
      WARNING: CPU: 2 PID: 1477 at ../kernel/locking/lockdep.c:4129 lockdep_hardirqs_on_prepare+0x160/0x1a0
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      Tested-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      d0c1c621
    • Philippe Gerum's avatar
      irq_pipeline: genirq: fix order of hard+virtual irq disable sequence · 19ca214a
      Philippe Gerum authored
      The way local_irq_disable_full() works may cause interrupt events to
      lag in the interrupt log inadvertently if the code path does not
      synchronize such log afterwards. As a result, some interrupts may not
      get played when they should, causing breakage.
      
      Since calling inband_irq_disable() with hard irqs off is deemed ok
      (unlike with inband_irq_enable()), invert the two operations so that
      hard irqs are disabled before the in-band stage is stalled, preventing
      any interrupt to be logged in between.
      
      See https://xenomai.org/pipermail/xenomai/2021-June/045476.html.
      
      This fixes this issue:
      https://xenomai.org/pipermail/xenomai/2021-May/045071.html
      
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      Reported-by: default avatarFlorian Bezdeka <florian.bezdeka@siemens.com>
      19ca214a
    • Jan Kiszka's avatar
      x86: dovetail: Fix TS flag reservation · 8994faca
      Jan Kiszka authored
      
      
      We had an overlap with compat flags so that, e.g., TS_COMPAT_RESTART
      made a 32-bit standard task also a dovetail one.
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      8994faca
    • Jan Kiszka's avatar
      irq_pipeline: Warn when calling irqentry_enter with oob stalled · 9dd32928
      Jan Kiszka authored
      
      
      Something must have went wrong if entering for an IRQ or an exception
      over oob and with this stage stalled. Warn when debugging
      Suggest-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      9dd32928
    • Jan Kiszka's avatar
      irq_pipeline: Account for stage migration across faults · ce4b7c6e
      Jan Kiszka authored
      
      
      We need to unstall the inband stage when we entered for a fault over OOB
      and then migrated to inband. So far we kept the inband stage stalled,
      causing a state corruption this way.
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      ce4b7c6e
    • Jan Kiszka's avatar
      irq_pipeline: Clean up stage_info field and users · bf1038cf
      Jan Kiszka authored
      
      
      This field represents mutually exclusive states, namely
      
      - IRQENTRY_INBAND_UNSTALLED
      - IRQENTRY_INBAND_STALLED
      - IRQENTRY_OOB
      
      Encodes them as enum and test against them, rather than against state
      bits that suggest they could be combined.
      
      Also flip the inverted naming of INBAND_STALLED vs. INBAND_UNSTALLED:
      Only when we entered under INBAND_UNSTALLED, certain actions need to be
      taken on exit.
      
      Finally, document the stage_info field of irqentry_state.
      Signed-off-by: Jan Kiszka's avatarJan Kiszka <jan.kiszka@siemens.com>
      bf1038cf
    • Philippe Gerum's avatar
      281d47f0
    • Philippe Gerum's avatar
      evl/trace: fix thread mode printer · 65c0db97
      Philippe Gerum authored
      
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      65c0db97
    • Philippe Gerum's avatar
      evl/trace: do not trace trace helpers · 189cd35a
      Philippe Gerum authored
      
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      189cd35a
    • Philippe Gerum's avatar
      evl/irq: fix SMP kernel boot on uniprocessor machine · 3995461c
      Philippe Gerum authored
      
      
      We may be running a SMP kernel on a uniprocessor machine whose
      interrupt controller supports no IPI. We should attempt to hook IPIs
      only if the hardware can support multiple CPUs, otherwise it is
      unneeded and poised to fail.
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      3995461c
    • Philippe Gerum's avatar
      evl/sched: refine tracepoints · 54848cde
      Philippe Gerum authored
      
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      54848cde
    • Philippe Gerum's avatar
      evl/syscall: remove indirection via pointer table · 8d1858b0
      Philippe Gerum authored
      
      
      We have only very few syscalls, prefer a plain switch to a pointer
      indirection which ends up being fairly costly due to exploit
      mitigations.
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      8d1858b0
    • Philippe Gerum's avatar
      evl: kconfig: clarify wording · 2c69b296
      Philippe Gerum authored
      
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      2c69b296
    • Philippe Gerum's avatar
      evl/wait: display waitqueue name in trace · 7b0812b3
      Philippe Gerum authored
      
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      7b0812b3
    • Philippe Gerum's avatar
      evl: kconfig: introduce high per-CPU concurrency switch · 382242af
      Philippe Gerum authored
      
      
      EVL_HIGH_PERCPU_CONCURRENCY optimizes the implementation for
      applications with many real-time threads running concurrently on any
      given CPU core (typically when eight or more threads may be sharing a
      single CPU core). This is a combination of the scalable scheduler and
      rb-tree timer indexing as a single configuration switch, since both
      aspects are normally coupled.
      
      If the application system runs only a few EVL threads per CPU core,
      then this option should be turned off, in order to minimize the cache
      footprint of the queuing operations performed by the scheduler and
      timer subsystems. Otherwise, it should be turned on in order to have
      constant-time queuing operations for a large number of runnable
      threads and outstanding timers.
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      382242af
    • Philippe Gerum's avatar
      evl/sched: enable fast linear thread scheduler (non-scalable) · 5029c829
      Philippe Gerum authored
      
      
      For applications with only few runnable tasks at any point in time, a
      linear queue ordering the latter for scheduling delivers better
      performance on low-end systems due to smaller CPU cache footprints,
      compared to the multi-level queue used by the scalable scheduler.
      
      Allow users to select between lightning-fast and scalable scheduler
      implementation depending on the runtime profile of the application.
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      
      # Please enter the commit message for your changes. Lines starting
      # with '#' will be ignored, and an empty message aborts the commit.
      #
      # On branch evl/master
      # Your branch is ahead of 'origin/evl/master' by 2 commits.
      #   (use "git push" to publish your local commits)
      #
      # Changes to be committed:
      #	modified:   include/evl/sched.h
      #	modified:   include/evl/sched/queue.h
      #	modified:   include/evl/sched/tp.h
      #	modified:   include/evl/sched/weak.h
      #	modified:   kernel/evl/Kconfig
      #	modified:   kernel/evl/sched/core.c
      #
      # Untracked files:
      #	include/trace/events/mm.h
      #
      5029c829
    • Philippe Gerum's avatar
      8a272917
    • Philippe Gerum's avatar
      evl/timer: add linear indexing method · 48a9426d
      Philippe Gerum authored
      
      
      Add (back) the ability to index timers either in a rb-tree or linked
      to a basic linked list.
      
      The latter delivers lower latency to applications systems with very
      few active timers at any point in time (typically less than 10 active
      timers, e.g. not more than a couple of timed loops, very few timed
      syscalls).
      Signed-off-by: Philippe Gerum's avatarPhilippe Gerum <rpm@xenomai.org>
      48a9426d
  3. 15 May, 2021 2 commits
  4. 03 May, 2021 15 commits