Linux-Architecture-Reference/LoPAR/ch_smp.xml

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Copyright (c) 2016, 2020 OpenPOWER Foundation

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.

-->
<chapter  xmlns="http://docbook.org/ns/docbook"
          xmlns:xl="http://www.w3.org/1999/xlink"
          xml:id="dbdoclet.50569340_14972"
          version="5.0"
          xml:lang="en">
  <title>The Symmetric Multiprocessor Option</title>

  <para>This   architecture supports the implementation of symmetric
  multiprocessor (SMP) systems as an optional feature. This Chapter provides
  information concerning the design and programming of such systems. For SMP OF
  binding information, see <xref linkend="dbdoclet.50569368_56107"/>.</para>
  <para>SMP systems differ from uniprocessors in a number of ways. These
  differences are not all covered in this chapter. Other chapters that cover
  SMP-related topics include:</para>

  <itemizedlist>
    <listitem>
      <para>Non-processor-related initialization and other requirements:
      <xref linkend="dbdoclet.50569327_31987"/></para>
    </listitem>

    <listitem>
      <para>Interrupts: <xref linkend="dbdoclet.50569331_37856"/></para>
    </listitem>

    <listitem>
      <para>Error handling: <xref linkend="dbdoclet.50569337_37595"/></para>
    </listitem>
  </itemizedlist>

  <para>Many other general characteristics of SMPs&#x2014;such as
  interprocessor communication, load/store ordering, and cache
  coherence&#x2014;are defined in <xref linkend="dbdoclet.50569387_99718"/>.
  Requirements and recommendations for system organization and time base synchronization are
  discussed here, along with SMP-specific aspects of the boot process.</para>
  <para>SMP platforms require SMP-specific OS support. An OS supporting only
  uniprocessor platforms
  may not be usable on an SMP, even when an SMP platform has only a single
  processor installed; conversely, an SMP-supporting OS may not be usable on a
  uniprocessor. It is, however, a requirement that uniprocessor OSs be able to
  run on one-processor SMPs, and that SMP-enabled OSs also run on uniprocessors.
  See the next section.</para>

  <section xml:id="dbdoclet.50569340_29104">
    <title>SMP System Organization</title>
    <para>This chapter only addresses SMP multiprocessor platforms. This is a
    computer system in which multiple processors equally share functional and
    timing access to and control over all other system components, including memory
    and I/O, as defined in the requirements below. Other
    multiprocessor organizations (&#x201C;asymmetric
    multiprocessors,&#x201D; &#x201C; attached
    processors,&#x201D; etc.) are not included in this architecture. These might,
    for example, include systems in which only one processor can perform I/O
    operations; or in which processors have private memory that is not accessible
    by other processors. </para>
    <para>Requirements <xref linkend="dbdoclet.50569340_73893"/> through
    <xref linkend="dbdoclet.50569340_79137"/>, further require that all processors
    be of (nearly) equal speed, type, cache characteristics, etc. Requirements for
    optional non-uniform multiprocessor platforms are found in
    <xref linkend="dbdoclet.50569346_35960"/>.</para>

    <variablelist>
     	<varlistentry xml:id="dbdoclet.50569340_80907">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-1.</emphasis></term>
          <listitem>
            <para>OSs that do not explicitly support the SMP option must support
            SMP-enabled platforms, actively using only one processor. </para>
          </listitem>
     	</varlistentry>

     	<varlistentry>
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-2.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor
            option:</emphasis> SMP OSs must support uniprocessor platforms.</para>
          </listitem>
     	</varlistentry>

     	<varlistentry>
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-3.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor
            option:</emphasis> The extensions defined in
            <xref linkend="dbdoclet.50569368_56107"/>, and the SMP support section of the RTAS
            specifications (see <xref linkend="dbdoclet.50569332_36251"/>) must be implemented.</para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_73893">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-4.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor or Power Management
            option:</emphasis> All processors in the configuration must have equal
            functional access and &#x201C;quasi-equal&#x201D;
            timing access to all of system memory,
            including other processors&#x2019; caches, via cache coherence.
            &#x201C;Quasi-equal&#x201D; means that the time required for processors to
            access memory is sufficiently close to being equal that all software can ignore
            the difference without a noticeable negative impact on system performance; and
            no software is expected to profitably exploit the difference in timing. </para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_25908">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-5.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
            All processors in the configuration must have equal functional and
            &#x201C;quasi-equal&#x201D;
            timing access to all I/O devices and IOAs.
            &#x201C;Quasi-equal&#x201D; is defined as in Requirement <xref linkend="dbdoclet.50569340_73893"/>,
            above, with I/O access replacing memory access for this case. </para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_63554">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-6.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
            SMP OSs must at least support SMPs with the same PVR contents and speed. The
            PVR contents includes both the PVN and the revision number.</para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_79137">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-7.</emphasis></term>
          <listitem>
            <para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
            All caches at the same hierarchical level must have the same OF properties.</para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_12670">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-8.</emphasis></term>
          <listitem>
            <para>Hardware for SMPs must provide a means for synchronizing all the
            time bases of all the processors in the platform, for use by platform firmware.
            See <xref linkend="dbdoclet.50569332_36251"/>. This is for purposes of clock synchronization
            at initialization and at times when the processor loses time base state.</para>
          </listitem>
     	</varlistentry>

     	<varlistentry xml:id="dbdoclet.50569340_88608">
        <term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
        xrefstyle="select: labelnumber nopage"/>-9.</emphasis></term>
          <listitem>
            <para>The platform must initialize and maintain the synchronization of
            the time bases and timers of all platform processors such that; for any code
            sequence &#x201C;C&#x201D;, run between any two platform processors
            &#x201C;A&#x201D; and &#x201C;B&#x201D;, where the reading of the time base or
            timer in processor &#x201C;A&#x201D; can be architecturally guaranteed to have
            happened later in time than the reading of the time base or timer in processor
            &#x201C;B&#x201D;, the value of the time base read by processor
            &#x201C;A&#x201D; is greater than or equal to the value of the time base read
            by processor &#x201C;B&#x201D;.</para>
          </listitem>
     	</varlistentry>
    </variablelist>

    <para><emphasis role="bold">Software Implementation Notes:</emphasis></para>

    <orderedlist>
      <listitem>
        <para>Requirement <xref linkend="dbdoclet.50569340_80907"/> has
        implications on the design of uniprocessor OSs, particularly regarding the
        handling of interrupts. See the sections that follow, particularly
        <xref linkend="dbdoclet.50569340_57956"/>.</para>
      </listitem>

      <listitem>
        <para>While Requirement <xref linkend="dbdoclet.50569340_63554"/> does
        not require this, OSs are encouraged to support processors of the same type but
        different PVR contents as long as their programming models are
        compatible.</para>
      </listitem>

      <listitem>
        <para>Because of performance penalties associated with inter-processor
        synchronization, the weakest synchronization primitive that produces correct
        operation should be used. For example, <emphasis>eieio</emphasis> can often be
        used as part of a sequence that unlocks a data structure, rather than the
        higher-overhead but more general <emphasis>sync</emphasis> instruction. </para>
      </listitem>
    </orderedlist>

    <para><emphasis role="bold">Hardware Implementation Notes:</emphasis></para>
    <orderedlist>
      <listitem>
        <para>Particularly when used as servers, SMP systems make heavy demands
        on the I/O and memory subsystems. Therefore, it is strongly recommended that
        the I/O and memory subsystem of an SMP platform should either be expandable as
        additional processors are added, or else designed to handle the load of the
        maximum system configuration.</para>
      </listitem>

      <listitem xml:id="dbdoclet.50569340_marker-513176">
        <para>Defining an exact numeric threshold for
        &#x201C;quasi-equal&#x201D; is not feasible because it depends on the
        application, compiler, subsystem, and OS software that the system is to run. It
        is highly likely that a wider range of timing differences can be absorbed in
        I/O access time than in memory access time. An illustrative example that is
        deliberately far from an upper bound: A 2% timing difference is certainly
        quasi-equal by this definition. While significantly larger timing differences
        are undoubtedly also quasi-equal, more conclusive statements must be the
        province of the OS and other software.</para>
      </listitem>
    </orderedlist>
  </section>

  <section xml:id="dbdoclet.50569340_19429">
    <title>An SMP Boot Process</title>
    <para>Booting
    an SMP entails considerations not present when booting a
    uniprocessor. This section indicates those considerations by describing a way
    in which an SMP system can be booted. It does not pretend to describe
    &#x201C;the&#x201D; way to boot an SMP, since there are a wide variety of ways
    to do this, depending on engineering choices that can differ from platform to
    platform. To illustrate the possibilities, several variations on the SMP
    booting theme will be described after the initial description.</para>
    <para>This section concentrates solely on SMP-related issues, and ignores a
    number of other initialization issues such as hibernation and suspension. See
    <xref linkend="dbdoclet.50569327_34213"/> for a discussion of those other
    issues. </para>

    <section xml:id="dbdoclet.50569340_33673">
      <title>SMP-Safe Boot</title>
      <para>The basic booting process described here is called
      &#x201C;SMP-Safe&#x201D; because it tolerates the presence of multiple
      processors, but does not exploit them. This process proceeds as follows: </para>

      <orderedlist>
        <listitem>
          <para>At power on, one or more finite state machines (FSMs) built into
          the system hardware initialize each processor independently. FSMs also perform
          basic initialization of other system elements, such as the memory and interrupt
          controllers. </para>
        </listitem>

        <listitem>
          <para>After the FSM initialization of each processor concludes, it
          begins execution at a location in ROM that the FSM has specified. This is the
          start of execution of the system firmware that eventually provides the OF
          interfaces to the OS. </para>
        </listitem>

        <listitem xml:id="dbdoclet.50569340_44228">
          <para>One of the first things that firmware does is establish one of the processors as the
          <emphasis>master</emphasis>: The
          <emphasis>master</emphasis> is a single processor which
          continues with the rest of the booting process; all the others are placed in a
          <emphasis>stopped</emphasis> state. A processor in this
          <emphasis>stopped</emphasis> state is out of the picture; it does nothing that affects
          the state of the system and will continue to be in that state until awakened by
          some outside force, such as an inter-processor interrupt (IPI).<footnote xml:id="pgfId-242214"><para>Another
            characteristic of the <emphasis>stopped</emphasis> state,
            defined in <xref linkend="dbdoclet.50569374_59715"/>, is that the
            processor remembers nothing of its prior life when placed in a
            <emphasis>stopped</emphasis> state; this distinguishes it from the
            <emphasis>idle</emphasis> state. That isn&#x2019;t strictly necessary for this booting
            process; <emphasis>idle</emphasis> could have been used. However, since the
            non-<emphasis>master</emphasis> processor must be in the
            <emphasis>stopped</emphasis> state when the OS is started,
            <emphasis>stopped</emphasis> might as well be used.</para></footnote></para>
          <para>One way to choose the <emphasis>master</emphasis> is to include a special register
          at a fixed address in the memory controller. That special register has the
          following properties: </para>

          <itemizedlist>
            <listitem>
              <para>The FSM initializing the memory controller sets this
              register&#x2019;s contents to 0 (zero). </para>
            </listitem>

            <listitem>
              <para>The first time that register is read, it returns the value 0 and
              then sets its own contents to non-zero. This is performed as an atomic
              operation; if two or more processors attempt to read the register at the same
              time, exactly one of them will get the 0 and the rest will get a non-zero
              value. </para>
            </listitem>

            <listitem>
              <para>After the first attempt, all attempts to read that
              register&#x2019;s contents return a non-zero value. </para>
            </listitem>
          </itemizedlist>

          <para>The <emphasis>master</emphasis> is then picked by having all the
          processors read from that special register. Exactly one of them will receive a
          0 and thereby become the <emphasis>master</emphasis>. </para>
          <para>Note that the operation of choosing the
          <emphasis>master</emphasis> cannot be done using the PA memory locking instructions,
          since at this point in the boot process the memory is not initialized. The
          advantage to using a register in the memory controller is that system bus
          serialization can be used to automatically provide the required
          atomicity.</para>
        </listitem>

        <listitem>
          <para>The <emphasis>master</emphasis> chosen in step
          <xref linkend="dbdoclet.50569340_44228"/> then proceeds to do the remainder of the
          system initialization. This includes, for example, the remainder of Power-On
          Self Test, initialization of OF, discovery of devices and construction of the
          OF device tree, loading the OS, starting it, and so on. Since one processor is
          performing all these functions, and the rest are in a state where they are not
          affecting anything, code that is at least very close to the uniprocessor code
          can be used for all of this (but see <xref linkend="dbdoclet.50569340_57956"/>
          below).</para>
        </listitem>

        <listitem>
          <para>The OS begins execution on the single
          <emphasis>master</emphasis> processor. It uses the OF Client Interface
          Services to start each of the other processors, taking them out of the
          <emphasis>stopped</emphasis> state and setting them loose on the SMP OS code.</para>
          <para>This completes the example SMP boot process. Variations are
          discussed beginning at <xref linkend="dbdoclet.50569340_69262"/>. Before
          discussing those variations, an element of the system initialization not
          discussed above will be covered.</para>
        </listitem>
      </orderedlist>

    </section>

    <section xml:id="dbdoclet.50569340_57956">
      <title>Finding the Processor Configuration</title>
      <para>Unlike uniprocessor initialization, SMP initialization must also
      discover the number and identities of the processors installed in the system.
      &#x201C;Identity&#x201D; means the interrupt address of each processor as seen
      by the interrupt controller; without that information, a processor cannot reset
      interrupts directed at it. This identity is determined by board wiring: The
      processor attached to the &#x201C;processor 0&#x201D; wire from the interrupt
      controller has identity 0. For information about how this identity is used, see
      <xref linkend="dbdoclet.50569368_56107"/>.</para>
      <para>The method used by a platform to identify its processors is
      dependent upon the platform hardware design and may be based upon service
      processor information, identification registers, inter-processor interrupts, or
      other novel techniques.</para>
    </section>

    <section xml:id="dbdoclet.50569340_69262">
      <title>SMP-Efficient Boot </title>
      <para>The booting
      process as described so far tolerates the existence of multiple processors but
      does not attempt to exploit them. It is possible that the booting process can
      be sped up by actively using multiple processors simultaneously. In that case,
      the pick-a-<emphasis>master</emphasis> technique must still be
      used to perform sufficient initialization that other inter-processor
      coordination facilities&#x2014;in-memory locks and IPIs&#x2014;can be used.
      Once that is accomplished, normal parallel SMP programming techniques can be
      used within the initialization process itself.</para>
    </section>

    <section>
      <title>Use of a Service Processor</title>
      <para>A system might contain a service processor that is distinct from the processors
      that form the SMP. If that service processor has suitably intimate access to
      and control over each of the SMP processors, it can perform the operations of
      choosing a <emphasis>master</emphasis> and discovering the SMP processor
      configuration.</para>
      <para>&#xA0;</para>
    </section>
  </section>
</chapter>