You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Linux-Architecture-Reference/Platform/ch_smp.xml

374 lines
20 KiB
XML

<?xml version="1.0"?>
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xl="http://www.w3.org/1999/xlink"
xml:id="dbdoclet.50569340_14972"
version="5.0"
xml:lang="en">
<title>The Symmetric Multiprocessor Option</title>
<para>This architecture supports the implementation of symmetric
multiprocessor (SMP) systems as an optional feature. This Chapter provides
information concerning the design and programming of such systems. For SMP OF
binding information, see <xref linkend="LoPAR.DeviceTree"/>.</para>
<para>SMP systems differ from uniprocessors in a number of ways. These
differences are not all covered in this chapter. Other chapters that cover
SMP-related topics include:</para>
<itemizedlist>
<listitem>
<para>Non-processor-related initialization and other requirements:
<xref linkend="dbdoclet.50569327_31987"/></para>
</listitem>
<listitem>
<para>Interrupts: <xref linkend="dbdoclet.50569331_37856"/></para>
</listitem>
<listitem>
<para>Error handling: <xref linkend="LoPAR.Error"/></para>
</listitem>
</itemizedlist>
<para>Many other general characteristics of SMPs&#x2014;such as
interprocessor communication, load/store ordering, and cache
coherence&#x2014;are defined in <xref linkend="dbdoclet.50569387_99718"/>.
Requirements and recommendations for system organization and time base synchronization are
discussed here, along with SMP-specific aspects of the boot process.</para>
<para>SMP platforms require SMP-specific OS support. An OS supporting only
uniprocessor platforms
may not be usable on an SMP, even when an SMP platform has only a single
processor installed; conversely, an SMP-supporting OS may not be usable on a
uniprocessor. It is, however, a requirement that uniprocessor OSs be able to
run on one-processor SMPs, and that SMP-enabled OSs also run on uniprocessors.
See the next section.</para>
<section xml:id="dbdoclet.50569340_29104">
<title>SMP System Organization</title>
<para>This chapter only addresses SMP multiprocessor platforms. This is a
computer system in which multiple processors equally share functional and
timing access to and control over all other system components, including memory
and I/O, as defined in the requirements below. Other
multiprocessor organizations (&#x201C;asymmetric
multiprocessors,&#x201D; &#x201C; attached
processors,&#x201D; etc.) are not included in this architecture. These might,
for example, include systems in which only one processor can perform I/O
operations; or in which processors have private memory that is not accessible
by other processors. </para>
<para>Requirements <xref linkend="dbdoclet.50569340_73893"/> through
<xref linkend="dbdoclet.50569340_79137"/>, further require that all processors
be of (nearly) equal speed, type, cache characteristics, etc. Requirements for
optional non-uniform multiprocessor platforms are found in
<xref linkend="dbdoclet.50569346_35960"/>.</para>
<variablelist>
<varlistentry xml:id="dbdoclet.50569340_80907">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-1.</emphasis></term>
<listitem>
<para>OSs that do not explicitly support the SMP option must support
SMP-enabled platforms, actively using only one processor. </para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-2.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor
option:</emphasis> SMP OSs must support uniprocessor platforms.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-3.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor
option:</emphasis> The extensions defined in
<xref linkend="LoPAR.DeviceTree"/>, and the SMP support section of the RTAS
specifications (see <xref linkend="LoPAR.RTAS"/>) must be implemented.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_73893">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-4.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor or Power Management
option:</emphasis> All processors in the configuration must have equal
functional access and &#x201C;quasi-equal&#x201D;
timing access to all of system memory,
including other processors&#x2019; caches, via cache coherence.
&#x201C;Quasi-equal&#x201D; means that the time required for processors to
access memory is sufficiently close to being equal that all software can ignore
the difference without a noticeable negative impact on system performance; and
no software is expected to profitably exploit the difference in timing. </para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_25908">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-5.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
All processors in the configuration must have equal functional and
&#x201C;quasi-equal&#x201D;
timing access to all I/O devices and IOAs.
&#x201C;Quasi-equal&#x201D; is defined as in Requirement <xref linkend="dbdoclet.50569340_73893"/>,
above, with I/O access replacing memory access for this case. </para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_63554">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-6.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
SMP OSs must at least support SMPs with the same PVR contents and speed. The
PVR contents includes both the PVN and the revision number.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_79137">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-7.</emphasis></term>
<listitem>
<para><emphasis role="bold">For the Symmetric Multiprocessor option:</emphasis>
All caches at the same hierarchical level must have the same OF properties.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_12670">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-8.</emphasis></term>
<listitem>
<para>Hardware for SMPs must provide a means for synchronizing all the
time bases of all the processors in the platform, for use by platform firmware.
See <xref linkend="LoPAR.RTAS"/>. This is for purposes of clock synchronization
at initialization and at times when the processor loses time base state.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="dbdoclet.50569340_88608">
<term><emphasis role="bold">R1-<xref linkend="dbdoclet.50569340_29104"
xrefstyle="select: labelnumber nopage"/>-9.</emphasis></term>
<listitem>
<para>The platform must initialize and maintain the synchronization of
the time bases and timers of all platform processors such that; for any code
sequence &#x201C;C&#x201D;, run between any two platform processors
&#x201C;A&#x201D; and &#x201C;B&#x201D;, where the reading of the time base or
timer in processor &#x201C;A&#x201D; can be architecturally guaranteed to have
happened later in time than the reading of the time base or timer in processor
&#x201C;B&#x201D;, the value of the time base read by processor
&#x201C;A&#x201D; is greater than or equal to the value of the time base read
by processor &#x201C;B&#x201D;.</para>
</listitem>
</varlistentry>
</variablelist>
<para><emphasis role="bold">Software Implementation Notes:</emphasis></para>
<orderedlist>
<listitem>
<para>Requirement <xref linkend="dbdoclet.50569340_80907"/> has
implications on the design of uniprocessor OSs, particularly regarding the
handling of interrupts. See the sections that follow, particularly
<xref linkend="dbdoclet.50569340_57956"/>.</para>
</listitem>
<listitem>
<para>While Requirement <xref linkend="dbdoclet.50569340_63554"/> does
not require this, OSs are encouraged to support processors of the same type but
different PVR contents as long as their programming models are
compatible.</para>
</listitem>
<listitem>
<para>Because of performance penalties associated with inter-processor
synchronization, the weakest synchronization primitive that produces correct
operation should be used. For example, <emphasis>eieio</emphasis> can often be
used as part of a sequence that unlocks a data structure, rather than the
higher-overhead but more general <emphasis>sync</emphasis> instruction. </para>
</listitem>
</orderedlist>
<para><emphasis role="bold">Hardware Implementation Notes:</emphasis></para>
<orderedlist>
<listitem>
<para>Particularly when used as servers, SMP systems make heavy demands
on the I/O and memory subsystems. Therefore, it is strongly recommended that
the I/O and memory subsystem of an SMP platform should either be expandable as
additional processors are added, or else designed to handle the load of the
maximum system configuration.</para>
</listitem>
<listitem xml:id="dbdoclet.50569340_marker-513176">
<para>Defining an exact numeric threshold for
&#x201C;quasi-equal&#x201D; is not feasible because it depends on the
application, compiler, subsystem, and OS software that the system is to run. It
is highly likely that a wider range of timing differences can be absorbed in
I/O access time than in memory access time. An illustrative example that is
deliberately far from an upper bound: A 2% timing difference is certainly
quasi-equal by this definition. While significantly larger timing differences
are undoubtedly also quasi-equal, more conclusive statements must be the
province of the OS and other software.</para>
</listitem>
</orderedlist>
</section>
<section xml:id="dbdoclet.50569340_19429">
<title>An SMP Boot Process</title>
<para>Booting
an SMP entails considerations not present when booting a
uniprocessor. This section indicates those considerations by describing a way
in which an SMP system can be booted. It does not pretend to describe
&#x201C;the&#x201D; way to boot an SMP, since there are a wide variety of ways
to do this, depending on engineering choices that can differ from platform to
platform. To illustrate the possibilities, several variations on the SMP
booting theme will be described after the initial description.</para>
<para>This section concentrates solely on SMP-related issues, and ignores a
number of other initialization issues such as hibernation and suspension. See
<xref linkend="dbdoclet.50569327_34213"/> for a discussion of those other
issues. </para>
<section xml:id="dbdoclet.50569340_33673">
<title>SMP-Safe Boot</title>
<para>The basic booting process described here is called
&#x201C;SMP-Safe&#x201D; because it tolerates the presence of multiple
processors, but does not exploit them. This process proceeds as follows: </para>
<orderedlist>
<listitem>
<para>At power on, one or more finite state machines (FSMs) built into
the system hardware initialize each processor independently. FSMs also perform
basic initialization of other system elements, such as the memory and interrupt
controllers. </para>
</listitem>
<listitem>
<para>After the FSM initialization of each processor concludes, it
begins execution at a location in ROM that the FSM has specified. This is the
start of execution of the system firmware that eventually provides the OF
interfaces to the OS. </para>
</listitem>
<listitem xml:id="dbdoclet.50569340_44228">
<para>One of the first things that firmware does is establish one of the processors as the
<emphasis>master</emphasis>: The
<emphasis>master</emphasis> is a single processor which
continues with the rest of the booting process; all the others are placed in a
<emphasis>stopped</emphasis> state. A processor in this
<emphasis>stopped</emphasis> state is out of the picture; it does nothing that affects
the state of the system and will continue to be in that state until awakened by
some outside force, such as an inter-processor interrupt (IPI).<footnote xml:id="pgfId-242214"><para>Another
characteristic of the <emphasis>stopped</emphasis> state,
defined in <xref linkend="LoPAR.RTAS"/>, is that the
processor remembers nothing of its prior life when placed in a
<emphasis>stopped</emphasis> state; this distinguishes it from the
<emphasis>idle</emphasis> state. That isn&#x2019;t strictly necessary for this booting
process; <emphasis>idle</emphasis> could have been used. However, since the
non-<emphasis>master</emphasis> processor must be in the
<emphasis>stopped</emphasis> state when the OS is started,
<emphasis>stopped</emphasis> might as well be used.</para></footnote></para>
<para>One way to choose the <emphasis>master</emphasis> is to include a special register
at a fixed address in the memory controller. That special register has the
following properties: </para>
<itemizedlist>
<listitem>
<para>The FSM initializing the memory controller sets this
register&#x2019;s contents to 0 (zero). </para>
</listitem>
<listitem>
<para>The first time that register is read, it returns the value 0 and
then sets its own contents to non-zero. This is performed as an atomic
operation; if two or more processors attempt to read the register at the same
time, exactly one of them will get the 0 and the rest will get a non-zero
value. </para>
</listitem>
<listitem>
<para>After the first attempt, all attempts to read that
register&#x2019;s contents return a non-zero value. </para>
</listitem>
</itemizedlist>
<para>The <emphasis>master</emphasis> is then picked by having all the
processors read from that special register. Exactly one of them will receive a
0 and thereby become the <emphasis>master</emphasis>. </para>
<para>Note that the operation of choosing the
<emphasis>master</emphasis> cannot be done using the PA memory locking instructions,
since at this point in the boot process the memory is not initialized. The
advantage to using a register in the memory controller is that system bus
serialization can be used to automatically provide the required
atomicity.</para>
</listitem>
<listitem>
<para>The <emphasis>master</emphasis> chosen in step
<xref linkend="dbdoclet.50569340_44228"/> then proceeds to do the remainder of the
system initialization. This includes, for example, the remainder of Power-On
Self Test, initialization of OF, discovery of devices and construction of the
OF device tree, loading the OS, starting it, and so on. Since one processor is
performing all these functions, and the rest are in a state where they are not
affecting anything, code that is at least very close to the uniprocessor code
can be used for all of this (but see <xref linkend="dbdoclet.50569340_57956"/>
below).</para>
</listitem>
<listitem>
<para>The OS begins execution on the single
<emphasis>master</emphasis> processor. It uses the OF Client Interface
Services to start each of the other processors, taking them out of the
<emphasis>stopped</emphasis> state and setting them loose on the SMP OS code.</para>
<para>This completes the example SMP boot process. Variations are
discussed beginning at <xref linkend="dbdoclet.50569340_69262"/>. Before
discussing those variations, an element of the system initialization not
discussed above will be covered.</para>
</listitem>
</orderedlist>
</section>
<section xml:id="dbdoclet.50569340_57956">
<title>Finding the Processor Configuration</title>
<para>Unlike uniprocessor initialization, SMP initialization must also
discover the number and identities of the processors installed in the system.
&#x201C;Identity&#x201D; means the interrupt address of each processor as seen
by the interrupt controller; without that information, a processor cannot reset
interrupts directed at it. This identity is determined by board wiring: The
processor attached to the &#x201C;processor 0&#x201D; wire from the interrupt
controller has identity 0. For information about how this identity is used, see
<xref linkend="LoPAR.DeviceTree"/>.</para>
<para>The method used by a platform to identify its processors is
dependent upon the platform hardware design and may be based upon service
processor information, identification registers, inter-processor interrupts, or
other novel techniques.</para>
</section>
<section xml:id="dbdoclet.50569340_69262">
<title>SMP-Efficient Boot </title>
<para>The booting
process as described so far tolerates the existence of multiple processors but
does not attempt to exploit them. It is possible that the booting process can
be sped up by actively using multiple processors simultaneously. In that case,
the pick-a-<emphasis>master</emphasis> technique must still be
used to perform sufficient initialization that other inter-processor
coordination facilities&#x2014;in-memory locks and IPIs&#x2014;can be used.
Once that is accomplished, normal parallel SMP programming techniques can be
used within the initialization process itself.</para>
</section>
<section>
<title>Use of a Service Processor</title>
<para>A system might contain a service processor that is distinct from the processors
that form the SMP. If that service processor has suitably intimate access to
and control over each of the SMP processors, it can perform the operations of
choosing a <emphasis>master</emphasis> and discovering the SMP processor
configuration.</para>
<para>&#xA0;</para>
</section>
</section>
</chapter>