This architecture supports the implementation of symmetric multiprocessor (SMP) systems as an optional feature. This Chapter provides information concerning the design and programming of such systems. For SMP OF binding information, see . SMP systems differ from uniprocessors in a number of ways. These differences are not all covered in this chapter. Other chapters that cover SMP-related topics include: Non-processor-related initialization and other requirements: Interrupts: Error handling: Many other general characteristics of SMPs—such as interprocessor communication, load/store ordering, and cache coherence—are defined in . Requirements and recommendations for system organization and time base synchronization are discussed here, along with SMP-specific aspects of the boot process. SMP platforms require SMP-specific OS support. An OS supporting only uniprocessor platforms may not be usable on an SMP, even when an SMP platform has only a single processor installed; conversely, an SMP-supporting OS may not be usable on a uniprocessor. It is, however, a requirement that uniprocessor OSs be able to run on one-processor SMPs, and that SMP-enabled OSs also run on uniprocessors. See the next section.

This chapter only addresses SMP multiprocessor platforms. This is a computer system in which multiple processors equally share functional and timing access to and control over all other system components, including memory and I/O, as defined in the requirements below. Other multiprocessor organizations (“asymmetric multiprocessors,” “ attached processors,” etc.) are not included in this architecture. These might, for example, include systems in which only one processor can perform I/O operations; or in which processors have private memory that is not accessible by other processors. Requirements through , further require that all processors be of (nearly) equal speed, type, cache characteristics, etc. Requirements for optional non-uniform multiprocessor platforms are found in . R1--1. OSs that do not explicitly support the SMP option must support SMP-enabled platforms, actively using only one processor. R1--2. For the Symmetric Multiprocessor option: SMP OSs must support uniprocessor platforms. R1--3. For the Symmetric Multiprocessor option: The extensions defined in , and the SMP support section of the RTAS specifications (see ) must be implemented. R1--4. For the Symmetric Multiprocessor or Power Management option: All processors in the configuration must have equal functional access and “quasi-equal” timing access to all of system memory, including other processors’ caches, via cache coherence. “Quasi-equal” means that the time required for processors to access memory is sufficiently close to being equal that all software can ignore the difference without a noticeable negative impact on system performance; and no software is expected to profitably exploit the difference in timing. R1--5. For the Symmetric Multiprocessor option: All processors in the configuration must have equal functional and “quasi-equal” timing access to all I/O devices and IOAs. “Quasi-equal” is defined as in Requirement , above, with I/O access replacing memory access for this case. R1--6. For the Symmetric Multiprocessor option: SMP OSs must at least support SMPs with the same PVR contents and speed. The PVR contents includes both the PVN and the revision number. R1--7. For the Symmetric Multiprocessor option: All caches at the same hierarchical level must have the same OF properties. R1--8. Hardware for SMPs must provide a means for synchronizing all the time bases of all the processors in the platform, for use by platform firmware. See . This is for purposes of clock synchronization at initialization and at times when the processor loses time base state. R1--9. The platform must initialize and maintain the synchronization of the time bases and timers of all platform processors such that; for any code sequence “C”, run between any two platform processors “A” and “B”, where the reading of the time base or timer in processor “A” can be architecturally guaranteed to have happened later in time than the reading of the time base or timer in processor “B”, the value of the time base read by processor “A” is greater than or equal to the value of the time base read by processor “B”. Software Implementation Notes: Requirement has implications on the design of uniprocessor OSs, particularly regarding the handling of interrupts. See the sections that follow, particularly . While Requirement does not require this, OSs are encouraged to support processors of the same type but different PVR contents as long as their programming models are compatible. Because of performance penalties associated with inter-processor synchronization, the weakest synchronization primitive that produces correct operation should be used. For example, eieio can often be used as part of a sequence that unlocks a data structure, rather than the higher-overhead but more general sync instruction. Hardware Implementation Notes: Particularly when used as servers, SMP systems make heavy demands on the I/O and memory subsystems. Therefore, it is strongly recommended that the I/O and memory subsystem of an SMP platform should either be expandable as additional processors are added, or else designed to handle the load of the maximum system configuration. Defining an exact numeric threshold for “quasi-equal” is not feasible because it depends on the application, compiler, subsystem, and OS software that the system is to run. It is highly likely that a wider range of timing differences can be absorbed in I/O access time than in memory access time. An illustrative example that is deliberately far from an upper bound: A 2% timing difference is certainly quasi-equal by this definition. While significantly larger timing differences are undoubtedly also quasi-equal, more conclusive statements must be the province of the OS and other software.

Booting an SMP entails considerations not present when booting a uniprocessor. This section indicates those considerations by describing a way in which an SMP system can be booted. It does not pretend to describe “the” way to boot an SMP, since there are a wide variety of ways to do this, depending on engineering choices that can differ from platform to platform. To illustrate the possibilities, several variations on the SMP booting theme will be described after the initial description. This section concentrates solely on SMP-related issues, and ignores a number of other initialization issues such as hibernation and suspension. See for a discussion of those other issues.

The basic booting process described here is called “SMP-Safe” because it tolerates the presence of multiple processors, but does not exploit them. This process proceeds as follows: At power on, one or more finite state machines (FSMs) built into the system hardware initialize each processor independently. FSMs also perform basic initialization of other system elements, such as the memory and interrupt controllers. After the FSM initialization of each processor concludes, it begins execution at a location in ROM that the FSM has specified. This is the start of execution of the system firmware that eventually provides the OF interfaces to the OS. One of the first things that firmware does is establish one of the processors as the master: The master is a single processor which continues with the rest of the booting process; all the others are placed in a stopped state. A processor in this stopped state is out of the picture; it does nothing that affects the state of the system and will continue to be in that state until awakened by some outside force, such as an inter-processor interrupt (IPI).Another characteristic of the stopped state, defined in , is that the processor remembers nothing of its prior life when placed in a stopped state; this distinguishes it from the idle state. That isn’t strictly necessary for this booting process; idle could have been used. However, since the non-master processor must be in the stopped state when the OS is started, stopped might as well be used. One way to choose the master is to include a special register at a fixed address in the memory controller. That special register has the following properties: The FSM initializing the memory controller sets this register’s contents to 0 (zero). The first time that register is read, it returns the value 0 and then sets its own contents to non-zero. This is performed as an atomic operation; if two or more processors attempt to read the register at the same time, exactly one of them will get the 0 and the rest will get a non-zero value. After the first attempt, all attempts to read that register’s contents return a non-zero value. The master is then picked by having all the processors read from that special register. Exactly one of them will receive a 0 and thereby become the master. Note that the operation of choosing the master cannot be done using the PA memory locking instructions, since at this point in the boot process the memory is not initialized. The advantage to using a register in the memory controller is that system bus serialization can be used to automatically provide the required atomicity. The master chosen in step then proceeds to do the remainder of the system initialization. This includes, for example, the remainder of Power-On Self Test, initialization of OF, discovery of devices and construction of the OF device tree, loading the OS, starting it, and so on. Since one processor is performing all these functions, and the rest are in a state where they are not affecting anything, code that is at least very close to the uniprocessor code can be used for all of this (but see below). The OS begins execution on the single master processor. It uses the OF Client Interface Services to start each of the other processors, taking them out of the stopped state and setting them loose on the SMP OS code. This completes the example SMP boot process. Variations are discussed beginning at . Before discussing those variations, an element of the system initialization not discussed above will be covered.

Unlike uniprocessor initialization, SMP initialization must also discover the number and identities of the processors installed in the system. “Identity” means the interrupt address of each processor as seen by the interrupt controller; without that information, a processor cannot reset interrupts directed at it. This identity is determined by board wiring: The processor attached to the “processor 0” wire from the interrupt controller has identity 0. For information about how this identity is used, see . The method used by a platform to identify its processors is dependent upon the platform hardware design and may be based upon service processor information, identification registers, inter-processor interrupts, or other novel techniques.

The booting process as described so far tolerates the existence of multiple processors but does not attempt to exploit them. It is possible that the booting process can be sped up by actively using multiple processors simultaneously. In that case, the pick-a-master technique must still be used to perform sufficient initialization that other inter-processor coordination facilities—in-memory locks and IPIs—can be used. Once that is accomplished, normal parallel SMP programming techniques can be used within the initialization process itself.

A system might contain a service processor that is distinct from the processors that form the SMP. If that service processor has suitably intimate access to and control over each of the SMP processors, it can perform the operations of choosing a master and discovering the SMP processor configuration.