Sun Blade Whitepaper

file:///C|/Documents%20and%20Settings/Administrator/Desktop/5_1_latest_whitepaper/sunblade.whitepaper.
html
The Sun Blade[tm] Workstation

and Sun Fire[tm] Server I/O
System Architecture
Version 1.0
Technical White Paper
file:///C|/Documents%20and%20Settings/Administrato...top/5_1_latest_whitepaper/sunblade.whitepaper.html (1 of 67) [5/7/2002 2:19:55 PM]

file:///C|/Documents%20and%20Settings/Administrator/Desktop/5_1_latest_whitepaper/sunblade.whitepaper.html
© 2002 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303 U.S.A. All rights reserved.
Printed in the United States of America.
The products described in this manual may be protected by one or more U.S. patents, foreign patents, or
pending applications.
Sun, Sun Microsystems, the Sun logo, Sun Blade, Sun Fire, Sun Fireplane, Solaris, Java, Ultra, and OpenBoot
are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.
FireWire is a trademark of Apple Computer, Inc., used under license.
UNIX is a registered trademark in the United States and other countries, exclusively licensed through X/Open
Company, Ltd.
All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC
International, Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon
an architecture developed by Sun Microsystems, Inc.
RESTRICTED RIGHTS: Use, duplication, or disclosure by the government is subject to restrictions as set forth in
subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013
and FAR 52.227-19.
THIS PUBLICATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.
THIS PUBLICATION COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS.

CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE
INCORPORATED IN NEW EDITIONS OF THE PUBLICATION. SUN MICROSYSTEMS, INC. MAY MAKE
IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR PROGRAM(S) DESCRIBED IN THIS
PUBLICATION AT ANY TIME.

Contents
_________
1. Introduction
1.1 Intent
1.2 Focus
1.3 Overview
2. UltraSPARC® II System Architecture
2.1 UPA Interconnect Bar Switch
2.2 Host PCI Bridge
2.3 PCI I/O Bridge
3. UltraSPARC III System Architecture
3.1 Sun[tm] Fireplane Interconnect
3.2 Host PCI Bridge
3.3 PCI I/O Bridge
3.4 UltraSPARC III Processor
3.5 FC-AL
3.6 Ethernet
4. Host PCI Bridge ASIC
4.1 PCI Bus Module
4.1.1 Supported PCI Features
4.1.2 Unsupported PCI features

4.1.3 Supported PCI Transactions
4.2 Streaming Cache
4.3 PCI IOMMU
4.4 Mondo Dispatch Unit
4.5 Interface Controller
4.6 Summary of PCI Characteristics
5. PCI I/O Bridge ASIC
5.1 Bus Adapter
5.2 Channel Engine Interface
5.3 1394 Channel Engine
5.4 USB Channel Engine
5.5 EBus Channel Engine
5.6 Ethernet Channel Engine
6. Open Boot Firmware
6.1 Device Properties
6.2 IEEE 1275 PCI Binding
6.2.1 Address Formats and Representations
6.2.2 Child Node Properties
6.3 IEEE 1275 USB Binding
6.3.1 Host Controller Node Properties
6.3.2 Device Node Properties
6.3.3 Child/Interface Node Properties

7. Solaris[tm] Device Tree
7.1 Tree Structure
7.2 Device Drivers
7.3 Displaying the Device Tree
7.4 Binding a Driver to a Device
7.5 Generic Device Names
8. Appendix A: Sun Blade Workstations
8.1 Sun Blade 100 Workstation
9. Appendix B: Sun Fire Servers
9.1 Sun Fire 280R Server
9.2 Sun Fire 880 Server
Figures
_________
Figure 2-1. Typical Sun Ultra PCI Bus Architecture Implementation
Figure 3-1. Typical Sun Fire PCI Bus Architecture Implementation
Figure 4-1. HPB Conceptual Block Diagram

Figure 4-2. HPB Byte Twisting
Figure 4-3. Virtual-to-Physical Address Translation
Figure 5-1. PIB Conceptual Block Diagram
Figure 6-1. Sun Blade 1000 Device Tree
Figure 6-2. PCI Bus B Properties
Figure 6-3. Address and Size Formats
Figure 6-4. Sun Blade 1000 SCSI Controller Properties
Figure 6-5. reg Property of scsi@6
Figure 6-6. Information Using Forth
Figure 6-7. Sun Blade 1000, USB Hub, Omega USB Predator CDRW
Figure 7-1. Example Device Tree
Figure 7-2 printf Example
Figure 7-3. /devices Hierarchy Sample
Figure 7-4. Device Node Names
Figure 7-5. Specific Driver Node Binding
Figure 7-6. Generic Driver Node Binding
Figure 8-1. Sun Blade 100 System Design
Figure 9-1. Sun Fire 280R System Design
Figure 9-2. Sun Fire 880 System Design
Figure 9-3. Sun Fire 880 Device Tree
Figure 9-4. Sun Fire 3800 Server cPCI I/O Assembly
Figure 9-5. Sun Fire 6800/4810/4800 Server PCI I/O Assembly

Figure 9-6. Block Diagram of the Sun Fire 6800/4810/4800/3800 Servers
Figure 9-7. Standard Operation of the Sun Fire 6800/4810/4800/3800 Servers
Figure 9-8. Sun Fire 15000 System Interconnect
Figure 9-9. Sun Fire 15000 Board Set Block Diagram
Tables
_________
Table 4-1. Supported PCI Transactions
Table 4-2. PCI Characteristics Summary
Table 6-1. Address and Size Format Codes
Table 6-2. Device Node Name Property
Table 6-3. Interface Node Name Property
Table 8-1. PCI Slot Configuration for Sun Fire Servers
Table 9-1. PCI Slot Configuration for Sun Blade Workstations

1. Introduction
________________________________________________________________________
1.1 Intent
This white paper is intended to be a central source for the I/O configuration in Sun Blade[tm] workstations and
Sun Fire[tm] servers. A new system architecture has been introduced to take advantage of the UltraSPARC® III
processor. The driving force to rearchitect the host bus architecture has been the ever-increasing demand for
more bandwidth, specifically I/O throughput and reduced memory latency. For more information on the evolution
of bandwidth, see "Gigabit Ethernet, Accelerating the Standard for Speed" [7]. Although PCI card and drivers
should not require any change, opportunities for performance enhancement may arise.
1.2 Focus
This paper will focus on the new technologies and components introduced in the architecture. The new
technologies now standard on these systems include the Universal Serial Bus (USB), Fibre Channel Arbitrated
Loop (FC-AL), and IEEE 1394. Gigabit Ethernet is now standard on the servers. Compact PCI (cPCI), a more
rugged version of PCI, is also being offered on several of the Sun Fire server models. The new components
include the Sun Fireplane[tm] interconnect bus, new Host PCI Bridge (HPB), and new PCI I/O Bridge (PIB).
Since its inception, Sun Microsystems has had a history of supporting industry-leading standard bus designs.
Beginning with the Ultra[tm] workstation series, the PCI local bus has been the standard I/O bus for peripheral
interconnection. PCI still remains the main bus architecture in the new systems, though a number of
enhancements have been made. The Sun Fire server architecture will be given primary focus over the Sun Blade
workstation architecture.
1.3 Overview
Section 2 gives a brief overview of the Ultra workstation system I/O architecture, while Section 3 gives a high-
level system overview of the new components introduced in the Sun Blade workstations and Sun Fire servers. In
section 4 and 5, the two key I/O components are examined in closer detail. Section 6 provides Open Boot PROM
(OBP) examples to explain the system from the firmware point of view and Section 7 from the operating system
level. Appendix A and B list the PCI I/O configurations of the Sun Blade and Sun Fire systems individually.
2. UltraSPARC II System Architecture

________________________________________________________________________
This section provides an overview of the Sun Ultra system architectures that use UltraSPARC II processors.
Components that are replaced in the UltraSPARC III system architecture as examined in Section 3 will be
discussed. These include the UPA Interconnect Bar Switch, the Host PCI Bridge, and the PIC I/O Bridge.

2.1 UPA Interconnect Cross Bar Switch

Starting in 1995, Sun Ultra systems featuring UltraSPARC II processors have been built around the Ultra Port
Architecture bus (see Figure 2-1), interconnecting the processor, main memory, and the PCI bus. This
architecture broke new ground in desktop systems by using a switch-based architecture instead of a single bus
as system interconnect. Before this, switch-based interconnects were only used in mainframe systems. The UPA
Interconnect Cross Bar Switch is also the main bus for the high-end graphics cards.
Figure 2-1. Typical Sun Ultra PCI Bus Architecture Implementation
2.2 Host PCI Bridge

The Host PCI Bridge provides an interconnect between the processor and peripheral components, enabling the
processor to access main memory independently of other PCI bus masters. This unique architecture supports
the separation of the I/O bus from the processor's host bus, speeding overall system throughput. The Host PCI
Bridge provides two PCI buses, a 32/64-bit, 33-MHz bus and a 32/64-bit, 66-MHz PCI bus. Internal storage and
other internal I/O devices share the 32/64-bit, 33-MHz PCI bus.
2.3 PCI I/O Bridge

The PCI I/O Bridge supports a wide range of devices -- including audio, 100-Mbit/sec Ethernet, serial ports, and

parallel ports. For more information and details on the Ultra Architecture, please refer to "The PCI Bus and
Sun[tm] Ultra[tm] Systems Technical Brief"[1] or "Writing Solaris[tm] PCI Device Drivers for Sun[tm] SPARC[tm]
Platforms"[2].
3. UltraSPARC III System Architecture

________________________________________________________________________
This section provides an overview of the Sun Fire and Sun Blade system architectures that house UltraSPARC
III technology, with the exception of the Sun Blade 100 workstation, which uses an UltraSPARC IIe processor.
Emphasis will be put on changes from the Sun Ultra workstation design as examined in Section 2. The Sun
Fireplane interconnect bus replaces the UPA Interconnect Bar Switch. The UltraSPARC III processor replaces
the UltraSPARC II processor. FC-AL, USB, IEEE 1394, and Gigabit Ethernet (on servers) are now standard. The
two key new components, the new Host PCI Bridge and new PCI I/O Bridge, are explained in detail in the next
two sections.
3.1 Sun Fireplane Interconnect
file:///C|/Documents%20and%20Settings/Administrat...op/5_1_latest_whitepaper/sunblade.whitepaper.html (10 of 67) [5/7/2002 2:19:56 PM]

Figure 3-1. Typical Sun Fire PCI Bus Architecture Implementation
Sun Fire and Sun Blade systems are built around the Sun Fireplane interconnect bus (see Figure 3-1),
interconnecting the new UltraSPARC III processors, memory, and I/O subsystem.
One of the major architectural innovations of the Fireplane interconnect bus is the ability to combine the
advantages of a switch-based interconnect, similar to the UPA bus, with the advantages of a single bus. A switch-
based architecture typically relies on a centralized controller, which has proven to be a limitation in achieving
higher bandwidths. In moving away from the centralized system controller, the Fireplane interconnect bus
pushed the envelope by making the data path independent from the address path. The order of data transfers
can be different from the issuing order of the requests. The Fireplane bus also minimizes latency by
implementing special techniques for delivering data to the caches, and the UltraSPARC III provides a
sophisticated write-buffer with headroom so that even worst-case circumstances do not impact performance.
Key features of the Fireplane interconnect bus include:
❍ 150-MHz operating frequency for increased performance over previous designs
❍ Total throughput of up to 4.8 Gbyte/sec (two processor systems) and beyond
❍ Low-latency memory access
❍ Separate address/control and data paths for flexible implementation
❍ Out-of-order transaction processing allowing multiple "in-flight" transactions on the bus at one time
❍ High-throughput paths to memory clocked at 150 MHz (576-bit wide paths, including ECC) with
integrated support for multiprocessor configurations
3.2 Host PCI Bridge

The Host PCI Bridge (HPB) is the primary connection between the Sun Fireplane interconnect bus and the PCI
I/O subsystem. The chip provides PCI support for both 32-bit and 64-bit as well as 33-MHz and 66-MHz device
connection. It also supports FC-AL for host and disk connection and Gigabit Ethernet for 1000-Mbit/sec Ethernet
connection. The HPB is explained in detail in Section 4.
3.3 PCI I/O Bridge

The PCI I/O Bridge (PIB) is a low-cost integrated I/O chip that provides core I/O functionality for the Sun Fire and
Sun Blade hardware systems. The chip provides support for both USB and IEEE-1394 serial bus. Both USB and
IEEE-1394 are popular peripheral bus architectures in the consumer market. USB is a hot-plug master slave
technology used to connect personal I/O peripherals to a workstation or server. USB supports 1.5-Mbit/sec and
12-Mbit/sec peripherals. Popular examples are USB keyboard, mouse, printer, removable storage, and audio.
IEEE-1394 serial bus, also found in Apple FireWire® technology or Sony i.LINK products, is a hot-plug serial
peer-to-peer bus that supports speeds up to 4000 Mbit/sec. IEEE-1394 is popular in the digital video (DV) and
high-end removable storage market. The PIB is explained in detail in Section 5.

3.4 UltraSPARC III Processor

At the processor end, the Fireplane interconnect bus includes one or more UltraSPARC III processors. This
processor was designed from the ground up for the new generation of Sun systems. Key features of the
UltraSPARC III processor include:
❍ Clock speeds from 600 to 900 MHz with up to 8 Mbyte external cache and with a published
roadmap of versions to 1.5+ GHz
❍ Higher performance than UltraSPARC II - SPECint95>35 and SPECfp95>60
❍ Industry-leading processor-to-memory bandwidth of 2.4 Gbyte/sec
❍ Large memory bandwidth and multiprocessor scalability, providing significant advantages for high-
performance computing applications such as electronic commerce, scientific computation, and data
mining
❍ Multiprocessing support for over 1000 processors
❍ Scalable shared memory (SSM), allowing systems to quickly expand from a few processors to
hundreds of processors without rewriting applications or using extensive additional circuitry
❍ 64-bit, SPARC® v9 processor architecture; full SPARC v9 Total Store Order (TSO) compliance
❍ 100 percent binary compatibility for the Solaris[tm] Operating Environment and application software
❍ 100 percent binary compatibility with previous versions of SPARC systems
❍ VIS instruction set for networking, media, imaging, and performance acceleration for Java[tm]
technology, enabling a two to three times performance boost, without recompilation, compared with
the UltraSPARC II
❍ Able to deliver 6 billion operations per second, pushing the limits of networking and media
acceleration
❍ Targeted to high-performance applications such as computer-aided design, computer-aided

manufacturing, electronic design automation, and scientific modeling applications
For more information, please refer to the UltraSPARC III Cu Processor page on sun.com.
3.5 FC-AL
Fibre Channel Arbitrated Loop is an industry-standard interface adopted by the American National Standards
Institute. It is usually thought of as a system-to-system or system-to-subsystem interconnection architecture. This
architecture provides high-performance links between computing nodes. The Fibre Channel loop topology was
developed to provide a low cost interconnection methodology for cost sensitive devices such as disk drives. The
Fibre Channel loop can have any combination of hosts and disks up to a maximum of 126 devices. For more

information on the FC-AL, please refer to the technical brief, "Fibre Channel Technology from Sun Microsystems"
[9].
3.6 Ethernet
Workstations still provide 10/100-Mbit/sec Ethernet as the primary LAN interface. In addition to 10/100-Mbit/sec
Ethernet, servers make Gigabit Ethernet available on the motherboard. For more information on Gigabit Ethernet
refer to "Gigabit Ethernet, Accelerating the Standard for Speed White Paper" [7].
4. Host PCI Bridge ASIC

________________________________________________________________________
The Host PCI Bridge (HPB) has been redesigned to interface with the Fireplane interconnect architecture. The
HPB is the primary connection between the Sun Fireplane interconnect bus and the PCI I/O subsystem.
Features in the chip include:
❍ Fireplane interface for full master and slave port connection to the high-speed Fireplane
interconnect bus offers a maximum data throughput of 1.0 Gbyte/sec.
❍ Ultra Port Architecture (UPA) interface for high-end video graphics, compliant with UPA
Specification rev 1.1; it provides support for two slave devices, and sustainable write data
throughput of 800 Mb/sec.
❍ Two independent PCI leafs: each provides an Interface Controller (IFC), a single PCI bus module
(PBM) with full master and slave support, PCI Specification Rev. 2.1 compatibility, 5V or 3.3V
signaling, and 64-bit with 32-bit device support.
❍ Each PCI leaf provides a 16-entry streaming cache (STC) for accelerating some kinds of PCI DVMA
activity.
❍ Each PCI leaf provides an I/O Memory Management Unit (IOMMU) with 16-entry TLB for mapping
DVMA addresses.
❍ Each PCI leaf provides an Interrupt Controller or "Mondo-Vector" Dispatch Unit (MDU) for delivering
interrupt requests from up to six external slots to a CPU module.
Figure 4-1 provides a conceptual block diagram of the HPB.

Figure 4-1. HPB Conceptual Block Diagram
4.1 PCI Bus Module

The PCI Bus Module (PBM) is the block directly responsible for operating on the PCI bus as a master and slave
device. It operates as a host PCI bridge and provides a number of central resources for the PCI bus that it
controls. There is one in each of the two PCI leafs, used to interface to the two independent PCI bus segments.
Internally, each PBM is identical: each is 64 bits wide and can support a 33-MHz or 66-MHz PCI bus, with up to
six external devices. Externally, not all capabilities of the PBM are available. In particular, PCI bus B is 64 bits
wide and supports 33/66-MHz operation with up to four master devices (only two at 66 MHz for electrical loading
reasons), while PCI bus A is 64 bits wide and supports only 33-MHz operation, but allows six master devices.
The PBM's main features are:
❍ Compliant with PCI Local Bus Specification, Revision 2.1.
❍ Supports 33-MHz and 66-MHz PCI operation, 64-bit DMA and PIO data operations, and 64-bit data
addressing as a target.
❍ As a PCI central resource, provides arbitration for up to six master devices and a mechanism for
generating PCI configuration cycles and PCI special cycles.
❍ Operates with a PCI clock rate of either 1x or 0.5x the internal clock rate.
❍ Internal interfaces to the IOMMU, STC, IFC, and MDU.

❍ Dual 64-byte DMA write buffers, quad 64-byte DMA read buffer, a dual 64-byte PIO write buffer, and
a dual 64-byte PIO read buffer.
❍ Little-endian access to the bus and to internal configuration space.
The PCI leaf's main internal data buses (which are big-endian) are connected to the PBM (a little-endian module)
in a "byte-twisted" fashion, where the logical byte lanes are connected together. For byte 0, the PCI leaf's data
bits [63:56] are connected to the PBM data bits [7:0], and so on. The PBM internal control registers (which are
big-endian) are byte-twisted again internally. Figure 4-2 shows HPB byte twisting.
Figure 4-2. HPB Byte Twisting
4.1.1 Supported PCI Features

❍ 64-bit PCI bus extensions
❍ 64-bit addressing (dual address cycle [DAC]), used for DMA bypass of IOMMU
❍ Required adapter and host-bridge configuration space header registers
❍ Fast back-to-back cycles as a target
❍ Arbitrary byte enables (consistent mode only)

❍ Ability to generate memory, I/O, and configuration read and write cycles
❍ Ability to generate special cycles
❍ Response only to memory space accesses
❍ Peer-to-peer DMA on the same PCI bus
4.1.2 Unsupported PCI Features

❍ Exclusive access to main memory (LOCK)
❍ Peer-to-peer DMA between bus segments
❍ PCI cache support
❍ External arbiter
❍ Cache-line wrap addressing mode
❍ Fast back-to-back cycles as a PIO master
❍ Address/data stepping
❍ Subtractive decode
❍ Any DOS compatibility features
4.1.3 Supported PCI Transactions

The PBM is leveraged from the previous design with a few extensive changes. There is some additional buffering
built in, larger than 64-byte bursts are supported in streaming mode, the 64-bit PCI data bus is utilized for block
PIOs, and support for PCI Delayed Read Transactions in consistent mode is added. The PBM does not respond
to any configuration read or write cycles on the PCI bus. Neither does the PBM respond to any PCI I/O space
transactions (IO Read/Write command types). It is assumed that the PBM is the only device generating
configuration cycles on the PCI bus. As a target on the PCI bus, the PBM only responds to PCI memory space
commands. Table 4-1 summarizes the PCI commands the PBM is able to generate, as well as its response as a
target.

Command/Byte Generated in
PCI Command Type Response in Target Mode
Enable (C/BE#) Master Mode
Interrupt Acknowledge 0000 No Ignored
Special Cycle 0001 Yes Ignored
I/O Read 0010 Yes Ignored
I/O Write 0011 Yes Ignored
Reserved 0100 No Ignored
Memory Read 0110 Yes Perform DMA Read access
Memory Write 0111 Yes Perform DMA Write access
Configuration Read 1010 Yes Ignored
Configuration Write 1011 Yes Ignored
Memory Read Multiple 1100 No Perform DMA Read, enables prefetch

if streamable
Dual Address Cycle 1101 No Bypass DMA access
Memory Read Line 1110 No Perform DMA Read, enables prefetch

if streamable
Memory Write and Invalidate 1111 No Equivalent to Memory Write

Command

Table 4-1. Supported PCI Transactions
The PBM detects parity errors during transactions with the PCI bus. PCI receive data parity errors can be
detected during PIO reads from a PCI device and DMA writes by a PCI device. PCI send data parity errors can
be detected during PIO writes to a PCI device and DMA reads by a PCI device. PCI Parity errors are not
detected during wait states.
4.2 Streaming Cache

The Streaming Cache (STC), implemented in the HPB, is a small-sized, fully associated cache managed by both
hardware and software, which accelerates certain PCI bus DVMA to and from system memory. Each PCI leaf
within the HPB contains an STC module. The streaming cache performs three primary functions:
1. Accumulate sequentially addressed PCI write bursts into quantities the size of a system block.
2. Speculatively prefetch the next sequential block of memory for an active PCI read stream.
3. Act as a local cache for PCI read accesses to the same block.
The STC block is leveraged from the previous design with some extensive changes. There is a second data
block in each STC entry, providing a context flush capability, and a change to the operating policy so that
"infinite" (page-sized) bursts are allowed. The implementation of the STC features:
● A fully associative pool of 16 entries shared among read and write streams
● Two 64-byte blocks of data per entry providing for the transfer of large (page-sized) bursts
● Dual-ported data RAM for concurrent write/flush and read/fill operations
● 64-bit-wide interface to both the PBM and IFC modules
● Least Recently Used (LRU) allocation scheme
● Virtual address tags for low lookup latency
● Physical address page translation for each entry to reduce fush and prefetch latencies
● One entry allotment per virtual page to reduce the problem of individual misbehaved devices from
thrashing the cache
● Individual byte write enabling support for PCI bus byte granularity
● Multiple cache line entries per page
Only accesses to virtual pages that are designated by software as streamable pages can use the streaming
cache's functions. The STC does not, however, participate in the system cache coherency protocol, and the data
in the cache is out of the coherency domain, so software intervention is required to ensure a consistent memory

image when transfers are complete. The order of commands, however, remains the same and does not require
software intervention; reads followed by writes and vice versa will see the correct results. Device drivers call
ddi_dma_sync(9F) to write to the stream cache registers after the DMA transfer is completed.
Extracting the most performance out of the streaming cache involves following several guidelines concerning
DMA accesses. Not conforming to these guidelines results in less than ideal observed performance. Some
access patterns may, in fact, incur a penalty that causes the streaming cache to yield poorer performance than
equivalent non-streamable accesses.
❍ DMA writes to a block within a streamable virtual page should always access memory in increasing
total sequential order (no gaps). Failure to do so will cause unnecessary flushing of the cache entry
(and byte holes within a single DMA write will cause errors). Better performance is exhibited when
writing large-sized bursts (6 bytes or larger). While writes within a block should be increasing and
sequential, no hardware-imposed performance impact is made in regard to accesses across blocks
or pages.
❍ DMA reads from a block within a streamable virtual page should access memory in increasing
sequential order, and should use the appropriate PCI bus commands based on the amount of data
to be read. Failure to do so will cause unnecessary prefetching. Reads across blocks should also be
in increasing and sequential order. Failure to do so will waste the prefetching, reducing the
maximum read bandwidth to no more than non-streaming reads. Prefetches are not launched if they
would cross a page boundary. Larger burst sizes will again improve performance.
❍ Since there is only one entry allotted per virtual page, multiple devices should not interleave their
accesses to the same virtual page. If it is desired to have multiple devices accessing non-
overlapping portions of the same page, aliasing should be used to map different virtual pages to the
same physical page.
❍ As a general rule, larger burst sizes (64 bytes or larger) will improve performance.
Byte holes within a single PCI write data stream (byte enable bit(s) is off while byte enable(s) to the left and right
are on), and zero byte writes are defined to be an error condition if the page is marked streamable. The PBM
detects these conditions, sets a status bit, and signals an interrupt. The transaction continues, however, and the
streaming cache behaves as if the byte holes didn't exist (it may overwrite data in memory that it shouldn't).
4.3 PCI IOMMU

The I/O Memory Management Unit (IOMMU) performs virtual-to-physical address translation during DVMA
cycles initiated by PCI masters. PCI master devices provide a 32-bit virtual address at the beginning of a DVMA
transfer. The IOMMU translates the virtual address into a 32-bit physical address plus an identifier for cacheable
vs. non-cacheable address space. There is a separate PCI IOMMU in each PCI leaf, which is responsible for
translating virtual addresses only from devices controlled by that PCI leaf. The IOMMU is leveraged from the
previous design with mostly minor changes (41-bit to 43-bit physical address) and one major change, which is
the addition of the context flush capability. The IOMMU consists of a 16-entry, fully associative Translation
Lookaside Buffer (TLB) implemented in hardware to cache recently used translations and a Translation Storage
Buffer (TSB), which is a software managed data structure (translation table) in memory. It contains one-level
mapping information for the virtual pages. IOMMU Hardware performs TSB lookup when the translation cannot
be found in the TLB. A TSB entry is called a Translation Table Entry (TTE) and takes 8 bytes. Supported TSB
table sizes include 1nK entries where n = 1 through 8 and is typically set to 8 Kbyte. The IOMMU supports two

different page sizes, 8 Kbyte and 64 Kbyte. Mixed page sizes can be used in the system, but the TSB table
lookup only assumes the smaller page size of 8 Kbyte. No overlapping of pages is allowed. This gives support
for DVMA address space of 8 Mbyte to 1 Gbyte for an 8-Kbyte page and 64 Kbyte to 2 Gbyte for a 64-Kbyte
page (DVMA space = page size * TSB table size). A DVMA space larger than 2 Gbyte is not supported, which
means that 64-Kbyte and 128-Kbyte TSB sizes are not supported with a 64-Kbyte page size. Software must set
up TSB before it allows translation to start. The TSB can be accessed in the IOMMU control register.
The PCI IOMMU can operate in three different modes: translation, bypass, and pass-through. Its operating mode
is determined by the values in IOMMU Control Register, the PCI addressing mode used (32 bits vs. 64 bits), and
values in the PCI virtual address. Translation is initiated by the PBM block by providing a 32-bit virtual address.
The IOMMU hardware performs a TLB lookup first. If the lookup results in TLB hit, the IOMMU returns a 43-bit
physical address to the PBM block. If a TLB miss happens, the IOMMU waits for the PBM to request a TSB
lookup. If the TSB locates a valid mapping for the virtual page, information in the TSB entry will be loaded into
TLB and translation continues. If the TSB lookup results in a miss, an error will be returned to the PBM. The
translation of a virtual address to a physical address is illustrated in Figure 4-3.
Figure 4-3. Virtual-to-Physical Address Translation
The implementation of the PCI IOMMU allows PCI devices to have their own MMU and bypass the IOMMU that
is in the HPB. A PCI device operating in bypass mode has direct access to the entire physical space of the
Fireplane interconnect bus. Pass-through mode allows access to the 2 Gbyte of memory address space. A
DVMA access in pass-through mode will always be cacheable to space.
4.4 Mondo Dispatch Unit

The Mondo Dispatch Unit (MDU) is responsible for fielding interrupts from external and internal HPB sources,
then building and sending the appropriate interrupt packet to the appropriate CPU via the Fireplane interface
block. External interrupt sources include interrupts from up to 8 PCI slots (four separate interrupts each) and up
to 16 interrupts from on-board I/O devices. These interrupts are concentrated externally to the HPB and are
presented to the Mondo Units one at a time. Internal interrupt sources include ECC errors signaled by the
Fireplane block, PCI bus errors signaled by the PBM, and the Timer/Counter interrupts. There is one MDU in
each PCI leaf within the HPB.
The MDU block is leveraged from the previous design with some substantial changes, the main one being a
reduction in outstanding interrupts from two to one (since the HPB has two copies of the MDU block). The Mondo
interrupt transfer mechanism proposed for future Sun systems is designed to reduce interrupt service overhead
through the use of processor and system-based supports. On the processor side, SPARC V9 CPUs provide a
dedicated set of registers to be used exclusively for servicing interrupts. This eliminates the need for the
processor to save its current register set to service an interrupt, and then restore it later. On the system side,
requests for interrupt service are converted into interrupt request packets, which are sent over the memory
interconnect to the processor. An interrupt packet contains a Mondo Vector, which contains data designed to
assist the processor in servicing the interrupt. Limitations of the Mondo Vector approach include:
❍ Only one interrupt request packet can be serviced at a time.
❍ There is no priority level associated with Mondo Vector interrupts; they are serviced on first-come,
first-served basis.
❍ Flow control must be done at the interconnect level to prevent loss of interrupt packets.
4.5 Interface Controller

The Interface Controller (IFC) is responsible for controlling all the transactions between the other blocks in the
PCI leaf and the asynchronous packet FIFOs to and from the Fireplane interface. This is almost entirely new
logic for the HPB. There is one IFC in each PCI leaf within the HPB. Responsibilities include:
❍ Receiving, decoding, and issuing PIO requests.
❍ Returning PIO read data and PIO write completion responses.
❍ Arbitrating among DMA requests.
❍ Forwarding DMA and interrupt requests, and keeping track of outstanding requests.
❍ Forwarding DMA write data.
❍ Receiving DMA read replies and routing them to the appropriate block.
❍ Tracking DMA write replies for retiring completed transactions and proper flow control.
❍ Forwarding interrupt replies (acks/nacks) to the MDU block.
❍ Handling PIO reads and writes, DMA reads and writes, MDU interrupts, acks and nacks.

4.6 Summary of PCI Characteristics

A summary of the PCI characteristics of the HPB is presented in Table 4-2, and new PCI features are indicated
in bold.
Characteristics PCI Bus A PCI Bus B
PCI Spec compliance Rev 2.1 Rev 2.1
Data Transfer Width 64-bit 64-bit
Clock Frequency 33 MHz capable 33/66MHz capable
# of external master devices 6 2 @ 66MHz / 4 @ 33MHz
Burst Size 128 bytes + 128 bytes +
Number of Read Buffers quad 64-byte for DMA quad 64-byte for DMA
dual 64-byte for PIO dual 64-byte for PIO
Number of Write Buffers dual 64-byte for DMA dual 64-byte for DMA
dual 64-byte for PIO dual 64-byte for PIO
Dual Address Cycles Bypass DMA only Bypass DMA only
Fast Back-to-Back Device In target mode only In target mode only
Endianness: Byte Twisting Enabled for both PIO and DMA Enabled for both PIO and DMA
transfers transfers
Interrupts Up to 8 PCI slots, each with 4 Up to 8 PCI slots, each with 4

interrupt lines interrupt lines
Cache Line Size 64-byte 64-byte

Number of Cache Lines 16 16
Disconnect End of cache line and/or page End of cache line and/or page
boundary for consistent mode; end boundary for consistent mode; end
of streaming cache on streaming of streaming cache on streaming
mode (2 x 64 byte) mode (2 x 64 byte)
Configuration Space 32 Mbyte 32 Mbyte
I/O Space 2 Gbyte 2 Gbyte
Memory Space 2 - 4 Gbyte 2 - 4 Gbyte
Configuration Cycles Master mode only Master mode only
Special Cycle Master mode only Master mode only
Arbitrary Byte Enables Consistent DMA only Consistent DMA only
Peer-to-Peer DMA On same PCI segment only On same PCI segment only
IOMMU Page Size 8 Kbyte and 64 Kbyte 8 Kbyte and 64 Kbyte
PIO Read Size Up to 16 bytes with arbitrary Up to 16 bytes with arbitrary

bytemasks or up to 64 bytes bytemasks or up to 64 bytes
without a bytemask without a bytemask
Up to 4 bytes for writes to I/O or Up to 4 bytes for writes to I/O or

Configuration space Configuration space
PIO Write Size Up to 16 bytes with arbitrary Up to 16 bytes with arbitrary

bytemasks or up to 64 byte without bytemasks or up to 64 byte without
a bytemask. a bytemask.
Up to 4 bytes for writes to I/O or Up to 4 bytes for writes to I/O or

Configuration space. Configuration space.
Cache Line Wrap Addressing Mode Not supported. Executed as one Not supported. Executed as one
data word at a time. data word at a time.

Local (on-PCI) Cache Not supported Not supported
Exclusive Access to Main Memory Not supported. Lock# signal is not Not supported. Lock# signal is not
connected. connected.
Address/Data Stepping Not supported Not supported
DOS Compatibility Hole Not supported Not supported
External Arbiter Not supported Not supported
Subtractive Decode Not supported Not supported
Table 4-2. PCI Characteristics Summary
5. PCI I/O Bridge ASIC

________________________________________________________________________
The PCI I/O Bridge (PIB) is a chipset comprising an I/O chip and a single-chip Ethernet transceiver. Figure 5-1
shows the major component blocks of the PIB. The PIB is built around an internal bus, the Channel Engine
Interface, which provides the key to its modularity. Above the Channel Engine Interface, the Bus Adapter
connects to the PCI bus. The four identical ports on the Channel Engine Interface are used for each of the PIB's
functional units: Ethernet, USB, 1394, and EBus. Each of these has its own set of control and status registers,
data buffers, and the core logic function. The PIB is a fully scannable design and provides power management
functions for the system. Ethernet transceiver integrates PCS, PMD, and PMA functions and some filtering
functions. The PIB is compliant to the JTAG 1149.1 test architecture.

Figure 5-1. PIB Conceptual Block Diagram
5.1 Bus Adapter

The Bus Adapter provides the layer between the bus-independent Channel Engine Interface and the PCI Local
Bus. The PCI bus interface is 64 bit and 33 MHz, fully compliant with the PCI Local Bus Specification, Revision
2.1. As a slave, it buffers write transactions internally. As a master, it is capable of 128-byte (16-word) bursts.
DMA writes are buffered in the Bus Adapter to support back transactions. The Bus Adapter also contains the PCI
bus Configuration Space. RIO presents itself to PCI as a multifunction device: EBus, Ethernet, 1394, and USB.
Each function has its own area in the configuration space. The bus adapter has two PCI request/grant pairs to
improve DMA latency for RIO on the PCI bus. A summary of its main features follows:
● Single time domain operating at the PCI Bus frequency
● PCI local Bus Revision 2.1 compatibility, 64-bit
● Full master and slave capabilities

● 256-byte bursts as initiator
● Two PCI request/grant pairs for better DMA latency
● Multifunction configuration space with independent address decoders for each function
● Dual-buffered DMA WRITE path
● Five Channel Engine Interface ports, 32/64-bit wide: EBus, 1394, USB, Ethernet transmit and Ethernet
receive
● Interrupt router for PCI Bus add-in card or motherboard modes
Like most other PCI Bus devices, the PIB conforms to the little-endian byte ordering. The channel engines within
the PIB are also little-endian. The PIB supports byte stacking when accessing the PROM and other devices on
the EBus. The 32-bit word assembled from successive reads of the 8-bit PROM EBus is returned in little-endian
format. This is still compatible with CPU code fetches, which are always big-endian, since the HPB incorporates
a fixed-byte lane alignment, converting the data in this case back to big-endian.
5.2 Channel Engine Interface

The Channel Engine Interface (CEI) is a modular, bus-independent interconnect intended to serve as the
backbone of the PIB design. Signals in the CEI are shared, going from the Bus Adapter to all of the Channel
Engines, or dedicated, going from one Channel Engine to the Bus Adapter, or vice versa. The CEI was designed
to be:
● Modular.
● Extensible (number of master and slave ports).
● Bus-independent and free of external timing constraints.
5.3 1394 Channel Engine

The implementation of the 1394 Channel interface provides the ability to attach 1394 devices capable of
transmitting and/or receiving asynchronous and/or isochronous data types. Asynchronous data transfer is
explicitly acknowledged and is used for applications that are not time critical but require guaranteed data
delivery. Isochronous data transfer is required in applications that need to have "on time" but not guaranteed
delivery such as real-time video transfer. The 1394 channel engine is capable of 100-, 200-, and 400- Mbit/sec
transfers. The engine contains 6 DMA channels: four for isochronous and two for asynchronous transfers. The
DMA engine implements the industry-standard DBDMA scheme. The design is based on Apple's core with
modifications for OpenHCI interface for 1394. The channel engine contains 1.5 Kbytes of internal buffering. The
engine is modularized to give isochronous channels priority over asynchronous channels. The PIB implements
the link portion of the IEEE 1394 standard.
5.4 USB Channel Engine

USB is an industry-standard, low-cost serial bus intended for 'slower' peripherals such as keyboards and mice. In
addition to supporting traditional asynchronous technology appropriately, the USB has provisions for supporting
isochronous devices as well. As such, it can be used for computer telephony integration (CTI) applications. The
USB has two speed modes of operation, 1.5 Mbit/sec and 12 Mbit/sec. The slow speed is intended for cost-
sensitive applications. The USB channel engine provides 1 DMA engine and 1 Kbyte of total internal buffering.
The USB interface provides the host controller for USB transfers and a four-port integrated hub. The USB host
controller manages the control flow and data flow. It also provides connection management and provides status
information. The hub enables tiered star topology and provides multiple connections. The USB interface also
provides auto resume from power-managed (suspended) state. The USB host controller is OpenHCI compliant.
There are some additional features provided for performance enhancement and making the core more palatable
to the Solaris OE.
5.5 EBus Channel Engine

The EBus Channel Engine interfaces standard off-the-shelf ISA devices to the PIB. This allows the ability to put
traditional Intel-style peripherals in a SPARC processor-based system with a minimal amount of glue logic. Up to
eight single- or multifunction slave devices and four DMA devices can be hooked to EBus. EBus has four DMA
engines, and each DMA engine has an internal 128-byte buffer. EBus in RIO has the following key differences
from the EBus interface provided by the previous-generation I/O chip in the Sun Ultra systems:
● Hooks to retry transactions on channel engine interface for PCI 2.1 compliance.
● Multiple posted write buffers.
A summary of the features supported by the EBus Channel Engine follows:
● Supports PCI 2.1 delayed read transactions.
● Capable of supporting EPROM, TOD/NVRAM, Audio CODEC, superIO, external serial ports and generic
Intel-style (ISA) slave and slave DMA devices.
● 8 chip selects.
● Programmable cycle time control for slave and DMA accesses.
● Posted write buffer supports up to four transactions of 8, 16, or 32 bits.
● Supports byte stacking for word and half-word slave cycles to EBus devices.
● Four DMA controllers.
● Chained or unchained mode of operation for each DMA controller.
● Programmable transfer size for each DMA controller.
● 128-byte buffer for each DMA controller.
● Programmable DMA priority algorithms.

● Multimaster capable.
5.6 Ethernet Channel Engine

The Ethernet Channel Engine provides a buffered full duplex DMA engine and a media access control function
based on a standard MAC. The descriptor-based DMA engine contains independent transmit and receive
channels, each with 2048 bytes of on-chip buffering. The MAC provides a 10- and 100-Mbit/sec CSMA/CD
protocol-based network interface conforming to IEEE 802.3 and IEEE 802.30. The MAC also provides full duplex
and flow control support as defined by the IEEE 802.3x Ethernet standard.
6. OpenBoot[tm] Firmware
________________________________________________________________________
Since 1989, Sun system firmware has been based upon IEEE 1275, a flexible and extensible boot PROM
architecture. OpenBoot system firmware is Sun's implementation of IEEE 1275. Open Firmware is the name
used for all firmware that complies with IEEE 1275. IEEE 1275 is a very powerful boot architecture: it supports
loading and executing of programs, including the operating system, from disks, tapes, and network devices,
providing complete boot flexibility. Its bus-independent method for identifying expansion devices and add-on
cards is extensible to the complex bus topologies possible with PCI and other new I/O technologies. Before
dissecting both a new workstation and new server, taking a look at the UltraSPARC III system architecture from
a firmware point of view, it is important to provide a little more background information about IEEE 1275.
6.1 Device Properties

OpenBoot firmware specifies a data structure known as the device tree to describe the set of devices attached to
the system. The device tree is a hierarchical data structure that describes the system hardware and its
associated configuration information and support routines. The device tree is also the main interface between
OpenBoot firmware and the operating system. Devices are attached to a system on a set of interconnected
buses and their attached devices as a tree of nodes. The root of the device tree is a representation of the
machine's main physical bus, the Fireplane interconnect in this case.
Each device node may have properties, methods, and data. Properties describe the characteristics of a
hardware device. Properties are externally visible to both OpenBoot procedures and client programs, that is, the
Solaris Operating Environment. Methods are named software procedures that control hardware devices or
provide other services. Data is used by methods to maintain internal information. Unlike properties, which can be
accessed by external software, data cannot be directly accessed from outside the device node package. As an
example, the device tree of a Sun Blade 1000 workstation is shown in Figure 6-1 and the properties of PCI bus B
in Figure 6-2.
{1} ok banner
Sun Blade 1000 (2 X UltraSPARC-III), No Keyboard
OpenBoot 4.0, 4096 MB memory installed, Serial #16458796.
Ethernet address 8:0:20:fb:24:2c, Host ID: 80fb242c.

{0} ok show-devs
/ppm@8,410050
/upa@8,480000 -> HPB-UPA
/pci@8,600000 -> HPB-PCI 64bit/66MHz
/pci@8,700000 -> HPB-PCI 32bit/33MHz
/memory-controller@1,400000
/SUNW,UltraSPARC-III@1,0 -> Processor 1 (P1)
/memory-controller@0,400000 -> Memory Controller on
Processor 0
/SUNW,UltraSPARC-III@0,0 -> Processor 0 (P0)
/virtual-memory
/memory@m0,0
/aliases
/options
/openprom
/chosen
/packages
/upa@8,480000/SUNW,ffb@0,0 -> Framebuffer on UPA slot
/pci@8,600000/TECH-SOURCE,gfxp@1
/pci@8,600000/SUNW,qlc@4 -> FC-AL controller
/pci@8,600000/SUNW,qlc@4/fp@0,0
/pci@8,600000/SUNW,qlc@4/fp@0,0/disk
/pci@8,700000/scsi@6,1
/pci@8,700000/scsi@6
/pci@8,700000/usb@5,3 -> USB part of PCI I/O Bridge
/pci@8,700000/firewire@5,2 -> IEEE-1394 on PCI I/O Bridge
/pci@8,700000/network@5,1 -> 10/100 Ethernet on PCI
I/O Bridge
/pci@8,700000/ebus@5 -> ebus on PCI I/O Bridge
/pci@8,700000/scsi@6,1/tape
/pci@8,700000/scsi@6,1/disk
/pci@8,700000/scsi@6/tape
/pci@8,700000/scsi@6/disk
/pci@8,700000/ebus@5/serial@1,400000
/pci@8,700000/ebus@5/parallel@1,300278
/pci@8,700000/ebus@5/floppy@1,3023f0
/pci@8,700000/ebus@5/pmc@1,300700

/pci@8,700000/ebus@5/gpio@1,300600
/pci@8,700000/ebus@5/rtc@1,300070
/pci@8,700000/ebus@5/audio@1,200000
/pci@8,700000/ebus@5/beep@1,32
/pci@8,700000/ebus@5/i2c@1,30
/pci@8,700000/ebus@5/i2c@1,2e
/pci@8,700000/ebus@5/ppm@1,e
/pci@8,700000/ebus@5/bbc@1,0
/pci@8,700000/ebus@5/flashprom@0,0
/pci@8,700000/ebus@5/i2c@1,30/i2c-bridge@0,60
/pci@8,700000/ebus@5/i2c@1,30/motherboard-fru@0,a8
/pci@8,700000/ebus@5/i2c@1,30/card-reader@0,40
/pci@8,700000/ebus@5/i2c@1,30/fan-control@0,48
/pci@8,700000/ebus@5/i2c@1,30/temperature@0,98
/pci@8,700000/ebus@5/i2c@1,30/cpu-fru@0,a2
/pci@8,700000/ebus@5/i2c@1,30/cpu-fru@0,a0
/pci@8,700000/ebus@5/i2c@1,2e/idprom@0,a0
/pci@8,700000/ebus@5/i2c@1,2e/nvram@0,a0
/pci@8,700000/ebus@5/i2c@1,2e/dimm-fru@1,ac
/pci@8,700000/ebus@5/i2c@1,2e/dimm-fru@1,a8
/openprom/client-services
/packages/ufs-file-system
/packages/kbd-translator
/packages/dropins
/packages/obp-tftp
/packages/terminal-emulator
/packages/disk-label
/packages/deblocker
/packages/SUNW,builtin-drivers
{0} ok
Figure 6-1. Sun Blade 1000 Device Tree

{0} ok dev /pci@8,700000

{0} ok .properties
available 81000000 00000000 00000500 00000000
0000fb00 82000000 00000000 0012c000
00000000 002d4000 82000000 00000000
02000000 00000000 7a000000 00006176
reg 00000400 04700000 00000000 00018000
00000400 04410000 00000000 00000050
000007ff ee000000 00000000 00000100
ranges 00000000 00000000 00000000 000007ff ee000000 00000000 01000000
01000000 00000000 00000000 000007ff ef000000 00000000 01000000
02000000 00000000 00000000 000007fe 00000000 00000001 00000000
03000000 00000000 00000000 000007fe 00000000 00000001 00000000
ino-bitmap f3fff000
001fecff
interrupt-map 00000800 00000000 00000000 00000001 f006ca40 0000000c
00000800 00000000 00000000 00000002 f006ca40 0000000d
00000800 00000000 00000000 00000003 f006ca40 0000000e
00000800 00000000 00000000 00000004 f006ca40 0000000f
00001000 00000000 00000000 00000001 f006ca40 00000010
00001000 00000000 00000000 00000002 f006ca40 00000011
00001000 00000000 00000000 00000003 f006ca40 00000012
00001000 00000000 00000000 00000004 f006ca40 00000013
00001800 00000000 00000000 00000001 f006ca40 00000014
00001800 00000000 00000000 00000002 f006ca40 00000015
00001800 00000000 00000000 00000003 f006ca40 00000016
00001800 00000000 00000000 00000004 f006ca40 00000017
00002800 00000000 00000000 00000002 f006ca40 0000001d
00002800 00000000 00000000 00000003 f006ca40 0000001e
00002800 00000000 00000000 00000004 f006ca40 0000001f
00003000 00000000 00000000 00000001 f006ca40 00000018
00003000 00000000 00000000 00000002 f006ca40 00000019

interrupt-map-mask 00fff800 00000000 00000000 00000007

#interrupt-cells 00000001
slot-names 00 00 00 1c 50 43 49 20 34 00 50 43 49 20 33 00
interrupts 00000033
00000030
00000031
00000034
00000026
clock-frequency 01f78a40
bus-range 00000000
00000000
bus-parity-generated
no-probe-list 0
compatible pci108e,8001
name pci
device_type pci
#address-cells 00000003
#size-cells 00000002
implementation# 00 00 00 2a
version# 00 00 00 03
portid 00 00 00 08
{0} ok
Figure 6-2. PCI Bus B Properties
6.2 IEEE 1275 PCI Binding

PCI Bus binding to IEEE Standard 1275: 1994 Standard for Boot (Initialization Configuration) Firmware, Rev. 2.1,
augments the standard boot firmware to specify the application of Open Firmware to PCO Local Bus
Specification, Rev. 2.1, June 1, 1995. It defines a set of properties specifically for PCI devices and specifies the
address formats and representations for these properties.
6.2.1 Address Formats and Representations

The numerical representation of the PCI address and the size for PCI devices as defined in PCI Bus Binding to
IEEE Standard 1275: 1994 Standard for Boot (Initialization Configuration) Firmware, Rev. 2.1, consists of three
cells and two cells, respectively. Applying the address formats to the reg property allows device drivers to
retrieve device information such as:
❍ Type of address space (I/O, memory, or configuration)
❍ Bus number
❍ Device number (slot number)
❍ Function number
❍ Register number
Figure 6-3 shows address and size formats, while Table 6-1 shows address and size format codes.
Figure 6-3. Address and Size Formats
n is 0 if the address is relocatable
p is 1 if the addressable region is prefetchable
t is 1 if the address region is aliased

ss type code, denoting the address space

00 - configuration space (n,p,t must be 0)
01 - I/O space (p must be 0)
10 - 32-bit address memory space
11 - 64-bit address memory space
bbbbbbbbb The 8-bit bus number
ddddd 5-bit device number
fff 3-bit function number
rrrrrrrr 8-bit register number
hhhhhhhh 32-bit unsigned number
llllllll 32-bit unsigned number
Table 6-1. Address and Size Format Codes
The bus number (8 bits) of each PCI bus is assigned a unique number during system initialization, when the bus
controllers for the PCU buses within the PCI domain are located. This number is written into a register in the bus
controller of that PCI bus. There is often a close relationship between the device number (5 bits) and the IDSEL
device select line. Hence, the device number will indicate which slot the PCI device is plugged into. Figure 6-4
shows the properties of the PCI SCSI controller in a Sun Blade 1000 workstation.
{0} ok pwd
/pci@8,700000
{0} ok cd scsi@6
{0} ok .properties
assigned-addresses 81003010 00000000 00000300 00000000 00000100
82003014 00000000 00124000 00000000 00002000
82003018 00000000 00126000 00000000 00002000
device_type scsi-2
clock-frequency 02625a00
reg 00003000 00000000 00000000 00000000 00000000
01003010 00000000 00000000 00000000 00000100

02003014 00000000 00000000 00000000 00002000

02003018 00000000 00000000 00000000 00002000
name scsi
compatible 70 63 69 31 30 30 30 2c 66 2e 33 37 00 70 63 69
devsel-speed 00000001
class-code 00010000 -> hence the device name scsi
interrupts 00000001
latency-timer 00000040
cache-line-size 00000010
max-latency 00000040
min-grant 00000011
revision-id 00000037
device-id 0000000f
vendor-id 00001000
{0} ok
Figure 6-4. Sun Blade 1000 SCSI Controller Properties
Figure 6-5 focuses in on the reg property which has a physical address-size pair for each of the four registers.
The bytes are displayed in hexidecimal format (0x00) and must be converted into binary format (0b00000000) to
match the address and size formats. Each row indicates a different register and each column indicates a different
cell. The rows, from top to bottom, represent register 0 through 3. The columns, from left to right, represent
phys.hi, phys.mid, phys.low, size.hi, and size.low. The first byte (00) in row 1, column 1, indicates that Register 0
is relocatable, not prefetchable, not aliased and is in configuration space. Register 1, 2, and 3 are also in
configuration space so the address of each register is relative to the base register address. To get the absolute
address of each of these registers, drivers have to use the assigned-address property. The second and third byte
in each register's phys.hi address is 0x00 (0b00000000) and 0x30 (0b00110000), respectively, indicating that the
device is in slot number 6 (hence scsi@6) of bus number 0 and performs only one function, function number 0.
Register 1 is in I/O space and has size 0x100 as indicated in the size.low address. Registers 2 and 3 are in 32-
bit address memory space, and each has size 0x2000.
reg 00003000 00000000 00000000 00000000 00000000 (reg 0)

01003010 00000000 00000000 00000000 00000100 (reg 1)
02003014 00000000 00000000 00000000 00002000 (reg 2)
02003018 00000000 00000000 00000000 00002000 (reg 3)
physical address size
Figure 6-5. reg property of scsi@6

6.2.2 Child Node Properties

name
This property defines the device's name. In accordance with the "Generic Names Recommended Practice" [6], it
should represent the general nature of the device. For devices with FCode, the name is defined by the FCode
program. For devices without FCode, this property is derived from the PCI "class code," as in the name property
for class code 0C00xx is firewire. Table 1 in the "PCI Bus Binding: to IEEE Standard 1275-1994 Standard for
Boot (Initialization Configuration) Firmware, Revision 2.1" [4] lists the different class code -name property
combinations. If none apply, the bus node should generate a name of the form pcivvvv,dddd as described
next in the compatible property. For more information about the name property, see "PCI Bus Binding: to IEEE
Standard 1275-1994 Standard for Boot (Initialization Configuration) Firmware, Revision 2.1" [4].
compatible
This property defines devices with which the device is compatible and has the form pcivvvv,dddd. If the
SubsystemID field in the configuration registers for this device is nonzero, vvvv,dddd should be the
SubsystemVendorID and SubsystemID, respectively; otherwise vvvv,dddd should be the value of the
VendorID and DeviceID fields.
reg
This property is used by the device to export its memory to the operating system for mapping. For devices
without FCode, this property is generated by the bus node by reading the base address register in the
configuration space. For devices with FCode, this property is generated by the FCode of the device. This
property is mandatory for PCI child nodes. The property value consists of a sequence of physical-address, size
pairs. This is similar to the device node properties explained earlier. In the first such pair, the physical-address
component is the configuration space address of the beginning of the function's set of configuration registers,
and the size component is 0. Note that there is no particular relationship between the PCI base registers and the
reg property. A particular base register may or may not be represented in the reg property, and the reg
property may contain entries referring to addresses not controlled by any base register. If the n bit is 0, the
address is relative to the value of the associated base register, and device drivers should read the assigned-
addresses property to get the absolute address.
assigned addresses
Each entry in this property corresponds to either one (32-bit address memory space) or two (64-bit address
memory space) of the function's configuration space base address registers. If the n bit is 1, the address is
absolute (within the PCI domain address space).
interrupts
The value of this property represents the interrupt line to which this function's interrupt is connected. This
property is present if the Configuration Interrupt Pin Register is nonzero, and is absent otherwise.
fcode-rom-offset

This property indicates the offset of the PCI Expansion ROM image within the device's Expansion ROM in which
the FCode image was found. This value is generated before the FCode is evaluated. The following properties
represent the values of standard PCI configuration registers. They are created during the probing process after
the device node has been created by OpenBoot, but before evaluating the device's FCode (if any). The property
is absent if the value of the property is 0.
❍ vendor-id
❍ device-id
❍ revision-id
❍ callse-code
❍ min-grant
❍ max-latency
❍ devsel-speed
❍ cache-line-size
❍ fast-back-to-back
❍ subsystem-id
❍ subsystem-vendor-id
❍ 66mhz-capable
❍ udf-supported
It is important to note that Sun Fire systems use OBP 4.x commands. A number of PCI specific commands are
no longer available, as these commands did not scale very well in more complex system architectures using
multiple HPBs. Among them are show-pci-devs, show-pci-devs-all, show-pci-config, show-pci-configs, and probe-
pci. However, the same information can still be obtained using Forth. (Figure 6-6 provides an example of
information obtained using Forth.)
{0} ok pwd
/pci@8,700000
{0} ok 1000 sph
PCI Header for Bus # 0, Dev # 2, Func # 0

Nothing There
{0} ok

Figure 6-6. Information Using Forth
6.3 IEEE 1275 USB Binding

IEEE 1275 published its recommended practice for Universal Serial Bus in June 1998 in "Open Firmware,
Recommended Practice: Universal Serial Bus" [5]. Since there is no provision for FCode on USB devices, this
document describes only the nodes and their properties, representing the USB device tree. There are four types
of device tree nodes for USB: host controller nodes, device nodes, interface nodes, and combined nodes.
Typically, a device will be represented by the device node, while one or more child interface nodes represent the
individual USB interfaces on the device.
Current OBP implementation for USB devices does not provide for hot plugging. Removing the USB keyboard
when the system is at ok prompt will hang the system. If the USB keyboard is plugged in again, OBP does not
recognize it. This is particularly dangerous if the system is live and healthy and someone drops the system to
kadb/ok prompt and removes the keyboard. The only solution in that case is to power-cycle the system. As a
rule, no USB device should be hot-plugged when system is at ok prompt in OBP.
See Figure 6-7 for output relating to Sun Blade 1000, USB hub, Iomega USB Predator CDRW.
{1} ok cd usb@5,3
{1} ok .properties
assigned-addresses 82002b10 00000000 01000000 00000000 01000000
82002b30 00000000 00c00000 00000000 00400000
sunw,find-fcode f0 0b 02 c0
maximum-frame# 00 00 ff ff
reg 00002b00 00000000 00000000 00000000 00000000
02002b10 00000000 00000000 00000000 01000000
compatible 70 63 69 31 30 38 65 2c 31 31 30 33 2e 31 00 70
name usb
interrupts 00000004
fast-back-to-back
devsel-speed 00000001
class-code 000c0310
latency-timer 00000040
cache-line-size 00000010

max-latency 00000005
min-grant 0000000a
revision-id 00000001
device-id 00001103
vendor-id 0000108e -> Sun Microsystem PCI vendor ID
{1} ok cd usb@5,3
{0} ok ls
f00b6260 storage@4
f00b1098 hub@3
{1} ok cd hub@3 -> USB hub in USB port 3
{1} ok .properties
endpoints 0,1
compatible 75 73 62 33 65 62 2c 33 33 30 31 2e 31 30 30 00
name hub
reg 00000003 -> connected to USB port 3
assigned-address 00 00 00 02
{0} ok cd ..
{0} ok cd storage@4 -> USB storage device in port 4
{0} ok .properties
endpoints 0,1,2,3
compatible 75 73 62 35 39 62 2c 35 30 2e 31 30 30 00 75 73
name storage
reg 00000004 -> connected to USB port 4
0} ok
assigned-address 00 00 00 03
Figure 6-7. Sun Blade 1000, USB Hub, Iomega USB Predator CDRW
6.3.1 Host Controller Node Properties

Host controller nodes specify two standard properties: #address-cells and #size-cells. The value of #size-cells for
host controller nodes is set to 0, representing the fact that host controller addresses are an enumeration rather
than memory-like address ranges. The value of #address-cells is 1.
6.3.2 Device Node Properties

Device nodes are shown in the device tree, using the number of the USB hub port or the USB host controller port
to which the USB device is attached (such as Storage@4 is a USB storage device plugged into port 4 of the USB
host controller or USB hub). The name, compatible and reg device properties are based upon information out of
the USB device descriptor (bDeviceClass, bDeviceSubclass, bDeviceProtocol).
name
The device node name property is determined using the information in Table 6-2.
bDeviceClass bDeviceSubclass bDeviceProtocol Name
9 any any hub
Mass 1 any storage
Mass 2 any cdrom
Mass 3 any tape
Mass 4 any solid-state
Mass any any storage
any any any device
Table 6-2. Device Node Name Property
compatible
The compatible property provides a list of alternate name property values. The value is a list of encoded and
concatenated strings. The lists may include combinations of vendorID, productID fields and class descriptor
fields. Both the name and compatible property are important for device driver binding.
reg
The reg property for a device node consists of the number of the USB hub port or the USB host controller port to
which the USB device is attached.

6.3.3 Child/Interface Node Properties

Interface nodes represent the USB interfaces present on a USB device. Normally, each interface represents
independent controller functionality, although for some device classes (such as audio), one interface may provide
'out of band' control for another interface.
name
The interface node name property is determined using the information in Table 6-3.
bInterfaceClass bInterfaceSubclass bInterfaceProtocol Name
1 1 any sound-control
1 2 any sound
1 3 any midi
1 any any sound
3 1 1 keyboard
3 1 2 mouse
7 any any printer
9 any any hub
Power any any power
Monitor any any display-control
Communications 1 any modem
Communications 2 any modem
Communications 3 any telephone

Communications any any communications
Data any any data
any any any interface
Table 6-3. Interface Node Name Property
compatible
Similarly to the device node compatible property, the interface node compatible property is a list of encoded and
concatenated strings. The lists may include combinations of vendorID, productID fields and interface class
descriptor fields.
reg
The reg property for a child node consists of two integers; the first contains the bInterfaceNumber value, the
second the bConfiguration value.
7. Solaris[tm] Device Tree

________________________________________________________________________
Devices in the Solaris OE are represented as a tree of interconnected device nodes. The tree begins at the 'root'
device node, which represents the platform. Below the root node are 'branches' of the device tree, where a
branch consists of one or more bus nexus devices and a terminating leaf device. The system builds a tree
structure that contains information about the devices connected to the machine at boot time. The device tree can
also be modified by dynamic reconfiguration operations while the system is in normal operation.
7.1 Tree Structure

The tree structure creates a parent-child relationship between nodes. This parent-child relationship is the key to
architectural independence. When a leaf or bus nexus driver requires a service that is architecturally dependent
in nature, it requests its parent to provide the service. This approach enables drivers to function regardless of the
architecture of the machine or processor.
Bus nexus devices are devices that provide bus mapping and translation services to devices that are subordinate
to it in the device tree. PCI-PCI bridges, PCMCIA adapters, and SCSI HBAs are all examples of nexus devices.

Leaf devices are typical peripheral devices such as disks, tapes, network adapters, frame buffers, and so forth.
Drivers for these devices export the traditional character and block driver interfaces for use by user processes to
read and write data to storage or communication devices.
Figure 7-1. Example Device Tree
In Figure 7-1, the root node is the parent node of the child SUNW,ffb leaf node (a frame buffer), a pseudo bus
nexus node, and a PCI bus nexus node. The SUNW,ffb leaf node represents a system frame buffer. The pseudo
bus nexus node is the parent of any pseudo device drivers (drivers without hardware). The PCI bus nexus node
further has two PCI bus nexus nodes as its children representing two PCI-to-PCI bridges. The lower-left PCI bus
nexus node is the parent of the child nodes; ebus bus nexus node, network leaf node (ethernet), and ide bus
nexus node. The ebus bus nexus node is the parent of the child nodes fdthree leaf node (a floppy disk device)
and sd leaf node (a CD-ROM device).
7.2 Device Drivers

Associated with each leaf or bus nexus node can be a device driver. Each driver exports a device operations
structure dev_ops that defines the operations that the device driver can perform. The device operations
structure contains function pointers for generic operations such as attach, detach, and getinfo. It also contains
a pointer to a set of operations specific to bus nexus drivers and a pointer to a set of operations specific to leaf
drivers.
7.3 Displaying the Device Tree

The device tree can be displayed in three ways. The libdevinfo library provides interfaces to access the
contents of the device tree programmatically. The prtconf command displays the complete contents of the
device tree. And, the /devices hierarchy is a representation of the device tree; use ls to view it. The
/devices hierarchy displays only the devices that have drivers configured into the system, while prtconf

shows all device nodes regardless of whether a driver for the device exists on the system or not.
7.3.1 libdevinfo
libdevinfo provides interfaces for accessing all public device configuration data. See libdevinfo(3LIB)
for a list of interfaces. See http://soldc.sun.com/developer/support/driver/docs/whitepapers.html for the
libdevinfo white paper.
7.3.2 prtconf
The prtconf command displays all the devices in the system as shown in Figure 7-2.
System Configuration: Sun Microsystems sun4u

Memory size: 128 Megabytes
packages (driver not attached)

terminal-emulator (driver not attached)
deblocker (driver not attached)
obp-tftp (driver not attached)
disk-label (driver not attached)
SUNW,builtin-drivers (driver not attached)
sun-keyboard (driver not attached)
ufs-file-system (driver not attached)
chosen (driver not attached)
openprom (driver not attached)
client-services (driver not attached)
options, instance #0
aliases (driver not attached)
memory (driver not attached)
virtual-memory (driver not attached)
pci, instance #0
pci, instance #0
ebus, instance #0
auxio (driver not attached)
power, instance #0
SUNW,pll (driver not attached)
se, instance #0
su, instance #0
su, instance #1

ecpp (driver not attached)

fdthree, instance #0
eeprom (driver not attached)
flashprom (driver not attached)
SUNW,CS4231 (driver not attached)
network, instance #0
SUNW,m64B (driver not attached)
ide, instance #0
disk (driver not attached)
cdrom (driver not attached)
dad, instance #0
sd, instance #15
pci, instance #1
pci, instance #0
pci108e,1000 (driver not attached)
SUNW,hme, instance #1
SUNW,isptwo, instance #0
sd (driver not attached)
st (driver not attached)
sd, instance #0 (driver not attached)
....
SUNW,UltraSPARC-IIi (driver not attached)
SUNW,ffb, instance #0
pseudo, instance #0
Figure 7-2 prtconf Example
7.3.3 /devices
The /devices hierarchy provides a name space representing the device tree. Figure 7-3 is an abbreviated
listing of the /devices name space. The sample output in Figure 7-3 corresponds to the example device tree in
Figure 7-1 and the prtconf output shown in Figure 7-2.

/devices
/devices/pseudo
/devices/pci@1f,0:devctl
/devices/SUNW,ffb@1e,0:ffb0
/devices/pci@1f,0
/devices/pci@1f,0/pci@1,1
/devices/pci@1f,0/pci@1,1/SUNW,m64B@2:m640
/devices/pci@1f,0/pci@1,1/ide@3:devctl
/devices/pci@1f,0/pci@1,1/ide@3:scsi
/devices/pci@1f,0/pci@1,1/ebus@1
/devices/pci@1f,0/pci@1,1/ebus@1/power@14,724000:power_button
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:a
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:b
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:0,hdlc
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:1,hdlc
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:a,cu
/devices/pci@1f,0/pci@1,1/ebus@1/se@14,400000:b,cu
/devices/pci@1f,0/pci@1,1/ebus@1/ecpp@14,3043bc:ecpp0
/devices/pci@1f,0/pci@1,1/ebus@1/fdthree@14,3023f0:a
/devices/pci@1f,0/pci@1,1/ebus@1/fdthree@14,3023f0:a,raw
/devices/pci@1f,0/pci@1,1/ebus@1/SUNW,CS4231@14,200000:sound,audio
/devices/pci@1f,0/pci@1,1/ebus@1/SUNW,CS4231@14,200000:sound,audioctl
/devices/pci@1f,0/pci@1,1/ide@3
/devices/pci@1f,0/pci@1,1/ide@3/sd@2,0:a
/devices/pci@1f,0/pci@1,1/ide@3/sd@2,0:a,raw
/devices/pci@1f,0/pci@1,1/ide@3/dad@0,0:a
/devices/pci@1f,0/pci@1,1/ide@3/dad@0,0:a,raw
/devices/pci@1f,0/pci@1
/devices/pci@1f,0/pci@1/pci@2
/devices/pci@1f,0/pci@1/pci@2/SUNW,isptwo@4:devctl
/devices/pci@1f,0/pci@1/pci@2/SUNW,isptwo@4:scsi
Figure 7-3. /devices Hierarchy Sample

7.4 Binding a Driver to a Device

In addition to constructing the device tree, the kernel also determines the drivers that will be used to manage the
devices. Binding a driver to a device refers to the process by which the system selects a driver to manage a
particular device. The driver binding name is the name that links a driver to a unique device node in the device
information tree. For each device in the device tree, the system attempts to choose a driver from a list of installed
drivers. Each device node has a name property associated with it. This property can be assigned either from an
external agent, such as the PROM, during system boot, or from a driver.conf configuration file. In either
case, the name property represents the node name assigned to a device in the device tree. The node name is
the name visible in /devices and listed in the prtconf output.
Figure 7-4. Device Node Names
A device node can also have a compatible property associated with it. The compatible property (if it exists)
contains an ordered list of one or more possible driver names or driver aliases for the device. The system uses
both the name and the compatible properties to select a driver for the device. If the compatible property exists,
the system first attempts to match the contents of the compatible property to a driver on the system. Beginning
with the first driver name on the compatible property list, the system attempts to match the driver name to a
known driver on the system. It processes each entry on the list until either a match is found or the end of the list
is reached. If the contents of either the name property or the compatible property match a driver on the system,
then that driver is bound to the device node. If no match is found, no driver is bound to the device node.
7.5 Generic Device Names

Some devices specify a generic device name as the value for the name property. Generic device names
describe the function of a device without actually identifying a specific driver for the device. For example, a SCSI
host bus adapter might have a generic device name of scsi. An Ethernet device might have a generic device
name of "ethernet." The compatible property allows the system to determine alternate driver names (like "glm" for
SCSI HBA device drivers or "hme" for Ethernet device drivers) for devices with a generic device name. Devices
with generic device names are required to supply a compatible property.
System Driver List

esp isp cgaix ad SUNW,ffb at pci ...
Figure 7-5 and Figure 7-6 show two device nodes: one node uses a specific device name, and the other uses a
generic device name.
For the device node with a specific device name, the driver binding name SUNW,ffb is the same name as the
device node name.
Figure 7-5. Specific Driver Node Binding
For the device node with the generic device name display, the driver binding name SUNW,ffb is the first name on the
compatible property driver list that matches a driver on the system driver list. In this case, display is a generic
device name for frame buffers.
Figure 7-6. Generic Driver Node Binding
8. Appendix A: Sun Blade Workstations

________________________________________________________________________
This appendix focuses on the PCI I/O configurations of each of the Sun Blade workstation models. Sun Blade
workstations are PCI 2.1 Specification compliant. Full-length slots are 12 inches and short slots are 7 inches.
Short PCI I/O cards will also fit in full-length PCI I/O slots. 33-MHz cards that are capable of 3.3V operation may
also be used in the 66-MHz slots but may impact disk performance. 32-bit cards that match or meet the
preceding voltage requirement can be used in the 64-bit slots without forcing the entire bus to operate in 32-bit
mode. Information regarding the SCSI controller, Ethernet, USB, and other peripheral device configurations for
each model can be found in the references listed within each section. Table 8-1 displays the PCI slot
configuration for Sun Blade workstations.

(MHz/bits/Volts) 100 1000
# of 66/64/3.3 PCI cards - 1
# of 33/64/5 PCI cards - 3
# of 33/32/5 PCI cards 3 -
total # of PCI cards 3 4
Table 8-1. PCI Slot Configuration for Sun Blade Workstations

The Sun Blade 100 I/O subsystem supports the PCI I/O assembly. One on-chip PCI controller can support up to
four PCI bus masters. One PCI bus running at 33 MHz, 32-bit, 3.3V I/O manages the on-board I/O devices.
Another bus running at 33 MHz, 32-bit, 5V I/O manages three full-length external PCI slots which are located in
the rear of the system. For more information about the Sun Blade 100 system architecture, see "The Sun Blade
100 Workstation Architecture Technical White Paper" [11]. Figure 8-1 shows the Sun Blade 100 system design.


The Sun Blade 1000 I/O subsystem supports the PCI I/O assembly. Two independent 64-bit buses control four
external PCI slots. Three slots (2 through 4) operate at 33 MHz, 64-bit, and 5V I/O. One slot (1) operates at 66
MHz, 64-bit, and 3.3V I/O. Each slot can supply up to 15W of power. The PCI I/O assembly is located in the rear
of the system. The Sun Blade 1000 workstation has essentially the same internal architecture as the Sun Fire
280R Server. For more information about the Sun Blade 1000 system architecture, see "The Sun Blade 1000
Workstation Architecture Technical White Paper" [12]. Figure 8-2 displays the Sun Blade 1000 system design.

9. Appendix B: Sun Fire Servers

________________________________________________________________________
This appendix focuses on the PCI and cPCI I/O configurations of each of the Sun Fire server models. Sun Fire
servers are PCI 2.1 Specification compliant. Full-length slots are 12 inches and short slots are 7 inches. Short
PCI I/O cards will also fit in full-length PCI I/O slots. 33-MHz cards that are capable of 3.3V operation may also
be used in the 66-MHz slots but may impact disk performance. 32-bit cards that match or meet the preceding
voltage requirement can be used in the 64-bit slots without forcing the entire bus to operate in 32-bit mode.
Information regarding the SCSI controller, Ethernet, USB, and other peripheral device configurations for each
model can be found in the references listed within each section. Table 9-1 displays the PCI slot configuration for

Sun Fire servers.
(MHz/bit/Volts) 280R 440 880 3800 4800 4810 6800 15000
# of I/O assemblies 1 1 1 2 2 2 4 18
# of 66/64/3.3 cPCI
- - - 2 1* 1* 1* -
cards per assembly
# of 33/ 64/5 cPCI

- - - 4 3* 3* 3* -
cards per assembly
# of 66/64/3.3 PCI
1 2 2 - 2 2 2 2
cards per assembly
# of 33/64/5 PCI
3 4 7 - 6 6 6 2
cards per assembly
total # of cPCI slots - - - 12 8* 8* 16* -
total # of PCI slots 4 6 9 - 16 16 32 72
Table 9-1. PCI Slot Configuration for Sun Fire Servers
* Sun Fire 6800/4810/4800 Servers support cPCI cards as well, allowing half the number of total cPCI cards as
PCI cards.
9.1 Sun Fire 280R Server

The Sun Fire 280R I/O subsystem supports the PCI I/O assembly. There are two independent 64-bit PCI buses
and four external PCI slots; three slots for full-length PCI I/O cards and one slot for short PCI I/O cards. Short
PCI cards can be installed in any of the three full-length slots as well. Peak I/O throughput for the full-length PCI
I/O slots (2 through 4) is 33 MHz, 64-bit, and 5V I/O. Peak I/O throughput for the short PCI I/O slot (1) is 66/33
MHz, 64-bit, and 3.3V I/O. Each slot can supply up to 15W of power. The PCI I/O assembly is located in the rear
of the system. The Sun Fire 280R Server has essentially the same internal architecture as the Sun Blade 1000
workstation. For more information about the Sun Fire 280R architecture, see "The Sun Fire 280R Server Product
Notes" [13]. Figure 9-1 displays the Sun Fire 280R system design.

Figure 9-1. Sun Fire 280R System Design

The Sun Fire 880 I/O subsystem supports only the PCI I/O assembly (no cPCI support) and operates at 1.2
GB/sec bandwidth. There are four independent 64-bit PCI buses and nine external PCI slots, all of which are hot
pluggable. Seven full-length slots (0 through 6) operate at 33 MHz, 64-bit, and 5V I/O. Two full-length slots (7
and 8) operate at 66 MHz, 64-bit, and 3.3V I/O. Each slot can supply up to 15W of power. All slots accept
adapters, either 64-bit or 32-bit wide, and either full-length or short cards. However, if a 33-MHz adapter is
inserted into either of the 66-MHz slots, it will cause that bus segment to operate at 33 MHz. During a hot plug
operation, a 33-MHz adapter cannot be connected to a 66-MHz slot. The PCI I/O assembly is located on the left
side of the system. For more information about the Sun Fire 880 Server, see the "Sun Fire 880 Server Product
Notes" [14]. Figure 9-2 displays the Sun Fire 880 system design, and Figure 9-3 shows the Sun Fire 880 device

tree.
Figure 9-2. Sun Fire 880 System Design
{0} ok banner
Sun Fire 880, No Keyboard
OpenBoot 4.0, 8192 MB memory installed, Serial #12980144.
Ethernet address 8:0:20:c6:f:b0, Host ID: 80c60fb0.
{0} ok .speed
CPU0 Speed: 750 MHz, 5:1 ClkMode
CPU2 Speed: 750 MHz, 5:1 ClkMode
Safari Speed: 150 MHz
{0} ok show-devs
/pci@9,600000 -> HPB # 1, PCI bus A
/pci@9,700000 -> HBP # 1, PCI bus B
/pci@8,600000 -> HBP # 2, PCI bus A

/pci@8,700000 -> HBP # 2, PCI bus B

/SUNW,UltraSPARC-III@2,0
/SUNW,UltraSPARC-III@0,0
/virtual-memory
/memory@m0,20
/aliases
/options
/openprom
/chosen
/packages
/pci@9,700000/usb@1,3 -> USB, part of PCI I/O bridge on HBP #1
/pci@9,700000/network@1,1 -> 10/100 Mbit/sec Ethernet
/pci@9,700000/ebus@1
/pci@9,700000/ebus@1/serial@1,400000
/pci@9,700000/ebus@1/rsc-console@1,3083f8
/pci@9,700000/ebus@1/rsc-control@1,3062f8
/pci@9,700000/ebus@1/pmc@1,300700
/pci@9,700000/ebus@1/gpio@1,300600
/pci@9,700000/ebus@1/rtc@1,300070
/pci@9,700000/ebus@1/i2c@1,500030
/pci@9,700000/ebus@1/i2c@1,50002e
/pci@9,700000/ebus@1/bbc@1,500000
/pci@9,700000/ebus@1/i2c@1,30
/pci@9,700000/ebus@1/i2c@1,2e
/pci@9,700000/ebus@1/power@1,30002e
/pci@9,700000/ebus@1/bbc@1,0
/pci@9,700000/ebus@1/flashprom@0,0
/pci@9,700000/ebus@1/i2c@1,500030/idprom@0,a0
/pci@9,700000/ebus@1/i2c@1,500030/nvram@0,a0
/pci@9,700000/ebus@1/i2c@1,50002e/temperature@4,56
/pci@9,700000/ebus@1/i2c@1,30/hotplug-controller@0,ec -> Hotplug Ctrl
/pci@9,700000/ebus@1/i2c@1,30/hotplug-controller@0,e8

/pci@9,700000/ebus@1/i2c@1,30/fru@0,ae
/pci@9,700000/ebus@1/i2c@1,30/fru@0,a8
/pci@9,700000/ebus@1/i2c@1,30/fru@0,a2
/pci@9,700000/ebus@1/i2c@1,30/fru@0,a0
/pci@9,700000/ebus@1/i2c@1,30/temperature-sensor@0,9c
/pci@9,700000/ebus@1/i2c@1,30/adio@0,96
/pci@9,700000/ebus@1/i2c@1,30/adio@0,92
/pci@9,700000/ebus@1/i2c@1,30/adio@0,90
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,8a
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,88
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,82
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,80
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,72
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,70
/pci@9,700000/ebus@1/i2c@1,30/i2c-bridge@0,60
/pci@9,700000/ebus@1/i2c@1,30/adio@0,5e
/pci@9,700000/ebus@1/i2c@1,30/adio@0,5a
/pci@9,700000/ebus@1/i2c@1,30/controller@0,58
/pci@9,700000/ebus@1/i2c@1,30/ioexp@0,46
/pci@9,700000/ebus@1/i2c@1,30/smbus-ara@0,18
/pci@9,700000/ebus@1/i2c@1,30/controller@0,16
/pci@9,700000/ebus@1/i2c@1,2e/fru@4,aa
/pci@9,700000/ebus@1/i2c@1,2e/fru@4,a8
/pci@9,700000/ebus@1/i2c@1,2e/fru@4,a0
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,ae
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,ac
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,a8

/pci@9,700000/ebus@1/i2c@1,2e/fru@2,a6
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,a4
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,a2
/pci@9,700000/ebus@1/i2c@1,2e/fru@2,a0
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,ae
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,ac
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,a8
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,a6
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,a4
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,a2
/pci@9,700000/ebus@1/i2c@1,2e/fru@0,a0
/pci@8,600000/SUNW,qlc@2
/pci@8,600000/network@1 -> Gigabit Ethernet on PCI 66/64 bus
/pci@8,600000/SUNW,qlc@2/fp@0,0 -> FC-AL controller on PCI 66/64 bus
/pci@8,600000/SUNW,qlc@2/fp@0,0/disk
/pci@8,700000/scsi@1
/pci@8,700000/scsi@1/tape
/pci@8,700000/scsi@1/disk
/openprom/client-services
/packages/ufs-file-system
/packages/kbd-translator
/packages/dropins
/packages/SUNW,debug
/packages/obp-tftp
/packages/terminal-emulator
/packages/disk-label
/packages/deblocker
/packages/SUNW,builtin-drivers
{0} ok
{0} ok cd /pci@9,600000
{0} ok ls
{0} ok cd ..
{0} ok cd /pci@9,700000
{0} ok ls
f00e360c usb@1,3
f00db25c network@1,1
f00a6f98 ebus@1

{0} ok cd ..
{0} ok cd /pci@8,600000
{0} ok ls
f00cf5c4 SUNW,qlc@2
f00c8cb0 network@1
{0} ok cd ..
{0} ok cd /pci@8,700000
{0} ok ls
f00d52e4 scsi@1
Figure 9-3. Sun Fire 880 Device Tree

The Sun Fire 3800 I/O subsystem supports only the CompactPCI I/O (cPCI) assembly and operates at 4.8-
Gbyte/sec bandwidth. There are six external slots per cPCI I/O assembly. Slots 0 and 1 are 66/33 MHz, 64-bit,
and 3.3V I/O. Slots 2 through 5 is 33 MHz, 64-bit, and 5V I/O. The 2 cPCI I/O assemblies are located at the front
of the system. When installing cPCI I/O cards, populate the left cPCI I/O assembly locations first and the right
cPCI I/O assembly locations last. For more information about the Sun Fire 3800 Server architecture, see "Sun
Fire 6800/4810/4800/3800 Systems Overview" [16]. Figure 9-4 displays the Sun Fire 3800 Server cPCI I/O
assembly.
Figure 9-4. Sun Fire 3800 Server cPCI I/O Assembly

The Sun Fire 4800 subsystem supports two PCI I/O assemblies and operates at 4.8-Gbyte/sec bandwidth. There
are two supported I/O card cage assemblies, a standard PCI assembly and a cPCI, Compact PCI assembly. The
cPCI assembly consists of four 3U slots. There are four cPCI buses, two 33-MHz slots and two 66-MHz slots.
Note that these cPCI assemblies are different than the cPCI assemblies used in the SF3800. The PCI I/O
assembly consists of eight external PCI slots per PCI I/O assembly; six slots for full-length PCI I/O cards and two
slots for short PCI I/O cards. Short PCI cards can be installed in any of the six full-length slots as well. The full-
length PCI I/O slots (0, 1, 2, 4, 5, 6) are 33 MHz, 64-bit, and 5V I/O. The short PCI I/O slots (3 and 7) are 66/33
MHz, 64-bit, and 3.3V I/O. Peak I/O is affected by operating frequency; total peak I/O throughput per PCI board
is 965 Mbyte/sec. The two PCI I/O assemblies are located in the front of the system. PCI I/O cards can be
installed in either the top PCI I/O assembly locations or the bottom PCI I/O assembly locations. For more
information about the Sun Fire 4800 Server architecture, see "Sun Fire 6800/4810/4800/3800 Systems
Overview" [16]. Figure 9-5 shows the Sun Fire 6800/4810/4800 server PCI I/O assembly.
Figure 9-5. Sun Fire 6800/4810/4800 Server PCI I/O Assembly

The Sun Fire 4810 sub-system supports two PCI I/O assemblies and operates at 4.8-Gbyte/sec bandwidth.
There are two supported I/O card cage assemblies, a standard PCI assembly and a cPCI, Compact PCI
assembly. The cPCI assembly consists of four 3U slots. There are four cPCI buses, two 33-MHz slots and two
66-MHz slots. Note that these cPCI assemblies are different than the cPCI assemblies used in the SF3800. The
PCI I/O assembly consists of eight external PCI slots per PCI I/O assembly; six slots for full-length PCI I/O cards
and two slots for short PCI I/O cards. Short PCI cards can be installed in any of the six full-length slots as well.
The full-length PCI I/O slots (0, 1, 2, 4, 5, 6) are 33 MHz, 64-bit, and 5V I/O. The short PCI I/O slots (3 and 7) are
66/33 MHz, 64-bit, and 3.3V I/O. Peak I/O is affected by operating frequency; total peak I/O throughput per PCI
board is 965 Mbyte/sec. The two PCI I/O assemblies are located in the front of the system. PCI I/O cards can be
installed in either the top PCI I/O assembly locations or the bottom PCI I/O assembly locations. For more
information about the Sun Fire 4810 Server architecture, see Sun Fire 6800/4810/4800/3800 Systems Overview
[16]. Figure 9-6 is a block diagram representing the Sun Fire 6800/4810/4800/3800 servers.

Figure 9-6. Block Diagram of the Sun Fire 6800/4810/4800/3800 Servers

The Sun Fire 6800 I/O subsystem supports the PCI I/O assembly and operates at 9.6-Gbyte/sec bandwidth.
There are two supported I/O card cage assemblies, a standard PCI assembly and a cPCI, Compact PCI
assembly. The cPCI assembly consists of four 3U slots. Each cPCI I/O assembly has four cPCI buses, two 33-
MHz slots and two 66-MHz slots. Note that these cPCI assemblies are different than the cPCI assemblies used
in the SF3800. The PCI I/O assembly consists of eight external PCI slots per PCI I/O assembly; six slots for full-
length PCI I/O cards and two slots for short PCI I/O cards. Short PCI cards can be installed in any of the six full-
length slots as well. The full-length PCI I/O slots (0, 1, 2, 4, 5, 6) are 33 MHz, 64-bit, and 5V I/O. The short PCI
I/O slots (3 and 7) are 66/33 MHz, 64-bit, and 3.3V I/O. Peak I/O is affected by operating frequency; total peak
I/O throughput per PCI board is 965 Mbyte/sec. The two PCI I/O assemblies are located in the front of the
system. PCI I/O cards can be installed in either the top PCI I/O assembly locations or the bottom PCI I/O
assembly locations. For more information about the Sun Fire 6800 Server architecture, see Sun Fire
6800/4810/4800/3800 Systems Overview [16]. Figure 9-7 shows the standard operation of the Sun Fire
6800/4810/4800/3800 servers.

Figure 9-7. Standard Operation of the Sun Fire 6800/4810/4800/3800 Servers

The Sun Fire 15000 I/O subsystem supports the PCI I/O assembly and operates at 2.4-Gbyte/sec bandwidth.
There are two 600-Mbyte total bandwidth capacity I/O controllers per I/O assembly; each controller offering one
66/33-MHz, 64-bit, 3.3V slot and one 33-MHz, 64-bit, 5V slot. Therefore, each I/O assembly contains four
external PCI slots. There are a total of 18 maximum I/O board assemblies, bringing the grand total of external
PCI slots to 72. Nine of the four PCI card capacity hot-swap cassettes are located in the front of the system, and
the other nine are located in the rear of the system. For more information about the Sun Fire 15000 Server
architecture, see Sun Fire 15K System Overview [19]. Figure 9-8 shows the Sun Fire 15000 system interconnect,
and Figure 9-9 is a block diagram of the Sun Fire 15000 board set.

Figure 9-8. Sun Fire 15000 System Interconnect

Figure 9-9. Sun Fire 15000 Board Set Block Diagram

Glossary
________________________________________________________________________
ack: acknowledge character, a signal sent by a station to a terminating station as an affirmative response that a
connection has been made, or that data has been received.
ASIC: application-specific integrated circuit
bus: a set of parallel communication lines that connect the major components of a computer system
CEI: Channel Engine Interface
cPCI: Compact PCI
DAC: dual address cycle
DDI: device driver interface
device driver: software that communicates with and manages a device on behalf of the user or application
DKI: driver kernel interface
DMA: Direct Memory Access
DV: digital video
DVMA: Direct Virtual Memory Access
ECC: error checking and correction
FC-AL: Fibre Channel Arbitrated Loop
FCodes: Forth bytecodes, a small program, usually a bootstrap loader, written in the Forth language and stored
in a PROM or EPROM
GDN: generic device name
HPB: Host PCI Bridge
IEEE: Institute of Electrical and Electronic Engineers

IEEE 1275: a standard that specifies how firmware should be applied to the PCI bus
IOMMU: I/O Memory Management Unit
LAN: Local area network
MBus: Multibus
nack: negative acknowledge character, a control code returned by a receiving station indicating that a station
with an established connection has sent incorrect information.
OBP: Open Boot PROM
OpenBoot: In SBus profiles, the facility by which the FCodes program can interrogate the host and determine the
state of various parameters it addresses
PBM: PCI bus module
PCI: Peripheral Component Interconnect
PCI SIG: PCI Special Interest Group
PIB: PCI I/O Bridge
PIO: peripheral I/O
PROM: programmable read-only memory
SBus: a 32-bit, self-identifying bus used mainly on SPARC technology-based workstations
SCSI: Small Computer Systems Interface
SSM: Scalable Shared Memory
TLB: Translation Lookaside Buffer
TSB: Translation Storage Buffer
TSO: Total Store Order
UPA: Ultra port architecture
USB: Universal Serial Bus
VIS: Visual Instruction Set

References
________________________________________________________________________
[1] "The PCI Bus and Sun[tm] Ultra[tm] Systems Technical Brief," Sun Microsystems, Inc., 1997:
http://sunsite.csi.forth.gr/sunsite/Sun/sun_product_papers/iotech/pci_ultra.pdf
[2] "Writing Solaris[tm] PCI Device Drivers for Sun[tm] SPARC[tm] Platforms," Chien-Hua Yen, Sun
Microsystems Computer Corporation, February 1997:
http://sunsite.csi.forth.gr/sunsite/Sun/sun_product_papers/iotech/
[3] "IEEE Standard 1275-1994 Standard for Boot (Initialization Configuration) Firmware: Core Requirements and
Practices," IEEE Standards Organization (ISBN Number: 1-55937-426-8):
http://playground.sun.com/1275/home.html
[4] "PCI Bus Binding: to IEEE Standard 1275-1994 Standard for Boot (Initialization Configuration) Firmware,
Revision 2.1," Open Firmware Working Group, 1998:
http://playground.sun.com/1275/bindings/pci/pci2_1.pdf
[5] "Open Firmware, Recommended Practice: Universal Serial Bus, Version 1," Open Firmware Working Group,
June 1, 1998: http://playground.sun.com/pub/1275/bindings/usb/
[6] "Generic Names Recommended Practice, Version 1.4," December 30, 1996:
http://playground.sun.com/pub/1275/practice/#gnames
[7] "Gigabit Ethernet, Accelerating the Standard for Speed," White Paper, Gigabit Ethernet Alliance, 1998:
http://www.10gea.org/GEA-Accel1999_rev-wp.pdf
[8] "FC-AL Selftest TOI," Sun Microsystems, Inc.:
http://fw.east/Docs/tois/IO-FCode/fcal-selftest/html/txtindex.html
[9] "Fibre Channel Technology from Sun Microsystems," Technical Brief, Sun Microsystems, Inc., 1997:
http://sunsite.csi.forth.gr/sunsite/Sun/sun_product_papers/SSA/fibre_channel.pdf
[10] "Writing Device Drivers," Sun Microsystems, Inc., 2000:
http://docs.sun.com/ab2/coll.45.13/DRIVER/@Ab2TocView?Ab2Lang=C&Ab2Enc=iso-8859-1&
DwebQuery=%22Writing+Device%22+OR+%22Device+Drivers%22&oqt=%22Writing+Device+Drivers%22

[11] "The Sun Blade 100 Workstation Architecture Technical White Paper," Sun Microsystems, Inc., March 2001:
http://www.sun.com/desktop/sunblade100/sb100_wp.pdf
[12] "The Sun Blade 1000 Workstation Architecture Technical White Paper," Sun Microsystems, Inc., 2000:
http://www.sun.com/desktop/sunblade1000/whitepapers/sb1000wp.pdf
[13] "Sun Fire 280R Server Product Notes," Sun Microsystems, Inc., September 2000:
http://www.sun.com/products-n-solutions/hardware/docs/Servers/Workgroup_Servers/Sun_Fire_280R/index.html
[14] "Sun Fire 880 Server Product Notes, "Sun Microsystems, Inc., May 2001:
http://www.sun.com/products-n-solutions/hardware/docs/pdf/806-6593-15.pdf
[15] "Sun Fire 880 Server Just The Facts," Sun Microsystems, Inc., July 2001
[16] "Sun Fire 6800/4800/3800 Systems Overview," Sun Microsystems, Inc., 2001
[17] "Sun Fire 6800/4800/3800 Systems Service Manual," Sun Microsystems, Inc., 2001
[18] "Sun Fire 3800-6800 Servers - Computing for the Net Effect," Technical White Paper, Sun Microsystems,
Inc., March 2001: http://www.sun.com/midframe/presskit/Sun_Fire_Technical_Overview_WP.pdf
[19] "Sun Fire 15K System Overview," Sun Microsystems, Inc., 2001:
http://sunsolve.sun.com/data/806/806-3509/pdf/806-3509-10.pdf
Sun Microsystems, Inc.

Sun Blade Whitepaper

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sun Blade Whitepaper

Uploaded by

Copyright:

Available Formats

file:///C|/Documents%20and%20Settings/Administrator/Desktop/5_1_latest_whitepaper/sunblade.whitepaper.

The Sun Blade[tm] Workstation

file:///C|/Documents%20and%20Settings/Administrato...top/5_1_latest_whitepaper/sunblade.whitepaper.html (1 of 67) [5/7/2002 2:19:55 PM]

Printed in the United States of America.

FireWire is a trademark of Apple Computer, Inc., used under license.

THIS PUBLICATION COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS.

file:///C|/Documents%20and%20Settings/Administrato...top/5_1_latest_whitepaper/sunblade.whitepaper.html (2 of 67) [5/7/2002 2:19:55 PM]

2. UltraSPARC® II System Architecture

2.1 UPA Interconnect Bar Switch

2.2 Host PCI Bridge

2.3 PCI I/O Bridge

3. UltraSPARC III System Architecture

3.1 Sun[tm] Fireplane Interconnect

3.2 Host PCI Bridge

3.3 PCI I/O Bridge

3.4 UltraSPARC III Processor

4. Host PCI Bridge ASIC

4.1 PCI Bus Module

4.1.1 Supported PCI Features

4.1.2 Unsupported PCI features

file:///C|/Documents%20and%20Settings/Administrato...top/5_1_latest_whitepaper/sunblade.whitepaper.html (3 of 67) [5/7/2002 2:19:55 PM]

4.1.3 Supported PCI Transactions

4.2 Streaming Cache

4.3 PCI IOMMU

4.4 Mondo Dispatch Unit

4.5 Interface Controller

4.6 Summary of PCI Characteristics

5. PCI I/O Bridge ASIC

5.1 Bus Adapter

5.2 Channel Engine Interface

5.3 1394 Channel Engine

5.4 USB Channel Engine

5.5 EBus Channel Engine

5.6 Ethernet Channel Engine

6. Open Boot Firmware

6.1 Device Properties

6.2 IEEE 1275 PCI Binding

6.2.1 Address Formats and Representations

6.2.2 Child Node Properties

6.3 IEEE 1275 USB Binding

6.3.1 Host Controller Node Properties

6.3.2 Device Node Properties

6.3.3 Child/Interface Node Properties

file:///C|/Documents%20and%20Settings/Administrato...top/5_1_latest_whitepaper/sunblade.whitepaper.html (4 of 67) [5/7/2002 2:19:55 PM]

7. Solaris[tm] Device Tree

7.1 Tree Structure

7.2 Device Drivers

7.3 Displaying the Device Tree

7.4 Binding a Driver to a Device

7.5 Generic Device Names

8. Appendix A: Sun Blade Workstations

8.1 Sun Blade 100 Workstation

8.2 Sun Blade 1000 Workstation

9. Appendix B: Sun Fire Servers

9.1 Sun Fire 280R Server

9.2 Sun Fire 880 Server

9.3 Sun Fire 3800 Server

9.4 Sun Fire 4800 Server

9.5 Sun Fire 4810 Server

9.6 Sun Fire 6800 Server

9.7 Sun Fire 15000 Server

Figure 2-1. Typical Sun Ultra PCI Bus Architecture Implementation