Print-friendly Version

Contents:

1. Introduction
2. System Physical Requirements
3 Typical Blade Server Interface Diagram
4 Electrical Interface Overview
   4.1 Midplane I/O
   4.2 Support Devices
   4.3 Power
   4.4 Interrupt Controller
   4.5 Indicators
5 FRU Elements and Behavior
   5.1 Management Interface
   5.2 Blade Support Chip (BSC)
6 PCI Bus Architecture
   6.1 South Bridge
7 Gigabit Ethernet
   7.1 MACs
   7.2 Broadcom BCM5704S
   7.3 SERDES and Physical Layer
   7.4 SERDES Control Signals
   7.5 SERDES Interface Specificationt
8 Internal I/O Devices
   8.1 LPC Super I/O
   8.2 1-Mbyte Flash PROM
   8.3 BSC and FRU-ID EEPROM
   8.4 BIOS EEPROM
   8.5 Temperature Monitor
   8.6 Blade Support Chip (BSC)
   8.7 Power Sequencing and Reset
   8.8 BSC Operating Mode
   8.9 Serial Ports
9 DC/DC Converters and Power Sequencing
   9.1 Diodes and Fusing
   9.2 Dual 12V Input Secondary Converters
   9.3 Hot Swap: Soft Start to Facilitate Hot Insertion
   9.4 Hot Swap Considerations
   9.5 Voltage Margining
   9.6 Secondary DC/DC Converter Requirements
   9.7 3.3V, 2.5V, 5V
   9.8 Secondary Power Sequencing
   9.9 3V3 Standby Regulator and H8 Reference
   9.10 Blade Server Total Power Dissipation
   9.11 CPU Power Dissipation
   9.12 Voltage Alarm Thresholds
10 Fans
   10.1 CPU Fan Speed Control
   10.2 CPU Fan Speed Monitoring
11 Indicators
12 Midplane Connector
13 Recommended Test Connections
14 References

1. Introduction

This documents describes the electrical interface requirements for specialized Blade FRUs intended for use in the Sun Fire B1600 Blade system chassis. The intended audience is electrical engineers responsible for designing new, solution-based products. These engineers should reference other documents, as well as the drawings in the References section (Section 14). This document is part of the Blade Computing Development Kit. Please send comments to: blade-dk-help@sun.com.

Vendors' Blade FRUs may be designed around different microcontroller and support chips. Vendor selection of microcontroller and support chips is best determined by the design engineer. Chips that are talked about in this document are only for design reference examples, and not a design recommended by Sun Microsystems. The design reference is used to discuss Blade FRU interfaces and interactions with the Sun Fire B1600 Blade system chassis only.

Blade FRUs are designed to provide a solution-based product. Most Blade FRUs are sealed units that require no customer intervention or tampering. The intention is that each Blade FRU is automatically configurable by software switches or environmental conditions, without customer intervention.

Each Blade FRU requires a Blade Support Chip (BSC) to handle out-of-band communication to the chassis. This communication is used to initialize the Blade FRU through power up sequence and hot swap conditions. Sun is making available BSC module code with jump table entries for interfaces to other program code. The bsc.c source code is scheduled to be made available under a source-level agreement. Please contact your account manager for assistance.

Both in-band and out-of-band communication are handled by two different electrical parts, but they still use the same connector on the midplane. In-band communication is dual differential gigabit Ethernet. Out-of-band communication is dual differential serial. All Blade FRUs must support both in-band and out-of-band communications.

Blade FRU power is supplied from two redundant 12V DC power units. The 12V DC unit is fused and diode or'ed as the primary power source for the Blade FRU. DC/DC converters are used to supply ASIC voltages to individual support chips.

2 System Physical Requirements

The way the major subsystems in the Sun Fire Blade Platform physically interface with each other is detailed in the document covering the system overview (see References section). It describes a midplane in which the Blade FRUs interface with one side, and the power supplies and system service processor (SSP) interface with the other side of the passive midplane.

The Blade FRU mechanical requirements are detailed in "Mechanical Requirements of Blade Servers and FRUs" (see References section), also included in the Blade Computing Development Kit.

Figure 1: Typical Blade Server Interface Diagram

3 Typical Blade Server Interface Diagram

The preceding figure (Figure 1) is a reference block diagram of the Blade Server functions that interface to a Sun Fire Blade System. Other acceptable configurations or variations can meet the interface requirements.

4 Electrical Interface Overview

4.1 Midplane I/0

• Two AC-coupled Gigabit SERDES Ethernet connections
• Two asynchronous serial ports from the Blade Server BSC microcontroller, one to each of two SSPs

4.2 Support Devices

• Real-time clock without battery (integrated with South Bridge)
• Blade Server BSC microcontroller
• 1-Mbyte Flash PROM
• FRU-ID EEPROM

4.3 Power

• Dual 12V input secondary converters
• Hot swap: soft start on power input

4.4 Interrupt Controller

The interrupt controller is built into the South Bridge and receives additional interrupt inputs from the BSC and BCM5704S.

4.5 Indicators

The Blade Server has the following indicators on the front panel:

• Blue -- OK to remove
• Green -- Normal operating mode
• Amber -- Fault detected, service action required

5 FRU Elements and Behavior

5.1 Management Interface

The management interface performs configuration, control, fault reporting and monitoring. Management can be administered over either the serial ports via CLI or over the Ethernet Links via CLI via Telnet or SNMP.

The management of a Sun Fire Blade unit is shared between a low-level 8-bit microcontroller and the Sun Fire Blade host or system processor. The host is responsible for higher-level management functions such as compute environment status and control, in the case of Blade Servers. The management system is described in the software document section of this Blade Computing Development Kit.

5.2 Blade Support Chip (BSC)

Sun Fire Blade units require the use of an 8-bit microcontroller as the system management interface, providing low-level board monitoring and control functions. The BSC contains two serial ports that connect to each of the redundant SSPs. Hardware details are described in "Electrical Requirements for Blade Server BSC" (see References section). The main purposes of the BSC are to:

1. Power up and down the Blade FRU.
2. Provide fault-management functions such as over-temperature shutdown, DC/DC voltage monitoring, and watchdog for the microcontroller host or higher-level system processor(s).

When a Blade FRU is first inserted into a Sun Fire Blade System, the BSC is powered up by a separate 5V supply and performs some low-level tests. After it boots up, it awaits the PowerON command from the SSP, after which the BSC powers up the rest of the FRU.

Detailed information about BSC function and control systems as relative to the Blade FRU can be found in "Electrical Requirements for Blade Server BSC" (see References).

6 PCI Bus Architecture

The reference architecture shown previously in Figure 1 uses a South Bridge component to support the PCI Bus Device.

6.1 South Bridge

The reference design South Bridge is the VIA VT8233A, packaged in a 376-pin plastic BGA (PBGA), and is connected to the North Bridge via a 66-MHz, 266-Mbyte/sec V-Link interface.

The South Bridge provides the following functions on the Blade Server:

• SM bus interface -- Access to the SPD (serial presence detect) feature of the DIMMs, which allows the BIOS to initialize the memory controller.
• LPC (low pin count) interface to LPC Super I/O device.
• IDE interface -- Provides an ATA-100 interface to the IDE disk; the South Bridge also supports ATA-133.
• XBus interface to flash PROM and BSC.
• Real-time clock -- No battery is provided; at boot-time the RTC is initialized by requesting the current time from the SSP via the BSC.
• 32-bit, 33-MHz PCI bus interface to BCM5704S.

7 Gigabit Ethernet

The Gigabit interface to system service processors is dual rail for the purpose of redundancy. Each rail is connected to the midplane and then to separate switch portions of the SSP. This allows the failover of both Gigabit Ethernet rails as well as the SSPs.

7.1 MACs

The design example found in Figure 1 shows two independent Ethernet MACs connected via the PCI Bus. They provide Ethernet MAC with PCS functionality, capable of operation up to Gigabit Ethernet.

7.2 Broadcom BCM5704S

The Broadcom BCM5704S is a dual Gigabit Ethernet MAC with integrated SERDES and is packaged in a 300-pin BGA. The 5704S has two interrupts, one for each channel, which connect to PCI bus INTA# and INTB#. The 5704S is currently fitted with an E2PROM to hold configuration information, including the MAC addresses in use.

7.3 SERDES and Physical Layer

The physical layer is implemented using a SERializer/DESerializer (SERDES) rather than the usual PHY device. The SERDES uses differential PECL TX+/- and RX+/- pairs to communicate over the midplane at 1.250 Gbit/sec.

The RX+/- pairs are AC coupled at the Blade Server, the TX+/- pairs are AC coupled at the Switch. This facilitates hot swap of the Blade Server and Switch.

Recommended AC coupling capacitors are 10nF 5% COG temperature characteristic. If independent SERDES devices are used, Sun recommends the use of Texas Instruments part number TLK2201RCP.

7.4 SERDES Control Signals

The Gigabit Ethernet SERDES will be compliant with the IEEE 802.3z specification. It should also support diagnostic features such as loop backs and PRBS generation and checking.

7.5 SERDES Interface Specification

Please see Broadcom Specification of the SERDES Link Gigabit Interconnection or contact Broadcom for information on the BCM5704S.

8 Internal I/O Devices

8.1 LPC Super I/O

This device provides many features; those used are:

• Serial console
• PROM attach
• General-purpose input pins for board version indication

8.2 1-Mbyte Flash PROM

The system requirement for flash is to provide enough capacity to hold two system code images. The system will provide a means to update the system image and, in the event of a failed update, the system can revert to the primary image. The system will allow updates from the midplane Ethernet interfaces or from the test connector.

• Footprint will support 1-Mbyte flash parts.
• The flash device should connect to the SIO via the LPC Bus.
• The BIOS is upgradable in the field.

8.3 BSC and FRU-ID EEPROM

A 16-Kbyte I2C EEPROM is accessible via the BSC microcontroller. This EEPROM contains the BSC variables and FRU-ID.

The EEPROM is nominally divided into 8 Kbyte for the FRU-ID and 8 Kbyte for the BSC variables.

The FRU-ID write-protection feature is implemented by the BSC firmware (this is probably best implemented by acknowledging the writes but not carrying them out).

I2C address: %1010 000

Since the RTC NVRAM in the South Bridge is not battery backed, the BIOS variables are stored in a second 16-Kbyte I2C EEPROM, accessible via the South Bridge SM Bus port.

8.5 Temperature Monitor

An Analog Devices ADM1021A monitors the CPU and ambient temperatures. This part is accessible via the onboard I2C bus from the BSC.

I2C address: %0011 000

8.6 Blade Support Chip (BSC)

This device is a Hitachi H8S/2148 microcontroller.

The BSC provides the following features:

• Dual access (for Blade Server and SSPs) to BIOS/FRU-ID EEPROM
• Communication channel between Blade FRU and SSPs
• Controls reset to the processor
• Controls the power, service_required and ready-to-remove LEDs
• Field-upgradable firmware, via host interface
• Implements a watchdog function for the OS
• Monitors the speed of the ASIC fan, ASIC and system temperature, and voltage rails
• Communicates with the BIOS and OS via the X-Bus
• Accesses the FRU-ID and DIMM SPD EEPROMS

8.7 Power Sequencing and Reset

The BSC is powered by 5VSB as soon as the Blade Server is fully inserted into the midplane. When instructed by the program, it turns on the rest of the DC/DC converters to provide power to the remainder of the Blade Server.

The BSC reset signal is derived from a simple POR generator that monitors the 5VSB rail.

8.8 BSC Operating Mode

The H8S/2148 is capable of operating in several modes, selected by the MD[1:0] pins. In this application the H8 is operated in Mode 2.

The chip selects are derived from a single South Bridge programmable chip-select signal (PCS0#) with the individual selects CS[4:1]# being decoded using a '139 logic IC.

Two of the four available BSC interrupt requests are connected to interrupt inputs on the South Bridge.

Figure 2: BSC Operating Mode

Please refer to the document on BSC requirements (see References section for "Electrical Requirements for Blade Server BSC") or the manufacturer's data for further information.

8.9 Serial Ports

The circuit in the following figure shows the configuration of the serial ports and the interface to the management processor (host) or system controller. The debug port is optional for devices that have dual serial ports.

COM2, if available, can be used for debug, and should be brought out to a pin header on the board, via an RS232 level shifter. These parts are not fitted in production.

The H8 I2C bus requires a pull-up to 5V to make the bus compatible with the 5V CMOS inputs on the H8. The I2C spec allows the connection of 3V3 devices to a 5V bus due to the open-drain nature of the bus.

The hexadecimal values listed are 7-bit values. They do not include the read/write bit, so they will require a left shift to convert to an 8-bit port value.

Figure 3: Serial Ports and Host or System Controller

9 DC/DC Converters and Power Sequencing

The Blade Server is powered from two fused and diode or'ed 12V supplies. DC/DC converters are used to provide the voltage levels required by the Blade FRU electronics.

9.1 Diodes and Fusing

The secondary DC/DC converters are supplied by dual 12V inputs, individually fused and then diode or'ed.

Blade FRU requires surface mount fuses. The diodes should have a forward voltage drop of less than 0.55V at rated current. Recommended part is MBRD1035CTL from ON Semiconductor. A small heat sink is likely to be required due to the amount of power dissipated.

9.2 Dual 12V Input Secondary Converters

If the BSC is reset (by a watchdog time-out or after a firmware download) the state of the DC/DC converters should not be affected. The 5V DC/DC converter is turned on as soon as the FRU is fully inserted, but only the BSC and support circuits are powered. An FET may be used to gate off the main 5V supply to the rest of the FRU.

The other DC/DC converters and the main 5V FET do not turn on until the BSC turns them on via a command from the SSP. This arrangement is illustrated in Figure 4.

1. The system controller is used rather than a GPIO pin on the BSC so that if the BSC is reset (by a watchdog time-out or after a firmware download), the state of the DC/DC converters is unchanged.
2. The 5V DC/DC converter is turned on as soon as the FRU is fully inserted, but only the BSC and the required portions of the system controller are powered (the 5VSB rail).
3. An FET is used to gate off the main 5V supply to the rest of the FRU.
4. When the output from the 3V3, 2V5, and voltage converters is within specification, the PWR_GOOD signal is asserted low to the BSC.
5. The Super IO has a resume circuit that runs from 3V3. A low-current, low-dropout regulator is required to generate this from the 5VSB supply.

Figure 4: Generic DE/DE Power Control

9.3 Hot Swap: Soft Start to Facilitate Hot Insertion

The Blade FRU is powered on when the BL_MATED_L signal is asserted low. This signal is on a short pin in the connector and is connected to GND on the midplane. When asserted, it generates a low signal to soft start the DC/DC converters, but not until the Blade Server connector is fully inserted.

BL_MATED_L enables the soft start circuit. When the FRU is inserted, the inrush current is limited to <1A and the rate of rise does not exceed 20A/s. The intent is to prevent damage to the connectors and to avoid generating noise due to sparking contacts.

9.4 Hot Swap Considerations

• When the Blade Server is off (BSC not powered or in reset) the CONS_IN and CONS_OUT signals should not sink or source >100uA. Tristate buffers should be used to control the outputs. The control signals are driven from a H8 output, SER_ENABLE_L, which must be held low to enable the Blade FRU serial outputs.
• The CONS_IN and CONS_OUT signals will be subject to voltages <=5V while the Blade Server is off. There must be no damage to the BSC under these conditions.
• The NETx_RX+/- signals must be AC coupled to the SERDES at the Blade Server.
• The NETx_TX+/- signals must be AC coupled at the switch.
• Soft start for 12V power.

9.5 Voltage Margining

Provision shall be made for voltage margining of the DC/DC converter outputs as part of the Blade FRU DVT.

FRU design guidelines require support to program all the rails together to a maximum and minimum voltage value, as detailed in the following table. No margining of the 12V input rail will be performed.

Eight signals are required, controlled by Port 20 to 27 on the BSC microcontroller. For simplicity the two signals should be MARGIN _vrail_UP/ and MARGIN_vrail_DOWN/.

The PWR_GOOD signal must be disabled in software during margining, to prevent false triggering while margining is in progress.

9.6 Secondary DC/DC Converter Requirements

• The secondary DC/DC converters must be protected against short circuit of their outputs so that no damage occurs.
• The input voltage range is 10-14V, and the converters must be able to withstand transient events to 28V without damage.
• Efficiency is very important and should be >90% at full load.

9.7 3.3V, 2.5V, 5V

Voltage regulation for full load range, input voltage range, and temperature range = +/-3% of nominal value. For instance +3.3V should remain between 3.20 and 3.40 volts for all combinations of load, input, and temperature.

Current changes should not cause more than +/-3% variation in output voltage for a load change of 25% (of full load rating) at a rate of 1A/microsecond. The transient must settle to within 1% of final voltage in 1 ms or less.

9.8 Secondary Power Sequencing

The secondary supplies must come up in the following order:

1. 5VSB (standby) first.
2. Your circuit requirements will determine the sequence of the other voltages.

9.9 3V3 Standby Regulator and H8 Reference

The 3.3V standby regulator (at the top in Figure 5) replaces the Zener diode dropper to create 3V3_SB. The regulator is capable of supplying 100mA, although around 10mA is used.

The voltage reference circuit shown at the bottom of Figure 5 feeds the analog reference in the H8 BSC microcontroller Aref (pin 36). The reference generates 2.5V, which forms the top end of the input range for the A to D converter. Where necessary the supply rails are divided down to less than 2.5V. Details of the division ratios are in the H8 pin list shown later on in this document.

Figure 5: Reference and Standby Regulators

9.10 Blade Server Total Power Dissipation

The total power dissipation of the Blade Server shall be no more than 48W.

9.11 CPU Power Dissipation

The dissipation of the CPU shall be no more than 25W.

9.12 Voltage Alarm Thresholds

The BSC monitors the actual voltage measured at each of the voltage rails. It compares these measured values to the thresholds programmed into the FRU-ID, and generates alarms that are signaled to environmental monitoring software. The programmed thresholds are as shown in the following diagram.

10 Fans

Air flow is from front to back of the Blade FRU. No ventilation is allowed on any other side. Fans are to assist in maintaining an operating temperature that is monitored by the BSC.

10.1 CPU Fan Speed Control

The Blade FRU fan is not speed controlled and runs at full speed. The intent is to minimize the temperature of the Blade FRU and the fan. However the PWM circuitry should be retained to allow future implementation of PWM speed control if required. With this circuit, the BSC must output a continuously low signal to the fan control port CPU_FAN_CTRL to enable the fan.

10.2 CPU Fan Speed Monitoring

The speed of the CPU blower/sink is monitored by the BSC, using the tachometer sense pin on the BSC microcontroller. In the event of the fan speed falling below 75 percent of its nominal speed, the BSC should alert the SSP module.

11 Indicators

The Blade Server has the following indicators on the front panel:

Active -- Green LED

Service Required -- Amber LED

Ready-To-Remove -- Blue LED

The indicators are mounted on the PCB. Light is conducted to the front panel using light pipes.

Figure 6: Front-Panel Indicators

12 Midplane Connector

The Blade Server connects to the midplane through a 40-pin SCA-2 connector.

Notes:

• The SCA-2 connector must be of the blind-mate type, with mate-first ground pins.
• The connector must be rated for at least 28VDC.
• The 12V inputs to the Blade Server are separated as far as possible from each other to avoid a safety hazard and minimize the possibility of a short between the rails.
• Seven power pins are required for a 6A maximum supply rail.

Notes:

• Long pins are shaded.
• Guide pins are connected to GND on Blade Server and midplane.

13 Recommended Test Connections

• H8 power bypass: Manually power up the main DC/DC converters. Fit a link to power up the DC/DC converters.
• BSC reset: Two-pin header. Insert jumper in J14 to reset the BSC.
• BSC boot mode: Use to program the H8 flash memory.
• PWRGOOD: Two-pin header. Insert jumper to hold the Blade Server in a powered down state (PWRGD not asserted). Remove for normal operation.

14 References

The documents in the following table are part of the Blade Computing Development Kit.