Trisha Shetty (Editor)

Fibre Channel

Updated on
Edit
Like
Comment
Share on FacebookTweet on TwitterShare on LinkedInShare on Reddit
Fibre Channel

Fibre Channel, or FC, is a high-speed network technology (commonly running at 1, 2, 4, 8, 16, 32, and 128 gigabit per second rates) primarily used to connect computer data storage to servers. Fibre Channel is mainly used in storage area networks (SAN) in commercial data centers. Fibre Channel networks form a switched fabric because they operate in unison as one big switch. Fibre Channel typically runs on optical fiber cables within and between data centers.

Contents

Most block storage runs over Fibre Channel Fabrics and supports many upper level protocols. Fibre Channel Protocol (FCP) is a transport protocol that predominantly transports SCSI commands over Fibre Channel networks. Mainframe computers run the FICON command set over Fibre Channel because of its high reliability and throughput. Fibre Channel can be used for flash memory being transported over the NVMe interface protocol.

Etymology

To promote the fiber optic aspects of the technology and to make a unique name, the industry decided to use the British English spelling fibre for the standard.

History

Fibre Channel is standardized in the T11 Technical Committee of the International Committee for Information Technology Standards (INCITS), an American National Standards Institute (ANSI)-accredited standards committee. Fibre Channel started in 1988, with ANSI standard approval in 1994, to merge the benefits of multiple physical layer implementations including SCSI, HIPPI and ESCON.

Fibre Channel was designed as a serial interface to overcome limitations of the SCSI and HIPPI interfaces. FC was developed with leading edge multi-mode fiber technologies that overcame the speed limitations of the ESCON protocol. By appealing to the large base of SCSI disk drives and leveraging mainframe technologies, Fibre Channel developed economies of scale for advanced technologies and deployments became economical and widespread.

Initially, the standard also ratified lower speed Fibre Channel versions with 132.8125 Mbit/s ("12,5 MB/s"), 265.625 Mbit/s ("25 MB/s"), and 531.25 Mbit/s ("50 MB/s") that were already growing out of use at the time. Fibre Channel saw adoption at 1 Gigabit/s Fibre Channel (1GFC) and its success grew with each successive speed. Fibre Channel has doubled in speed every few years since 1996.

Fibre channel has seen active development since its inception, with numerous speed improvements on a variety of underlying transport media. For example, the following table shows native Fibre Channel speeds:

In addition to a cutting edge physical layer, Fibre Channel also added support for any number of "upper layer" protocols, including ATM, IP and FICON, with SCSI being the predominant usage.

Topologies

There are three major Fibre Channel topologies, describing how a number of ports are connected together. A port in Fibre Channel terminology is any entity that actively communicates over the network, not necessarily a hardware port. This port is usually implemented in a device such as disk storage, an HBA on a server or a Fibre Channel switch.

  • Point-to-point (see FC-FS-3). Two devices are connected directly to each other. This is the simplest topology, with limited connectivity.
  • Arbitrated Loop (see FC-AL-2). In this design, all devices are in a loop or ring, similar to token ring networking. Adding or removing a device from the loop causes all activity on the loop to be interrupted. The failure of one device causes a break in the ring. Fibre Channel hubs exist to connect multiple devices together and may bypass failed ports. A loop may also be made by cabling each port to the next in a ring.
  • A minimal loop containing only two ports, while appearing to be similar to point-to-point, differs considerably in terms of the protocol.
  • Only one pair of ports can communicate concurrently on a loop.
  • Maximum speed of 8GFC.
  • Arbitrated Loop has been rarely used after 2010.
  • Switched Fabric (see FC-SW-6). In this design, all devices are connected to Fibre Channel switches, similar conceptually to modern Ethernet implementations. Advantages of this topology over point-to-point or Arbitrated Loop include:
  • The Fabric can scale to tens of thousands of ports.
  • The switches manage the state of the Fabric, providing optimized paths via Fabric Shortest Path First (FSPF).
  • The traffic between two ports flows through the switches and not through any other ports like in Arbitrated Loop.
  • Failure of a port is isolated to a link and should not affect operation of other ports.
  • Multiple pairs of ports may communicate simultaneously in a Fabric.

    • Layers

    Fibre Channel does not follow the OSI model layering, and is split into five layers:

  • FC-4 – Protocol-mapping layer, in which upper level protocols such as SCSI, IP or FICON, are encapsulated into Information Units (IUs) for delivery to FC-2. Current FC-4s include FCP-4, FC-SB-5, and FC-NVMe.
  • FC-3 – Common services layer, a thin layer that could eventually implement functions like encryption or RAID redundancy algorithms; multiport connections;
  • FC-2 – Signaling Protocol, defined by the Fibre Channel Framing and Signaling 4 (FC-FS-4) standard, consists of the low level Fibre Channel protocols; port to port connections;
  • FC-1 – Transmission Protocol, which implements line coding of signals;
  • FC-0 – PHY, includes cabling, connectors etc.;

    • This diagram from FC-FS-4 defines the layers.

    Layers FC-0 are defined in Fibre Channel Physical Interfaces (FC-PI-6), the physical layers of Fibre Channel.

    Fibre Channel products are available at 1, 2, 4, 8, 10, 16 and 32 and 128 Gbit/s; these protocol flavors are called accordingly 1GFC, 2GFC, 4GFC, 8GFC, 10GFC, 16GFC, 32GFC or 128GFC. The 32GFC standard was approved by the INCITS T11 committee in 2013, and those products became available in 2016. The 1GFC, 2GFC, 4GFC, 8GFC designs all use 8b/10b encoding, while the 10GFC and 16GFC standard uses 64b/66b encoding. Unlike the 10GFC standards, 16GFC provides backward compatibility with 4GFC and 8GFC since it provides exactly twice the throughput of 8GFC or four times that of 4GFC.

    Ports

    Fibre Channel ports come in a variety of logical configurations. The most common types of ports are:

  • N_Port (Node port) An N_Port is typically an HBA port that connects to a switch's F_Port or another N_Port. Nx_Port communicating through a PN_Port that is not operating a Loop Port State Machine.
  • F_Port (Fabric port) An F_Port is a switch port that is connected to an N_Port.
  • E_Port (Expansion port) Switch port that attaches to another E_Port to create an Inter-Switch Link.

    • Fibre Channel Loop protocols create multiple types of Loop Ports:

  • L_Port (Loop port) FC_Port that contains Arbitrated Loop functions associated with the Arbitrated Loop topology.
  • FL_Port (Fabric Loop port) L_Port that is able to perform the function of an F_Port, attached via a link to one or more NL_Ports in an Arbitrated Loop topology.
  • NL_Port (Node Loop port) PN_Port that is operating a Loop port state machine.

    • If a port can support loop and non-loop functionality, the port is known as:

  • Fx_Port switch port capable of operating as an F_Port or FL_Port.
  • Nx_Port end point for Fibre Channel frame communication, having a distinct address identifier and Name_Identifier,providing an independent set of FC-2V functions to higher levels, and having the ability to act as an Originator, a Responder, or both.

    • Ports have virtual components and physical components and are described as:

  • PN_Port entity that includes a Link_Control_Facility and one or more Nx_Ports.
  • VF_Port (Virtual F_Port) instance of the FC-2V sublevel that connects to one or more VN_Ports.
  • VN_Port (Virtual N_Port) instance of the FC-2V sublevel. VN_Port is used when it is desired to emphasize support for multiple Nx_Ports on a single Multiplexer (e.g., via a single PN_Port).
  • VE_Port (Virtual E_Port) instance of the FC-2V sublevel that connects to another VE_Port or to a B_Port to create an Inter-Switch Link.

    • The following types of ports are also used in Fibre Channel:

  • A_Port (Adjacent port) combination of one PA_Port and one VA_Port operating together.
  • B_Port (Bridge Port) Fabric inter-element port used to connect bridge devices with E_Ports on a Switch.
  • D_Port (Diagnostic Port) A configured port used to perform diagnostic tests on a link with another D_Port.
  • EX_Port A type of E_Port used to connect to an FC router fabric.
  • G_Port (Generic Fabric port) Switch port that may function either as an E_Port, A_Port, or as an F_Port.
  • GL_Port (Generic Fabric Loop port) Switch port that may function either as an E_Port, A_Port, or as an Fx_Port.
  • PE_Port LCF within the Fabric that attaches to another PE_Port or to a B_Port through a link.
  • PF_Port LCF within a Fabric that attaches to a PN_Port through a link.
  • TE_Port (Trunking E_Port) A trunking expansion port that expands the functionality of E ports to support VSAN trunking, Transport quality of service (QoS) parameters, and Fibre Channel trace (fctrace) feature.
  • U_Port (Universal port) A port waiting to become another port type
  • VA_Port (Virtual A_Port) instance of the FC-2V sublevel of Fibre Channel that connects to another VA_Port.
  • VEX_Port VEX_Ports are no different from EX_Ports, except underlying transport is IP rather than FC.

    • Media and modules

    The Fibre Channel physical layer is based on serial connections that use corresponding modules. The small form-factor pluggable transceiver (SFP) module and its enhanced version SFP+ are common form factors for ports, supporting a variety of distances via multi-mode and single-mode fiber as shown in the table below. The SFP module uses duplex fiber cabling that has LC connectors.

    The quad small form-factor pluggable (QSFP) module began being used for 4-lane implementations of 128GFC. The QSFP uses either the LC connector for 128GFC-CWDM4 or an MPO connector for 128GFC-SW4 or 128GFC-PSM4. The MPO cabling uses 8- or 12-fiber cabling infrastructure that connects to another 128GFC port or may be broken out into four duplex LC connections to 32GFC SFP+ ports. Fibre Channel switches use either SFP or QSFP modules.

    Modern Fibre Channel devices support SFP transceiver, mainly with LC (Lucent Connector) fiber connector. Older 1GFC devices used GBIC transceiver, mainly with SC (Subscriber Connector) fiber connector.

    Fibre Channel Storage Area Networks

    The goal of Fibre Channel is to create a storage area network (SAN) to connect servers to storage.

    The SAN is a dedicated network that enables multiple servers to access the same storage. Enterprise storage uses the SAN to backup to tape libraries while the storage is still accessible to the server. Servers may access storage from multiple storage devices over the network as well.

    SANs are often designed with dual fabrics to increase fault tolerance. Two completely separate fabrics are operational and if the primary fabric fails, then the second fabric becomes the primary.

    Fibre Channel Switches

    Fibre Channel switches can be divided into two classes. These classes are not part of the standard, and the classification of every switch is a marketing decision of the manufacturer:

  • Directors offer a high port-count in a modular (slot-based) chassis with no single point of failure (high availability).
  • Switches are typically smaller, fixed-configuration (sometimes semi-modular), less redundant devices.

    • A fabric consisting entirely of one vendors products is considered to be homogeneous. This is often referred to as operating in its "native mode" and allows the vendor to add proprietary features which may not be compliant with the Fibre Channel standard.

    If multiple switch vendors are used within the same fabric it is heterogeneous, the switches may only achieve adjacency if all switches are placed into their interoperability modes. This is called the "open fabric" mode as each vendor's switch may have to disable its proprietary features to comply with the Fibre Channel standard.

    Some switch manufacturers offer a variety of interoperability modes above and beyond the "native" and "open fabric" states. These "native interoperability" modes allow switches to operate in the native mode of another vendor and still maintain some of the proprietary behaviors of both. However, running in native interoperability mode may still disable some proprietary features and can produce fabrics of questionable stability.

    Fibre Channel Host Bus Adapters

    Fibre Channel HBAs, as well as CNAs, are available for all major open systems, computer architectures, and buses, including PCI and SBus. Some are OS dependent. Each HBA has a unique World Wide Name (WWN), which is similar to an Ethernet MAC address in that it uses an Organizationally Unique Identifier (OUI) assigned by the IEEE. However, WWNs are longer (8 bytes). There are two types of WWNs on a HBA; a node WWN (WWNN), which can be shared by some or all ports of a device, and a port WWN (WWPN), which is necessarily unique to each port.

    References

    Fibre Channel Wikipedia
    (Text) CC BY-SA