A just released revision 2 of the RoCE standard specifies an incompatible layering over UDP/IP to provide routability. However, it does not specify how lossless operation will be provided over an IP network, or how congestion control will be handled.
In contrast, iWARP is a stable and mature open standard, ratified by the IETF in 2007. iWARP was designed to leverage TCP/IP from the outset, which allows it to operate over cost-effective, regular Ethernet infrastructure and deliver on the promise of true convergence, with Ethernet transporting LAN, SAN and RDMA traffic over a single wire.
iWARP provides all the key RDMA technologies that lower latency and improve efficiency by offloading networking tasks from a server processor:
- Direct data placement, which reads/writes data directly from/to application memory
- Kernel bypass, which alleviates the cost of context switching from user space to kernel space
- Transport acceleration, which leverages protocol engines on a network controller to offload packet processing from the system CPU
iWARP RDMA is deployed on network controllers that have the capability to perform all of the processing of the network stack, including connection context, segmenting and reassembling packets and interrupt handling. With RDMA, the server processor no longer needs to touch the data or copy payload from a receive buffer to the application buffer.
The iWARP standard was recently updated with the adoption by IETF of RFC 7306, which adds two RDMA extensions: atomic operations and immediate data messages. These additions bring in RDMA functionality that is supported in IB, but were not part of the original iWARP specification.
Atomic operations allow clients to implement synchronized access to shared remote memory locations, i.e. guarantee that the operations performed on these memory locations by different users do not interleave such that the end result is undefined.
Atomics enhance the usability of iWARP in distributed system environments, such as message passing interface (MPI) applications, where jobs executing in parallel across the network can more readily implement synchronization points.
The new RFC also provides support for immediate data messages. This capability allows the upper-layer protocols (ULP) at the sender to provide a small amount of data to be delivered as part of the completion of the RDMA operation, improving the efficiency of delivering small notifications. With this enhancement, iWARP can support the RDMA Write with Immediate message that is found in other RDMA transport protocols.
iWARP use is rising in cloud environments, such as Microsoft's recently launched Cloud Platform System (CPS), a turnkey scalable private cloud appliance. CPS is a complete solution consisting of up to 4 racks of Dell hardware running Windows Server 2012, with the hypervisor, the management tools, and the automation needed for a large-scale, software-defined data center. The CPS networking infrastructure includes an Ethernet fabric and an iWARP-enabled storage fabric, which utilizes the SMB Direct protocol for high efficiency, high performance storage networking.
Sign up for Computerworld eNewsletters.