A startup that recently released a novel flash memory card is working on a method to pack terabytes of storage cost-effectively in a compact server design. The work will require that server operating systems update their storage capabilities, something already in the works for Linux.

"There has been no innovation in operating system I/O structures for years, because it didn't matter; but now it does, and we are finding there are a lot of old cobwebs that were left hanging around," said David Flynn, chief technology officer for Fusion-io (Salt Lake City). "The poor I/O subsystems in the OS are the real storage bottleneck today."

Competitors say the approach makes sense. But they caution it could take years and is not likely to supplant bigger markets for disk drives and hybrid drives that embed a small amount of flash.

Fusion-io released its ioDrive, which packs 80 to 640 Gbytes of NAND flash on a PCI Express card, aiming to replace both hard drives and traditional solid-state drives. Its novel controller design allows performance of up to 100,000 I/O operations per second (IOPS), matching the throughput of the PCI Express bus the card rides.

The card pushes the limits for to- day's server architectures. Just one of Fusion-io's high-end 640-Gbyte cards will swamp a host processor in I/O handling jobs for block-based storage.

"We have the opportunity to fix something that's been broken in storage--and the reason to do it, because memory is moving from drives to silicon," said Flynn.

Although hard drives provide storage at much lower cost (about 50 cents to $2 per Gbyte) than flash (about $20 per Gbyte), disks trail in performance and power consumption. Those factors are opening up new frontiers for flash in computers, particularly in portable and high-end servers--as long as the PC underpinnings support solid-state technology.

If OSes did not spend so much time moving and copying block storage structures, they could handle more throughput, said Flynn. That would open the door for today's blade servers to pack enough of Fusion-io's Express cards to create, in effect, a storage farm of 20 Tbytes inside the existing 10U partial rack that houses the server.

Today, that amount of storage re- quires as many as four full racks of spinning hard disks linked over dedicated networks of Fibre Channel arrays connected to Fibre Channel switches. The cost of building and running that network could be as much as $1 million, or $80/Gbyte. By contrast, an equivalent set of the Fusion-io cards inside a server--with the right OS changes--would cost about $30,000, or $30/Gbyte.

"We are dissolving the storage-area network back into the server," said Flynn.

Fusion-io is working with the Linux community to make the required changes in that OS. The changes may not be available until at least the end of the year. Microsoft is believed to be considering similar changes for Windows Server. However, the company was not available to comment on the issue by press time.

The changes amount to a way to handle block storage more efficiently. Flynn suggested the block storage structures could be moved to a higher OS level, such as creating address offsets in files. The capability will also require changes in the hardware controller Fusion-io has designed.

"We think storage on PCI Express makes a lot of sense; it's just a question of when," said Pat Wilkison, vice president of marketing and business development at Stec Inc. (Santa Ana, Calif.). "It's not in the next 18 months."

Wilkison noted that "there are no protocols for handling a drive on Express. Some big changes need to happen on the software side, and you will rely on people like Microsoft to make that happen."

Stec makes the Zeus solid-state drive (SSD), which packs up to 500 Gbytes of flash in what looks and acts like a 3.5-inch disk drive riding standard 3-Gbit/second serial ATA and Serial-attached SCSI drive interconnects. The company claims the drive handles up to 40,000 IOPS using 8-kbyte packets.

By early 2008, OEMs will be using the flash drives to put the equivalent of a storage-area network in a server chassis without needing to retool the operating system, said Wilkison. "This is happening with our Zeus product now, and adoption rates of these high-end SSDs are looking good," he said.

Indeed, even leading hard drive makers such as Seagate Technology plan to get into the market for SSDs soon, though they are not disclosing details yet.

"We plan to ship SSD products in fiscal-year 2008," said a spokesman for Seagate. "Looking ahead, analysts and customers believe that hard drives remain the dominant storage solution in terms of units and petabytes shipped, followed by hybrid hard drives and then solid-state drives."

According to Flynn, the SSD designers have it all wrong. "People using block data over SATA buses and SCSI protocols are missing the point," he said.

The block protocols and drive interconnects are bottlenecks to the performance that flash can deliver over Express. Fusion-io employs the kind of direct-memory access transactions used in processor caching operations to move bulk storage, providing its 100,000-IOPS performance.

"At these rates, the CPU cannot keep up because it is handling too many block data functions in the OS," said Flynn.

Single-chip controller
Fusion-io implemented its flash controller as a single chip with an embedded 1-GHz PowerPC 405 core, a DMA engine, a PCI Express block and miscellaneous control logic. The controller manages a fixed, interleaved matrix of interconnects to each of up to 160 flash die on the Express card.

The up to 160 links are separated into multiple levels, each level executing given transactions in lockstep. Separate levels are independent and act in parallel to keep the internal bus full. That bus is 160 bits wide and runs at 50 MHz, essentially marrying the speed of the flash chips to the throughput capabilities of 2.5-Gbit/s Express.

"Fundamentally, it is moving 160 separate processes in NAND flash in parallel," said Flynn.

Fusion-io is among the few providers of flash disks that support multilevel-cell flash. MLC is 30 to 40 percent less expensive than the more popular, single-cell flash, but wears out more quickly.

The startup claims its MLC drive will last longer than a conventional hard drive. Fusion-io has not yet tested the throughput of its MLC products but believes it could be less than the 100,000 IOPS for its SLC designs.

"We support MLC, but we are not sure what our product mix will be like yet," Flynn said.