How is reliability achieved with SCSI and SAN?

Question

I am pretty new to SCSI and actually not even sure if this is the correct forum to ask. (I did because I found some SCSI questions :) So please feel free to improve/migrate this question.

I am playing with Fibre Channel transmission, and read in an internal document that unlike TCP, SCSI over FCP-3 is not guaranteed deliviery, hence my questions:

1. does this mean that the SCSI standard/protocol itself is not reliable? But I think it was once very popular for hard disks. How was the issue of reliability solved?
2. similarly, how is reliability handled in a SAN environment?

Answer 1

I found an informal article making the same comparison, which matches the rough impression I had. The article also shows multi-pathing to survive failure of an FC switch, or of one of the cables into a big SAN, as @Sobrique mentions.

SCSI doesn’t take kindly to dropped commands. It’s a bit of a misconception that SCSI can’t tolerate a lost command. It can, it just takes a long time to recover (relatively speaking). I’ve seen plenty of SCSI errors, and they’ll slow a system down to a crawl. So it’s best not to lose any SCSI commands.

https://datacenteroverlords.com/2011/09/14/fibre-channel-and-ethernet-the-odd-couple/

It sounds like FCP does not formally guarantee delivery, but... If you read the Wikipedia article, FibreChannel specifies a certain Bit Error Rate (as acceptable / expected). TCP is designed to operate over links which take much, much less care with packets than FibreChannel does.

FibreChannel also includes flow control, to avoid dropping packets due to congestion / overflowing buffers. IP doesn't (and doesn't expect the underlying networks to do so in any way).

For example, Google have this cool paper where they were measuring TCP packet loss rates averaging between 1% and 20% or even higher. (They advocate that ISPs use the technology which resulted in 1% losses (shaping) and not the 20%+ losses (policing). These technologies are how IP networks generally handle the problem of congestion).

I guess most SCSI implementations retry failed commands. I don't know how standardised it is - I expect it's possible to tune in various ways. Technically this doesn't guarantee delivery either, in the same way you can expect TCP to eventually time out (TCP would give up and close the connection).

Answer 2

It might be one for ServerFault.

But you're quite correct. Fiber Channel doesn't have the same protection mechanisms as TCP. It's more like UDP in that regard (although that's a bit of a weak analogy) and for many of the same reasons - for some applications, TCP is bad solution because of those reliability mechanisms - your stream can 'stall' for a retransmit, and that hurts a near-real-time application more than a dropped packet does. Storage IO latency you start to 'hurt' when you're more than about 20ms, and that's not enough time for TCP to do it's thing really.

What happens for FCP is that the SCSI driver on the endpoint handles the reliability, because as part of that it can also do load balancing. Commonly, you won't single-attach a fiber, but instead have dual HBAs with dual independent paths to storage.

So your driver routes packets however it likes (some are smarter than others - most do multipathing these days, but some do some quite clever adaptive multipathing), and keeps track of which IOs have been acknowledged or not. The OS can queue IO where appropriate, or ... well, not, if it thinks that's a bad idea. Practically speaking it does this anyway as part of routine filesystem caching mechanisms anyway.

This is why, for example open has the O_DIRECT option:

   O_DIRECT (since Linux 2.4.10)
          Try to minimize cache effects of the I/O to and from this
          file.  In general this will degrade performance, but it is
          useful in special situations, such as when applications do
          their own caching.  File I/O is done directly to/from user-
          space buffers.  The O_DIRECT flag on its own makes an effort
          to transfer data synchronously, but does not give the
          guarantees of the O_SYNC flag that data and necessary metadata
          are transferred.  To guarantee synchronous I/O, O_SYNC must be
          used in addition to O_DIRECT.  See NOTES below for further
          discussion.