With Windows Server 2012 R2, Microsoft introduces support for tiered storage and write back caching. With only rudimentary details available, this blog examines some highlights and also asks a few questions that I hope to make the content of a future blog.
Tiered storage and write back caching with Windows Server 2012 R2 requires:
- A Storage Spaces capable set of rotating hard disks i.e. SAS, SATA, or USB hard disks. Obviously USB disks have their limitations in terms of IOPS.
- A Storage Spaces (set) of flash storage – the word “flash” is used to loosely include SLC, MLC, and other kinds of SSD; again these must be SAS, SATA, or USB
- Creation of a Storage Space that includes both rotating hard disk and flash disks
Tiered storage in Windows Server 2012 R2 provides just two levels. At any given time, a particular file may be
- Fully on SSD because that is how Windows decided the file should be
- Fully on HDD because that is how Windows decided the file should be
- Partly on SSD and partly on HDD because that is how Windows decided the file should be
- Pinned fully to either HDD or SSD by the administrator
With tiered storage, Windows tracks access to file ranges with a granularity of 1MB ranges. By default, a scheduled job runs at 1AM and moves the often accessed ranges of the file to SSD and the less often accessed parts of the file to HDD. The retiering can also be run on demand by the system administrator.
Windows Server 2012 R2 also introduces write back caching along with tiered storage. When writes happen, some (or all) of them end up with the new data on SSD tier. Presumably, at a later time when the scheduled optimization job runs, the data is moved to HDD.
The pros of the Windows Server 2012 tiered storage and write back caching:
- Built into the operating system and free
- If the understanding that write back caching simply places data on SSD and uses regular file structures is correct, the likelihood of data corruption due to cache coherency and cache corruption is minimized
The cons of Windows server 2012 tiered storage and write back caching:
- Only works with Storage spaces which requires SAS, SATA or USB, and in addition requires all storage to be non-RAID
- Does not work with dedicated SSD designated as cache or in other words, the likelihood of the SSD becoming full and then write back caching being turned off is higher
- Is not “real time” in the sense that potentially all writes go into the SSD. I could be wrong here since not enough details are available. But certainly, the process of moving often accessed file ranges to SSD, and less often accessed file ranges to HDD is not real time. It does allow the file to still be used during this retiering process, but it is still only periodic and by default only once per day.
- The retiering process and also the process of monitoring and logging statistics as to what file ranges are actively accessed and what file ranges are less actively accessed may be resource intensive.
Jose Baretto from Microsoft has put out numerous blogs and talks, including some on SMB 3.0 and Multichannel. Some examples include The basics of SMB 3.0 Multichannel and Windows Server 2012 NIC Teaming and Multichannel. While it is difficult to add to the voluminous material Jose has contributed, this blog highlights the Hyper-V 2012 operational scenarios where SMB 3.0 Multi-Channel is useful and also points out an often overlooked fact of a configuration that is SMB 3.0 Multichannel capable. The goal of the blog is to draw attention to things that Jose has adequately already explained, but people have missed for some reason.
The Hyper-V 2012 team rewrote pieces of Hyper-V to support placing Hyper-V workloads on an SMB 3.0 share. Significantly, Hyper-V uses SMB 3.0 as a transport to move large amount of data/files from one SMB 3.0 NAS to another during live migration. But SMB 3.0 Multichannel is useful in scenarios other than live migration as well.
With Windows Server 2012, Microsoft also rewrote APIs such as the CopyFile API to leverage SMB 3.0 and its performance. CopyFile ensures there are multiple 1MB I/Os in flight and using the SMB 2/3 credit algorithm, the number of I/Os in flight can be increased. Also, these multiple I/Os in flight can flow on parallel TCP connections using SMB 3.0 Multi-Channel. Deploying a VM would typically involve copying a large vhdx file from say the test server to the production server. Another example would be Microsoft System Center copying a large (10s of GBs) vhdx file from a System Center Library Server to the production Hyper-V server. These file copies would benefit from SMB Multichannel.
And that brings me to the last point in this blog. Many folks I have talked to miss the fact that there is a case that allows for Multi-Channel without requiring multiple NICs. Here are the cases where SMB 3.0 Multichannel can come into play:
- Either client OR server has multiple 1GB NICs. Note that it is not necessary for BOTH client and server to have multiple NICs.
- Either client OR server has multiple 10GB NICs. Note that it is not necessary for BOTH client and server to have multiple NICs.
- Both client AND server have a single 10 GB NIC and both the client and server NICs are RSS capable
As Jose Baretto has pointed out multiple times, even when both client and server each have only a single 10GB RSS capable NIC, that is sufficient to enable Multichannel. Microsoft observed that without RSS, TCP send/receive completion interrupts get serviced by a single CPU and that CPU becomes a bottleneck and prevents a typical Windows system from pumping a full 10Gbps through the NIC. A single TCP channel using RSS can alleviate the single CPU bottleneck, but it cannot use the full capability of the 10Gbps NIC. But a Multichannel TCP using RSS does have a chance of saturating the full 10Gbps capability.
Given that backup and vhdx file copy scenarios will occur often in a typical Hyper-V 2012 environment, SMB Multichannel thus not only plays an important role, but is very likely to be enabled with the hardware used for commercial deployment of Hyper-V 2012 VM workloads.
winntfs 