Category Archives: File copy

Backup performance and SMB 3 Multi Channel

March 3, 2014 – 8:30 pm

In this day and age of exploding data amounts, backup and restore is both increasingly important, and becoming more common and taken for granted. But not all backup “target systems” i.e. the systems to which data is backed up are created equal. Especially so, when the system being backed up is Windows based.

  1.  If your backup target system is based upon CIFS (also sometimes referred to as SMB 1), backup (and restore) is limited to 64kb serial I/O. In other words, the backup/restore software does a 64kb I/O, waits for the I/O to complete, and only then issues the next I/O. In fact it is worse than this. The total payload is limited to 64kb and hence well behaved apps that want to perform I/O in 4MB block size will only use a 60kb payload (data).
  2. If your backup target system is running SMB 2.0, the I/O is 1MB serial, which is certainly an improvement.
  3. If your backup target is SMB 2.1, the I/O is again 1MB, but SMB 2.1 has a server issuing multiple credits which means the client can issue multiple I/Os without having to wait for any one of the I/Os to complete. A typical Windows to Windows flow will show 10 1MB I/Os on the wire at the same time. Note that this is all on a single TCP channel. So the backup/restore speed is significantly higher
  4. Now recall that in most cases, BOTH the system being backed up AND the backup target are servers. For example, you could be backing up a file server or SQL server or Hyper-V server, and of course, the backup target also operates typically as a NAS (file server).  Thus it is very likely that at least one of the two has multiple NICs. If any one (or both) ends of an SMB 3 connection have multiple NICs, and provided these NICs are 10GB RSS capable (which are fairly cheap now), SMB 3 Multi Channel will kick in. SMB 3 Multi Channel establishes multiple TCP channels and engages multiple credits on each TCP channel. So with just 2 TCP channels, you could now have 20MB I/O in flight at any given moment.

In short, if Windows and especially so Windows 2012 is part of your IT environment (or planned environment), make sure your backup target has an upgrade path to SMB 3! And don’t be fooled by just the SMB 3 label! Ask your vendor if it is SMB 3 Multi Channel. The SMB 3 protocol allows a storage device to negotiate SMB 3, but not support SMB 3 Multi Channel!

Wishing you higher backup/restore speeds with SMB 3 Multi Channel!

Advertisement
By dilipcnaik | Also posted in Hyper-V, SMB, SMB 2.2, SMB 3.0, Windows 8, Windows Server 2012 | Comments (0)

Hyper-V 2012 operations and the importance of SMB 3.0 Multichannel

July 12, 2013 – 12:03 am

Jose Baretto from Microsoft  has put out numerous blogs and talks, including some on SMB 3.0 and Multichannel. Some examples include The basics of SMB 3.0 Multichannel and Windows Server 2012 NIC Teaming and Multichannel. While it is difficult to add to the voluminous material Jose has contributed, this blog highlights the Hyper-V 2012 operational scenarios where SMB 3.0 Multi-Channel is useful and also points out an often overlooked fact of a configuration that is SMB 3.0 Multichannel capable. The goal of the blog is to draw attention to things that Jose has adequately already explained, but people have missed for some reason.

The Hyper-V 2012 team rewrote pieces of Hyper-V to support placing Hyper-V workloads on an SMB 3.0 share. Significantly, Hyper-V uses SMB 3.0 as a transport to move large amount of data/files from one SMB 3.0 NAS to another during live migration. But SMB 3.0 Multichannel is useful in scenarios other than live migration as well.

With Windows Server 2012, Microsoft also rewrote APIs such as the CopyFile API to leverage SMB 3.0 and its performance. CopyFile ensures there are multiple 1MB I/Os in flight and using the SMB 2/3 credit algorithm, the number of I/Os in flight can be increased. Also, these multiple I/Os in flight can flow on parallel TCP connections using SMB 3.0 Multi-Channel. Deploying a VM would typically involve copying a large vhdx file from say the test server to the production server. Another example would be Microsoft System Center copying a large (10s of GBs) vhdx file from a System Center Library Server to the production Hyper-V server. These file copies would benefit from SMB Multichannel.

And that brings me to the last point in this blog. Many folks I have talked to miss the fact that there is a case that allows for Multi-Channel without requiring multiple NICs. Here are the cases where SMB 3.0 Multichannel can come into play:

  • Either client OR server has multiple 1GB NICs. Note that it is not necessary for BOTH client and server to have multiple NICs.
  • Either client OR server has multiple 10GB NICs. Note that it is not necessary for BOTH client and server to have multiple NICs.
  • Both client AND server have a single 10 GB NIC and both the client and server NICs are RSS capable

As Jose Baretto has pointed out multiple times, even when both client and server each have only a single 10GB RSS capable NIC, that is sufficient to enable Multichannel. Microsoft observed that without RSS, TCP send/receive completion interrupts get serviced by a single CPU and that CPU becomes a bottleneck and prevents a typical Windows system from pumping a full 10Gbps through the NIC. A single TCP channel using RSS can alleviate the single CPU bottleneck, but it cannot use the full capability of the 10Gbps NIC. But a Multichannel TCP using RSS does have a chance of saturating the full 10Gbps capability.

Given that backup and vhdx file copy scenarios will occur often in a typical Hyper-V 2012 environment, SMB Multichannel thus not only plays an important role, but is very likely to be enabled with the hardware used for commercial deployment of Hyper-V 2012 VM workloads.

By dilipcnaik | Also posted in Hyper-V, SMB, SMB 3.0, VHDX | Comments (0)

Windows Write Caching – Part 3 – An Overview For System Administrators

December 1, 2012 – 4:21 pm

The Windows Cache Manager (also referred to as System Cache) acts as a single system-wide cache that contains driver code, application code, data for both, user mode applications as well as driver data. While an application can make API calls in a manner that guarantees the application data bypasses this cache, there is no way for an application to guarantee that its data WILL be cached. Because the behavior of the cache depends upon a number of factors and is very often non repeatable, the application and system administrator can only increase the likelihood that the application data will be cached. In other words, executing the same program multiple times is very likely to result in slightly different cache behavior each time. This is part of the reason why applications such as Microsoft SQL and Microsoft Exchange bypass the System Cache.

To illustrate the complexities involved, consider the seemingly simple act of copying a file from one volume to another. Some, but not all, of these have been originally described in References 8 and 9.

  • Either the source or the destination volume may be a local volume or a network volume
  • The access speed for the source and destination volumes may either be the same, or one may be significantly slower than the other. Further, the access speed can change depending upon a variety of factors such as network load, system load in terms of other application execution, resource usage e.g. I/O may switch from being cached to non cached and vice versa.
  • The optimum I/O size for the source and destination volumes may be either the same or significantly different
  • If both the source and destination volumes are on a Windows system, then the System Cache is involved in both reads and writes
  • The Windows team has spent a considerable amount of resources fine tuning the CopyFile and CopyFileEx APIs. Details are described in Reference 8, but the lesson to take away is the complexity of the issue and that further changes are probably forthcoming

Applications may

  • Use the CopyFile or CopyFileEx APIs and utilize the system cache
  • Use the CreateFile, ReadFile, WriteFile APIs and utilize the system cache for the source only, destination only, or both, or none.

Once you combine all of the various permutations and combinations offered by the above mentioned elements, the following situations can and do occur, when large files are being copied:

  • The system cache on the computer hosting the source file gets filled to a large extent with data from the source file. At the very least, this will affect other programs executing on that system.
  • The system cache on the computer system hosting the destination file gets filled with data for the destination file. This occurs fairly often since in the beginning, all of the destination file data is cached and thus writes appear to complete quickly. Once the destination file system cache hits a limit, disk writes (for flushing that cache) may occur slowly because the disk subsystem may be relatively slow
  • To complicate matters further, even when the data is flushed from system cache, it may be cached inside the block storage device (storage array)

When suspecting problems that may involve the System Cache, an administrator can

  • Inspect the application being used and switch to using a different application that explicitly does not use the System Cache. The Microsoft Server Performance Team Blog (Reference 7) explicitly suggests using Microsoft Exchange EseUtil as a file copy tool. The legal implications of using software shipped with Microsoft Exchange on a regular file server are beyond the scope of this document and best decided by your legal department
  • Use some other means to affect the System Cache e.g. use some other application that will consume up the System Cache, but not otherwise unduly load the system.
  • Attempt to administer the System Cache behavior utilizing in built utilities and/or registry keys

System caching can be controlled using administrative utilities and or a registry key.

To change the setting by editing the registry – as always beware of making registry changes and do so at your own risk – edit the registry key

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache

By default this DWORD is set to one (enabled) on Server SKUs and to zero (disabled)  on desktop SKUs.

On Windows XP, Microsoft provides a GUI to make the same changes which is preferable to making these changes via registry edits. Figure 2 shows the GUI that results from launching the Control Panel System Applet and then clicking the Advanced Tab

wincachefig2

Figure 2 Windows XP Control Panel System Applet Advanced Tab

Figure 3 shows the resulting System Cache size adjustment GUI when the advanced tab is clicked in Figure 2 on a Windows XP system

WinCacheFig3

Figure 3 Windows XP Control Panel System Applet Advanced Tab to adjust System Cache Size

Note that this GUI to change the System Cache size has been removed in Windows Vista.

Windows Server 2003, Windows Vista, and Windows Server 2008 Block Storage Cache Administration

Recall the earlier explanation of the bug in previous versions of Windows that ignored application requests to ensure data/metadata got committed to storage media and the subsequent fix made in Windows Server 2000 SP3 and also Windows XP SP2. To allow system administrators an informed choice, Microsoft made available a cache administration utility called DskCache.exe. This utility was only available by calling Microsoft PSS and could be obtained without incurring any monetary charge. To make it very clear that the DskCache utility should only be used in rare circumstances, Microsoft labeled it the “Power Protected Write Cache” and shipped it natively with Windows Server 2003 and higher versions of Windows. The new utility name emphasizes that it should be used only when the administrator is sure that the disk storage cache has a battery backup to ensure data integrity.

For Windows Server 2003 and higher versions of Windows, Microsoft has provided the equivalent of the DskCache.exe tool built into Windows. To use this feature:

  • Start Device Manager
  • Select the drive for which you wish to administer the caching policy
  • Select Properties
  • Click on Policies tab
  • Look for the option  “Enable write caching on the disk”  and make sure it is selected
  • And just below that, look for an option “Enable advanced performance”. This  option favors throughput/speed at the potential risk of data corruption.

The resulting GUI from following these steps is shown in Figure 4.

wincachefig4

 

Figure 4 – Windows Server 2003, Windows Vista & Windows Server 2008 disk caching policy administration

For Windows Server 2012, here is what the disk caching policy GUI looks like

WinCacheFig5

Figure 5 – Windows Server 2012 disk caching policy administration

Conclusion

This article described means by which application programmers can

  • Ensure that their file level data does not get cached in the Windows System Cache
  • Ensure that their file data does not get cached in the block storage layer and does get committed to storage media, given the correct hardware
  • Attempt to ensure, with no guarantee of success, that their file data does indeed get cached in the Windows System Cache

This article also describes means by which system administrators can attempt to ensure that data gets committed to storage media and does not get cached at either the System Cache or any block storage cache.

References

  1. Microsoft KB 241374 (http://support.microsoft.com/kb/241374/EN-US/) : Read and Write Access Required for SCSI Pass Through Request
  2. Microsoft KB 8373314: About Cache Manager in Windows Server 2003
  3. Microsoft KB 332023 Slow Disk Performance When Write Caching Is Disabled
  4. Nuances of Windows NT and SCSI disk performance article by Dilip Naik
  5. Force Unit Access Proposal
  6. Microsoft KB  870894 You receive a “Delayed Write Failed” error message in Windows XP Service Pack 2 or Windows XP Tablet PC Edition 2005
  7. Slow Large File Copy Issues – Microsoft Server Technical Support Performance Team Blog http://blogs.technet.com/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx
  8. Inside Vista SP1 File Copy Improvements – Mark Russinovich Blog http://blogs.technet.com/markrussinovich/
  9. Server Generates Delayed Errors Copying Very Large Files http://www.eggheadcafe.com/software/aspnet/32252624/server-generates-delayed.aspx
  10. Microsoft KB 920739  http://support.microsoft.com/kb/920739  Decreased Performance when copying files larger than 500 MB
  11. Serial ATA Program Revision 1.2 http://www.sata-io.org/documents/Interop_UnifiedTest_Rev1_2_v10_091707_000.pdf
  12. Disks, Lies, and damn disks http://perspectives.mvdirona.com/2008/04/17/DisksLiesAndDamnDisks.aspx
  13. Serial ATA in the Microsoft operating system environment http://www.microsoft.com/whdc/device/storage/serialATA_FAQ.mspx
  14. Enforcing Database Recoverability on Disks that lack Write-Through ftp://ftp.research.microsoft.com/pub/tr/TR-2008-36.pdf

 

 

 

By dilipcnaik | Also posted in Hyper-V, NTFS, SSD, VHDX, Windows 8 | Comments (2)

Tips for copying large VHD and VHDX files

January 21, 2012 – 6:09 pm

I have been copying VHD files for a while and have been partly putting up with some issues, but finally devoted the time to look at the issues a little closer.

I have 2 different systems running Windows Server 2008 R2, one with 4GB of physical RAM and one with 16GB of physical RAM. Obviously these are developer systems and certainly not production systems. The problem happens when I copy large VHD files, large being defined as anything significantly larger than the amount of physical RAM on the system that is doing the copying. So for these 2 systems, say anything larger than 20GB in size.

I used copy or xcopy with the default options to copy the large VHD file from one local volume to another. Due investigation showed that the physical RAM in use grew to 100% and stayed there while the copy was happening. I was careful not to run any other tasks on either system. It also appeared that the physical RAM was all being consumed by the Cache Manager.

Once the physical RAM usage hit 100% (as observed via PerfMon), I tried starting up NotePad. In a highly unscientific study that does not have enough data points, I found that at least half the times, the system “hiccupped” noticeably before NotePad ran – it took a while even for me to be able to type “Start Run notepad”.  There certainly were times, especially so early in the copy process, that the systems were highly responsive, even with physical RAM usage at 100%.

My speculation – and I have not done any investigation to verify – is that the physical RAM is being consumed by the Cache Manager for 2 different purposes. One is read ahead of the source VHD file, and the second is caching the data written into the destination VHD file. The cache manager is more willing – and able – to give up memory that has read ahead cached data. The cache manager has to work harder – and will consume system resources – when it needs to free up RAM that has cached data written to the destination VHD file.

Looking around, I noticed that the problem of copying large files on Windows seems to be a well known problem. The Microsoft performance team has written a blog “Slow large file copy issues”. Clearly they conclude that the solution is to copy the large file non cached. While the Microsoft Performance team suggests using Exchange EseUtil, I am not sure how many of my readers have access to that utility. I will also point out that I don’t understand the legal issues in taking an utility that ships with Microsoft Exchange and copying it to a non Microsoft Exchange system!

The simpler solution, again as the Microsoft Performance Team advocates, is using the Windows 7 or Windows Server 2008 R2 xcopy utility and making sure to specify the /J option indicating that the file should be opened and copied in a non-cached manner.

My same unscientific testing shows that xcopy /J works well in copying large VHD files. Someday, I will trace this to figure out whether xcopy /J performs non-cached I/O on both the source and destination VHD files, or on just one of them.

In the meanwhile, do certainly consider using xcopy /J to copy large VHD  and VHDX files.

By dilipcnaik | Also posted in Hyper-V, NTFS | Comments (1)

NTFS volume defragmentation – Part 3 – supply NTFS even more information

January 21, 2012 – 1:59 am

Earlier blogs described the problem of supplying NTFS enough information so that it could make an appropriate placement of a file on an NTFS volume to avoid fragmentation. We concluded that setting the file size information using a SetFileInformation API was not sufficient.

I updated my code – here is the pseudo code from the previous blog with one modification – this in attempt to provide NTFS even more information.

Open source file

Open destination file

GetFileSizeInformationForSourceFile();

SetFileSizeInformationForDestinationFile();

SetFileValidData();          // for destination file

While (!EndOfSourceFile)

{

                Read(SourceFile)

                CheckForEndOfFile

                WriteToDestinationFile (including write a partial buffer if any)

}

Close source file

Close destination file

The one modification is the call to SetFileValidData. As the MSDN documentation points out, this has security implications in that any data “left over” on the disk blocks allocated to the file can now be potentially available to anybody who can open the file. But since our copy applet will fully write all blocks of the file, this security concern does not apply in our case.

Contig now shows that the file is unfragmented!

Would it be possible for NTFS to determine the file size and allocated disk blocks accordingly without the need of this API? Only somebody from the NTFS team at Microsoft can answer that question.

But in the meanwhile, application developers have a way to make sure that they supply NTFS all the information it needs to try and avoid file fragmentation.

 

 

By dilipcnaik | Also posted in NTFS | Comments (0)

NTFS volume defragmentation – Part 2 – supplying NTFS more information

January 21, 2012 – 1:54 am

In an earlier blog, I described how developers typically ask NTFS to place a file on a volume without providing NTFS enough information to ensure the file placement does not lead to fragmentation. In particular, a typical application such as a file copy application does not provide the file size information before the first few blocks of the file are placed on disk.

Here is the same program from the earlier blog with a few additional steps

Open source file

Open destination file

GetFileSizeInformationForSourceFile();

SetFileSizeInformationForDestinationFile();

While (!EndOfSourceFile)

{

                Read(SourceFile)

                CheckForEndOfFile

                WriteToDestinationFile (including write a partial buffer if any)

}

Close source file

Close destination file

Obviously this is pseudo code that is meant to convey intentions and not code that can be compiled. The main step here is to determine the size of the source file, and then set the size of the destination file to that size – and especially so, do that before the first write occurs to the destination file.

After doing this, I inspected the destination file fragments using the SysInternals tool contig – and found that the file still tended to be fragmented. The expectation was that when the Cache Manager flushes and asks NTFS to commit some parts of the file to disk, NTFS could perhaps retrieve the file size from the open file handle – the file size being set via the SetFileSizeInformationForDestinationFile call. But this is clearly not the case. At least not for Windows 7 or Windows Vista or Windows XP that I tested with.

The third and last part of the blog will examine how to provide NTFS the information it needs to properly place the file on volume and avoid fragmentation.

By dilipcnaik | Also posted in NTFS | Comments (0)

Slow file copy or slow file transfer with various Windows versions 2k8, 2k8R2, 2k3

October 7, 2011 – 3:37 am

There are various posts in the Microsoft Technet File Systems and Storage forum (and other forums) indicating slow file copy/transfer speed. The details vary, but most of them involve either Windows Server 2008, Windows Server 2008 R2, or Windows Server 2003. This document is an attempt to coalesce all the various solutions into a single place. Note that the author of this document has not personally tested any of the solutions, but various posts in the forum indicate these to have solved the problem for some users.

Symptom: File copy or File transfer speed is slow, either to a local drive or a network drive, especially so when a Windows Server 2008 or newer Windows file server is involved.

Possible Solutions in no particular Order

  • Make sure both IPv6 and IPv4 are running on 2008 R2, even if the 2008 R2 server is the lone IPv6 device on the network! There may be alternate solutions but this solution has been reported to work [1] [2]
  • Tune TCP on client – set autotuning to off “netsh interface tcp set global autotuninglevel=disabled” [2]
  • Tune TCP on client and server – turn off receive side scaling “netsh interface tcp set global rss=disabled” [3]
  • Tune TCP disable large send offload [5] [17]
  • Tune TCP disable large receive offload [11]
  • Tune TCP – disable  offload [5] [7][9] [11] [16]

Edit the registry key Edit the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters; create a new DWORD value named DisableTaskOffload and set its value to 1

  • Use non-cached writes while copying the file e.g. run Windows 7 xcopy which supports a new /J switch to do non-cached writes. The Microsoft Performance Team uses the Microsoft Exchange utility eseutil to copy files using non-cached writes. However, some Microsoft customers may not have Microsoft Exchange. Also, the legalities of copying files from a Microsoft Exchange server to a File Server as the Microsoft Performance Team blog in Reference [6] suggests are beyond the scope of this compilation of solutions. [6] [14]
  • Install all hotfixes from Microsoft [8]
  • Don’t use File Explorer to do copy/paste – and if you do, ensure Remote Differential Copy is turned off [4]
  • If one of the drives involved in the file copy is an external USB drive, ensure that you are using the USB 2 protocol and not the older slower USB 1. Start Device Manager, find USB Controller, clock USB2 Enhanced Controller Host, and right click properties; make sure it is enabled. Repeat these steps for all USB2 Enhanced controller hosts.
  • Make sure drive compression is turned off [10]
  • Make sure antivirus is fully updated or if possible, remove antivirus solutions from the equation to see if that is the cause [12]
  • Install all updated drivers from vendors especially NIC vendors and RAID drivers [13] [16]

References:

  1. http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/7fd2a924-b1b7-400a-acab-398d09bdb462 What causes Windows 2008 R2 slow file and folder copy
  2. Server 2008 R2 core slow file copy http://social.technet.microsoft.com/Forums/en-US/winservercore/thread/b2462e82-710d-4357-827f-d97172e921f4
  3. Windows 2008 R2 File and Folder Copy very slow over 1000MBs link http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2general/thread/2e14aa58-cb9b-462c-8a9e-7b10be2de3cb/
  4. Why is Windows 7 still as slow as Vista with File Copy http://answers.microsoft.com/en-us/windows/forum/windows_7-performance/why-is-windows-7-still-as-slow-as-vista-with-file/a5b548ad-9102-40f3-a320-6cd1cabd1f3d?page=3
  5. Slow network file copy on Windows 7 http://serverfault.com/questions/85965/slow-network-file-copy-on-windows-7
  6. Slow large file copy issues  Microsoft Performance team blog http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx
  7. Slow file copy between Vista and Small Biz Server http://www.smallbizserver.net/Forums/tabid/53/aff/57/aft/62300/afv/topic/Default.aspx 
  8. Low performance when you transfer files from external IEEE 1394 device to server http://support.microsoft.com/kb/975500
  9. Very slow DFSR on Windows Server 2008 http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/e55022a4-db65-4dc9-a2f1-96b7f5d8e2fa
  10. Server 2008 File transfer is slow http://windows2008forum.com/topic/339-server-2008-file-share-transfer-is-slow/
  11. File Copy very slow to Windows Server 2008 R2 http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_Server_2008/Q_25149934.html
  12. IPS driver causes network slowdown http://community.norton.com/t5/Norton-Internet-Security-Norton/Bug-IPS-device-driver-causes-network-slowdown/td-p/151735
  13. Network transfers start fast then slow down http://hardforum.com/showthread.php?t=1566589
  14. Windows 2008 R2 large file copy uses all available memory and slows down http://www.networksteve.com/forum/topic.php/Windows_2008_R2_-_large_file_copy_uses_all_available_memory_and/?TopicId=15382&Posts=5  
  15. Slow LAN transfer Windows Server 2008 R2 http://social.technet.microsoft.com/Forums/en-US/w7itpronetworking/thread/96fe7d15-2bde-451b-8a12-796682a6cea8 – enable IPv6
  16. Windows Server 2008 R2 file/folder copy/paste very slow http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2general/thread/2e14aa58-cb9b-462c-8a9e-7b10be2de3cb 
  17. Windows 7 Large File copy painfully slow http://social.technet.microsoft.com/Forums/en/w7itpronetworking/thread/c30f6649-a0d9-4f5e-8671-904a8f5469bb 

 

By dilipcnaik | Also posted in SMB | Comments (2)