Category Archives: NTFS

Intel Ultrabooks, SSD based laptops, and file system needs

January 21, 2012 – 9:08 pm

This blog is partly triggered by the new Resilient File System (ReFS) that Microsoft just announced for Windows 8. At least for now, the new file system appears to be more for servers than any laptops or tablets, and that too, particularly SSD based laptops and tablets. More about the ReFs in some other blogs.

For the record, I believe Intel holds a trademark on the term  Ultrabook.

I am not sure my Windows 7 and Windows 8 Developer Preview NTFS based laptops need a better metadata checksum mechanism, let alone a better user data checksum mechanism. But here is what I do believe my NTFS based laptops (Win 7 and presumably also the Win 8 based laptops) need, especially so when the hard disk is an SSD.

  • Can the OEMs please stop bundling and/or stop offering a disk defragmentation utility with SSD based systems? SSD based volumes do not need to be defragmented, indeed, they reduce the life of the SSD! Further, maybe the Microsoft OEM division, especially so for Windows 8, as well as the Intel Ultrabook division can do something about this?
  • Microsoft, thank you for disabling the built in defrag code in Windows 7 when an SSD based NTFS volume is detected. Hopefully the same is true in Windows 8 as well.
  • SSDs have this need for the unused data blocks to be erased. That is just the nature of the physics involved. Doing so makes the writes faster. After a user buys an SSD based laptop, after a while, all of the blocks have been written and it would be advantageous for the disk firmware to know which blocks are unused as seen by the file system (NTFS) so that it can go ahead and erase them. The idea is that the disk blocks are erased and ready for me to write to when I download the latest movie, whether from iTunes, YouTube, or my DVD drive. Enter the Windows 7 “TRIM” command where Windows 7 passes down the information as to what disk blocks it just “released” and that can be erased. The problem is that it is not clear which drive vendors make use of that TRIM command? Or when a new version of the driver firmware makes use of that TRIM command, do the laptop OEMs bother to use that firmware? I understand there are profits to be made, and that goal may at least temporarily result in a situation where a Windows 7 SSD volume is simply ignoring the TRIM command. It would be interesting to get those statistics – whether it be from Microsoft, or a drive vendor, or a laptop OEM, or for that matter Intel for its Ultrabook branded OEMs.
  • What would be even more useful would be to have the same insight for Windows 8 SSD based laptops and tablets, whenever those are commercially available. While I will not buy a laptop or tablet simply because it makes proper use of the TRIM command, I do know that 64GB and 128 GB SSDs tend to fill very quickly, and hence the TRIM command will help. So it is certainly an important consideration. Perhaps this is one way an OEM can differentiate their offering.

More about ReFS and SSD in a new blog  at a later date.

Advertisement
By dilipcnaik | Also posted in SSD, Windows 8 | Comments (0)

Tips for copying large VHD and VHDX files

January 21, 2012 – 6:09 pm

I have been copying VHD files for a while and have been partly putting up with some issues, but finally devoted the time to look at the issues a little closer.

I have 2 different systems running Windows Server 2008 R2, one with 4GB of physical RAM and one with 16GB of physical RAM. Obviously these are developer systems and certainly not production systems. The problem happens when I copy large VHD files, large being defined as anything significantly larger than the amount of physical RAM on the system that is doing the copying. So for these 2 systems, say anything larger than 20GB in size.

I used copy or xcopy with the default options to copy the large VHD file from one local volume to another. Due investigation showed that the physical RAM in use grew to 100% and stayed there while the copy was happening. I was careful not to run any other tasks on either system. It also appeared that the physical RAM was all being consumed by the Cache Manager.

Once the physical RAM usage hit 100% (as observed via PerfMon), I tried starting up NotePad. In a highly unscientific study that does not have enough data points, I found that at least half the times, the system “hiccupped” noticeably before NotePad ran – it took a while even for me to be able to type “Start Run notepad”.  There certainly were times, especially so early in the copy process, that the systems were highly responsive, even with physical RAM usage at 100%.

My speculation – and I have not done any investigation to verify – is that the physical RAM is being consumed by the Cache Manager for 2 different purposes. One is read ahead of the source VHD file, and the second is caching the data written into the destination VHD file. The cache manager is more willing – and able – to give up memory that has read ahead cached data. The cache manager has to work harder – and will consume system resources – when it needs to free up RAM that has cached data written to the destination VHD file.

Looking around, I noticed that the problem of copying large files on Windows seems to be a well known problem. The Microsoft performance team has written a blog “Slow large file copy issues”. Clearly they conclude that the solution is to copy the large file non cached. While the Microsoft Performance team suggests using Exchange EseUtil, I am not sure how many of my readers have access to that utility. I will also point out that I don’t understand the legal issues in taking an utility that ships with Microsoft Exchange and copying it to a non Microsoft Exchange system!

The simpler solution, again as the Microsoft Performance Team advocates, is using the Windows 7 or Windows Server 2008 R2 xcopy utility and making sure to specify the /J option indicating that the file should be opened and copied in a non-cached manner.

My same unscientific testing shows that xcopy /J works well in copying large VHD files. Someday, I will trace this to figure out whether xcopy /J performs non-cached I/O on both the source and destination VHD files, or on just one of them.

In the meanwhile, do certainly consider using xcopy /J to copy large VHD  and VHDX files.

By dilipcnaik | Also posted in File copy, Hyper-V | Comments (1)

NTFS volume defragmentation – Part 3 – supply NTFS even more information

January 21, 2012 – 1:59 am

Earlier blogs described the problem of supplying NTFS enough information so that it could make an appropriate placement of a file on an NTFS volume to avoid fragmentation. We concluded that setting the file size information using a SetFileInformation API was not sufficient.

I updated my code – here is the pseudo code from the previous blog with one modification – this in attempt to provide NTFS even more information.

Open source file

Open destination file

GetFileSizeInformationForSourceFile();

SetFileSizeInformationForDestinationFile();

SetFileValidData();          // for destination file

While (!EndOfSourceFile)

{

                Read(SourceFile)

                CheckForEndOfFile

                WriteToDestinationFile (including write a partial buffer if any)

}

Close source file

Close destination file

The one modification is the call to SetFileValidData. As the MSDN documentation points out, this has security implications in that any data “left over” on the disk blocks allocated to the file can now be potentially available to anybody who can open the file. But since our copy applet will fully write all blocks of the file, this security concern does not apply in our case.

Contig now shows that the file is unfragmented!

Would it be possible for NTFS to determine the file size and allocated disk blocks accordingly without the need of this API? Only somebody from the NTFS team at Microsoft can answer that question.

But in the meanwhile, application developers have a way to make sure that they supply NTFS all the information it needs to try and avoid file fragmentation.

 

 

By dilipcnaik | Also posted in File copy | Comments (0)

NTFS volume defragmentation – Part 2 – supplying NTFS more information

January 21, 2012 – 1:54 am

In an earlier blog, I described how developers typically ask NTFS to place a file on a volume without providing NTFS enough information to ensure the file placement does not lead to fragmentation. In particular, a typical application such as a file copy application does not provide the file size information before the first few blocks of the file are placed on disk.

Here is the same program from the earlier blog with a few additional steps

Open source file

Open destination file

GetFileSizeInformationForSourceFile();

SetFileSizeInformationForDestinationFile();

While (!EndOfSourceFile)

{

                Read(SourceFile)

                CheckForEndOfFile

                WriteToDestinationFile (including write a partial buffer if any)

}

Close source file

Close destination file

Obviously this is pseudo code that is meant to convey intentions and not code that can be compiled. The main step here is to determine the size of the source file, and then set the size of the destination file to that size – and especially so, do that before the first write occurs to the destination file.

After doing this, I inspected the destination file fragments using the SysInternals tool contig – and found that the file still tended to be fragmented. The expectation was that when the Cache Manager flushes and asks NTFS to commit some parts of the file to disk, NTFS could perhaps retrieve the file size from the open file handle – the file size being set via the SetFileSizeInformationForDestinationFile call. But this is clearly not the case. At least not for Windows 7 or Windows Vista or Windows XP that I tested with.

The third and last part of the blog will examine how to provide NTFS the information it needs to properly place the file on volume and avoid fragmentation.

By dilipcnaik | Also posted in File copy | Comments (0)

Notes on Windows 8 VHDX file format

November 16, 2011 – 1:16 am

Last year, I wrote an article on Technet regarding the performance issues with the dynamic VHD file format. That article can be fund here . The main point made in the article was that the dynamic VHD file format ensured that 3 out of every 4 2MB data blocks were misaligned in the sense that they did not begin on a 4096 byte boundary. With the future of large sector disks, this will become even more important.

With Windows 8, Microsoft appears to have addressed this issue with the new VHDX file format. As disclosed in some talks at the BUILD conference, the VHDX file format

  • Extends VHD file based volumes to 16TB from the current 2TB limit
  • Improves performance
  • Makes the file format more resilient to corruption

While the best way to evaluate these features would be to compare the VHDX file format with the VHD file format, unfortunately Microsoft has not yet released the VHDX file format.

I decided to look into this a little further and wrote a little applet that writes 4kb at offset zero in a file, and 4kb at offset 2MB in a file. I ran the program twice

  • Once with the file housed in a VHD based volume
  • Once with the file housed in a VHDX based volume

 Given that the VHD file format in Windows Server 2008 R2 uses 2MB data blocks, the applet effectively writes at the beginning of the first 2 data blocks.

While I plan to analyze the I/Os in more detail, for now, there is one interesting observation. Here are the I/Os on the VHD file traced using Process Monitor in the Windows Server 2008 R2 parent.

And here are the I/Os traced with the file housed in a VHDX based volume

The immediate observation is that there are some 512 byte writes on the VHD based volume whereas the VHDX based volume shows no 512 byte writes at all. The 512 byte writes are presumably the writes to the Sector bitmap of the VHD (file format). While the conclusion is not definitive, one is drawn to believe that the 512 byte sector bitmap has been replaced and/or moved – maybe the sector bitmaps are now all together and not interspersed between data blocks.

More on this topic in a later blog.

By dilipcnaik | Also posted in Hyper-V, Windows 8 | Comments (0)

NTFS improvements for high availability in Windows 8

October 6, 2011 – 11:07 pm

Caveat: While efforts are made to make the content accurate, you peruse the contents of this blog at your own risk and responsibility. Any actions you take, or do not take based upon the blog contents are also your own responsibility.

Microsoft has strived to improve NTFS resiliency and availability through the various versions of Windows. This post focusses on one aspect: the availability of an NTFS volume when corruption is detected.

Prior to Windows Server 2008 R2, when an NTFS volume corruption is detected, the volume is taken offline and a process known as Chkdsk run to fix the corruption. The amount of time Chkdsk took to run was somewhat proportional to the number of files and directories within the NTFS volume. Thus smarter administrators sized NTFS volumes in a way that limited the maximum amount of files and directories within a given volume. For some corporations, this meant additional NTFS volumes with the associated management overhead. Starting with Windows NT4 and up to Windows 2008, Microsoft invested in efforts to reduce the amount of time it took to run Chkdsk.

With Windows Server 2008 R2, Microsoft changed tack and focused on improving NTFS volume availability by:

  1. Further reducing the amount of time it took Chkdsk to run. In particular, this was done by reading NTFS meta data in bigger chunks and by better caching it, thus avoiding disk IOPS. Note that most of the time spent in Chkdsk is scanning the volume to detect issues, rather than in actually fixing the issues. This is where the time taken to run Chkdsk being proportional to the number of files & directories arises from.
  2. Reducing the frequency with which Chkdsk needed to run by better detection of whether the volume was corrupted or not
  3. Fixing at least some of the corruption issues without taking the volume off line, a process Microsoft termed “self healing”

Windows 8 takes this logical progression a step further

  1. Scanning to detect errors is done while the volume is online and available.
  2. As in step 3 above, self healing fixes some of the corruption issues. Indications are that a further set of errors is fixed by self healing in Windows 8 as compared to prior versions of Windows
  3. The errors that cannot be corrected are logged
  4. The volume is taken offline and the errors identified in phase C are corrected. This implies the volume is unavailable only for the duration taken to fix the errors logged in phase C.
  5. In rare instances, the scanning determines that the corruption is too serious and needs a full offline scan, just as in Windows Server 2008 R2 and earlier versions of Windows.

Diagram 1 is a slide from a talk at the Windows 8 BUILD conference and the whole slide deck is available at this link

The item to note from the diagram is that the amount of time taken to fix a corrupted volume is now independent of the number of files and directories within the volume with one caveat. The caveat is that the corruption is not of an extremely serious nature – those still require the old style full scan while the volume is unavailable.

Microsoft now supports NTFS volumes of up to 24TB in size with Windows 8 and effectively, the number of files and directories within a given volume may also be increased without fear of the volume being unavailable for a long time due to corruption.

The changes in NTFS imply that an NTFS volume has more states in Windows 8 compared to prior versions of Windows. Tables 1 and 2 summarize the NTFS volume states in Windows 8 and prior versions of Windows

Table 1 Windows Server 2008R2/Windows 7 and earlier version of Windows NTFS volume states

NTFS Volume State 2k8R2/Win7 & earlier versions of Windows Description
Clean/Healthy “All is well” – no corrective action needed
Dirty/Potentially needs repair Can result even from an incorrect shutdown – indicates a scan and potential fixing needed

 Table 2 Windows 8 NTFS Volume states

Windows 8 NTFS volume states Description
Clean/Healthy “All is well” – no corrective action needed
Scan Needed/Dirty Indicates scan and potential fixing needed, but the scan will be done while the volume is mounted
Spotfix needed Scan has identified the errors, some were “self healed”, some marked for quick offline fixing, the volume needs to be briefly taken offline and the logged errors fixed
Full Chkdsk needed  The errors are too serious to spot fix and an old style full scan with the volume being offline for a longer time is needed. Presumably this happens only rarely.

 Note that NTFS volumes are still “compatible” – you can take an NTFS volume created with prior versions of Windows and mount it on a Windows 8 system. You can also take an NTFS volume created with Windows 8 and use it with earlier versions of Windows.

By dilipcnaik | Also posted in Windows 8 | Comments (1)