Case Sensitive File Names Windows(POSIX)

Question

Is there an undocumented way to work with case sensitive files in Windows(If not is this a bug - it doesn't appear to match OSX or Linux behavior)?

First enable case sensitivity in Windows. Then create 2 files named file using cygywin.

echo file > file
echo File > File

Now run

FilePrint["File"]
> file

Notice it doesn't refer to the correct file. I'm mainly interested in the Read,Write & Import.

Answer 1

This is decidedly not an answer, so please do not give me the bounty. It is also not a solution to the problem, for the simple reason that my C++, Win32, and LibraryLink skills are virtually nonexistent, and certainly not sufficient to write this robustly. Rather, my intent is simply to show that the answer given by Eric Towers contains some serious misconceptions about how Win32 and NT work, and is not a solution either.*

As I mentioned in my comments under his answer, the only reason why some (although not all) of Mathematica's file-handling functions fail to respect case-sensitivity is because they call the Win32 API function CreateFile without supplying the flag FILE_FLAG_POSIX_SEMANTICS. The Microsoft documentation for the flag is the following:

Access will occur according to POSIX rules. This includes allowing multiple files with names differing only in case, for file systems that support that naming. Use care when using this option, because files created with this flag may not be accessible by applications that are written for MS-DOS or 16-bit Windows.

(Note that it is not entirely clear whether there is any other semantic distinction apart from case-sensitivity. POSIX also does not normally lock files that are opened, whereas Windows does, which seems to me to fall under the category of semantics.)

Microsoft is absolutely correct that this capability (and creating and using case-distinguished files in general) should be approached with some care. Most Win32 applications do not supply this flag, even though there is nothing preventing them from doing so. This includes Explorer and many other utilities provided with Windows itself. These programs will thus not be able to distinguish between filenames that differ only in their case, and in fact they normally behave as if you had specified all of the applicable files. This can be a serious problem when the operation one wants to perform is e.g. deletion.

With that caveat, let us proceed to the program. For this I will use some Microsoft example code available on MSDN.

Let us note explicitly:

• This is Win32 code. Not POSIX. Not NT. (Most of the NT API is undocumented anyway.)
• The function called is CreateFile, part of the Win32 API. Not NtCreateFile (which is case-sensitive by default, as NT is in general).
• It is a user-mode application, not a driver. It does not run in kernel mode.

First we create the files. I used Cygwin, consistent with the question:

Olek@core2 /mnt/c/Users/Olek/Desktop/fileprint
$ echo file > file

Olek@core2 /mnt/c/Users/Olek/Desktop/fileprint
$ echo File > File

Now compile the program (I used MinGW g++):

C:\Users\Olek\Desktop\fileprint>g++ -D__in= -o fileprint.exe fileprint.cpp

Try it out:

C:\Users\Olek\Desktop\fileprint>fileprint.exe File

Error code:     0
Number of bytes:        4
Data read from File (4 bytes):
file

What's that? It doesn't work? We must have forgotten to specify FILE_FLAG_POSIX_SEMANTICS! Let's add it (on line 53):

FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED, // normal file

becomes

FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_POSIX_SEMANTICS,

After recompiling:

C:\Users\Olek\Desktop\fileprint>fileprint.exe File

Error code:     0
Number of bytes:        4
Data read from File (4 bytes):
File

Well, look at that! Apparently the presence or absence of FILE_FLAG_POSIX_SEMANTICS really does make all the difference. Who'd have guessed?

But still, one can reasonably ask, how does this relate to Mathematica? The answer is that it is obviously possible to write the same C++/Win32 code as a DLL and load it into Mathematica using LibraryLink. With the aid of the (new-in-9) functions for defining stream methods, it would be relatively straightforward to manipulate files with arbitrarily-cased names as Mathematica streams. Relatively straightforward, that is, for someone who is more confident of their ability to write correct Win32 code than I am of mine.

Finally, the Microsoft example code is given below for reference, in case the page on which it currently exists is changed or moved in future:

#include <windows.h>
#include <tchar.h>
#include <stdio.h>
#include <strsafe.h>

#define BUFFERSIZE 5
DWORD g_BytesTransferred = 0;

void DisplayError(LPTSTR lpszFunction);

VOID CALLBACK FileIOCompletionRoutine(
  __in  DWORD dwErrorCode,
  __in  DWORD dwNumberOfBytesTransfered,
  __in  LPOVERLAPPED lpOverlapped
);

VOID CALLBACK FileIOCompletionRoutine(
  __in  DWORD dwErrorCode,
  __in  DWORD dwNumberOfBytesTransfered,
  __in  LPOVERLAPPED lpOverlapped )
 {
  _tprintf(TEXT("Error code:\t%x\n"), dwErrorCode);
  _tprintf(TEXT("Number of bytes:\t%x\n"), dwNumberOfBytesTransfered);
  g_BytesTransferred = dwNumberOfBytesTransfered;
 }

//
// Note: this simplified sample assumes the file to read is an ANSI text file
// only for the purposes of output to the screen. CreateFile and ReadFile
// do not use parameters to differentiate between text and binary file types.
//

void __cdecl _tmain(int argc, TCHAR *argv[])
{
    HANDLE hFile; 
    DWORD  dwBytesRead = 0;
    char   ReadBuffer[BUFFERSIZE] = {0};
    OVERLAPPED ol = {0};

    printf("\n");
    if( argc != 2 )
    {
        printf("Usage Error: Incorrect number of arguments\n\n");
        _tprintf(TEXT("Usage:\n\t%s <text_file_name>\n"), argv[0]);
        return;
    }

    hFile = CreateFile(argv[1],               // file to open
                       GENERIC_READ,          // open for reading
                       FILE_SHARE_READ,       // share for reading
                       NULL,                  // default security
                       OPEN_EXISTING,         // existing file only
                       FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED, // normal file
                       NULL);                 // no attr. template

    if (hFile == INVALID_HANDLE_VALUE) 
    { 
        DisplayError(TEXT("CreateFile"));
        _tprintf(TEXT("Terminal failure: unable to open file \"%s\" for read.\n"), argv[1]);
        return; 
    }

    // Read one character less than the buffer size to save room for
    // the terminating NULL character. 

    if( FALSE == ReadFileEx(hFile, ReadBuffer, BUFFERSIZE-1, &ol, FileIOCompletionRoutine) )
    {
        DisplayError(TEXT("ReadFile"));
        printf("Terminal failure: Unable to read from file.\n GetLastError=%08x\n", GetLastError());
        CloseHandle(hFile);
        return;
    }
    SleepEx(5000, TRUE);
    dwBytesRead = g_BytesTransferred;
    // This is the section of code that assumes the file is ANSI text. 
    // Modify this block for other data types if needed.

    if (dwBytesRead > 0 && dwBytesRead <= BUFFERSIZE-1)
    {
        ReadBuffer[dwBytesRead]='\0'; // NULL character

        _tprintf(TEXT("Data read from %s (%d bytes): \n"), argv[1], dwBytesRead);
        printf("%s\n", ReadBuffer);
    }
    else if (dwBytesRead == 0)
    {
        _tprintf(TEXT("No data read from file %s\n"), argv[1]);
    }
    else
    {
        printf("\n ** Unexpected value for dwBytesRead ** \n");
    }

    // It is always good practice to close the open file handles even though
    // the app will exit here and clean up open handles anyway.

    CloseHandle(hFile);
}

void DisplayError(LPTSTR lpszFunction) 
// Routine Description:
// Retrieve and output the system error message for the last-error code
{ 
    LPVOID lpMsgBuf;
    LPVOID lpDisplayBuf;
    DWORD dw = GetLastError(); 

    FormatMessage(
        FORMAT_MESSAGE_ALLOCATE_BUFFER | 
        FORMAT_MESSAGE_FROM_SYSTEM |
        FORMAT_MESSAGE_IGNORE_INSERTS,
        NULL,
        dw,
        MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
        (LPTSTR) &lpMsgBuf,
        0, 
        NULL );

    lpDisplayBuf = 
        (LPVOID)LocalAlloc( LMEM_ZEROINIT, 
                            ( lstrlen((LPCTSTR)lpMsgBuf)
                              + lstrlen((LPCTSTR)lpszFunction)
                              + 40) // account for format string
                            * sizeof(TCHAR) );

    if (FAILED( StringCchPrintf((LPTSTR)lpDisplayBuf, 
                     LocalSize(lpDisplayBuf) / sizeof(TCHAR),
                     TEXT("%s failed with error code %d as follows:\n%s"), 
                     lpszFunction, 
                     dw, 
                     lpMsgBuf)))
    {
        printf("FATAL ERROR: Unable to output error code.\n");
    }

    _tprintf(TEXT("ERROR: %s\n"), (LPCTSTR)lpDisplayBuf);

    LocalFree(lpMsgBuf);
    LocalFree(lpDisplayBuf);
}

---

* Eric's workaround of using the short file names was necessary because, in the case he describes, he had obviously handled the drag-and-drop operation incorrectly and did not retrieve the file names (which include the extensions) but rather the text displayed by Explorer, which is merely part of the user interface. Handling drag-and-drop input is not trivial in Win32, but even so, Microsoft cannot really be blamed if it is done wrongly, because all of the documentation and ample examples are provided. And why should we not rely on short file names? Because they don't necessarily even exist! Short file names are completely optional, and implemented as hard links, on NTFS volumes.

Answer 2

I don't have a Windows machine to test this on anymore. I've seen this problem before in the form: .Net programs and DOS batch files cannot (or, at least, could not) distinguish upper and lower case filenames, so SFNs (8.3 filenames) had to be used for disambiguation or the ignore case and hide extensions options had to be disabled in File Explorer and/or the Registry. There was no way to get case discrimination because the userland filesystem interfaces refused to recognize a difference.

(We had to un-Hide Extensions of Known Filetypes so that programs could discriminate foo.tar and foo.tgz, both of which were "Foo" to batch files and other clients of the filesystem. SFNs could be used, but were unpredictable; how many "foo"s were previously in that directory?)

Filename prettification is the Windows filesystem's method of making many files have the same apparent name and making it impossible for programs to use the names the users are seeing. Attachment named "ARCHIVE.ZIP"? Well it's "Archive" (or its SFN) in every program that touches it through the userland FS interfaces. (... unless you unhide extensions and turn off prettification). NTFS added mixed case filenames, but the userland interfaces had to normalize names so that bazillions of legacy programs would continue to do the right thing.

Edit: Still don't have Windows running (and that's not about to change). However:

• Q100625: "In NTFS, you can create unique file names, stored in the same directory, that differ only in case. For example, the following filenames can coexist in one directory on an NTFS volume: "CASE.TXT", "case.txt", "case.TXT". However, if you attempt to open one of these files in a Win32 application, such as Notepad, you would only have access to one of the files, regardless of the case of the filename you type in the Open File dialog box.
• SuperUser question 266110: (Note that this is the article OP links.) "If it uses WIN32 API (which 99.9% of the software does) it will be case-insensitive whatever you do. All built-in software in Windows (like Explorer, command prompt, Internet Explorer and etc.) and all consumer software out there uses WIN32 and is always case-insensitive."
• Understanding case sensitivity: "It is worth noting that most existing Win32 applications will not set the FILE_FLAG_POSIX_SEMANTICS flag or the FIND_FIRST_EX_CASE_SENSITIVE flag when calling these APIs, or will rely on other APIs that internally do not. Because of this, the OBJ_CASE_INSENSITIVE flag will always end up being set by the Win32 subsystem in the OBJECT_ATTRIBUTES structures that it creates to call NT Native APIs. This always enforces insensitivity for such applications."
• MS Threats and Vulnerabilities Mitigation: "Vulnerability: Because Windows is case-insensitive but the POSIX subsystem will support case sensitivity, failure to enable this policy setting would make it possible for a user of that subsystem to create a file with the same name as another file but with a different mix of upper and lower case letters. Such a situation could potentially confuse users when they try to access such files from normal Win32 tools because only one of the files will be available."
• Cygwin: Special Filenames: And Cygwin specifically mentions that you can turn off case sensitivity in the NT Executive, but you can't fix userland: "However, there are limitations: while two programs Abc.exe and aBc.exe can be created and accessed like other files, starting applications is still case-insensitive due to Windows limitations and so the program you try to launch may not be the one actually started."

Potentially relevant for resolving OP's difficulties:

• Cygwin: Special Filenames: "[I]f you use case-sensitivity on the /cygdrive prefix, your shell might claim that it can't find Windows commands like attrib or net. To ease the pain, the /cygdrive path is case-insensitive by default and you have to use the 'posix=1' setting explicitly in /etc/fstab or /etc/fstab.d/$USER to switch it to case-sensitivity, or you have to make sure that the native Win32 %PATH% environment variable is using the correct case for all paths throughout."
• Understanding case sensitivity: Probably less relevant for Cygwin use: "Once support for case sensitivity system-wide has been established, the ability to use files in a case sensitive way is then determined by the file system in use on a per-volume basis. This is established by the presence of the FILE_CASE_SENSITIVE_SEARCH file system attribute flag for a volume, documented as meaning that 'the specified volume supports case-sensitive file names'."

So, first, regardless of one commenter's persistent claims to the contrary, this is not a UI problem in File Explorer. It is a uniform problem across non-POSIX applications. Second, fixing it for Cygwin will likely break Win32 access to case-distinct-only filenames.

A bit of clarification of my first text: We were taking piles of data files from a POSIX compliant network filesystem to various Windows client machines (XP x32, XP x64, Server 2003, 7 x32, 7 x64) and batch processing subpiles. Relevant observations: third-party Win32 applications were unable to resolve case-distinct-only filenames. If filename extensions were hidden, the multi-select drag-and-drop files passed by the shell and File Explorer (You'd think those are the same thing and you'd almost be right.) did not include extensions, but the files were not openable with extensionless filenames. (Batch file receives list of dozens of filenames, all without their extensions. Batch file tries to pass them to other programs for processing. Other programs repeatedly observe no such files exist. Programmer thanks Microsoft for conflating presentation markup and content markup. Of course, filename prettification also altered the filenames passed to the batch files, repeating MS's error and leading to another prerequisites check in the batch processing. (Yeah, the batch processing eventually became about 70% "check that the user's computer will not do something stupid and change that if possible" and 30% "get any actual work done".))

Finally, from the OP's linked article: "I was trying to emulate a Linux case-sensitive file-system for debugging purposes during development. It seems the better approach is to use VMWare with an instance of Ubuntu for development." Yes, if I could have peered 6 months into the future at the time we started that batch processing, I would have started working on a batch processing VM (running some Linux distro) to distribute to the client machines.

Edit: All that having been said... What's the resolution? Scroll back to the top of this answer. The answer is to use short filenames (SFNs). These are guaranteed to be unique and to work in Win32. For more detail, including how to use NetLink to make functions to retrieve these names from the filesystem, see this answer to a different question. Short version (via cut and paste):

Needs["NETLink`"]
InstallNET[];
$longPrefix = "\\\\?\\";
getLastError = DefineDLLFunction["GetLastError", "kernel32.dll", "DWORD", {}];

toLongPath[filename_] := $longPrefix <> ExpandFileName[filename]

fromLongPath[filename_] := StringReplace[filename, StartOfString~~$longPrefix -> ""]

getShortPathNameW =
  DefineDLLFunction["GetShortPathNameW"
  , "kernel32.dll", "DWORD", {"LPCTSTR", "System.Text.StringBuilder", "DWORD"}
  , MarshalStringsAs -> "Unicode"
  ];

shortPath[name_String] :=
  NETBlock @ Module[{result = NETNew["System.Text.StringBuilder", 260]}
  , getShortPathNameW[toLongPath@name, result, result@Capacity]
  ; getLastError[] /. 0 :> fromLongPath@result@ToString[]
  ]

then use, for instance,

FilePrint[shortPath["some long case sensitive filename"]]