January 2002/Encapsulating Virtual Memory in Windows

Features

Joey Rogers

Encapsulating Virtual Memory in Windows

This class gives you convenient, direct control over memory-mapped files in Win32.

Virtual memory is a powerful feature in Windows. It extends available memory resources beyond their physical limits, provides a flat memory addressing scheme for all processes, and hides the complicated mechanics of managing memory from applications. Virtual memory works by mapping a large memory address space (possibly larger than the available physical memory) to a file on disk. The virtual memory manager swaps pages of memory to and from disk to give an application the appearance that its entire address space exists. While virtual memory is essential to the normal operation of any application, it may be overlooked as a programming tool in Windows development. Using the operating system’s virtual memory manager to map any file to a virtual memory address space allows an application to treat the mapped file as physical memory. This ability can simplify coding when you must process data from memory and from files in the same way or when your application needs to be scalable beyond physical memory limits. The CVirtMemFile class (Listing 1) presented in this article encapsulates the commonly used virtual memory functionality to simplify its use in application development.

The Win32 API provides several functions to utilize Windows’ virtual memory manager. These functions are not overly complex, but their usage can be simplified significantly by encapsulating the basic functionality of mapping a file on disk to virtual memory in a class. The CVirtMemFile class shown in Listing 1 is small yet generic enough to use in a variety of applications. Its purpose is to make mapping a file to virtual memory and using it as easy as opening and using a file.

The CVirtMemFile class provides five methods:

long Open( const char filename[], long mode )

void Close( void )

long GetStatus( void )

DWORD GetFileSize( void )

const LPVOID GetBasePtr( void )

The Open method maps a file on disk to virtual memory. The filename parameter is the full path to the file you wish to map. The mode parameter specifies for the file to be Read Only or Read Write and must have one of the following values provided by the CVirtMemFile class as a public enumeration:
READ_ONLY_MODE = 0
READ_WRITE_MODE = 1
Open returns a true value if the file is mapped successfully and a false value otherwise.

The Close method unmaps and closes the open file.

The GetStatus method returns the current status with one of the following enumerated values:
FILE_NOT_OPEN = 0
FILE_OPEN = 1
The GetFileSize method returns the size in bytes of the open file.

The GetBasePtr method returns the base address of the file as a void pointer. This pointer is valid only if Open successfully maps a file to virtual memory. The base memory address points to the first byte of the file in virtual memory. The beauty of using a virtual memory-mapped file is that it is used just like dynamically allocated memory. There is no seeking, reading data to a buffer, or writing data from a buffer. As you reference memory relative to the base pointer, the appropriate portions of the file are automatically paged into memory as they are needed. If the file is opened in Read Write mode, any changes made to the virtual memory will also be made to the file.

Since the focus of this article is to illustrate how easy it is to use virtual memory in your applications with the CVirtMemFile class, the details of directly using the Win32 API virtual memory functions (CreateFileMapping, MapViewOfFile, and UnmapViewOfFile) are omitted. Information on these functions can be found at: <http://msdn.microsoft.com/library/psdk/winbase/filemap_0583.htm>.

Examples

The following code segment maps a file to virtual memory and accesses the file’s contents like an array of characters. This example is trivial, but it shows how easy it is to use virtual memory.
CVirtMemFile vmFile;

vmFile.Open("c:\\temp\\test.dta",
  CVirtMemFile::READ_ONLY_MODE);

char value;

for (int i=0; i<100; i++)
{
  value=
    static_cast<char *>
      (vmFile.GetBasePtr())[i];
  cout << "Pos: "
       << i
       << "  Value: "
       << (long)value
       << endl;    
  }

vmFile.Close();
Listing 2 contains a more practical example. It illustrates how a potentially complex task can be performed with the CVirtMemFile class. Sorting large or very large files is a task with a scalability problem. The sorting algorithms normally used with smaller amounts of data are still applicable, but the techniques used to implement these algorithms usually do not scale to cope with large amounts of data. If you can sort files containing 100,000 records with your favorite implementation, can you also sort files containing 50 million or even more? Using the CVirtMemFile class, a file can be mapped to virtual memory and sorted just like it were an array in memory. The following code segment demonstrates how the C++ provided qsort function (which is designed to work with data in memory) can easily sort a file of any size:
CVirtMemFile vmFile;

if (vmFile.Open(filename, CVirtMemFile::READ_WRITE_MODE))
  {
  unsigned long count=
    vmFile.GetFileSize()/sizeof(sampleRec);
  qsort( vmFile.GetBasePtr(),
    count, sizeof(sampleRec), compareSampleRec );
  vmFile.Close();
  printf("%ld records sorted.\n", count);
  }
The first step is to map the file on disk with the Open method. Since we are rearranging the data as it is sorted, the file must be opened in the Read Write mode. The second step is to sort the file with the qsort function, which requires a pointer to the data in memory (returned by GetBasePtr), the number of records to sort (GetFileSize’s return value divided by the record size), the size of the records, and a comparison function. The last step is to close the file with the Close method. While not all sorting algorithms will work well with virtual memory, the Quick Sort algorithm used by qsort and its locality of reference characteristic work very well with paged memory. However, the limitations of the Quick Sort algorithm (such as poor performance with ordered values) still apply.

Conclusion

By directly using the Win32 API or by using a class like CVirtMemFile, the powerful features provided by the operating system’s virtual memory manager can be directly leveraged by your applications. Whether you need tools to solve a scalability problem or to simply write homogenous code to handle data in memory and on disk, virtual memory may be useful to your application’s development; both of these benefits are seen in the sorting example. Applications using virtual memory not only exploit some of the best features offered by Windows, they will be cleaner, smaller, and more powerful.

Joey Rogers is an application developer for Graphic Technologies, Inc. in Huntsville, Alabama. He can be reached at Joey.Rogers@gti-us.com.