Dr. Dobb's Journal April 1997
In the early days of digital video, there was a good deal of scoffing at the grainy, jerky animations passed off as multimedia. Having come to PC-based multimedia from a production business focused on broadcast audio and video formats, I did my fair share of guffawing. But seeing the possibilities of interactive media as compelling, I decided to stick with PC multimedia until the hardware and software platforms could support a reasonable standard.
But even with today's high-performance PCs and multimedia environments, Macintosh and Windows apps billed as "multimedia" are still unresponsive, jerky, and unstable. Why? In part, because the average CD-ROM isn't fast enough to keep the data rate to an acceptable level, and seek delays cause problems when changing sequences. Additionally, layers of software, in the form of multimedia-authoring programs, sit on top of the multimedia engine and eat up CPU cycles.
In this article, I'll address problems such as these by presenting a reusable framework for preparing and presenting a high-performance multimedia production that's based on Apple's QuickTime multimedia engine. The benefit of using the QuickTime engine is that it is not only universally available on both Macintosh and Windows, but offers a long-term solution to media storage and playback demands. With its handling of multitrack video of practically any depth and resolution, mattes, sprites, text, real-time effects, multitrack hi-fi sound, MIDI (and more to follow, like 3-D tracks), as well as the wide range of audio and video compressors/decompressors from which to choose, the QuickTime engine is making other approaches to 2-D multimedia look obsolete and inefficient. Just as important, QuickTime's support for user data types frees programmers from being restricted to existing media formats.
Still, there are caveats. First, I've used a subset of the current QuickTime API that is more or less common across Macintosh and Windows. This is not a permanent necessity, since Apple has committed to a common API. Since the current Windows API doesn't support callbacks, I've used the MovieController component for transport control on both platforms, rather than making the direct movie calls available on Macintosh. To keep the project simple, I've stuck to the basics -- video and audio. Once you've got the framework running, though, it is straightforward to add text and MIDI handlers, sprites, or whatever.
The development environment I use is MetroWerks CodeWarrior Gold 10, which is Macintosh hosted, but lets you target applications for Macintosh (68K and PowerPC), Windows 95/NT, Magic Cap, and BeBox. CodeWarrior Gold is now also available hosted on Windows 95/NT, and lets you target Windows 95/NT and Macintosh (68K and PowerPC). Both packages support C/C++, Pascal, and Java for most target platforms. Processor support includes Motorola 680x0, PowerPC 601/603/604, x86/Pentium, MIPS R3000/R4200/R4600, and the Java Virtual Machine.
Producing the executable binaries for Macintosh 68K, Macintosh PowerPC, and Win32 projects here involves only a button-click each. It's also a good compiler to know if you're planning to develop for Sony Playstation, BeOS, Magic Cap, or Java, since you can use the same API across the board. If you use the Visual C++ cross compiler, it shouldn't be a problem working the other way around. While I like Visual C++, I find the Macintosh kit a bit pricey. When comparing CodeWarrior-compiled x86 code with VC-compiled code using digital video or real-time 3-D libraries, I've found no performance penalty. If you're writing your own inner-loop renderers and rasterizers, however, you may get different results. Finally, I haven't included a Windows 3.1 version of the project presented here for two reasons: The CodeWarrior x86 compiler supports only 32-bit Windows, and Microsoft is not exactly making it easy to support dual compilation of 16- and 32-bit projects.
I'll base the package presented here on QuickTime 2.1, the hottest version at this writing. Interestingly, the QuickTime SDK, which is available to APDA members on CD-ROM, covers only Version 2. Consequently, my only guide for this framework was the QTW.h header file that I downloaded. The rest was trial and error. While I could refer back to my developer notes on my Mac help files, the conversion was not totally intuitive. As you will see in the code, there are subtle differences in parameter passing between the two platforms. I'm pleased to say that you don't have to worry about any of that, since I've provided a liberal sprinkling of #ifdefs.
There are two CodeWarrior projects that make up this package. MediaPacker is a Mac-only program that prepares the media for the player. Most of the code deals with compressing PICT files and "stitching" movies into a single file. MediaPacker also generates a playlist that the player program references.
The second project group consists of MCPWin32, MCPPPC, and MCP68K, which create the executables for the three environments. If you are an MPW freak, you could use one makefile (with ToolServer on CodeWarrior) to create the lot in one go. Since I've put them into three distinct projects, you only have to hit the Make button for each.
I recently used MediaPacker for an art-installation product that required looping various movies and synchronized sound files. Performance on older machines was a concern since the product was also to go on CD-ROM. The product contained several hundred high-resolution PICT files, exported from SoftImage (a graphics package distributed by Microsoft) and stored on CD. Unfortunately, none of the QuickTime editing programs I have would accept the unusual numbering sequence of the PICT files. This prompted me to write my own compressor (MediaPacker) which allows users to type in a "filter" to parse the file names and sort them into some sort of numerical order. Another reason for writing MediaPacker was to pack the many PICT sequences into a single movie. This turned out to be the most efficient way to randomly access the video sequences, while putting minimal strain on the operating system. This also overcame seek-time problems from having movies scattered over a disk.
Likewise, each audio piece was converted from SoundEdit format to QuickTime. The QuickTime sound movies were then stitched together with MediaPacker, and a playlist was generated. While you could manually collect the start and end points of each movie in a digital video editor, it's not so easy with audio.
Since I don't like dialog boxes, and multiselection from a huge list isn't fun, I made the file load interface drag-and-drop. If you are new to PowerMac programming, there are traps when using callbacks with the Mixed Mode Manager that aren't thoroughly documented in the operating system help files. Previously, all that was required was casting to a UniversalProcPointer. Now you must use the NewDragReceiveHandler() macros provided. (In this case you'll find them in the relevant header file.) So to compress a bunch of PICT files into a single QuickTime movie, and generate a playlist that corresponds with the start and end points of each movie, the user simply has to select the first (lowest number) of each group, and drag this collection to the MediaPacker window. Feedback is provided, using DragTrackingHandler, to notify the user if the program can accept files they are about to drop. The program currently accepts only PICT or movie files, so I look for a file type of either "PICT" or "MooV" in the FlavorData field of the drag structure. (Refer to main.cpp, available electronically; see "Availability," page 3.)
In the userdata field of the drag receiver ((void *)handlerRefCon), you pass a pointer to the BatchPack object, so the callback (wherever it is) has access to essential properties and methods to set up the compression process; see Example 1.
The BatchPack object, a C++ structure, is defined in Listing One One of the more useful objects within BatchPack is the Cruncher class. Listing Two shows how Cruncher gets instantiated. Depending on the type of media, the appropriate derived class is used. The main window is passed to Cruncher so it can offer user feedback during lengthy compression sessions. The descriptive p parameter is an SCParams structure that contains user preferences and/or defaults for all the compression settings for the session.
To keep the program multitasking-friendly (since it could be tied up for some time), the processing is done in slices during idle time; see Listing Three. This is standard Mac fare for handling any task that takes more than a couple of seconds. During the compression process, the playlist is created and appended to using the code in Listing Four.
I simply paste the current movie into the end of the output movie and note the new end point. In the PICTCruncher version, there is more work since you're compressing frame-by-frame, rather than appending a whole movie (see PictCruncher.cpp, also available electronically). Another significant task for Cruncher is to find the next file according to the filter the user specified. These utilities are in HandyStuff.cpp (Listing Five).
To wrap up MediaPacker, the file is "flattened" to make it cross platform, and the playlist is pasted as a resource into the movie as well as being saved into a .txt file. While not satisfactory for Windows, it does serve to keep the playlist tied to the movie until you prepare the player program.
All the player code is in a single file called "MacWin," which compiles for Mac68K, PowerMac, and Win32. A lot of the useful data is kept in a C++ structure called EditList. Listing Six presents its structure.
The movie files and playlists can be specified in several ways. Information about the media location can be stored in a Mac or Win text resource, an external file, or in the local folder with the preset filenames of "video.mov" and "audio.mov." The first attempt looks in the resource in InitStrings(). Example 2 is the Windows version.
Next, InstallMovies() is called. The second place that is searched for the video movie is a specified file name in MacWin.cpp. OverrideSoundSource() is called in case the user has put special instructions in a config file that is created in a folder called MCP in the Windows directory (or in Preferences in the System Folder on the Mac). Example 3, the config file, recognizes two instructions. A complete HTML-style script could be built on this base using the functions in Example 4, which are called from OverrideSoundSource() (which really should be called OverrideDefaults); see Listing Seven.
The CDPath function makes no change on the Mac, as the CD can be named and found by name. Since Windows uses a single letter to identify a drive, there is no way to directly call the correct volume, so a search is made of any mounted CDs. (If you know a better way, please tell me!)
OpenEditListFromResource() loads the text resource and passes it in a buffer to OpenEditListFromBuffer(), which reads through the end point list and creates an array of EditItems within the EditList structure for that movie. The end point of each EditItem pair is set to an arbitrary amount of [next start (or end of movie)] minus GAP (defined in the MacMain.cpp file). This is to avoid any "spill" into the next sequence. The frame resolution is not that fine, so nothing is lost, although QuickTime will be advised not to play the subsequent keyframe (explicitly set in MediaPacker). As you can see from Example4, if all else fails, the entire movie becomes the play selection. The heart of the program is appropriately called Pump(). Most of the code handles the interaction and timeout. The last two lines of Listing Eight do the work.
For this example, I used a small mouse sweep to trigger for a switch in sequences. It's a nice, soft way of interacting (less jarring than a key hit or mouse click). Because Windows is multitasking, I set up a standard event loop. You could bypass this and the system would still multitask, but the application would get no further important information (such as paint messages) if the user did switch. To optimize performance on the Mac, I run a tight loop with a CheckForQuit() that scans the keyboard for a quit combination. Without any tricky system additions like WindowPicker, the user would have a difficult time switching programs anyway, although in reality, the system loop does not affect performance much unless another program is running concurrently. To make it less of a hard core kiosk application, just pop in a WaitNextEvent().
Since all the real decision making is in Pump(), that's where you can customize the application without messing with its cross-platform properties. For demonstration, the function RandomChangeMovie() is doing the orchestrating at present, and should provide you with a starting point for customization. Having gotten the fiddly parts out of the way, from here on, it should be a lot easier to put together some multimedia material without having to be a propeller-head programmer or resort to an overweight scripting language.
Metrowerks, Inc.
2201 Donley, Suite 310
512-873-4900
http://www.metrowerks.com/
Apple Computer Inc.
1 Infinite Loop
Cupertino, CA 95014
408-996-1010
http://www.quicktime.apple.com/
DDJ
struct BatchPack { Rect rect, originalPictFrame;
Boolean batching, batchFileOpened, scale_set;
int batchFileElements;
Cruncher *cruncher;
FSSpec *batchFileList;
OSType type;
BatchPack() :
rect(),
originalPictFrame(),
batching(false),
batchFileOpened(false),
scale_set(false),
batchFileElements(0),
cruncher(NULL),
batchFileList(NULL)
{
rect.right = 320; rect.bottom = 240;
originalPictFrame.right = 320;
originalPictFrame.bottom = 240;
}
~BatchPack()
{
Reset();
}
void PartialReset()
{
if (cruncher)
delete cruncher;
cruncher = NULL;
batching= false;
}
void Reset()
{
PartialReset();
if (batchFileList)
delete batchFileList;
batchFileList = NULL;
batchFileElements = 0;
batchFileOpened = false;
}
};
if (batchPack.batchFileOpened) { if (!batchPack.cruncher) {
if (batchPack.type == 'PICT')
batchPack.cruncher = new PictCruncher(gSrcWindow, p);
else if (batchPack.type == 'MooV')
batchPack.cruncher = new MovieCruncher(gSrcWindow, p);
}
batchPack.cruncher->CreateBatch(batchPack.batchFileList,
batchPack.batchFileElements, &batchPack.rect);
...
switch (myEvent->what) {case nullEvent:
if (batchPack.cruncher && allow_processing) {
if (batchPack.cruncher->crunching)
if (!(batchPack.cruncher->Process(gFilterString))) {
// means it's just finished
// reset all process info
// leave edit list full in case more copies required
batchPack.PartialReset();
allow_processing = false;
FixMenus();
}
}
break;
...
TimeScale t = GetMovieTimeScale(newMovie);TimeValue v = GetMovieDuration(newMovie); // get end time of editee
SetMovieSelection (newMovie, v, (TimeValue)0); // set insertion point
PasteMovieSelection(newMovie, current_movie);
v = GetMovieDuration(newMovie);
if (logFile) {
sprintf(pBuff, "%d\n", (int)v);
fwrite(pBuff, strlen(pBuff), sizeof(char), logFile);
}
if (batching) { P2Ccpy(batchFilter, headerFile.name);
Trim2Filter(batchFilter, filterString);
thisNum = FindNextNumberedFile(&headerFile, lastNum, batchFilter);
}
else
thisNum = FindNextNumberedFile(&headerFile, lastNum, filterString);
if (thisNum == -1) { // no more numbers for this prefix
if (!NextInBatch()) {
finishedProcess = true;
headerFileOpened = false;
crunching = 0;
}
else {
set_key = true; // make beginning of next pict sequence a key frame
goto keyset;
}
}
...
typedef struct _qtedititem { long start;
long end;
} EditItem;
struct EditList
{
EditList() {Zero();}
~EditList() {Clear();}
void Set(int count)
{
total = count;
edits = new EditItem[total];
index = 0;
}
void Randy() {index = rand() % total;} // ints are in pairs
long& Start() {return edits[index].start;}
long& End() {return edits[index].end;}
void Clear()
{
if (edits) delete [] edits;
Zero();
}
void Zero()
{
index = mode = total = 0;
edits = (EditItem*)NULL;
}
long index;
long mode;
long total;
Movie mov;
MovieController mc;
EditItem *edits;
CallBackStuff cbs;
static bool noChangeYet;
};
bool EditList::noChangeYet = false; // prevents jumping too early
void InstallMovies(){
if (!SetupMovie(THA_MOOFIE, &editList))
if (!SetupMovie(DEFAULT_VIDEO, &editList))
editList.mov = NULL;
if (OverrideSoundSource(sndeditList))
goto end_sound_init;
if (SetupMovie(THA_SOUND, &sndeditList))
goto end_sound_init;
cdPath = CDPath(THA_SOUND);
if (*cdPath)
if (SetupMovie(cdPath, &sndeditList))
goto end_sound_init;
if (SetupMovie(DEFAULT_SOUND, &sndeditList))
goto end_sound_init;
cdPath = CDPath(DEFAULT_SOUND);
if (*cdPath)
if (SetupMovie(cdPath, &sndeditList))
goto end_sound_init;
sndeditList.mov = NULL;
end_sound_init:
if (editList.mov) {
if (OpenEditListFromFile(DEFAULT_VIDEO_EDITS, editList) == 0)
if (OpenEditListFromResource(VIDEO_RESOURCE, editList) == 0) {
editList.Set(1);
editList.Start() = (long) 0;
editList.End() = (long) GetMovieDuration(editList.mov);
}
}
if (sndeditList.mov) {
if (OpenEditListFromFile(DEFAULT_SOUND_EDITS, sndeditList) == 0)
if (OpenEditListFromResource(AUDIO_RESOURCE, sndeditList) == 0) {
sndeditList.Set(1);
sndeditList.Start() = (long) 0;
sndeditList.End() = (long) GetMovieDuration(sndeditList.mov);
}
}
}
void Pump(){
static char buff[20];
static bool disp = false;
int point_x, point_y;
clock_t thisTime, diffTime;
if (++gRandCounter & 0x0F) { // minimize frequency of checks
thisTime = clock();
if (((diffTime = thisTime - gLastTime)) >= gTime_change) {
gLastTime = thisTime;
RandomChangeMovie(sndeditList);
RandomChangeMovie(editList);
#ifdef WIN32
GetCursorPos(&gLastMouse);
#else
Point mp;
GetMouse(&mp);
gLastMouse.x = mp.h;
gLastMouse.y = mp.v;
#endif
EditList::noChangeYet = false;
}
else { // check for fresh mouse drag
POINT p;
#ifdef WIN32
GetCursorPos(&p);
#else
Point mp;
GetMouse(&mp);
p.x = mp.h;
p.y = mp.v;
#endif
if ((p.x == gLastMouse.x) && (p.y == gLastMouse.y))
EditList::noChangeYet = false; // ok for another mouse sweep
else if ((abs(p.x - gLastMouse.x) > MIN_MOUSE_MOVE) ||
(abs(p.y - gLastMouse.y) > MIN_MOUSE_MOVE)) {
if (EditList::noChangeYet == false) {
RandomChangeMovie(sndeditList);
EditList::noChangeYet = true;
RandomChangeMovie(editList);
gLastTime = thisTime;
}
}
gLastMouse = p; // keep pointer up to date
}
}
if (editList.mc)
MCDoAction(editList.mc, mcActionIdle, NULL);
if (sndeditList.mc)
MCDoAction(sndeditList.mc, mcActionIdle, NULL);
}