Doug owns Access Microsystems, a software-development house specializing in C/C++ software development. He is also the author of the BTFILER and BTVIEWER Btrieve file utilities. Doug can be contacted at Access Microsystems Inc., 404 Midstreams Road, Brick, NJ 08724, or on CompuServe at 74040,607.
It's the call all Btrieve developers dread: "I have a Btrieve file with 98,000 records, it's damaged, and I don't have a current back-up!" (or "...it's damaged and my tape back-up didn't work!"). In either case, your client--and possibly you as well--has a problem.
Let's begin with some definitions. Btrieve from Novell is a key-indexed record-management system. A Btrieve file can have between 0 and 24 keys. An individual key can be composed of more than one "segment," or contiguous section of a file, as long as there are not more than a total of 24 segments. Unlike some systems used to do similar jobs, Btrieve is a record manager only. It has no user interface, but instead provides an API that allows Btrieve to be used from C/C++, Pascal, Cobol, Basic, and assembler. Btrieve comes in a client-based version (BTRIEVE.EXE) and a server-based version (BREQUEST.EXE). For our purposes, both versions can be considered identical since the resulting files are identical.
Btrieve files consist of "pages." When a Btrieve file is created, a page size is specified. The page size must be a number between 512 and 4096 that is divisible by 512, and fit at least one of the fixed-length portions of the record being created. Btrieve allows variable-length records, but each variable-length record has a fixed-length portion as well. There are three general types of pages: a single-header page (possibly with an additional page for an "Alternate Collating Sequence"), 0 or more index pages, and 0 or more data pages. (In fact, most files have one or more index and data pages, but Btrieve allows data-only and key-only files, too.) All pages within a Btrieve file are the same size. Figure 3 - Figure 5 are hex dumps of pages from a Btrieve file with a page size of 512 and a single index starting at position 1 for 11 bytes. The variable data page (Figure 5) is from a variable-length data file.
Most of what we are about to discuss is undocumented by Novell. A warning: As with any undocumented element of a system, some aspects of Btrieve files are undocumented to hide details of implementation from the outside world in order to ensure that the end user is shielded from changes to the details. Most of the information presented here is valid for Btrieve files created by version 4.11 through version 5.1x. Btrieve version 6.0 (currently available only as part of Netware SQL) will almost certainly change some of these details. Of course, there are good reasons to dive into the undocumented aspects of a system now and then; one reason is to recover data unavailable elsewhere.
The first page of any Btrieve file is the "header" page or File Control Record; see Figure 1. The details of what is where on the header page are undocumented. A second header-like page will be present if an alternate collating sequence is defined to allow the file to be sorted in other than strict alphabetical order; see Figure 2. For instance, a standard alternate collating sequence is defined in a file, UPPER.ALT, included with the Btrieve developer's kit. In addition to some overhead bytes, the page containing the alternate collating sequence has the hex value AC as an identifier, an eight-character name of the sequence associated with the file (in this case "UPPER") and one byte for each of the 256 possible values for a byte. In Figure 2, note that the bytes describing this alternate collating sequence start at 0 and end at FF hex, with the exception of the fact that the uppercase alphabetic characters (41 through 5A hex) replace the lowercase alphabetic characters (61 through 7A hex). This allows upper- and lowercase characters to be sorted identically.
The next type of page in a Btrieve file is the "index" page; see Figure 3. The number of index pages (if any) depends upon the number of indexes in the file, as well as the number of records. These are of little use independent of the actual data, but one might find it interesting to track the pointers near each "key" on the index pages back to the data they represent. In a data-only record, it's possible no index page will exist.
The next type of page is the "data" page. There are at least two types of data pages: the "normal" data page (see Figure 4) for fixed-length records (or fixed-length portions of variable-length records) and the variable-length data page (see Figure 5). In a key-only record, no data pages will exist. The header can be rebuilt (as we will see) and indexes can be recreated (given the correct data and enough time), but the data is the one thing that cannot be derived from the other components.
The organization of records on data pages gives us some more information we can use to recover Btrieve files, and also allows for the selection of the page size that will make the best use of each byte of the page. Fixed-length records, or fixed-length portions of variable, length records, are stored on the data page one after another, with some other information. Depending upon page size, more or less space will be left over at the end of the page. Determining the "best" page size, from a space utilization standpoint, requires calculating the number of records that will fit on each page, and from that number, the number of bytes wasted.
Let's use as an example a file with a fixed record length of 128 with two keys that allow duplicates. There is an overhead of six bytes for each fixed-length data page. (This information will be useful when trying to recover data.) The physical record length, that is, the actual number of bytes taken up by a Btrieve record is: (logical_record_length+ (number_of_dup_keys*8)) or in our example (128+(2*8)) or 144. The eight bytes for each key that allows duplicates are used for "prev" and "next" pointers. Since each page starts out with six bytes less than the stated size, we next search for numbers between 506 and 4090 that have the smallest remainders when divided by 144. The best fit allows seven records on a page size of 1024, with ten bytes wasted. If 128 is not a "magic number" for our record, we can use more of the page by using a record length of 129, thus wasting only three bytes. There are shareware and public-domain utilities that do this calculation for you. The important thing to remember is that on every data page, six bytes are used by the system, and eight bytes are added on to the physical record length for every key that allows duplicates.
The nature of variable-length records is that the data is placed on the data page such that no "optimum page size" exists for the variable portion of a Btrieve record. Btrieve also allows compressed records, with which the entire record is placed on a variable data page, and the fixed data pages contain only pointers to the variable pages. It is important to note that Btrieve's data compression is not nearly as sophisticated as that in programs like PKZIP. Btrieve compresses only those strings of data that contain repeated characters. Other data compression techniques take advantage of the fact that, for instance, English text has many "e" and "i" characters and relatively few "z" and "x" characters. Therefore, they encode the common characters in less than one byte, and the less-common characters in one byte (or, I suppose, possibly more than one byte).
Btrieve compression only occurs when you have five or more characters repeated one right after another! In one sample purchase-order file I looked at, the only time data compression kicked in was when the PO number had at least five consecutive identical digits (like "144444" or "777777"). Not much of a savings there. Keep this in mind as we discuss possible data-recovery scenarios and how data compression affects them.
With that background, we can go back to the Btrieve-based system-developer's dilemma described at the start of this article. The first step, of course, is to make sure that the user has a backup of this damaged file. This is critical. If the user is unable to copy the damaged file, then we have a DOS problem, and Btrieve-specific cures must await its resolution.
The next step is to determine what sort of error is occurring. Btrieve has many status codes; those related to damaged files are listed in Table 1. Several of the status codes that could lead to a damaged Btrieve file refer to a "pre-image" file.
Code Meaning 2 I/O Error 14 Pre-image Open Error 15 Pre-image I/O Error 30 Not a Btrieve File Error 42 Incomplete Accelerated Access 56 Incomplete Index
One way to protect data in Btrieve files is through the use of a pre-image file created when a file is first updated. The pre-image file contains images of the pages in the file before the update occurred. If the operation is interrupted (for instance, the program hangs or the power fails), the next time the file is opened the pre-image file is read, and the file is restored to the condition it was in just after the last complete operation. If these pre-image files are disturbed before the next successful open, the file cannot be automatically restored by Btrieve. Pre-image files share the filename with the Btrieve data file, with a .PRE extension.
The most common (and maddening) Btrieve error code is status 2, an I/O error. This could be caused by almost anything. Novell has a list of 33 distinct causes of status 2, and it's possible they missed a couple. Look back at Figure 1 through Figure 4 and notice that in byte 3 of each record, there's a number that acts like a "page count," with the header page having a 0, the alternate collating sequence having a 1, the index page having a 2, and so on. Btrieve appears to use this number as a "sanity check." On the index page, change that 3 to a 7 and a status 2 will appear. Your data is still just fine, but changing that one byte (or any of the four bytes that are part of the page count) will cause a Btrieve status 2.
Start with the easiest recovery option first. Btrieve allows reading of records by logical positioning (that is, via an index) or by physical positioning. To recover data from a damaged Btrieve file, physical positioning must be used; see the pseudocode in Listing One (page 106). To recover the damaged file, we start from the first physical record and read towards the end of the file. If we encounter an error and the error is not an end-of-file error, we save the physical-positioning information from the most recent successfully read record and get the last physical record (using STEP_LAST, Btrieve function 34). We read towards the beginning of the file, stopping when a Btrieve error occurs or when we read the last record we read previously. Now we close the files and exit.
The recovery procedure just described is what Novell's utility BUTIL does during a -RECOVER operation, saving records to a sequential file in a special format. In many cases that is all that needs to be done. But what happens in the case of multiple nonadjacent errors in the file? In that case, we start at the beginning of the file, step forward until we find an error, then go to the end of the file and step back until we find an error or read the same record as the last successfully read record when we were stepping forward. If the error we find stepping back is not the error we find stepping forward, the records between the two errors will not be read. Here is where our knowledge of Btrieve files (and some undocumented aspects of Btrieve files) comes in handy.
In Listing One, we talk about saving physical-positioning information. This takes the form of a four-byte value that is, in fact, the offset in the file where the record occurs. (We'll ignore for a moment the fact that Btrieve for DOS allows files to extend over more than one disk drive.) Btrieve also allows you to set the position in a file. Usually, this set position function (GET_DIRECT, Btrieve function 23) uses a 4-byte block obtained from a previous call to GET_POSITION (Btrieve function 22). But this time we have to figure it out ourselves.
We know that every Btrieve file has a 6-byte overhead on every data page. We can prove this by creating a new Btrieve file and then seeing where the record is placed on the data page. From this we know that the only possible place the first record on a page can be found is six bytes from the start of the data page. But how do we know what pages are data pages and what pages are index pages? We could use DOS functions to read through the file, checking the bytes at the beginning of each page. Data pages do have some distinctive values in two of the six overhead bytes that are not the "page count" discussed above, but this seems a little too low level. Thankfully, there is a better way.
We can ignore the first page of the file, since we know it will always be a header page. We can calculate the total number of pages in the file by dividing the file length by the page size. Then for each page we do a GET_DIRECT at six bytes from the beginning of the page. If there is no error, or the error that does occur is not a status 43 (Invalid Positioning Information), we have found a data page. We save the record through the GET_DIRECT operation, and calculate where the next record will start. From our earlier discussion, we can tell the physical record length of any record: (logical_record_length+ (number_of_dup_keys*8)).
We can calculate the number of keys that allow duplicates by using a Btrieve status function (STAT, Btrieve function 15). Then we can simply take the position of the first record, add the physical record length, and call GET_DIRECT again. We can repeat this until the position is past the page on which we are working, and we can repeat the entire procedure (look for a record at six bytes into the page) for every page in the file.
A second type of problem can occur in Btrieve files. This problem will show itself by reporting a status 30 error (Not a Btrieve File) during an OPEN call. You can try to recover a file using the procedures just described, but this will not work since Btrieve can't open the file. When Btrieve reports a status 30, it is really saying that the header of the file is not consistent with what Btrieve expects in the header.
The solution uses a "clone" of the damaged file--a Btrieve file with the exact same page size, record length, key structure, and so on. Copy the header from the good clone file over the header in the damaged file, remembering that a page is some number divisible by 512 less than or equal to 4096. Now we have to do one more thing before we can use the data-recovery method described earlier.
In bytes 26 hex through 29 hex, Btrieve stores the length of the file, measured in pages. If we do a directory listing on the file used to generate Figure 11, we find that it is 9216 bytes. Dividing this value by 1536, we get 6. So, if we were to repair this file, we'd start with the 4-byte hex value, as in Table 2. In our example in Figure 1, bytes 26 through 29 do contain "00 00 06 00" (hex values). If the header page was damaged and we inserted the "00 00 06 00" into the file after replacing the header page with a header page from a clone file, we would then be able to use the file-recovery procedure described earlier. File recovery is still needed because some information needed for accessing the file (for instance, actual record counts) is still missing. C code to rebuild a damaged file header is provided in Listing Two (page 106).
26 27 28 29
Start with 6 00 00 00 06
Flip byte 26 with byte 27 00 00 00 06
Flip byte 28 with byte 29 00 00 06 00
Btrieve as a file manager is fairly robust. Many Btrieve developers go for years without seeing some of these conditions. The Btrieve developer's kit provides the BUTIL program that handles the most common of the file-recovery procedures. In addition, several commercial, shareware, and public-domain programs exist to recover damaged Btrieve files.
But what you don't know can hurt you. For instance, some of the data-recovery methods described depend upon knowing where records start. With Btrieve-compressed records such methods cannot be used, since the starting position for records depends upon the data in previous records. If, as a developer, you do not understand exactly what it means to use Btrieve data compression, you can end up with a file that is no smaller than a similar file without compression, and somewhat slower to boot. Look as you might, you will not find an adequate description of the implementation details of Btrieve's data compression. It is for times like these that getting out the HEX editor and rooting around our Btrieve data files can help make you a hero when the dreaded call for help comes.
_INSIDE BTRIEVE FILES_ by Douglas Reilly[LISTING ONE]
OPEN (Btrieve function 0) damaged file
Open destination file (Could be Btrieve file, could be DOS Sequential file)
if STEP_FIRST (Btrieve function 33) does not return an error
DO
write record to destination file
save the physical position from call to GET_POSITION (Btrieve function
22) in last_saved_pos.
WHILE STEP_NEXT (Btrieve function 24) does not return an error.
if last Btrieve error is not an end of file error
if STEP_LAST (Btrieve function 34) does not return an error
DO
write record to destination file
save the physical position from call to GET_POSITION (Btrieve function 22) in cur_pos.
WHILE STEP_PREV (Btrieve function 35) does not return an error AND cur_pos is not equal to
last_saved_pos
CLOSE (Btrieve function 1) source
close destination file (either Btrieve or DOS sequential file).
[LISTING TWO]
#include "stdio.h"
#include "stdio.h"
/*
Given a FILE pointer, return the length of the file.
*/
long filelen(FILE *t)
{
int i;
long loc;
long begloc;
extern errno;
errno=0;
if ( (begloc=ftell(t))==-1L ||
(fseek(t,0L,SEEK_END))!=0 ||
(loc=ftell(t))==-1L ||
(fseek(t,begloc,SEEK_SET))!=0 )
{
return(-1L);
}
return(loc);
}
/*
Given the name of a damaged file, a good "clone" of the damaged file,
rebuild the header on the damaged file. Don't use this file directly,
but next do a simple recover on the file so that the rest of the
information in the header is updated.
This should NOT be tried on any file except one that returns a Btrieve
status 2 (I/O error) or 30 (Not a Btrieve file) on OPEN.
*/
int do_hdr_rebld(char *damagedFile,char *goodClone,int pageSize)
{
int ret;
char headerbuf[4096];
long damaged_size=0L;
long num_pages=0L;
long b1,b2,b3,b4;
FILE *in;
FILE *out;
/* Open the good clone file, if you can */
if ( (in=fopen(goodClone,"rb"))==NULL )
{
printf("\nCan't open undamaged file to read header. Exiting");
return(0);
}
/* read the header from the good file */
fread(headerbuf,1,pageSize,in);
fclose(in);
if ( (out=fopen(damagedFile,"r+b"))==NULL )
{
printf("\nCan't open damaged file. Exiting.");
return(0);
}
/* get the length of the file, in bytes. */
if ( (damaged_size=filelen(out))==-1 )
{
printf("\nCan't read damaged file. Exiting.");
return(0);
}
/* convert to pages. We could do a sanity check to ensure
that the file length is evenly divisible by the page
length.
*/
num_pages=damaged_size/(long)pageSize;
/* isolate each byte. Yep, there probably is a better way
(shifting bits), but this seems clearer to me, and we won't
use this routine very often.
*/
b1=(num_pages&0xFF000000L);
b1=b1/0x1000000L;
b2=(num_pages&0x00FF0000L);
b2=b2/0x10000L;
b3=(num_pages&0x0000FF00L);
b3=b3/0x100;
b4=(num_pages&0x000000FFL);
headerbuf[0x26]=(char)(b2);
headerbuf[0x27]=(char)(b1);
headerbuf[0x28]=(char)(b4);
headerbuf[0x29]=(char)(b3);
/* get to the start of the file... */
fseek(out,0L,SEEK_SET);
/* write the header */
ret=fwrite(headerbuf,1,pageSize,out);
fclose(out);
return(ret);
}
Copyright © 1993, Dr. Dobb's Journal