Making a Cross-Platform Installer with Perl

I did my best to choose modules for this project that appeared to be both stable and available on both the CPAN and Activestate PPM repositories. Coding the system went very quickly and I felt confident that the delivery would be on time and relatively stable. After the coding was complete, I began writing the installer.

I started with the code that would install any third-party modules onto the target system. I felt this would be the most interesting, fun, and easy part of the installer. I soon discovered otherwise.

Dealing with Dependencies

My troubles began with the XML modules I chose. The project needed a fast and easy way to select elements or subsets of a complete XML document. XML::XPath came to mind immediately. I had used XPath in conjunction with Cocoon, in Java, at my last job. This module depends on XML::Parser (almost all XML namespace CPAN modules depend on XML::Parser directly or indirectly) and XML::RegExp. I also chose XML::EasyOBJ for programmatically writing XML documents, as it is very easy to use and has a clean syntax. XML::EasyOBJ depends on XML::DOM, which depends on XML::Parser. XML::Parser, for those of you who haven't used it, is a glue module for the very fast (in my experience), very functional C-based expat library.

I encountered no problems in Windows with the XML::Parser installation. The PPD (ActiveState package file) for XML::XPath comes with a precompiled expat DLL. On *nix-based systems, no binary distributions were available that I could find, so I had to do the following to autoinstall the module and the expat library:

1. Hard-code the latest stable expat distribution in our distribution to avoid writing code to download it dynamically.

2. Write a shell script to explode the distribution, call the included configure script to ready the source for compilation, then call make to make and install it.

3. Hard-code the latest XML::Parser distribution in the source-code tree, and add code to the shell script to build and install it.

The script I used to do the install is shown in Listing 1.

I used the Bourne shell for this because it was quicker than writing it in Perl, and the Bourne shell is available on all flavors of UNIX/Linux. I called this script from the setup.pl script using the system() function (see Listing 2).

XML::EasyOBJ seemed to be a no-brainer; both the CPAN and Activestate repositories had it listed for download. I was wrong. I tried to install the module using ppm and discovered that the PPD file for this package was not installable. The quick workaround for this was to hard install the module in our distribution. So much for autoinstalling all third-party modules.

The system we wrote also depends on Crypt::SSLeay and LWP for posting XML to two gateways over HTTP using secure sockets (SSL). Crypt::SSLeay depends on the OpenSSL C library. PPM includes the OpenSSL DLL with the Crypt::SSLeay PPD, but CPAN does not. Our solution was to write another shell script to drive the install after adding the latest openssl and Crypt::SSLeay distributions in our package.

Problem solved? Not quite. The Crypt::SSLeay Makefile.pl looked for OpenSSL in directories usually only writable by root. We had to be able to run in a shared hosting environment with potentially very limited permissions. Our solution was to edit Makefile.pl so that it would find our private, nonroot OpenSSL install and build against that. The shell script to do the install is shown in Listing 3.

Here is the altered portion of Crypt::SSLeay Makefile.pl that lets it find our local install of OpenSSL:

if (exists $ENV{'OPENSSL_DIR'}) {
    unshift(@POSSIBLE_SSL_DIRS, $ENV{'OPENSSL_DIR'});
}

For session management, I chose the CGI::Session module with the file-based backend for persistence, as I felt that would be the most portable and easiest to set up of the available CGI::Session backends. CGI::Session is a wonderful module—I really enjoyed integrating it into this system. However, on Windows, calls to clear session objects of all data or delete data from a session would mysteriously hang. My first thought was that I had encountered a file-locking issue since I was developing this under cygwin on Windows. I reviewed the CGI::Sesson::File source, removed the file-locking code, and tried again. No luck. I then went to the CGI::Session online mailing list archive and found the issue addressed.

CGI::Session uses Autoloader and has two sets of clear and delete routines, the names of which only differ in case: CLEAR()/clear() and DELETE()/delete(). Under Windows, one of the routines gets overwritten due to the case-insensitive nature of the platform; this causes calls to the like-named methods to fail. To fix this, we moved the lowercase named pair above the __END__ tag in the Session.pm module so Autoloader never bothers with it. The result was that we had to hard install yet another module into our software package (this issue has since been fixed in the CGI::Session distribution).

It was my understanding that any module accepted into CPAN had to detect and execute CPAN installation for any missing dependent modules. It turns out this isn't entirely true. The XML modules I picked did not autoinstall all of their needed dependencies, and I found that if Crypt::SSLeay and XML::Parser were not on the system, I had to install them first so any dependent modules would find them. I remembered that Bundle::LWP solved my dependency tree problem for LWP on *nix, but I then discovered that the bundle doesn't exist for Windows and isn't needed, as LWP is installed as a standard module with ActiveState Perl. The list of modules needed by the software this script installs is stored as a hash table in the script. It looks like this:

    my %modules = qw(
        CGI			2.49
        Storable		2.00
        CGI::Session		2.94
        DB_File		1.73
        Crypt::CBC		2.00
        Crypt::SSLeay		0.17
        Crypt::Blowfish_PP	1.10
        HTML::Template		2.4
        Bundle::LWP		1.6
        LWP			5.44
        Test::More		0.47
        XML::Parser		2.27
        XML::RegExp		0.03
        XML::DOM		1.06
        XML::XPath		1.04
        XML::EasyOBJ		1.12
    );

I thought it would then be easy to just edit the list in the script and let the generic installer code install every module. Due to the XML::Parser and Crypt::SSLeay dependency issues, I had to add this code deep in the install routines themselves to make sure XML::Parser/expat and openssl/Crypt::SSLeay were installed before any other needed modules:

if (grep(/XML::Parser/, @need)) {
    @need = grep(!/XML::Parser/, @need);
    install_xml_parser($perl, $lib);
}

if (grep(/Crypt::SSLeay/, @need)) {
    @need = grep(!/Crypt::SSLeay/, @need);
    install_crypt_ssleay($perl, $lib);
}

@need is a list of modules the target system does not have, and is populated using the code below. Notice the code that deletes the LWP package is not needed for a given platform and also notice I have to use use lib in the system call so that Perl will be able to see the modules we had to hard install into the software distribution; the Perl binary the user picks for the CGI scripts may not be the same one that is running the setup script. Finally, the variable $lib holds the absolute pathname to the lib directory in the install tree and is set earlier in the script, as shown in Listing 4.

Up to this point, I had been testing the install process only under cygwin with CPAN. The first time I ran the script under a DOS prompt on Windows NT, ppm2 complained that some of the PPDs I tried to install were corrupt, and "force install" failed for some reason when called through the install script. I debugged my own ppm2 automation code and searched for information on PPM and answers online for several hours. No luck. I then searched the Windows Perl installation tree, hoping to find something to help me out. I found a ppm3 batch file. I hadn't been aware of an upgrade. (Of course, the install notes talked about it in depth, but unfortunately, I had skipped them.) This worked, but I was in a huge time crunch, and the method of setting the root directory for installs had changed between ppm2 and ppm3. I was having a hard time figuring out how to set this option with ppm3, partially because I was anxious about the looming deadline. I recently revisited this, and it turns out to be easy to do.

Setting the root under ppm2:

C:> ppm set root <directory name>
Root is now <directory name> [Was <previous directory>]

open(S,"$perl -S ppm.bat set root $dir |") || die "Can't read from PPM: $!\n";
my $line = <S>;
close(S);
$orig_root = ($line =~ m/\[was\s+(.+?)\]/i)[0];

Under ppm3:

ppm3 target set root <directory name>

If I hadn't been in a rush, I would have used ppm3 to do nonadministrator installs; since delivery day was very near and I couldn't afford more time to learn ppm3, I decided instead to just document that the person installing this package under Windows would have to have administrator privileges also or have an administrator install it for them. I plan to change this in the next release.

Cross-Platform Woes

For *nix environments, I stayed with doing either a local (nonroot) install or a root install based on the UID of the person running the script. QA started testing this part of the installer in different *nix environments. The first error they uncovered involved installing the package as a nonroot user on a machine that already has a systemwide CPAN Config.pm, but no local CPAN configuration for the user running the install. The install failed because the installer couldn't write to system directories such as /etc. Our fix was to set the environment variable PERL_MM_OPT to LIB=<path to distribution lib directory> and start CPAN initialization for the nonroot user. This is shown in Listing 5.

QA then found another case: a root user who has su'd without "su -" so that their username variable still points to their real user ID. This would cause the CPAN installer to try and use their local CPAN configuration over the systemwide configuration. To fix this, we manually set the username variable in the script to "root" and set their effective UID to be the same as their real UID in the script:

$ENV{USER} = 'root';
$< = $>; 

On Solaris, the bootstrap shell script that calls setup.pl failed because the Bourne shell on Solaris does not have a built-in [ operator; instead it calls the shell utility test, which is symlinked to the character [. I had left the quotes out of the right-hand side of a conditional I used to determine if the script was using a BSD or SysV flavor of echo. This caused the conditional to be malformed when it was passed to test on Solaris, which in turn caused the bootstrap script to abort. This was an easy fix—I just quoted the right-hand side of the expression. Here is the fixed code I used to detect which echo was in use in the bootstrap script.

#  To get SysV/BSD echo sans-newline right

vnl=
bnl="\c"

if [ "'echo \"\c\"'" = "\c" ]
then
    vnl=-n
    bnl=
              
fi

echonl() {
    echo $vnl "$*$bnl"
}

QA then moved on to test the install under Windows. They found that when my setup script tried to find all existing Perl binaries on a Windows 2000 system (the second step in the install process), the install died with an error about a unicode module not being found. This was puzzling, as we were not doing anything with Unicode in this release. I discovered that when I used glob() on Windows 2000 to get file names, I would get a list of long file names. When I used a long Windows-style file name in a regular expression, Perl would try to interpret some backslash followed by a letter sequences, such as

\l

as unicode characters. It would find that it did not have the unicode decoder loaded for that character, and die. We wrote our own mini glob() that handles this case correctly by quoting metacharacters in the regular expression matcher:

sub our_glob {
    my $dir = shift;
    my $pat = shift;
    local(*D);
    local($_);
    opendir(D,$dir);
    my @files = grep(/\Q$pat/, readdir(D));
    return map {File::Spec->catfile($dir, $_);} 			@files;
};

With the install then working correctly on both *nix and Windows, QA discovered that even after the install succeeded, the CGIs would fail to find the modules we installed. I forgot to include system-dependent library directories, for example 'i386-linux', in my use lib statements in all the CGIs for the modules the script installs locally. The question was how to do this without knowing the system type ahead of time (scramble for Perl in a Nutshell). A combination of FindBin.pm and Config.pm does that for us:

use Config;
use FindBin;

use lib "$FindBin::Bin/lib";
use lib "$FindBin::Bin/lib/$Config{archname}"

Conclusion

I haven't related all of the problems I encountered in creating this script, which also has six additional installation steps, each with character-based screens and a fair amount of user interaction. The problems I encountered haven't dampened my enthusiasm for Perl, and I think CPAN and PPM are wonderful repositories. But I also see that one of the downsides of having such an open, do-it-how-you-want-to system is that it allows many implementation differences among modules and even Perls.

Writing and debugging the installer took approximately 30 percent of the time allocated to this project. I encountered problems with autoinstalling modules, issues with bugs in the third-party modules I used, issues with my own lack of awareness of Windows long path names, and regular expression interaction. I also failed to imagine the full variety of possible installation environments in which this script would be run, given that it is supposed to run "anywhere." I hope I have helped others avoid some of the same mistakes.

The online documentation for this project (which was also part of the five-week project), is viewable at http://vrpp.support .netsol.com/docs/.

TPJ

Listing 1

#!/bin/sh

PATH=$PATH:/usr/local/bin:/usr/local/gnu/bin:/usr/gnu/bin: 	/opt/bin:/opt/gnu/bin
PATH=$PATH:/usr/bin:/bin:/usr/ccs/bin
export PATH

CC=gcc
export CC

PERL=$1
DIR=`pwd`

gzip -d -c ./expat-1.95.5.tar.gz | tar xvf -
cd ./expat-1.95.5
./configure --prefix=$DIR
make
make install
cd ..
rm -rf ./expat-1.95.5

gzip -d -c ./XML-Parser-2.31.tar.gz | tar xvf -
cd ./XML-Parser-2.31/
$PERL Makefile.PL EXPATLIBPATH="$DIR/lib" EXPATINCPATH= 	"$DIR/include" CC=gcc LD=gcc
make 
make install
cd ..
rm -rf ./XML-Parser-2.31

Back to Article

Listing 2

sub install_xml_parser {
    my $perl = shift;
    my $dir = shift;

    Screen::clear();
    print "Installing expat library and XML::Parser .. \n";
    Screen::pause();

    if (exists $ENV{'LD_LIBRARY_PATH'}) {
        $ENV{'LD_LIBRARY_PATH'} .= ":$dir";
    } else {
        $ENV{'LD_LIBRARY_PATH'} = "$dir";
    }

    system("cd $dir; sh ./make_expat $perl");

    print "XML::Parser installed.\n";
    Screen::pause();
}

Back to Article

Listing 3

#!/bin/sh

PATH=$PATH:/usr/local/bin:/usr/local/gnu/bin:/usr/gnu/bin: 	/opt/bin:/opt/gnu/bin
PATH=$PATH:/usr/bin:/bin:/usr/ccs/bin
export PATH

CC=gcc
export CC

PERL=$1
DIR=`pwd`

gzip -d -c ./openssl-0.9.6g.tar.gz | tar xvf -
cd ./openssl-0.9.6g
./config --prefix=$DIR
make
make install
cd ..
rm -rf ./openssl-0.9.6g

gzip -d -c ./Crypt-SSLeay-0.45.tar.gz | tar xvf -
cd ./Crypt-SSLeay-0.45/

OPENSSL_DIR=$DIR
export OPENSSL_DIR

$PERL Makefile.PL CC=gcc LD=gcc
make 
make install
cd ..

rm -rf ./Crypt-SSLeay-0.45

exit 0

Back to Article

Listing 4

my $perl = $pool->get('PERL');

#  Close standard error so user doesn't see failure messages for modules
#  that don't exist.

close(STDERR);

my $cmd;

#  On windows, no such thing as Bundle::LWP, on UNIX/Linux,
#  just want to test for Bundle so dependencies are done correctly.

if (Util::iswin()) {
    delete $modules{'Bundle::LWP'};
} else {
    delete $modules{'LWP'};
}

for (sort keys %modules) {
    $cmd = "$perl -e \"use lib q!$lib!; use $_ $modules{$_};\"";
    if (system($cmd)) {
        push(@need, $_);
    } else {
        push(@have, $_);
    }
}

open(STDERR,'>&0');

Back to Article

Listing 5

#  For local, non-root installs
$ENV{'PERL_MM_OPT'} =" LIB=$dir" if defined $dir;

my $cpanrc = "$ENV{'HOME'}/.cpan/CPAN/MyConfig.pm";
unless (-f $cpanrc) {
    initialize_cpan($cpanrc, $perl);
}

sub initialize_cpan {
    my $cpanrc = shift;
    my $perl   = shift;

    use File::Basename;

    Screen::clear();
    print "Initializing CPAN (follow on-screen instructions:\n";
    Screen::pause();

    my $dir = File::Basename::dirname($cpanrc);
    DirCopy::make_path($dir, 0700);

    system("$perl -MCPAN::FirstTime -e 'CPAN::FirstTime::init(q!$cpanrc!)'");

    print "Initialization complete!  On to module install.\n";
    Screen::pause();
}

Back to Article