Mike Gilson is a senior system test engineer for the Allen-Bradley Co. and is responsible for developing test software on VAX/VMS and UNIX platforms. You may contact him at 747 Alpha Drive, Highland Heights, Ohio 44143.
This article describes the design and VAX/VMS implementation of interprocess communication (IPC) primitives that allow processes to share information. It also describes an architecture that reduces the number of IPC links between processes but still allows any process to communicate with any other. The implementation uses VAX/VMS I/O services and mailboxes as the IPC mechanisms.
Mailboxes: A VAX/VMS IPC Mechanism
Mailboxes are virtual I/O devices that can be used for communication among processes. Unlike other I/O devices, which are hardware devices, mailboxes are implemented in software. Mailbox operations include receiving mail, sending mail, and rejecting mail. The receive and send operations may be performed synchronously or asynchronously through a variety of system services. In this application, data transfer is performed by a call to the SYS$QIO system service. VAX/VMS provides higher level I/O system services, but all of them eventually call SYS$QIO or SYS$QIOW (the synchronous version of SYS$QIO).Mailboxes are flexible in size, message format, and protection. When a mailbox is created, the programmer may specify the mailbox's size (in bytes) or use the system default. Mailboxes place no restriction on message format, so the programmer may read/write any structure to a mailbox as long as the message is within the mailbox's size restrictions. Mailbox access is controlled through the usual VAX/VMS protection strategies.
Although a mailbox may be used for two-way communication, this is not always convenient, especially if asynchronous communication is desired. If process A and process B are communicating through a single mailbox, both processes will be notified that a message is in the mailbox regardless of which process sent the message. Applying mailboxes as if they were one-way devices is often easier. Using this approach, two mailboxes link processes A and B: one for A-to-B transfers, another for B-to-A.
Architecture
When designing an IPC solution, the programmer needs to consider the tradeoffs between minimizing the number of IPC links and maximizing performance. Mailboxes require memory, which is deducted from process resources. Too many mailboxes will slow down a system by eating up memory. Minimizing the number of links requires more processing overhead. Two of the designs I'll discuss in this article aim for maximum performance, and for minimum links. Our implementation, which is a compromise of the two, is a third design.When a link is established from each process to every other process in the IPC network as in Figure 1, the total number of links for n processes is n * (n - 1)/2. This solution obtains a constant, minimum transfer time but requires a large number of links. If the links are to be bidirectional, two mailboxes will be required for each link: a total of n * (n - 1) mailboxes.
The minimum link design, shown in Figure 2, yields n - 1 links for n processes. Since the links are bidirectional, 2 * (n - 1) mailboxes are required. In this scheme, messages are passed from one process to another until the destination is eventually reached. This approach requires message handling by each process in the path between sender and receiver. The number of links is minimized, but transfer time now depends on the number of processes that handle the message. One transfer in the best case, n-2 transfers in the worst case, n/2 in the average case.
The architecture used in our application requires near-minimum links and provides fast transfer time (see Figure 3) . For n processes, this solution requires n + 1 links and n + 1 mailboxes. IPC between client processes 0 and 1 requires a transfer from the client 0 to the server, then from the server to client 1. This architecture minimizes mailboxes since all clients write to a single server mailbox.
Server Functions
The server provides pass-through and broadcast functions to perform IPC between any two processes. The pass-through function allows any process A to send a message to any process B via the server, assuming both A and B have established IPC links with the server. The broadcast function allows any process to broadcast a message to all other processes linked to the server.
Server Startup
During startup, the server creates its receive mailbox (see Listing 2) by calling the open_comm_link() primitive shown in Listing 3. open_comm_link() creates a VAX/VMS mailbox using the SYS$CREMBX system service, which returns the completion status code. The mailbox is created with the following attributes: it is temporary and both protection and access mode are set to lowest levels. If the mailbox already exists, the SYS$CREMBX service assigns a channel to that mailbox. Thus cooperating processes need not consider which process must execute first to create the mailbox. Table 1 shows arguments passed to open_comm_link().The server then calls the SYS$DCLAST system service to enable the asynchronous receipt of messages. SYS$DCLAST causes an interrupt, declares the receive() primitive as the interrupt handler, and passes rcvmbx as an argument to receive(). The server maintains a message queue for buffering incoming messages. The queue is a fixed length array which is treated as a ring buffer or circular queue. Messages sent to the server's mailbox are received and placed in the queue asynchronously by receive(). The SYS$QIO system service is the key to the receive() routine. Table 2 shows the parameters passed to SYS$QIO.
The example makes use of two powerful features of SYS$QIO, receiving I/O asynchronously and calling a user-defined interrupt service routine when the I/O completes. The server calls receive() (through SYS$DCLAST) immediately after creating its receive mailbox. Receive() posts a read request and returns control to the main loop. When the mailbox receives a message, the SYS$QIO sets the RCVEF local event flag, copies the message and the final I/O completion status into the message queue, and calls the interrupt service routine, passing it mbxid as a parameter. In this case, the interrupt service routine is receive(). Every time the mailbox receives a message, receive() places it in the messge queue and posts a new read request. It might appear to create an infinite loop, but it posts a new read request only after the last request is completed.
Establishing The IPC Link
After startup, the server places itself in a wait state. When the server receives a message, its RCVEF event flag is set and the server drops into the main processing loop. The first message a client process sends is an ADDMBX command (see Listing 4) . This message requests that the server assign a channel to the server-to-client mailbox created by the client process in Listing 4. The name of this mailbox is CLIENTMBX concatenated with the client process' process number. Before assigning the channel to the mailbox, the server first examines its client list, which is a linked list of processes that it is serving. If the process is not in the client list, it adds it to the list and assigns the channel, storing the mbxid in the new list entry.
Pass-through
When client A wants to send a message to client B (and only client B), client A initializes MSGBUF variable and sends it to the server. In the following code fragment, assume that A's process number is 0, that B's is 1, and that the message type is TEXT.
msgbuf.cmdtyp = PASSTHRU; msgbuf.msgtyp = TEXT; msgbuf.xmt_prcnum = 0; msgbuf.rcv_prcnum = 1; strcpy(msgbuf.msg.text, "\nTesting"); send(xmtbmx, sizeof(MSGBUF), msgbuf);The server first checks to see that B is a valid destination. If it is, then the server sends the message to client B by calling the send() primitive (see Listing 3) .Send() uses the SYS$QIOW system service to write a message to a mailbox. SYS$QIOW waits for completion of the write transaction and returns completion status in the variable's status and iosb. In this example, the status returned in iosb is used for debugging purposes only. Send() expects
- mbxid: the mailbox identification.
- msg: address of the message to be sent.
- msgsiz: the size (in bytes) of the message.
Broadcast
A client may broadcast a message to all processes linked to the server. A message is initialized as above, with the cmdtyp set to BROADCAST. rcv_prcnum need not be assigned a value. The server passes the message, message size, and a pointer to the beginning of the client list to the broadcast() primitive shown in Listing 3. broadcast() simply traverses the client list, passing the mbxid for that entry in the linked list to send(), which performs the actual I/O. broadcast() sends the message to all the clients, including the originator of the request.
Deleting The IPC Link
When a client performs a graceful shutdown, it should notify the server of its impending shutdown. Once notified, the server can remove the client from its client list and close its corresponding mailbox. Both client and server call close_comm_link() to close the mailbox. The client closes both of its mailboxes after sending the server a DELMBX request.Close_comm_link() deassigns the channel between a process and the specified VAX/VMS mailbox using the SYS$DASSGN service. In this application, mailboxes are created with the TEMPORARY_MBX attribute, which allows the system to automatically delete the mailbox when all channels to the mailbox have been deassigned. Channels may be deassigned explicitly using close_comm_link() or implicitly upon process termination. The mailbox identification, mbxid, is the only argument passed to close_comm_link().
Server Shutdown
The server performs a graceful shutdown when it receives a SHUTDOWN command from a client. The server broadcasts a SHUTDOWN command to all its clients, closes all mailboxes in the client list, and closes its received mailbox. It then exits.
Flexible Messages
VAX/VMS mailboxes make no restriction on message format. A programmer can take advantage of this flexibility through the union data structure. The MSGBUF typedef may be logically divided into three parts: the message header, the msgtyp field, and the message body (see Listing 1) . The clients and the server use the message header for routing. The message body is a union of whatever types of data may be transferred between processes. The msgtyp field allows the receiving process to intelligently process the data.Listing 4 demonstrates the message format's flexibility. The client process switches on the msgtyp field, which may be either TEXT or INT_ARRAY. In either case, the receiver can access the correct member of the union. This allows the programmer to create any number of message types by declaring a union that includes each message type and by using a corresponding definition that identifies each union member.
Conclusion
VAX/VMS mailboxes offer the programmer a powerful and flexible solution in IPC applications. These primitives provide the programmer with basic IPC functionality. The server architecture offers the advantages of a near-minimum number of links and reduced message passing, yielding predictable performance and good use of resources.