Dr. Dobb's Sourcebook January/February 1997
Within the current World Wide Web environment, hypertext links are used to glue together resources that are primarily static, read only, and file based. Although the Web has been successful in serving these "standard" resources, it is questionable whether the current system can scale indefinitely, given the Web's massive growth. For instance, one highly visible problem is broken links between documents, caused by the lack of referential integrity within the current hypertext implementation. With the predicted growth in the number of resources, this problem will increase in significance, and the Web will become more difficult to use and manage.
The future is also likely to bring new demands on the Web in terms of the types of resources that it will be required to serve. The Web has proved its usefulness in the organization of distributed documentation, and users will wish to access other kinds of resources, with richer interfaces within the same environment. Furthermore, service providers will wish to take advantage of the Web to tap the potentially huge customer base.
In this article, we describe how the current web implementation is object based, with a single interface comprising HTTP operations. Although extensions have been implemented to allow nonstandard resources, the system does not exhibit the necessary extensibility to cope with more-advanced resources and services. Consequently, we will examine how making the change to an object-oriented system can yield an extensible infrastructure capable of supporting existing functionality, while allowing the seamless integration of more-complex resources and services. We aim to use proven technical solutions from the distributed-object-oriented community to show how many of the current problems with the Web can be addressed within the proposed model.
Currently, the Web is primarily populated with standard resources such as HTML documents, GIF images, PostScript files, audio files, and the like -- all of which share the common characteristics of being passive, read only, and static. All resources are currently named using URLs, a locational naming scheme that describes the protocol used to access the resource, the Internet address of the server containing the resource, and the location of the resource within that server. HTML documents contain hypertext links that are either intraresource (for providing multiple navigational paths through a document) or interresource (providing the "webglue" that allows related resources to be connected). One of the major advantages of the Web is the speed and ease with which information can be made available. In this environment, web resources are frequently created, moved, and destroyed in an ad hoc manner, suiting the owner of the information or publishing site. These changes are usually made autonomously, with little regard for information users. The distinct problems related to this are: referential integrity, migration transparency, resource management, and quality of service.
Referential Integrity. A system supports referential integrity if it guarantees that resources will continue to exist as long as there are outstanding references to the resources. The Web does not support this property and cannot do so since the system is unaware of the number of references that exist to a particular resource.
It is impractical to maintain every resource that has ever been published on a particular server forever; this simply does not scale. Resources that are no longer of value, for whatever reason, become garbage and need to be collected. This may involve moving the resources to back-up storage, or in some cases, deleting the resources entirely. Access-pattern information (currently available through examination of server logs) is not a sufficient basis to decide whether or not an object is safe to garbage collect, as important (though rarely used) references to a resource may exist. Safe garbage collection can only be performed if referencing information is available.
The consequences of deleting resources that are still referenced affect both users and information providers. In the web environment, deleting a resource that is referenced by another resource results in a broken hypertext link. Such broken links are the single most annoying problem faced by users. Broken links result in a tarnished reputation for the provider of the document containing the link, annoyance for document users, and possible lost opportunity for the owner of the resource pointed to by the link.
Migration Transparency. In addition to the problems associated with deleting web resources, migrating resources (either intra- or interserver) also have the potential to break hypertext links. When a resource moves with the URL naming scheme, its identity also changes. Thus, hypertext links to the old name break.
A partial solution to this problem is provided by the HTTP redirect directive: a forwarding pointer to the new location that allows clients to rebind to the resource (automatically, in the case of redirection-aware browsers). However, this is only a partial solution because documents containing references to the old location of the resource are not automatically updated, and future requests will continue to access the old location first. Also, even if there were an automatic-update mechanism, the lack of referential integrity means that the redirector can never be safely removed, since it is impossible to determine whether all of the links have been updated. There is also the possibility that the URL may be reallocated following the migration.
The IETF Uniform Resource Identifiers (URI) Working Group's work on Uniform Resource Names (URN), an alternative naming scheme based on logical rather than locational naming, attempts to address the problem of migration transparency. The mapping from logical name (URN) to locational name (URL) is stored within a name server that is updated with the new URL when the resource moves. The disadvantage of this approach is the performance penalty associated with name-server lookups, updates, and access bottlenecks. Furthermore, this scheme does not address the issue of referential integrity.
Resource and Service Management. Recall that most web-resource accesses are read-only operations. Updates (using the native commands and editing tools of the server machine) to the resources by information providers are performed using mechanisms orthogonal to the Web. In effect, web resources reside in two distinct domains in parallel: the traditional structure of the file system and the complex interlinking web of hypertext. Within these two environments, the interfaces to the resources -- as well as the relationships between the resources -- are fundamentally different; see Figure 1.
Maintenance operations carried out by information providers or site maintainers typically require manipulation of the resources within both domains. Moving a resource within the file system has the side effect of changing its name in the web domain. To reflect such changes, other web resources must also be modified. Further, the internal state of the moved resource may also require modification due to the use of relative naming. At present, such changes are performed manually and are prone to mistakes and inconsistencies (although tools such as MOMSpider, at http://www.ics.uci.edu/ WebSoft/MOMspider/WWW94/paper.html, help detect some of these). This is the result of having two parallel interfaces without any support for maintaining consistency.
Quality of Service. The perceived quality of service (QoS) of the Web is influenced by many factors, including the broken-link problems. Even if users hold a correct reference to an existing web resource, it may still be unavailable for a number of reasons, such as unavailability of the machine serving the resource, or partitions in the network between the client and server. Partitions may either be real (caused by breaks in the physical network) or virtual (due to excessive network or server load), making communications between the client and server impossible. Even if communication is possible, poor response characteristics may effectively make the resource unusable. QoS will become more of an issue as the Web continues its transformation into a commercially oriented system.
Technical solutions for improving QoS are fairly well understood, including caching for responsiveness, replication for availability, and migration for load balancing. Caching in the Web is reasonably common, through the use of browser memory and disk caches, and the use of caching servers. Effective caching is not trivial since there are many subtle problems that need to be addressed, including cache consistency, accounting, and so on. Current caching servers use a heuristic approach for consistency management, where resources can only apply coarse-grained tuning based on expiry dates. The IETF URI working group is implementing a framework for resource replication, but appears to only be addressing the problem within the realm of read-only, static resources where a read-from-any policy is appropriate. Replicating more complex read/write resources is much more difficult due to the problems associated with maintaining consistency between concurrent users and in the presence of failures.
Web resources that possess a richer interface than standard resources are manipulated through CGI protocol, which sits above HTTP. Through this protocol, it is possible to perform arbitrary operations not supported directly by HTTP. Such resources are still identified by URLs, but the HTTP daemon is able to distinguish CGI URLs from standard resources. Upon receiving a request containing a CGI URL, the daemon invokes a CGI script identified in the URL rather than dealing with the request itself. Additional parameters can be included in the URL that the daemon passes directly to the script, and can be used in an application-specific manner in the servicing of the request. When the script finishes the request, it returns a reply message to the client and terminates.
Unlike publishing standard resources, writing CGI scripts is not simple; resources manipulated no longer need to be read-only, and the issues involved in interacting with them are more complex. There is no support for controlled sharing of resources, and because HTTP is still the underlying communication protocol, connections between client and resource last only for each request. This forces users to adopt ad hoc solutions to problems such as persistence, concurrency control, and sharing of resources between requests.
The HTTP protocol specifies the operations a client can attempt to perform on a web resource -- GET, POST, and the like. There is also support for an extension method that allows user-specific operations to be added to the protocol. Resource providers using this feature must register these operations with the Internet Assigned Numbers Authority (IANA). The IETF has also proposed new standard methods to be incorporated into HTTP in response to user requests. Irrespective of the approach used to add new methods to the protocol, changes will be required to both the daemon and browsers. Further, new methods may be inappropriate for the majority of web resources, and there will still be requirements that they do not address. This will result in another round of protocol negotiations and changes to web software.
The result of this model and of the constraints it imposes is that people may change the daemons in an ad hoc manner. This makes migration of web resources from modified daemons difficult because modifications need to be made to every daemon that services them. There is also no easy route for code reuse, leading to individual implementation of similar methods in different ways.
From an object perspective, the HTTP daemon provides a single interface through which all objects it manages are accessed. In effect, the current Web may be viewed as an object-based system with a single class of object. What is needed is the ability to modify this interface on a per-resource basis. Within the object-oriented paradigm, such specializations are achieved through the use of inheritance, and it is the absence of this that distinguishes an object-based from an object-oriented system.
We believe that the model and implementation we present here represents a cleaner way of addressing the issues of flexibility and extensibility in a uniform manner, and can be used to provide solutions to the aforementioned. They can also be applied to the current Web with little or no modification from users of these resources, and with minor requirements from their providers.
The primary objective of our research is to develop an extensible web infrastructure capable of supporting a range of resources and services. Our model makes extensive use of the concepts of object orientation to achieve the necessary extensibility characteristics. Within this object-oriented framework, proven concepts from the distributed-object-oriented community can be applied to the problems currently facing the Web.
In our proposed model, web resources are transformed from file-based resources into objects called "W3Objects" -- encapsulated resources possessing internal state and well-defined behaviors. The objects themselves are responsible for managing their own state transitions and properties in response to method invocations. This model supports abstraction, since clients only interact with W3Objects through the published interfaces (the implementation of a particular operation is not externally visible).
Different classes of W3Objects support different operational interfaces obtained through the use of interface inheritance. Abstract classes are used to define an interface to a particular object abstraction, without specifying any particular implementation of the operations. Different classes of W3Objects may share conformance to a particular abstract interface, but may implement the operations differently, in a manner appropriate to the particular class. Consider, for example, a Manageable interface (that includes a migrate operation) for moving objects from one location to another. While the same interface is appropriate for many classes of W3Objects, the implementations may differ; for example, migration of a hypertext object may require some internal link-manipulation operations in addition to the operations required by, say, a text file.
The use of interface inheritance provides polymorphism; all derived classes that conform to an interface provided by some base class may be treated as instances of that base class without regard for any other aspect of that class's behavior. Continuing with the previous example, consider a dedicated GUI-based web-site management tool that provides a graphical interface for performing management-style operations on the objects (one such operation may be object migration). The management tool is able to abstract away from other features of the different objects (supported through various other interfaces) and simply address all of the different objects as instances of the Manageable interface. In addition to inheritance of interface, the model also supports behavioral inheritance, thereby supporting code reuse. For example, object properties (such as persistence and concurrency control) may be provided by mixin base classes to be inherited as required. mixin classes are not designed to be instantiated themselves. They are used to augment the functionality of the derived class by providing some particular behavior, usually orthogonal to the primary function of the class.
Figure 2 illustrates the key points of our object model by showing how two W3Object classes, Spreadsheet and Play, are composed using both interface and behavioral inheritance. The abstract class Manageable provides the interface description for management-style operations (only a single operation, migrate, is shown). Both derived classes inherit this interface, providing their own implementations. In addition, both derived classes provide other interfaces, describing the primary operations of the classes. The Play class, representing a theatrical performance, provides a bookSeat operation, and Spreadsheet provides get and set operations for manipulating the contents of the spreadsheet cells. Further, the Spreadsheet class is concurrency controlled, having derived this property from the mixin class, ConcControllable. The operations setLock and releaseLock are used in the implementation of the get and set operations to ensure the integrity of the spreadsheet in the event of concurrent access.
Figure 2 also shows three clients that manipulate instances of Play and Spreadsheet. A web-site management tool is solely concerned with the operations provided through the Manageable interface. The tool is able to invoke the migrate operation on instances of either derived class without knowledge of the nature of the class. Two other clients, a theater-booking application and spreadsheet tool, manipulate instances of Play and Spreadsheet, respectively. That these classes also conform to the Manageable interface is of no consequence to those clients interacting with the objects solely via the interfaces supporting the classes' primary function.
The proposed W3Object architecture consists of three basic entity types: clients, servers, and published objects. In the current web environment, these types correspond to web browsers, daemons (CERN HTTPD, for instance), and documentation resources (HTML documents), respectively. Our architecture supports both client-object (client-server) and interobject (peer-to-peer) communication.
Client-Object Interactions. Figure 3 illustrates the logical view of client-object interactions within the W3Object architecture. A single server process is shown managing a single W3Object (although servers are capable of managing multiple objects of different types) that is being accessed via two different clients: a standard web browser and a dedicated bespoke application. Figure 3 highlights interoperability as one of the key concepts of the architecture -- the support for object accessibility via different applications using multiple protocols.
W3Objects are encapsulated, meaning that they are responsible for managing their own properties (security, persistence, concurrency control, and the like) rather than the application accessing the object. In the case of concurrency, for example, the object manages its own access control based on its internal policy, irrespective of which application method invocations originate from.
The local representation of an object, together with the available operations, may vary depending upon the particular type of client accessing it. The web browser uses a URL to bind to the particular object in the server. The operations permitted on the object (via the URL) are defined by the HTTP protocol. The HTTP communication end point of the server may perform name mapping between the URL and the internal name for the object and may also map HTTP requests to appropriate method invocations on the object. Within the bespoke application, a client-stub object acts as the local representation of the remote object. From the application's point of view, this stub object creates the illusion that the remote object is actually within the address space of the client. Like any other object, the stub presents a well-defined interface describing the supported operations. This interface has the potential to be much richer than that provided through HTTP. Operation invocations on the stub are passed to the object using the RPC protocol. Client-stub objects may be automatically generated from a description of an object interface. Our implementation uses C++ as the definition language and we provide stub-generation support for creating client- and server-side stubs to handle operation invocation and parameter marshaling. Other possible interface-definition languages include CORBA IDL (ftp://ftp.omg.org/ pub/docs/93-12-43.ps) and ILU ISL (ftp:// parcftp.parc.xerox.com /pub/ilu/ilu.html).
To support access via conventional browsers using HTTP, the example theater object would conform to an HTTP interface supporting an operation, say, httpGet. An HTTP GET request from the browser would be mapped to an invocation of the httpGet method of the object. Through this limited interface, the program of events and availability information may be accessed. A client-stub object for the theater may support more advanced operations, such as seat booking, using atomic actions (transactions).
A number of other client implementations are possible. CGI could be used to provide a richer client-side interface than is readily available through HTTP. Again, we believe that CGI is too low-level for direct programming and that CGI interfaces to remote objects can be automatically created using stub-generation tools. We have implemented a basic stub generator that uses an abstract definition of the remote object, and ANSA (http:// www .ansa.co.uk/ANSA/ISF/1464/1464prt1.html) has recently released a more complete tool based on CORBA IDL. Recent developments using interpreted languages, including Java and SafeTcl, are potentially useful for developing client-side interfaces to W3Objects. Using such languages, complex, architecture-neutral front ends dedicated to a particular W3Object class can be developed, supporting rich sets of operations.
Interobject Interactions. In addition to client-object communication, our architecture also supports interobject communication, regardless of the objects' locations. In effect, the architecture may be viewed as a single distributed service, partitioned over different hosts; see Figure 4. Interobject communication is used for a variety of purposes, including referencing, migration, caching, and replication.
In addition to W3Objects, servers may contain W3Object stubs (or aliases): named objects that simply forward operation invocations to another object, transparently, to clients. One particular use of aliases is in the implementation of name servers, since a name server may be viewed as a collection of named objects that alias other objects with alternative names (Server S1 in Figure 4). Objects may also contain stubs to other objects (S2 in Figure 4). This feature is used in our implementation of referencing.
Implementation Considerations. A particular W3Object server is capable of holding W3Objects belonging to a set of specific types. At the time it is created, it is unlikely that you can predict all of the object types that a particular server will be required to serve over its lifetime. This raises the question of how new types are introduced to the system. The usual approach is to bring down a server, rebuild it with the code that supports the new object types, and then restart it. This approach is less than ideal for an application such as the Web, where there is a requirement for keeping downtime to a minimum. An alternative technique relies on dynamic loading, allowing an active process to load new code as necessary. Although this approach is clearly more elegant and we do not preclude its use, dynamic loaders currently suffer from lack of standardization and, therefore, lack of portability. A third and more pragmatic approach is to create another server to support the new object types. This approach may be desirable even if suitable dynamic-loading support is available due to requirements for fault tolerance and security: Since a single server for all object types is a single point of failure, multiple servers can be used to protect users against such failures.
One method of interfacing with multiple servers is to make use of an HTTP gateway that uses stub objects to forward object invocations to the appropriate server. We have implemented such a gateway as a plug-in module for the Apache web server. The server is configured to pass requests for part of the server's URL space (for example, URLs beginning "/w3o/") to the gateway module. In response to such a request, the W3Objects module binds to the appropriate object and invokes a particular method (for example, httpGet), passing all parameters and environment information. The object's response is returned to the client. This is illustrated in Figure 5, where Server S1 manages a number of different types of objects (illustrated by different shapes) and Server S2 manages objects of a single type. As the processing of operations is entirely the responsibility of the individual object, the introduction of new object types is transparent to the gateway.
We have identified the set of properties required by W3Objects and classified them into three categories: core properties, common properties, and class-specific properties.
Core Properties. Four properties have been identified as being the core requirements for W3Objects: Naming, Sharing, Mobility, and Referencing. The implementation of these properties is divided between the objects themselves and the supporting infrastructure that manages the objects.
Common Properties. There are a potentially large number of common properties for W3Objects that can be encapsulated within appropriate base classes.
Having described our model, we now illustrate how two of the core properties -- referencing and mobility -- are implemented within the model. Our aim is to address the current problem of broken links and provide transparent object migration.
In our model, web resources are represented as W3Objects and may be referenced from some root, either directly via W3Object stubs or by being contained within another W3Object (W3Object stubs are themselves W3Objects). Figure 6 shows a number of objects (all reachable from some root). Our service maintains the distributed-referencing graph and uses reference counting to detect unreferenced objects. When created, stubs perform an explicit bind operation on the object they refer to (thereby incrementing the object's reference count) and perform an unbind operation whenever the stub is deleted (thereby decrementing the count).
Reference from a root represents the service's continued interest in the object while stubs represent hypertext links from one object to another. Therefore an object is not available for deletion until the service has expressed its lack of further interest in the object (by no longer referencing it from a root), or there are no hypertext links from other objects. Once these conditions are met, the object is marked as unreferenced and is available for deletion.
An object may be moved from one location to another (potentially to some remote site). Our service guarantees that object moves are transparent to all other objects (references to the moved object continue to be valid). Whenever an object is moved, a stub representing the object is left behind at the old location, automatically forwarding invocations (without involving the invoker) to the new location. Further moves of the object may cause a chain of such stubs to be created (in a similar manner to that used by SSP chains), ultimately leading to the real object. For example, in Figure 7, object Y has been moved, first to server B, then to server C, leaving behind the forwarding chain shown. Whenever an invocation follows such a chain, these indirections are short cut; for instance, the invoker's stub communicates directly with the W3Object at the new location. Following a short cut, intermediate stubs no longer referenced are automatically garbage collected.
The short-cutting mechanism stubs will generally point directly at their W3Object; however, there may be circumstances where the chain becomes longer than is desirable (for instance, the object moves a number of times between invocations from a particular stub). Long chains have more points of failure and a performance cost in the necessary indirections at invocation time; consequently there may be situations where long chains should be avoided. Therefore as an extension to our basic forwarding location mechanism, we allow stubs to register callbacks with objects so that whenever an object moves, the registered stubs are automatically informed of the move. These stubs may represent hypertext links from other resources and/or belong to a location service that offers the current position of the object to any of its users.
Our use of forward references is an entirely distributed solution that scales well by not relying on centralized-location services or broadcasts of moves. An alternative approach proposed by Hyper-G (http://info.iicm.tu-graz.ac.at/) is to ensure that all references to all resources are (eventually) updated to reflect any movement. However, this requires additional network traffic that may be appropriate for individual enterprises but does not scale to a worldwide solution. Our system also guarantees that any object movement is transparent to any reference holder and does not require the handling of object faults and subsequent rebinding. Our policy is to provide inexpensive forward referencing (introducing no extra network traffic when an object is moved) as the default location service, while also offering alternative, potentially more-expensive solutions such as callback and the use of location servers for resources with specific performance or fault-tolerance requirements. Our model allows the choice of appropriate location mechanisms for an application and the ability to vary that choice at run time as required.
The mechanisms just described offer a consistent view of referencing within the system so long as the referenced objects and their stubs cannot be lost through failures, and the bind and unbind operations are totally reliable. We guarantee that W3Objects and stubs recover from failures by saving them on stable store (a file system, for example). The bind and unbind operations use an RPC to access the W3Object, which may be held within a remote server (optimized to a local procedure call whenever the object is held in the same server). Our remote object-invocation layer guarantees "at-most-once" semantics for the RPC (despite failures operations will eventually be performed).
Using the Shadows lightweight object-support system, we have started the implementation of the core W3Object properties, concentrating initially on referential integrity and object mobility. During this prototyping stage, we have developed a system that operates in parallel with conventional HTTP daemons. Object states are stored in their native formats, (HTML files, and the like) and are available for direct manipulation through standard file system and web interfaces. The advantage of this approach is that we are able to develop and test our design without requiring site maintainers to alter existing web configurations. The disadvantage is the inherent loose coupling between the two systems, requiring users to register entities with our service and to be sufficiently disciplined so as not to delete entities directly through the file system without consulting the service. After this prototyping stage, we will introduce the HTTP gateway and provide user-level tools to replace the direct manipulation of the object through the file system, without restricting the use of existing resource-manipulation tools (text editors, for instance).
In the prototype system, an additional daemon per host provides the referencing service (administered using a GUI-based tool). This tool allows site maintainers to browse through the entities currently being served by the HTTP daemon and to include them within the service. On being instructed to include a web entity, the service creates a corresponding W3Object that is registered, causing it to be referenced from a root (thereby ensuring it is not immediately marked as unreferenced). The administration tool then parses the resource to identify any hyper-text links. If the links point to resources already included in the service, then references to their corresponding objects are created and stored within the newly created object (thereby extending the referencing graph). Links to resources not already within the service are reported back to users who have the option of explicitly including them if they are under user control. A deregister operation expresses the site maintainer's lack of further interest in a resource; the reference count for the resource is then decremented. If the reference count reaches zero, the site maintainer is informed that it is available for deletion. If the site maintainer confirms it is to be deleted, then the W3Object is removed from the service and the corresponding resource removed from the file system. Deleting an object may result in the reference counts of other objects being decremented. The site maintainer may move a resource from one location to another (potentially, to some remote site where the resource may come under new ownership). The service also removes the resource from the file system, inserts an HTTP redirector, transforms all relative URLs contained within the moving resource, and places the resource in the file system at the new location.
The next stage of our development is to implement the full W3Object model as described in this article, integrating the referencing work with other software subsystems developed locally as part of the Arjuna Project (http://arjuna.ncl.ac.uk/). To provide the flexibility and extensibility we have described, we shall make use of the Gandiva system (http://arjuna.ncl.ac.uk/arjuna/ papers/framework-extensible-software.ps), which supports the separation of object interfaces from implementation and enables reconfiguration of this relationship without requiring the rebuilding of objects or applications. For the common W3Object properties, we will make use of the work developed for the Arjuna system, which provides the necessary fault-tolerance and replication support.
The W3Object model is intended to provide a flexible and extensible way of building web applications, with web resources encapsulated as objects with well-defined interfaces. Objects inherit desirable characteristics, redefining operations as necessary. Users interact with these objects in a uniform manner. A key feature of our design is support for interoperability. In addition to sophisticated clients that can use the rich-object interfaces that our model provides, our implementation will allow W3Objects to continue to be accessed using existing web browsers.
This article is adapted from the paper entitled "W3Objects: Bringing Object-Oriented Technology to the Web," first published in the proceedings of the Fourth International World Wide Web Conference in Boston, December 1995. The work reported here has been partially funded by grants from the Engineering and Physical Sciences Research Council (EPSRC) and the United Kingdom Ministry of Defense (Grant Numbers GR/H81078 and GR/K34863) and GEC-Plessey Telecommunications.
DDJ