Securing Web Consoles with Apache

Michael Heironimus

While many Unix administrators prefer to avoid such things, Web-based administration tools are becoming a standard part of life in many environments. Unfortunately, some vendors' push to put their products on the Web has come at the expense of security. Some Web consoles have security flaws; others offer only very limited authentication options. Some sites also have specific needs that general-purpose products do not address. Luckily, the industry-standard Apache Web server can be used as a reverse proxy to replace or augment the security of many Web consoles.

What Is a Reverse Proxy?

A reverse proxy is a middleman. Like a conventional proxy (a "forward" proxy), it accepts a request from a client (a browser). The proxy opens a new connection to the corresponding back-end server and sends a request matching the one it received, possibly with some remapping of the URL or other processing, and returns the server's response to the client.

While a forward proxy handles outbound connections for clients, a reverse proxy handles inbound connections for servers. A reverse proxy looks to the client like an ordinary Web server. In fact, a properly configured reverse proxy is nearly indistinguishable from the real server. URLs on the proxy are mapped to back-end server URLs, and response headers have the back-end URL replaced with the proxy URL before being returned to the client. This allows cookies and redirects, but not content, to be rewritten to match the proxy instead of the back-end server.

Protecting Web Consoles

A reverse proxy can be used with most Web interfaces, with some limitations. Why would anybody need to do such a thing? Putting an Apache reverse proxy in front of another server adds the capabilities of Apache while requiring few or no changes to the server or application being proxied. The basic process of protecting a Web console is simple -- create a proxy to pass requests through, then block direct access to the console.

There are a variety of options for blocking console access. If the servers are in a protected network segment, the console port can simply be blocked at the firewall. If there is no firewall or if users have access to other machines in the same segment, a host-based firewall such as IPFilter, IPTables, or PF could be used.

However, a simpler, and often less-intrusive, method is to bind the console to the loopback interface instead of to the external interface. The procedure for this will often be different from one console to another, and the vendor may or may not provide usable documentation. This also allows the proxy to use the original console's port, providing a nearly seamless transition from direct console access to proxied access. This will not protect the console from access by users who can log in to the server itself; however, granting shell access to anybody other than administrators (who would have legitimate console access) creates a wide variety of other issues for most applications anyway.

Proxy Capabilities

For those not already familiar with Apache, here is a short list of some of the options it can offer to Web consoles.

Authentication

Most current Web consoles provide for authentication, but it is often limited to simple username/password logins against LDAP, local OS accounts, or a flat file. In one case, non-privileged accounts could reside in LDAP, but administrative accounts could only be defined in a local flat file. In another, the vendor told administrators on a support call that the master administrative login could not be changed from the shipped default and that no password could be set without causing problems.

Apache can provide many more options for authentication. Support for plain passwords, LDAP (which can be used with Active Directory), and SSL client certificates are all included in the Apache release. With third-party modules, Apache can also authenticate using PAM, database tables, Kerberos, SecurID tokens, various third-party provisioning systems, or nearly anything else. Depending on exactly which modules are used, multiple authentication systems can sometimes be used for a single path. Furthermore, different URLs within the same proxy can be protected with completely different authentication systems to meet the needs of different groups.

This may not be a clean solution -- it will often force two logins, the first to the proxy and the second to the administrative console itself. This, like most aspects of security, is a trade-off that each site must consider.

Access Control

Apache can be configured with ACLs to limit access to specific networks or IPs, or to explicitly block access from specific networks or IPs. Few Web consoles support host- or network-based access control, and some of those that do are based on Apache. As with user authentication methods, different URLs can be protected with different ACLs. Apache can also provide a standardized environment for these ACLs, reducing the amount of training and research that must be done to implement such controls on a new console.

Logging

Many Web consoles offer little or no flexibility in logging. Apache provides many options for detailed logging, especially when used with mod_log_forensic or a third-party auditing module such as mod_security. Log analysis software can be used to report on activity. mod_security auditing can provide a complete trace of all activity, including POST submissions. With the appropriate third-party module, Apache can even log to a remote database. This can be used to create a single repository for console access logs.

SSL

Some Web consoles still do not use SSL by default, even for administrative logins. Others cannot enable SSL for the console without making other changes, which the administrators may or may not be prepared to make.

One well-known Java application server supports SSL for communication among all components. It even goes so far as to support mutual authentication. However, the console certificates are overloaded -- the same SSL configuration is used for administrative logins as for connections between the administrative server and other components of the application server.

This presents a problem when administrators attempt to secure all of the internal communications using SSL certificate authentication. If the console is configured not to require mutual authentication, anybody who can access the port can attempt to access the same configuration data that the other server components use. If the console does require authentication, client certificates will need to be issued to all administrators, potentially creating a nightmare of certificate management.

Instead of issuing client certificates to administrators, a single client certificate can be issued for an Apache proxy. Administrators will connect to the proxy, which will not require a client certificate. The proxy will then initiate a new connection to the console, presenting its client certificate for access.

Administration "Portals" and Firewalls

Most companies will protect some or all of their servers with firewalls. More Web consoles require more firewall rules -- more work to manage, more work to install new consoles, and more rules to be processed for network traffic. The term "Web portal" has become a technology buzzword, but in the most general sense a Web portal is a single entry point to a variety of related services or applications. A Web-based administrative console may have much greater authority than a simple Web application, but it is still a Web application. A Web proxy can provide a single entry point to all consoles in a given network segment though a single port.

Different consoles can be protected with different access rules, providing the necessary flexibility for multiple console types in the same segment. This also makes it possible to prevent users from accessing sensitive consoles by connecting from other machines in the same network -- each back-end console can itself be protected by another Apache proxy that only allows access from the entry point.

This is a security trade-off. The two options are to allow a large number of specific ports or to grant more general access through a single control point, most likely a hardened server. Different companies will have different policies regarding this type of configuration.

Proxy Troubleshooting

One of the nice things about an Apache proxy is that it will either work or fail with great consistency. Once it works, it will continue working until somebody makes a change. The overwhelming majority of proxy problems tend to be caused by the network or the back-end server, not the proxy itself.

502 (Bad Gateway) Errors

The Apache proxy has no way to detect the status of the back-end server. If the server can not be contacted, the proxy returns a 502 error code. Users will no longer get "connection refused" errors from their browsers, nor will they generally get timeout messages. Users not familiar with proxies will commonly assume this to be a problem with the proxy, but in reality it is almost always a problem with the application or console being proxied.

Unresponsive Consoles

Just as with the 502 error, new users will often suspect the proxy. Sadly, most current Web consoles are far less reliable than an Apache proxy. Generally, the first thing to check when a proxied Web console is slow or otherwise unresponsive is the back-end console. A browser installed on the server or an SSH tunnel can bypass the proxy for a quick test.

Apache Directives for Reverse Proxies

Before any proxying can be configured, the appropriate Apache modules must be enabled. mod_proxy is the main proxy module. Web proxies need mod_proxy_http; the other proxy modules are used for other types of proxied connections. mod_ssl is necessary when the back-end server uses SSL.

The main directives for configuring a reverse proxy are ProxyPass and ProxyPassReverse. The first maps the path on the proxy to the back-end server URL. The second controls the rewriting of the response headers to replace the back-end server with the proxy. These directives are processed in order from top to bottom, and the first match is used.

mod_rewrite can also be used for more advanced proxying by using the P flag in the RewriteRule. The ProxyPassReverse directive is still used when the proxying is controlled by mod_rewrite.

Another important directive is SSLProxyEngine. This must be enabled if any of the back-end servers are using SSL. Other SSL-related proxy directives allow the proxy to present a client certificate to the server and control what validation, if any, the proxy does on the certificate presented by the back-end server.

One specific directive not used with reverse proxies is worth noting to avoid confusion. The ProxyRequests directive is used only for forward proxies; it should never be enabled on a reverse proxy. Enabling ProxyRequests without the necessary proxy security would allow anybody who can connect to the proxy to use it as a relay point to access other systems.

Remapping URLs

Mapping different paths on the proxy than the application itself uses can sometimes create new problems. As previously mentioned, Apache itself does not modify the content of the Web pages, only the headers. This means that content that includes absolute URLs will generally not work correctly. However, there is a third-party module that provides the necessary functionality. mod_proxy_html acts as a filter on the response being sent back to the client to replace paths in image tags, links, and other content.

One Web-based tool that exhibits this behavior is Webmin. A single proxy cannot provide access to more than one Webmin server without being able to map each server to a different path. A few simple lines in the server configuration can accomplish this:

ProxyPass /webmin/server1/ https://server1:10000/
ProxyPassReverse /webmin/server1/ https://server1:10000/
<Location /webmin/server1/>
    SetOutputFilter proxy-html
    ProxyHTMLURLMap / /webmin/server1/
    ProxyHTMLURLMap /webmin/server1/ /webmin/server1/
</Location>

Case Study 1: Information Exposure

At a customer site, I found myself supporting several servers hosting one particular piece of middleware. The vendor provided an administrative console running in Tomcat. Modifications to the configuration required an administrative login, but some configuration variables were visible to unauthenticated users. On this particular system this interface exposed database logins and passwords. The password was stored in an obfuscated format in the configuration files, but the Web interface "helpfully" decoded it and presented it in clear to anybody who knew the path.

The application administrator had contacted the vendor, who declined to offer any solution, saying only that those resources might be secured in a future release. The problem was brought to me, and I quickly put together a plan.

The first step was to reconfigure the Tomcat server for the console to bind only to the loopback interface. As with most embedded components, the Tomcat installation was buried in a subdirectory of the product installation. However, it was largely unchanged from a standard Tomcat installation. A quick visit to the documentation on the Tomcat Web site revealed the necessary option in the server.xml to explicitly bind the listener to a specific IP instead of to all interfaces:

Original:

    <Connector className="org.apache.catalina.connector.http.HttpConnector"
        port="8080" minProcessors="5" maxProcessors="75"
        enableLookups="true" redirectPort="8443" acceptCount="10" debug="0"
        connectionTimeout="60000" allowChunking="false"/>

    <Connector className="org.apache.catalina.connector.http.HttpConnector"
        address="127.0.0.1" port="8080" minProcessors="5" maxProcessors="75"
        enableLookups="true" redirectPort="8443" acceptCount="10" debug="0"
        connectionTimeout="60000" allowChunking="false"/>

The final step was to block proxy access to the path containing sensitive information. The path could actually be protected with a login or by IP range, but in this particular case there was no need for anybody to access this information. The path was configured to deny all requests and return a 403 (Forbidden) error. A simple Location block within the virtual host definition was sufficient, although a RewriteRule with the F flag could also have been used:

<Location /path/to/protect>
    Order deny,allow
    Deny from all
</Location>
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

This situation arose with another Unix administrator. After a Windows worm spread through the client's sites over the WAN, the Nessus security scanning tool was used to scan entire networks for unpatched Windows machines. The longer-term plan was to provide on-demand security scanning from a central Unix server running Nessus. The administrator had created a Webmin custom command and a wrapper script to allow people to submit scan requests through a Web interface. The plan was to grant Windows administrators access to run scans through a simple form in Webmin. However, Webmin's built-in security did not provide any easy way to authenticate users against the Windows domain. The only available mechanism for this was PAM, which would not have allowed Webmin access to be granted based on Windows group membership.

Our solution was to move the security out of Webmin and into Apache. Webmin was configured to allow access to the scan script without authentication. An Apache reverse proxy authenticated users against one of the Active Directory LDAP servers. This required no local users and created almost no administrative overhead -- the users already existed in Active Directory.

LDAP-based authentication in Apache is provided by mod_auth_ldap. A setup such as this would normally use LDAP over SSL (ldaps), but SSL was unavailable on the Active Directory server at this site. Configuring Apache to use SSL would be a simple matter of changing the AuthLDAPURL, assuming that the LDAP module was compiled to support it. If anonymous searches were enabled the AuthLDAPBindDN and AuthLDAPBindPassword directives could also have been omitted:

AuthType Basic
AuthName AD
AuthLDAPEnabled on
AuthLDAPURL ldap://activedirectory:389/ \
  ou=users,o=company?sAMAccountName?sub?(objectClass=*)
AuthLDAPBindDN cn=adread,ou=users,o=company
AuthLDAPBindPassword adread
Require group cn=scangroup,ou=groups,o=company
ProxyPass / https://localhost:10000/
ProxyPassReverse / https://localhost:10000/

Case Study 3: Remapping URLs

The next step with the Webmin scan interface was to map the custom command page, which is /custom in Webmin, to the root level on the proxy. Webmin's interface also includes graphics under /images and a login page at /session_login.cgi. While these last two could probably be omitted in this situation, the system is more complete with both included. One additional benefit to this setup is that the other functionality of Webmin is not accessible through the proxy at all:

ProxyPass /session_login.cgi https://server1:10000/session_login.cgi
ProxyPassReverse /session_login.cgi \
  https://server1:10000/session_login.cgi
ProxyPass /images/ https://server1:10000/images/
ProxyPassReverse /images/ https://server1:10000/images/
ProxyPass / https://server1:10000/custom/
ProxyPassReverse / https://server1:10000/custom/

Fancy Tricks

Real-Time Notifications

The previous examples were fairly straightforward. Apache also allows more advanced configurations. For example, the following directives would execute a CGI any time a restricted path was requested:

ScriptAlias /kshexH7 /web/cgi-bin
<Directory /web/cgi-bin>
    AllowOverride None
    Options ExecCGI
    Order allow,deny
    Allow from all
</Directory>
RewriteRule "^/restricted/path" "/kshexH7/log_request.cgi?" [PT]
RewriteRule "^/kshexH7/.*" - [T=application/x-httpd-cgi,L]

printf "Request logged `date`\n\nURI: $SCRIPT_URI \nHost: $REMOTE_ADDR \
  \n" | mail -s "`hostname` security" administrator@example.com

cat <<EOF
Content-Type: text/html; charset=iso-8859-1
Status: 403 Forbidden

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access ${SCRIPT_URL}
on this server.</p>
</body></html>

EOF

Attempts to access restricted paths can also be written to separate logs. A SetEnv, SetEnvIf, or RewriteRule directive can be used to set a variable on those requests. An additional CustomLog directive with the appropriate "env=" clause will then log all of those requests to a dedicated file.

Apache -- The Swiss Army Knife

Apache reverse proxies offer many options to enhance the security of Web consoles. The same techniques work with other Apache-based servers, such as IBM HTTP Server, and can be applied to nearly any Web-based tool or application. The work described here is only the tip of the iceberg -- a little imagination and experience will open a whole world of possibilities.

Related Links

Apache Tomcat -- http://tomcat.apache.org/

Apache Web Server -- http://httpd.apache.org/

Mod_proxy_html -- http://apache.webthing.com/mod_proxy_html/

Mod_security -- http://www.modsecurity.org/

Webmin -- http://www.webmin.com/

Michael Heironimus has been working in various areas of systems administration for more than seven years. He has supported a number of Unix systems and Web technologies in both large and small environments, using a mixture of free and commercial tools.