A workable way to do HTTPS from AOLserver

The big picure

We want to do something like <a href=”http://www.aolserver.com/doc/3.0/tcldev/tapi-c68.htm#107131” >ns_httpopen or <a href=”http://www.aolserver.com/doc/3.0/tcldev/tapi-c67.htm#68227” >ns_httpget from the standard <a href=”http://www.aolserver.com”>AOLserver API. We don’t want to implement SSL in Tcl, so we are thankful that it’s already been implemented in C and Perl.

The way I did it and explain here is stupid but it’s quick ‘n easy and it works. See end for more along these lines.

Under the hood

It’s rat-simple in theory but takes a little tweaking to make it work in reality. We need the following components:

• A <a href=”http://www.pl.com/pub/language/info/software.html”>Perl 5 interpreter (probably 5.005 or something). We assume that’s already in place.
  
  
• The OpenSSL module (check with Net:SSLeay what version is required, probably 0.9.3a).
  
  
• The Net:SSLeay module from CPAN (check the <a href=”http://www.cpan.org/modules/01modules.index.html”>module list if you want the latest version of the module).
  
  
• A little Perl hacking
  
  
• A little Tcl hacking
  
  

Installing OpenSSL and the Net:SSLeay module

Installation instructions are included in the tar files. One tip, though: If you don’t have root access on the machine and want to install it in a private directory, you want to say:

$ ./config --prefix=~/bin/openssl
$ make
$ make test
$ make install

for OpenSSL, and

./Makefile.PL LIB=~/lib
make install

for Net::SSLeay.

Beware that Net::SSleay poses requirements to the version of OpenSSL that you use — not always the latest version!

The Perl script

This is the script I used. The problematic part was passing the args. We have a bunch of things, such as host, port, path, headers and body that we want to pass. The way I found simplest for a general solution was passing the args in url encoded form. Everything returned by the foreign server is simply output to stdout.

If you have a more specific need, e.g. one fixed request and a few fixed informations you want to pull out of the reply, you’re much better off hacking together a specialized Perl script, since you don’t have to wait for everything to stream in and pass as much data via arguments and stdout.

• The Perl script (is gone)

The Tcl script

If the returned value doesn’t contain nulls, it’s really easy. If it does, things are a little more difficult.

The problem is that Tcl (v7.4) strings can’t handle binary data in strings, since they terminate at null. What you can do is to pipe the output of the Perl script to a file then slurp up everything from the file instead of from the Tcl string. Remember that with my perl script, the file will contain everything returned from the connection, i.e. both headers and body.

set filename [ns_mktemp /tmp/https-XXXXXX]
exec ~/local/https/https-do.pl [export_url_vars mathod url headers] > $filename

• The Tcl script I used (is gone)

Shortcomings

• Forking is bad. We fork when we exec the Perl script.
  
  
• Blocking until the whole page has been read is bad if we only need something in the top. Modify the Perl script to suit your particular need.
  
  
• Passing arguments to the Perl script on the command line instead of to stdin is probably bad if the body to a HTTPS POST request is big. Modify the Perl script to suit your particular needs if possible.
  
  

  Problems you’ll run into if you try to generalize this idea

  
  
  
• Passing request and reply bodies as Tcl strings is bad if they might contain null characters. It’s probably also very inefficient in large chunks, unless you really want them in memory.
  
  
• Making the choice of HTTP or HTTPS transparent is not really possible, mainly because Tcl’s inability to handle binary data means that we must be able to stream the data. In HTTP this will be from a socket, while it’ll be from a file with HTTPS. Also, HTTPS as done here will require that everything’s received and streamed out to the file before we can start reading it. On the other hand, we know that size of the body without reading it, if that’s all we’re interested in (and how often is that not the case :-)).
  
  

Why this is really stupid

Having a C program interpret Tcl that forks a C program to interpret Perl to call a C program is obviously stupid. Don’t expect it to be very reliable or scalable.

A better approach would probably be to link in the OpenSSL or another SSL library and expose a few Tcl procs that will do secure sockets (like the ns_sock procs). But you should probably make sure the SSL library you link in is thread-safe, that it won’t make AOLserver crash too often (which, I heard, was what happened when someone tried to link in the CyberCash API library) and that you don’t run into legal problems like annoying export restrictions.

Branimir Dolicki suggested another, safer approach: A proxy running as a separate process on the same box as AOLserver, that will accept HTTP connections from localhost on an agreed port and do HTTPS connections on the other end.