Python Secret Weblog for 2005 November 11 (entry 0)

(8) Fri Nov 11 2005 16:41 Tramline source code now available:

At the Plone conference 2005 I gave a lightning talk about tramline, a lightweight up and download accelator for web applications. Now at last I've found some time to put the source code online. This is not a proper release yet, but it's there for interested people to take a look at it.

What is tramline about? From the readme:

Tramline is a upload and download accelerator that plugs into Apache, using mod_python. Its aim is to make downloading and uploading large media to an application server easy and fast, without overloading the application server with large amounts of binary data.

Tramline integrates into Apache using mod_python. The application server is assumed to sit behind Apache, for instance hooked up using mod_proxy or mod_rewrite.

Tramline takes over uploading and downloading files, handling these within Apache. Only a small configuration change in Apache should be necessary to enable tramline.

The application server remains in complete control over security, page and form rendering, and everything else. Minimal changes are necessary to any application to enable it to work with tramline; in fact it's just setting two response headers in a few places in the code.

Tramline is generic code, but is particularly useful for Zope applications. Zope's object database, the ZODB, has one drawback: it doesn't scale very well when large binary files are put into it. In addition, many appservers have only limited resources available to handle large upload or download processes. Tramline works around both issues by letting Apache and the filesystem handle both.

Pay careful attention to the installation instructions. Tramline currently needs a one-line patch in mod_python (a Python file so that's easy, thankfully). It also needs the latest version of Apache 2.0 (2.0.55), that was released last month.

Here's where to find the source:

http://codespeak.net/svn/rr/tramline/trunk

If you want to give feedback or help out, please do! We have a mailing list, here:

http://codespeak.net/mailman/listinfo/tramline-dev

Update: credit where credit's due: this is of course a project I built for Infrae, like almost all of the code I write. Tramline was conceived by myself and Jan-Wijbrand Kolman.

Comments:

Posted by Ian Bicking at Fri Nov 11 2005 19:28

Is mod_python necessary, really? It seems like a combination of signed forms and some simple scripts should be enough (maybe even just CGI scripts). For downloads it's possible you have to do some tricks with authentication, if you want to use GET and don't want to put the signature in the link, and perhaps don't want to route it through a CGI script and reimplement Apache code for efficient downloading (ranges and whatnot). mod_auth_script could potentially be used in that situation.

Posted by Peter Bengtsson at Sat Nov 12 2005 00:02

surely there's a point where actually overhead to use tramline such as 'logo.gif'. Any idea what kbyte upper-limit where tramline becomes a benefit? (...for Zope)

Posted by Graham Dumpleton at Sat Nov 12 2005 07:29

You might want to explain to the mod_python mailing list why the FilterDispatch change is required. Is it indicative of some sort of bug in mod_python that needs fixing? If it isn't a bug, but some arbitrary change just to suit your package, you might find it harder to get people to try your package out as many will not want to modify mod_python. If the change is something reasonable that enables a feature that other people might also want to use, you might be better to talk to the mod_python developers about adding an option to allow a user to disable flushing in that context.

Posted by Martijn Faassen at Sat Nov 12 2005 12:45

Graham, I contacted the mod_python people three times a
few months ago, entered a bug in their bugtracker, but to my knowledge haven't received any response yet whatsoever.
I don't know why this change is required; I think there's
a bug somewhere but that's how far I got...

I offered help in tracking it down, but nobody has taken me up on it.

Posted by Martijn Faassen at Sat Nov 12 2005 12:55

Ian, the point about tramline is performance *and*
transparency when integrating with a backend appserver such as Zope. Railroad, its big brother, takes the way of hooking into authentication and CGI scripts, but it turned out to be hard to integrate.

Tramline integrates easily into any arbitrary web form and authentication and the like just work. You don't have to GET from or POST to special places, and no signing is necessary either.

I'm sure one can get either efficiency or transparency
by just using scripts, but for both at the same time I'd need a bit more of a sketch of what you actually have in mind.

Posted by Martijn Faassen at Sat Nov 12 2005 12:58

Peter, I don't know whether there's a point where the overhead of tramline in downloading a small file actually costs you compared to Zope. I'm not sure it exists (Apache is pretty fast and tramline isn't doing much compared to Zope), but I haven't done any measurement of this. It's possible a small file that's in the object cache can actually be served up faster, as tramline will have to open the file on disk. Then again, the OS also caches stuff there...

Anyway, for downloads, if you don't set the tramline header, tramline is not in use, so if this turns out to be the case there's no problem.

For file uploads in a POST right now tramline would take over always. It might indeed be useful to have a way to turn on or off tramline explicitly.

Posted by Graham Dumpleton at Fri Nov 18 2005 05:13

Martijn, I see that you did indeed log a bug report and post a couple of messages on the mod_python mailing list about it. I plead ignorance on the basis that I was overseas on holiday around that time and only had Internet access occassionally so couldn't keep up with all my email. I have added to your bug report references to the original mailing list posts, to this page and the Tramline source code. If you do discover anything else about the issue, please keep adding any additional information to the bug report.

Posted by Martijn Faassen at Fri Nov 18 2005 16:34

Graham, thanks! It's good to hear back from mod_python people. I realize it's not always easy to respond to each and every message in time; I have my own trouble keeping up with some lists as well. I'll try to find time to do some more experimenting.