Alden Bates' Weblog

Today's lesson: get the right tool for the job and the job will become much easier.

A long while ago now I opted to mirror Paul Harman's Web Guide to Doctor Who in order to avoid having to maintain my own directory of Doctor Who links. This didn't quite work for two reasons:

1. People still emailed me to ask me to add a link to their web site, despite the word "Mirror" in large letters and a link to the page for adding sites to the Web Guide, and
2. I ended up volunteering to assist Paul in maintaining the Web Guide.

The Web Guide has 21 sections and almost 700 sites listed in it. It's pretty kick ass and I don't think there's another comparable Doctor Who directory in existence. Most other sites mirroring the Guide would simply use a copy of the HTML-Version Paul sends out on a mailing list every update, but I had to be different and split the guide into sections, thus making my job slightly more difficult.

My first technique for doing the update was as follows:

1. Take the HTML web guide and...
2. Replace all the links to graphics to point to the same graphics on my server, avoiding the sticky issue of stealing bandwidth.
3. Copy and paste each section separately into the individual pages.
4. Process the pages into real HTML.
5. Upload all the pages.

But this took too long, and I decided to write a program to make things easier. The technique now became:

1. Copy and paste the HTML web guide into a file with my site format.
2. Feed the resulting page into my Windows-based program, which produces some CSV files to be used to generate the section pages dynamically on the site.
3. Copy and paste the compile date into the mirror index.html.
4. Copy and paste the changes into the changes.html page.
5. Process the pages into real HTML.
6. Upload the three pages and CSV files.

This resulted in a process which was not actually a hell of a lot faster than the first process. Tonight I wrote a completely new perl-based program to turn the process into the following:

1. Upload the HTML web guide.
2. Run the script to process it into all the separate files.

Leaving me more time to play Unreal Tournament 2004, swim the Amazon, invent cold fusion, achieve world peace, etc. Teh skillz.