Posts Tagged n3qo

While troubleshooting the filtering functions of my DXspider installation, I found there was very little documentation on how to get filtering to actually work. For the uninitiated, browse to the bottom of this post if you only require an updated version of usdbraw.gz from 12/23/09.

First I started with G1TLH’s download documentation. It references a file named usdbraw.gz which was unfortunately unavailable at the time. After a brief google search, I found the file, expecting it to only contain states and cities, but instead it IS the US FCC callsign database. The newest one I found was from 2004. Importing this database would make state filtering spuratic at best since so many new callsigns have been issued in the US of the course of the last 5 years and DXspider would only filter callsigns it knew about.

The FCC provides a daily updated database archive on their site available for download. Their database structure by itself is not compatible with DXspider and requires massaging of the data to ready it for import.

A huge find in this process was G1TLH’s USDB generation script. Dirk’s script takes the compressed FCC archive and converts it into DXspider format which happens to be output as usdbraw.gz.

Taken that process, then reverting back to the download documentation, then importing the database renders a newly up-to-date reference to apply state filters against.

This is great, but new callsigns are generated daily and I wanted to keep DXspider as up-to-date as possible with little to no administrative intervention from me. Introduce a little scripting and the use of cron.

My shell scripting skills a disasterous at best but it does function as advertised. I named my script usdb-update.sh and wrote it in /spider/local_cmds. It seemed to be the most obvious place for it at the time. Now let’s step through the script:

#!/bin/sh
SPIDERDATA=/spider/data
SPIDERPERL=/spider/perl

wget -P $SPIDERDATA http://wireless.fcc.gov/uls/data/complete/l_amat.zip

$SPIDERPERL/gen_usdb_data.pl $SPIDERDATA/l_amat.zip

$SPIDERPERL/create_usdb.pl $SPIDERDATA/usdbraw.gz

rm $SPIDERDATA/l_amat.zip
rm $SPIDERDATA/usdbraw.gz

At this point if all has gone swimmingly you have the US callsign database waiting for you in DXspider. We now have to load it to make it available to our cluster users.

Telnet to your cluster, login as your privileged callsign and load the database:
load/usdb

30 5 * * * run_cmd('load/usdb')
So my cron entry above executes the command load/usdb within the cluster environment every day at 5:30AM GMT.

Since the previous cron entry only works within the cluster environment, I need to execute the data collection process external to DXspider.

[sysop@dxspider1 ~]$ crontab -e
Append the following entry into the sysop user’s crontab
00 5 * * * /spider/local_cmd/usdb-update.sh

I’ve decided to run the usdb-update.sh we detailed earlier at 5:00AM GMT. In practice I’ve found the data collection and processing only took 5 minutes so to execute the actual load of the database in DXspider 30 minutes later was plenty of time. This allows for internet or server slowness. Feel free to set these times however it suits your needs.

Below are the files I’ve referenced in the above documentation:
G1TLH’s gen_usdb_data Perl script and my data collection script

usdbraw.gz created 12/23/09