Posts Tagged gen_usdb_data.pl

DXspider US callsign database

Posted by jeff in DX Spider on December 23rd, 2009

While troubleshooting the filtering functions of my DXspider installation, I found there was very little documentation on how to get filtering to actually work. For the uninitiated, browse to the bottom of this post if you only require an updated version of usdbraw.gz from 12/23/09.

First I started with G1TLH’s download documentation. It references a file named usdbraw.gz which was unfortunately unavailable at the time. After a brief google search, I found the file, expecting it to only contain states and cities, but instead it IS the US FCC callsign database. The newest one I found was from 2004. Importing this database would make state filtering spuratic at best since so many new callsigns have been issued in the US of the course of the last 5 years and DXspider would only filter callsigns it knew about.

The FCC provides a daily updated database archive on their site available for download. Their database structure by itself is not compatible with DXspider and requires massaging of the data to ready it for import.

A huge find in this process was G1TLH’s USDB generation script. Dirk’s script takes the compressed FCC archive and converts it into DXspider format which happens to be output as usdbraw.gz.

Taken that process, then reverting back to the download documentation, then importing the database renders a newly up-to-date reference to apply state filters against.

This is great, but new callsigns are generated daily and I wanted to keep DXspider as up-to-date as possible with little to no administrative intervention from me. Introduce a little scripting and the use of cron.

My shell scripting skills a disasterous at best but it does function as advertised. I named my script usdb-update.sh and wrote it in /spider/local_cmds. It seemed to be the most obvious place for it at the time. Now let’s step through the script:

  • We start with the usual shell scripting stuff. Let’s set variables to our paths. Just in case one day we want to move all this stuff to a new location, it makes life a little easier.

    • #!/bin/sh
    SPIDERDATA=/spider/data
    SPIDERPERL=/spider/perl

  • Let’s download the the FCC database via wget and store it in /spider/data. Don’t mind formating below…

    • wget -P $SPIDERDATA http://wireless.fcc.gov/uls/data/complete/l_amat.zip

  • I placed Dirk’s script in /spider/perl will all the other scripts. Let’s call that script against our newly downloaded FCC database and convert it into something DXspider can understand. This actually creates the usdbraw.gz file you’ve probably been scavenging the internet for. It stores it in /spider/data

    • $SPIDERPERL/gen_usdb_data.pl $SPIDERDATA/l_amat.zip

  • Now take that output and create the USDB from it.

    • $SPIDERPERL/create_usdb.pl $SPIDERDATA/usdbraw.gz

  • Now cleanup our mess. Disk space is valuable.

    • rm $SPIDERDATA/l_amat.zip
    rm $SPIDERDATA/usdbraw.gz

    At this point if all has gone swimmingly you have the US callsign database waiting for you in DXspider. We now have to load it to make it available to our cluster users.

    Telnet to your cluster, login as your privileged callsign and load the database:
    load/usdb

  • OK we’re functional, but let’s automate it starting in reverse. Edit your /spider/local_cmd/crontab and append the following entry:

    • 30 5 * * * run_cmd('load/usdb')
    So my cron entry above executes the command load/usdb within the cluster environment every day at 5:30AM GMT.

    Since the previous cron entry only works within the cluster environment, I need to execute the data collection process external to DXspider.

  • Log into the shell of your DXspider server as the sysop user and create your crontab (typing crontab -e).

    • [sysop@dxspider1 ~]$ crontab -e
    Append the following entry into the sysop user’s crontab
    00 5 * * * /spider/local_cmd/usdb-update.sh

  • Save and close your cron.

    • I’ve decided to run the usdb-update.sh we detailed earlier at 5:00AM GMT. In practice I’ve found the data collection and processing only took 5 minutes so to execute the actual load of the database in DXspider 30 minutes later was plenty of time. This allows for internet or server slowness. Feel free to set these times however it suits your needs.

    Below are the files I’ve referenced in the above documentation:
    G1TLH’s gen_usdb_data Perl script and my data collection script

    usdbraw.gz created 12/23/09

    , , , , , , , ,

    No Comments