The ab1jx Radio Spectrogram page


8/21/2012 image

These are used in radio astronomy and they can be interesting in general too. I've got Hamlib driving my Icom IC-7000 so it scans the area between 30 and 60 MHz, stopping to grab an audio sample for 50 milliseconds every 250 KHz. AGC is turned off from the front panel menu on the radio and the selectivity is opened up all the way to 10 KHz. Is there anything interesting in the picture above? Probably not.

This is NOT Plug and Play. Don't expect to just download it and run it. There's a good chance it may not work with your radio at all, even if Hamlib supports it for other things. With my IC-7000 I can tune certain frequency ranges entirely electronicly. No mechanical parts are moving inside the radio.

Why is this important? Both to avoid wear and tear on the radio and for speed reasons. A "real" radio spectrograph covers its range about 10 times a second, this is on the order of 10 times a minute. It will capture slow things but miss fast ones, or at least you won't be able to pinpoint exactly when they started. Solar flares may come out OK, some of the very brief bursts from Jupiter you'll miss.

Will it work with your radio? I haven't any idea. Sit down with your radio in a quiet room and tune it from its lowest to its highest frequency. The radio should be on with the volume down all the way, and don't wear headphones. Every time you hear a relay click inside, write down the frequency. When you get done, the ranges between relay clicks are the ranges you can use. My IC-7000 has clicks at 20, 22, 30 and 60 MHz (among others), so scanning the usual Jove frequency of 20.15 MHz is out of the question, and I had to settle for 30-60 MHz. These frequencies aren't reliably documented in the usual literature but if anyone wants to send me their list I'll post it here. The IC-7000 partial list: clicks at 22, 30, antenna change at 60, then goes 60 - 199, then 400 - 470. It isn't anything you normally care about so it isn't in user manuals, maybe some service manuals. As things are now, I'm thinking about buying a another radio to use for spectrograms but without knowing what radio can scan what region I'd have to buy another IC-7000.

This doesn't use any "scanning" capability that may be built into the radio, or tie up any of the radio's memories. There's an array of frequencies built into the program which it fills by running some loops, then it feeds those frequencies to the radio. In my case the radio's connected over a 9600 baud serial line, so that's another source of slowness. For 30 - 60 MHz coverage I feed 600 frequencies to the radio, over and over again. You may want to use different sets of frequencies on occasion. I just spotted what looks like a 6 meter repeater in one of my scans (also above) so I might check that out. Scanning in 50 KHz steps is too specific for looking at man-made radio communications in general, but most natural astronomical sources are broadband in nature. You could use this to scan the 2 meter ham band every 5 KHz looking for open frequencies, or repeaters you don't know about. With repeaters you often see both the input and output frequencies. Could this work as a spectrum analyzer? Maybe, sort of.

Why do some of my images have the "window screen" effect and vertical lines? It has to do with the way I'm scanning. If I scan from 30 to 60 MHz normally it takes a long time and there's a good chance of missing stuff. So I do it in several passes that are 30 MHz long but 50 KHz apart, which lets me cover the entire region in a short time, then I roughly cover the same region again a little higher, then again, and again. Every 50 KHz point gets scanned, but only on one of 5 passes. The vertical lines have to do with the way I'm plotting. Those vary depending on the length of scan and how much data I cram into 875 pixels. For an example of "real" spectrograms see Learmonth's archive. That's where I stole the palette I'm using from. Oh, and see the actual thursday.jpg elsewhere on this page: when you put more data into the same number of pixels the windowscreen and moire go away.

             5      ^      ^
            4       |      |
           3        |      |
         1          Frequency
        5    5      
       4    4       
      3    3        
     2    2
    1    1
        5    5
       4    4
      3    3
     2    2
    1    1
        5    5
       4    4
      3    3
     2    2
    1    1    Time -->
The palette I'm using
Here's the scanning pattern, drawn out as ascii art. Start at the lower left corner and imagine that time passes as you go to the right, and frequency is upwards. On the first pass you start with the 1 in the lower left, then go to the 1 that's 5 rows above, then the 1 that's 5 rows above that. Next pass you hit all the 2s starting at the lower left and working upward, then the 3s and 4s and 5s on later passes. Then the next bunch of 1s, etc. Right now it's getting through the frequency range from 30 to 60 MHz about 10 times a minute this way. Each individual frequency gets hit every 5th pass, but remember that these are natural phenomenon so they're broadband in nature. Time resolution is probably more important than frequency resolution.

The other little trick I figured out to make the radio effectively scan faster is this: You start at a certain frequency, get x number of samples from the sound card, convert the negative values to positive, average them together and store. But, you can take your samples, then scan to the next frequency so the radio can be settling in while you're processing the last batch.

This also isn't Plug and Play because I had a bear of a time with Hamlib. It would be nice to have it control the radio's AGC and bandwidth among other things, but it took me 2 days to figure out how to just set the frequency so that's enough for now. Icom and a few other radios need a CI-V address which is fairly well buried in Hamlib. Each model number has its own default address, but the defaults weren't getting loaded so even with everything else right the computer wasn't talking to the radio.

But, I didn't spend thousands of dollars on this, in fact I didn't spend a cent. It's just one more digital mode for the radio to do. What do the scans sound like? Background noise that increases every time you hit certain frequency ranges then quiets down again. Nothing exciting in the least, but if I listen long enough I hear changes in the pattern. Once in a while I hear a beep that's part of a CW ID on a 6 meter repeater.

What do you use for an antenna? That depends on what you want to do, how serious you get about doing it, what the frequency capabilities of your radio are. It may also depend on your local noise conditions. I've started out by using a 15-17-20 meter fan dipole and alternately a 150 foot longwire. I'd like more directivity for this so I can point it up and avoid some of the terrestrial junk. Then again I'll probably lose interest in this in some number of months and want an antenna I can use for other things. Because I cover a 30 MHz range I'm looking at broadband antennas like a T2FD, log periodic, discone. I guess I'm leaning toward a T2FD because I've got most of the stuff to build and hang one in a tree, but I'd like to have a log periodic pointed up too. Except just "up" probably isn't the direction to point it in. Use what you've got and as you gain more experience and get interested in a certain thing to listen to ideas will materialize. A low band HF dipole or long wire is a good start if you've got one, otherwise string up 30 feet of wire and see what you get. In several weeks of listening, writing this software and researching antennas I still haven't decided. I've got a few more months before there's snow on the ground though.

Actually, I've sort of decided on a fan dipole as a first antenna for this. It doesn't actually fan, but it has 17 dipoles connected to a common feed point, covering the 30 - 60 MHz range every 2 MHz plus one for Jove's 20.15 MHz. I'm making it out of used network wire from the dump. The sizes of the conductors don't matter much and this is all stranded wire so it will stand being flexed unlike solid wire. The biggest question is whether this will stand up to ice storms in the winter. I've got everything cut out and soldered, now I'm threading the individual wires through bits of heat shrink tubing to neaten things up. Because the Jove dipole is longest I made it out of galvanized steel electric fence wire so it's the backbone supporting everything else. I'll hang it between two trees down low enough so they don't sway much in the wind at that point, probably about 2 meters above ground. Hams used to cut multi-band dipoles out of flat 4 and 5 conductor rotor cable and supposedly that worked. If you're curious about making one, you could come close by cutting wires for each 1/4 wave half in 2 inch increments from 4 feet to 8 feet.

About the programs

As with sc5 there are multiple programs, in this case a backend that scans the radio and a plotting program that makes JPEGs. The backend needs Hamlib, the plotting program needs the JPEG library which you probably have already for Firefox, or The Gimp or GTK. These are written so you can put the scanning backend one place and maybe call it from a cron job to run most of the time, and where you do the plotting may be on a different machine.

Even scanning away at full tilt and crunching audio samples down to datapoints every 50 ms, it doesn't take much CPU. On my 3.2 GHz P4 it barely shows in Top at all, which shows the machine between 99.6% and 100% idle. It does tie up a soundcard and a serial port or USB port to drive the radio, and of course a radio. It would probably run happily on an old laptop which will probably use less electricity than leaving your main computer on all the time, and you can transfer the files off once a day or so.

You set how long the scanning program runs for in the program itself. You'll probably have to set a lot of other things too, since this isn't plug and play. I only have my IC-7000, so I'm not trying to make this compatible with everything that's Hamlib compatible, it's more of an example. You'll need some ability to program in C, and probably as much Hamlib documentation as you can find, including hamlib source code. I didn't modify anything in Hamlib to make life easier. The CI-V default address is part of a struct which is part of another struct, something like 5 levels deep.

The scanning backend will automatically make filenames like which are the time when the recording started. And they're in UTC, which is probably not what you run your clock set to. This leads to confusing things like

-rw-r--r--  1 root  wheel   3166495 Aug 17 11:59
in an ls -lrt because the ls -lrt shows the time of the last change to the file (local time) and the filename shows when it was created (UTC). Because of the way they're named a mask like spec_2012-08-16*.tab will get all the raw files for that day.

The output files really are tab-delimited, which makes them compatible with Excel and PostgreSQL to name a couple of things. The lines inside look like:

1345188424 59300000 642 1345188424 59550000 659 1345188424 59800000 654 1345188424 30100000 627 1345188424 30350000 600 1345188424 30600000 568 1345188424 30850000 599 1345188424 31100000 611 1345188425 31350000 616 1345188425 31600000 525 1345188425 31850000 580 1345188425 32100000 563 1345188425 32350000 634

The first column is the time, in good old unix time_t format, or the number of seconds since 1/1/1970 midnight GMT. This is in GMT/UTC, done by getting the computer's time with gettimeofday() to check GMT and DST offsets, then using those offsets to modify the time_t from doing time(). You can see that there are many data points per second. I tried to get an example that had both a second change and a frequency wrap-around from 60 MHz back to 30.

The second column is frequency. Your radio may work differently, but these are Hz because I use an int64_t (or long long int) format mostly because that's what worked. There's mention in Hamlib of some radios needing BCD and some binary. So it's a little inconvenient and there's a lot of extra data to transfer over the serial port, but I've got all these extra zeroes I can't live without. These are the actual numbers fed to the radio, at least in my case. They originate in fillfreqs() which you'll need to change to match the frequency range you're going to use.

The third column is the audio level. There aren't any real units here, the numbers depend on the sound card and level settings. 6000 is a big number here on this scale, and may indicate the lower edge of some clipping or compression according to NTP (Network Time Protocol) sources. Stay well below that. If you want to use a noise generator and try to calibrate to degrees Kelvin be my guest. This is a 16 bit integer that's had negative values flipped to positive and it's the average of probably 400 samples (see SAMPPOINT) from the soundcard, which by default is sampling 8000 times per second.

Because the file format is such a simple one, you could set the run times small like an hour, then concatenate each's day's files together to do one plot covering the whole day. Something like "cat spec_2012-08-16*.tab > /tmp/" will get a day's files into one. The order doesn't even matter since each line is self-contained with its own time/date value. This example ends Fri Aug 17 03:27:05 2012 according to the time_t. The files do get big, like 10 megs for 6 hours. The image at the top of the page is from concatenating 6 files together to make a 39 meg file for 24 hours. If you need to interrupt a run, it's not a big deal: you just get a black line.

The plotting program
The plotting program, as of this writing, is called plt5. It takes the .tab file name on the command line, or it will prompt for one if you don't supply one. There aren't any options other than changes you make in the program itself and recompile. You'll probably want to replace my call.

The plotting program autoscales everything. It takes off everything after the last . in the input file name, adds .jpg and sends the plot to that file name, so my earlier example of concatenating files into would produce thursday.jpg. The program reads the input file twice, once to do a min/max on the values inside and once again to plot the actual values. There is of course a bit of history to that. I wrote a version that read in the file and tucked everything neatly away in arrays of structs, so it could do the min/max then the plot. For some reason some of the arrays of structs got clobbered (overwritten), I could add extra printfs to the program and see that I'd store one set of values and when I read them back some of them would be changed. I haven't seen that since Turbo Pascal in the 90s, this is with gcc 4.2.1. So I just lived with it, at the expense of plotting speed.

The plot is autoscaled to all 3 fields in the input file, although the frequency labels are fixed. There's a little feature I added called BIAS which causes the lowest intensity values to not be drawn as black, right now it's set at 50. The intensity values are autoscaled into the range of BIAS to 255 instead of 0 to 255. The learmonth.pal file is included, you can replace it if you want. A blackbody or hot iron color scheme might be appropriate. It was generated by running gifclrmp (part of giflib) on a gif from Learmonth.

When you look at the source of this you may notice functions called writeraw() and make_gif(). These in principle should still work, although you need giflib for make_gif(). writeraw() is the simplest way to save an array into an image so that came first. You can load a raw file into The Gimp and save in some compressed format, as long as you know the size (1000x500). Then came make_gif(), but this is 2012, and OpenBSD ports haven't replaced the version of giflib that avoids LZW compression because of the Unisys patent issue. So make_gif works, but with the giflib I have the GIFs are bigger than the raw files. I don't like PNG, and those weren't any smaller than the raw files either, so I used JPEG. I do my own graphics (no Gnuplot) by writing into a malloced one dimensional array then passing the array to make_jpeg, make_gif or writeraw. Accessing 2 dimensions is done by writing to img[(width * y) + x] for x,y. I stole the text writing and font for the labels from giflib, converting them to work without giflib.

The plot program could also be called from a cron job. If you look at the date man page it can work almost like strftime, and if you play around with that it can be quite versatile in making filename masks.

full day image

OK, here's thursday.jpg. I'm still in the process of writing stuff, so this is a collection of files made at different parts of the day. From where plt5 was I did:

cat spec_2012-08-17_*.tab > /tmp/
./plt5 /tmp/
qiv /tmp/thursday.jpg

Probably the first thing to notice is that the files were made using 2 different antennas. The 2 (actually 3 if you look close) parts at the left were made on my 150 foot longwire which goes up to about 50 feet at the high points, the rest was from my 15-17-20 meter fan dipole about 10 feet high. The black vertical separations are time gaps because I wasn't scanning continuously. I won't rule out the possililty that there's some bad coax or connection, but I can't quite put my finger on it. The picture at the top of the page was on the low antenna.

On both sides of the big vertical gap you can see 2 horizontal lines close together around 53 MHz, which are possibly the input and output frequencies of a 6 meter repeater. I can see Mt Greylock from the yard, about 20 miles away, and Mt Tom about the same distance to the south.

Somewhere around midnight UTC there's a thundershower. See the vertical group of dots that appear at all frequencies but mostly the upper ones. From this I can see I need more time labels on the X axis, that's what I mean about still changing things. Also curious are the faint horizontal lines about in the center, just below 40 MHz. Then too there's a bit of something a couple hours later going from 58 MHz down to 45 with a sharp turn-on and turn-off. ET? Probably bad coax.

But see what I mean about the data points being self-contained: there are 9 different data files here and everything just falls into place and works. Because plt5 autoscales the concatenated all the scales are compatible. Oh, and this partial is 38 megs, but if you notice the start and end times it covers about 26 hours. You can see the effect of my BIAS setting in the main image: the background is a nice blue instead of black. You see the same blue in the bottom image but you wouldn't know it's background because there's so little of it. That longwire hooked to a scope puts out about 200 millivolts peak-peak and pulls in a lot of junk.

Having the data points self-contained in each line has another potential use. Remember that where the points are in the file doesn't matter. I was testing another antenna but getting almost no background noise because I had some noise peaks that were messing up the autoscaling. So I did "sort -n -k 3 < >". Then I could edit in a text editor and work at deleting the lines with the biggest signals, which were all at the bottom of the file after the sort. I could replot, delete some more, replot again. I'm building a new antenna.

My crontab section that deals with this. I've finally got jobs starting at midnight UTC instead of haphazardly. I'm in EDT, so the offset is 4 hours (right now).

# new scheme every 4 hours starting at midnight UTC 8/20/2012 0 20 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 0 0 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 0 4 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 0 8 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 0 12 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 0 16 * * * cd /usr/programs/c/hamlib/spec1/cron ; ./sp1 and in sp1.c I've now got: stop_t = cur_t + 14380; // 20 seconds under 4 hours

After a week or two of regular use I've settled on making 4 hour data files. I can plot 1 - 6 data files per image to get 4 - 24 hours per plot. But, the images are "only" 1000 pixels wide. There are 10 data points per second. Something has to give somewhere. What happens is that newer data points scaled into certain pixels overwrite older ones. The displacement timewise isn't significant and the plots look ok. I had noticed some regularly-spaced signals if I did a partial plot during a day, but not in the full day plots. So I've settled on doing morning and afternoon plots which is a convenient compromise. I do something like cat spec_2012-09-03_0* > and spec_2012-09-03_1* >, then when I plot I get am.jpg and pm.jpg which of course need to be renamed before archiving, but I could cat into then the jpgs would have the right names automatically. Notice the difference here: PM times (24 hour clock) start with a 1, AM times don't. Except 20:00 starts with a 2. Maybe I should go to 12 hour data files.

Because this scans frequencies that are just calculated then loaded into an array, they could also be loaded from a file. If you live in an area that's heavily congested with man-made RF you might consider a companion program that repeatedly scans say every 5 KHz and logs into a database. After running that for a few days then about once a week afterward it should be possible to pull a set of quiet frequencies for this program out of the database. You need to be sure to sample both weekend and weekdays, both initially and routinely. That program sounds almost more interesting than this, so I might start on something that does it soon. The database could be as fancy as an SQL database or something more like histogram bins. I don't live in an urban area, but I'm on top of a mountain, so it's almost the same problem. I can't listen to the local fire departments without hearing some bunch of cops 50 miles away in a different state. Same frequency, different repeaters.

Another thing to consider might be to look over articles that have been written about building frequency analyzers cheaply over recent years. With an increasing number of varactor/varicap tuners being available for both radio and TV, all it should take is something like a 555 timer to generate a frequency control voltage ramp waveform. As long as it produces some some of a sync pulse once per cycle and you know its characteristics you can figure out where it is in the cycle. So you feed audio into an old laptop, hook the sync pulse into a serial or parallel handshaking line (or even game port), write some fairly simple software and you've got your spectrometer. And you get your radio back. Add an MMIC from an old satellite dish for more gain.

If you need a CI-V adapter, take a look at G3VGR's schematic. I haven't tried it because I bought one before I read about this. He built it in a DB-9 shell. It may have trouble in some situations because it powers itself from the RS-232 port: try taking power from RTS instead of DTR if you have trouble.

If anybody's got a good working "Hot Iron" or "black body"palette file I'd love a copy to include.

For compressing your data files (if you want to keep them), I took one example file and tried gzip, bzip2, and xz on it. Now I see why FreeBSD uses xz:

-rw-r--r--   1 root  wheel  6791043 Aug 20 10:43
-rw-r--r--   1 root  wheel  1096016 Aug 20 10:43
-rw-r--r--   1 root  wheel  1390332 Aug 20 10:43
-rw-r--r--   1 root  wheel   523684 Aug 20 10:43

This is for OpenBSD but might work under NetBSD. I don't use big multi-part version numbers much, but I'll change the date here when I update. 8/22/2012: spectra.tar.gz.

And one more, just because I like staring at them (sometimes it doesn't load over a modem connection):

Sunday image part 1 Sunday image part 2

AB1JX / toys/