WHAT IS QRSS VD?
QRSS VD is a free, open-source, cross-platform QRSS spectrograph and spectrogram analyzer written entirely in Python (distributed as source .py’s or compiled .exe’s) by Scott Harden, AJ4VD. I wrote this software because I wanted to build a 30m QRSS grabber for weak signal extraction experimentation but was dissatisfied by the limitations placed on my creativity by currently-available spectrograph software commonly used for QRSS. While I greatly admire and respect the makers of SpectrumLab and Argo, I simply wanted the flexibility of an open source option.
Why shouldn’t I be able to create spectrograms spanning thousands of hertz (several kilohertz) instead of just 100 hertz? Why shouldn’t I be able to automatically stitch together hundreds spectrogram images so I can browse through them as a single, giant image thousands of pixels in size in each directions? Why shouldn’t I be able to run the spectrograph on Linux, Mac, or Windows, with or without a gui? While questions like these are frustrating to many, I saw these challenges as an opportunity to write some profound code. It was from my deepest frustration that QRSS VD was born!
What does QRSS VD do?
QRSS VD visualizes sounds in such a way that you can visually see unique properties of the audio that you could never notice by listening to it directly. As far as its application to amateur (ham) radio goes, QRSS VD allows you to visually distinguish between frequencies (along a vertical axis) with extremely high resolution. Additionally, by averaging the frequency readings several-fold with respect to time, extremely weak signals (normally far below the noise level) begin to emerge. By sending data extremely slowly (the Morse code Q-code for “slow down” is QRS, and QRSS implies incredibly slow transmission of information), low power transmitters (perhaps running only a few milliwatts) can be detected thousands of miles away!
This is an image created by QRSS VD. On the vertical axis we have frequency (with respect to my radio’s frequency at the time, 10.140mHz), and on the horizontal axis we have time (about 2 and a half minutes worth). The signals are from QRSS transmitters around the world. Whether or not we see certain transmitters depends on many things, especially atmospheric conditions which affect propagation. In the image above we see the “flying W” from W1BW in MA, USA some letters below the W which are from IW4DXW in Italy (putting out a quarter of a watt!), and the dashed line below that shifts only a few Hz is AA5CK running a 122mW transmitter in OK, USA, and the two lines which shift up and down by ~10Hz are KC7VHS and WA5DJJ in NM, USA running ~250 milliwatts. These signals are interpreted by Morse code, and take a long time to come across (about 1 letter per minute). Here’s an example:
However, the image above is only a small, teeny, tiny part of the radio spectrum. It goes on far in all directions (along both the frequency and time axis). Although QRSS VD did create the images above, here’s the same audio processed by QRSS VD differently, allowing the creation of a much larger image. The blue square represents the region where the green image above (with the flying W) was taken. The quality of this image was greatly reduced to make it web-friendly. It’s over 8,000 pixels wide and over 1,000 pixels high!
Note that as large as that image is, it could have been much larger. QRSS VD has the capability (and yes, I’ve tried it) to generate spectrograms over 8,000 pixels high (spanning ~4kHz) and many thousands of pixels wide. I intentionally chose a smaller region to display for the website, and that’s okay! The beauty of QRSS VD is that it lets you make these decisions later, without having to destroy any data. While QRSS VD is recording you can scroll up, down, left, and right without messing-up any of the images.
How does it to everything? It listens to you! You tell it how much audio to record, how much to analyze, what region of the band (or the whole thing!), low pass filters, high pass filters, Fourier transformation methods, frequency-domain smoothing methods, time-domain smoothing methods, colors, intensity adjustments, etc. (don’t worry, they’re all preset for common QRSS use!). While you analyze, you can adjust most of these settings in real time. The GUI is seen on the right.
While it runs, QRSS VD generates and saves spectrograms in ~10 minute chunks. Note that you can make it save each chunk as the same filename (good for a web server QRSS grabber), or you can have it save them all consecutively with timestamps (seconds since epoch) in the filename. The result is a folder full of BMP files! (pictured on the left) QRSS VD viewer can then open a folder filled with these BMP files, assemble them as a giant map, and let you scroll around in all directions (somewhat like Google maps!, pictured below).
If you see a cool region you want to extract, Just click on the top left side of the region, then the bottom right side. The QRSS VD Viewer automatically makes a new, cropped image, adds scale bars, and opens it for you to save! How convenient is that?
- IQ4DJ (straight CW)
- G6AVK (Triangles, up is a dot down is a dash)
- G3ZJO (mountains)
- AA5CK (3 Hz frequency shift)
- ??? (10 Hz frequency shift, too weak to copy)
- WA5DJJ(10 Hz frequency shift)
Screen shots of QRSS VD in action!
At the UF ARC station QRSS VD is often on a Flex 5000a software-defined radio. Here you can see two instances of QRSS VD Spectrograph running, one of them set to low resolution (bitrate power set to 2^10) for fast-update larger-spectrum viewing while the other is configured to record slow, high quality images of the small QRSS region (note the differences in scale of the vertical axis of each instance).
I use Argo as the “golden standard” to assess the quality of the spectrograms produced by QRSS VD. Here you can see the two being run simultaneously. The output on both is similar (score!) but with QRSS VD, you can scroll up and down without affecting the incoming image. See the cool straight-line morse code above the rest of the QRSS, even above the letters? With Argo, you’d never even know that was there, and even if you tried you couldn’t record it and the text and the lines at the same time. I love Argo, and it’s great at what it’s designed to do, but it just doesn’t meet all of my needs for a QRSS grabber.
Here you can see me during the early development of QRSS VD. I’ve got Argo running to compare its output to mine, and you can see the code (being edited in Notepad++ for Windows) being edited on the right. As a personal note, at home I use exclusively Linux, but a lot of the work for this project was done at the University of Florida Gator Amateur Radio Club Sataion, W4DFU and all of their PCs run Windows. I can’t complain – it just further solidifies the need for cross-platform software.
Current Version: 1.05 (Mar 13, 2010)
Setup: qrss_vd_1.05_setup.exe (12.8 MB, Windows only)
Source: qrss_vd_1.05.tar.gz (21 Kb, any OS)
New features in this version: sound card selection, non-crashing image viewer, web-friendly QRSS grabber, and more…
Audio: qrss.mp3 is 1hr of QRSS (3.4 Mb centered at 10.140 MHz) if you don’t have a radio
The source package is a collection of two python scripts. They should work on virtually any operating system. To run them you need to install Python 2.6, Matplotlib for Python 2.6, PyAudio for Python 2.6, the Python Imaging Library (PIL) for Python 2.6, Numpy for Python 2.6, and Scipy for Python 2.6.
UPDATE: I found an easy way to get PyAudio on my current version of ubuntu. According to this page, I just added the lines
deb http://ppa.launchpad.net/gezakovacs/pyaudio/ubuntu lucid mainto /etc/apt/sources.list then ran
deb-src http://ppa.launchpad.net/gezakovacs/pyaudio/ubuntu lucid main
sudo apt-get updatethen installed it with
sudo apt-get install python-pyaudioand it works!
I run Ubuntu 9.10 at home and QRSS VD runs fine for me, but Michael (DL4MGM) reported that he got it running successfully on Ubuntu 8.10 and 9.10 as well. He made a list of the packages required for this software. To install all of the required packages on an Ubuntu system, run the following command:
sudo apt-get install python-matplotlib python-imaging python-imaging-tk python-numpy python-scipy python-pyaudio. Next, install PyAudio from the PyAudio website using their instructions (add the repository to the Ubuntu sources file and apt-get install it). Once installed, launch the spectrograph by navigating to the program folder in the console and typing
Fire-up some QRSS audio
The most commonly-used QRSS frequency is 10.140 MHz. Set your radio to CW mode and listen-in on that frequency. If you don’t have a radio or the band isn’t very active, you can download a 1 hr audio file (mp3 format) of QRSS audio from the QRSS VD downloads section.
Create some spectrograms
Launch QRSS VD Spectrograph (through the start menu). Marvel at the dynamically-generated splash screen which changes every time you load the program. Next, right-click on the body of the program and select ” >> RESUME” to begin generating data. The program will slowly create a large green image which will eventually fill the entire window. If you don’t see what you think you should be seeing, try using different sound card selection settings by selecting “Set Sound Card” from the right-click menu.
If the image is too dark or light, you may have your sound card settings wrong. Double-check to make sure your default recording device is properly set. Then you can adjust the intensity of the image by adjusting the gain. Right-click the spectrogram, select “general settings”, and change the “Auto Gain” value up or down (lower makes it brighter, higher makes it darker, it’s weird I know – I like 7) and press SET afterwords.
Select spectrograms to view
Once several minutes have gone by, QRSS VD should have begun generating images. Right-click on the spectrogram and select “Program Statistics” to see the number images captures on the top line. Once you have several images captured, you can view them by opening the “QRSS VD Viewer” (from the start menu). When it opens, you’ll see a screen similar to this one:
Explore large spectrograms with QRSS VD Viewer
Create a snapshot of a region of interest
Let’s find an interesting region that we want to isolate. The cursor should be an “upper-left” cursor, so click on the upper-left of the interesting area. The cursor will change to a “lower-right” cursor, allowing you to click on the lower-right of the interesting area. When you do this, QRSS VD will automatically make a copy of this region, add scale bars, and open it in your default image viewing program so you can save it in the format you desire. Note that the right-click options menu in the QRSS VD Spectrogram Viewer allows you to set a custom message to be displayed on the generated image.
Since it’s so easy to do, I encourage you to E-mail QRSS beacon owners nice, clean, neat, well-labled QRSS captures – they’re much more informative than numerical signal reports!
Explore QRSS VD!
There are a lot of options in the right-click menus. For a start, you can expand the vertical view of QRSS VD far beyond what the default region is using the bandpass options. You can adjust the color of the spectrum, or the intensity using many different means. The documentation further down the page may be helpful.
- Adjust brightness and contrast by right-clicking, selecting “general settings”, entering a new value for “auto gain”, and clicking “SET”. Changes are reflected immediately on new, incoming data.
- After capturing a long series of images, keep them organized by moving them into a subfolder you create within ./output/ and move the “scale_info.txt” file too! That scale_info.txt file is required to recreate scalebars and must be kept with those images or it will be over-written when you begin saving new images with different settings!
QRSS VD Spectrograph: Right-Click Menu
Right-click on the spectrograph to display the following menu…
Return – does nothing, exits menu immediately (often clicked accidentally)
Launch Viewer – executes “QRSS_VD_Viewer.exe” for quick access
|| Pause – prevents the recorder from launching new threads
>> Resume – allows the recorder to poll the sound card
() Capture Now – takes an image capture of the spectrograph immediately
Set Sound Card – displays all sound input devices on your system and allows you to switch between them in real time, without restarting your program.
General Settings – math configuration (details)
Autosave Options – customize how QRSS VD saves images automatically (details)
Set Base Frequency – allows you to tell QRSS VD what frequency your radio is tuned to and adjusts the scale bar accordingly. You can use this option to fine-tune the calibration as well.
Set as Default – saves the current settings in a file so when you launch the program again they’re restored
Reload from Default – loads settings from the saved file (similar to closing and re-opening the program)
Factory Reset – deletes the default configuration file to force QRSS VD to create a new one from scratch
Program Statistics – Displays long-running information (details)
Development Console – Displays immediate information, TONS if debug mode is enabled (details)
Open Program Folder – Launches the folder where QRSS VD is installed
Open Output Folder – Launches the folder where QRSS VD saves images
The rest – are just links to various portions of this website.
Bitrate – This relates to the vertical resolution of the image, a.k.a. how good QRSS VD is at discerning between frequencies. The lower the number is, the less information QRSS VD uses to make the spectrograph, so it makes it faster with less CPU load but at a lower quality. A high value produces EXTREMELY (virtually uselessly) high resolution images, with each Hz being several pixels, takes a HUGE amount of CPU to compute, and crawls along at a few horizontal pixels a minute. I recommend capturing QRSS at 13 (2^13=8,192 samples), and casual browsing at 10 (2^10=1,024 samples). I often run both simultaneously, allowing me to view the entire band at low resolution, and focus on an important region at high resolution.
Window width (px) – the width of the spectrograph window. A smaller window takes slightly less CPU and produces narrow image captures.
band pass low/high – allows QRSS VD to concentrate its efforts on only analyzing a specified region of frequencies. If you’re doing exclusively QRSS work and only looking in a 200 hz window, setting this to a narrow pass band will improve CPU usage and dramatically reduce the size of the output files. Conversely, large images are great for monitoring entire chunks of radio bands. Running a narrow filter risks missing something exciting on the air!
Colormap – changes the color of the spectrograph. Currently only 3 options are available, but they’re relatively easy to generate yourself for implementation at the python code level.
Gain type – Automatic is the most intelligent to use in most cases.
- – Automatic Gain – The brightest pixel your value times the mode (most common value) of the data. Lowering the automatic gain value increases brightness of the image. Because this is based on the mode (~average values) of the data, changes in the audio level, noise level, or band are automatically compensated for to produce images of uniform intensity (and sensitivity) over time. The downside is that extremely strong signals in the passband may cause a darkening of the weak signals in the image.
- – Simple Gain – The maximum pixel intensity is set as the maximum value of the FFT data. Because the value of the maximum FFT data point varies greatly, the images created in this mode are very fluttery. It’s virtually useless.
- – Manual Gain – In cases of scientific analysis where you want to measure and compare signal strength between two time points, you want to manually set the gain to keep everything consistent. The value is simply the value of the FFT data which will be converted to maximum pixel intensity. The higher the number, the darker the image. ~1700 is a good value.
Transformation – FFT () should be used, but for completeness I provided an earlier version of this method “Real FFT” which discards the imaginary component of the transformation. That’s like throwing the baby out with the bath water. If we hold on to the imaginary component, we get twice as much data to work with, and produce far superior spectrograms.
Time-domain smoothing – Probably should be set to 0, but if you want to further reduce noise by averaging FFT values in the dime domain, set this to the number of FFTs (vertical columns of pixels) to save in memory. Smoothing is performed at the FFT level BEFORE normalization, not at the pixel-intensity level after normalization, so it is very different than just loading an image and applying a “wind-blow” type of effect, even though the output looks similar. The following photo demonstrates the same audio (music!) being processed without and with smoothing (value: 15)
NOTES ON GAIN: You can mathematically tell how your gain is affecting the values of the FFT by viewing the before and after normalization values in the statistics window in real time.
Image Capture – If it’s automatic, it captures images and saves them in ./output/ automatically. If set to manual, images are only saved with you manually right-click and select “capture now”.
File Type – Currently supports BMP, PNG, and JPG output. JPG is intentionally compressed at low quality to minimize file size (I figured it’d be for web server use). If space is an issue, consider PNG format. It’s lossless (like BMP) but usually smaller.
Filename prefix – This string is appended to the beginning of your saved image files. IT MUST CONTAIN AN UNDERSCORE IF IT IS TO BE READ BY THE QRSS VD VIEWER PROGRAM! That might change in the future, but for now it’s true. You’ve been warned!!
Resize Output – If enabled, the image (which could be HUGE) will be resized to dimensions of your choice. The algorithm used for downscaling is bilinear (not nearest neighbor) so they’re very smooth looking.
CLICK TO APPLY ALL – must be pressed for any of these setting changes to take effect
The stats window was designed to give you a peek at information which you may find useful. It shows how many images have been saved and some other info. Of particular interest is the values of the FFT before and after normalization. Remember how our gain controls work? This is how you can tell numerically where you stand. If your average value is 1330, it makes sense that 1700 might be your maximum value for your manual gain setting.
The developer console is only for developers. Are you a developer? Didn’t think so. It’s more useful if you’re working on the code at the Python script level. Its default state is to show realtime configuration information. If you manually open-up the “qrss_vd.cfg” file and change
'logAll': False, to
'logAll': True, (caps are important), it displays min/max/avg information for EVERY row of pixels in realtime (too processor intensive to leave on by default). It’s set to only display the last 5 events so it isn’t very functional in that way.
QRSS VD Viewer Right-Click Menu
set custom message – CURRENTLY BROKEN! D’OH!!! It’s supposed to let you add a message (like your own callsign) to the saved images. It’s on my todo list to fix it.
save ENTIRE image – does what it says, but adds a special minimally-intrusive timestamp to the top of the image periodically. Wow, this would be a great thing to display live on a website… [ponders]
Important Files and Folders
the following files are a few of the gajillion that come with QRSSVD…
C:\Program Files (x86)\QRSS VD\ is the base directory on Windows. Within this directory is a bunch of compiled junk which is a (likely vestigial) remnant of py2exe‘s glorious but awkward attempt to convert Python scripts into machine code for mass distribution.
QRSS_VD_spectrograph.exe – the main spectrograph program
QRSS_VD_spectrograph.exe.log – when it crashes, it logs why in this file
QRSS_VD_viewer.exe – the spectrogram browser program
QRSS_VD_viewer.exe.log – when it crashes, it logs why in this file
TEMP.png – the image the viewer creates when you crop a large image. This is what’s opened in your default image viewer.
./output/ – This is the folder that contains everything QRSS VD generates
./output/capt_1268456622.42.bmp – Filenames of captured images usually contain a prefix, underscore, timestamp, and extension. (details)
./output/scale_info.txt – This saves the scale OF YOUR MOST RECENT IMAGE CAPTURE ONLY. This is required to be in the same folder as collections of images which will be assembled by the QRSS VD Viewer. It contains information on how to redraw the vertical and horizontal axis labels.
./output/qrss_vd.cfg – This is the default configuration file, saved when you right-click and select “Set as Default”. The contents should not need to be edited directly, as most options either get overwritten automatically or can be set through the menus within the program. If your program crashes instantly upon loading, try deleting this file and QRSS VD will recreate it with factory defaults.
QRSS VD IS CRASHING!!!
Oh no! Don’t panic, we’ve got options. If it crashes often (or every time you try to start it), try deleting the ./output/qrss_vd.cfg file and starting it again. Sometimes changing settings outside of the capabilities of the program will cause it to crash. I intentionally didn’t limit a lot of the values. If you try to set the sample bitrate to 2^5423894, I’ll bet it would let you try!
You might want to check out I2NDT’s awesome collection of QRSS grabs all in one place (many of which use other software)
QRSS VD Grabber.exe is a bare-bones preview of what’s to come (a full-featured GUI-based app with FTP uploading capabilities), but for now it is what it is. Below is a screen-shot of me using QRSS VD Grabber.exe and in it you can see the highlighted image (grabber.jpg), which is an automatically-updated JPEG with scale bars ready for web-upload.
To do this, I simply ran the spectrograph monitor (from start menu) and get it recording how you like, then I run the QRSS VD Grabber (from start menu). Boom! The QRSS VD Grabber will watch your output folder for new BMPs, and every time it sees one it will create a new JPEG (web-friendly) of the latest few (user-determinable) captures, add scale bars, and save it as a full size JPEG and a thumbnail JPEG automatically.
- Even though I refer to “bitrate”, I really mean “sample size”. Bitrate is defaulted to 5500 hz. You can change this if you’re adventurous by manually editing the qrss_vd.cfg file and changing
'sampleRate': 5500to something psychotic like
'sampleRate': 48100. Theoretically the software is only limited by your soudcard’s ability to record that fast, and your CPU’s ability to keep up with the FFT. Increasing this can help increase maximum displayable frequency, and should be played with if you have a wide-range software defined radio (SDR) with a fancy wide-receive highspeed soundcard.
- Colormaps can be edited by opening-up the Python sources. They’re on the first few lines of the spectrograph code, and they’re just arrays (256 in length, representing pixel intensities from 0 to 255) of RGB tuples. How easy is that? So easy to expand! I know!
- If you’re a coder, let me have you look a look at the normalize() function. Heck, I’ll post it here. It represents such potential! After the FFT values are calculated for a segment of sound, the array (ffty) is sent to normalize() so it can figure out what pixel intensity/values to assign to each y position in the single-pixel-wide band. This represents an AWESOME place for improvement. It’s intentionally simple and modular. What about using special logarithmic, sinusoidal, sigmoidal, or creative combinations of these and other functions to help emphasize weak signals while still displaying, but not putting too much emphasis on the strong ones. At this level we could add moving window de-trending, smoothing, or other linear data manipulation techniques to help weak signals jump out of the noise! How cool would it be to be able to begin to see signals pop out of regions which look blank right now? Think about it – that data could be there – you just need to think of creative ways to extract it! Here’s the normalize() function. Work with it!
def normalize(ffty): #COULD USE TECHNIQUE WORK TO IMPROVE WEAKSIGNALS!!!! """Eats crazy large values and poops-out values 0 to 255.""" """This represents an area which could use GUI improvement. This function does everything to the FFT from the moment its created until it's given 8-bit pixel values. By adjusting how this works you can either hide accentuate weak signals. experiment!!!""" global fftmins,fftmaxs #fft=scipy.log(ffty+10)#log magnifies small peaks (weak signals) fftmin=ffty.min() fftmin=0 if config["gainType"]=="auto": fftmax=scipy.median(ffty) fftmax=(fftmax-fftmin)*config["gain"]+fftmin ffty=255.0*(ffty-fftmin)/(fftmax-fftmin) return ffty if config["gainType"]=="simple": fftmax=ffty.max() ffty=255.0*(ffty-fftmin)/(fftmax-fftmin) return ffty if config["gainType"]=="manual": fftmax=config["manGain"] ffty=255.0*(ffty-fftmin)/(fftmax-fftmin) return ffty
- Full source code has been posted in the download section. I encourage you to take a look at the sources.
- Please note that Python wasn’t necessarily designed for or intended to be run as a GUI, and ESPECIALLY not as compiled .exe’s for Windows-only distribution. If you have the opportunity, I encourage you to run this script from the sources. It’s cool, it’s fun, if it crashes you get to see where and why, and you get to change up the code to make it fit your needs.
- If you’re interested in getting started learning Python and you’re already tech-savvy, I recommend you visit DiveIntoPython.org. It’s an actual book that’s been made available online. It’s great! So clear, and it dives right in (it’s meant for programmers to learn a new language).
- If you have feature requests, bug reports, suggestions, comments, or captured some really cool stuff with QRSS VD I’d love to hear about it! I encourage you to post these things on the QRSS VD Google Group.
73! Scott Harden, AJ4VD
Comments should be directed toward
the QRSS VD Google Group.
/* QRSS VD Changelog Written by Scott Harden, AJ4VD website: http://www.SWHarden.com/qrssvd #################### ### VERSION 1.05 ### #################### Primary reasons for update: - added options (i.e., sound card selection) - prevented crashes - added qrss grabber program Detailed changes: Added a "launch viewer" button to the right-click menu made the "set message" option work on the viewer right-click menu added sound card detection and selection options added time domain smoothing capabilities prevented crashing from entering too-high bandpass filter added open program / output buttons to right-click menu added google group link to right-click menu spell correction for "transformation" added loading confirmation screen included new "qrss grabber" program to produce web-friendly graphics #################### ### VERSION 1.04 ### #################### Primary reasons for update: - first public release Notes: compiled at 5pm Friday afternoon (March 12, 2010), the last weekday of Spring break. Published online several hours later. */