Impeeded Progress

Spring break is over and dental school classes have resumed. An unexpected side-effect of spending all of spring break working so hard on my QRSS VD software project is that when I resumed dental school, I actually feel more relaxed. It’s like spring break was my period of intense work, and the rest of dental school is my “break”. It makes me a little disappointed though – I feel like dental school isn’t pushing me very hard, and I don’t feel like I’m learning or growing like I could be. With the estimated cost of tuition totaling ~$200,000 for four years of dental school, I’d have expected classes to feel a little less trite. Anyhow, it is what it is, and I’ll make the best of it.

I’m spending the weekend polishing-up code and reading circuit diagrams. I have a few projects on my plate (involving QRSS in some way or another), and I was moved by a song I heard enough to post it here. It’s a variation on Pachelbel’s Canon in D, it starts out slow, but half way through it gets so good. I’m trying to figure out if the emotional response to this song comes from the music, or weirdly misappropriated longings of my teenage self who watches a movie with this version of the song in it as a primary theme. Regardless, I’ll post it because I enjoy the compilation and because I’m impressed with how well this person plays it. Back to work. Cheers!


     

Lubnaan Shaikhto's Project

I’ve been exchanging emails with Lubnaan Shaikhto for a little while regarding a project involving a realtime spectrograph controlled by a microcontroller (ATTiny2313). He had a few questions, I had a few recommendations, and several weeks went by and I heard nothing… until today when I saw this awesome youtube video of his incredible success! The guy built his own USB AVR programmer just for this project too. Double score! Way to go Lubnaan!

LedDisp_schematics
100_3478 copy
100_3479 copy


     

How to Destroy a Radio Operator

After cwQslspriding myself on my ingenuity a few weeks ago for documenting my homemade stealth indoor apartment antenna for 40m and 20m, it seems that the green movement has contrived a plan to cripple my successes. So far I’ve made a few dozen contacts in Morse code with my humble little setup (~20 watts of power, direct conversion receiver, indoor homemade antenna). The photo shows some QSL cards I’ve gotten. Anyhow, my apartment manager decided that my apartment needed to have solar panels added to it. It’s too early to tell for sure, but spinning the dial a few times and hearing *nothing* makes me think that it dramatically impacted my reception (and likely transmission) in a dramatic way.

A few days ago…
before

yesterday…
workers

today
after

R.I.P. AJ4VD station… [sigh]


     

Overseas QRP Transmission

While working on the spectrograph software I’m so psychotic about completing this spring break, I happened to capture a cool signal from Italy. IW4DXW was sending some cool signals that I captured around 10.140 MHz. See how his callsign is written “visually” on the spectrogram? I thought I’d post it because it’s an encouraging sign that my software is going in the right direction. Also note the numerous QRSS FSK signals around it! So cool.
iw4dxw

UPDATE! The guy emailed me his station information.

Hi Scott! Thank you so much for nice report. This is a short description of my homebrew beacon. The audio source is a normal MP3/WAV player. The file (.WAV) of my hell message (8×8 char) is generated using Chirphel (program by DF6NM) with a fmax~=800Hz and BW=5Hz, and processed with Csound to obtain the phase relation R = L + 90° between channels (hilbert function). A 10.140MHz oven xtal oscillator (adjusted for +880Hz) is a PLL reference used to obtain a x4 frequency. This clock with the 2 audio channels are applied to a “Softrock style” SSB phasing modulator (double H-mixer with 74HC4053 ic).

The LSB output signal (10.140,080MHz @ ~ -3dBm) is amplified by a 2N2222 (driver stage) and a 2SC2314 (linear PA stage ~1W P.E.P. max, but 100 – 200 mW of average power). I’m using a 30m dipole (inverted vee @ about 50 feet). 30 meters looks good today! My signal heard in VK2 too…

Greetings from north Italy, dear Scott!
73
Riccardo, IW4DXW

Holy… 200mW? That’s so awesome. Maybe one day I’ll build something equally as cool and epically useless =oD I will now return to my psychotic programming project what has been eating-up my spring break. [boom] I just snapped a screenshot for the future memory of this break. Yes, I’m on a XP machine. I’m at the W4DFU radio club station (they have nicer computers than I do!)
epicProgramming


     

I'm working too hard

Several days straight of coding all day every day has started to mess with my mind. I’ve got to incorporate some relaxation time! Something good ‘ol youtube can assist with. I share with you the following videos today instead of Python code (there’s always time for that later).


     

Dynamic Image Generation with Python

So here’s the problem. I’m working on a software project and I’d love the startup screen to be flashy, but not tacky. I want the program to be ENTIRELY scripted, so pretty bitmap images are out. I have to generate it from the script, but how can I reliably print text on an image with fonts vary across operating systems? (i.e., many OS’s don’t have “arial.ttf”, or no truetype fonts at all!) The solution was a creative process which will unfold before you.

First, I created a 2D binary array to represent the alphabet using pixel fonts such as this as a reference. The word I’m trying to create is “QRSS VD” (the name of my program). I store the data in strings, as seen below. I use 1’s to mark pixels, and spaces to mark empty spaces.

data="""
1111 1111 1111 1111   1  1 111
1  1 1  1 1    1      1  1 1  1
1  1 1111 1111 1111   1  1 1  1
1 11 1 1     1    1   1 1  1  1
1111 1 11 1111 1111    1   111
"""

Once I think it looks nice, I replace the spaces with zeros to make it take up a lot of visual space…

data2="""
1111011110111101111000100101110
1001010010100001000000100101001
1001011110111101111000100101001
1011010100000100001000101001001
1111010110111101111000010001110
"""

Then I further obscure it by replacing linebreaks with a different symbol, such as the number 2, then break the lines so they’re not lined up…

b="1111011110111101111000100101110210010100101000010000001001010012"
b+="1001011110111101111000100101001210110101000001000010001010010012"
b+="1111010110111101111000010001110"

The result is a pretty cool way to obscure the text. I don’t know why you’d want to, but if you want to make sure that no one goes in and changes the letters around (at least without making them think pretty hard about it) you could look at ways to further encrypt this data stream. From here, I create an image using the Python Imaging Library, setting pixel values to 255*b[x,y] (so 0 stays 0 and 1 becomes 255, perfect for an 8-bit image). After enlarging, here’s the result:

nearest

That’s cool huh? Now let’s make it a little bit less pixelated. It’s not a cure-all method, but blurring it up a little with a bilinear filter helps a lot…

bilinear

Then I apply the code below which applies a cool colormap to the pixel values. I’ll provide cleaner code for this later (I have a really cool way of generating colormaps and saving them as arrays of RGB tuples). Then I go through and plot some random sin wavs on top of it. Sweet! Here are 6 images generated from the program run 6 times. Notice the randomness of the sine wavs!

qrss vd

qrss vd

qrss vd

qrss vd

qrss vd

qrss vd

Vwa la! A different image is generated every time the script runs, and it requires no external files (bitmaps or fonts) and should work well on all operating systems. Take the idea and run with it!

from PIL import Image
from PIL import ImageOps
from PIL import ImageFilter
from random import randint
import scipy

def genLogo():
	colormap=[(0, 0, 129), (0, 0, 134), (0, 0, 139), (0, 0, 143), (0, 0, 148), (0, 0, 152), (0, 0, 157), (0, 0, 161), (0, 0, 166), (0, 0, 170), (0, 0, 175), (0, 0, 180), (0, 0, 184), (0, 0, 189), (0, 0, 193), (0, 0, 198), (0, 0, 202), (0, 0, 207), (0, 0, 211), (0, 0, 216), (0, 0, 220), (0, 0, 225), (0, 0, 230), (0, 0, 234), (0, 0, 239), (0, 0, 243), (0, 0, 248), (0, 0, 252), (0, 0, 255), (0, 0, 255), (0, 0, 255), (0, 0, 255), (0, 2, 255), (0, 7, 255), (0, 11, 255), (0, 14, 255), (0, 18, 255), (0, 23, 255), (0, 27, 255), (0, 31, 255), (0, 34, 255), (0, 39, 255), (0, 43, 255), (0, 47, 255), (0, 51, 255), (0, 54, 255), (0, 59, 255), (0, 63, 255), (0, 67, 255), (0, 71, 255), (0, 75, 255), (0, 79, 255), (0, 83, 255), (0, 87, 255), (0, 91, 255), (0, 95, 255), (0, 99, 255), (0, 103, 255), (0, 107, 255), (0, 111, 255), (0, 115, 255), (0, 119, 255), (0, 123, 255), (0, 127, 255), (0, 131, 255), (0, 135, 255), (0, 139, 255), (0, 143, 255), (0, 147, 255), (0, 151, 255), (0, 155, 255), (0, 159, 255), (0, 163, 255), (0, 167, 255), (0, 171, 255), (0, 175, 255), (0, 179, 255), (0, 183, 255), (0, 187, 255), (0, 191, 255), (0, 195, 255), (0, 199, 255), (0, 203, 255), (0, 207, 255), (0, 211, 255), (0, 215, 255), (0, 219, 254), (0, 223, 251), (0, 227, 248), (2, 231, 245), (5, 235, 241), (7, 239, 238), (11, 243, 235), (14, 247, 232), (18, 251, 228), (21, 255, 225), (23, 255, 222), (27, 255, 219), (31, 255, 215), (34, 255, 212), (37, 255, 208), (40, 255, 205), (44, 255, 203), (47, 255, 199), (50, 255, 195), (54, 255, 192), (57, 255, 189), (60, 255, 186), (63, 255, 183), (66, 255, 179), (70, 255, 176), (73, 255, 173), (76, 255, 170), (79, 255, 166), (83, 255, 163), (86, 255, 160), (89, 255, 157), (92, 255, 154), (95, 255, 150), (99, 255, 147), (102, 255, 144), (105, 255, 141), (108, 255, 137), (112, 255, 134), (115, 255, 131), (118, 255, 128), (121, 255, 125), (124, 255, 121), (128, 255, 118), (131, 255, 115), (134, 255, 112), (137, 255, 108), (141, 255, 105), (144, 255, 102), (147, 255, 99), (150, 255, 95), (154, 255, 92), (157, 255, 89), (160, 255, 86), (163, 255, 83), (166, 255, 79), (170, 255, 76), (173, 255, 73), (176, 255, 70), (179, 255, 66), (183, 255, 63), (186, 255, 60), (189, 255, 57), (192, 255, 54), (195, 255, 50), (199, 255, 47), (202, 255, 44), (205, 255, 41), (208, 255, 37), (212, 255, 34), (215, 255, 31), (218, 255, 28), (221, 255, 24), (224, 255, 21), (228, 255, 18), (231, 255, 15), (234, 255, 12), (238, 255, 8), (241, 252, 5), (244, 248, 2), (247, 244, 0), (250, 240, 0), (254, 236, 0), (255, 233, 0), (255, 229, 0), (255, 226, 0), (255, 221, 0), (255, 218, 0), (255, 215, 0), (255, 211, 0), (255, 207, 0), (255, 203, 0), (255, 199, 0), (255, 196, 0), (255, 192, 0), (255, 188, 0), (255, 184, 0), (255, 180, 0), (255, 177, 0), (255, 173, 0), (255, 169, 0), (255, 165, 0), (255, 162, 0), (255, 159, 0), (255, 155, 0), (255, 151, 0), (255, 147, 0), (255, 143, 0), (255, 140, 0), (255, 136, 0), (255, 132, 0), (255, 128, 0), (255, 125, 0), (255, 121, 0), (255, 117, 0), (255, 114, 0), (255, 110, 0), (255, 106, 0), (255, 102, 0), (255, 99, 0), (255, 95, 0), (255, 91, 0), (255, 88, 0), (255, 84, 0), (255, 80, 0), (255, 76, 0), (255, 73, 0), (255, 69, 0), (255, 65, 0), (255, 62, 0), (255, 58, 0), (255, 54, 0), (255, 51, 0), (255, 47, 0), (255, 43, 0), (255, 39, 0), (255, 36, 0), (255, 32, 0), (255, 28, 0), (255, 25, 0), (255, 21, 0), (253, 17, 0), (248, 14, 0), (244, 10, 0), (240, 6, 0), (235, 2, 0), (230, 0, 0), (225, 0, 0), (221, 0, 0), (217, 0, 0), (212, 0, 0), (207, 0, 0), (203, 0, 0), (198, 0, 0), (194, 0, 0), (189, 0, 0), (185, 0, 0), (180, 0, 0), (175, 0, 0), (171, 0, 0), (166, 0, 0), (162, 0, 0), (157, 0, 0), (152, 0, 0), (148, 0, 0), (144, 0, 0), (139, 0, 0), (134, 0, 0), (130, 0, 0), (134, 0, 0), (130, 0, 0)]
	def red(val):
		return colormap[val][0]
	def green(val):
		return colormap[val][1]
	def blue(val):
		return colormap[val][2]
	def colorize(im):
		r=Image.eval(im,red)
		g=Image.eval(im,green)
		b=Image.eval(im,blue)
		im=Image.merge("RGB",(r,g,b))
		return im
	b="1111011110111101111000100101110210010100101000010000001001010012"
	b+="1001011110111101111000100101001210110101000001000010001010010012"
	b+="1111010110111101111000010001110"
	b=b.split("2")
	im=Image.new("L",(33+15,7+13))
	data=im.load()
	for y in range(len(b)):
		for x in range(len(b[y])):
			data[x+6,y+6]=int(b[y][x])*255
	scale=15
	im=im.resize((im.size[0]*scale,im.size[1]*scale))
	data=im.load()
	def drawSin(width,height,vertoffset,horizoffset,thickness,darkness):
		for x in range(im.size[0]):
			y=scipy.sin((x-horizoffset)/float(width))*height+vertoffset
			for i in range(thickness):
				if 0<=y+i<im.size[1] and 0<=x<im.size[0]:
					#print x,im.size[0],y+i,im.size[1]
					data[x,y+i]=data[x,y+i]+darkness

	for i in range(5):
		print "line",i
		drawSin(randint(5,75),randint(-100,200),randint(0,im.size[1]),
				randint(0,im.size[0]),randint(3,15),70)
	for i in range(10):
		im=im.filter(ImageFilter.SMOOTH_MORE)
	im=colorize(im)
	return im

im=genLogo()
im.save('logo.png',"PNG")

     

Brewing Excitement

Wow, I can’t believe I took-on such a massive challenge this week! Some irony lies in the fact that I’ve worked harder and learned more in the last 4 days of all-day work (researching, reading, skimming other peoples’ code, and writing my own) than of the last 9 months of dental school. This is an incredible feeling of accomplishment. My program, *MINE*, which I coded 100% from scratch (using Python’s scripting platform as a strong base coupled with the Python Imaging Library (PIL), Tk bindings (Tkinter)). It polls the soundcard continuously and makes incredibly large spectrographs. I’ll explain more about it and its rationale later. It’s not finished, but it’s working… and working pretty darn well. I’m floored!

Here’s what it looks like when it’s running…
gotit

Pretty nice ‘eh? Yeah, that’s a GUI, but it can be run entirely from a headless server through a console as well. (see where I’m going with this?) Here’s some more of the output, cropped to emphasize QRSS signals. Keep in mind that the image below was cropped to less than 3,000 pixels high, whereas the original is over 8,000 pixels high!!!
qrss_big


     

Animated Realtime Spectrograph with Scrolling Waterfall Display in Python

WARNING: this project is largely outdated, and some of the modules are no longer supported by modern distributions of Python.

For a more modern, cleaner, and more complete GUI-based viewer of realtime audio data (and the FFT frequency data), check out my Python Real-time Audio Frequency Monitor project.

My project is coming along nicely. This isn’t an incredibly robust spectrograph program, but it sure gets the job done quickly and easily. The code below will produce a realtime scrolling spectrograph entirely with Python! spectrogram scrollbarsIt polls the microphone (or default recording device), should work on any OS, and can be adjusted for vertical resolution / FFT frequency discretion resolution. It has some simple functions for filtering (check out the detrend filter!) and might serve as a good start to a spectrograph / frequency analysis project. It took my a long time to reach this point! I’ve worked with Python before, and dabbled with the Python Imaging Library (PIL), but this is my first experience with realtime linear data analysis and high-demand multi-threading. I hope it helps you. Below are screenshots of the program (two running at the same time) listening to the same radio signals (mostly Morse code) with standard output and with the “detrending filter” activated.
nofilter
filter

And the code…

import pyaudio
import scipy
import struct
import scipy.fftpack

from Tkinter import *
import threading
import time, datetime
import wckgraph
import math

import Image, ImageTk
from PIL import ImageOps
from PIL import ImageChops
import time
import random
import threading
import scipy


#ADJUST RESOLUTION OF VERTICAL FFT
bufferSize=2**11
#bufferSize=2**8

#ADJUSTS AVERAGING SPEED NOT VERTICAL RESOLUTION
#REDUCE HERE IF YOUR PC CANT KEEP UP
sampleRate=24000
#sampleRate=64000

p = pyaudio.PyAudio()
chunks=[]
ffts=[]
def stream():
        global chunks, inStream, bufferSize
        while True:
                chunks.append(inStream.read(bufferSize))

def record():
        global w, inStream, p, bufferSize
        inStream = p.open(format=pyaudio.paInt16,channels=1,
                rate=sampleRate,input=True,frames_per_buffer=bufferSize)
        threading.Thread(target=stream).start()
        #stream()

def downSample(fftx,ffty,degree=10):
        x,y=[],[]
        for i in range(len(ffty)/degree-1):
                x.append(fftx[i*degree+degree/2])
                y.append(sum(ffty[i*degree:(i+1)*degree])/degree)
        return [x,y]

def smoothWindow(fftx,ffty,degree=10):
        lx,ly=fftx[degree:-degree],[]
        for i in range(degree,len(ffty)-degree):
                ly.append(sum(ffty[i-degree:i+degree]))
        return [lx,ly]

def smoothMemory(ffty,degree=3):
        global ffts
        ffts = ffts+[ffty]
        if len(ffts)< =degree: return ffty ffts=ffts[1:] return scipy.average(scipy.array(ffts),0) def detrend(fftx,ffty,degree=10): lx,ly=fftx[degree:-degree],[] for i in range(degree,len(ffty)-degree): ly.append((ffty[i]-sum(ffty[i-degree:i+degree])/(degree*2)) *2+128) #ly.append(fft[i]-(ffty[i-degree]+ffty[i+degree])/2) return [lx,ly] def graph(): global chunks, bufferSize, fftx,ffty, w if len(chunks)>0:
                data = chunks.pop(0)
                data=scipy.array(struct.unpack("%dB"%(bufferSize*2),data))
                #print "RECORDED",len(data)/float(sampleRate),"SEC"
                ffty=scipy.fftpack.fft(data)
                fftx=scipy.fftpack.rfftfreq(bufferSize*2, 1.0/sampleRate)
                fftx=fftx[0:len(fftx)/4]
                ffty=abs(ffty[0:len(ffty)/2])/1000
                ffty1=ffty[:len(ffty)/2]
                ffty2=ffty[len(ffty)/2::]+2
                ffty2=ffty2[::-1]
                ffty=ffty1+ffty2
                ffty=(scipy.log(ffty)-1)*120
                fftx,ffty=downSample(fftx,ffty,2)
                #fftx,ffty=detrend(fftx,ffty,30)
                #fftx,ffty=smoothWindow(fftx,ffty,10)
                #ffty=smoothMemory(ffty,3)
                #fftx,ffty=detrend(fftx,ffty,3)
                #print len(ffty)
                #print min(ffty),max(ffty)
                updatePic(fftx,ffty)
                reloadPic()
                #w.clear()
                #w.add(wckgraph.Axes(extent=(0, -1, 6000, 3)))
                #w.add(wckgraph.LineGraph([fftx,ffty]))
                #w.update()


        if len(chunks)>20:
                print "falling behind...",len(chunks)

def go(x=None):
        global w,fftx,ffty
        print "STARTING!"
        threading.Thread(target=record).start()
        while True:
                #record()
                graph()


def updatePic(datax,data):
     global im, iwidth, iheight
     strip=Image.new("L",(1,iheight))
     if len(data)>iheight:
             data=data[:iheight-1]
     #print "MAX FREQ:",datax[-1]
     strip.putdata(data)
     #print "%03d, %03d" % (max(data[-100:]), min(data[-100:]))
     im.paste(strip,(iwidth-1,0))
     im=im.offset(-1,0)
     root.update()

def reloadPic():
     global im, lab
     lab.image = ImageTk.PhotoImage(im)
     lab.config(image=lab.image)


root = Tk()
im=Image.open('./ramp.tif')
im=im.convert("L")
iwidth,iheight=im.size
im=im.crop((0,0,500,480))
#im=Image.new("L",(100,1024))
iwidth,iheight=im.size
root.geometry('%dx%d' % (iwidth,iheight))
lab=Label(root)
lab.place(x=0,y=0,width=iwidth,height=iheight)
go()

UPDATE! I’m not going to post the code for this yet (it’s very messy) but I got this thing to display a spectrograph on a canvas. What’s the advantage of that? Huge, massive spectrographs (thousands of pixels in all directions) can now be browsed in real time using scrollbars, and when you scroll it doesn’t stop recording, and you don’t lose any data! Super cool.

spectrogram scrollbars