Archive for June, 2010

These screenshots show me running the Py2EXE-compiled script I wrote last weekend on a Windows 7 machine. Additionally there is a screenshot of the “Add/Remove Programs” window demonstrating which versions of which libraries were required.

Wow, that’s a mouthfull. This is a total hack, but it works — and barely I might add! I spent all night jumping through hoops to get this thing to run on Windows. The problem is that I designed my previous UI in a version of GLADE which is newer than that supported by Windows. It looks like it’s not backward-compatible, so I have to re-design the GUI from scratch using an earlier version of GLADE. I’ll probably stick to GTK version 2.12 and Python version 2.6 because they play nicely on Windows. It’s a quick and dirty script, but I was able to make the following run on Windows as a single EXE file!

WHAT A NIGHTMARE

While continuing to investigate my options for the new version of QRSS VD, I re-visited Glade, the GTK GUI designer. In short, it lets you draw widgets (combo boxes, scrollbars, labels, images, buttons, etc) onto windows and then makes it easy to add code to the GUI. I *hated* the old QRSS VD development because of the ridiculously large amount of time I had to spend coding the UI. Hopefully by migrating from TKinter to GTK – while it opens a whole new can of worms – will let me add functionality rapidly without hesitation.

Here’s a quick screenshot of my running this new version of the software with a GUI I made in less than an hour. The bars for brightness and contrast can be adjusted which modify the spectrograph in real time. The audio is whatever is playing in Pandora. I like the “fantastic plastic machine” radio station!

I had TKinter all wrong! While my initial tests with PyGame’s rapid ability to render Numpy arrays in the form of pixel maps proved impressive, it was only because I was comparing it to poor TK code. I don’t know what I was doing wrong, but when I decided to give TKinter one more shot I was blown away — it’s as smooth or smoother as PyGame. Forget PyGame! I’m rendering everything in raw TK from now on. This utilizes the Python Imaging Library (PIL) so it’s EXTREMELY flexible (supports fancy operations, alpha channels, etc).

The screenshot shows me running the script (below) generating random noise and “scrolling” it horizontally (like my spectrograph software does) at a fast rate smoothly (almost 90 FPS!). Basically, it launches a window, creates a canvas widget (which I’m told is faster to update than a label and reduces flickering that’s often associated with rapid redraws because it uses double-buffering). Also, it uses threading to handle the calculations/redraws without lagging the GUI. The code speaks for itself.

import Tkinter
from PIL import Image, ImageTk
import numpy
import time

class mainWindow():
	times=1
	timestart=time.clock()
	data=numpy.array(numpy.random.random((400,500))*100,dtype=int)

	def __init__(self):
		self.root = Tkinter.Tk()
		self.frame = Tkinter.Frame(self.root, width=500, height=400)
		self.frame.pack()
		self.canvas = Tkinter.Canvas(self.frame, width=500,height=400)
		self.canvas.place(x=-2,y=-2)
		self.root.after(0,self.start) # INCREASE THE 0 TO SLOW IT DOWN
		self.root.mainloop()

	def start(self):
		global data
		self.im=Image.fromstring('L', (self.data.shape[1],\
			self.data.shape[0]), self.data.astype('b').tostring())
		self.photo = ImageTk.PhotoImage(image=self.im)
		self.canvas.create_image(0,0,image=self.photo,anchor=Tkinter.NW)
		self.root.update()
		self.times+=1
		if self.times%33==0:
			print "%.02f FPS"%(self.times/(time.clock()-self.timestart))
		self.root.after(10,self.start)
		self.data=numpy.roll(self.data,-1,1)

if __name__ == '__main__':
    x=mainWindow()

While continuing my quest into the world of linear data analysis and signal processing, I came to a point where I wanted to emphasize variations in FFT traces. While I am keeping my original data for scientific reference, visually I want to represent it emphasizing variations rather than concentrating on trends. I wrote a detrending function which I’m sure will be useful for many applications:

def detrend(data,degree=10):
	detrended=[None]*degree
	for i in range(degree,len(data)-degree):
		chunk=data[i-degree:i+degree]
		chunk=sum(chunk)/len(chunk)
		detrended.append(data[i]-chunk)
	return detrended+[None]*degree

However, this method is extremely slow. I need to think of a way to accomplish this same thing much faster. [ponders]

UPDATE: It looks like I’ve once again re-invented the wheel. All of this has been done already, and FAR more efficiently I might add. Simply:

import scipy.signal
ffty=scipy.signal.detrend(ffty)

Now I’m looking into scipy.signal.triang()

I’m attempting to thoroughly re-write the data assessment portions of my QRSS VD software, and rather than rushing to code it (like I did last time) I’m working hard on every step trying to optimize the code. I came across some notes I made about Fast Fourier Transformations from the first time I coded the software, and though I’d post some code I found helpful. Of particular satisfaction is an email I received from Alberto, I2PHD, the creator of Argo (the “gold standard” QRSS spectrograph software for Windows). In it he notes:

These concepts are simple to visualize when graphed. Here I’ve written a short Python script to listen to the microphone (which is being fed a 2kHz sine wave), perform the FFT, and graph the real FFT component, imaginary FFT component, and their sum. The output is:

Of particular interest to me is the beautiful complementarity of the two curves. It makes me wonder what types of data can be extracted by the individual curves (or perhaps their difference?) down the road. I wonder if phase measurements would be useful in extracting weak carries from beneath the noise floor?

Here’s the code I used to generate the image above. Note that my microphone device was set to listen to my stereo output, and I generated a 2kHz sine wave using the command speaker-test -t sine -f 2000 on a PC running Linux. I hope you find it useful!

import numpy
import pyaudio
import pylab
import numpy

### RECORD AUDIO FROM MICROPHONE ###
rate=44100
soundcard=1 #CUSTOMIZE THIS!!!
p=pyaudio.PyAudio()
strm=p.open(format=pyaudio.paInt16,channels=1,rate=rate,\
			input_device_index=soundcard,input=True)
strm.read(1024) #prime the sound card this way
pcm=numpy.fromstring(strm.read(1024), dtype=numpy.int16)

### DO THE FFT ANALYSIS ###
fft=numpy.fft.fft(pcm)
fftr=10*numpy.log10(abs(fft.real))[:len(pcm)/2]
ffti=10*numpy.log10(abs(fft.imag))[:len(pcm)/2]
fftb=10*numpy.log10(numpy.sqrt(fft.imag**2+fft.real**2))[:len(pcm)/2]
freq=numpy.fft.fftfreq(numpy.arange(len(pcm)).shape[-1])[:len(pcm)/2]
freq=freq*rate/1000 #make the frequency scale

### GRAPH THIS STUFF ###
pylab.subplot(411)
pylab.title("Original Data")
pylab.grid()
pylab.plot(numpy.arange(len(pcm))/float(rate)*1000,pcm,'r-',alpha=1)
pylab.xlabel("Time (milliseconds)")
pylab.ylabel("Amplitude")
pylab.subplot(412)
pylab.title("Real FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq,fftr,'b-',alpha=1)
pylab.subplot(413)
pylab.title("Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq,ffti,'g-',alpha=1)
pylab.subplot(414)
pylab.title("Real+Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq,fftb,'k-',alpha=1)
pylab.show()

After fighting for a while long with a “shifty baseline” of the FFT, I came to another understanding. Let me first address the problem. Taking the FFT of different regions of the 2kHz wave I got traces with the peak in the identical location, but the “baselines” completely different.

Like many things, I re-invented the wheel. Since I knew the PCM values weren’t changing, the only variable was the starting/stopping point of the linear sample. “Hard edges”, I imagined, must be the problem. I then wrote the following function to shape the PCM audio like a triangle, silencing the edges and sweeping the volume up toward the middle of the sample:

def shapeTriangle(data):
	triangle=numpy.array(range(len(data)/2)+range(len(data)/2)[::-1])+1
	return data*triangle

After shaping the data BEFORE I applied the FFT, I made the subsequent traces MUCH more acceptable. Observe:

Now that I’ve done all this experimentation/thinking, I remembered that this is nothing new! Everyone talks about shaping the wave to minimize hard edges before taking the FFT. BAH! Another case of me re-inventing the wheel because I’m too lazy to read others’ work. However, in my defense, I learned a lot by trying all this stuff — far more than I would have learned simply by copying someone else’s code into my script. Experimentation is the key to discovery!

While I wrote a pervious post on linear data smoothing with python, those scripts were never fully polished. Fred (KJ4LFJ) asked me about this today and I felt bad I had nothing to send him. While I might add that the script below isn’t polished, at least it’s clean. I’ve been using this method for all of my smoothing recently. Funny enough, none of my code was clean enough to copy and paste, so I wrote this from scratch tonight. It’s a function to take a list in (any size) and smooth it with a triangle window (of any size, given by “degree”) and return the smoothed data with or without flanking copies of data to make it the identical length as before. The script also graphs the original data vs. smoothed traces of varying degrees. The output is below. I hope it helps whoever wants it!

import numpy
import pylab

def smoothTriangle(data,degree,dropVals=False):
	"""performs moving triangle smoothing with a variable degree."""
	"""note that if dropVals is False, output length will be identical
	to input length, but with copies of data at the flanking regions"""
	triangle=numpy.array(range(degree)+[degree]+range(degree)[::-1])+1
	smoothed=[]
	for i in range(degree,len(data)-degree*2):
		point=data[i:i+len(triangle)]*triangle
		smoothed.append(sum(point)/sum(triangle))
	if dropVals: return smoothed
	smoothed=[smoothed[0]]*(degree+degree/2)+smoothed
	while len(smoothed)<len(data):smoothed.append(smoothed[-1])
	return smoothed

### CREATE SOME DATA ###
data=numpy.random.random(100) #make 100 random numbers from 0-1
data=numpy.array(data*100,dtype=int) #make them integers from 1 to 100
for i in range(100):
	data[i]=data[i]+i**((150-i)/80.0) #give it a funny trend

### GRAPH ORIGINAL/SMOOTHED DATA ###
pylab.plot(data,"k.-",label="original data",alpha=.3)
pylab.plot(smoothTriangle(data,3),"-",label="smoothed d=3")
pylab.plot(smoothTriangle(data,5),"-",label="smoothed d=5")
pylab.plot(smoothTriangle(data,10),"-",label="smoothed d=10")
pylab.title("Moving Triangle Smoothing")
pylab.grid(alpha=.3)
pylab.axis([20,80,50,300])
pylab.legend()
pylab.show()

While thinking of ways to improve my QRSS VD high-definitions spectrograph software, I often wish I had a better way to display large spectrographs. Currently I’m using PIL (the Python Imaging Library) with TK and it’s slow as heck. I looked into the PyGame project, and it seems to be designed with speed in mind. I whipped-up this quick demo, and it’s a simple case audio spectrograph which takes in audio from your sound card and graphs it time vs. frequency. This method is far superior to the method I was using previously to display the data, because while QRSS VD can only update the entire GUI (500px by 8,000 px) every 3 seconds, early tests with PyGame suggests it can do it about 20 times a second (wow!). With less time/CPU going into the GUI, the program can be more responsivle and my software can be less of a drain.

import pygame
import numpy
import threading
import pyaudio
import scipy
import scipy.fftpack
import scipy.io.wavfile
import wave
rate=12000 #try 5000 for HD data, 48000 for realtime
soundcard=2
windowWidth=500
fftsize=512
currentCol=0
scooter=[]
overlap=5 #1 for raw, realtime - 8 or 16 for high-definition
def graphFFT(pcm):
	global currentCol, data
	ffty=scipy.fftpack.fft(pcm) #convert WAV to FFT
	ffty=abs(ffty[0:len(ffty)/2])/500 #FFT is mirror-imaged
	#ffty=(scipy.log(ffty))*30-50 # if you want uniform data
	print "MIN:\t%s\tMAX:\t%s"%(min(ffty),max(ffty))
	for i in range(len(ffty)):
		if ffty[i]<0: ffty[i]=0
		if ffty[i]>255: ffty[i]=255
	scooter.append(ffty)
	if len(scooter)<6:return
	scooter.pop(0)
	ffty=(scooter[0]+scooter[1]*2+scooter[2]*3+scooter[3]*2+scooter[4])/9
	data=numpy.roll(data,-1,0)
	data[-1]=ffty[::-1]
	currentCol+=1
	if currentCol==windowWidth: currentCol=0

def record():
	p = pyaudio.PyAudio()
	inStream = p.open(format=pyaudio.paInt16,channels=1,rate=rate,\
						input_device_index=soundcard,input=True)
	linear=[0]*fftsize
	while True:
		linear=linear[fftsize/overlap:]
		pcm=numpy.fromstring(inStream.read(fftsize/overlap), dtype=numpy.int16)
		linear=numpy.append(linear,pcm)
		graphFFT(linear)

pal = [(max((x-128)*2,0),x,min(x*2,255)) for x in xrange(256)]
print max(pal),min(pal)
data=numpy.array(numpy.zeros((windowWidth,fftsize/2)),dtype=int)
#data=Numeric.array(data) # for older PyGame that requires Numeric
pygame.init() #crank up PyGame
pygame.display.set_caption("Simple Spectrograph")
screen=pygame.display.set_mode((windowWidth,fftsize/2))
world=pygame.Surface((windowWidth,fftsize/2),depth=8) # MAIN SURFACE
world.set_palette(pal)
t_rec=threading.Thread(target=record) # make thread for record()
t_rec.daemon=True # daemon mode forces thread to quit with program
t_rec.start() #launch thread
clk=pygame.time.Clock()
while 1:
	for event in pygame.event.get(): #check if we need to exit
		if event.type == pygame.QUIT:pygame.quit();sys.exit()
	pygame.surfarray.blit_array(world,data) #place data in window
	screen.blit(world, (0,0))
	pygame.display.flip() #RENDER WINDOW
	clk.tick(30) #limit to 30FPS

After spending nearly my entire teen-age life bitterly rejecting any Microsoft products (none of the computers in my room had any MS software on any of them for years), I gradually eased back into the Windows life. My wife got a new PC, it ran Windows, she was happy, I was happy. She’s since gotten a better PC, and I settled into using her old one. Windows (Vista) was junky as heck, so I replaced it with XP, and never got around to loading Linux to dual boot with. While I wasn’t a fan of Windows, it seemed easy enough, and I tolerated it. Today I come home after a long day of dental school and the darn thing won’t boot. It tries to boot, and goes into some weird endless reboot cycle. After getting mad and just letting it reboot on its own a bunch of times, it finally managed to boot — but with no icons, and explorer.exe wouldn’t load. I booted in safe mode, consolidated all the files I wanted into a single folder, then booted from a USB drive I carry with me at all times which has an Ubuntu LiveCD on it. I’m installing Linux as I write this. [snap]

There are several things ironic about this:
1.) I’m surfing the net / blogging WHILE installing Ubuntu – try being productive while Windows is installing!
2.) The Internet works. Even with windows installed, the Internet doesn’t work with this USB wireless adapter – it requires drivers. In fact, it doesn’t work on Windows 7 AT ALL! And here it is running out of the box.
3.) I can determine my graphics cards with a single command: lspci – try doing that with Windows. MAYBE you’ll see “UNKNOWN DISPLAY ADAPTER” in the Device Manager at best. Here I have drivers up and ready to go.
4.) I’m not even a fan of Linux. At this point, I’m more disappointed in Microsoft. I don’t care what software I run – I just want it to work, and I feel I’m getting let down again and again. Even if my problem were caused by a virus, it begs the question of why it can happen in the first place. All I do is browse the web! Are you kidding me?

Bah! I hate computers -_-

update: Kyle Walker posted a comic I had to share. Thanks Kyle!

I’m starting to investigate PyGame as an alternative to PIL and K for my QRSS VD spectrograph project. This sample code makes a box bounce around a window.

import pygame, sys
pygame.init() #load pygame modules
size = width, height = 320, 240 #size of window
speed = [2, 2] #speed and direction
screen = pygame.display.set_mode(size) #make window
s=pygame.Surface((100,50)) #create surface 100px by 50px
s.fill((33,66,99)) #color the surface blue
r=s.get_rect() #get the rectangle bounds for the surface
clock=pygame.time.Clock() #make a clock
while 1: #infinite loop
        clock.tick(30) #limit framerate to 30 FPS
        for event in pygame.event.get(): #if something clicked
                if event.type == pygame.QUIT: #if EXIT clicked
                        sys.exit() #close cleanly
        r=r.move(speed) #move the box by the "speed" coordinates
        #if we hit a  wall, change direction
        if r.left < 0 or r.right > width: speed[0] = -speed[0]
        if r.top < 0 or r.bottom > height: speed[1] = -speed[1]
        screen.fill((0,0,0)) #make redraw background black
        screen.blit(s,r) #render the surface into the rectangle
        pygame.display.flip() #update the screen

While it may not be perfect, it’s a whole lot better. Below is a capture from this morning of my signal (the waves near the bottom). Compare that to how it was before and you should notice a dramatic improvement! The MEPT is inside a metal box inside a 1-inch-thick Styrofoam box. Very cool!

As if I didn’t ALREADY have enough against me, my oscilloscope decided to die on my RIGHT as I finally was able to view my 10MHz waveform. (I used a piece of coax with a load at the connector to the o-scope, and ran the coax to my test points.) It was beautiful! … and lasted about 30 seconds. The culprit seems to be a failing “focus” knob. My images had been getting blurrier by the day, and now it’s completely black unless I twist pretty hard on the focus knob. I’d stick a small POT in there, but I have no idea how much voltage/current is being regulated. I’m sure the schematics are posted somewhere, but for now I’m going to try to clean out the potentiometer manually and see if the situation improves. Here are some photos of the circuitry inside this old scope – too bad they don’t make stuff like this anymore!

I completed work on my first RF receiver, and for what it is it seems to work decently. It should be self-explanatory from the photos. It’s based around an SA602. As with everything, I don’t plan on posting schematics until the project is complete because I don’t want people re-creating junky circuits! It’s stationed at the University of Florida’s club station W4DFU and its spectrograph can be viewed in real time from the QRSS VD – Web Grabber – W4DFU page. Back to work!

And some music to complete the day!

Now that my minimalist QRSS transmitter is mostly functional, I’m shifting gears toward building a minimalist receiver. These are some early tests, but I’m amazed I managed to hack something together that actually works! Once it’s finished I’ll post schematics. For now, here are some photos. This receiver is based upon an SA602 and although there *IS* an op-amp on the board, I actually bypassed it completely! The SA602 seems to put out enough juice to make my PC microphone jack happy, and those cheap op-amps are noisy anyway, so awesome! Go minimalism!

Here it’s pictured with its power supply:

Here’s a close-up. Remember, the op-amp is there but NOT used!

Here’s the output from 7.040 MHz. Conditions are pretty bad right now, and I’m at my apartment using my crazy indoor antenna [pic1] [pic2]

This minimal Python script will convert a directory filled with tiny image captures such as this into gorgeous montages as seen below! I whipped-up this script tonight because I wanted to assess the regularity of my transmitter’s embarrassing drift. I hope you find it useful.

full-size output:

10x squished output:

Script to assemble a folder of images into a single, large image:

import os
from PIL import Image

x1,y1,x2,y2=[0,0,800,534] #crop from (x,y) 0,0 to 800x534
squish=10 #how much to squish it horizontally

### LOAD LIST OF FILES ###
workwith=[]
for fname in os.listdir('./'):
    if ".jpg" in fname and not "assembled" in fname:
        workwith.append(fname)
workwith.sort()

### MAKE NEW IMAGE ###
im=Image.new("RGB",(x2*len(workwith),y2))
for i in range(len(workwith)):
    print "Loading",workwith[i]
    im2=Image.open(workwith[i])
    im2=im2.crop((x1,y1,x2,y2))
    im.paste(im2,(i*x2,0))
print "saving BIG image"
im.save("assembled.jpg")
print "saving SQUISHED image"
im=im.resize((im.size[0]/10,im.size[1]),Image.ANTIALIAS)
im.save("assembled-squished.jpg")
print "DONE"

Script to download every image linked to from a webpage:

import urllib2
import os

suckFrom="http://w1bw.org/grabber/archive/2010-06-08/"

f=urllib2.urlopen(suckFrom)
s=f.read().split("'")
f.close()
download=[]

for line in s:
    if ".jpg" in line and not line in download and not "thumb" in line:
        download.append(line)

for url in download:
    fname = url.split("/")[-1].replace(":","-")
    if fname in os.listdir('./'):
        print "I already downloaded",fname
    else:
        print "downloading",fname
        output=open(fname,'wb')
        output.write(urllib2.urlopen(url).read())
        output.close()

These should speak for themselves. Obviously I’m the crazy person who thinks it’s funny to merge molecular biology with amateur radio.

Belgium JO10UX:

England G4CWX:

France JN39AB:

Massachusetts W1BW:

Nevada KK7CC:

Netherlands JO22DA:

Alaska KL1X:

Italia I2NDT:

New Zealand ZL2IK:

Germany DL4MGM:

I’m still working on this project, and although progress is slow I’m learning a lot and the circuit is getting better with time. I’m still not yet ready to post the schematics, but you can get an idea of what’s going on from the picture. It can handle 255 levels of frequency shift and has the ability to turn the tone on and off. 6 capacitors, 3 resistors, 4 transistors, a single inductor, and a micro-controller. Boom!

… and yeah, that’s a double helix

UPDATE I spotted myself on W4BHK’s Grabber about 300 miles away…

#include <avr /io.h>
#include <util /delay.h>

char dotlen=5; // ultimately the speeed of transmission
char call[]={0,1,1,1,2,0,2,1,1,0}; // 0 for space, 1 for dit, 2 for dah

void setfor(char freq, char ticks){
	OCR1A=freq;
	while (ticks>0){
		sleep();
		ticks--;
	}
}

void sleep(){
	for  (char i=0;i<dotlen ;i++){
		_delay_loop_2(65000);
	}
}

void slideto(char freq, char ticks){
	freq=freq+30;
	char step=1;
	if (OCR1A>freq){step=-1;}
	while (OCR1A!=freq){
		OCR1A+=step;
		setfor(OCR1A, 1);
	}
	setfor(freq, ticks);
}

void DNA(){
	char a[]={4,5,5,6,6,6,7,7,7,7,8,8,8,8,8,7,7,7,7,6,6,6,5,5,4,3,3,2,2,2,1,1,1,1,0,0,0,0,0,1,1,1,1,2,2,2,3,3};
	char b[]={1,1,1,1,2,2,2,3,3,4,5,5,6,6,6,7,7,7,7,8,8,8,8,8,7,7,7,7,6,6,6,5,5,4,3,3,2,2,2,1,1,1,1,0,0,0,0,0};
	for (char i=0;i<sizeof (a);i++){
		//slideto(a[i]*4,2);
		//slideto(b[i]*4,2);
		setfor(a[i]*2+5, 10);
		setfor(b[i]*2+5, 10);
	}
}

void ID(){
	for (char i=0;i<sizeof(call);i++){
		setfor(10,50);
		if (call[i]==0){setfor(10,100);}
		if (call[i]==1){setfor(15,100);}
		if (call[i]==2){setfor(15,250);}
		setfor(10,50);
	}
}

void ID2(){
	for (char i=0;i<sizeof(call);i++){
		if (call[i]==0){ampOFF();setfor(10,50);}
		if (call[i]==1){ampON();setfor(10,100);}
		if (call[i]==2){ampON();setfor(13,100);}
		ampOFF();setfor(OCR1A,30);
	}
	ampON();
}

void ampON(){PORTA|=(1<<PA7);PORTA|=(1<<PA0);PORTA&=~(1<<PA1);_delay_loop_2(10000);}
void ampOFF(){PORTA&=~(1<<PA7);PORTA|=(1<<PA1);PORTA&=~(1<<PA0);_delay_loop_2(10000);}

int main(void)
{
    DDRA = 255;
	OCR1A = 75;TCCR1A = 0x81;TCCR1B = 1;
	while (1){
		ID2();
		ID();
		for (char i=0;i<3;i++){
			DNA();
		}
	}

}

These TV shows always get it right!