Here’s a rough approximation of the current schematic of the prime number calculator I’m working on. Last night I finished wiring all 12 shift registers for the primary display, so now it’s time to start working on software. Notice that we have a lot of pins free still. This will be advantageous if I decide to go crazy adding extraneous functionality, such as fancy displays (LCD?, 7-segment LEDs?, VFD?, all 3?!) or creative input systems (how about a numerical keypad?). After feeling the stink of paying almost $15 for 100′ of yellow, 24 gauge, solid-core wire from DigiKey I was relieved (and a little embarrassed) to find I could score 1,000′ of yellow, 24 gauge, threaded wire for $10 at Skycraft! Anyway, here’s the current schematic:
Last weekend was field day, a disaster simulation / competition for amateur radio operators. In a sentence, people are encouraged to make as many contacts as they can around the world (earning points) using emergency radio preparations (battery and solar powered radios, temporary antennas, etc) for a full 24 hours (2pm to 2pm). I spent the time with the UCF Amateur Radio Club who set up big antennas in a grassy field on campus. It was a fun experience, and the first time I ever got to see a HF rig in operation. A representative for the UCF newspaper showed up, took some interviews, and I ended-up being quoted in the article. I can also be seen in the photo, if you look close enough (yellow square).
Being that amateur radio was something I got into independently (I didn’t know anyone else with a license) I was (and still am) very isolated in the hobby. I’m really thankful I found the UCF ARC, even though it wasn’t until I’d already been going to UCF for 2 years and was already on my way out. Seeing (and actually get to use) a HF rig was an eye-opening experience for me, and one I’m a little regretful I participated in. Before yesterday, I had already come to terms with my situation (going to dental school in a few weeks and virtually dropping all of my hobbies) and was content with my summer accomplishments so far. My summer goal was to get into radio, and before yesterday I felt I had. I studied for my exam, got my license, learned how to use repeaters on VHF to easily make local contacts, and I was satisfied. I knew HF was out there, and that it allowed communication over thousands of miles, but I ignored it knowing I wouldn’t get into it this summer (the equipment is just too expensive for me to justify purchasing). Now, after sitting in front of a rig for several hours, I wish I had the time to upgrade my license, earn a little cash to blow on a HF radio, and spend a few weeks sitting in front of it scouring the waves for random voices around the world. I know it’s a little morbid, but I’d probably have to compare the feeling I’m experiencing with what an old person feels like when they realize their end is near and that they won’t be able to do the things they always dreamed they would. Oh well, at least I’ll be able to fill holes in teeth soon. [smiles convincingly]
I’ll update my progress on this project as I go. I added a lot more light bars to the shift registers on my prime number generator project. I’m up to 5 daisy-chained shift registers completed (powering 40 LEDs) with 7 more to go! I’m using 22 gauge solid-core (fancy and expensive, from digikey, 100′ 14$!) wire for the back of this project. Being that I plan to keep it for many years, I want it to look crazy awesome. Remember, I’m only about 1/3 done so far…
I powered the device up and it produced proper output. Yay! I was so discouraged yesterday when I wired-up an entire row (the top one), powered it on, and 1/2 the LEDs didn’t work. At first I thought it was software, but then I realized that I burned the LEDs out in the soldering process by getting them too hot. I had to de-solder EVERYTHING, rip out the destroyed LED bars, and start over. I’ll have to pick up some more light bars at Skycraft soon. This is what it looks like currently:
I’m making this project a priority because I only have a few weeks before I move to Gainesville, FL for dental school (the cutoff date for all electronics/radio/programming projects). I’ll be busy the next few days with other obligations (work, apartment hunting, field day, etc.) but I hope to resume this project soon.
UPDATE (June 26, 2009 @ 7:30pm): I finished wiring all the light bars I have. I need to purchase 3 more 10-led bars at Skycraft to replace the ones I melted with my soldering iron. D’oh! Anyway, here’s the beaut:
Bitwise programming techniques (manipulating binary numbers) is simple in theory, but it’s often hard to remember how to do specific tasks if you don’t do them often. Recently in my microcontroller programming endeavors (where you’re pressed to conserve every bit of memory) I’ve needed to perform a lot of bitwise operations. If I’m storing true/false (1-bit) information in variables, it’s a waste to assign a whole variable to the task (even a char, the smallest variable in C is a waste because it uses 8 bits of memory!). When cramming multiple values into individual variables, it’s nice to know how to manipulate each bit of a variable.
Questions like “how do I retrieve the value of a certain bit in a variable”, “how do I set the value of a certain bit in a variable”, and “how do I flip a certain bit in a variable” can eventually be answered by twiddling around with bitwise operators in C, but often the solutions you randomly discover this way are not elegant or efficient. This afternoon I ran across the following chart on an Arduino help site and although I’m not a fan of Arduino, I can certainly appreciate the chart. I hope you find it as useful as I did.
y = (x>>n)&1; // stores nth bit of x in y. y becomes 0 or 1.
x&=~(1<<n); // forces nth bit of x to be 0. all other bits left alone.
x&=(1<<(n+1))-1; // leaves lowest n bits of x; all higher bits set to 0.
x|=(1<<n); // forces nth bit of x to be 1. all other bits left alone.
x^=(1<<n); // toggles nth bit of x. all other bits left alone.
x=~x; // toggles ALL the bits in x.
I’m currently challenging myself by creating a microcontroller-based project a few orders of magnitude more complex than anything I’ve ever done before. Although this is probably on par with projects you might see being created by senior electrical engineering seniors, keep in mind that I have no formal training in engineering, and that my MS is in molecular biology. I just started learning about circuitry / microcontrollers a few months ago, and challenge myself to learn more by continually attacking greater and greater challenges. Here’s what I began working on last night:
This if the first entry describing the creation of my non-prototype microcontroller-powered prime number generator. I made a proof of concept device a few weeks ago which calculates prime numbers (up to 2^25, about 33.5 million) and displays the results in binary form using 25 LEDs assembled in a 5×5 matrix. I added an extra column of 5 LEDs for a final matrix size of 6×5, illuminated by multiplexing through 11 IO pins of an ATMEL ATTiny2313. My new project will do the same thing, except it can calculate prime numbers up to 2^30 (over 1 billion!). Instead of only displaying 1 number, it will display 3 numbers (last prime, test number, and the divisor) using 90 LEDs. The picture above is of the main circuit board before I began soldering. The empty sockets will house a combination of 8-bit shift registers, binary-to-digital converters, and 7-segment display drivers all powered by an ATMEL ATMega8 microcontroller crystal-clocked at 10.042 MHz (arbitrary, but stable). Here’s what the underside looked like before I began soldering:
I anticipate that this project will develop into a soldering nightmare. The board is nearly too small as it is, and I don’t have good wire for soldering. (I’m actually using the small wires from an old phone cord right now.) I included a potentiometer, 2 buttons, and 3 switches to aid with various settings (brightness, menus, etc). 3 Rows of LEDs (60 pins each) requires 12 shift registers (16 pins each) plus the 28-pin microcontroller makes about 400 solder points (YIKES!), so I anticipate the underside of this project will quickly grow to become a daunting mess of wires. Last night I finished the connections necessary to program the microcontroller, and for the microcontroller to control a single 8-bit shift register, allowing the first 8 LEDs of the first number to be controlled. Here’s what the soldering looked like. Remember, the dense clump of connections only controls 8 LEDs, so multiply this by more than 10 and that’s what I’ll have to do JUST to power the display.
After programming with a straight-through DAPA style parallel-port programmer, I was able to shift data out to the single HC595 I had wired. I spent hours banging my head against the wall because nothing I did in the software would make the LEDs illuminate. I thought I was sending signals to the shift register wrong, or that I soldered something wrong. I finally concluded that somehow (probably when I was troubleshooting by applying 5v of power directly to the pins of the shift registers) I managed to burn out all 8 LEDs of the first light bar. I had to de-solder ALL of the connections you see in that picture, replace the bar with a new one (thank goodness I had an extra), and re-solder everything. I have a feeling that by the end of this project, I’ll be an expert at soldering. Here’s the program running controlling the first 8 bits only:
Supposedly the hard part is done for the display. The software was written in such a way that it will automatically begin lighting-up more LEDs as I wire them. The small node for the first shift register and 8 LEDs will be identical to the other 11, and as I solder them one by one I’ll get closer to my end goal. It will probably be many hours of soldering. In retrospect, I wish I purchased a bigger perfboard. Actually, in retrospect I wish I made a PCB!!
Update: After a few more hours (of soldering, troubleshooting, desoldering, and rewiring, and resoldering) I have my second 8-bit segment working. Note all the yellow (newly-added) wires. Multiply this by 10, and that’s what I have left to wire for the display alone!
When I figured this out I figured it was simply way too easy and way to helpful to keep to myself. Here I post (for the benefit of friends, family, and random Googlers alike) two examples of super-simplistic ways to read PCM data from Python using Numpy to handle the data and Matplotlib to display it. First, get some junk audio in PCM format (test.pcm).
import numpy
data = numpy.memmap("test.pcm", dtype='h', mode='r')
print "VALUES:",data
This code prints the values of the PCM file. Output is similar to: VALUES: [-115 -129 -130 ..., -72 -72 -72]
To graph this data, use matplotlib like so:
import numpy, pylab
data = numpy.memmap("test.pcm", dtype='h', mode='r')
print data
pylab.plot(data)
pylab.show()
This will produce a graph that looks like this:
Could it have been ANY easier? I’m so in love with python I could cry right now. With the powerful tools Numpy provides to rapidly and efficiently analyze large arrays (PCM potential values) combined with the easy-to-use graphing tools Matplotlib provides, I’d say you can get well on your way to analyzing PCM audio for your project in no time. Good luck!
Let’s get fancy and use this concept to determine the number of seconds in a 1-minute PCM file in which a radio transmission occurs. I was given a 1-minute PCM file with a ~45 second transmission in the middle. Here’s the graph of the result of the code posted below it. (Detailed descriptions are at the bottom) Figure description: The top trace (light blue) is the absolute value of the raw sound trace from the PCM file. The solid black line is the average (per second) of the raw audio trace. The horizontal dotted line represents the threshold, a value I selected. If the average volume for a second is above the threshold, that second is considered as “transmission” (1), if it’s below the threshold it’s “silent” (0). By graphing these 60 values in bar graph form (bottom window) we get a good idea of when the transmission starts and ends. Note that the ENTIRE graphing steps are for demonstration purposes only, and all the math can be done in the 1st half of the code. Graphing may be useful when determining the optimal threshold though. Even when the radio is silent, the microphone is a little noisy. The optimal threshold is one which would consider microphone noise as silent, but consider a silent radio transmission as a transmission.
### THIS CODE DETERMINES THE NUMBER OF SECONDS OF TRANSMISSION
### FROM A 60 SECOND PCM FILE (MAKE SURE PCM IS 60 SEC LONG!)
import numpy
threshold=80 # set this to suit your audio levels
dataY=numpy.memmap("test.pcm", dtype='h', mode='r') #read PCM
dataY=dataY-numpy.average(dataY) #adjust the sound vertically the avg is at 0
dataY=numpy.absolute(dataY) #no negative values
valsPerSec=float(len(dataY)/60) #assume audio is 60 seconds long
dataX=numpy.arange(len(dataY))/(valsPerSec) #time axis from 0 to 60
secY,secX,secA=[],[],[]
for sec in xrange(60):
secData=dataY[valsPerSec*sec:valsPerSec*(sec+1)]
val=numpy.average(secData)
secY.append(val)
secX.append(sec)
if val>threshold: secA.append(1)
else: secA.append(0)
print "%d sec of 60 used = %0.02f"%(sum(secA),sum(secA)/60.0)
raw_input("press ENTER to graph this junk...")
### CODE FROM HERE IS ONLY USED TO GRAPH THE DATA
### IT MAY BE USEFUL FOR DETERMINING OPTIMAL THRESHOLD
import pylab
ax=pylab.subplot(211)
pylab.title("PCM Data Fitted to 60 Sec")
pylab.plot(dataX,dataY,'b',alpha=.5,label="sound")
pylab.axhline(threshold,color='k',ls=":",label="threshold")
pylab.plot(secX,secY,'k',label="average/sec",alpha=.5)
pylab.legend()
pylab.grid(alpha=.2)
pylab.axis([None,None,-1000,10000])
pylab.subplot(212,sharex=ax)
pylab.title("Activity (Yes/No) per Second")
pylab.grid(alpha=.2)
pylab.bar(secX,secA,width=1,linewidth=0,alpha=.8)
pylab.axis([None,None,-0.5,1.5])
pylab.show()
I’ve been working on the pySquelch project which is basically a method to graph frequency usage with respect to time. The code I’m sharing below listens to the microphone jack on the sound card (hooked up to a radio) and determines when transmissions begin and end. First, I’ll entice you by showing some nice graphs of the output! I ran the code below for 24 hours and this is the result… Pretty good ‘eh? This graph represents traces of the frequency activity with respect to time. The semi-transparent gray line represents the raw frequency usage in fractional minutes the frequency was tied-up by transmissions. The solid blue line represents the same data but smoothed by 10 minutes (in both directions) by the Gaussian smoothing method modified slightly from my linear data smoothing with Python page.
I used the code below to generate the log, and the code further below to create the graph from the log file. Assuming your microphone is enabled and everything else is working, this software will require you to determine your own threshold for talking vs. no talking. Read the code and you’ll figure out how test your sound card settings.
If you want to try this yourself you need a Linux system (a Windows system version could be created simply by replacing getVolEach() with a Windows-based audio level detection system) with Python and the alsaaudio, numpy, and matplotlib libraries. Try running the code on your own, and if it doesn’t recognize a library “aptitude search” for it. Everything you need can be installed from packages in the common repository.
#pySquelchLogger.py
import time, random, alsaaudio, audioop
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE,alsaaudio.PCM_NONBLOCK)
inp.setchannels(2)
inp.setrate(1000)
inp.setformat(alsaaudio.PCM_FORMAT_S8)
inp.setperiodsize(100)
addToLog=""
lastLogTime=0
testLevel=False ### SET THIS TO 'True' TO TEST YOUR SOUNDCARD
def getVolEach():
# this is a quick way to detect activity.
# modify this function to use alternate methods of detection.
while True:
l,data = inp.read() # poll the audio device
if l>0: break
vol = audioop.max(data,1) # get the maximum amplitude
if testLevel: print vol
if vol>10: return True ### SET THIS NUMBER TO SUIT YOUR NEEDS ###
return False
def getVol():
# reliably detect activity by getting 3 consistant readings.
a,b,c=True,False,False
while True:
a=getVolEach()
b=getVolEach()
c=getVolEach()
if a==b==c:
if testLevel: print "RESULT:",a
break
if a==True: time.sleep(1)
return a
def updateLog():
# open the log file, append the new data, and save it again.
global addToLog,lastLogTime
#print "UPDATING LOG"
if len(addToLog)>0:
f=open('log.txt','a')
f.write(addToLog)
f.close()
addToLog=""
lastLogTime=time.mktime(time.localtime())
def findSquelch():
# this will record a single transmission and store its data.
global addToLog
while True: # loop until we hear talking
time.sleep(.5)
if getVol()==True:
start=time.mktime(time.localtime())
print start,
break
while True: # loop until talking stops
time.sleep(.1)
if getVol()==False:
length=time.mktime(time.localtime())-start
print length
break
newLine="%d,%d "%(start,length)
addToLog+=newLine
if start-lastLogTime>30: updateLog() # update the log
while True:
findSquelch()
The logging code (above) produces a log file like this (below). The values represent the start time of each transmission (in seconds since epoch) followed by the duration of the transmission.
This log file is only 7.3 KB. At this rate, a years’ worth of log data can be stored in less than 3MB of plain text files. Awesome! The data presented here can be graphed (producing the image at the top of the page) using the following code:
#pySquelchGrapher.py
print "loading libraries...",
import pylab, datetime, numpy
print "complete"
def loadData(fname="log.txt"):
print "loading data...",
# load signal/duration from log file
f=open(fname)
raw=f.read()
f.close()
raw=raw.replace('\n',' ')
raw=raw.split(" ")
signals=[]
for line in raw:
if len(line)<3: continue
line=line.split(',')
sec=datetime.datetime.fromtimestamp(int(line[0]))
dur=int(line[1])
signals.append([sec,dur])
print "complete"
return signals
def findDays(signals):
# determine which days are in the log file
print "finding days...",
days=[]
for signal in signals:
day = signal[0].date()
if not day in days:
days.append(day)
print "complete"
return days
def genMins(day):
# generate an array for every minute in a certain day
print "generating bins...",
mins=[]
startTime=datetime.datetime(day.year,day.month,day.day)
minute=datetime.timedelta(minutes=1)
for i in xrange(60*60):
mins.append(startTime+minute*i)
print "complete"
return mins
def fillMins(mins,signals):
print "filling bins...",
vals=[0]*len(mins)
dayToDo=signals[0][0].date()
for signal in signals:
if not signal[0].date() == dayToDo: continue
sec=signal[0]
dur=signal[1]
prebuf = sec.second
minOfDay=sec.hour*60+sec.minute
if dur+prebuf<60: # simple case, no rollover seconds
vals[minOfDay]=dur
else: # if duration exceeds the minute the signal started in
vals[minOfDay]=60-prebuf
dur=dur+prebuf
while (dur>0): # add rollover seconds to subsequent minutes
minOfDay+=1
dur=dur-60
if dur< =0: break
if dur>=60: vals[minOfDay]=60
else: vals[minOfDay]=dur
print "complete"
return vals
def normalize(vals):
print "normalizing data...",
divBy=float(max(vals))
for i in xrange(len(vals)):
vals[i]=vals[i]/divBy
print "complete"
return vals
def smoothListGaussian(list,degree=10):
print "smoothing...",
window=degree*2-1
weight=numpy.array([1.0]*window)
weightGauss=[]
for i in range(window):
i=i-degree+1
frac=i/float(window)
gauss=1/(numpy.exp((4*(frac))**2))
weightGauss.append(gauss)
weight=numpy.array(weightGauss)*weight
smoothed=[0.0]*(len(list)-window)
for i in range(len(smoothed)):
smoothed[i]=sum(numpy.array(list[i:i+window])*weight)/sum(weight)
while len(list)>len(smoothed)+int(window/2):
smoothed.insert(0,smoothed[0])
while len(list)>len(smoothed):
smoothed.append(smoothed[0])
print "complete"
return smoothed
signals=loadData()
days=findDays(signals)
for day in days:
mins=genMins(day)
vals=normalize(fillMins(mins,signals))
fig=pylab.figure()
pylab.grid(alpha=.2)
pylab.plot(mins,vals,'k',alpha=.1)
pylab.plot(mins,smoothListGaussian(vals),'b',lw=1)
pylab.axis([day,day+datetime.timedelta(days=1),None,None])
fig.autofmt_xdate()
pylab.title("147.120 MHz Usage for "+str(day))
pylab.xlabel("time of day")
pylab.ylabel("fractional usage")
pylab.show()
If you have any questions, Google first, then feel free to contact me if you still can’t get it. Good luck!!
While working to improve the python-based frequency activity logger I wrote a couple of entries back, I greatly increased its accuracy. I won’t go into the details other than saying that it simply polls the sound card a few times a second to listen for when a transmission begins and ends. The new data file format appears like this…
The major change is that the date and time is recorded as seconds since epoch (Unix time) and the duration of the transmission is given after the comma. This greatly simplifies post-processing. Here is the revised python code to poll the sound card and generate such a log file:
##################################
##### UPDATED CODE IS IN A NEWER ENTRY #####
### CHECK THE PYTHON AND RADIO CATEGORIES ###
##################################
import time, random, alsaaudio, audioop
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE,alsaaudio.PCM_NONBLOCK)
inp.setchannels(2)
inp.setrate(1000)
inp.setformat(alsaaudio.PCM_FORMAT_S8)
inp.setperiodsize(100)
addToLog=""
lastLogTime=0
testLevel=False ### SET THIS TO TRUE TO TEST YOUR SOUNDCARD
def getVolEach():
# this is a quick way to detect activity.
# modify this function to use alternate methods of detection.
while True:
l,data = inp.read() # poll the audio device
if l>0: break
vol = audioop.max(data,1) # get the maximum amplitude
if testLevel: print vol
if vol>10: return True ### SET THIS NUMBER TO SUIT YOUR NEEDS ###
return False
def getVol():
# reliably detect activity by getting 3 consistant readings.
a,b,c=True,False,False
while True:
a=getVolEach()
b=getVolEach()
c=getVolEach()
if a==b==c:
if testLevel: print "RESULT:",a
break
if a==True: time.sleep(1)
return a
def updateLog():
# open the log file, append the new data, and save it again.
global addToLog,lastLogTime
#print "UPDATING LOG"
if len(addToLog)>0:
f=open('log.txt','a')
f.write(addToLog)
f.close()
addToLog=""
lastLogTime=time.mktime(time.localtime())
def findSquelch():
# this will record a single transmission and store its data.
global addToLog
while True: # loop until we hear talking
time.sleep(.5)
if getVol()==True:
start=time.mktime(time.localtime())
print start,
break
while True: # loop until talking stops
time.sleep(.1)
if getVol()==False:
length=time.mktime(time.localtime())-start
print length
break
newLine="%d,%d\n"%(start,length)
addToLog+=newLine
if start-lastLogTime>30: updateLog() # update the log
while True:
findSquelch()
To test the functionality of the code visually, you can use this quick and dirty method of graphing the log files output by this program. Keep in mind THIS IS NOT INTENDED TO BE USED other than to simply test the program.
##################################
##### UPDATED CODE IS IN A NEWER ENTRY #####
### CHECK THE PYTHON AND RADIO CATEGORIES ###
##################################
import pylab
import time, datetime
f=open('log.txt')
raw=f.read()
f.close()
raw=raw.split("\n")
pylab.figure()
for line in raw:
if len(line)<5:continue
line=line.split(",")
sec=datetime.datetime.fromtimestamp(int(line[0]))
dur=int(line[1])
sec2=datetime.datetime.fromtimestamp(int(line[0])+int(line[1]))
pylab.fill([sec,sec,sec2,sec2],[0,1,1,0],'k',lw=0,alpha=.5)
pylab.show()
This morning I woke up at 4:45am, hopped out of bed, and raced to the university parking lot for field day. It’s pretty much a flea market with an emphasis in ham radio and associated electronics. This is a panorama of the parking lot the tailgate was held in, taken from the roof of a parking garage at about 9am. The UCF ARC (the amateur radio club which sponsored the event) is stationed under the white tent.
My goal was to purchase a [working] oscilloscope, and I lucked-out. I ended-up purchasing two, and I’m glad I did! The 1st one (the one with the green circular screen) crapped-out on me after literally 1 minute. (By crapped-out I mean it started spurring thick gray smoke and made my whole apartment smell like a burned marshmallow). At $5, I’m not crying over it. The second one is a 1969 Tektronix 561A 10 MHz oscilloscope. Just think, these things just started started being produced the same year Neil Armstrong walked on the moon. I tested it and it seems to be functioning well. At $10, I’m very happy! The back one there is the one that died on me. The closer one works beautifully, but a lot of the knobs are staticky and there’s a lot of dirt/dust all over it. Inside it looks like a dust bomb detonated. It’ll take enough of my time to restore that I’m thinking of ignoring the older, non-working one. I don’t want to trash it, but I don’t want to store it. Maybe I’ll salvage parts from it? Hmm, vacuum tubes.
Here you can see it attached to my prime number generator described in agonizingly-boring detail over the last several weeks’ posts. It’s attached to one of the microcontroller pins responsible for multiplexing the LED display. Finally, a way to assess high speed power output as a function of time. The output of the microcontroller isn’t performing like I expected, and since it’s a series of pulses I can’t use a volt meter to measure its output. Thus, the need [more like desire] for an oscilloscope.
As anyone who visits this website can tell you, I’m inexplicably drawn toward inventing new ways of using Python (often in conjunction with MatPlotLib) to graphically visualize data in new ways. When I found out a fellow ham in Orlando was using his computer to stream (and serve recorded streams of) a popular local repeater frequency over the internet. I got excited because of the potential for generating [quasi-useful, and at least interesting] data from the existing setup. Since this guy already has his radio connected to his PC’s microphone jack, I figured I could write a Python app. to check the microphone input to determine if anyone is using the frequency. By recording when people start and stop talking, I can create a log of frequency activity. Later I can write software to visualize this data. I’ll talk about that in a later post. For now, here’s how I used Python and a Linux box (Ubuntu, with the python-alsaaudio package installed) to generate such logs.
We can visualize this data using some more simple Python code. Long term it would be useful to visualize frequency activity similarly to how I graphed computer usage at work over the last year but for now since I don’t have any large amount of data to work with. I’ll just write cote to visualize a QSO (conversation) with respect to time. It should be self-explanatory. This data came from data points displayed in the video (provided at the end of this post too).
And, of course, the code I used to generate the log files (seen running in video above): Briefly, this program checks the microphone many times every second to determine if its state has changed (talking/no talking) and records this data in a text file (which it updates every 10 seconds). Matplotlib can EASILY be used to graph data from such a text file.
##################################
##### UPDATED CODE IS IN A NEWER ENTRY #####
### CHECK THE PYTHON AND RADIO CATEGORIES ###
##################################
import alsaaudio, time, audioop, datetime
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE,alsaaudio.PCM_NONBLOCK)
inp.setchannels(1)
inp.setrate(4000)
inp.setformat(alsaaudio.PCM_FORMAT_S16_LE)
inp.setperiodsize(1)
squelch = False
lastLog = 0
dataToLog = ""
def logIt(nowSquelch):
global dataToLog, lastLog
timeNow = datetime.datetime.now()
epoch = time.mktime(timeNow.timetuple())
if nowSquelch==True: nowSquelch=1
else: nowSquelch=0
logLine="%s %d\n"%(timeNow, nowSquelch)
print timeNow, nowSquelch
dataToLog+=logLine
if epoch-lastLog>10:
#print "LOGGING..."
f=open('squelch.txt','a')
f.write(dataToLog)
f.close()
lastLog = epoch
dataToLog=""
while True:
l,data = inp.read()
if l:
vol = audioop.max(data,2)
#print vol #USED FOR CALIBRATION
if vol>800: nowSquelch = True
else: nowSquelch = False
if not nowSquelch == squelch:
logIt(nowSquelch)
squelch = nowSquelch
time.sleep(.01)
To use this code make sure that you’ve properly calibrated it. See the “vol>800″ line? That means that if the volume in the microphone is at least 800, it’s counted as talking, and less than it’s silence. Hopefully you can find a value that counts as silence when the squelch is active, but as talking when the squelch is broken (even if there’s silence). This is probably best achieved with the radio outputting at maximum volume. You’ll have to run the program live with that line un-commented to view the data values live. Find which values occur for squelch on/off, and pick your threshold accordingly.
After you run the code, it should output data like this:
After that you can visualize the data with the following code. Note that this is SEVERELY LIMITED and is only useful when graphing a few minutes of data. I don’t have hours/days of data to work with right now, so I won’t bother writing code to graph it. This code produced the graph seen earlier in this page. Make sure matplotlib is installed on your box.
##################################
##### UPDATED CODE IS IN A NEWER ENTRY #####
### CHECK THE PYTHON AND RADIO CATEGORIES ###
##################################
import pylab
def loadData():
#returns Xs
import time, datetime, pylab
f=open('good.txt')
raw=f.readlines()
f.close()
onTimes=[]
timeStart=None
lastOn=False
for line in raw:
if len(line)<10: continue
line = line.strip('\n').split(" ")
t=line[0]+" "+line[1]
t=t.split('.')
thisDay=time.strptime(t[0], "%Y-%m-%d %H:%M:%S")
e=time.mktime(thisDay)+float("."+t[1])
if timeStart==None: timeStart=e
if line[-1]==1: stat=True
else: stat=False
if not lastOn and line[-1]=="1":
lastOn=e
else:
onTimes.append([(lastOn-timeStart)/60.0,\
(e-timeStart)/60.0])
lastOn=False
return onTimes
times = loadData()
pylab.figure(figsize=(8,3))
for t in times:
pylab.fill([t[0],t[0],t[1],t[1]],[0,1,1,0],'k',lw=0,alpha=.5)
pylab.axis([None,None,-3,4])
pylab.title("A little QSO")
pylab.xlabel("Time (minutes)")
pylab.show()
For the last several weeks I’ve been hacking together a small prototype of a microcontroller (ATTiny2313)-powered prime number generator. I can finally say that it (the prototype) is complete, functional, and elegant [get the song I'm referencing while it's up!]. The name says what it does, so I won’t waste time describing it. If you’re interested in how it does what it does, check out the other posts with the same category tag for a full (and I mean full as in “more than you ever wanted to know”) description. A schematic is soon to come. Code is below. For now, here’s a video of the completed project:
And the source code I used. I chose the ATTiny2313 because it was cheap (~$2) and has 18 IO pins to work with. I had to fight memory usage every step of the way. I’m limited to 2kb, and this program compiles within 1% of that! For future projects (more LEDs, more menus) I’ll use the 20-pin and/or 40-pin ATMega series. I’m thinking about having one be in charge of the display (multiplexing) leaving the other to think about generating prime numbers.
#define F_CPU 10240000
#include <avr/interrupt.h>
#include <util/delay.h>
#include <math.h>
// ~~~ PORT D ~~~
#define r1 0b00000100
#define r2 0b00000010
#define r3 0b00001000
#define r4 0b00100000
#define r5 0b00010000
#define f1 0b01000000
#define id 0b00000001
#define f0 0b10111111
// ~~~ PORT B ~~~
#define c1 0b00010000
#define c2 0b00000001
#define c3 0b00000010
#define c4 0b00000100
#define c5 0b00001000
char delay=1;
int redo=0;
char rows[] = {r1,r2,r3,r4,r5};
char cols[] = {c1,c2,c3,c4,c5};
char vals[] = {0,0,0,0,0};
char funcs=f1+id;
unsigned long showNum=5;//33554431;
void displayNum();
void incrimentNum();
void menu_pause();
void menuCheck();
char button();
char isPrime(unsigned long);
unsigned int twoTo(char);
int main(void) {
DDRD = r1+r2+r3+r4+r5+f1+id;
DDRB = c1+c2+c3+c4+c5;
DDRB &= ~_BV(DDB7);
PORTB = 0;
PORTD = 0;
unsigned int i=0;
int j;
//convertNum();
//showNum=5;
while (1) {
showNum+=1;
if (isPrime(showNum)){
convertNum();
}
for (j=0;j<redo;j++) {
displayNum();
menu();
}
if (i%10==0){funcs^=r1;
funcs^=id;
if (i%100==0){funcs^=r2;
if (i%1000==0){funcs^=r3;i=0;
}
}
}
i++;
}
return 0;
}
char isPrime(unsigned long test){
if (test%2==0) return 0;
unsigned long div = 3;
while(div*div<test+1){
if (test%div==0) return 0;
div+=2;
displayNum();
}
return 1;
}
void menu(){
//return;
char j,but;
but = button();
if (but==0) return;
else if (but==1){
while (1) {
PORTD=id;
if (PINB & _BV(PB7)) {
funcs=f1;
for (j=0;j<200;j++){
if (j%25==0) funcs^=r1;
displayNum();
}
}
else {
if (redo==0) redo=200;
else redo=0;
_delay_ms(300);
return;
}
}
}
return;
}
char button(){
PORTD=id;
if (PINB & _BV(PB7)) return 0; // not pressed
_delay_ms(1000);
if (PINB & _BV(PB7)) return 1; // pressed
return 2; // held down
}
void convertNum(){
char col,row,rowcode;
unsigned long msk=1;
for (col=0;col<5;col++){
rowcode=0;
for (row=0;row<5;row++){
if (showNum&msk) rowcode+=rows[row];
msk = msk < < 1;
}
vals[col]=rowcode;
}
return;
}
void displayNum(){
char i,j;
PORTB = 0b0;
PORTD = funcs;
_delay_ms(delay);
for (i=0;i<5;i++){
PORTD = vals[i];
PORTB = cols[i];
_delay_ms(delay);
}
return;
}
I decided to crank up the power on my prime number identifier and attempt to document its efficiency in both C (GCC) and Python. I used the [very inefficient] code from the previous entry, modified it to test every integer up to 1×10^8, and let them both run on my laboratory computer overnight (an AMD Athlon 64-bit X2 Dual Core 2 GHz PC with 2GB of ram). The Python program took 3.24 processor hours, and the C program took 1.85 processor hours to complete. Each step along the way each program recorded how long it took to test the last 10,000 integers, creating a logfile with 1,000 data points, which can be easily graphed (see previous entry). On this nice, fast computer the data look very clean and curve-like. I curve-fitted the data using my new favorite website, ZunZun.com. Here’s what I came up with…
Coeffecients:
### C (GCC) ###
a = 1.4193535434065586E-03
b = -1.2708466972699874E-07
c = -4.9489218065978358E-16
d = 1.3032518272036044E-11
e = 1.3231610935638881E-06
### PYTHON ###
a = 2.6579973323520396E-03
b = -2.8299592901357803E-07
c = -1.2080374779761445E-15
d = 3.0281624700595501E-11
e = 2.4778595094623538E-06
### Generating Data Points: ###
Ys=[]
for x in range(len(values)):
Ys.append(a*(x**.5)+b*(x)+c*(x**2)+d*(x**1.5)+e)
Note that this fitted curve is the representation of how long (in seconds, vertical axis) it takes to systematically test 10,000 integers for primeness (starting at an integer on the horizontal axis), This does NOT represent how long it takes to reach a certain X. To properly calculate this, one would need to integrate the equation. This would allow for the easy prediction of how long it would take to reach a certain integer (the solution would be the integral of the provided equation from 0 to the integer). I’ll let someone else attack that. It’s time for me to get back to work!
While working on my current microcontroller project I’ve become a little obsessed with the prospect of rapidly generating lists of prime numbers. I’ve been testing various strategies for speeding up the process using logic modifications to a base concept. I’ve been doing my conceptual work in Python because it’s so fast and easy to rapidly develop applications with. Now that I have a good understanding of my strategy (code scheme) of choice, I’m starting to wonder how much better the same code would run in C. It’s no secret that Python’s strength is its versatility and ease of development of scritps, not its speed. C on the other hand can be very cumbersome and awkward to develop with, but its compiled code is very efficient so it’s better suited for mathematical applications which require billions of repetitive tasks.
To visualize the speed difference between C and Python, I wrote the same prime-number-testing code in both languages and timed their output. Basically I wrote identical code in each language (variable names and functions are the same, almost a line-by-line translation) and timed how long it took to find all prime numbers up to 10,000,000. (It reported the length of runtime at every 100,000 integers, totalling 100 time points).
The results:
The black line is code written in C and the blue line is Python. According to this output, Pythin is about 4 times slower than C at identifying primes utilizing the identical method. I’m not a coding expert, and there may be better ways to code/compile things in each language, so I’ll asy this is an indication of speed rather than a detailed, scientific study comparing the two languages. For fairness, I’ll provide my code.
Python code to generate prime numbers up to 10^7:
import time
def isPrime(testPrime):
for div in xrange(2,int(testPrime**.5)+1):
if testPrime%div==0: return False
return True
def testSpeed(startAt, stopAt):
primesFound=0
startTime = time.clock()
for testPrime in xrange(startAt,stopAt):
if isPrime(testPrime):
primesFound+=1
s="%d,%d,%d,%fn"%(primesFound,startAt,stopAt,time.clock()-startTime)
print s
f=open("log_py.txt",'a')
f.write(s)
f.close()
return
for i in range(100):
testSpeed(100000*i,100000*(i+1))
raw_input()
C code to generate prime numbers up to 10^7:
#include <stdio.h>
#include <math.h>
#include <time.h>
char isPrime(int testPrime){
int div;
for (div=2;div<(int)sqrt(testPrime)+1;div++){
if (testPrime%div==0) return 0;
}
return 1;
}
void testSpeed(unsigned int startAt, unsigned int stopAt){
int primesFound=0;
char buf[256],len;
unsigned int testPrime;
clock_t startTime = clock();
for(testPrime=startAt;testPrime<stopAt;testPrime++){
if (isPrime(testPrime)) primesFound++;
}
len=sprintf(buf,"n%i,%u,%u,%f",primesFound,startAt,stopAt,
(float)(clock()-startTime)/CLOCKS_PER_SEC);
printf("%s",buf);
FILE *fptr = fopen("log_c.txt","a");
fputs(buf,fptr);
fclose(fptr);
return;
}
int main(){
char i;
for (i=0;i<100;i++){
testSpeed(100000*i,100000*(i+1));
}
return 0;
}
Python code to graph the difference utilizing MatPlotLib:
import pylab
def graphIt(fname,col):
f=open(fname,'r')
data=f.readlines()
f.close()
val,time=[],[]
for line in data:
if line.count(',')<3: continue
line=line.strip('n').split(',')
val.append(int(line[1]))
time.append(float(line[3]))
pylab.plot(val,time,color=col)
return [val,time]
pylab.figure()
pylab.subplot(311)
v,tc=graphIt('log_c.txt','k');
v,tp=graphIt('log_py.txt','b');
pylab.title("C vs Python: test 10,000 numbers for primeness")
pylab.ylabel("time (seconds)")
pylab.xlabel("starting value")
pylab.grid(alpha=.3)
pylab.subplot(312)
pylab.grid(alpha=.3)
graphIt('log_c.txt','k');
graphIt('log_py.txt','b');
pylab.ylabel("time (seconds)")
pylab.xlabel("starting value")
pylab.semilogy()
pylab.subplot(313)
ratios=[]
for x in range(len(v)):
ratios.append(float(tp[x])/tc[x])
pylab.plot(v,ratios,'k')
pylab.ylabel("speed ratio")
pylab.xlabel("starting value")
pylab.axis([None,None,None,5])
pylab.grid(alpha=.3)
pylab.show()
I don’t think I’ll keep the slang title McPPNG for my Microcontroller Powered Prime Number Generator, however I have some good news to report. I have a FULLY FUNCTIONAL prototype of this project and it looks better than I ever could have imagined, especially for the prototype. There are a few more features I want to add, and I want to work on the code some more, but I hope to be done tomorrow. I have a ton more photos taken and some videos in the works, but I’m saving them until this prototype is complete before I share it. The coolest part is that I’ve included an internal button which drives a pause/resume and speed-controller menu based upon the length of button presses! There’s a lot of awesome stuff I want to write, but once again, I’ll save it for the completed project page. Here are some pics to entice your curiosity… Before you lose your cool about the number on the screen not being prime, calm down. I mislabeled all of the LEDs. The first LED should be 2^0 (1), and the second should be 2^1 (2), etc. Also, 2^22 and 2^23 are mislabeled – oops! But the thing really does generate, differentiate, and display [only[ prime numbers. Once again, videos (including demonstration of the menus and the programming options) and source code will be posted soon.
This week has been a hectic week of ups and downs, with the stress never letting up. Work is stressful (I’m being challenged to collect representative data to write & submit two manuscripts for publication in JCN ASAP) and home is stressful. My wife and I are down to one car right now, which has no AC and pumps out heat! There’s always stuff I need to do around the house (cleaning, cooking, laundry, etc). Bills are actually getting a little intimidating since I haven’t gotten paid accurately in over a month and a half due to a payroll problem at UCF. On top of everything, I only have a few minutes here and there to devote to my microcontroller projects. When I have time to work on them, I’m rushed, and often make bad mistakes (such a yesterday when I destroyed 6 of my 2222 NPN transistors).
Thankfully, I’ve had a couple lucky breaks lately. Yesterday I finished the hardware enclosure and wiring for my microcontroller-powered prime number generator prototype. The code is sufficient to blink each of the 30 LEDs individually. When I do it fast, they all appear on. I’m golden – all I need to do to finish the [prototype] project is software side. Time to refresh myself on how to pass arrays with pointers in C! In other news, I had a little mental breakthrough this afternoon allowing me to generate prime numbers significantly more sufficiently. The screenshot below shows me identifying the 2,364,346′th prime number (38,790,019) in a little under two and a half hours. Yay! [makes dinner]
After a few hours running the code described above (writing down some time points along the way) I decided to try and visualize the curve of the rate of primes discovered with respect to time. I figured it would resemble an inverse exponential curve but it was surprisingly linear. Before I go to bed tonight I’ll re-write the code to make it automatically log time points so I can graph it in higher accuracy tomorrow. For now, this is what it looks like:
As much as I’d like to toot my own horn about how fast and efficient my computational method is. The linear estimate based upon this data (an overshoot of the actual speed) suggests that it finds 10,000 prime numbers per minute. That sounds lovely, but according to my math in order to get to the 10^12′th prime (the goal) it would take 190 years. I hope I have a full battery.
The code to generate the graph (requires MatPlotLib):
data="""
112.5,2062841
141.0,2364346
168.75,2656766
177.5,2745573
214.5,3123373
232.5,3290106
247.5,3419679
256.5,3493450
"""
import pylab
x,y,fx,fy=[],[],[],[]
for line in data.split('n'):
if not "," in line: continue
nx,ny = line.split(",")
x.append(float(nx))
y.append(float(ny))
pylab.figure(figsize=(8,4))
pylab.plot(x,y,'k.-',label='real data',lw=1)
pylab.plot([x[0],x[-1]],[y[0],y[-1]],'b:',label='linear estimate')
pylab.ylabel("number of primes found")
pylab.xlabel("time (minutes)")
pylab.title("pyPrime Nth Prime Speed (bin=[2,3,5])")
pylab.legend(loc=8)
pylab.grid(alpha=.2)
pylab.show()
After letting the program run overnight using a larger prime bin (primes=[2,3,5,7]) and recording the progress every 100 bins I have a better idea of the speed (and rate of decay) of this method.
Now that I’ve worked-out the software side of the ATMEL microcontroller-powered prime number generator, it’s time to start working on the hardware. Mind you that this is a prototype and not the final project. This is far smaller and simpler than the final version. For starters, I’m only multiplexing 30 LEDs (the full version will have at least 80). Also, this is being run by an ATTiny2313 microcontroller, and the full version will be powered by an ATMEega8. I picked up an unfinished wooden box with a magnetic latch from Michaels. I think it’s balsa wood. It’s really delicate and tends to chip when you drill it.
I used InkScape to make the layout of the LEDs. I simply made an 8.5×11” document, measured the box lid, drew a square that size, then made some Xs on a grid. I printed it out, taped it to the top of my box, and used my uber-fancy dremel drill press to drill perfectly-aligned holes. I’m so impressed by how easy this was that I wished I used the hexagonal layout I proposed earlier! Here are some photos:
I used my dremel drill press to drill nice holes in the lid of the box
The template I designed in InkScape was taped onto the lid of the box so that I'd have a guide to drill the holes - it worked beautifully!
Holes were enlarged by a hand drill with a bit the size of the cylindrical LEDs
After LEDs were inserted I added some little support posts with screw holes in them to the base so I'd have a place to screw-on my perfboard/circuitry down the road
You can get an idea for how the finished project will look by falsely-illuminating the lEDs (they're tinted plastic after all!)
The circuitry controlling the display involved an ATTiny2313 microcontroller (clocked at 10.2MHz with an oscillator and two capacitors for consistancy) and some transistors controlling grounding
I included some extra wires for prototyping/debugging/programming that will be later discarded. The chip in the breadboard (ATMega8) does nothing.
This is what it looks like in the light...
... and in the dark
my incomplete code didn't illuminate the entire display
So you’re pretty close to being done with the prototype, right? Yes and no. Yes in the sense that I’ve made the enclosure, basic circuitry, and basic code. No in the sense that I still have to improve the enclosure, circuitry, and code. For starters, after I took these pictures I touched the microcontroller and burned my finger!! It was running hot. I’m surprised I didn’t fry it altogether! I quickly powered it down and started inspecting the circuitry. Apparently Mr. I-got-a-masters-degree-and-am-going-to-be-a-dentist-soon doesn’t pay attention to detail [I'm losing prospective dental patients by writing this right now] and managed to wire every single one of six transistors backwards [shriek!] I guess I was pumping current out one side of the microcontroller and into the other side. Live and learn I guess. I have to go home tonight, cut all of them out (they were “permanently ghettorigged” in such a way that simple desoldering techniques will not remove them safely), find another batch of NPN transistors and solder them all in correctly.
This is the circuit concept. The chip is an ATTiny2313, sourced with 5V, where the left pins control the columns (by providing current) and the right pins control the rows (by providing ground). The “holes” at the top of the circuit represent where I hook up my PC and external power for testing purposes.
Being the semi-poor graduate student that I am, I have to be extremely selective what I spend my money on. Non-essential purchases (such as PIC microcontrollers, voltage regulators, wire, switches, buttons, LEDs) always take a back seat to essential purchases (such as food, rent, and ATMEL microcontrollers). Therefore, when I begin a new project, I’m forced to recycle parts from my old projects. This is the secondary reason why I don’t have any good completed projects on display at my house (the primary reason being that I never make them look pretty enough to be considered anything other than an eyesore in the first place). It’s always sad though when I begin tearing something apart. When I began making it, I thought it would be amazing! Here I am, a few months later, caring so little about it that I just want to rip it open for parts. Before I tear this guy apart (I don’t want to even describe what it does – it’s embarrassing) I took some photos so I could remember it. At least it looks pretty! Functionally, I programmed the microcontroller to blink LEDs and output sound in response to button presses. It also has a power supply circuit, which I’m harvesting in addition to the potentiometer.
While contemplating the feasibility of my most recent project (the microcontroller-powered prime number generator featured in the previous entry) I began to wonder exactly how efficient the current design is. The same chip doing the math is also running a fairly complicated 9-layer multiplexed display. I toyed with the idea of utilizing two microcontrollers – one to handle the display, and another to be the mathematical powerhouse. Theoretically this would be a very efficient way to display data on an LCD, because all of that processing power required to send parallel data to the display could instead be devoted to calculation. However, the bottom line is that in order to accurately determine the best setup for my needs, I need to first determine what my needs are!
Here’s a simple question: “How many calculations will be required?” In other words, if I’m counting from 1 to 2^25 (33.5 million), dividing each number by every integer between 2 and its square root, by the time I get to the end how many calculations will I have run? If I can determine the speed at which the microcontroller calculates divisions I can multiply it by the number of required divisions and have an estimate of how long (years?) it will take to reach 2^25. Finally, a use for basic calculus! Let’s assume that no numbers are eliminated, and every test number is divided by each integer between 0 and its square root. Therefore the number of divisions for each test number (Phi, the squiggly line) is its square root. Summing up the square roots of Phi from 0 to 2^25 requires simple integration:
Alas! It appears that approximately 388 billion calculations are required. Let’s round down to 300 billion, considering that many of these numbers are not prime and will be excluded quickly. I know this is a rough estimation, but at least it’s numerical! Before I did the math, my only guess would be “a really big number”. Now, let’s guesstimate some efficiency. If my previous project (with the LCD screen) didn’t require parallel communication with the display and could concentrate just on the path, I’m guessing I could do 1,000 divisions a second. At that rate, 300 billion divisions would take 3472 days (9 and a half years). Yeah, I think that’ll keep me busy for a while. How awesome would it be to wait around for a decade and actually be there watching when it finishes! I wonder if it would accurately loop around to 0, or crash… I’ll have to try it to find out. Look me up in 10 years.
Seeing (and actually get to use) a HF rig was an eye-opening experience for me, and one I’m a little regretful I participated in. Before yesterday, I had already come to terms with my situation (going to dental school in a few weeks and virtually dropping all of my hobbies) and was content with my summer accomplishments so far. My summer goal was to get into radio, and before yesterday I felt I had. I studied for my exam, got my license, learned how to use repeaters on VHF to easily make local contacts, and I was satisfied. I knew HF was out there, and that it allowed communication over thousands of miles, but I ignored it knowing I wouldn’t get into it this summer (the equipment is just too expensive for me to justify purchasing). Now, after sitting in front of a rig for several hours, I wish I had the time to upgrade my license, earn a little cash to blow on a HF radio, and spend a few weeks sitting in front of it scouring the waves for random voices around the world. I know it’s a little morbid, but I’d probably have to compare the feeling I’m experiencing with what an old person feels like when they realize their end is near and that they won’t be able to do the things they always dreamed they would. Oh well, at least I’ll be able to fill holes in teeth soon. [smiles convincingly]
