Raspberry Pi RF Frequency Counter

I build a lot of RF circuits, and often it’s convenient to measure and log frequency with a computer. Previously I’ve built standalone frequency counters, frequency counters with a PC interface, and even hacked a classic frequency counter to add USB interface (twice, actually). My latest device uses only 2 microchips to provide a Raspberry Pi with RF frequency measurement capabilities. The RF signal clocks a 32-bit counter SN74LV8154 ($1.04 on Mouser) connected to a 16-bit IO expander MCP23017 ($1.26 on Mouser) accessable to the Raspberry Pi (via I²C) to provide real-time frequency measurements from a python script for $2.30 in components! Well, plus the cost of the Raspberry Pi. All files for this project are on my GitHub page.

img_8773

The entire circuit is only two microchips! I have a few passives to clean up the RF signal (the RF input is loaded with a 1k resistor to ground, decoupled through a series 100 nF capacitor, and balanced at VCC/2 through a voltage divider of two 47k resistors), but if the measured signal is already a strong square wave they could be omitted. The circuit requires a gate pulse which typically will be 1 pulse per second (1PPS) and can be generated by dividing-down a 32.768kHz oscillator, a spare pin on a microcontroller, a fancy 1PPS time reference, or like in my case a GPS module (Neo-6M) with 1PPS output to provide an extremely accurate gate.

schem

The connections are intuitive! The I2C address is 0x20 when A0, A1, and A2 are grounded. GPB(1-4) control the register select of the counter, and GPA(0-7) reads each bit of the selected register. The whole thing is controlled from Python, but could be trivially written in any language.

img_8777

Here’s a quick summary describing how the code works: First I send bytes to address 0 and 1 to set all pins of GPIO A as inputs, and GPIO B as outputs. Note that only 4 of 8 pins are used for the output, so technically 4 extra pins could be used for things like blinking LEDs or controlling other devices. I then set the register select pins by sending a value to 0x13 (GPIO B), and read the entire GPIO A bus (INTCAPB, 0x18). For address details, consult the datasheet. I do this 4 times (1 for each byte of the 32-bit counter), do a little math to turn it into a frequency value, and compare the current value with the last value and take the difference to display as the measured frequency.

screenshot

An advantage of this continuously running mode is that no clock cycles are lost, so a gate which accidentally fires a bit early due to jitter and cuts-off a cycle will compensate for it on a subsequent read. This is shown above, as a very stable 10MHz frequency reference is measured with this method. A “slow” 1PPS clock tick causes a reading slightly higher, compensated-for by the next reading being slightly lower. In this way, clock sources which are extremely accurate but suffer from low precision (like GPS time sources) are able to maximize the long-term measurement of frequency. Combining this frequency measurement technique with the ability to generate an analog voltage with a Raspberry Pi will allow me to perform some interesting experiments with a voltage controlled crystal oscillator.

Useful Links:

 


     

VHF Frequency Counter with PC Interface

Projects I build often involve frequency synthesis, and one of the most useful tools to have around is a good frequency counter. Being a budding programmer and data analysis guru, I love the idea of being able to access / log / analyze frequency readings on my computer too. Commercial frequency counters can be large, expensive, and their calibration is a chicken-and-egg problem (you need a calibrated frequency counter to calibrate a frequency reference you use to calibrate a frequency counter!). For about the cost of a latte I made a surprisingly good frequency frequency counter (which directly counts >100 MHz without dividing-down the input signal) by blending a SN74LV8154 dual 16-bit counter (which can double as a 32-bit counter, $1.04 on mouser) and an ATMega328 microcontroller ($3.37 on Mouser). Although these two chips are all you need to count something, the accuracy of your counts depend on your gate. If you can generate a signal of 1 pulse per second (1PPS), you can count anything, but your accuracy depends on the accuracy of your 1PPS signal. To eliminate the need for calibration (and to provide the 1PPS signal with the accuracy of an atomic clock) I’m utilizing the 1PPS signal originating from a GPS unit which I already had distributed throughout my shack (using a 74HC240 IC as a line driver). If you don’t have a GPS unit, consider getting one! I’m using a NEO-6M module ($17.66 on Amazon) to generate the 1PPS gate, and if you include its cost we’re up to $22.07. Also, all of the code for this project (schematics, C that runs on the microcontroller, and a Python to interact with the serial port) is shared on GitHub! You may be wondering, “why do GPS units have incredibly accurate 1PPS signals?” It’s a good question, but a subject for another day. For now, trust me when I say they’re fantastically accurate (but slightly less precise due to jitter) if you’re interested in learning more read up on GPS timing.

 

pc frequency counter schem

This is the general idea behind how this frequency counter works. It’s so simple! It’s entirely digital, and needs very few passive components. sn74lv8154 is configured in 32-bit mode (by chaining together its two 16-bit counters, see the datasheet for details) and acts as the front-end directly taking in the measured frequency. This chip is “rare” in the sense I find very few internet projects using it, and they’re not available on ebay. However they’re cheap and plentiful on mouser, so I highly encourage others to look into using it! The datasheet isn’t very clear about its maximum frequency, but in my own tests I was able to measure in excess of 100 MHz from a breadboarded circuit! This utilized two cascaded ICS501 PLL frequency multiplier ICs to multiply a signal I had available (the 11.0592 MHz crystal the MCU was running from) by ten, yielding 110 MHz, which it was able to measure (screenshot is down on the page).

neo-60 gps 1pps

The 1PPS gate signal is generated from an inexpensive GPS module available on AmazonI’ve hinted at the construction of this device before and made a post about how to send output signals like the 1PPS signal generated here throughout your shack via coax using a line driver, so I won’t re-hash all of those details here. I will say that this module has only VCC, GND, and TX/RX pins, so to get access to the 1PPS signal you have to desolder the SMT LED and solder a wire to its pad. It requires a bit of finesse. If you look closely, you can see it in this picture (purple wire).

IMG_8207

I first built this device on a breadboard, and despite the rats nest of wires it worked great! Look closely and you can see the ICS501 frequency multiplier ICs I wrote about before. In this case it’s measuring the 10x multiplied crystal frequency clocking the MCU (11 MHz -> 110 MHz) and reporting these readings every 1 second to the computer via a serial interface.

ss

Frequency measurements of the VHF signal are reported once per second. Measurements are transmitted through a USB serial adapter, and captured by a Python script. Note that I’m calling this signal VHF because it’s >30 MHz. I am unsure if this device will work up to 300 MHz (the border between VHF and UHF), but I look forward to testing that out! Each line contains two numbers: the actual count of the counter (which is configured to simply count continuously and overflow at 2^32=4,294,967,296), and the gated count (calculated by the microcontroller) which is the actual frequency in Hz.

This screenshot shows that my ~11.05 MHz crystal is actually running at 11,061,669.4 Hz. See how I capture the 0.4 Hz unit at the end? That level of precision is the advantage of using this VHF-capable counter in conjunction with a 10x frequency multiplier!

Once I confirmed everything was working, I built this device in a nice enclosure. I definitely splurge every few months and buy extruded split body aluminum enclosures in bulk (ebay), but they’re great to have on hand because they make projects look so nice. I added some rubber feet (cabinet bumpers from Walmart), drilled holes for all the connectors with a continuous step drill bit, made a square hole for the serial port using a nibbler, and the rest is pretty self-evident. Labels are made with a DYMO LetraTag (Target) and clear labels (Target, Amazon) using a style inspired by PA2OHH. I tend to build one-off projects like this dead-bug / Manhattan style.

IMG_8277

IMG_8282

I super-glued a female header to the aluminum frame to make in-circuit serial programming (ICSP) easy. I can’t believe I never thought to do this before! Programming (and reprogramming) was so convenient. I’m going to start doing this with every enclosed project I build from now on. FYI I’m using a USBTiny ISP ($10.99, Amazon) to do the programming (no longer the BusPirate, it’s too slow) like I describe here for 64-bit Windows 7 (although I’m now using Windows 10 and it works the same).

IMG_8330

The front of the device has LEDs indicating power, serial transmission, and gating. Without a 1PPS gate, the device is set to send a count (of 0) every 5 seconds. In this case, the TX light will illuminate. If a gate is detected, the TX and GATE LEDs will illuminate simultaneously. In reality I just drilled 3 holes when I really needed two, so I had to make-up a function for the third LED (d’oh!)

IMG_8286

The back of the device has serial output, frequency input, gate input, and power. Inside is a LM7805 voltage regulator, and careful attention was paid to decoupling and keeping ripple out of the power supply (mostly so our gate input wouldn’t be affected). I’m starting to get in the habit of labeling all serial output ports with the level (TTL vs CMOS, which makes a HUGE difference as MAX232 level converter may be needed, or a USB serial adapter which is capable of reading TTL voltages), as well as the baud rate (119200), byte size (8), parity (N), and stop bit (1). I just realized there’s a typo! The label should read 8N1. I don’t feel like fixing it, so I’ll use a marker to turn the 2 into an 8. I guess I’m only human after all.

IMG_8297

I should have tried connecting all these things before I drilled the holes. I got so lucky that everything fit, with about 2mm to spare between those BNC jacks. Phew!

IMG_8316

This is an easy test frequency source. I have a dozen canned oscillators of various frequencies. This is actually actually a voltage controlled oscillator (VCO) with adjustment pin (not connected), and it won’t be exactly 50 MHz without adjustment. It’s close enough to test with though! As this is >30 MHz, we can call the signal VHF.

IMG_8318

You can see on the screen it’s having no trouble reading the ~50 MHz frequency. You’ll notice I’m using RealTerm (with a good write-up on sparkfun) which is my go-to terminal program instead of HyperTerminal (which really needs to go away forever). In reviewing this photo, I’m appreciating how much unpopulated room I have on the main board. I’m half tempted to build-in a frequency multiplier circuit, and place it under control of the microcontroller such that if an input frequency from 1-20MHz is received, it will engage the 10x multiplier. That’s a mod for another day though! Actually, since those chips are SMT, if I really wanted to do this I would make this whole thing a really small SMT PCB and greatly simplify construction. That sounds like a project for another day though…

IMG_8335

Before closing it up I added some extra ripple protection on the primary counter chip. There’s a 560 uH series inductor with the power supply, followed by a 100 nF capacitor parallel with ground. I also added ferrite beads to the MCU power line and gate input line. I appreciate how the beads are unsecured and that this is a potential weakness in the construction of this device (they’re heavy, so consider what would happen if you shook this enclosure). However, anything that would yank-away cables in the event of shaking the device would probably also break half the other stuff in this thing, so I think it’s on par with the less-than-rugged construction used for all the other components in this device. It will live a peaceful life on my shelf. I am not concerned.

IMG_8340

This is the final device counting frequency and continuously outputting the result to my computer. In the background you can see the 12V power supply (yellow) indicating it is drawing only 20 mA, and also the GPS unit is in a separate enclosure on the bottom right. Click here to peek inside the GPS 1PPS enclosure.

IMG_8344

I’m already loving this new frequency counter! It’s small, light, and nicely enclosed (meaning it’s safe from me screwing with it too much!). I think this will prove to be a valuable piece of test equipment in my shack for years to come. I hope this build log encourages other people to consider building their own equipment. I learned a lot from this build, saved a lot of money not buying something commercial, had a great time making this device, and I have a beautiful piece of custom test equipment that does exactly what I want.

Microcontroller code (AVR-GCC), schematics, and a Python script to interface with the serial port are all available on this project’s GitHub page



Afterthought: Using without GPS

One of the great advantages of this project is that it uses GPS for an extremely accurate 1 PPS signal, but what options exist to adapt this project to not rely on GPS? The GPS unit is expensive (though still <$20) and GPS lock is not always feasible (underground, in a Faraday cage, etc). Barring fancy things like dividing-down rubidium frequency standards or oven controlled oscillators, consider having your microcontroller handle the gating using either interrupts and timers precisely configured to count seconds. Since this project uses a serial port with a 11.0592 MHz crystal, your 1PPS stability will depend on the stability of your oscillator (which is pretty good!). Perhaps more elegantly you could use a 32.768 kHz crystal oscillator to create a 1 PPS signal. This frequency can be divided by 2 over and over to yield 1 Hz perfectly. This is what most modern wristwatches do. Many AVRs have a separate oscillator which can accomodate a 32 kHz crystal and throw interrupts every 1 second without messing with the system clock. Alternatively, the 74GC4060 (a 14 stage ripple counter) can divide 32k into 1 Hz and even can be arranged as an oscillator (check the datasheet). It would be possible to have both options enabled (local clock and GPS) and only engage the local clock if the GPS signal is absent. If anyone likes the idea of this simple VHF frequency counter with PC interface but doesn’t want to bother with the GPS, there are plenty of options to have something almost as accurate. That really would cut the cost of the final device down too, keeping it under the $5 mark.

Update: Integrating Counter Serial Output with GPS Serial Output

The NEO-M8 GPS module is capable of outputting serial data at 9600 baud and continuously dumps NEMA formatted GPS data. While this isn’t really useful for location information (whose frequency counter requires knowing latitude and longitude?) it’s great for tracking things like signal strength, fix quality, and number of satellites. After using this system to automatically log frequency of my frequency reference, I realized that sometimes I’d get 1-2 hours of really odd data (off by kHz, not just a few Hz). Power cycling the GPS receiver fixes the problem, so my guess it that it’s a satellite issue. If I combine the GPS RX and counter in 1 box, I could detect this automatically and have the microcontroller power cycle the GPS receiver (or at the least illuminate a red error LED). I don’t feel like running 2 USB serial adapters continuously. I don’t feel like programming my AVR to listen to the output from the GPS device (although that’s probably the correct way to do things).  Instead I had a simpler idea that worked really well, allowing me to simultaneously log serial data from my GPS unit and microcontroller (frequency counter) using 1 USB serial adapter.

img_8401

The first thing I did was open up the frequency counter and reconnect my microcontroller programmer. This is exactly what I promised myself I wouldn’t do, and why I have a nice enclosure in the first place! Scott, stop fidgeting with things! The last time I screwed this enclosure together I considered adding super glue to the screw threads to make sure I didn’t open it again. I’ll keep my modifications brief! For now, this is a test of a concept. When it’s done, I’ll revert the circuitry to how it was and close it up again. I’ll take what I learn and build it into future projects.

img_8402

I peeked at the serial signals of both the frequency counter (yellow) and the GPS unit output (blue). To my delight, there was enough dead space that I thought I could stick both in the same signal. After a code modification, I was able to tighten it up a lot, so the frequency counter never conflicts with the GPS unit by sending data at the same time.

img_8403

I had to slow the baud rate to 9600, but I programmed it to send fewer characters. This leaves an easy ~50ms padding between my frequency counter signal and the GPS signal. Time to mix the two! This takes a little thought, as I can’t just connect the two wires together. Serial protocol means the lines are usually high, and only pulled down when data is being sent. I had to implement an active circuit.

fullsizerender-2

Using a few components, I built an AND gate to combine signals from the two serial lines. For some reason it took some thought before I realized an AND gate was what I needed here, but it makes sense. The output is high (meaning no serial signal) only when both inputs are high (no serial signals on the input). When either signal drops low, the output drops low. This is perfect. My first thought was that I’d need a NOR gate, but an inverted AND gate is a NOR gate.

img_8404

Here’s my quick and dirty implementation. A reminder again is that this will be removed after this test. For now, it’s good enough.

img_8405

After connecting the GPS serial output and frequency counter serial output to the AND gate (which outputs to the computer), I instantly got the result I wanted!

serial-combine

RealTerm shows that both inputs are being received. It’s a mess though. If you want to know what everything is, read up on NEMA formatted GPS data.

combined-python

I whipped-up a python program to parse, display, and log key information. This display updates every 1 second. The bottom line is what is appended to the log file on ever read. It’s clunky, but again this is just for testing and debugging. I am eager to let this run for as long as I can (days?) so I can track how changes in satellite signal / number / fix quality influence measured frequency.


     

ICS501 Simple Frequency Multiplier

Today I made a high frequency multiplier using a single component: the ICS501 PLL clock multiplier IC. This chip provides 2x, 5x, 8x (and more) clock multiplication using an internal phased-lock loop (PLL). At less than a dollar on eBay$1.55 on mouser, and $0.67 on Digikey, they don’t break the bank and I’m glad I have a few in my junk box! I have a 10MHz frequency standard which I want to use to measure some 1Hz (1pps) pulses with higher precision, so my general idea is to use a frequency multiplier circuit to increase the frequency (to 80 MHz) and use this to run a counter IC to measure the number of clock pulses between the PPS pulses. I spent a lot of time working with the CD4046 micro-power phased lock loop IC which has a phase comparator and a voltage controlled oscillator built in. It seemed this chip was the go-to for many years, but it requires external circuitry (ICs in my case) to divide by N and is intended to adjust a VCO output voltage based on the phase difference of two different inputs. Although I made some great progress using this chip, I found a few SMT ICS501 ICs in my junk box and decided to give them a try. I was impressed how easy it was to use! I just fed it 5V and my clock signal, and it output 8x my clock signal! Since I don’t have my 10MHz reference frequency running at the moment, I tested it with a 1MHz canned oscillator. It worked great, and was so easy! I’ll definitely be using this chip to multiply-up crystal oscillator frequencies to improve the precision of frequency counting.

datasheet

The pin connections are straightforward: +5V and GND to pins 2 and 3, no connection for pins 7 and 8, clock goes in 1 and comes out on 5. Pins 4 and 6 are both set to +5V to yield a x8 multiplier, according to the chart. All of this is in the datasheet for the chip.

IMG_8104

The IC I had on hand was SOIC. I don’t think they make this IC in DIP. Luckily, I have breadboardable breakout boards on hand. These breakout boards are identical to those sold on dipmicro but I got mine from ebay and they’re all over ebay!

IMG_8111

I didn’t feel like changing my soldering iron tip so I gave it a go with a huge wedge, and it worked pretty well! I first melted a little bit of solder on all the rails, waited until it cooled, pressed the IC into the solder, then re-melted it with the iron. It was relatively easy and I had no shorts. I do have a hot air gun (which I also didn’t feel like setting up and waiting for to get warm) but this worked fine…

IMG_8113

Here’s the test circuit. I added a 100nF power decoupling capacitor and a SMT LED (with a 1 kOhm current limiting resistor) so I could tell when it was powered. I am using a 1MHz can oscillator at the input of the ICS501, and capturing both outputs through a 0.1uF capacitors terminating in a 50 ohm loads (at the oscilloscope, seen better in the next photo).

IMG_8121

It worked immediately with no trouble! The top trace is the original 1MHz clock signal, and the bottom is the 8MHz trace.

ics501-demo

The frequency isn’t exactly 1MHz because the adjustment pin of the can oscillator has been left floating. Also, I recognize the power supply is noisy which is also getting noise in the signals. None of that matters, I’m just testing the concept here. The bottom line is that the ICS501 is an extremely easy way to multiply a clock frequency to beyond 100 MHz and it requires no external components! I will definitely be using this IC in some of my future designs. I’m glad I have it! I had to search my email to see when I ordered it because I had no memory of doing so. It looks like I got them in August 2013 (3 years ago!) and never used them. Regardless, I’m happy to have found them in my junk box, and will definitely be using them from now on.

Update: Cascading Two ICS501s for 10x Frequency Multiplication

My ultimate goal is to build a frequency counter using a 10 MHz frequency source, multiplied to a higher value for greater precision. Although I could achieve 8x frequency multiplication with a single ICS501, I didn’t like the idea of frequency steps not being decimal. I decided to try to cascade two ICS501 chips configured to multiply by 2 then by 5 to yield 10. Supposedly this could work on a range of frequencies up through 64x multiplication, but for me generating 100 MHz from a 10 MHz reference is exactly what I need.

IMG_8179

Here’s my design. It’s simple. I configure S0 or S1 as floating, grounded, or high to set the multiplication factor (see the chart above).

IMG_8175

Here’s my implementation. I didn’t have enough space on the breakout board to fit the whole chip (I was missing a single row!). Luckily the SMT perf board is spaced perfectly for SOIC. I was surprised how easy this thing was to solder on the SMT perf board. I’m going to have to buy some more and try prototyping with it. It would be cool to get good at it. That’s another story for another day though…

IMG_8162

The breadboard design got way easier! This thing now just needs power (+5V and GND), an input signal (1 MHz in this demo), and the output signal is 10x the input (10 MHz).

IMG_8161

This is what the output looks like. Signals terminate into a 10 ohm load at the level of the oscilloscope.

SDS00018

I had the USB drive in the thing so I went ahead and pushed the print button. Here’s the actual screen capture.

SDS00005

Here it is converting 10 MHz into 100 MHz. The signals are a bit noisy, likely because both ICs are being powered together (behind the same inductor/capacitor). In a production device, each IC should have its own inductor/capacitor to isolate it from ripple on the power rail. Regardless, this works great in cascading arrangement to multiply HF frequencies to VHF frequencies. The 10MHz source is my oven controlled crystal oscillator (OCXO) which I haven’t written about yet.

IMG_8166

All in all, the ICS501 was an easy cheap single-component solution to frequency multiplication, and cascading two in series easily allows me to multiply the frequency of an incoming signal. I look forward to posting details of my frequency counter build soon!


     

Breadboard Line Driver Module

Sometimes I rapidly want to amplify a signal, but building amplifiers, buffers, and line drivers can be a hassle, especially on a breadboard! It’s important to know how to carefully design build tuned and untuned amplifier circuits, but sometimes you just want to analyze or work with a signal without modifying it by sinking too much current, so being able to rapidly drop in a buffer stage would be a great help. Sometimes I want to buffer a signal so I can analyze it (with an oscilloscope or frequency counter) or use use it (perhaps to drive or gate something), but the signal source is across the room, so I need a beefy amplifier to drive it into coax as I run it across my ceiling while I’m experimenting. A MOSFET voltage follower or a Darlington transistor may do the job, but I have to worry about input conditioning, biasing, output voltage limiting, class A, B, C, D, etc., RF vs DC, copying this circuit multiple times for multiple signals, and before you know it I’m sinking more time into my task than I need to. Line driver chips are one of my go-tos for quickly amplifying digital signals because they’re so fast to drop in a breadboard and they provide a strong output with very high impedance inputs and need no external components. Individual buffer of the integrated chip can be paralleled to multiply their current handling capabilities too. One of the common variants is the 74HC240. I don’t know why it’s so popular (I still find its pinout odd), but because it is popular it is cheap. They’re $0.50 on Mouser.com (perhaps cheaper on ebay) and according to their datasheet they can be run up to 7V to deliver or sink 20mA/pin with a maximum dissipation of 500mW. With propagation, enable, and disable times of tens of nanoseconds, they’re not awful for lower-range radio frequencies (HF RF). This specific chip (somewhat comically at the exclusion of almost all others) has been latched onto by amateur radio operators who use it as an amplifier stage of low power (QRP) Morse code radio transmitters often pushing it to achieve ~1 watt of power output. A quick google reveals thousands of web pages discussing this! This Portuguese site is one of the most thorough. Even if not used as the final amplifier, they’re a convenient intermediate stage along an amplifier chain as they can directly drive FET final stages very well (probably best for class C operation). If you’re interested, definitely check out The Handiman’s Guide to MOSFET “Switched Mode” Amplifiers guide by Paul Harden (no relation). Also his part 2.

Typical configuration for RF or TTL amplification with a 74HC240 buffer / line driver

This is the circuit I commonly build. I have one variant on hand for RF (extremely fast oscillations which are continuously fed into the device and often decoupled through a series capacitor), and one for TTL signals (extremely fast). I find myself paralleling line driver outputs all the time. On a breadboard, this means tons of wires! It becomes repetitive and a pain. I’ve started pre-packaging highly parallel line drivers into little modules which I find really convenient. I have a half dozen of these soldered and ready to go, and I can use them by simply dropping them into a breadboard and applying ground, power (+5V), and input signal, and it amplifies it and returns an output signal. Note that in the “Case 2: RF input” example, the inverted output of the first stage is continuously fed back into the input. This will result in continuous oscillation and undesired output if no input is supplied. In case 2, RF must be continuously applied. The advantage is that the feedback network holds the input near the threshold voltage, so very little voltage swing through the decoupling capacitor is required to generate strong output.

Although I have made this entirely floating, I prefer using copper-clad board. Not only does it aid heat dissipation and provide better mechanical structure, but it also serves as a partial RF shield to minimize noise in the input and output signals. A Dremel with a diamond wheel does a good job at cutting out notches in the copper-clad board.

The best way to replicate this is to look at the picture. It’s surprisingly difficult to get it right just by looking at the datasheet, because when it’s upside down it’s mirror-imaged and very easy to make mistakes. I just connect all inputs and all outputs in parallel, for 7 of 8 gates. For one gate, I connect its output to the parallel inputs. I added some passives (including a ferrite bead and decoupling capacitor on the VCC pin) and it’s good to go.

Here's a closer look at the final amplifier
With only 4 pins (GND, +5V, IN, and OUT) this amplifier is easy to drop in a breadboard

Although I often use it in a breadboard, it’s easy to stick in a project. Since the back side is unpopulated, you can use a dot of super glue and stick it anywhere you want. In this example, I had a GPS receiver module which blinked a LED at exactly one pulse per second (1PPS) [check out why] and I wanted to do some measurements on its output. I couldn’t send this line signal out a coax line because it was so low current (in reality, I didn’t know what it could deliver). This is a perfect use for a buffer / line driver. I glued this board inside my temporary project enclosure (which admittedly looks nicer and more permanent than it’s actually intended to be) and set the output to deliver through 50 Ohm coax. It works beautifully!


     

Realtime Audio Visualization in Python

Python’s “batteries included” nature makes it easy to interact with just about anything… except speakers and a microphone! As of this moment, there still are not standard libraries which which allow cross-platform interfacing with audio devices. There are some pretty convenient third-party modules, but I hope in the future a standard solution will be distributed with python. I appreciate the differences of Linux architectures such as ALSA and OSS, but toss in Windows and MacOS in the mix and it gets to be a huge mess. For Linux, would I even need anything fancy? I can run “cat file.wav > /dev/dsp” from a command prompt to play audio. There are some standard libraries for operating system specific sound (i.e., winsound), but I want something more versatile. The official audio wiki page on the subject lists a small collection of third-party platform-independent libraries. After excluding those which don’t support microphone access (the ultimate goal of all my poking around in this subject), I dove a little deeper into sounddevice and PyAudio. Both of these I installed with pip (i.e., pip install pyaudio)

For a more modern, cleaner, and more complete GUI-based viewer of realtime audio data (and the FFT frequency data), check out my Python Real-time Audio Frequency Monitor project.

I really like the structure and documentation of sounddevice, but I decided to keep developing with PyAudio for now. Sounddevice seemed to take more system resources than PyAudio (in my limited test conditions: Windows 10 with very fast and modern hardware, Python 3), and would audibly “glitch” music as it was being played every time it attached or detached from the microphone stream. I tried streaming, but after about an hour I couldn’t get clean live access to the microphone without glitching audio playback. Furthermore, every few times I ran this script it crashed my python kernel! I very rarely see this happening. iPython complained: “It seems the kernel died unexpectedly. Use ‘Restart kernel’ to continue using this console” and I eventually moved back to PyAudio. For a less “realtime” application, sounddevice might be a great solution. Here’s the minimal case sounddevice script I tested with (that crashed sometimes). If you have a better one to do live high-speed audio capture, let me know!

import sounddevice #pip install sounddevice

for i in range(30): #30 updates in 1 second
    rec = sounddevice.rec(44100/30)
    sounddevice.wait()
    print(rec.shape)

Here’s a simple demo to show how I get realtime microphone audio into numpy arrays using PyAudio. This isn’t really that special. It’s a good starting point though. Note that rather than have the user define a microphone source in the python script (I had a fancy menu system handling this for a while), I allow PyAudio to just look at the operating system’s default input device. This seems like a realistic expectation, and saves time as long as you don’t expect your user to be recording from two different devices at the same time. This script gets some audio from the microphone and shows the values in the console (ten times).

import pyaudio
import numpy as np

CHUNK = 4096 # number of data points to read at a time
RATE = 44100 # time resolution of the recording device (Hz)

p=pyaudio.PyAudio() # start the PyAudio class
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK) #uses default input device

# create a numpy array holding a single read of audio data
for i in range(10): #to it a few times just to see
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    print(data)

# close the stream gracefully
stream.stop_stream()
stream.close()
p.terminate()

01

I tried to push the limit a little bit and see how much useful data I could get from this console window. It turns out that it’s pretty responsive! Here’s a slight modification of the code, made to turn the console window into an impromptu VU meter.

import pyaudio
import numpy as np

CHUNK = 2**11
RATE = 44100

p=pyaudio.PyAudio()
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK)

for i in range(int(10*44100/1024)): #go for a few seconds
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    peak=np.average(np.abs(data))*2
    bars="#"*int(50*peak/2**16)
    print("%04d %05d %s"%(i,peak,bars))

stream.stop_stream()
stream.close()
p.terminate()

The results are pretty good! The advantage here is that no libraries are required except PyAudio. For people interested in doing simple math (peak detection, frequency detection, etc.) this is a perfect starting point. Here’s a quick cellphone video:

I’ve made realtime audio visualization (realtime FFT) scripts with Python before, but 80% of that code was creating a GUI. I want to see data in real time while I’m developing this code, but I really don’t want to mess with GUI programming. I then had a crazy idea. Everyone has a web browser, which is a pretty good GUI… with a Python script to analyze audio and save graphs (a lot of them, quickly) and some JavaScript running in a browser to keep refreshing those graphs, I could get an idea of what the audio stream is doing in something kind of like real time. It was intended to be a hack, but I never expected it to work so well! Check this out…

Here’s the python script to listen to the microphone and generate graphs:

import pyaudio
import numpy as np
import pylab
import time

RATE = 44100
CHUNK = int(RATE/20) # RATE / number of updates per second

def soundplot(stream):
    t1=time.time()
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    pylab.plot(data)
    pylab.title(i)
    pylab.grid()
    pylab.axis([0,len(data),-2**16/2,2**16/2])
    pylab.savefig("03.png",dpi=50)
    pylab.close('all')
    print("took %.02f ms"%((time.time()-t1)*1000))

if __name__=="__main__":
    p=pyaudio.PyAudio()
    stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
                  frames_per_buffer=CHUNK)
    for i in range(int(20*RATE/CHUNK)): #do this for 10 seconds
        soundplot(stream)
    stream.stop_stream()
    stream.close()
    p.terminate()

Here’s the HTML file with JavaScript to keep reloading the image… 

<html>
<script language="javascript">
function RefreshImage(){
document.pic0.src="03.png?a=" + String(Math.random()*99999999);
setTimeout('RefreshImage()',50);
}
</script>
<body onload="RefreshImage()">
<img name="pic0" src="03.png">
</body>
</html>

Here’s the result! I couldn’t believe my eyes. It’s not elegant, but it’s kind of functional!

Why stop there? I went ahead and wrote a microphone listening and processing class which makes this stuff easier. My ultimate goal hasn’t been revealed yet, but I’m sure it’ll be clear in a few weeks. Let’s just say there’s a lot of use in me visualizing streams of continuous data. Anyway, this class is the truly terrible attempt at a word pun by merging the words “SWH”, “ear”, and “Hear”, into the official title “SWHear” which seems to be unique on Google. This class is minimal case, but can be easily modified to implement threaded recording (which won’t cause the rest of the functions to hang) as well as mathematical manipulation of data, such as FFT. With the same HTML file as used above, here’s the new python script and some video of the output:

import pyaudio
import time
import pylab
import numpy as np

class SWHear(object):
    """
    The SWHear class is made to provide access to continuously recorded
    (and mathematically processed) microphone data.
    """

    def __init__(self,device=None,startStreaming=True):
        """fire up the SWHear class."""
        print(" -- initializing SWHear")

        self.chunk = 4096 # number of data points to read at a time
        self.rate = 44100 # time resolution of the recording device (Hz)

        # for tape recording (continuous "tape" of recent audio)
        self.tapeLength=2 #seconds
        self.tape=np.empty(self.rate*self.tapeLength)*np.nan

        self.p=pyaudio.PyAudio() # start the PyAudio class
        if startStreaming:
            self.stream_start()

    ### LOWEST LEVEL AUDIO ACCESS
    # pure access to microphone and stream operations
    # keep math, plotting, FFT, etc out of here.

    def stream_read(self):
        """return values for a single chunk"""
        data = np.fromstring(self.stream.read(self.chunk),dtype=np.int16)
        #print(data)
        return data

    def stream_start(self):
        """connect to the audio device and start a stream"""
        print(" -- stream started")
        self.stream=self.p.open(format=pyaudio.paInt16,channels=1,
                                rate=self.rate,input=True,
                                frames_per_buffer=self.chunk)

    def stream_stop(self):
        """close the stream but keep the PyAudio instance alive."""
        if 'stream' in locals():
            self.stream.stop_stream()
            self.stream.close()
        print(" -- stream CLOSED")

    def close(self):
        """gently detach from things."""
        self.stream_stop()
        self.p.terminate()

    ### TAPE METHODS
    # tape is like a circular magnetic ribbon of tape that's continously
    # recorded and recorded over in a loop. self.tape contains this data.
    # the newest data is always at the end. Don't modify data on the type,
    # but rather do math on it (like FFT) as you read from it.

    def tape_add(self):
        """add a single chunk to the tape."""
        self.tape[:-self.chunk]=self.tape[self.chunk:]
        self.tape[-self.chunk:]=self.stream_read()

    def tape_flush(self):
        """completely fill tape with new data."""
        readsInTape=int(self.rate*self.tapeLength/self.chunk)
        print(" -- flushing %d s tape with %dx%.2f ms reads"%\
                  (self.tapeLength,readsInTape,self.chunk/self.rate))
        for i in range(readsInTape):
            self.tape_add()

    def tape_forever(self,plotSec=.25):
        t1=0
        try:
            while True:
                self.tape_add()
                if (time.time()-t1)>plotSec:
                    t1=time.time()
                    self.tape_plot()
        except:
            print(" ~~ exception (keyboard?)")
            return

    def tape_plot(self,saveAs="03.png"):
        """plot what's in the tape."""
        pylab.plot(np.arange(len(self.tape))/self.rate,self.tape)
        pylab.axis([0,self.tapeLength,-2**16/2,2**16/2])
        if saveAs:
            t1=time.time()
            pylab.savefig(saveAs,dpi=50)
            print("plotting saving took %.02f ms"%((time.time()-t1)*1000))
        else:
            pylab.show()
            print() #good for IPython
        pylab.close('all')

if __name__=="__main__":
    ear=SWHear()
    ear.tape_forever()
    ear.close()
    print("DONE")

I don’t really intend anyone to actually do this, but it’s a cool alternative to recording a small portion of audio, plotting it in a pop-up matplotlib window, and waiting for the user to close it to record a new fraction. I had a lot more text in here demonstrating real-time FFT, but I’d rather consolidate everything FFT related into a single post. For now, I’m happy pursuing microphone-related python projects with PyAudio.

UPDATE: Displaying a single frequency

Use Numpy’s FFT() and FFTFREQ() to turn the linear data into frequency. Set that target and grab the FFT value corresponding to that frequency. I haven’t tested this to be sure it’s working, but it should at least be close…

import pyaudio
import numpy as np
np.set_printoptions(suppress=True) # don't use scientific notation

CHUNK = 4096 # number of data points to read at a time
RATE = 44100 # time resolution of the recording device (Hz)
TARGET = 2100 # show only this one frequency

p=pyaudio.PyAudio() # start the PyAudio class
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK) #uses default input device

# create a numpy array holding a single read of audio data
for i in range(10): #to it a few times just to see
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    fft = abs(np.fft.fft(data).real)
    fft = fft[:int(len(fft)/2)] # keep only first half
    freq = np.fft.fftfreq(CHUNK,1.0/RATE)
    freq = freq[:int(len(freq)/2)] # keep only first half
    assert freq[-1]>TARGET, "ERROR: increase chunk size"
    val = fft[np.where(freq>TARGET)[0][0]]
    print(val)

# close the stream gracefully
stream.stop_stream()
stream.close()
p.terminate()

UPDATE: Display peak frequency

If your goal is to determine which frequency is producing the loudest tone, use this function. I also added a few lines to graph the output in case you want to observe how it operates. I recommend testing this script with a tone generator, or a YouTube video containing tones of a range of frequencies like this one.

import pyaudio
import numpy as np
import matplotlib.pyplot as plt

np.set_printoptions(suppress=True) # don't use scientific notation

CHUNK = 4096 # number of data points to read at a time
RATE = 44100 # time resolution of the recording device (Hz)

p=pyaudio.PyAudio() # start the PyAudio class
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK) #uses default input device

# create a numpy array holding a single read of audio data
for i in range(10): #to it a few times just to see
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    data = data * np.hanning(len(data)) # smooth the FFT by windowing data
    fft = abs(np.fft.fft(data).real)
    fft = fft[:int(len(fft)/2)] # keep only first half
    freq = np.fft.fftfreq(CHUNK,1.0/RATE)
    freq = freq[:int(len(freq)/2)] # keep only first half
    freqPeak = freq[np.where(fft==np.max(fft))[0][0]]+1
    print("peak frequency: %d Hz"%freqPeak)

    # uncomment this if you want to see what the freq vs FFT looks like
    #plt.plot(freq,fft)
    #plt.axis([0,4000,None,None])
    #plt.show()
    #plt.close()

# close the stream gracefully
stream.stop_stream()
stream.close()
p.terminate()

 

This program shows left vs right audio level:

import pyaudio
import numpy as np

maxValue = 2**16
p=pyaudio.PyAudio()
stream=p.open(format=pyaudio.paInt16,channels=2,rate=44100,
              input=True, frames_per_buffer=1024)
while True:
    data = np.fromstring(stream.read(1024),dtype=np.int16)
    dataL = data[0::2]
    dataR = data[1::2]
    peakL = np.abs(np.max(dataL)-np.min(dataL))/maxValue
    peakR = np.abs(np.max(dataR)-np.min(dataR))/maxValue
    print("L:%00.02f R:%00.02f"%(peakL*100, peakR*100))

Output

L:47.26 R:45.17
L:47.55 R:45.63
L:49.44 R:45.98
L:45.27 R:49.80
L:44.39 R:45.75
L:47.50 R:46.96
L:41.49 R:42.64
L:42.95 R:41.39
L:49.56 R:49.62
L:48.29 R:48.80
L:45.03 R:47.62
L:47.99 R:49.35
L:41.58 R:49.21

Or with a tweak…

import pyaudio
import numpy as np

maxValue = 2**16
bars = 35
p=pyaudio.PyAudio()
stream=p.open(format=pyaudio.paInt16,channels=2,rate=44100,
              input=True, frames_per_buffer=1024)
while True:
    data = np.fromstring(stream.read(1024),dtype=np.int16)
    dataL = data[0::2]
    dataR = data[1::2]
    peakL = np.abs(np.max(dataL)-np.min(dataL))/maxValue
    peakR = np.abs(np.max(dataR)-np.min(dataR))/maxValue
    lString = "#"*int(peakL*bars)+"-"*int(bars-peakL*bars)
    rString = "#"*int(peakR*bars)+"-"*int(bars-peakR*bars)
    print("L=[%s]\tR=[%s]"%(lString, rString))

graphical output: