Logging I2C Data with Bus Pirate and Python

I’m working on a project which requires I measure temperature via a computer, and I accomplished this with minimal complexity using a BusPirate and LM75A I2C temperature sensor. I already had some LM75A breakout boards I got on eBay (from China) a while back. A current eBay search reveals these boards are a couple dollars with free shipping. The IC itself is available on Mouser for $0.61 each. The LM75A datasheet reveals it can be powered from 2.8V-5.5V and has a resolution of 1/8 ºC (about 1/4 ºF). I attached the device to the Bus Pirate according to the Bus Pirate I/O Pin Descriptions page (SCL->CLOCK and SDA->MOSI) and started interacting with it according to the Bus Pirate I2C page. Since Phillips developed the I2C protocol, a lot of manufacturers avoid legal trouble and call it TWI (two-wire interface).

Here I show how to pull data from this I2C device directly via a serial terminal, then show my way of automating the process with Python. Note that there are multiple python packages out there that claim to make this easy, but in my experience they are either OS-specific or no longer supported or too confusing to figure out rapidly. For these reasons, I ended up just writing a script that uses common Python libraries so nothing special has to be installed.

Reading data directly from a serial terminal

Before automating anything, I figured out what I2C address this chip was using and got some sample temperature readings directly from the serial terminal. I used RealTerm to connect to the Bus Pirate. The sequence of keystrokes I used are:

  • # – to reset the device
  • m – to enter the mode selection screen
    • 4 – to select I2C mode
    • 3 – to select 100KHz
  • W – to turn the power on
  • P – to enable pull-up resistors
  • (1) – to scan I2C devices
    • this showed the device listening on 0x91
  • [0x91 r:2] – to read 2 bytes from I2C address 0x91
    • this showed bytes like 0x1D and 0x20
    • 0x1D20 in decimal is 7456
    • according to datasheet, must divide by 2^8 (256)
    • 7456/256 = 29.125 C = 84.425 F

Automating Temperature Reads with Python

There should be an easy way to capture this data from Python. The Bus Pirate website even has a page showing how to read data from LM75, but it uses a pyBusPirateLite python package which has to be manually installed (it doesn’t seem to be listed in pypi). Furthermore, they only have a screenshot of a partial code example (nothing I can copy or paste) and their link to the original article is broken. I found a cool pypy-indexed python module pyElectronics which should allow easy reading/writing from I2C devices via BusPirate and Raspberry Pi. However, it crashed immediately on my windows system due to attempting to load Linux-only python modules. I improved the code and issued a pull request, but I can’t encourage use of this package at this time if you intend to log data in Windows. Therefore, I’m keeping it simple and using a self-contained script to interact with the Bus Pirate, make temperature reads, and graph the data over time. You can code-in more advanced features as needed. The graphical output of my script shows what happens when I breathe on the sensor (raising the temperature), then what happens when I cool it (by placing a TV dinner on top of it for a minute). Below is the code used to set up the Bus Pirate to log and graph temperature data. It’s not fast, but for temperature readings it doesn’t have to be! It captures about 10 reads a second, and the rate-limiting step is the timeout value which is currently set to 0.1 sec.

NOTE: The Bus Pirate has a convenient binary scripting mode which can speed all this up. I’m not using that mode in this script, simply because I’m trying to closely mirror the functionality of directly typing things into the serial console.

import serial
import matplotlib.pyplot as plt

BUSPIRATE_PORT = 'com3' #customize this! Find it in device manager.

def send(ser,cmd,silent=False):
    """
    send the command and listen to the response.
    returns a list of the returned lines. 
    The first item is always the command sent.
    """
    ser.write(str(cmd+'\n').encode('ascii')) # send our command
    lines=[]
    for line in ser.readlines(): # while there's a response
        lines.append(line.decode('utf-8').strip())
    if not silent:
        print("\n".join(lines))
        print('-'*60)
    return lines

def getTemp(ser,address='0x91',silent=True,fahrenheit=False):
    """return the temperature read from an LM75"""
    unit=" F" if fahrenheit else " C"
    lines=send(ser,'[%s r:2]'%address,silent=silent) # read two bytes
    for line in lines:
        if line.startswith("READ:"):
            line=line.split(" ",1)[1].replace("ACK",'')
            while "  " in line:
                line=" "+line.strip().replace("  "," ")
            line=line.split(" 0x")
            val=int("".join(line),16)
            # conversion to C according to the datasheet
            if val < 2**15:
                val = val/2**8
            else:
                val =  (val-2**16)/2**8
            if fahrenheit:
                val=val*9/5+32
            print("%.03f"%val+unit)
            return val
    
    
# the speed of sequential commands is determined by this timeout
ser=serial.Serial(BUSPIRATE_PORT, 115200, timeout=.1)

# have a clean starting point
send(ser,'#',silent=True) # reset bus pirate (slow, maybe not needed)
#send(ser,'v') # show current voltages

# set mode to I2C
send(ser,'m',silent=True) # change mode (goal is to get away from HiZ)
send(ser,'4',silent=True) # mode 4 is I2C
send(ser,'3',silent=True) # 100KHz
send(ser,'W',silent=True) # turn power supply to ON. Lowercase w for OFF.
send(ser,'P',silent=True) # enable pull-up resistors
send(ser,'(1)') # scan I2C devices. Returns "0x90(0x48 W) 0x91(0x48 R)"

data=[]
try:
    print("reading data until CTRL+C is pressed...")
    while True:
        data.append(getTemp(ser,fahrenheit=True))
except:
    print("exception broke continuous reading.")
    print("read %d data points"%len(data))

ser.close() # disconnect so we can access it from another app

plt.figure(figsize=(6,4))
plt.grid()
plt.plot(data,'.-',alpha=.5)
plt.title("LM75 data from Bus Pirate")
plt.ylabel("temperature")
plt.xlabel("number of reads")
plt.show()

print("disconnected!") # let the user know we're done.

Experiment: Measuring Heater Efficacy

This project now now ready for an actual application test. I made a simple heater circuit which could be driven by an analog input, PWM, or digital ON/OFF. Powered from 12V it can pass 80 mA to produce up to 1W of heat. This may dissipate up to 250 mW of heat in the transistor if partially driven, so keep this in mind if an analog signal drive is used (i.e., thermistor / op-amp circuit). Anyhow, I soldered this up with SMT components on a copper-clad PCB with slots drilled on it and decided to give it a go. It’s screwed tightly to the temperature sensor board, but nothing special was done to ensure additional thermal conductivity. This is a pretty crude test.

I ran an experiment to compare open-air heating/cooling vs. igloo conditions, as well as low vs. high heater drive conditions. The graph below shows these results. The “heating” ranges are indicated by shaded areas. The exposed condition is when the device is sitting on the desk without any insulation. A 47k resistor is used to drive the base of the transistor (producing less than maximal heating). I then repeated the same thing after the device was moved inside the igloo. I killed the heater power when it reached the same peak temperature as the first time, noticing that it took less time to reach this temperature. Finally, I used a 1k resistor on the base of the transistor and got near-peak heating power (about 1W). This resulted in faster heating and a higher maximum temperature. If I clean this enclosure up a bit, this will be a nice way to test software-based PID temperature control with slow PWM driving the base of the transistor.

Code to create file logging (csv data with timestamps and temperatures) and produce plots lives in the ‘file logging’ folder of the Bus Pirate LM75A project on the GitHub page.

Experiment: Challenging LM7805 Thermal Shutdown

The ubiquitous LM7805 linear voltage regulator offers internal current limiting (1.5A) and thermal shutdown. I’ve wondered for a long time if I could use this element as a heater. It’s TO-220 package is quite convenient to mount onto enclosures. To see how quickly it heats up and what temperature it rests at, screwed a LM7805 directly to the LM75A breakout board (with a dab of thermal compound). I soldered the output pin to ground (!!!) and recorded temperature while it was plugged in.

Power (12V) was applied to the LM7805 over the red-shaded region. It looks like it took about 2 minutes to reach maximum temperature, and settled around 225F. After disconnecting power, it settled back to room temperature after about 5 minutes. I’m curious if this type of power dissipation is sustainable long term…

Update: Reading LM75A values directly into an AVR

This topic probably doesn’t belong inside this post, but it didn’t fit anywhere else and I don’t want to make it its own post. Now that I have this I2C sensor mounted where I want it, I want a microcontroller to read its value and send it (along with some other data) via serial USART to an FT232 (USB serial adapter). Ultimately I want to take advantage of its comparator thermostat function so I can have a USB-interfaced PC-controllable heater with multiple LM75A ICs providing temperature readings at different sites in my project. To do this, I had to write code to interface my microcontroller to the LM75A. I am using an ATMega328 (ATMega328P) with AVR-GCC (not Arduino). Although there are multiple LM75A senor libraries for Arduino [link] [link] [link] I couldn’t find any examples which didn’t rely on Arduino libraries. I ended up writing functions around g4lvanix’s L2C-master-lib.

Here’s a relevant code snippit. See the full code (with compile notes) on this GitHub page:

uint8_t data[2]; // prepare variable to hold sensor data
uint8_t address=0x91; // this is the i2c address of the sensor
i2c_receive(address,data,2); // read and store two bytes
temperature=(data[0]*256+data[1])/32; // convert two bytes to temperature

 

 

This project lives on my growing GitHub page for microcontroller projects:

https://github.com/swharden/AVR-projects/

 


     

Raspberry Pi RF Frequency Counter

I build a lot of RF circuits, and often it’s convenient to measure and log frequency with a computer. Previously I’ve built standalone frequency counters, frequency counters with a PC interface, and even hacked a classic frequency counter to add USB interface (twice, actually). My latest device uses only 2 microchips to provide a Raspberry Pi with RF frequency measurement capabilities. The RF signal clocks a 32-bit counter SN74LV8154 ($1.04 on Mouser) connected to a 16-bit IO expander MCP23017 ($1.26 on Mouser) accessable to the Raspberry Pi (via I²C) to provide real-time frequency measurements from a python script for $2.30 in components! Well, plus the cost of the Raspberry Pi. All files for this project are on my GitHub page.

img_8773

The entire circuit is only two microchips! I have a few passives to clean up the RF signal (the RF input is loaded with a 1k resistor to ground, decoupled through a series 100 nF capacitor, and balanced at VCC/2 through a voltage divider of two 47k resistors), but if the measured signal is already a strong square wave they could be omitted. The circuit requires a gate pulse which typically will be 1 pulse per second (1PPS) and can be generated by dividing-down a 32.768kHz oscillator, a spare pin on a microcontroller, a fancy 1PPS time reference, or like in my case a GPS module (Neo-6M) with 1PPS output to provide an extremely accurate gate.

schem

The connections are intuitive! The I2C address is 0x20 when A0, A1, and A2 are grounded. GPB(1-4) control the register select of the counter, and GPA(0-7) reads each bit of the selected register. The whole thing is controlled from Python, but could be trivially written in any language.

img_8777

Here’s a quick summary describing how the code works: First I send bytes to address 0 and 1 to set all pins of GPIO A as inputs, and GPIO B as outputs. Note that only 4 of 8 pins are used for the output, so technically 4 extra pins could be used for things like blinking LEDs or controlling other devices. I then set the register select pins by sending a value to 0x13 (GPIO B), and read the entire GPIO A bus (INTCAPB, 0x18). For address details, consult the datasheet. I do this 4 times (1 for each byte of the 32-bit counter), do a little math to turn it into a frequency value, and compare the current value with the last value and take the difference to display as the measured frequency.

screenshot

An advantage of this continuously running mode is that no clock cycles are lost, so a gate which accidentally fires a bit early due to jitter and cuts-off a cycle will compensate for it on a subsequent read. This is shown above, as a very stable 10MHz frequency reference is measured with this method. A “slow” 1PPS clock tick causes a reading slightly higher, compensated-for by the next reading being slightly lower. In this way, clock sources which are extremely accurate but suffer from low precision (like GPS time sources) are able to maximize the long-term measurement of frequency. Combining this frequency measurement technique with the ability to generate an analog voltage with a Raspberry Pi will allow me to perform some interesting experiments with a voltage controlled crystal oscillator.

Useful Links:

 


     

Generating Analog Voltage with Raspberry Pi

I recently had the need to generate analog voltages from the Raspberry PI, which has rich GPIO digital outputs but no analog outputs. I looked into the RPi.GPIO project which can create PWM (which I wanted to smooth using a low pass filter to create the analog voltage), but its output on the oscilloscope looked terrible! It stuttered all over the place, likely because the duty is continuously under software control. I ended up solving my problem with a MCP4921 12-bit DAC chip (about $1.50 on eBay). It’s controlled via SPI, and although I could have written a python program to bit-bang its protocol with RPi.GPIO I realized I could write directly to the Raspberry Pi SPI device using the echo command. Dividing 3.3V into 12-bits (4096) means that I can control voltage in steps of less than 1mV each, right from the bash console!

img_8696

Video: The Problem (RPi PWM jitters)

Video: My Solution (SPI DAC)

Hardware Connection

There’s very little magic in how the microchip is connected to the Pi. It’s a straight shot to its SPI bus! Here’s a quick drawing showing which pins to connect. Check your device against the Raspberry Pi GPIO pinout diagram for different devices.

img_8701-1

Controlling the DAC with a Bus Pirate

Before I used a Raspberry Pi to control the DAC chip, I tested it out with a Bus Pirate. I don’t have a lot of pictures of the project, but I have a screenshot of a serial console used to send commands to the chip. One advantage of the Bus Pirate is that I can type bytes in binary, which helps to see the individual bits. I don’t have this ability when I’m working in the bash console.

serial

I’m less familiar with the Bus Pirate, but this was a good opportunity to get to know it a little better. It look me a long time (requiring I pull out the logic analyzer) to realize that I had to manually enable/disable the chip-select line, using the “[” and “]” commands. When I set up the SPI mode (command m5) I told it to use active low, but I wasn’t sure how to reverse the active level of the chip-select commands, so I just did ]this[ instead of [this] and it worked great.

frompi

This is the signal probed when it was controlled by the Raspberry Pi, but it looked essentially identical when values were sent via the Bus Pirate. The only difference is there was an appreciable delay between the “]” commands and each of the bytes. It worked fine though.

Controlling the DAC with Console Commands

Once the hardware was configured, the software was trivial. I could control analog voltages by sending two properly-formatted bytes to the SPI hardware device. Importantly, you must use raspi-config to enable SPI.

# set analog voltage to minimum value (about 0V)
echo -ne "\x30\x00" > /dev/spidev0.0 # minimum

# set analog voltage to something a little higher
echo -ne "\x30\xAB" > /dev/spidev0.0 

# set analog voltage to maximum value (about 3.3V)
echo -ne "\x3F\xFF" > /dev/spidev0.0

Helpful Links:

 


     

VHF Frequency Counter with PC Interface

Projects I build often involve frequency synthesis, and one of the most useful tools to have around is a good frequency counter. Being a budding programmer and data analysis guru, I love the idea of being able to access / log / analyze frequency readings on my computer too. Commercial frequency counters can be large, expensive, and their calibration is a chicken-and-egg problem (you need a calibrated frequency counter to calibrate a frequency reference you use to calibrate a frequency counter!). For about the cost of a latte I made a surprisingly good frequency frequency counter (which directly counts >100 MHz without dividing-down the input signal) by blending a SN74LV8154 dual 16-bit counter (which can double as a 32-bit counter, $1.04 on mouser) and an ATMega328 microcontroller ($3.37 on Mouser). Although these two chips are all you need to count something, the accuracy of your counts depend on your gate. If you can generate a signal of 1 pulse per second (1PPS), you can count anything, but your accuracy depends on the accuracy of your 1PPS signal. To eliminate the need for calibration (and to provide the 1PPS signal with the accuracy of an atomic clock) I’m utilizing the 1PPS signal originating from a GPS unit which I already had distributed throughout my shack (using a 74HC240 IC as a line driver). If you don’t have a GPS unit, consider getting one! I’m using a NEO-6M module ($17.66 on Amazon) to generate the 1PPS gate, and if you include its cost we’re up to $22.07. Also, all of the code for this project (schematics, C that runs on the microcontroller, and a Python to interact with the serial port) is shared on GitHub! You may be wondering, “why do GPS units have incredibly accurate 1PPS signals?” It’s a good question, but a subject for another day. For now, trust me when I say they’re fantastically accurate (but slightly less precise due to jitter) if you’re interested in learning more read up on GPS timing.

 

pc frequency counter schem

This is the general idea behind how this frequency counter works. It’s so simple! It’s entirely digital, and needs very few passive components. sn74lv8154 is configured in 32-bit mode (by chaining together its two 16-bit counters, see the datasheet for details) and acts as the front-end directly taking in the measured frequency. This chip is “rare” in the sense I find very few internet projects using it, and they’re not available on ebay. However they’re cheap and plentiful on mouser, so I highly encourage others to look into using it! The datasheet isn’t very clear about its maximum frequency, but in my own tests I was able to measure in excess of 100 MHz from a breadboarded circuit! This utilized two cascaded ICS501 PLL frequency multiplier ICs to multiply a signal I had available (the 11.0592 MHz crystal the MCU was running from) by ten, yielding 110 MHz, which it was able to measure (screenshot is down on the page).

neo-60 gps 1pps

The 1PPS gate signal is generated from an inexpensive GPS module available on AmazonI’ve hinted at the construction of this device before and made a post about how to send output signals like the 1PPS signal generated here throughout your shack via coax using a line driver, so I won’t re-hash all of those details here. I will say that this module has only VCC, GND, and TX/RX pins, so to get access to the 1PPS signal you have to desolder the SMT LED and solder a wire to its pad. It requires a bit of finesse. If you look closely, you can see it in this picture (purple wire).

IMG_8207

I first built this device on a breadboard, and despite the rats nest of wires it worked great! Look closely and you can see the ICS501 frequency multiplier ICs I wrote about before. In this case it’s measuring the 10x multiplied crystal frequency clocking the MCU (11 MHz -> 110 MHz) and reporting these readings every 1 second to the computer via a serial interface.

ss

Frequency measurements of the VHF signal are reported once per second. Measurements are transmitted through a USB serial adapter, and captured by a Python script. Note that I’m calling this signal VHF because it’s >30 MHz. I am unsure if this device will work up to 300 MHz (the border between VHF and UHF), but I look forward to testing that out! Each line contains two numbers: the actual count of the counter (which is configured to simply count continuously and overflow at 2^32=4,294,967,296), and the gated count (calculated by the microcontroller) which is the actual frequency in Hz.

This screenshot shows that my ~11.05 MHz crystal is actually running at 11,061,669.4 Hz. See how I capture the 0.4 Hz unit at the end? That level of precision is the advantage of using this VHF-capable counter in conjunction with a 10x frequency multiplier!

Once I confirmed everything was working, I built this device in a nice enclosure. I definitely splurge every few months and buy extruded split body aluminum enclosures in bulk (ebay), but they’re great to have on hand because they make projects look so nice. I added some rubber feet (cabinet bumpers from Walmart), drilled holes for all the connectors with a continuous step drill bit, made a square hole for the serial port using a nibbler, and the rest is pretty self-evident. Labels are made with a DYMO LetraTag (Target) and clear labels (Target, Amazon) using a style inspired by PA2OHH. I tend to build one-off projects like this dead-bug / Manhattan style.

IMG_8277

IMG_8282

I super-glued a female header to the aluminum frame to make in-circuit serial programming (ICSP) easy. I can’t believe I never thought to do this before! Programming (and reprogramming) was so convenient. I’m going to start doing this with every enclosed project I build from now on. FYI I’m using a USBTiny ISP ($10.99, Amazon) to do the programming (no longer the BusPirate, it’s too slow) like I describe here for 64-bit Windows 7 (although I’m now using Windows 10 and it works the same).

IMG_8330

The front of the device has LEDs indicating power, serial transmission, and gating. Without a 1PPS gate, the device is set to send a count (of 0) every 5 seconds. In this case, the TX light will illuminate. If a gate is detected, the TX and GATE LEDs will illuminate simultaneously. In reality I just drilled 3 holes when I really needed two, so I had to make-up a function for the third LED (d’oh!)

IMG_8286

The back of the device has serial output, frequency input, gate input, and power. Inside is a LM7805 voltage regulator, and careful attention was paid to decoupling and keeping ripple out of the power supply (mostly so our gate input wouldn’t be affected). I’m starting to get in the habit of labeling all serial output ports with the level (TTL vs CMOS, which makes a HUGE difference as MAX232 level converter may be needed, or a USB serial adapter which is capable of reading TTL voltages), as well as the baud rate (119200), byte size (8), parity (N), and stop bit (1). I just realized there’s a typo! The label should read 8N1. I don’t feel like fixing it, so I’ll use a marker to turn the 2 into an 8. I guess I’m only human after all.

IMG_8297

I should have tried connecting all these things before I drilled the holes. I got so lucky that everything fit, with about 2mm to spare between those BNC jacks. Phew!

IMG_8316

This is an easy test frequency source. I have a dozen canned oscillators of various frequencies. This is actually actually a voltage controlled oscillator (VCO) with adjustment pin (not connected), and it won’t be exactly 50 MHz without adjustment. It’s close enough to test with though! As this is >30 MHz, we can call the signal VHF.

IMG_8318

You can see on the screen it’s having no trouble reading the ~50 MHz frequency. You’ll notice I’m using RealTerm (with a good write-up on sparkfun) which is my go-to terminal program instead of HyperTerminal (which really needs to go away forever). In reviewing this photo, I’m appreciating how much unpopulated room I have on the main board. I’m half tempted to build-in a frequency multiplier circuit, and place it under control of the microcontroller such that if an input frequency from 1-20MHz is received, it will engage the 10x multiplier. That’s a mod for another day though! Actually, since those chips are SMT, if I really wanted to do this I would make this whole thing a really small SMT PCB and greatly simplify construction. That sounds like a project for another day though…

IMG_8335

Before closing it up I added some extra ripple protection on the primary counter chip. There’s a 560 uH series inductor with the power supply, followed by a 100 nF capacitor parallel with ground. I also added ferrite beads to the MCU power line and gate input line. I appreciate how the beads are unsecured and that this is a potential weakness in the construction of this device (they’re heavy, so consider what would happen if you shook this enclosure). However, anything that would yank-away cables in the event of shaking the device would probably also break half the other stuff in this thing, so I think it’s on par with the less-than-rugged construction used for all the other components in this device. It will live a peaceful life on my shelf. I am not concerned.

IMG_8340

This is the final device counting frequency and continuously outputting the result to my computer. In the background you can see the 12V power supply (yellow) indicating it is drawing only 20 mA, and also the GPS unit is in a separate enclosure on the bottom right. Click here to peek inside the GPS 1PPS enclosure.

IMG_8344

I’m already loving this new frequency counter! It’s small, light, and nicely enclosed (meaning it’s safe from me screwing with it too much!). I think this will prove to be a valuable piece of test equipment in my shack for years to come. I hope this build log encourages other people to consider building their own equipment. I learned a lot from this build, saved a lot of money not buying something commercial, had a great time making this device, and I have a beautiful piece of custom test equipment that does exactly what I want.

Microcontroller code (AVR-GCC), schematics, and a Python script to interface with the serial port are all available on this project’s GitHub page



Afterthought: Using without GPS

One of the great advantages of this project is that it uses GPS for an extremely accurate 1 PPS signal, but what options exist to adapt this project to not rely on GPS? The GPS unit is expensive (though still <$20) and GPS lock is not always feasible (underground, in a Faraday cage, etc). Barring fancy things like dividing-down rubidium frequency standards or oven controlled oscillators, consider having your microcontroller handle the gating using either interrupts and timers precisely configured to count seconds. Since this project uses a serial port with a 11.0592 MHz crystal, your 1PPS stability will depend on the stability of your oscillator (which is pretty good!). Perhaps more elegantly you could use a 32.768 kHz crystal oscillator to create a 1 PPS signal. This frequency can be divided by 2 over and over to yield 1 Hz perfectly. This is what most modern wristwatches do. Many AVRs have a separate oscillator which can accomodate a 32 kHz crystal and throw interrupts every 1 second without messing with the system clock. Alternatively, the 74GC4060 (a 14 stage ripple counter) can divide 32k into 1 Hz and even can be arranged as an oscillator (check the datasheet). It would be possible to have both options enabled (local clock and GPS) and only engage the local clock if the GPS signal is absent. If anyone likes the idea of this simple VHF frequency counter with PC interface but doesn’t want to bother with the GPS, there are plenty of options to have something almost as accurate. That really would cut the cost of the final device down too, keeping it under the $5 mark.

Update: Integrating Counter Serial Output with GPS Serial Output

The NEO-M8 GPS module is capable of outputting serial data at 9600 baud and continuously dumps NEMA formatted GPS data. While this isn’t really useful for location information (whose frequency counter requires knowing latitude and longitude?) it’s great for tracking things like signal strength, fix quality, and number of satellites. After using this system to automatically log frequency of my frequency reference, I realized that sometimes I’d get 1-2 hours of really odd data (off by kHz, not just a few Hz). Power cycling the GPS receiver fixes the problem, so my guess it that it’s a satellite issue. If I combine the GPS RX and counter in 1 box, I could detect this automatically and have the microcontroller power cycle the GPS receiver (or at the least illuminate a red error LED). I don’t feel like running 2 USB serial adapters continuously. I don’t feel like programming my AVR to listen to the output from the GPS device (although that’s probably the correct way to do things).  Instead I had a simpler idea that worked really well, allowing me to simultaneously log serial data from my GPS unit and microcontroller (frequency counter) using 1 USB serial adapter.

img_8401

The first thing I did was open up the frequency counter and reconnect my microcontroller programmer. This is exactly what I promised myself I wouldn’t do, and why I have a nice enclosure in the first place! Scott, stop fidgeting with things! The last time I screwed this enclosure together I considered adding super glue to the screw threads to make sure I didn’t open it again. I’ll keep my modifications brief! For now, this is a test of a concept. When it’s done, I’ll revert the circuitry to how it was and close it up again. I’ll take what I learn and build it into future projects.

img_8402

I peeked at the serial signals of both the frequency counter (yellow) and the GPS unit output (blue). To my delight, there was enough dead space that I thought I could stick both in the same signal. After a code modification, I was able to tighten it up a lot, so the frequency counter never conflicts with the GPS unit by sending data at the same time.

img_8403

I had to slow the baud rate to 9600, but I programmed it to send fewer characters. This leaves an easy ~50ms padding between my frequency counter signal and the GPS signal. Time to mix the two! This takes a little thought, as I can’t just connect the two wires together. Serial protocol means the lines are usually high, and only pulled down when data is being sent. I had to implement an active circuit.

fullsizerender-2

Using a few components, I built an AND gate to combine signals from the two serial lines. For some reason it took some thought before I realized an AND gate was what I needed here, but it makes sense. The output is high (meaning no serial signal) only when both inputs are high (no serial signals on the input). When either signal drops low, the output drops low. This is perfect. My first thought was that I’d need a NOR gate, but an inverted AND gate is a NOR gate.

img_8404

Here’s my quick and dirty implementation. A reminder again is that this will be removed after this test. For now, it’s good enough.

img_8405

After connecting the GPS serial output and frequency counter serial output to the AND gate (which outputs to the computer), I instantly got the result I wanted!

serial-combine

RealTerm shows that both inputs are being received. It’s a mess though. If you want to know what everything is, read up on NEMA formatted GPS data.

combined-python

I whipped-up a python program to parse, display, and log key information. This display updates every 1 second. The bottom line is what is appended to the log file on ever read. It’s clunky, but again this is just for testing and debugging. I am eager to let this run for as long as I can (days?) so I can track how changes in satellite signal / number / fix quality influence measured frequency.


     

TENMA Multimeter Serial Hack

I just spent the afternoon reverse-engineering the 72 series TENMA multimeter serial interface, and can now access all of its readings from a standalone Python script. This lets me send all measurements made with the multimeter to my computer in real time (using an optically isolated connection), and eliminates the need for the TENMA PC interface software. In addition to allowing the development of custom software to use measurements from TENMA multimeters in real time, this project also lets allows TENMA multimeters to interface with Linux computers (such as the raspberry pi). I’ve had a TENMA 72-7750 multimeter for several years, and over all I’ve been happy with it! To be honest, 90% of my multimeter needs are just using a continuity tester or checking to see if there is voltage on a line. For checking electrical signals, I love my no-name (actually it’s branded “KOMEC”) $15 eBay special multimeter. The screen updates about 4 times a second, and I don’t care if it’s off by 10%, it’s cheap and light and fast and easy for simple tasks. However, when I’m going to use a multimeter to actually measure something, I reach for a higher quality meter like my TENMA 72-7750. Although similar TENMA models may be more popular, I went with this particular one because it could measure frequency which is convenient when building RF circuits. While big fancy frequency counters are nice to have on your workbench, I liked the idea of having that functionality built into my multimeter. I believe my particular model is discontinued, but it looks like the 72-7745 is a similar product, and there are many TENMA multimeters on Amazon. Back in April of 2013 I mentioned on my website that I’d consider writing interface software in Python. Now that I’m [finally] out of school and have a little more free time, I decided to pick up the project again. I ran into a few tangles along the way, but I’m happy to report this project is now working beautifully! The pyTENMA project is open-sourced on my GitHub. I’m excited to see what kind of data I can get out of this thing!

IMG_7956

This is my multimeter taking a measurement (resistance) and sending the data to my computer using the optically-isolated serial connector (which ships with the multimeter). In this picture, it’s interacting with the official TENMA software. To try to figure out what was going on, I probed pins of the serial port while data was being exchanged. The yellow trace is the data signal. There was a problem, and this problem took me hours to figure it out, but now that I realize what’s going on it seems so obvious. The problem was that I could never get the multimeter to send my Python script data, despite the fact that the exact same configuration would send the commercial program data. I used serial port sniffing software to view the data too! I matched the baud rate (19200 / 19230), data bits (7), and parity (odd), and I just couldn’t figure out why the heck this thing wouldn’t work. I resorted to using an oscilloscope to probe the pins of the serial cable directly. I made a small man-in-the-middle test jig to give me headers I could easily probe or solder wires to. After poking around, I learned two things. (1) I really need a logic analyzer. They’re so cheap now, I went ahead and ordered one. (2) The RTS line goes low and the DSR line goes high when data is being sent. I realized that the Python software was disregarding these pins. You wouldn’t think you needed them if you’re just going to be receiving data with software control… but I immediately realized that those pins may be important for powering the optoelectronics (likely a phototransistor and some passive components) underlying the data exchange. After all, it’s not like the multimeter is able to source or sink appreciable current through an optical connection! I’ll note that some sketchy schematics are floating around Hackaday (pun intended), but the web page they link to doesn’t look very complete so I’m not sure how far that author got toward the same endeavor I’m chasing.

IMG_7961

Here you can see some of the adjacent (non-data) pins change their voltage state during transmissions. Once I realized replicating these states was also necessary, everything quickly fell into place. After manually commanding the RTS pin to lie low (1 line of code), the data starting coming in! I finished writing a basic pyTENMA class (which does a lot of hardware detection, string parsing, etc. to generate simple no-nonsense value/unit pairs to return to the user as well as log values to disk automatically) and tried to make it as simple as possible. Without going into too much detail (see the note in the top of my source code for more information), the multimeter just sends a 9-character ASCII string every second. I refer to this string as ABBBBCDEF. Byte 1 is a multiplier and bytes 2-5 are the value displayed on the screen. The actual value of a read is BBBB*10^A. The units depend on the mode (resistance, capacitance, etc), which is indicated by byte 6. It’s a little funny in that “4” means temperature and “;” means voltage, but once I figured out (through trial and error) which symbols match with which mode it was pretty easy to make it work for me. D is the sign (negative, zero, or positive), and I still haven’t really figured what E and F are. I thought they might be things like backlight or perhaps indicators of the range setting. I didn’t care to figure it out, because I already had access to the data I wanted!

To use the pyTENMA script, just drop it alongside a Python script you want to work on. Import it, tell it a COM port to use (if not, it’ll try to guess one) and a log file (optional). This is all the code you need:

import pyTENMA # make sure pyTENMA.py is in the same folder
PT=pyTENMA.pyTenma("COM4","log.txt")
PT.readUntilBroken()

The output is very simple. Here it is compared to the commercial TENMA software. PyroElectro has a good demonstration of the PC interface software that ships with this unit. While the TENMA software is functional, it has some serious limitations that motivate me to improve upon it. (1) It’s Windows only. (2) It doesn’t automatically log data (you have to manually click save to write it to disk). (3) It seems to be limited to COM1-COM4. My USB serial adapter was on COM7 and inaccessible to this program. I had to go in the device manager and change the advanced settings to allow the commercial software to read my device. (4) The graphs are poor, non-interactive, and often broken. (5) Data output format is only an Excel spreadsheet (.xls), and I don’t have control to save in other formats like CSV. If I’m going to use this on a raspberry pi, I don’t want to fumble around with Microsoft Office! Yeah I know I can get modules (even for Python) to access data in excel spreadsheets, but it seems like an unnecessary complexity just to retrieve some voltage readings. Over all it seems a little unfortunate that a relatively great product is pulled down when its weakest link is its software. It’s okay, we are on our way to can fixing this with pyTENMA!

pyTENMA
pyTENMA

 

official TENMA software
official TENMA software

Simple Example: Measuring capacitor leakage

I set up an experiment to demonstrate how logging data works. I charged a 22uF capacitor on a breadboard and let it sit there disconnected, slowly draining through leakage (and perhaps micro current draw from the multimeter). After a while I slowly charged it (using my body as a resistor, touching the +5V line and touching the capacitor lead with my fingers) and watched it discharge again. You can set pyTENMA software to save as little or often as you want. It defaults to every 10 reads, but I adjust it to every 100 reads for longer experiments. Also note that if you break it (with CTRL+C) it gently disconnects the serial device, logs remaining data to disk, then exits gracefully.
IMG_7981

 

In this demonstration, voltage across the capacitor on the breadboard is being measured by the multimeter, and reported (and logged) in real time by pyTENMA seen on the screen. Here is what that data looks like after about a half hour of run time. The code to read the log file and make graphs from it (using numpy and matplotlib) is in the logPlot source code.

logDemo

 

Real World Example: Measuring voltage and current during warm-up of an oven controlled crystal oscillator (OCXO)

Now that I know everything is up and running, I can use this device to make some measurements I’m actually interested in! In reality, this usage case is the reason I went through all the trouble to write custom data logging software for this multimeter is specifically for this case. I’m working on a large project involving a GPS-disciplined oven controlled crystal oscillator (OCXO) for a 1pps frequency reference, and spoiler alert it involves a raspberry pi to plot and upload live graphs of real-time frequency and accuracy statistics to my website. I don’t want to discuss it yet (it’s not complete), but I can’t avoid mentioning it since I’m showing photos of it. I’ll surely make a follow-up post when that project is complete and well documented. For now, the only relevant thing is that the device is an oven which takes a lot of current to heat from room temperature to a high temperature, and a smaller amount of current to maintain it at that temperature. I wanted to know how long it takes the current to stabilize over time (on a scale of hours), determine if its current draw oscillates, and also assess what the voltage at the oscillator reads during warm-up (high current draw) vs. stable conditions.

FullSizeRender (2)

My test setup uses the TENMA multimeter in current measuring configuration. Note the configuration of the multimeter test leads as being in series with the power supply.  This meter has two current measurement settings, one for <600 mA and one for up to 10 A. I know that the oscillator draws about 2 A during warm-up (this is because I’m intentionally limiting it to 2A), and stabilizes to somewhere near 200 mA after several minutes. To maximize my sample resolution, I started the recording using the 10 A setting, then after it dropped well below 600 mA I switched to the lower current setting. The data is colored red and blue, respectively:

current stabilizes within 10 minutes
current stabilizes within 10 minutes
Current is maxed-out for a few minutes, then oscillates (cool!) then stabilizes
Current is maxed-out for a few minutes, oscillates then stabilizes. 10 A / 600 mA measuring settings are in red and blue (respectively).
once stable, is stable for hours
once stable, current draw is stable for hours

I concluded that this thing stabilizes to within 10% of its final current draw well within 10 minutes. From there, it seems really stable, but slowly oscillates on a time scale of tens of minutes. I suspect this correlates with the AC unit of my house turning on and off. A similar recording of temperature of the oscillator (which the TENMA 72-7750 can also do with the thermocouple it was shipped with) may provide more insight as to whether or not the oscillator itself is actually changing temperature during these current oscillations. Now I’m curious what the voltage does during the warm-up period while the current is maxed out. I guess I need to reveal that my current limit is provided by two parallel LM7809 voltage regulators each in series with a 2 Ohm current limiting resistor before connecting to a common +9V rail which is running the oscillator. Since each regulator is current limited to about 1A, it’s no surprise my maximum current is about 2A, but I’d be interested to learn what the voltage is doing during that period.

I measured voltage just downstream of the voltage regulators.
I measured voltage just downstream of the voltage regulators.
The voltage reading is less exciting
The voltage reading is less exciting
During current max-out, the voltage is <<9V
During current max-out, the voltage is <<9V
voltage stabilizes after about 10 minutes
voltage stabilizes after about 10 minutes

I am interested in seeing what of these measurements (with more such as temperature and OCXO frequency) look like when they are all measured simultaneously. The TENMA multimeter I’m using can’t measure voltage and current at the same time (which would require a third lead, if you think about it), so this solution will require alternative equipment. Stay tuned, because I have a cool solution for that in the works! For now, I couldn’t be happier with my TENMA multimeter’s ability to log data to text files using pyTENMA and the ease in which numpy/matplotlib can read and graph them. A data logging multimeter is a great tool to have in any engineer’s toolbox, and I’m glad I now have one that plays nicely with Python.


     

DIY ECG with 1 op-amp

I made surprisingly good ECG from a single op-amp and 5 resistors! An ECG (electrocardiograph, sometimes called EKG) is a graph of the electrical potential your heart produces as it beats. Seven years ago I posted DIY ECG Machine on the Cheap which showed a discernible ECG I obtained using an op-amp, two resistors, and a capacitor outputting to a PC sound card’s microphone input. It didn’t work well, but the fact that it worked at all was impressive! It has been one of the most popular posts of my website ever since, and I get 1-2 emails a month from people trying to recreate these results (some of them are during the last week of a college design course and sound pretty desperate). Sometimes people get good results with that old circuit, but more often than not the output isn’t what people expected. I decided to revisit this project (with more patience and experience under my belt) and see if I could improve it. My goal was not to create the highest quality ECG machine I could, but rather to create the simplest one I could with emphasis on predictable and reproducible results. The finished project is a blend of improved hardware and custom cross-platform open-source software (which runs on Windows, Linux, and MacOS), and an impressively good ECG considering the circuit is so simple and runs on a breadboard! Furthermore, the schematics and custom software are all open-sourced on my github!

ECG_1470609065
my heartbeat recorded while filming the YouTube video shown below

Here’s a video demonstrating how the output is shown in real time with custom Python software. The video is quite long, but you can see the device in action immediately, so even if you only watch the first few seconds you will see this circuit in action with the custom software. In short, the amplifier circuit (described in detail below) outputs to the computer’s microphone and a Python script I wrote analyzes the audio data, performs low-pass filtering, and graphs the output in real time. The result is a live electrocardiograph!

The circuit is simple, but a lot of time and thought and experimentation went into it. I settled on this design because it produced the best and most reliable results, and it has a few nuances which might not be obvious at first. Although I discuss it in detail in the video, here are the highlights:

circuit

  • The output goes to the microphone jack of your computer.
  • There’s nothing special about the op-amp I used (LM741). A single unit of an LM324 (or any general purpose op-amp) should work just as well.
  • Resistor values were chosen because I had them on hand. You can probably change them a lot as long as they’re in the same ballpark of the values shown here. Just make sure R1 and R2 are matched, and R3 should be at least 10MOhm.
  • Do not use a bench power supply! “BAT+” and “BAT-” are the leads of a single 9V battery.
  • Note that the leg electrode is ground (same ground as the computer’s microphone ground)
  • R5 and R4 form a traditional voltage divider like you’d expect for an op-amp with a gain of about 50.
    • You’d expect R4 to connect to ground, but since your body is grounded, chest 2 is essentially the same
    • R3 must be extremely high value, but it pulls your body potential near the optimal input voltage for amplification by the op-amp.
    • R1 and R2 split the 9V battery’s voltage in half and center it at ground, creating -4.5V and +4.5V.
  • altogether, your body stays grounded, and the op-amp becomes powered by -4.5V and +4.5V, and your body is conveniently near the middle and ready to have small signals from CHEST1 amplified. Amplification is with respect to CHEST2 (roughly ground), rather than actual ground, so that a lot of noise (with respect to ground) is eliminated.
DIY ECG made from 1 op-amp, 5 resistors, a 9V battery, and 3 penny electrodes

For those of you who would rather see a picture than a schematic, here’s a diagram of how to assemble it graphically. This should be very easy to reproduce. Although breadboards are typically not recommended for small signal amplification projects, there is so much noise already in these signals that it doesn’t really matter much either way. Check out how good the signals look in my video, and consider that I use a breadboard the entire time.

design

The most comfortable electrodes I used were made for muscle simulators. A friend of mine showed me some muscle stimulator pads he got for a back pain relief device he uses. As soon as I saw those pads, I immediately thought they would be perfect for building an ECG! They’re a little bit expensive, but very comfortable, reusable, last a long time, and produce brilliant results. They also have 3.5 mm (headphone jack) connectors which is perfect for DIY projects. On Amazon.com you can get 16 pads for $11 with free shipping. I decided not to include links, because sometimes the pads and cords are sold separately, and sometimes they have barrel connectors and sometimes they have snap connectors. Just get any adhesive reusable electrodes intended for transcutaneous electrical nerve stimulation (TENS) that you can find! They should all work fine.

IMG_7576

You can make your own electrodes for $0.03! Okay that’s a terrible joke, but it’s true. I made not-awful electrodes by soldering wires to copper pennies, adding strength by super-gluing the wire to the penny, and using electrical tape to attach them to my chest. Unless you want a tattoo of an old guy’s face on your torso, wait until they cool sufficiently after soldering before proceeding to the adhesion step. I suspect that super gluing the penny to your chest would also work, but please do not do this. Ironically, because the adhesive pads of the TENS electrodes wear away over time, the penny solution is probably “more reusable” than the commercial electrode option.

I put pennies on wood to help them get hot as I solder to them.
I put pennies on wood to help them get hot as I solder to them.

 

penny electrodes match the minimalist style of this project
penny electrodes match the minimalist style of this project

 

this ECG was captures using penny electrodes
This ECG was captured using penny electrodes. It’s pretty darn good!

 

Notes on filtering: Why didn’t I just use a hardware low-pass filter?

  1. It would have required extra components, which goes against the theme of this project
  2. It would require specific value components, which is also undesirable for a junkbox project
  3. I’m partial to the Chebyshev filter, but getting an extremely sharp roll-off a few Hz shy of 50Hz would take multiple poles (of closely matched passive components) and not be as trivial as it sounds.

Notes on software: This a really cool use of Python! I lean on some of my favorite packages numpy, scipy, matplotlib, pyqrgraph, and PyQt4. I’ve recently made posts describing how to perform real-time data graphing in Python using these libraries, so I won’t go into that here. If you’re interested, check out my real-time audio monitor, notes on using PlotWidget, and notes on using MatPlotLib widget. I tried using PyInstaller to package this project into a single .EXE for all my windows readers who might want to recreate this project, but the resulting EXE was over 160MB! That’s crazy! It makes sense considering packagers like PyInstaller and Py2EXE work by building your entire python interpreter and all imported libraries. With all those fun libraries I listed above, it’s no wonder it came out so huge. It may be convenient for local quick-fixes, but not a good way to distribute code over the internet. To use this software, just run it in Python. It was tested to work with out-of-the-box WinPython-64bit-3.5.2.1 (not the Qt5 version), so if you want to run it yourself start there.

Notes on safety. How safe is this project? I’m conflicted on this subject. I want to be as conservative as I can (leaning on the side of caution), but I also want to be as realistic as possible. I’m going to play it safe and say “this may not be safe, so don’t build or use it”. As an exercise, let’s consider the pros and cons:

  • PROS:
    • It’s powered from a 9V battery which is safer than a bench power supply (but see the matching con).
    • The only connections to your body are:
      • leg – ground. you ground yourself all the time. using a wrist grounding strap is the same thing.
      • chest 1 – extremely high impedance. You’re attaching your chest to the high impedance input of an op-amp (which I feel fine with), and also to a floating battery through a 10MOhm resistor (which also I feel fine with)
      • chest 2 – raises an eyebrow. In addition to a high impedance input, you’re connected to an op-amp through a 100k resistor. Even if the op-amp were putting out a full 4.5V, that’s 0.045mA (which doesn’t concern me a whole lot).
    • I don’t know where to stick this, but I wonder what type of voltages / currents TENS actually provide.
  • CONS / WARNINGS:
    • It’s powered from a 9V battery. So are many stun guns.
    • If the op-amp oscillates, oscillations may enter your body. Personally I feel this may be the most concerning issue.
    • Small currents can kill. I found a curiously colored website that describes this. It seems like the most dangerous potential effect is induction of cardiac fibrillation, which can occur around 100mA.

Improving safety through optical isolation: The safety of this device may be improved (albeit with increased complexity) through the implementation of opto-isolators. I may consider a follow-up post demonstrating how I do this. Unlike digital signals which I’ve optically isolated before, I’ve never personally isolated analog signals. Although I’m sure there are fully analog means to do this, I suspect I’d accomplish it by turning it into a digital signal (with a voltage-to-frequency converter), pulsing the output across the optoisolator, and turning it back into voltage with a frequency-to-voltage converter or perhaps even a passive low-pass filter. Analog Devices has a good write-up about optical isolation techniques.

Do you have comments regarding the safety of this device? Write your thoughts concisely and send them to me in an email! I’d be happy to share your knowledge with everyone by posting it here.

Did you build this or a device similar to it? Send me some pictures! I’ll post them here.

Source code and project files: https://github.com/swharden/diyECG-1opAmp/

LEGAL: This website is for educational purposes only. Do not build or use any electrical devices shown. Attaching non-compliant electronic devices to your body may be dangerous. Consult a physician regarding proper usage of medical equipment.


     

Python Real-time Audio Frequency Monitor

A new project I’m working on requires real-time analysis of soundcard input data, and I made a minimal case example of how to do this in a cross-platform way using python 3, numpy, and PyQt. Previous posts compared performance of the matplotlib widget vs PyQtGraph plotwidget and I’ve been working with PyQtGraph ever since. For static figures matplotlib is wonderful, but for fast responsive applications I’m leaning toward PyQtGraph. The full source for this project is on a github page, but here’s a summary of the project.

demo

 

I made the UI with QT Designer. The graphs are QGraphicsView widgets promoted to a pyqtgraph PlotWidget. I describe how to do this in my previous post. Here’s the content of the primary program:

 

from PyQt4 import QtGui,QtCore
import sys
import ui_main
import numpy as np
import pyqtgraph
import SWHear

class ExampleApp(QtGui.QMainWindow, ui_main.Ui_MainWindow):
    def __init__(self, parent=None):
        pyqtgraph.setConfigOption('background', 'w') #before loading widget
        super(ExampleApp, self).__init__(parent)
        self.setupUi(self)
        self.grFFT.plotItem.showGrid(True, True, 0.7)
        self.grPCM.plotItem.showGrid(True, True, 0.7)
        self.maxFFT=0
        self.maxPCM=0
        self.ear = SWHear.SWHear()
        self.ear.stream_start()

    def update(self):
        if not self.ear.data is None and not self.ear.fft is None:
            pcmMax=np.max(np.abs(self.ear.data))
            if pcmMax>self.maxPCM:
                self.maxPCM=pcmMax
                self.grPCM.plotItem.setRange(yRange=[-pcmMax,pcmMax])
            if np.max(self.ear.fft)>self.maxFFT:
                self.maxFFT=np.max(np.abs(self.ear.fft))
                self.grFFT.plotItem.setRange(yRange=[0,self.maxFFT])
            self.pbLevel.setValue(1000*pcmMax/self.maxPCM)
            pen=pyqtgraph.mkPen(color='b')
            self.grPCM.plot(self.ear.datax,self.ear.data,
                            pen=pen,clear=True)
            pen=pyqtgraph.mkPen(color='r')
            self.grFFT.plot(self.ear.fftx[:500],self.ear.fft[:500],
                            pen=pen,clear=True)
        QtCore.QTimer.singleShot(1, self.update) # QUICKLY repeat

if __name__=="__main__":
    app = QtGui.QApplication(sys.argv)
    form = ExampleApp()
    form.show()
    form.update() #start with something
    app.exec_()
    print("DONE")

note: this project uses a gutted version of the SWHEar class which I still haven’t released on githib yet. It will likely have its own project folder. For now, take this project with a grain of salt. The primary advantage of this class is that it makes it easy to use PyAudio to automatically detect input sound cards, channels, and sample rates which are likely to succeed without requiring the user to enter any information.

All files used for this project are in a GitHub folder

2016-09-05: Okko adapted this project into a screenlet (cross platform) which also includes an installer for Windows. That Githib page is here: https://github.com/ninlith/audio-visualizer-screenlet Below is a screenshot of me running it on my Windows 10 machine

widget


     

Live Data in PyQt4 with PlotWidget

After spending a day comparing performance of MatplotlibWidget with PlotWidget, when it comes to speed PlotWidget wins by a mile! Glance over my last post where I describe how to set up the window with QT Designer and convert the .ui file to a .py file. With only a few changes to the code, I swapped the matplotlib MatplotlibWidget with the pyqtgraph PlotWidget. I easily got a 20x increase in speed (frame rate), and I’m likely to favor PyQtGraph over matpltolib for python applications involving realtime display of data. Just like the previous example using matplotlib, this one uses the system time to control the phase and color of a sine wave in real time. You can grab the full code from my github folder.

demo2

demo2cmd

When designing the GUI with QT Designer, add a QGraphicsView widget then assign it to become a PyQtGraph object. To do this, follow the steps found on the pyqtgraph docs page:

  1. In Designer, create a QGraphicsView widget.
  2. Right-click on the QGraphicsView and select “Promote To…”.
  3. Set “Promoted class name” to “PlotWidget”.
  4. Under “Header file”, enter “pyqtgraph”.
  5. Click “Add”, then click “Promote”.
  6. apparently this only needs to be done once per project

promoted widgets

In addition to faster frame rate, the PyQtGraph method is easy to interact with. Clicking and dragging scrolls the data, and right-clicking and dragging zooms on each axis. Here’s the program code:

from PyQt4 import QtGui,QtCore
import sys
import ui_main
import numpy as np
import pylab
import time
import pyqtgraph

class ExampleApp(QtGui.QMainWindow, ui_main.Ui_MainWindow):
    def __init__(self, parent=None):
        pyqtgraph.setConfigOption('background', 'w') #before loading widget
        super(ExampleApp, self).__init__(parent)
        self.setupUi(self)
        self.btnAdd.clicked.connect(self.update)
        self.grPlot.plotItem.showGrid(True, True, 0.7)

    def update(self):
        t1=time.clock()
        points=100 #number of data points
        X=np.arange(points)
        Y=np.sin(np.arange(points)/points*3*np.pi+time.time())
        C=pyqtgraph.hsvColor(time.time()/5%1,alpha=.5)
        pen=pyqtgraph.mkPen(color=C,width=10)
        self.grPlot.plot(X,Y,pen=pen,clear=True)
        print("update took %.02f ms"%((time.clock()-t1)*1000))
        if self.chkMore.isChecked():
            QtCore.QTimer.singleShot(1, self.update) # QUICKLY repeat

if __name__=="__main__":
    app = QtGui.QApplication(sys.argv)
    form = ExampleApp()
    form.show()
    form.update() #start with something
    app.exec_()
    print("DONE")

All files used in project can be downloaded from the GitHub page.


     

Live Data in PyQt4 with MatplotlibWidget

I keep getting involved in projects which require live data to be graphed in real time. Since most of my back-end is written in Python, it makes sense to have a Pythonic front-end. Cross-platform GUI programming in Python is frustratingly non-trivial, as there multiple window frameworks available (TK, GTK, and QT) and their respective graphical designers (torture, Glade, and QT Designer) and each has its own way of doing things. Add different ways to plot data in the mix (gnuplot, matplotlib, and custom widgets) and it can become a complicated mess. Different framework combinations favor different features (with unique speed / simplicity / elegance), so my goal is to slowly test out a few combinations most likely to work for my needs, and add my findings to a growing github repository. The first stab is using PyQt4 and matplotlib’s widget (MatplotlibWidget). Rather than capture data from the sound card (my ultimate goal), I’m going to generate a sine wave whose phase and color is related to the system time. Matplotlib plotting is a bit slow, but the output is beautiful, and their framework is so robust. Here’s the output of my first test showing the sine wave generated as well as the console output (showing that each call to the plotting function takes about 40 ms. At this rate, I can expect a maximum update rate of ~25 Hz.

demo

qt4

Designing this project was easy, but it was surprisingly hard to figure out how to do this based on examples I found on the internet. This is part of why I wanted to place this example here. The downside of many internet examples is that they did not use Qt Designer to make the window, so their code to create a window and insert the MatplotlibWidget wasn’t copy/paste compatible with my goals, and often more complex than I needed. Some internet examples did use Qt Designer to make the window, but left a frame empty which they later manually filled with a widget and attached to a matplotlib canvas. This is fine, but more complex than I need to get started.

First, I designed a GUI with Qt Designer. I dropped a MatplotlibWidget somewhere, and used its default name. I saved this file as ui_main.ui (which is an XML file, ready to be used for multiple programming languages).

pyqt4 designer

Next, I converted the UI file into a .py file with a standalone python script that’s an alternative to using pyuic from the command line. The script to do this is ui_convert.py and it calls PyQt4.uic.compileUi():

from PyQt4 import uic 
fin = open('ui_main.ui','r')
fout = open('ui_main.py','w')
uic.compileUi(fin,fout,execute=False)
fin.close()
fout.close()

I then created my main program file which populates the matplotlib widget with data. I called it go.py and running it will launch the app. The code explains it all, and there’s not much more to say! It produces the output at the top of this post.

from PyQt4 import QtGui,QtCore
import sys
import ui_main
import numpy as np
import pylab
import time

class ExampleApp(QtGui.QMainWindow, ui_main.Ui_MainWindow):
    def __init__(self, parent=None):
        super(ExampleApp, self).__init__(parent)
        self.setupUi(self)
        self.btnAdd.clicked.connect(self.update)
        self.matplotlibwidget.axes.hold(False) #clear on plot()

    def update(self):
        t1=time.time()
        points=100 #number of data points
        X=np.arange(points)
        Y=np.sin(np.arange(points)/points*3*np.pi+time.time())
        C=pylab.cm.jet(time.time()%10/10) # random color
        self.matplotlibwidget.axes.plot(X,Y,ms=100,color=C,lw=10,alpha=.8)
        self.matplotlibwidget.axes.grid()
        self.matplotlibwidget.axes.get_figure().tight_layout() # fill space
        self.matplotlibwidget.draw() # required to update the window
        print("update took %.02f ms"%((time.time()-t1)*1000))
        if self.chkMore.isChecked():
            QtCore.QTimer.singleShot(10, self.update) # QUICKLY repeat

if __name__=="__main__":
    app = QtGui.QApplication(sys.argv)
    form = ExampleApp()
    form.show()
    form.update() #start with something
    app.exec_()
    print("DONE")

All files used in project are available on GitHub


     

Realtime Audio Visualization in Python

Python’s “batteries included” nature makes it easy to interact with just about anything… except speakers and a microphone! As of this moment, there still are not standard libraries which which allow cross-platform interfacing with audio devices. There are some pretty convenient third-party modules, but I hope in the future a standard solution will be distributed with python. I appreciate the differences of Linux architectures such as ALSA and OSS, but toss in Windows and MacOS in the mix and it gets to be a huge mess. For Linux, would I even need anything fancy? I can run “cat file.wav > /dev/dsp” from a command prompt to play audio. There are some standard libraries for operating system specific sound (i.e., winsound), but I want something more versatile. The official audio wiki page on the subject lists a small collection of third-party platform-independent libraries. After excluding those which don’t support microphone access (the ultimate goal of all my poking around in this subject), I dove a little deeper into sounddevice and PyAudio. Both of these I installed with pip (i.e., pip install pyaudio)

For a more modern, cleaner, and more complete GUI-based viewer of realtime audio data (and the FFT frequency data), check out my Python Real-time Audio Frequency Monitor project.

I really like the structure and documentation of sounddevice, but I decided to keep developing with PyAudio for now. Sounddevice seemed to take more system resources than PyAudio (in my limited test conditions: Windows 10 with very fast and modern hardware, Python 3), and would audibly “glitch” music as it was being played every time it attached or detached from the microphone stream. I tried streaming, but after about an hour I couldn’t get clean live access to the microphone without glitching audio playback. Furthermore, every few times I ran this script it crashed my python kernel! I very rarely see this happening. iPython complained: “It seems the kernel died unexpectedly. Use ‘Restart kernel’ to continue using this console” and I eventually moved back to PyAudio. For a less “realtime” application, sounddevice might be a great solution. Here’s the minimal case sounddevice script I tested with (that crashed sometimes). If you have a better one to do live high-speed audio capture, let me know!

import sounddevice #pip install sounddevice

for i in range(30): #30 updates in 1 second
    rec = sounddevice.rec(44100/30)
    sounddevice.wait()
    print(rec.shape)

Here’s a simple demo to show how I get realtime microphone audio into numpy arrays using PyAudio. This isn’t really that special. It’s a good starting point though. Note that rather than have the user define a microphone source in the python script (I had a fancy menu system handling this for a while), I allow PyAudio to just look at the operating system’s default input device. This seems like a realistic expectation, and saves time as long as you don’t expect your user to be recording from two different devices at the same time. This script gets some audio from the microphone and shows the values in the console (ten times).

import pyaudio
import numpy as np

CHUNK = 4096 # number of data points to read at a time
RATE = 44100 # time resolution of the recording device (Hz)

p=pyaudio.PyAudio() # start the PyAudio class
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK) #uses default input device

# create a numpy array holding a single read of audio data
for i in range(10): #to it a few times just to see
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    print(data)

# close the stream gracefully
stream.stop_stream()
stream.close()
p.terminate()

01

I tried to push the limit a little bit and see how much useful data I could get from this console window. It turns out that it’s pretty responsive! Here’s a slight modification of the code, made to turn the console window into an impromptu VU meter.

import pyaudio
import numpy as np

CHUNK = 2**11
RATE = 44100

p=pyaudio.PyAudio()
stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
              frames_per_buffer=CHUNK)

for i in range(int(10*44100/1024)): #go for a few seconds
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    peak=np.average(np.abs(data))*2
    bars="#"*int(50*peak/2**16)
    print("%04d %05d %s"%(i,peak,bars))

stream.stop_stream()
stream.close()
p.terminate()

The results are pretty good! The advantage here is that no libraries are required except PyAudio. For people interested in doing simple math (peak detection, frequency detection, etc.) this is a perfect starting point. Here’s a quick cellphone video:

I’ve made realtime audio visualization (realtime FFT) scripts with Python before, but 80% of that code was creating a GUI. I want to see data in real time while I’m developing this code, but I really don’t want to mess with GUI programming. I then had a crazy idea. Everyone has a web browser, which is a pretty good GUI… with a Python script to analyze audio and save graphs (a lot of them, quickly) and some JavaScript running in a browser to keep refreshing those graphs, I could get an idea of what the audio stream is doing in something kind of like real time. It was intended to be a hack, but I never expected it to work so well! Check this out…

Here’s the python script to listen to the microphone and generate graphs:

import pyaudio
import numpy as np
import pylab
import time

RATE = 44100
CHUNK = int(RATE/20) # RATE / number of updates per second

def soundplot(stream):
    t1=time.time()
    data = np.fromstring(stream.read(CHUNK),dtype=np.int16)
    pylab.plot(data)
    pylab.title(i)
    pylab.grid()
    pylab.axis([0,len(data),-2**16/2,2**16/2])
    pylab.savefig("03.png",dpi=50)
    pylab.close('all')
    print("took %.02f ms"%((time.time()-t1)*1000))

if __name__=="__main__":
    p=pyaudio.PyAudio()
    stream=p.open(format=pyaudio.paInt16,channels=1,rate=RATE,input=True,
                  frames_per_buffer=CHUNK)
    for i in range(int(20*RATE/CHUNK)): #do this for 10 seconds
        soundplot(stream)
    stream.stop_stream()
    stream.close()
    p.terminate()

Here’s the HTML file with JavaScript to keep reloading the image… 

<html>
<script language="javascript">
function RefreshImage(){
document.pic0.src="03.png?a=" + String(Math.random()*99999999);
setTimeout('RefreshImage()',50);
}
</script>
<body onload="RefreshImage()">
<img name="pic0" src="03.png">
</body>
</html>

Here’s the result! I couldn’t believe my eyes. It’s not elegant, but it’s kind of functional!

Why stop there? I went ahead and wrote a microphone listening and processing class which makes this stuff easier. My ultimate goal hasn’t been revealed yet, but I’m sure it’ll be clear in a few weeks. Let’s just say there’s a lot of use in me visualizing streams of continuous data. Anyway, this class is the truly terrible attempt at a word pun by merging the words “SWH”, “ear”, and “Hear”, into the official title “SWHear” which seems to be unique on Google. This class is minimal case, but can be easily modified to implement threaded recording (which won’t cause the rest of the functions to hang) as well as mathematical manipulation of data, such as FFT. With the same HTML file as used above, here’s the new python script and some video of the output:

import pyaudio
import time
import pylab
import numpy as np

class SWHear(object):
    """
    The SWHear class is made to provide access to continuously recorded
    (and mathematically processed) microphone data.
    """

    def __init__(self,device=None,startStreaming=True):
        """fire up the SWHear class."""
        print(" -- initializing SWHear")

        self.chunk = 4096 # number of data points to read at a time
        self.rate = 44100 # time resolution of the recording device (Hz)

        # for tape recording (continuous "tape" of recent audio)
        self.tapeLength=2 #seconds
        self.tape=np.empty(self.rate*self.tapeLength)*np.nan

        self.p=pyaudio.PyAudio() # start the PyAudio class
        if startStreaming:
            self.stream_start()

    ### LOWEST LEVEL AUDIO ACCESS
    # pure access to microphone and stream operations
    # keep math, plotting, FFT, etc out of here.

    def stream_read(self):
        """return values for a single chunk"""
        data = np.fromstring(self.stream.read(self.chunk),dtype=np.int16)
        #print(data)
        return data

    def stream_start(self):
        """connect to the audio device and start a stream"""
        print(" -- stream started")
        self.stream=self.p.open(format=pyaudio.paInt16,channels=1,
                                rate=self.rate,input=True,
                                frames_per_buffer=self.chunk)

    def stream_stop(self):
        """close the stream but keep the PyAudio instance alive."""
        if 'stream' in locals():
            self.stream.stop_stream()
            self.stream.close()
        print(" -- stream CLOSED")

    def close(self):
        """gently detach from things."""
        self.stream_stop()
        self.p.terminate()

    ### TAPE METHODS
    # tape is like a circular magnetic ribbon of tape that's continously
    # recorded and recorded over in a loop. self.tape contains this data.
    # the newest data is always at the end. Don't modify data on the type,
    # but rather do math on it (like FFT) as you read from it.

    def tape_add(self):
        """add a single chunk to the tape."""
        self.tape[:-self.chunk]=self.tape[self.chunk:]
        self.tape[-self.chunk:]=self.stream_read()

    def tape_flush(self):
        """completely fill tape with new data."""
        readsInTape=int(self.rate*self.tapeLength/self.chunk)
        print(" -- flushing %d s tape with %dx%.2f ms reads"%\
                  (self.tapeLength,readsInTape,self.chunk/self.rate))
        for i in range(readsInTape):
            self.tape_add()

    def tape_forever(self,plotSec=.25):
        t1=0
        try:
            while True:
                self.tape_add()
                if (time.time()-t1)>plotSec:
                    t1=time.time()
                    self.tape_plot()
        except:
            print(" ~~ exception (keyboard?)")
            return

    def tape_plot(self,saveAs="03.png"):
        """plot what's in the tape."""
        pylab.plot(np.arange(len(self.tape))/self.rate,self.tape)
        pylab.axis([0,self.tapeLength,-2**16/2,2**16/2])
        if saveAs:
            t1=time.time()
            pylab.savefig(saveAs,dpi=50)
            print("plotting saving took %.02f ms"%((time.time()-t1)*1000))
        else:
            pylab.show()
            print() #good for IPython
        pylab.close('all')

if __name__=="__main__":
    ear=SWHear()
    ear.tape_forever()
    ear.close()
    print("DONE")

I don’t really intend anyone to actually do this, but it’s a cool alternative to recording a small portion of audio, plotting it in a pop-up matplotlib window, and waiting for the user to close it to record a new fraction. I had a lot more text in here demonstrating real-time FFT, but I’d rather consolidate everything FFT related into a single post. For now, I’m happy pursuing microphone-related python projects with PyAudio.