I realized that the C code from yesterday wasn’t showing-up properly because of textile, a rapid, inline, tag-based formatting system. The app converted blog code from [“text”:http://www.SWHarden.com/ *like* _this_] to [text like this. ] While it’s fun and convenient to use, it’s not always practical. The problem I was having was that in C code, variable names (such as _delay_) were becoming irrevocably italicized, and nothing I did could prevent textile from ignoring code while styling text. The kicker is that I couldn’t disable it easily, because I’ve been writing in this style for over four years! I decided that the time was now to put my mad Python skills to the test and write code to handle the conversion from textile-format to raw HTML.
I accomplished this feat in a number of steps. Yeah, I could have done hours of research to find a “faster way”, but it simply wouldn’t have been as creative. In a nutshell, I backed-up the SQL database using PHPMyAdmin to a single “x.sql” file. I then wrote a pythons script to parse this [massive] file and output “o.sql”, the same data but with all of the textile tags I commonly used replaced by their HTML equivalent. It’s not 100% perfect, but it’s 99.999% perfect. I’ll accept that. The output? You’re viewing it! Here’s the code I used to do it:

## This Python (1.0) script removes *SOME* textile formatting from WordPress
## backups in plain text SQL format (dumped from PHP MyAdmin). Specifically,
## it corrects bold and itallic fonts and corrects links. It should be easy
## to expand if you need to do something else with it.
## Enjoy! --Scott Harden (www.SWHarden.com)

infile = 'x.sql' # < < THIS IS THE INPUT FILE NAME!

replacements=   ["r"," "],["n"," n "],["*:","* :"],["_:","_ :"],
                ["n","<br>n"],[">*","> *"],["*< ","* <"],
                [">_","> _"],["_< ","_ <"],
                [" *"," <b>"],["* "," "],[" _"," <i>"],["_ ","</i> "]
                #These are the easy replacements

def fixLinks(line):
    ## replace ["links":URL] with [<a href="URL">links</a>]. ##
    words = line.split(" ")
    for i in range(len(words)):
        word = words[i]
        if '":' in word:
            upto=1
            while (word.count('"')&lt;2):
                word = words[i-upto]+" "+word
                upto+=1
            word_orig = word
            extra=""
            word = word.split('":')
            word[0]=word[0][1:]
            for char in ".),'":
                if word[1][-1]==char: extra=char
            if len(extra)>0: word[1]=word[1][:-1]
            word_new='<a href="%s">%s</a>'%(word[1],word[0])+extra
            line=line.replace(word_orig,word_new)
    return line

def stripTextile(orig):
    ## Handle the replacements and link fixing for each line. ##
    if not orig.count("', '") == 13: return orig #non-normal post
    line=orig
    temp = line.split
    line = line.split("', '",5)[2]
    if len(line)&lt;10:return orig #non-normal post
    origline = line
    line = " "+line
    for replacement in replacements:
        line = line.replace(replacement[0],replacement[1])
    line=fixLinks(line)
    line = orig.replace(origline,line)
    return line

f=open(infile)
raw=f.readlines()
f.close
posts=0
for raw_i in range(len(raw)):
    if raw[raw_i][:11]=="INSERT INTO":
        if "wp_posts" in raw[raw_i]: #if it's a post, handle it!
            posts+=1
            print "on post",posts
            raw[raw_i]=stripTextile(raw[raw_i])


print "WRITING..."
out = ""
for line in raw:
    out+=line
f=open('o.sql','w')
f.write(out)
f.close()

I certainly held my breath while the thing ran. As I previously mentioned, this thing modified SQL tables. Therefore, when I uploaded the “corrected” versions, I kept breaking the site until I got all the bugs worked out. Here’s an image from earlier today when my site was totally dead (0 blog posts)

hostingwork





Additional Resources

THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!

>>> CHECK OUT THE NEW CODE < << [Generate Apache-Style HTTP Access Logs via SQL and PHP]

OBSOLETE CODE IS BELOW…

A few months ago I wrote about a way I use PHP to generate apache-style access.log files since my web host blocks access to them. Since then I’ve forgotten it was even running! I now have some pretty cool-looking graphs generated by Python and Matplotlib. For details (and the messy script) check the original posting.

This image represents the number of requests (php pages) made per hour since I implemented the script. It might be a good idea to perform some linear data smoothing techniques (which I love writing about), but for now I’ll leave it as it is so it most accurately reflects the actual data.





Additional Resources

UPDATE: I found a method of PC/microcontroller communication which I feel is simpler, easier, and definitely cheaper than this! It’s not good for everything, but worth looking at. It’s a way to communicate with a PC using your sound card and zero components!

Another page HERE has more info on how to do this with an intermediate IC if yours doesn’t have RX/TX pins…

I recently had the desire to be able to see data from an ATMEL AVR microcontroller (the ATTiny2313) for development and debugging purposes. I wanted an easy way to have my microcontroller talk to my PC (and vise versa) with a minimum number of parts. The easiest way to do this was to utilize the UART capabilities of the ATTiny2313 to talk to my PC through the serial port. One problem is that the ATTiny2313(as with most microcontrollers) puts out 5V for “high” (on) and 0V for “low” (off). The RS-232 standard (which PC serial ports use) required -15V for high and +15v for low! Obviously the microcontroller needs some help to achieve this. The easiest way was to use the MAX232 serial level converter which costs about 3 bucks at DigiKey. Note that it requires a few 10uF capacitors to function properly.

Here’s a more general schematic:

I connected my ATTiny2313 to the MAX232 in a very standard way. (photo) MAX232 pins 13 and 14 go to the serial port, and the ATTiny2313 pins 2 and 3 go to the MAX232 pins 12 and 11 respectively. I will note that they used a oscillator value (3.6864MHz) different than mine (9.216MHz).

Determining the speed of serial communication is important. This is dependent on your oscillator frequency! I said I used a 9.216Mhz oscillator. First, a crystal or ceramic oscillator is required over the internal RC oscillator because the internal RC oscillator is not accurate enough for serial communication. The oscillator you select should be a perfect multiple of 1.8432MHz. Mine is 5x this value. Many people use 2x this value (3.6864Mhz) and that’s okay! You just have to make sure your microchip knows (1) to use the external oscillator (google around for how to burn the fuses on your chip to do this) and (2) what the frequency of your oscillator is and how fast it should be sending data. This is done by setting the UBRRL value. The formula to do this is here:

The datasheet of your microcontroller may list a lot of common crystal frequencies, bandwidths, and their appropriate UBRR values. However my datasheet lacked an entry for a 9.216MHz crystal, so I had to do the math myself. I Googled around and no “table” is available! Why not make one? (picture, below). Anyway, for my case I determined that if I set the UBRR value to 239, I could transmit data at 2800 baud (bits/second). This is slow enough to ensure accuracy, but fast enough to quickly dump a large amount of text to a PC terminal.

>>> IMPORTANT UPDATE!

This will make your life easier. The page wormfood.net/avrbaudcalc.php has a chart of common crystals and the baud rates they work best with! Try to pick a combination that provides the least error possible...

This is the bare-minimum code to test out my setup. Just load the code (written in C, compiled with avr-gcc) onto your chip and it’s ready to go. Be sure you set your fuses to use an external oscillator and that you set your UBRRL value correctly.

  

  #include <avr/io.h>  

  #include <avr/interrupt.h>  

  #include <util/delay.h>  

   

 int main (void)  

 {  

   unsigned char data=0;  

   UBRRL = 239;  

   UCSRB = (1 < < RXEN) | (1 << TXEN);  

   UCSRC = (1 < < UCSZ1) | (1 << UCSZ0);  

   for (;;)  

     {  

     if (data>'Z'||data< 'A')  

     {  

       UDR = 10; UDR = 13; data='A';_delay_ms(100);  

     }  

     UDR = data;  

     data += 1;  

     _delay_ms(100);  

     }  

 }  

 

Once you load it, it’s ready to roll! It continuously dumps letters to the serial port. To receive them, open HyperTerminal (on windows, under accessories) or minicom (on Linux, look it up!). Set your baud rate to 2800 (or whatever you selected) and you’re in business. This (picture below) is the output of the microcontroller to HyperTerminal on my PC. Forgive the image quality, I photographed the LCD screen instead of taking a screenshot.

This is the circuit which generates the output of the previous image. I have a few extra components. I have an LED which I used for debugging purposes, and also a switch (labeled “R”). The switch (when pressed) grounds pin 1 of the ATTiny2313 which resets it. If I want to program the chip, I hold “R” down and the PC can program it with the inline programmer “parallel port, straight-through, DAPA style. One cable going into the circuit is for the parallel port programmer, one cable is for the serial port (data transfer), and one is for power (5v which I stole from a USB port).

I hope you found this information useful. Feel free to contact me with any questions you may have, but realize that I’m no expert, and I’m merely documenting my successes chronologically on this website.





Additional Resources

I placed another online order to DigiKey yesterday. They have virtually every electronic component available for next-day delivery. How awesome is that? The frustrating thing is that I’m always trying to get my order in before 6pm CMT so that my package will ship same day, so I rush, and I always forget something! It never fails. 30 minutes after I place an order, I always remember something else I wish I’d added to the order (to save on shipping). So, instead of buying it when I want it (now) I wait a few more days to think of some more stuff to add. The last couple weeks I’ve placed 3 orders totaling about $30. You never think you’re spending a lot, but all the sudden you realize that all these 40-cent parts add up when you go to check out. Shipping is usually under $3, which makes me happy. What did I get? I got a couple MAX232 chips so I can easialy shuffle data back and forth from my PC to an AVR chip using the serial port using a simple terminal application like hyperterminal. This will allow me to do some pretty fancy stuff and certainly help with the prototyping/debugging/development steps of my various microcontroller-based projects.

Some of the things which I forgot to order include a few 1F supercapacitors for my solar-powered QRSS beacon project and a random collection of parts needed to build a DIY AVR microcontroller programmer with a USB PC interface and I also wanted to buy at least one ATMega8 microcontroller because they seem to be a good step above the ATTiny 2313 chips I already have and are at the center of so many microcontroller projects. I’m sure there are are few more items I need, but can’t think of them right now, so they don’t make the list.

A little random, but worth noting is a cool website I found earlier which details (in a very basic way) the various types of HF antennas and does a good job (IMHO) of explaining the basic theory behind them. The page can be found at deltadx.net.

Let’s see, what have I been up to? Today’s work (similar to much of last week) involves “damage control” of scientific images. Today’s project is recovering images from a project that cost hundreds of thousands of dollars to complete. Hundreds of animals were killed. Hundreds of man hours went into this project. Numerous animal surgeries on old and sick mice. This project was one of the most massive, time-consuming, and resource-consuming projects ever undertaken by this laboratory. The results (significant) are represented numerically (some graphs) and visually (some representative microscope scans of neurons innervating tissue). But wait! What’s this? The only representative images we have are in highly-compressed JPEG format! (extremely damaged from the start) Holy crap, some are even in 256-color GIF format! To add to the frustration, scale bars, labels, arrows, and titles (some of then incorrect and requiring modification) are permanently burned into the images. Yet, these are all that remain from the project. I’ve got my work cut out for me huh? Check out this example image. This (originally) was a gorgeous TIFF file showing an axon gracefully innervating its target tissue. However, the conversion to extremely-poor-quality JPEG format caused the image to be fragmented into thousands of small squares which (due to the compression algorithm) no longer line up. If you want to get technical, one might be able to argue that since they no longer line-up, it demonstrates that they no longer reflect accurate, actual signals. In other words, this is no longer a scientifically-valid image. In some pixels you’re not seeing things were originally there, and not seeing some things that were originally there. Supposedly backups of the original images were never made, and this is all we have to work with. It’s certainly a challenge!

On that note I encourage everyone to take a couple minutes out of your day and browse the Wikipedia entry describing the JPEG compression algorithm. Seriously, it’s an educational, enlightening, and entertaining read.





Additional Resources

Man, what a long day! Work is so tedious sometimes. This week I’ve been proofing scientific literature (revising scientific manuscripts in an attempt to improve them as much as possible to increase their probability of acceptance and timely publication). I’ve been using Office 2003 (with “track changes”) to do this. I make changes, my boss makes changes, I make more changes, and it goes back and forth a few times. I wonder why office 2007 is so bad. Does anybody truly like it, and find it to be a significant improvement upon 2003? … or Vista over XP? [sigh] Maybe I’m just getting old, inflexible, and grumpy.

Here, take a look at what I’m working on [snapps screenshot]. I had to blur the content for intellectual property protection and to avoid possible future copyright violations. The light bubbles on the right are deletions. The dark bubbles on the right are comments. The red text is insertions/modifications I made. Pretty intense, huh? Pages and pages of this. And, upon successful completion of a manuscript, my reward is to begin working on another one! Luckily we’re almost caught-up on manuscripts… but that means we get to start writing grants… I’m starting to grasp the daunting amount of time a scientist must spend writing in the laboratory as opposed to performing actual experiments or even doing literature research.

Last night I assembled a Pixie II circuit similar to the one pictured here. I must say that I’m a little disappointed with the information available on the internet regarding simple RF theory in relation to transceiver circuits. I’m probably just not looking in the right places though. (Yes, I know about the ARRL handbook.) The thing is that I’m just now starting to get into RF circuitry and the concept looking at solid-state circuits and imagining a combination of AC and DC flowing through it is warping my brain. I have everything I need to build an ultra-simple Pixie II transceiver (which is supposedly capable of morse code transmissions over 300 miles, and QRSS applications over 3,000 miles) but I refuse to use it. No, it’s not because of moral obligations preventing me from powering it up before I get a general class radio license (shhhh). It’s because building something is useless unless you understand what you’re building.

I’m trying to break this circuit down into its primary components. I understand the role of the lowpass pi filter (before antenna). I understand the role of the 1st transistor and related circuitry in amplifying the output of the oscillator (left side). I totally get the audio amplifier circuitry (bottom). It’s that center transistor (which supposedly handles signal amplification, receiving, and mixing) that I can’t get my mind around. Every time I think figure it out for one mode (sending or receiving), I lose the other one, and visa versa. It has me very frustrated (and a little depressed about the whole thing) because this should be much easier than I’m making it. There’s no thourough documentation on this circuit! I selected it because it was extremely simple and I assumed I’d be smart enough to figure it out. I guess I was wrong. I wish I had an oscilloscope so I could probe the RF passing through various stages of this circuit [sigh]. Back to the ARRL handbook. Maybe if I read chapters 5-11 a couple more times I’ll magically understand it.