Don't Yell at Your PC Like I Do

I’ve been stuck in the laboratory all day! I’m currently writing software to convert data from images (microscope scans) into massive 4-dimensional arrays (handled by numpy and Python) which are then analyzed statistically (that’s the code I’m working on today). Since these data files are so huge, it usually takes at least 30 seconds just to load these massive arrays into memory before the calculations can be performed (which only take a second). The frustrating part is that the calculations don’t work right, so I make a change in the code and try again, and have to wait half a minute before another failed result. After doing this for hours (with only about 10 minutes of actual work – the rest of the time spent waiting) I began screaming at my PC. It’s ironic that (in my frustration-spawned break) I logged into Facebook and noticed that Tom Hayward posted a video entitled “shouting at your computer increases hard drive latency”. Do I need to say more? [dies laughing]


     

Fixing Slow Internet in Ubuntu

I recently swapped my two main PCs in my house. The “headless” (no monitor) media PC (whose job consists of downloading, storing, and playing movies) connected directly to my TV, and our standard desktop PC which my wife uses most of the time. I decided to do the swap because the media PC was way nicer than our desktop PC, and since the media PC is just playing movies and downloading torrents, I figured the extra processing power / ram / video acceleration could be put to better use. Anyhow, I decided (in both cases) to completely start fresh by wiping hard drives clean and reinstalling Ubuntu linux (I’m using 8.10 currently). However, after the installation I noticed a peculiar problem. I’ll quote it to emphasize it…

Browsing the internet was very slow. When I’d click a link on a website, it would take several seconds before it seemed to even try to go to the next page. The same thing would happen if I manually typed-in a new website. I tried disabling IPv6 in firefox’s about:config and in the /etc/init.d/aliases file, but it didn’t help!

The solution for me was simple, and since I spent a lot of time searching forums I know I’m not the only one with this problem. Disabling IPv6 was suggested in 99% of similar posts. My solution took a while to uncover, so I figured I’d write it here. The basic problem is that my DHCP (auto-configured IP address) settings were screwed up, and my manually setting them I fixed the problem. Here’s what I did…

Start by right-clicking your network icon (wireless in my case) and selecting connection information

Check out your current configuration. Is a local address (192.168.*.*) set for the primary DNS server? If so, that’s your problem! Note your secondary server. We’ll set it as your primary…

Continue by right-clicking your network icon (wireless in my case) and selecting edit connections*. Open the tab corresponding to your internet connection (wired or wireless – wireless in my case), select your connection, and click Edit

Use this screen to manually enter the information from the information screen you saw earlier, but making sure not to list any local IP addresses as the DNS servers. Save your settings, close the windows, and the problem should be immediately corrected. Leave “search domains” blank, that’s important too. Good luck!!!


     

Compress Strings and Store to Files in Python

While writing code for my graduate research thesis I came across the need to lightly compress a huge and complex variable (a massive 3D data array) and store it in a text file for later retrieval. I decided to use the zlib compression library because it’s open source and works pretty much on every platform. I ran into a snag for a while though, because whenever I loaded data from a text file it wouldn’t properly decompress. I fixed this problem by adding the “rb” to the open line, forcing python to read the text file as binary data rather than ascii data. Below is my code, written in two functions to save/load compressed string data to/from files in Python.

  

 import zlib  

   

 def saveIt(data,fname):  

     data=str(data)  

     data=zlib.compress(data)  

     f=open(fname,'wb')  

     f.write(data)  

     f.close()  

     return  

   

 def openIt(fname,evaluate=True):  

     f=open(fname,'rb')  

     data=f.read()  

     f.close()  

     data=zlib.decompress(data)  

     if evaluate: data=eval(data)  

     return data  

 

Oh yeah, don’t forget the evaluate option in the openIt function. If set to True (default), the returned variable will be an evaluated object. For example, “[[1,2],[3,4]]” will be returned as an actual 2D list, not just a string. How convenient is that?


     

Run Ubuntu Live CD From a USB Drive

I accidentally nuked my laptop’s 80G hard drive this morning (D’OH!) while shuffling around partitions. Supposedly there’s a valid windows (XP) installation on there still that’s about 20G. I’d love to repair it so I can use it today while I’m in the confocal room, but I don’t have an Ubuntu CD, Windows CD, or any CD for that matter! I looked around, but I guess blank CD-Rs aren’t something that’s standard in molecular biology laboratories. Anyhow, I wanted to install the new Ubuntu 8.10 Linux distribution, and I’ve downloaded the ISO, but since I can’t find a CD to burn it to I decided to try booting from a USB drive (something I’ve never done before). I found an AWESOME program which specialized in putting ISO files onto bootable USB drives. It’s called UNetBootin and it’s free (of course), runs on Linux or Windows, and has some built-in options for various linux distributions. I can repair my PC now! Yay!


     

Linear Data Smoothing in Python

Here’s a scrumptious morsel of juicy python code for even the most stoic of scientists to get excited about. Granted, it’s a very simple concept and has surely been done countless times before, but there aren’t any good resources for this code on the internet. Since I had to write my own code to perform a variety of different linear 1-dimensional array data smoothing in python, I decided it would be nice to share it. At the bottom of this post you can see a PNG image which is the file output by the code listen even further below. If you copy/paste the code into an empty text file and run it in Python, it will generate the exact same PNG file (assuming you have pylab and numpy libraries configured).

  

 ### This is the Gaussian data smoothing function I wrote ###  

 def smoothListGaussian(list,degree=5):  

     window=degree*2-1  

     weight=numpy.array([1.0]*window)  

     weightGauss=[]  

     for i in range(window):  

         i=i-degree+1  

         frac=i/float(window)  

         gauss=1/(numpy.exp((4*(frac))**2))  

         weightGauss.append(gauss)  

     weight=numpy.array(weightGauss)*weight  

     smoothed=[0.0]*(len(list)-window)  

     for i in range(len(smoothed)):  

         smoothed[i]=sum(numpy.array(list[i:i+window])*weight)/sum(weight)  

     return smoothed  

 

Basically, you feed it a list (it doesn’t matter how long it is) and it will return a smoother version of the data. The Gaussian smoothing function I wrote is leagues better than a moving window average method, for reasons that are obvious when viewing the chart below. Surprisingly, the moving triangle method appears to be very similar to the Gaussian function at low degrees of spread. However, for huge numbers of data points, the Gaussian function should perform better.

  

 ### This is the code to produce the image displayed above ###  

 import pylab,numpy  

     

 def smoothList(list,strippedXs=False,degree=10):  

     if strippedXs==True: return Xs[0:-(len(list)-(len(list)-degree+1))]  

     smoothed=[0]*(len(list)-degree+1)  

     for i in range(len(smoothed)):  

         smoothed[i]=sum(list[i:i+degree])/float(degree)  

     return smoothed  

   

 def smoothListTriangle(list,strippedXs=False,degree=5):  

     weight=[]  

     window=degree*2-1  

     smoothed=[0.0]*(len(list)-window)  

     for x in range(1,2*degree):weight.append(degree-abs(degree-x))  

     w=numpy.array(weight)  

     for i in range(len(smoothed)):  

         smoothed[i]=sum(numpy.array(list[i:i+window])*w)/float(sum(w))  

     return smoothed  

   

 def smoothListGaussian(list,strippedXs=False,degree=5):  

     window=degree*2-1  

     weight=numpy.array([1.0]*window)  

     weightGauss=[]  

     for i in range(window):  

         i=i-degree+1  

         frac=i/float(window)  

         gauss=1/(numpy.exp((4*(frac))**2))  

         weightGauss.append(gauss)  

     weight=numpy.array(weightGauss)*weight  

     smoothed=[0.0]*(len(list)-window)  

     for i in range(len(smoothed)):  

         smoothed[i]=sum(numpy.array(list[i:i+window])*weight)/sum(weight)  

     return smoothed  

   

 ### DUMMY DATA ###  

 data = [0]*30 #30 "0"s in a row  

 data[15]=1    #the middle one is "1"  

   

 ### PLOT DIFFERENT SMOOTHING FUNCTIONS ###  

   

 pylab.figure(figsize=(550/80,700/80))  

 pylab.suptitle('1D Data Smoothing', fontsize=16)  

   

 pylab.subplot(4,1,1)  

 p1=pylab.plot(data,".k")  

 p1=pylab.plot(data,"-k")  

 a=pylab.axis()  

 pylab.axis([a[0],a[1],-.1,1.1])  

 pylab.text(2,.8,"raw data",fontsize=14)  

   

 pylab.subplot(4,1,2)  

 p1=pylab.plot(smoothList(data),".k")  

 p1=pylab.plot(smoothList(data),"-k")  

 a=pylab.axis()  

 pylab.axis([a[0],a[1],-.1,.4])  

 pylab.text(2,.3,"moving window average",fontsize=14)  

   

 pylab.subplot(4,1,3)  

 p1=pylab.plot(smoothListTriangle(data),".k")  

 p1=pylab.plot(smoothListTriangle(data),"-k")  

 pylab.axis([a[0],a[1],-.1,.4])  

 pylab.text(2,.3,"moving triangle",fontsize=14)  

   

 pylab.subplot(4,1,4)  

 p1=pylab.plot(smoothListGaussian(data),".k")  

 p1=pylab.plot(smoothListGaussian(data),"-k")  

 pylab.axis([a[0],a[1],-.1,.4])  

 pylab.text(2,.3,"moving gaussian",fontsize=14)  

   

 #pylab.show()  

 pylab.savefig("smooth.png",dpi=80)  

 

Hey, I had a great idea, why don’t I test it on some of my own data? Due to the fact that I don’t want the details of my thesis work getting out onto the internet too early, I can’t reveal exactly what this data is from. It will suffice to say that it’s fractional density of neurite coverage in thick muscle tissue. Anyhow, this data is wild and in desperate need of some smoothing. Below is a visual representation of the differences in the methods of smoothing. Yayness! I like the gaussian function the best.

I should note that the degree of window coverage for the moving window average, moving triangle, and gaussian functions are 10, 5, and 5 respectively. Also note that (due to the handling of the “degree” variable between the different functions) the actual number of data points assessed in these three functions are 10, 9, and 9 respectively. The degree for the last two functions represents “spread” from each point, whereas the first one represents the total number of points to be averaged for the moving average. Enjoy.


     

Free Damask Seamless Tiling Backgrounds

If you’re in the mood for some 18’th century textile patterns you’ve stumbled upon the right place! Surprisingly, it’s incredibly difficult to find functional (seamless, tiling, free) damask-style patterns on the internet. If you don’t believe me, just Google / image search for it! It took me over an hour to find a functional pattern that tiled properly. Actually, to correct myself there, the image I downloaded didn’t even tile correctly!!! I had to manually modify it to make it seamless. So, free for all website makers, webmasters, wallpaper collectors, and Louis XVI enthusiasts: I give you a plethora of different colors of damask-style tiling backgrounds for whatever you want to do with it!