Resuming Life

I’m staring deeply into a large, ominous, empty white text box on my screen as a small vertical text cursor blinks, staring back at me. I faintly remember writing my first blog entry eleven years ago (to the week), as a 15 year old boy typing on his keyboard in the middle of the night. Every time I log-in to write on my website, I see the same cursor, monotonous in its dependable, unvarying appearance.  What does the cursor see in me? Definitely not the same person it saw the last time I logged-in. What collection of words can even begin to describe the jumble that’s in my head? My name is Scott Harden, my future is changing before my eyes, and I’ve chosen to use this website to continue to document my life as it unfolds.

It’s been 298 days since I’ve last written. For anyone who’s been through a hard time, part of getting better is acting like ones self again. For me, that’s doing the things that I used to do: designing circuits, writing code, building radios, and documenting my projects on my website. Me, starting to write again – forcing myself to write again – is a step in a positive direction, and an indication that I am starting to be okay. My intention is to resume building and sharing projects on this website like I used to.

I feel it is important to address some of the recent changes in my life in a clear manner. I refuse to resume posting code and pictures of circuits all the sudden without acknowledging the serious issues that I’ve dealt with and am continuing to deal with. I choose not to ignore these things – a decision made out of respect, and out of logic. It’s important that it’s obvious that (a) these things happened, (b) I’m working to be okay with them, and (c) I’m not ignoring them. I have noticed that ignoring serious issues by pretending that they do not exist is one of the most dangerous and destructive coping mechanisms a person can exhibit. People behave this way around me often, but I, in an effort to improve my situation, refuse to ignore my challenges.

I will attempt to describe a few of the highlights of the last nine months of my life. It’s certainly not an exhaustive list, but it touches on a few of the significant experiences I have gone through, changes which have affected me, and a few of the events that made me happy since I last wrote here.

I began a new Ph.D. program in Neuroscience through the Interdisciplinary Program in Biomedical Science through the College of Medicine at the University of Florida. I am currently in my 4th year of dental school (in my 10th year of colletge currently), and I’m considering starting a new program from scratch. Rather than throw-away my dental degree (which I’ve been working on for several years and spent quite a lot of money to pursue), I am seeking a combined D.M.D/Ph.D. degree. This will allow me to graduate with a degree in dental medicine (D.M.D.) through the UF College of Dentistry about the same time I get my Ph.D. in Neuroscience through the UF College of Medicine. I begin this August, and I couldn’t be more excited. As my Google Scholar author page reports, I’ve had a lot of excellent experience publishing research in the past, and I’m eager to begin life again as a biomolecular graduate student. I have yet to decide the specific direction of my research, but if I could do anything in the world, I would try to find a project which lets me dip into bioinformatics. The pictures below are of me on the day of my PhD interview. I think I nailed it 🙂

It looks like I’m kind of sick. To make a long story short, I’m in a strange diagnostic limbo where it’s looking like I may have a pretty serious, rare, and somewhat difficult to diagnose form of cancer (stage 4 peripheral T-cell lymphoma), but at the same time it seems there is a chance it could be a weird, rare ailment that’s not as serious (an obscure autoimmune condition or abnormal presentation of chronic infection). The type of cancer it is currently proposed to be is pretty bad, and people usually don’t live too long after their diagnosis. Intriguingly, my symptoms and clinical presentation don’t reflect the rapidly destructive nature commonly associated with this disease, leaving open the possibility that it might not be that form of cancer after all. I have hard, enlarged lymph nodes (with abnormal histology demonstrated by 1 needle biopsy and 2 separate surgeries) in all quadrants of my body, and have a lot of fatigue (which I combat with large volumes of coffee). Other than that, I’m relatively normal (physically, at least). The picture to the right is me on the first visit to the oncologist. I didn’t take it too seriously. I kept thinking (and still do to some degree) there’s no way it could be that bad.  I mean, who my age gets sick like this? And right on the heels of losing my wife? It’s too coincidental, it can’t be that serious! Realistically we still don’t know for sure exactly what it is, and it’s too early to tell how this will affect my life. It’s a strange experience grappling with the idea that I might only live a few more years, knowing at the same time that my cancer might not be that bad and I may have a long, relatively normal life. I will know more a few months from now. In the mean time, I’m content planning my life as if I will be completely fine (which is certainly may be the case), and if I get sick along the way I’ll re-evaluate my situation if the need arises. For now, I’m quite positive. If anyone were to get a one in a million disease and get through it unscathed, it would be me!

Here are a few relevant images from my medical experiences within the last few months. The left image is my PET CT, which depicts active metabolism. I was injected with radioactive glucose, and the tissues that work hard absorb it and appear as “hot” on the image. I was lying down (for about 30 minutes), so the only active organs should be my heart and brain. The kidneys/bladder are illuminated because it filtered some of the material from the blood. Actively growing tumors also uptake glucose, and are labeled on this image. While the majority of my tumors are benign and slow growing, this image shows a few rapidly growing tumors: one on my upper right leg (which was removed with surgery #1, second picture), one small one at the base of my neck (which was removed with surgery #2, third picture), and a few scattered ones under each of my arms. The active ones were removed to aid in diagnosis, and it’s not feasible to remove all of the affected lymph nodes. The last picture shows me after my most recent surgery coming out of anesthesia. I don’t even remember it being taken – I was pretty high at the time, as you can infer from the photo 🙂

Many people have asked me how I am doing after the death of my wife Angelina. Unexpectedly losing a spouse at any age is difficult, but at such a young age (I was 25, she was 24, we were married for 4 years) it’s even harder to rationalize sometimes. No words I write can even begin to describe the complex array of emotions I carry. Although I wish things had turned-out differently, I still feel fortunate to have had Angelina in my life, and I know unquestionably that I am a better person for having known her, and am thankful we got to share as much as we did. The picture on the right is the last one I have of us together. (I’m in the blue shirt, she’s in the black shirt). It was taken about two weekends before she died. We went with a group of my dental school classmates to Islands of Adventure (part of Universal Studios) in Orlando. If you had told me then that I would be a widower in a couple weeks, I couldn’t have even begun to comprehend it. I will always miss her, but I know that she would not want me to destroy my life because she is gone. In a note she wrote for me the day she died, she said “I love you, Scott, more than you will ever know.” It’s a phrase I remember and repeat in my mind every day, and it’s given me comfort over the last several months. I know that I will always carry part of her with me.

Several people have e-mailed me in concern due to the fact that I had stopped posting over the last few months. Again, I was surprised and honored the unexpected support I’ve been getting over the internet from people I don’t really even know. The following quote is one that came in this morning.

It has been a while since I have last visited and I really hope you’re doing OK since your last post. I can’t pretend to remotely know how you feel, even with the time that has passed since then. Just wanted to say I wish you well and miss your posts. All I can say is that I know that there have been periods of time in my life where I put down the soldering iron due to life… even for a couple of years. But, I always come back around because its part of who I am… and I hope it will be for you as well.

–Jeremy

The truth is, when Angelina died, life stopped. I stopped answering email, I stopped adding to my website, I stopped making phone calls, I even distanced myself from my own family for a while. Slowly, one by one, I’m resuming these activities. A few months ago, I unpacked my electronics workstation in my new apartment. It was a little premature, as I only worked at my station 1 day and took a few more months before I came back to it, but now it’s a weekly process. For a long time I felt guilty even thinking about picking-up my silly projects again. I felt that, after everything she and I had gone through, it seemed fickle. With a little time and some self-reflection, I realized that Angelina encouraged me to do these things throughout our marriage. I remember her telling me that she went to college and bragged “my husband is on hack-a-day“. I told her that my academic publications on PubMed are more impressive, and I’ll always remember her response: “Yeah, but you get more excited every time you make it on hack-a-day”. I remember this and remind myself that, if she were here, I think she’d still continue to encourage me to do what I enjoy. While silly little projects might not seem significant, they do make me happy, and I think she would be proud of me for working on them.

Angelina left a few special reminders on my workbench. A few months after she died, and as I would unpacking my workstation again, I noticed a carving she made in my workbench. I almost threw away this table when I moved, and I’m so glad I didn’t. I think Angelina carved it with a car key, probably on the day this post was made, when I left her unattended for a while soldering at my station. I also found a message she wrote on my roll of solder. I didn’t remember seeing it before, so I imagine it was something she wrote the last few days she was alive. It is really special to me.

 One of the things I’ve done that’s really helped me a lot is to get out of my comfort zone a little bit and go places and do things I’ve never done before, often on a whim. This has led me to have a lot of a lot of fantastic experiences, and give me some new things to occupy myself with and remind me that there’s still a lot of life out there left to live. Over the last several months, I tried to minimize the amount of time I spent doing things I’d done before. I noticed that the more time I spent doing the same old things, the more I felt like a part of me was missing, and the worse I felt. Changing scenery and being in new places, it felt less like something was missing, and began letting me establish the feeling of being okay on my own. I stayed away from Gainesville as much as I could, stayed away from family as much as I reasonably could (sorry mom and dad if you read this one day), and kept the amount of time I spent interacting with my old friends from dental school to a minimum. For some reason, nothing made me feel worse inside than being around my old environment, and avoiding it for a while was my way of trying to heal. For several months, this behavior was the only thing that made me feel better. Luckily, within the last few weeks, things have started to settle down, and I’m a little more comfortable settling-back into some of my old environments. I’m going to toss-in a sizable collection of random photos from the last several months. I’m not going to describe them in detail – it’s more an indication that yes, I’m alive, and yes, I’m getting out there a little. I’ll post them in little clusters with descriptions in between…

I marched in the UF homecoming parade with the University of Florida Gator Amateur Radio Club. I participated in the “out of the darkness” walk for suicide awareness sponsored by the Suicide Survivors of Northeast Florida group. I volunteered to operate as net control for Gator Amateur Radio Society during the 5 points of life “relay for life” marathon (official site). I spent some time in Georgia and caught a sunrise on the east coast of Florida over the ocean.

I practiced advanced suturing techniques utilizing a pig jaw. I actually messed it up pretty badly… oops? Next time I’ll do it on a person. Scary, right? I took an awesome photo of the dental school early in the morning, and the radio antennas are visible on the top. I have a picture at the 2011 “art walk” in downtown Gainesville, Florida. A few days after my wife died, I got a new apartment and moved in with a dental school classmate. His name is Ray, and he’s a pretty cool guy. I’m very lucky to have him. He moved out of his old apartment and into my apartment in less than a week’s notice. I spent some time in Georgia and had the opportunity to operate my Yaesu 857-D portable. I have a 25 foot boom with an inverted-vee wire antenna fed by 300-ohm ladder line. I worked Italy on 40m (7MHz) CW (Morse code) on 20 watts.

I visited the base of the Appalachian Trail in Georgia, Bok Tower in Lake Wales, and a high-rise retirement home in Tampa. Another photo is of me with my two sisters (who were obviously adopted). You can see me operating as net control for the 5-points of life relay marathon early in the morning. Also there is a picture with me beside my friend Bill, W4HBK. He and his wife were wonderfully kind in offering me a place to stay over a portion of the Christmas break. Bill is a fellow QRSS’er (who mans the Pensacola Snapper) and we had a great time working with antennas, looking at QRP equipment, and testing 160M WSPR. It seemed like a random thing for me to do (drive 6 hours to spend several days with a family I hadn’t met before), but was a wonderful experience and I am thankful to both of them.

Here are some more photos of me working as net control for the race. I also went shooting with a sharpshooters group at a shooting range after hours. That’s a scoped 22, and it’s accurate as heck. It was pretty fun to shoot. I also began working on some new micro-controller projects, and I even purchased an arduino. I’ve had the arduino for a few months and haven’t used it for anything yet. Between you and me, it’s an annoyance since I’m so used to programming these ATMEL microcontrollers in AVR-GCC, but I figure that I should do at least *one* project with Arduino for the sake of doing it. It’s what everyone is using these days.

I started getting into electronics again a couple months ago. I learned how to etch PCBs at home (using the toner transfer method with hot ferric chloride) and am making digital QRSS transmitter designs. My signal is the one that looks drunk, swerving all over the QRSS road. The picture of me by a fence was in Ocala at a gokart racetrack. One of my professors took a few students, and it was a pretty cool day. It was the week I found out I was sick.

There I am shooting again at a friends house. I don’t remember the details of that weapon (I shot quite a few that day), but I think it was a 9mm with a red-dot scope. I spent a little more time in the wilderness up in Georgia, and had some time to stop by Georgia Tech two days ago. I visited to watch a friend’s friend give her master’s thesis defense. She did really well, and while I was there I snapped a picture of the campus indicating the proximity of the Georgia Tech Amateur Radio Club (W4AQL) to the Coca-Cola factory. Also pictured is my QRSS transmitter, a somewhat novel design using no potentiometers. The oscillator is a 74HC240 with a varactor diode, allowing frequency adjustment via a potentiometer, and a lowpass-filtered PWM line from the microcontroller to provide modulation. The degree of modulation is adjustable via the second potentiometer.

The nighttime image is a small restaurant a few friends took me to. The lights were spectacular. I’m also eating a frozen banana. I get my hair cut in my friend’s kitchen. His name is Don and he is a dental student too. My hair cuts are $5. I visited the coast and have a strange picture of me touching my hair. I operated the W4DFU radio station during the CQWW contest. I also visited a museum. Can you believe I went my entire life without ever actually seeing a dinosaur skeleton in person?

The photo from space indicates a successful retrieval of a camera that was launched back in August. It was placed on a high altitude balloon launched the same day Angelina and I drove to Orlando for the weekend (the next day the photo of us above was taken). It took photos as it ascended, then landed, and it was lost for several months. Some boys playing in the woods found it and gave it to their mom, who contacted us. The photos are pretty impressive, and I’ll post details in a future entry. There’s another photo of me on a beach, and one of me at a jellyfish exhibit. I look funny in the car playing with my camera. I recently realized that if I pushed the correct button, there was a camera facing me that could take a picture. Also there’s a photo of me at an old artillery battle fortress in Pensacola, Florida. I’m not much of a history buff, I’m not sure what it’s formal name is.

It’s nice being in the woods. A friend of my owns a house far enough out of town that we can shoot in the back yard. Bok tower in Lake Wales as seen from beneath a tree. I went there for a carillon concert one night after the sun went down. It was absolutely beautiful. I visited a small Irish pub in North Florida one day and listened to a small guitar player. It was so sparsely populated that, at one point, I think my table was the only one occupied. It was nice though. The picture of me sitting on the couch is from Christmas at a big family get-together. That was an experience. Like I mentioned above, my wife’s death hurts the most when I’m in a place where I feel she should be. When I do “old things”, and go to the places we used to go together, and be in situations where we always were together, her absence is unignorably overwhelming and painful. I imagine holidays will continue to hurt for a while. I don’t know how I got this photo, did my sister take it? Anyhow, that’s that. In another photo my fiend Ben sits at my workstation in one of the photos. Occasionally I invite engineering students to spend time at my apartment. With all the silly old electrical equipment I’ve amassed over the years, my apartment is turning into quite the electronics workshop! I’m posing in a tux with my sister, sporting a fine mustache.

At the March for Babies walk in Gainesville, Florida (where I served again as net control station for the event), there was a military representative who brought a rock climbing wall. I made it up two of the three paths. That third one was rough, and I never got to the top. My sister got married a few weeks ago. She went from Kelly Harden to Kelly Tran, and I got to be at the wedding. Actually, I was part of the wedding. I read Corinthians 13 during the service. It’s commonly known for “Love is patient, love is kind…“, but there were some ironic parts of the chapter that were a little hard to read: “If I have everything but do not have love, I am nothing” (1-3), “love always perseveres” (7), and “Love never fails” (8). I took it as a personal challenge, and made myself to be okay with the way life is. The harder something is for me emotionally, the more I feel like I’ve grown after getting through it. There’s another photo of me at my workstation. I don’t think I knew it was being taken. There’s a picture of a poster I found in a shoe store. The message was pretty cool, so I snapped a picture. One of the days working in the dental clinic, let’s just say I had “computer problems”.

Kicking-off the transition back into internet productivity, I will share some code which I utilized to create today’s entry. I wanted to rapidly browse through numerous high resolution images to choose which ones to share, but pressing the left and right arrows was too slow (some of the large images took 5+ seconds to load). I therefore copied all of the images (*.jpg) created within the last 6 months and copied them into a single folder. I whipped-up a script to use ImageMagick’s convert feature (which is available for linux and windows) to create 300px wide thumbnails and place them on a static HTML page for viewing in a browser. Each thumbnail was clickable, revealing the original, large image. Below is the script I used to do this.

import os
out="<html><body>"
fnames=os.listdir("./")
fnames.sort() # alphabetize them
for fname in fnames:
if ".jpg" in fname or ".JPG" in fname:
if "sml_" in fname: continue
cmd="convert -resize 300 %s sml_%s"%(fname,fname)
print cmd #note that this requires ImageMagik installed
os.system(cmd)
out+='<a href="%s">'%(fname)
out+='<img src="sml_%s"></a><br>'%(fname)
out+='%s<br><br><br>n'%(os.path.split(fname)[1])
out+="</body></html>"
f=open("pics.html",'w')
f.write(out)
f.close()
print "DONE"

What do I plan to do from here? I’m starting my PhD program in August, and I plan to hit it hard. I feel lucky that I have the opportunity to study biomolecular science again. I’m holding my breath for it to start, and am looking forward to having something academically and intellectually challenging to devote myself to. I plan to continue making my electronics and programming projects, and publishing them here on my website. If my health changes, it all might change, but I’m holding out that I’ll be okay.  All in all, I’ve lived through an unbelievably rough year, but I look forward to getting back into the game and pushing into life head first.


     

Generate Apache-Style HTTP Access Logs via SQL and PHP

Does your web hosting company block access to access.log, the text file containing raw website log files? If so, you’re like me, and it sucks. There’s a plethora of gorgeous and extremely insightful website traffic analyzers, but all of them require access to raw HTTP access logs. Today I propose a semi-efficient way to generate such logs utilizing PHP to determine page load data (time, user IP, requested page, referring page, user client, etc) and SQL to save such data for easy retrieval later. Note that this method is a HUGE improvement of my previous project which used PHP scripts to store HTTP access logs as flat files. Although it worked in theory, in all practicality the process of opening, writing to, and closing a text file (which grew a few MB a week) was too cumbersome for my server to comfortable handle. The method described on this page utilizes SQL, a database engine well-suited to meet these exact demands. When we’re done, you’ll be able to use a web interface to view your access log (pictured, converting long, complicated search queries to web search and image search strings automatically), or have the option to export it directly to an access.log text file in a standard Apache-style format.
sql_php_http_log_viewer

First, make sure your database is structured appropriately. This page is written for those with a working knowledge of PHP and SQL, but if you’re new to the field I encourage you to learn! W3Schools.com is an awesome resource to rapidly learn new languages. Also, when starting-out with SQL (like me), phpMyAdmin is a awesome. The code, as it’s currently written (below) is designed to store data in the “nibjb” database under the “logs” table. Briefly, it uses PHP to determine user data (time, ip, requested page, etc.) and injects this information into the SQL database. In fact, it’s doing it to you right now! Don’t believe me? View the source of this web page and scroll to the bottom. BAM! There you are.

// logme.php
<?php

if ( !isset($wp_did_header) ) {
	$wp_did_header = true;
	require_once( '/home/content/n/i/b/nibjb/html/blog/wp-load.php' );
	//wp();
	//require_once( '/home/content/n/i/b/nibjb/html/blog/wp-includes/template-loader.php' );
}

function logwriter_handlevar($varname,$defaultvalue){
    $tempvar = getenv($varname);
    if(!empty($tempvar)) {
        return $tempvar;
    } else {
        return $defaultvalue;
    }
}

if (!empty($REMOTE_HOST)) {
$logwriter_remote_vistor = $REMOTE_HOST;
}else{
$logwriter_remote_vistor = logwriter_handlevar("REMOTE_ADDR","-");
}

$logwriter_remote_ident = logwriter_handlevar("REMOTE_IDENT","-");
$logwriter_remote_user = logwriter_handlevar("REMOTE_USER","-");
$logwriter_date = date("d/M/Y:H:i:s");
$logwriter_request_method = logwriter_handlevar("REQUEST_METHOD","GET");
$logwriter_request_uri = logwriter_handlevar("REQUEST_URI","");
$logwriter_server_protocol = logwriter_handlevar("SERVER_PROTOCOL","HTTP/1.1");
$logwriter_http_referer = logwriter_handlevar("HTTP_REFERER","-");
$logwriter_http_user_agent = logwriter_handlevar("HTTP_USER_AGENT","");
$logwriter_logstring = "$logwriter_remote_vistor $logwriter_remote_ident $logwriter_remote_user [$logwriter_date $logwriter_timezone] "$logwriter_request_method $logwriter_request_uri $logwriter_server_protocol" 200 - "$logwriter_http_referer" "$logwriter_http_user_agent"n";
?>

<?php
$username="YOUR_USERNAME";
$password="YOUR_PASSWORD";
$database="nibjb";
mysql_connect('mysql157.secureserver.net',$username,$password);
//mysql_connect(localhost,$username,$password);

$query = "INSERT INTO logs VALUES ('','$logwriter_date','$logwriter_remote_vistor','$logwriter_request_method','$logwriter_request_uri','$logwriter_server_protocol','$logwriter_http_referer','$logwriter_http_user_agent')";
mysql_query($query);
mysql_close();
?>

<!--
LOG DETAILS:
time: <?php echo($logwriter_date); ?>
vistor: <?php echo($logwriter_remote_vistor); ?>
method: <?php echo($logwriter_request_method); ?>
request: <?php echo($logwriter_request_uri); ?>
protocol: <?php echo($logwriter_server_protocol); ?>
referrer: <?php echo($logwriter_http_referer); ?>
agent: <?php echo($logwriter_http_user_agent); ?>
HTML LOG LINE:
<?php echo($logwriter_logstring); ?>
 -->

All right, that was easy. Every time we load logme.php, it adds the data to the SQL database. To add data every time you go to a particular web page, you could use a PHP include() statement in each webpage, or you could take advantage of the PHP’s auto_append_file feature! Simply insert the following line into your php.ini file if you have access to yours:

auto_append_file = "/path/to/html/logme.php"

How do we access this data once it’s been loaded into the database? There are many different ways, but I’ve chosen to get a little creative with a sleek, yet minimalistic web-based fronted. It basically just shows the last [x] number of entries in the access log. You can adjust the number of entries displayed by slapping on some arguments to the URL, transforming viewLast.php into viewLast.php?limit=123 or something (see the screenshot above). I won’t discuss the details of this script. It’s self-explanatory.

// viewLast.php
<html>
<head>
<style type="text/css">
td {
font-family: verdana, arial;
font-size:10px;
}
</style>
</head>
<body>
<?php

$limit = (int)$_GET['limit'];
if ($limit===0) {$limit=25;}

$username="YOUR_USERNAME";
$password="YOUR_PASSWORD";
$database="nibjb";
mysql_connect('mysql157.secureserver.net',$username,$password);
mysql_select_db($database) or die( "Unable to select database");
$query="
SELECT * FROM logs WHERE
request NOT LIKE "%testlog.php%"
AND request NOT LIKE  "%/logs/%"
AND request NOT LIKE "%/wp-admin/%"
ORDER BY ID DESC LIMIT 0,$limit
";
//$query="SELECT * FROM logs WHERE referrer LIKE "%&q=%" or referrer LIKE "%&prev=%" ";
$result=mysql_query($query);
$num=mysql_numrows($result);
mysql_close();
?>

<b><?php echo($query); ?></b>
<table border="1">
<tr>
<td>id</td>
<td>time</td>
<td>visitor</td>
<td>request</td>
<td>referrer</td>
</tr>

<?php
$i=1;
while ($i<$num) {
$id=mysql_result($result,$i,"id");
$time=mysql_result($result,$i,"time");
$visitor=mysql_result($result,$i,"visitor");
$method=mysql_result($result,$i,"method");
$request=mysql_result($result,$i,"request");
$protocol=mysql_result($result,$i,"protocol");
$referrer=mysql_result($result,$i,"referrer");
$referrer2=str_replace("&", "& ", $referrer);
$agent=mysql_result($result,$i,"agent");
$searchWords="";
$searchEngine="";
if (strpos($referrer, "q=")>0 and strpos($referrer, "google")>0) {$searchEngine="Google Web Search: ";}
if (strpos($referrer, "prev=/images")>0 and strpos($referrer, "google")>0) {$searchEngine="Google Image Search: ";}

// SEARCH EXTRACTION //
$j=0;
$rTemp=str_replace("prev=/images%3Fq%3D", "q=", $referrer);
$rTemp=str_replace("?q=","&q=", $rTemp);
$rTemp=str_replace("%2B"," ", $rTemp);
$rTemp=str_replace("%26"," ", $rTemp);
$rTemp=str_replace("%3D"," ", $rTemp);
$rTemp=str_replace("+"," ", $rTemp);
$wvars=split("&",$rTemp);
while ($j<count($wvars)){
	if (substr($wvars[$j],0,2) === "q=") {
		$searchWords = $searchWords . $wvars[$j] . " ";
		}
	$j++;
}

$searchWords=substr($searchWords,strpos($searchWords, "q=")+2);
if (strlen($searchWords)<3) {$searchWords=$referrer;}
////////////////////////

echo "
<tr>
<td>$id</td>
<td>$time</td>
<td>$visitor</td>
<td><a href='$request'>$request</a></td>
<td>$searchEngine <a href='$referrer'>$searchWords</a></td>
</td>
";
$i++;
}
?>
</table>
</body>
</html>

And you’re done! This example is a simplified, bare bones example. You can take this a long way if you’d like. My goal is lite & flexible. A quick query from Python and Matplotlib (for example) yields gorgeous visual representations of otherwise-convoluted data!

If you have any questions, or end-up developing something awesome with this code, shoot me an email! It’s not luxurious, but this code works for me, and I share it with the best of intentions.


     

Transition

I’m briefly suspending entries on this website. I currently have no projects I’m working on, and I’m going to try to keep it that way for a few weeks. I really need to re-gear my brain and get ready for dental school next month. I’m struggling with a plethora of random emotions, and I think the best thing for me is to take it easy for a little bit and try to let go of the things I feel are important to me (projects, electrical, mechanical, computational, painting, or otherwise). I’m going to try my best to organize data from my past life (about a decade worth) in an attempt to preserve it. I’ve been thrown back into my early teen years by uncovering ~10 GB of music I used to listen to. Nostalgia? Yeah, I’m feeling it. I had totally forgotten about random, obscure Japanese bands such as Rip Slyme. For example, Hot Chocolate [a must hear / must see youtube video]. In fact, [youtubes some more], check out this randomness [embeds below]. I love non-mainstream awkwardness. What’s that I hear? 8-bit tones?


     

Removing Textile From WordPress

I realized that the C code from yesterday wasn’t showing-up properly because of textile, a rapid, inline, tag-based formatting system. The app converted blog code from [“text”:http://www.SWHarden.com/ *like* _this_] to [text like this. ] While it’s fun and convenient to use, it’s not always practical. The problem I was having was that in C code, variable names (such as _delay_) were becoming irrevocably italicized, and nothing I did could prevent textile from ignoring code while styling text. The kicker is that I couldn’t disable it easily, because I’ve been writing in this style for over four years! I decided that the time was now to put my mad Python skills to the test and write code to handle the conversion from textile-format to raw HTML.
I accomplished this feat in a number of steps. Yeah, I could have done hours of research to find a “faster way”, but it simply wouldn’t have been as creative. In a nutshell, I backed-up the SQL database using PHPMyAdmin to a single “x.sql” file. I then wrote a pythons script to parse this [massive] file and output “o.sql”, the same data but with all of the textile tags I commonly used replaced by their HTML equivalent. It’s not 100% perfect, but it’s 99.999% perfect. I’ll accept that. The output? You’re viewing it! Here’s the code I used to do it:

## This Python (1.0) script removes *SOME* textile formatting from WordPress
## backups in plain text SQL format (dumped from PHP MyAdmin). Specifically,
## it corrects bold and itallic fonts and corrects links. It should be easy
## to expand if you need to do something else with it.
## Enjoy! --Scott Harden (www.SWHarden.com)

infile = 'x.sql' # < < THIS IS THE INPUT FILE NAME!

replacements=   ["r"," "],["n"," n "],["*:","* :"],["_:","_ :"],
                ["n","<br>n"],[">*","> *"],["*< ","* <"],
                [">_","> _"],["_< ","_ <"],
                [" *"," <b>"],["* "," "],[" _"," <i>"],["_ ","</i> "]
                #These are the easy replacements

def fixLinks(line):
    ## replace ["links":URL] with [<a href="URL">links</a>]. ##
    words = line.split(" ")
    for i in range(len(words)):
        word = words[i]
        if '":' in word:
            upto=1
            while (word.count('"')&lt;2):
                word = words[i-upto]+" "+word
                upto+=1
            word_orig = word
            extra=""
            word = word.split('":')
            word[0]=word[0][1:]
            for char in ".),'":
                if word[1][-1]==char: extra=char
            if len(extra)>0: word[1]=word[1][:-1]
            word_new='<a href="%s">%s</a>'%(word[1],word[0])+extra
            line=line.replace(word_orig,word_new)
    return line

def stripTextile(orig):
    ## Handle the replacements and link fixing for each line. ##
    if not orig.count("', '") == 13: return orig #non-normal post
    line=orig
    temp = line.split
    line = line.split("', '",5)[2]
    if len(line)&lt;10:return orig #non-normal post
    origline = line
    line = " "+line
    for replacement in replacements:
        line = line.replace(replacement[0],replacement[1])
    line=fixLinks(line)
    line = orig.replace(origline,line)
    return line

f=open(infile)
raw=f.readlines()
f.close
posts=0
for raw_i in range(len(raw)):
    if raw[raw_i][:11]=="INSERT INTO":
        if "wp_posts" in raw[raw_i]: #if it's a post, handle it!
            posts+=1
            print "on post",posts
            raw[raw_i]=stripTextile(raw[raw_i])


print "WRITING..."
out = ""
for line in raw:
    out+=line
f=open('o.sql','w')
f.write(out)
f.close()

I certainly held my breath while the thing ran. As I previously mentioned, this thing modified SQL tables. Therefore, when I uploaded the “corrected” versions, I kept breaking the site until I got all the bugs worked out. Here’s an image from earlier today when my site was totally dead (0 blog posts)

hostingwork


     

PHP-Generated Access.log is a Success

THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!

>>> CHECK OUT THE NEW CODE < << [Generate Apache-Style HTTP Access Logs via SQL and PHP]

OBSOLETE CODE IS BELOW…

A few months ago I wrote about a way I use PHP to generate apache-style access.log files since my web host blocks access to them. Since then I’ve forgotten it was even running! I now have some pretty cool-looking graphs generated by Python and Matplotlib. For details (and the messy script) check the original posting.

This image represents the number of requests (php pages) made per hour since I implemented the script. It might be a good idea to perform some linear data smoothing techniques (which I love writing about), but for now I’ll leave it as it is so it most accurately reflects the actual data.


     

Is Google Re-Indexing My Life?

After several years of persistent writing on this website I was forced (by my undergraduate university’s difficult course loads) to stop adding to this blog – something I consider to be one of the most significant projects I’ve ever worked on, with brain-to-text recordings of my thoughts spanning almost a decade of time. After a few years of suspended writing, Google went from loving me (sending me thousands of pageviews daily) to forgetting about me (nothing. silence. nada.). Now that my thesis requirements have been completed, I’m trying to re-energize my writing in an attempt to document the projects I work on which, without this website, would likely be forever forgotten even by me. It appears that the burst of new writing has regained Google’s attention. Google for terms such as “data smoothing in python” and it favors my site. Google is slowly, but surely, re-indexing my pages and assigning them values of relevance which are approaching (but still a tiny fraction of) what they were before my hiatus. Here’s a chart from google’s analytics demonstrating an estimation of IP visits per day (visitors) and their locations. Do I have fans in South Africa? I didn’t know they had computers in South Africa! (I’m sorry if you are that person in South Africa, and were offended by that statement)


     

Analyzing my Writings with Python

*I spent the day* in the lab with some random time on my hands between adding reagents to an ongoing immunohistochemical reaction I was performing. At one point I decided to further investigate the field of bioinformatics (is it worth seeking a PhD in this field if I don’t get into dental school again?). UCF offers a PhD in bioinformatics but it’s a new and small department (I think there are only 4 faculty). The degree itself is a degree in computer science (the logic side of computers, more programming than designing hardware). A degree in bioinformatics combines molecular biology (DNA, proteins, etc), computer science (programming), and statistics (developing code to analyze biological data). I feel a need to express what it is, because it’s not something that is commonly understood. Do you know what people who study bioinformatics do?

*I came across a paper* today “Structural Alignment of Pseudoknotted RNA”:http://cseweb.ucsd.edu/users/shzhang/app/RECOMB2005_pseudoknot.pdf (by Han B, Dost B, Bafna V, and Zhang S.) which is a good example of the practice of bioinformatics. Think about what goes on in a cell… the sequence of a gene (a short region of DNA) is copied (letter-by-letter) onto an RNA molecule. The RNA molecule is later read by an enzyme (called a ribosome) and converted into a protein based on its sequence. (This process is the central dogma of molecular biology) Traditionally, it was believed that RNA molecules’ only function was to copy gene sequences from DNA to ribosomes, but recently (the last several years) it was discovered that some small RNA molecules are never read and turned into proteins, but rather serve their own unique functions! For example, some RNA molecules (siRNAs) can actually turn genes on and off, and have been assosiated with cancer development and other immune diseases. Given the human genome (the ~3 billion letter long sequence all of our DNA), how can we determine what regions form these functional RNA molecules which don’t get converted into proteins? The paper I mentioned earlier addresses this. An algorithm was developed and used to test regions of DNA and predict its probability of forming small RNA molecules. Spikes in this trace (figure 7 of the paper) represent areas of the DNA which are likely to form these RNA molecules. (Is this useful? What if you were to compare these results between normal person and someone with cancer?)

*After reading the article* I thought to myself “Hmmm… logically manipulating large amounts of linear data… why does this seem familiar?” Then I realized how similar my current programming projects are with this one. (see “my latest DIY ECG data”:http://www.swharden.com/blog/images/ecg_goodie.png posted a couple days ago) Consider the trace (pictured, figure 7 in “Structural Alignment of Pseudoknotted RNA”:http://cseweb.ucsd.edu/users/shzhang/app/RECOMB2005_pseudoknot.pdf) of score (the likelihood that a region of DNA forms an RNA molecule), where peaks represent likely locations of RNA formation. Just generate the trace, determine the positions of the peaks, and you’re golden. How similar is this to the work I’ve been doing with my homemade ECG machine, where I perform signal analysis to eliminate electrical noise and then analyze the resulting trace to isolate and identify peaks corresponding to heartbeats?

*After reading* I shivered from mental-overload. There are so many exciting Python projects in the field of bioinformatics that are just waiting for me to begin work on! I know I’m like a child sometimes, but hey it’s my personality. I get excited. It’s just that I get excited about tacky things these days. Anyway, I got the itch to write a string-analysis program. What does it do? It reads the content of my website (exported in the form of a SQL backup query generated by PHPmyAdmin, pictured), splits it up by date, and allows for its analysis. Ultimately I want to track the usage of certain words (i.e.: the inverse relationship between the words “girls” and “python”), but for now I wrote a script which plots the number of words I wrote. Observe the output.

*Pretty cool huh?* Check out all those spikes between 2004 and 2005! (previous figure) Not only are they numerous (meaning many posts), but they’re also high (meaning many words per post). As you can see by the top trace, the most significant contribution to my site occurred during this time. So, let’s zoom in on it! (next figure)

*And of course, the code to produce this…* (obviously you have to have a wordpress backup SQL file in the same folder – if you want mine let me know and I’ll email it to ya’)

import datetime, pylab, numpy
# Let's convert SQL-backups of my WordPress blog into charts! yay!
class blogChrono():
    baseUrl="http://www.SWHarden.com/blog"
    posts=[]
    dates=[]
    def __init__(self,fname):
        self.fname=fname
        self.load()
    def load(self):
        print "loading [%s]..."%self.fname,
        f=open(self.fname)
        raw=f.readlines()
        f.close()
        for line in raw:
            if "INSERT INTO" in line
            and';' in line[-2:-1]
            and " 'post'," in line[-20:-1]:
                post={}
                line=line.split("VALUES(",1)[1][:-3]
                line=line.replace(', NULL',', None')
                line=line.replace(", '',",", None,")
                line=line.replace("''","")
                c= line.split(',',4)[4][::-1]
                c= c.split(" ,",21)
                text=c[-1]
                text=text[::-1]
                text=text[2:-1]
                text=text.replace('"""','###')
                line=line.replace(text,'blogtext')
                line=line.replace(', ,',', None,')
                line=eval("["+line+"]")
                if len(line[4])>len('blogtext'):
                    x=str(line[4].split(', '))[2:-2]
                    raw=str(line)
                    raw=raw.replace(line[4],x)
                    line=eval(raw)
                post["id"]=int(line[0])
                post["date"]=datetime.datetime.strptime(line[2],
                                                        "%Y-%m-%d %H:%M:%S")
                post["text"]=eval('"""'+text+' """')
                post["title"]=line[5]
                post["url"]=line[21]
                post["comm"]=int(line[25])
                post["words"]=post["text"].count(" ")
                self.dates.append(post["date"])
                self.posts.append(post)
        self.dates.sort()
        d=self.dates[:]
        i,newposts=0,[]
        while len(self.posts)>0:
            die=min(self.dates)
            for post in self.posts:
                if post["date"]==die:
                    self.dates.remove(die)
                    newposts.append(post)
                    self.posts.remove(post)
        self.posts,self.dates=newposts,d
        print "read %d posts!n"%len(self.posts)

#d=blogChrono('sml.sql')
d=blogChrono('test.sql')

fig=pylab.figure(figsize=(7,5))
dates,lengths,words,ltot,wtot=[],[],[],[0],[0]
for post in d.posts:
    dates.append(post["date"])
    lengths.append(len(post["text"]))
    ltot.append(ltot[-1]+lengths[-1])
    words.append(post["words"])
    wtot.append(wtot[-1]+words[-1])
ltot,wtot=ltot[1:],wtot[1:]

pylab.subplot(211)
#pylab.plot(dates,numpy.array(ltot)/(10.0**6),label="letters")
pylab.plot(dates,numpy.array(wtot)/(10.0**3),label="words")
pylab.ylabel("Thousand")
pylab.title("Total Blogged Words")
pylab.grid(alpha=.2)
#pylab.legend()
fig.autofmt_xdate()
pylab.subplot(212,sharex=pylab.subplot(211))
pylab.bar(dates,numpy.array(words)/(10.0**3))
pylab.title("Words Per Entry")
pylab.ylabel("Thousand")
pylab.xlabel("Date")
pylab.grid(alpha=.2)
#pylab.axis([min(d.dates),max(d.dates),None,20])
fig.autofmt_xdate()
pylab.subplots_adjust(left=.1,bottom=.13,right=.98,top=.92,hspace=.25)
width=675
pylab.savefig('out.png',dpi=675/7)
pylab.show()

print "DONE"

*I wrote a Python script to analyze the word frequency* of the blogs in my website (extracted from an SQL query WordPress backup file) for frequency. “This is what I came up with”:http://swharden.com/little/worddump.html I then took my giant list over to “Wordie”:http://www.wordle.net/create and had them create a super-cool little word jumble. Neat, huh? Here’s “a picture”:http://www.SWHarden.com/blog/images/wordie2.png that’s cool but not worth posting.

*This is the script to make the worddump:*

import datetime, pylab, numpy
f=open('dump.txt')
body=f.read()
f.close()
body=body.lower()
body=body.split(" ")
tot=float(len(body))
words={}
for word in body:
    for i in word:
        if 65< =ord(i)<=90 or 97<=ord(i)<=122: pass
        else: word=None
    if word:
        if not word in words:words[word]=0
        words[word]=words[word]+1
data=[]
for word in words: data.append([words[word],word])
data.sort()
data.reverse()
out= "<b>Out of %d words...n"%tot
xs=[]
for i in range(1000):
    d=data[i]
    out += '<b>"%s"</b> ranks #%d used <b>%d</b> times (%.05f%%)
n'%
                (d[1],i+1,d[0],d[0]/tot)
f=open("dump.html",'w')
f.write(out)
f.close()
print "DONE"

     

Celebrity Dwarf Gouramis

So I was reviewing my website statistics generated by a Python script I wrote when I noticed a peculiarity so bizarre that it made me questin the very purpose of my life. Okay maybe it wasn’t that bizarre, but it was interesting. The python script (which is automatically run every hour) downloads my latest access.log and saves it to its own folder. It then analyzes the data, creates some charts and graphs, and dumps out a bare-bones results file displaying some of the information I found useful. Of note is the number of times each page is hit.

This is where things get funny. Outperforming my home page by nearly double was indexOld.php (now indexOld22.php) – a simple webpage I tossed of for about a year before I put my big blog back online! Why were people still going to this page? Further investigation (from the referring sites section of my stats page) revealed a lot of hits from Google image-searches. I started looking at the actual requests and realized that many of these hits were people searching for the term Dwarf Gouramis “a type of freshwater aquarium fish) which was mentioned on that old webpage. The ironic part about it is what happens when you google image search for dwarf gouramis there is a picture of an extremely rare zebra pleco which is actually a link to my website! However the link APPEARS to be to wallpaperfishtalk.com because on my page I just linked to their image.

My conclusion: People are Google image-searching for ‘dwarf gouramis’, and an amazing picture of a zebra pleco is coming up which links to my site (due to the fact that months ago I talked about dwarf gouramis but posted a photo of a zebra pleco) and people (in their awe at this amazing fish) are clicking on it. So what did I do? I pulled a bait-and-switch! You bet I did. Now when you go to indexOld2.php it just forwards you to my current website – mua ha ha ha ha

PS: I’m appending to this entry at 2:17pm to note that I made a wonderful breakthrough in the lab today. Due to intellectual property protection blah blah and the fact that I don’t want anyone else to beat me to my research goal I will not describe what this is, I’ll just say that it took months of preparation and today – presto! It worked beautifully =oD


     

Using PHP to Create Apache-Style Access.log

THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!
THIS CODE HAS BEEN UPDATED!

>>> CHECK OUT THE NEW CODE < << [Generate Apache-Style HTTP Access Logs via SQL and PHP]

OBSOLETE CODE IS BELOW…

My web server blocks access to my apache-generated visitor logs (commonly stored in “access.log”). Therefore, many great site usage stats generators (such as awstats – see this example) cannot be used to analyze web traffic to my site. (How many people go what pages? Where do they come from? What search phrases do they type into Google to find my website?) My web host does allow PHP, and access to php.ini, so I figured that I could generate my own access.log using PHP code. I succeeded, but had a hard time doing this because it’s not clearly documented elsewhere – so I’ll make it clear.

Sample line from access.log generated by my PHP script:
132.170.10.227 – – [22/Jan/2009:11:58:49 +0800] “GET /blog/2005-06-29-eva-05-attack-scotts-sanity/ HTTP/1.1” 200 – “http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=8Lk&q=swharden+eva-05&btnG=Search” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5”

All I had to do was insert the following line at the end of my php.ini file:

  

 auto_append_file = "/home/content/n/i/b/nibjb/html/logme.php"  

 

And I placed logme.php in my root folder with the following code:

  

 $logwriter_logformat = "combined"; // log format,combined or common  

 $logwriter_logdir = "/home/content/n/i/b/nibjb/html/logs/"; // physical path where your log file located  

 $logwriter_logfilename = "access.log"; // your log file's filename  

 $logwriter_timezone = "+0800"; // your server's time zone. +0800 means GMT+8  

   

 function logwriter_writelog($logstring){  

   

 global $logwriter_logdir,$logwriter_logfilename;  

 $fullpathfilename = $logwriter_logdir.$logwriter_logfilename;  

   

 if (!is_file($fullpathfilename)) {  

 print "Log file doesn't exist or file is corrupt.";  

 return;  

 }  

   

 if (!is_writeable($fullpathfilename)) {  

 print "Log file is not writable,please change its permission.";  

 return;  

 }  

   

 if($fp = @fopen($fullpathfilename, "a")) {  

 flock($fp, 2);  

 fputs($fp, $logstring);  

 fclose($fp);  

 }  

 }  

   

 function logwriter_handlevar($varname,$defaultvalue) {  

 $tempvar = getenv($varname);  

 if(!empty($tempvar)) {  

 return $tempvar;  

 } else {  

 return $defaultvalue;  

 }  

 }  

   

 if (!empty($REMOTE_HOST)) {  

 $logwriter_remote_vistor = $REMOTE_HOST;  

 }else{  

 $logwriter_remote_vistor = logwriter_handlevar("REMOTE_ADDR","-");  

 }  

   

 $logwriter_remote_ident = logwriter_handlevar("REMOTE_IDENT","-");  

 $logwriter_remote_user = logwriter_handlevar("REMOTE_USER","-");  

 $logwriter_date = date("d/M/Y:H:i:s");  

   

 $logwriter_server_port = logwriter_handlevar("SERVER_PORT","80");  

 if($logwriter_server_port!="80") {  

 $logwriter_server_port = <a href="".$logwriter_server_port;"></a>  

 }else{  

 $logwriter_server_port = "";  

 }  

   

 $logwriter_request_method = logwriter_handlevar("REQUEST_METHOD","GET");  

 $logwriter_request_uri = logwriter_handlevar("REQUEST_URI","");  

 $logwriter_server_protocol = logwriter_handlevar("SERVER_PROTOCOL","HTTP/1.1");  

   

 if ($logwriter_logformat=="common") {  

 $logwriter_logstring = "$logwriter_remote_vistor $logwriter_remote_ident $logwriter_remote_user [$logwriter_date $logwriter_timezone] "$logwriter_request_method $logwriter_request_uri $logwriter_server_protocol" 200 - 

 ";  

 }else{  

   

 $logwriter_http_referer = logwriter_handlevar("HTTP_REFERER","-");  

 $logwriter_http_user_agent = logwriter_handlevar("HTTP_USER_AGENT","");  

   

 $logwriter_logstring = "$logwriter_remote_vistor $logwriter_remote_ident $logwriter_remote_user [$logwriter_date $logwriter_timezone] "$logwriter_request_method $logwriter_request_uri $logwriter_server_protocol" 200 - "$logwriter_http_referer" "$logwriter_http_user_agent" 

 ";  

   

 }  

   

 logwriter_writelog($logwriter_logstring);  

 

Note that the PHP code must be surrounded with < ? php ?> as demonstrated here

The result? As you can tell, my logme.php dumps data to www.swharden.com/logs/access.log – if you browse a few pages on my website, or even use Google to search for me (ie: google for ‘swharden’ and ‘minidisc’) you can see yourself in the logfile – pretty cool huh? Once I have a good volume of log data I’ll demonstrate how to turn it into useful information.


     

Free Damask Seamless Tiling Backgrounds

If you’re in the mood for some 18’th century textile patterns you’ve stumbled upon the right place! Surprisingly, it’s incredibly difficult to find functional (seamless, tiling, free) damask-style patterns on the internet. If you don’t believe me, just Google / image search for it! It took me over an hour to find a functional pattern that tiled properly. Actually, to correct myself there, the image I downloaded didn’t even tile correctly!!! I had to manually modify it to make it seamless. So, free for all website makers, webmasters, wallpaper collectors, and Louis XVI enthusiasts: I give you a plethora of different colors of damask-style tiling backgrounds for whatever you want to do with it!