Thursday, August 7, 2014

Applying old properties to new files in Windows

A while back, I used some kind of "cp -r" command in Linux to copy files from an old IDE hard drive to a newer SATA drive.  The IDE drive is 10 years old, and thus many of the files on there are of that vintage.  Unfortunately, I was not aware that cp does not properly transfer file attributes between Windows files (as these are NTFS partitions).  I could have simply used Windows to copy the files, but I didn't want this to happen:

Hence why the choice to use Linux.  Nevertheless, being somewhat of an archivist at heart, I was interested in retaining some of these original file properties, namely Created, Modified, and Accessed times.  I didn't want all my files looking like they came from 12/2/2013 when some of them are as old as 2002.  So, I went back into Windows to solve my problem.

The idea for this was derived from one of many related StackOverflow posts on the topic, plus some additional information was acquired in order to walk the directory trees recursively.  I have pasted the code below.  One issue I haven't been able to work through is that directory attributes are not copied properly, claiming you do not have enough permissions to modify the file.  I even tried running VS 2013 as Administrator, but it didn't help.  So, I solved this problem by taking any errors you encounter along the way and writing them to a CSV file which you can sort & search later on.

As for the IDE hard drive, it's been zeroed out and will be donated along with other old drives targeted in my downsizing plan.

I'm sure it could be much better-optimized for speed & multitasking, but nevertheless, the code below is for Visual C# 2013:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace WindowsFormsApplication1
    public partial class Form1 : Form
        public long filesCount;
        public StreamWriter writer;

        public Form1()
            filesCount = 0;

        private void button1_Click(object sender, EventArgs e)
            // Recursively go through all the files on the source drive/folder
            //File.SetCreationTime(tgtFile, File.GetCreationTime(srcFile));
            //File.SetLastAccessTime(tgtFile, File.GetLastAccessTime(srcFile));
            //File.SetLastWriteTime(tgtFile, File.GetLastWriteTime(srcFile));
            string driveLetter = "E";
            writer = new StreamWriter("c:\\Notes-On-Migration-Of-" + driveLetter + ".csv");
            DirectoryInfo dirSrc = new DirectoryInfo(driveLetter + ":\\");
            WalkDirectoryTree(dirSrc, "D:\\Win7\\");
            lblStatus.Text = "Number of files: " + filesCount;

        private void WalkDirectoryTree(System.IO.DirectoryInfo root, string dirTgt)
            System.IO.FileInfo[] files = null;
            System.IO.DirectoryInfo[] subDirs = null;

            DateTime ss = File.GetCreationTime(root.FullName);
            DateTime ee = File.GetLastAccessTime(root.FullName);
            DateTime xx = File.GetLastWriteTime(root.FullName);
            // First, process all the files directly under this folder 
                files = root.GetFiles("*");
                filesCount += files.LongLength;
                lblStatus.Text = "On " + root.FullName + "; " + filesCount + " files done";
            // This is thrown if even one of the files requires permissions greater 
            // than the application provides. 
            catch (Exception e)
                // This code just writes out the message and continues to recurse. 
                // You may decide to do something different here. For example, you 
                // can try to elevate your privileges and access the file again.
                textBox1.Text = textBox1.Text += e.Message + "\r\n";

            if (files != null)
                foreach (System.IO.FileInfo fi in files)
                    // In this example, we only access the existing FileInfo object. If we 
                    // want to open, delete or modify the file, then 
                    // a try-catch block is required here to handle the case 
                    // where the file has been deleted since the call to TraverseTree().
                        File.SetCreationTime(dirTgt + fi.Name, File.GetCreationTime(fi.FullName));
                        File.SetLastAccessTime(dirTgt + fi.Name, File.GetLastAccessTime(fi.FullName));
                        File.SetLastWriteTime(dirTgt + fi.Name, File.GetLastWriteTime(fi.FullName));
                    catch (Exception e)
                        textBox1.Text = textBox1.Text += e.Message + "\r\n";
                 * */

                // Now find all the subdirectories under this directory.
                    subDirs = root.GetDirectories();
                catch (Exception e)
                    writer.WriteLine("Corrupt directory," + root.FullName);
                    textBox1.Text = textBox1.Text += e.Message + "\r\n";

                foreach (System.IO.DirectoryInfo dirInfo in subDirs)
                    // Write the directory attributes too.
                        //MessageBox.Show("Target: " + dirTgt + dirInfo.Name + "\r\nSource: " + dirInfo.FullName);
                        File.SetCreationTime(dirTgt + dirInfo.Name, File.GetCreationTime(dirInfo.FullName));
                        File.SetLastAccessTime(dirTgt + dirInfo.Name, File.GetLastAccessTime(dirInfo.FullName));
                        File.SetLastWriteTime(dirTgt + dirInfo.Name, File.GetLastWriteTime(dirInfo.FullName));
                    catch (System.UnauthorizedAccessException e)
                        // Write to the CSV file the "Access Denied" tag, what we were trying to modify, and its original properties from the source hard drive
                        writer.WriteLine("Access denied," + dirTgt + dirInfo.Name + "," + File.GetCreationTime(dirInfo.FullName) + "," + File.GetLastAccessTime(dirInfo.FullName) + "," + File.GetLastWriteTime(dirInfo.FullName));
                        textBox1.Text = textBox1.Text += e.Message + "\r\n";
                    catch (System.IO.FileNotFoundException e)
                        // Attempt to copy the file from the source to the destination
                            File.Copy(dirInfo.FullName, dirTgt + dirInfo.Name, false);
                            string ifei = "kjbhwe";
                        catch (Exception e2)
                            // This happens when we don't have enough permission to copy the file for some reason
                            writer.WriteLine("File copy denied," + e.Message);
                            textBox1.Text = textBox1.Text += e.Message + "\r\n";
                    catch (Exception e)
                        // We don't really know what happened, so just write it down
                        textBox1.Text = textBox1.Text += e.Message + "\r\n";
                    // Recursive call for each subdirectory.
                    WalkDirectoryTree(dirInfo, dirTgt + dirInfo.Name + "\\");

Thursday, July 10, 2014

Waking the Sore, Groggy Dog

Well, it's reached the point where I finally had enough of computers and everything I was doing on them for just a bit, so I closed the lid of my laptop on Tuesday right at 5 PM and sat outside in the lawn chair, doing absolutely nothing for about 30 minutes until my dog started panting up a storm.  I let her back inside and she gulped down tons of water quickly.  Still not knowing what to do with myself while trying to stay away from the computer, I just about dozed off on the couch when inspiration struck me: I needed to do something physical to stay awake.

Now I'm a huge fan of sitting, but sitting still in one position at a desk for a long time leads to unhappy muscles.  Plus, I find tedious or uninteresting work to be soporific, especially if I just had a heavy lunch or didn't get much sleep the night before.  However, taking a nap usually leads to undesirable outcomes: I wake up about 2 or 3 hours later, still feeling groggy, and often incapable of accomplishing my desired tasks for the night.  But what am I supposed to do if I don't want to be on the computer but still want to sit?

How about take a bicycle ride?


In this hot weather, my dog isn't capable of going more than about a mile before she gets utterly & completely pooped, so I just went solo.  (The reason she gets so tired is because she hates being behind me; she's a dominant little creature and likes to lead the way.  However, I can always ride as fast as she can run, thus she burns herself out very quickly at the beginning and never goes quite so fast until the next walk.)  So on my own, I was able to log over 5 miles in about 35 minutes -- not too bad, considering my bike (and I) aren't in the best of shape, it was about 100 degrees outside (since I was riding at about 6 PM), and I had to cross at a busy street a couple times and wait on a long light.

Anyway, I guess I'll shut up now.  I haven't been feeling incredibly prolific with regards to writing, and none of the draft articles in my queue to finish really struck my fancy.  Stay tuned for an article called "Speech Pipe: My First (And Probably Last) Windows Phone 8 App"... but, alas, I can't really remember too much about the pain points I encountered that made that experience so fraught with tedium.  Let's see what has happened since then:
  • My last blog post was one of those pesky "draft" posts that I write when I'm feeling a surplus of creativity so that I can save it for days when I don't have enough time or don't feel like writing anything on Thursday night.  I submitted it after arriving at the hotel in San Antonio for the SoHacks hackathon, and after our flight had been delayed for several hours (thus we probably could have just driven to San Antonio by the time our plane landed), doing anything besides laying in bed and watching TV didn't seem very appealing.
  • It was good that I'd written that canned article ahead of time, but I didn't have anything technical to write about because I'd been working on Speech Pipe.  I was trying to get the app done in time for the DVLUP Day Challenge in order to be eligible for a Nokia Lumia 1020, but two things went against me: I had a lot of work to do for LEDgoes as well, plus I didn't just want to crank out yet another turd-ball type of application for Windows Phone, which many of these challenges seem to inspire since people always try to do the bare minimum necessary in order to win points and get free stuff.  Nevertheless, I've since received a different model of Windows Phone 8 device from the local Developer Evangelist, and Speech Pipe is currently making money in the app store.  So there. >-D
  • You may have seen how I posted the game of Tetris those kids developed with LEDgoes during SoHacks.  If not, what are you waiting for -- check that link!  And you can read more about SoHacks from DoesItPew's perspective and from Randy Arnold's perspective.  Apparently, one of the kids wrote this about me as a mentor:

    Stephen is f#@king smart!
    How eloquent. :-P
  • I completely redesigned the LEDgoes Battery POWAH board, in cahoots with some overseas engineers at TI who have helped me fine-tune the schematic and will shortly be helping me fine-tune the PCB layout.  I also did like four other big things on that project, which you can see on our latest Kickstarter update to date.
  • The freaking Udoo board that I hooked up to the Yamaha for my IoT project stopped working.  I dunno what the deal is, but the screen is all garbled and it's not responding to MQTT messages anymore.  I think it needs to be rebooted, but really I think there's a memory leak that needs to be fixed with the Python MQTT client that's running on there.  Or in an ideal world, I'd just make it into an app you can run on XBMC and make the Udoo run XBMC instead of Ubuntu or whatever distro the Udoo has on it now.
  • I have helped close to a half dozen people track down long-lost episodes of The Price is Right over the past month or so by referring them to a contact I made at FremantleMedia who works in their tape vault, or sending them links to their shows on YouTube, or what have you.  It turns out that my "Price Is Right Episode Guide" (if you can call it that) has been very popular on Google lately, usually ranking among the top 10 whenever you search for "the price is right <specific-production-number>".  It's really nice to be able to help so many people track down their shows, and what's even more fun is if they manage to find their own long-lost copies and share bits of them with me!
  • I started taking a class on Lean Six Sigma, Green Belt.  @DoesItPew has already taken her green belt, so of course she sounds like a know-it-all in the class. :-P  Not to mention, she's a vocal person anyway, but she thought there was a "continuing education" component to keeping your Six Sigma green belt certification alive, and if she's right, this'll help her keep/retain it.
  • We have had jam sessions, beer sessions, attended classes and talks held by other sucessful startups, had friends poke holes in their success stories and claim to be able to do in 3 days what took them 3 years, but we're taking a break from that: 3 out of the 5 weekdays this week will have involved some sort of evening drinking or huge dinner, and only 1 was class (the Six Sigma).  It's been nice to cut loose.
  • My dad celebrated 30 years of working for Mercedes-Benz last week.  It's amazing anyone can stay in the car business that long these days, much less work for the same company.  At this point, even after he retires, he'll be able to keep getting the "friends & family" discount on Mercedes vehicles for the rest of his life.
So... until next time... keep your pants on at work!

Thursday, June 12, 2014

The original Super Mario Bros Theme... for Bass Drum!

Back in high school, marching band took up even more time than I spent on homework in any given week. Being a member of the drumline added yet another level of demands and responsibility -- but that's all part of what it takes to play on a line that, if memory serves me correctly, would have beaten the University of North Texas' drumline at PASIC's drumline competition in 2002 had we not been penalized for crossing the physical boundary lines of the competition at some point.  (Nevertheless, we still won among high schools.)  Even though it has been almost 10 years since I graduated high school, their drumline is still kicking butt and constantly taking top honors at every contest they enter.

In middle school, I discovered I had the ability to play along with songs by ear as long as I heard them emough times and knew how they went.  In 9th grade, the realization that I had perfect (or really darn good relative) pitch always fascinated the other band members; I was often tested with random notes or chords, and never failed to impress.  My drum instructor would even ask me for the pitches of the drums -- yes, drums have pitches too, especially timpani & bass drums!  (Now I will admit it's much harder to tell a snare drum's pitch, so usually I'd jokingly say it's a "Bang Flat.")

I also became married to the bass drum in marching band, and became well-known as the one who carried arguably the heaviest instrument on the field -- the 30" bass.  Only the tuba section or tenor drums came close in weight, and this group of players carried a special level of respect for each other.  As I played the 30" bass for my second, and then my third, year, I was used as an inspiration tool on the field (i.e. called out as an example by the directors & techs) to those who carried a lighter instrument yet whose marching technique was not as good.  One of the drum majors admitted to watching my feet to get the tempo on one of the songs we played --- Wow!!!  I thought they always watched the center snare drum player (who is usually the highest position or best player on the line) to get the time, so I took that as a very high honor.

Ok, enough gloating.  During that time, I also dabbled with music composition on the side (more on that in a future post perhaps ;), and found inspiration one football-game night when it sounded like the opposing team was playing a few bars from the Super Mario Bros. theme.  I thought to myself, "Why aren't we doing this?"  Over the summer, after the long, sweaty rehearsal hours, I brought my microphone to the drumline equipment room and took samples of most of the drums to make a drumline "SoundFont."  This soundfont was the basis for my experimentation in MIDI and composition of the Super Mario Bros. Theme for bass drum, which you will find below.

I then formally notated it using whatever version of Sibelius was around 10 years ago.  You're probably better off simply transcribing it for yourself from the image here, because me trying to find the actual Sibelius file along with installing some archaic software on hardware that has long since moved on, would likely be an exercise fraught with disappointments.

So, at last, here it is!

So, one note about reading this:

The score is written for 5 bass drums, with the biggest (lowest-pitch) one notated below the staff ( partially highlighted -- makes it easier to spot your notes), and the other drums are on each "space" on the staff in decreasing-size (increasing-pitch) order.  A note on the middle line (bass clef "D") means that all basses will play.  Thus, it's not the real notes of the Mario Bros. theme. 

Unfortunately, one of the bass drummers left the drumline right after I rolled this out, so we never got to play it.  You just can't do Mario very well with only four bass drums, so maybe it got passed down over the years and played by others.  I really don't know what happened with it since.  However, this copy still sits in a page protector in my marching music binder, which has pretty much remained untouched since I graduated.

For the record, the (C) 2004 is just for show.  It makes things look a little more official (which is cool to do when you're in high school), but it's more for posterity's sake than actually claiming any rights on anything.  You should know I'm way bigger into open-source and open sharing than copyrights anymore.

Thursday, June 5, 2014

Teaching Arduino to the lil' ones

Maybe you've figured out by now that I like to make blog posts on Thursdays.  I'll skip a week or two here & there if I don't have new content or anything prepared, but last Thursday, I simply forgot to post some of my "canned content" due to sheer exhaustion from a week culminating with teaching Arduino to a group of fourth-graders!

Making a Difference

You never know what opportunities will knock on your doorstep from demonstrating at various events around town, and of course, simply showing up to meet people could lead to that million-dollar introduction.  As such, we were invited to present LEDgoes at the Frisco Mini Maker Faire back in April, and were busy showing off this:

The maximum-length LEDgoes marquee, 64 panels, 8 feet long!

Area educators were on the prowl looking for fun things to engage their students, and one of them just happened to wander by our booth and inquire into the details of our project.  We mentioned the boards are based on Arduino microcontrollers, and that got the gears turning: she is looking to build curriculum for 4th & 5th-grade gifted/talented students around Arduino, microcontrollers, and programming.  We traded contact info, and eventually scheduled a date for... last Thursday.

It was a bit of an exhausting week for me anyway, as Stacy's car was broken and I needed to shuttle her everywhere too.  It's a good thing our office buildings are only about 1/2 mile apart, but she works different hours than I do and adjusting to her wake-up time was not easy.  However, I'm blessed to work at a company which allows for up to 4 paid volunteer hours per month, so I took advantage of these paid hours last Thursday to drive to the elementary school with some of my own gear to share and projects to amaze with.

I got a warm welcome from the students on the way in, and jumped in telling them a little bit about myself, what I do, what I'm interested in, and how I got there.  I showed them the completed "Inklings of a Project" that I posted here a while back and told them that even though it looks like all the LEDs are on, in fact only one LED among the 104 in the charlieplexed LED array are turned on at any given instant in time.  They were, of course, quite fascinated by this, plus by the tiny Bluetooth module I passed around among them (yes, I didn't need to tell these kids what Bluetooth was -- they are started early on technology these days!)  There were many more gasps of amazement when I told them to compare how fast you can see things change (less than 30 times a second, usually) to how fast the Arduino's chip can change things: up to 16 million times per second.  They were blown away by that, but told them that was nothing: the Asus Eee netbooks they all had in front of them ran much faster, up to a couple billion instructions per second!

The truly smart ones can explain anything to anyone...

Then began the fun part: there is an awful lot that needs to be explained to anyone just starting to learn this stuff, whether you're in fourth grade or even an adult looking for a new hobby.  I needed to explain the basics of inputs & outputs on the chip, and how these could be used to read & write data in & out of sensors, lights, or any other electronic widget you can imagine.  These kids probably had not been exposed to the concepts of electricity yet (i.e. voltage & potential), so I had to carefully explain how the pins behave, and why they do what they do.  The first time I tried explaining electronics to them, I got back puzzled looks across the room -- oh no, time to reformulate!  I paused for a couple moments and picked another analogy that would resonate better.  Forget about declaring the pins as inputs & outputs: think of them like light switches on the wall.  You can flip them on & off as fast as you want.  The chip can flip the switch, or another device you connect to the chip can flip the switch.  Their faces started lighting up, and so I knew I could proceed.  The awkward "What?  I don't get it." phase was over.

Now that the chip had been discussed, it was time to demonstrate how the chip would "toggle the light switch."  The kids opened up the Arduino IDE and loaded the simple "Blink" example.  I walked them through each line of the code, told them what setup() and loop() does, and then had to explain what a loop was.  (Actually, the teacher would sometimes jump in and walk the kids through a brief exercise to help shed light on some of these concepts, and the loop was one of them.)  Then, I noticed that Blink has some simplistic use of variables, which I'm sure the students would not learn about in math for many years, so I phrased the line "int ledPin = 13" as "the secret word for 13 is LEDPIN; LEDPIN means 13".  Once we walked through the code, I had them load it onto the Arduino Unos, and soon, all of them were getting it to blink.

Afterward, I decided to do a little more advanced exercise in variables with them, and engage them further by finding exactly what blink rate can be set before the LED appears to be on constantly.  I solicited a name for the variable from among the students, and an initial value.  At this point, I needed to explain microseconds to them, as the argument for the delay() function is in milliseconds and they hadn't been introduced to these scientific-notation prefixes yet.  To check their understanding, I picked some random values and asked them if the light would blink on & off faster or slower than 1 second, and they seemed to get it.  Once we had our variable in place and the value, I had them replace delay(1000) with delay(waitTimer), and experiment with different values of waitTimer to see at what point the LED would appear to be solidly on.  They got it down to about 15 milliseconds before it appeared to be solid.

Soon thereafter, the students wanted to experiment with longer delays -- one group tried to set it as 1 billion.  I told them that wasn't likely to yield the result they expected, and then tried to explain the concept of binary to them in order to convey why the chip can only store numbers up to a certain value.  I even emphasized that in my field, you don't need to know a lot of advanced math like calculus; just knowing how to count in different number systems and your Powers of Two go a long way.  For instance, the maximum value for an "int" number on an Arduino Uno is 32,767, or 2^15, because the 16th bit is reserved for the sign (+/-).  Any number beyond 32767 would suffer from aliasing, and possibly become negative, due to two's complement.  (I didn't dare attempt to explain two's complement to them. :-P)  To put this in a base-ten perspective, say the largest number you can hold is 9999 (four places), and you provide it 106531: the system is only going to remember 6531 (the first four places).  Later on, for the teacher, I explained how the 16th bit in the binary number representing 1 billion is in fact a 1, thus the processor interpreted that number as some negative value and the kids did not see any delay whatsoever.

At this point, some of the kids had to go off to lunch.  Bummer.  Some brought theirs back to continue, and then I showed them how to wire up another LED in series with the blinking LED so they could begin to understand the layout of a breadboard.  Now I had to teach them which side of an LED is the anode, and which is the cathode, so they could wire Pin 13 into the anode and Ground into the cathode.  One of them put the jumpers in backwards, so I simply flipped around the LED on the board and showed them that was perfectly valid too.  Voila!

Well, that was about it.  I felt like we could have pressed on with more material, but realized they'd probably had enough; what I had already showed them would probably be enough for them to think about for the whole summer.

However, that's not the end of the story: to save you from a bunch of interruptions, I omitted all the times when the kids would blurt out some tangentially-related questions during my talk, including "What college did you go to?", "How much money do you make?", and "What other types of Arduinos are there?"  Many times, these questions were tempting to answer, and would lead me down some long tangent; I had to be careful to stay on-topic.

Finally, as standardized testing was over, the teachers pretty much had free reign over the kids to cover additional material, give them extra recess time, or whatever suited their fancy.  Some of the kids seemed to be in "summer break" mode already, and really wanted to play the Cookie Clicker game.  As I was chatting with the teacher over lunch (including a quip that these kids will probably replace me at work in 15 years), it came out that I played percussion in high school and had fast hands; the kids caught onto this, and soon I was recruited to click on everyone's cookies.  (I was wondering why the kids were talking about buying grandmas, which apparently you can do in this game with so many cookies, and quipped to the teacher that whatever this site is needs to be added to the school proxy's blacklist. :-P) It was a darn shame that I couldn't put my test automation skills to work here: as someone who is now doing Web UI testing for a living, I could have written them a really good script to click on that cookie faster than they could ever dream.  Sadly, the IT permissions on these computers prevent them from installing any new software.

On the way out, the kids presented me with this neat booklet containing thank-you letters thanking me for spending time with them to teach Arduino.  It was a sweet gesture that a couple of them actually brainstormed, created, and put together for me.  I also told the teacher to devote some time to teaching them how to count in base 2 so they'll be better prepared for my explanations on why certain things happen, and possibly when it comes time to read & write values from & to entire output ports on the ATmega chips.

Thursday, May 22, 2014

Getting your significant other's attention in a large house

The physics of sound in our house lend well to Stacy being able to shout up to me from downstairs, but not so well for me shouting back downstairs.  It makes communication inefficient, and often times she's not exactly watching for messages on GChat either.  I don't like going halfway downstairs when I'm trying to hold very skinny wires in a special arrangement when poking around for defects in LEDgoes, much less just to get a Yes/No answer on a simple question.

Recently, we picked up a Yamaha HTR-4065 stereo receiver for our living room.  Stacy was not happy with the sound quality of the original receiver, and hearing how the Yamaha can fill up the room while not vibrating every tiny appendage on your body with obnoxious bass, it was definitely money well-spent.  Hopefully it will add some more years to our natural hearing capability before we have to get hearing aids or whatever magical, mystical implant they develop by the time we're that old.  However, there is something else that piqued my interest about this receiver in particular that can solve the problem I proposed in the preceding paragraph.

You'd think the communication problem would be simple to solve; just install an intercom, right?  Sure, our house was built in the '90s, but the folks who built it weren't that wealthy.  Thus, it's not wired for an intercom.  We are both licensed ham radio operators, but neither of us listen for messages from each other unless we have previously agreed to, and usually the 2m or 70cm simplex frequency we use doesn't get a great range -- after all, we are two great big bags of water that excel at attenuating RF signals.  There are other aspects of our life, though, that elevate the Yamaha receiver into an excellent, effective means of communication: you can change its input source over WiFi.

Most of the time, the Yamaha is blasting music or TV when she's home (which also describes why I have such a hard time getting messages downstairs); thus it's always on.  All of our entertainment devices are routed through that receiver -- our turntable, "cable" (fiber?) box, DVR, Google TV, and PlayStations are switched at the receiver so now there's only one HDMI cable going into the TV.  We also hoard single-board computers such as the Raspberry Pi, Beaglebone Black, Udoo, pcDuino, and even some exotic builds of the Panda board that were never released to the public.  Most of the time, these boards sit idle, to be honest; now one of them is about to be put to work for me.

How will this be done?

Our devices are all Internet-connected, making them perfect targets for IoT/M2M protocols such as MQTT or AT&T's M2X platform.  Gone are the days when you need to use socket programming and open up ports on your router to have external devices connect to them; now I can run an application on my Android smartphone without bothering to turn its WiFi radio on, and through the magic of these M2M protocols, I can use it to control these devices in my house (or out of my house, or someone else's devices, or what have you).

The mobile app simply takes a recording from the microphone.  This recording is stuffed into a JSON object by storing the binary data as two letters representing each byte's hexadecimal value (done efficiently thanks to this StackOverflow post).  Hence, we can send the message in plain text and not have to worry about any special UTF-8 or Base64 encoding involved with sending binary values that would break the JSON object's structure.  (Not only that, but I wasn't able to figure out what string encoding they were using, and trying to guess & check in Android Java was becoming painful).  The string representing the audio recording gets submitted as the payload to the 2lemetry API.  The specific API call I use


ultimately publishes the message to their MQTT service, so that anyone subscribed to the prescribed {domain}/{stuff}/{thing} on their MQTT endpoint will receive the message.  Their service doesn't have a nice blob-storage capability (like MySQL has) that could straight-up store a binary file (but it does support JSON nicely), hence the need for a bit of additional legwork.  AT&T's M2X service should support blob storage by the end of 2014, at which time it might be worth re-evaluating the services.

An MQTT client, written in Python, is left to run perpetually on the Udoo single-board computer.  It subscribes to 2lemetry's MQTT server (or broker, in proper terms) to listen for new messages on the desired channel & pathway.  Once the mobile app finishes uploading the voice data onto the broker, the Udoo client will receive an asynchronous message and commence downloading the JSON-encoded data from the broker.  The JSON data (particularly the value containing the audio data) will be converted from "hex string" format back to a list of byte-sized numbers (0 to 255), and the series of numbers will be saved into an audio file.

The Yamaha HTR-4065 can be controlled via their AV Controller mobile app, but several eager developers have already taken the time to discover the ins & outs of the Yamaha stereo's API.  A great list of many (if not all) of the possible commands is here:  After the Udoo has converted my JSON-encoded message back into a binary file, it will be responsible for switching the HDMI input in such a way as to get the precise timing of when the message can start playing, then play the message, and finally switch the receiver back to the original HDMI input when the message is done playing.  This is achieved using a couple of POST requests, described in the previous link, to the stereo's API.  Sadly, it also means that the video will also go black (or switch to the Udoo's desktop) temporarily while the message is playing.

Do we expect this to be effective?

Stacy doesn't tend to look at the TV screen very much; she runs shows just to have background noise.  Nevertheless, the sudden change in the environment should, from a psychological perspective, wake up sensory neurons and put them on full alert, especially if what's currently playing involves loud music or an intense action scene: to suddenly hear silence will be startling.  Even if the scene is quiet, it'll be shocking to hear my voice coming through the speakers -- how did I just work my way into the show?  Since the receiver is totally remote-controllable, I can even power it on if it's off, and the message will still get through!  The only way for it not to work is if the power goes out, in which case my shouting will probably be more effective than usual.

This seems like the equivalent of a modern-day Rube Goldberg device!

Damn straight, and that's the way I like it.  It could have been worse, though: I might have had to use TCP sockets, or perhaps develop a robot to attach to the front of the stereo receiver in order to select the right inputs, plus use a camera with optical character recognition to determine what HDMI input to switch back to after the message has finished playing.  Even though it may seem to fulfill a niche market now, having to design a robot to press buttons would be the epitome of over-engineering and over-thinking.  My implementation should easily apply to several different applications, devices, and "user stories" as-is or with little modification.

Other things worth noting

  • 2lemetry now requires that the "password" of a user looking to authenticate themselves on the MQTT broker be hashed with MD5 before being sent to the server.  This threw me off for a couple days.
  • In Python, to convert the "hex string" back into a list of numeric values between 0 and 255, I used an interesting array operation to convert each pair of letters to integers, then each integer into a byte:intArr = [int(i, 16) for i in hexArr[0:len(hexArr):2]]
    audioByteArray = bytearray(intArr)

    It evidently wasn't good enough to write the integer array straight to the file: some of the data was getting misinterpreted and corrupted, so I needed to use the approach above.  There is a competing approach I could potentially use straight into the output file writer:

    from binascii import unhexlify
    output.write(unhexlify(''.join(format(i[2:], '>02s') for i in b)))

    However, its use of "format" leads me to believe it could be a bit less efficient.  I might do a benchmark to see which method runs faster.
  • I managed to debug the Android side of this application by downloading a hex editor to see if the file it saved locally matched what the MQTT broker had stored as the last message it had received in my particular topic pertaining to audio messages.
  • The length of the audio file doesn't need to be calculated.  I looked into MP4 headers for a little bit before realizing I can just spin up an instance of VLC in its own thread (with some particular arguments so that VLC only plays the audio file once and then quits), then write thread.join() so my Python client will resume regular operation when VLC is done.  The interesting question is what happens if a message comes in while VLC is playing audio already; will the MQTT client hold the message, or discard it?
  • Before using the MQTT client in Python, make sure you've run

    pip install paho-mqtt

    to install the required Python library.
  • The Android app is mostly a mashup of two previous applications I wrote: Street Beats and SpeechPipe.  This is why I could get it done in two weeks with so many other distractions, even amidst all the new learnings.  The features I did for this application will be rolled into SpeechPipe.
  • The Udoo allows several choices of Linux distributions to be installed on the device by means of SD card.  I chose Linaro Ubuntu 12.04 but had terrible luck getting any sort of audio to come out of it, even with VLC.  VLC was even seg-faulting on me!  I then tried xbmc media player, but didn't feel like going through the trouble of writing a plugin (even though it shouldn't take long).  Luckily, I already had an SD card in the Udoo formatted with some flavor of Ubuntu already, and that one seemed to work fine.  The MQTT client I wrote in Python had no trouble running on Linux.
  • The audio quality from whatever compression Android is using sounds terrible.  I need to keep playing with codecs & compression rates.  Nevertheless, a few seconds of audio is only taking up 8KB of space right now!
  • Sorry I'm not showing very much of my code in this post.  I need to clean it up and get it production-ready before I am comfortable open-sourcing it.
  • Every time we switch the configuration of our entertainment devices, it takes me about a week to figure out the new setup regarding the remote controls.  Now, to watch regular TV, I have to:
    • turn on the receiver with the Yamaha remote,
    • turn on the TV with the Samsung or FiOS remote,
    • switch to regular TV mode with the Google TV remote, and
    • wake up the FiOS receiver with the FiOS remote.

Thursday, May 15, 2014

The Options Scalper

The Options Scalper was another project that had a lot of potential during its time, but would have required some startup capital (VC attention) and more care and nurturing in order to really get off the ground.  Now I feel like the guys at Quantopian are quite on the right track, but they too have recognized that designing an intuitive interface for stock options analysis will be difficult... plus there's just so much data!!!  Nevertheless, Dan Dunn at Quantopian seemed to like the interface I provided, but they're designing the interface I ultimately wanted to provide so I'm just going to let them finish it up. ;)  (Plus, they know a little more Python than I do, which seems a great programming language choice for this task.)

The scalper was designed as a means to learn and study option decay in a graphical manner.  As weekly options were relatively new back in 2011, I decided those would be a great target because there's not a lot of data you need to store for weekly options before they expire, plus as being new instruments, perhaps one could pick up on potential arbitrage from market inefficiencies.  It was also to show the motion of the options themselves, or an entire spread, in a manner that brings it up at the same level as a stock's price.  Graphing tools I was familiar with back then typically would put the option way down on the Y axis compared to the stock's current price, and thus it's hard to draw comparisons when you're trying to study the dynamics of a cheap out-of-the-money call or a butterfly spread on a very expensive stock.  This solves the problem by smartly transposing the option's X axis ($0 price level) up to its strike price on the Y axis of the stock's graph.  At the time, there were 12 stocks whose weekly options I cared about; nowadays, I'd probably switch some of them out for more trendy or volatile securities.

The fundamental backbone of the options scalper is the options data.  A great source for options data is Yahoo! Finance, but writing a crawler & scraper for each of those pages could prove unreliable as everyone knows how many times sites change their markup, and I didn't want to have to deal that much with maintenance.  Luckily I happened to walk down the sidewalk at school once during my masters with another fellow who was an intern at Yahoo! complaining about how he perceived very few people were using the Yahoo! Query Language (YQL), so I ended up exploring this and finding out you can obtain stock and in fact options data using YQL queries.  Great!

Here's the exact query I used to fetch the prices of the stocks tracked on the Scalper:^DJX%22%2C%22BIDU%22%2C%22GS%22%2C%22C%22%2C%22PCLN%22%2C%22AAPL%22%2C%22SLV%22)&

This query simply returns the last trade price, and I store this information along with a timestamp into a MySQL database in order to present a graph of these stocks over the desired time frame.  Here's the query used to collect the options prices for AAPL in particular:*'AAPL'&

The options query collects only the data for the current month (weekly series & monthly series if unexpired), but gets much more information than the stock query.  It fetches the options's symbol, whether it's a put or call, its strike price, last traded price, change from yesterday's close, bid & ask price, volume, and open interest.  In general, I ran the collection cron job every 15 minutes.  It was a compromise between smooth data points & exceeding the storage limits on my server. :-P

The other interesting facet about the Scalper was the user didn't directly have access to the underlying data, much like the model Quantopian has adopted.  In fact, Dan told me that dealers of the historical stock & options data are likely to work with you for much cheaper if you don't directly redistribute the data, so simply being able to query & visualize the data without the user needing to know the value of every single point is important.  This also means you need a really good front-end, which I whipped up using JavaScript and Flot.js charts.

The boxy elements on the front page were meant to summarize the stock price and the 4 nearest to at-the-money contracts in the front-week expiration.  The different elements would highlight in red or green depending on what had moved up or down.  With no new data in years, they all pretty much show N/A.  I had begun experimenting with some different graphical elements on before I abandoned the project.  The top bar, listing the ticker symbols tracked, is the gateway to all the historical information hosted on  From here, you can see a graph of the stock, and select an expiration day to play with the options for that particular week.  Once you select an expiration week, it will load the option chain on the bottom left.  You can click on any strike price to see the motion of that option graphed on top of the stock's motion in a graph right below the main stock graph.

By far, the most interesting feature is the big white graph at the bottom.  This is where you can click any of the +1/-1 fields in the option chain to simulate going long or short on any of the puts or calls in the option chain.  You can put together spreads on the fly including verticals and lopsided butterflies, or whatever you desire (as long as it doesn't involve shares of the underlying security), and the total value of that spread over the course of the whole week will be represented in the white graph.

I ended up using some of this data in a project for my master's where we tried to predict which iron condor spreads would be most ideal to sell on a given week, assuming low market volatility.  None of the visualizations were incredibly useful for it, but we were able to churn through a lot of data real fast and feed it into some pattern analyzers and machine learning algorithms.  As it turned out, the classification data worked very well on the real-world example just in time for our final presentation, but the following week, the market dynamics changed and our algorithm would have lost us a bunch of money!  There must have been an earnings surprise we didn't account for before picking.

Thursday, May 1, 2014

Three Obscure Pricing Games On "The Price Is Right"

Everyone has their favorite pricing game on The Price Is Right, whether it be Plinko, Lucky $even, Hole In One, Clock Game, or whatever the case may be.  Some may fancy retired games such as Hurdles (1976-83) or Super Ball!!! (1981-98).  Long-time producer Roger Dobkowitz once said "every pricing game is somebody's favorite," and with over 100 pricing games played over the show's long history, it's been sufficient to keep millions of viewers still watching loyally every morning.

For as many memorable games as the show has, they have certainly had some, shall we say, "experiments" that did not go so well.  Some games were dull; others simply did not mesh well with the spirit of Price is Right (like Professor Price or Double Bullseye).  Still others have grown old (like Barker's Bargain Bar), were killed off due to inflation (like Walk of Fame), were plagued with mechanical issues (like Hurdles), or became too confusing for today's contestants to win with regularity (like Hit Me).  And it's not for a lack of effort on the part of the staff; some of these involved detailed game props or an interesting game play, and as such, they still make for exciting fun to watch--if not for the gameplay, then for the "vintage factor" or even the "kitsch factor" (though it is hard to find kitsch on Price is Right if you ask me!).

Most of these retired games can be viewed by the public by means of tape trading or on YouTube.  As such, the game where two people bid on a car (Double Bullseye), and many other games which you may have never heard of, can be seen and enjoyed by die-hard fans.  And, this month, thanks to venerable emcee Wink Martindale (who I'm surprised is this into Price Is Right and its fans to have done this!), three very short-lived games from 1978 have now been recovered from the vaults to be mused over by fans.

For 35 years, these three games had remained unseen since they originally aired because Game Show Network did not air them for one reason or another and no original broadcast tapes of these episodes had appeared on the trading circuit.  These games include Shower Game, Telephone Game, and Finish Line.  (By the way, Wink recently posted another video detailing the original rules for Punch-A-Bunch.)

Tell me more!

If you plan to watch these videos with all the curiosity of someone whose imagination has intrigued them about these games for years, you'd better not read on.  However, I'm guessing most of you would go either way, and perhaps some of you wouldn't even make it through the whole video. :-P  So, here I shall describe my initial reactions to these videos.

First, I started with Shower Game.  This game was only played ten times, all airing between September 4 and November 30, 1978.  The man who played Shower Game on its premiere sported the quintessential '70s style: poofy hair, mustache, and tinted glasses indoors.  His shirt, with big pointy collars, was already unbuttoned quite far, and became even moreso once Bob told him he would be playing the Shower Game.  The game is played for a car, and its prop is quite large.  There is one price listed above each of six shower stalls, and the contestant must enter the stall and pull the cord to "start the shower."  The shower could consist of confetti (in 3 stalls), 100 $1 bills (in 2 stalls), or a large prop resembling a car key in one stall.  The game continues if the contestant draws confetti; once they win an actual prize, they stop.  As for the key, it doesn't fall all the way down; it's on a short hook so it simply swings out.  It was amusing to see the camera swing from price to price over this huge prop in a "Jeopardy" style, as if it were revealing the categories in the round.

Anyway, it took Bob Barker longer to explain the rules of the game than it took the contestant to jump in a stall and win the car.  You would expect more from a prize reveal, and as it's up to the contestant to pull the rope, there's really no way Bob could have built it up in such the way as he was masterful at doing.  Not only that, but what if the contestant could look up and see what prize was there?  I'd have to imagine the big car key was hard to hide.  Nevertheless, it was great to see the contestant win, but it would have (hopefully) been more amusing if he'd have gotten some confetti on him first.  The game was obviously based on sheer dumb luck, with no strategy involved; not to mention, the shower stalls evoked imagery of the Holocaust among some viewers.  This would explain its quick demise.

I hastily proceeded to watch Finish Line, which had been posted next.  Finish Line was played 16 times, seen from February 21 through September 25, 1978.  Among the trio of obscure games I'm presenting, this was the longest-running one.  This game definitely had flair, and would have been a great replacement for Give Or Keep (seeing as how the rules were identical), had it not suffered from mechanical problems and just taken too darn long to play.  Give Or Keep itself was not a flashy game by any means, but the essence of the game in Give or Keep was not lost in all kinds of superfluous activities such as moving a yellow bar slowly down the track to indicate the Finish Line (i.e. how far the horse has to go, which is based on the sum of the prizes the contestant thinks has the lowest price) and then moving a tacky plastic horse slowly down this track until it either crosses the finish line... or doesn't.  Give Or Keep revealed this information right away; instead, on Finish Line, it takes a long time to learn the outcome.

On the surface, both games have quite uninteresting gameplay compared to some of the more modern games, which boast more challenges than simply picking the cheapest prize among three pairs of prizes.  Nevertheless, it's still a solid concept, totally fits the Price is Right theme, and is a quintessential game.  They might have luck bringing Finish Line back, in particular, if someone would rework the prop to use solid-state electronics and stepper motors instead of analog electromechanical devices (an entire wall full of relays, for instance) which are more prone to failure and inaccuracy.  (And, they'd need to pick a theme with a broader appeal than horse racing; plus I can see how in later years, as Bob became more of an outspoken animal rights activist, he'd want nothing to do with horse racing anyway.)  As for Give or Keep, that one's pretty much toast.  The staff tried for years to get it retired, probably because it was so simplistic, and ended up finally getting their way after 18 years of it in the Price is Right rotation.  At least one other game (Trader Bob) was built around the same concept, but it too only managed to last for a couple years.

Finally, I viewed the game among these three I was most excited to see: Telephone Game.  Even though it was only played three times, all airing within November 1978 (and played more times than only Professor Price), it has the most novel gameplay among these games.  Several elements contributed to this factor.  In a cue from Five Price Tags, you had to win at least one choice of prices among the three that could be for the car; except on this game, you had to buy two grocery items from among four with a budget of only $1 and save at least 10 cents to make a phone call on the payphone.  Then, in a twist unlike any other game I can think of, Bob would present the contestant with a phone book containing the price of the car in dollars, and the price of the other two items in dollars and cents (prizes less than $100, as new cars cost only about $4,000 or $5,000 back in 1978).  The contestant would then dial the price they chose on a rotary phone, and after a painful amount of waiting around, a telephone would ring next to the prize that had the price the contestant picked, and the model standing next to that prize would answer the phone.

The game started out solid, with picking out the grocery items, but then quickly went downhill from there.  Bob giving the contestant the dime seemed hokey, and then he had to tell the contestant to actually put the dime in the phone!  That seemed superfluous because it doesn't actually need to happen (unlike Master Key, where the contestant needs the key to find out what prize they've won).  Finally, in another stroke of "dumb luck," the contestant managed to win the car, only after arbitrarily picking the right price and waiting quite a while to see if he was right.  It was reminiscent of waiting for a cell phone call to go through when the network is really busy; sometimes it takes a while for the phone to start ringing!  Maybe they only strung this game along through three playings to see if anyone would actually win it, because after this game was finally won (on the third try), it was retired and never to be seen or heard from again until clips of its third playing finally resurfaced in 2014.  This one could possibly be revived if they could somehow remove the painful elements of watching the game unfold; perhaps this means taking away the whole "telephone" theme and playing through it just for what it is.  However, then it'd become a little too close to Five Price Tags except that you would win something on the first try as opposed to continuing to try for the car (or getting nothing).

Hey, that was cool!  May I have another?  How about that Punch-A-Bunch?

Since Wink Martindale has posted additional rarities beyond these three obscure games, I'd like to go into one more as a bonus.  The first game played on The Price is Right to offer a big cash prize had remarkably different rules in its first 11 playings than it does now.  The original rules for Punch-A-Bunch throughout 1978 had a higher "suspense beta" (where "beta" is a stock-market term for volatility) than it does now.  While Punch-A-Bunch is (and has been for a long time) a very exciting game, it could have been originally super-suspenseful or super-boring.  The contestant gets to choose which one of the four prizes they would like to attempt to earn a punch with, and then before punching, they must pick a letter on the top row among "PUNCHBOARD."  Each letter contains a number from 1-10, so if the contestant picks a disappointingly low number, it becomes lame (kind of like in the mid to late '80s when someone wins the $6,000 showcase instead of the $20,000 showcase).  This number will then be multiplied by their punch from among the 50 slots -- where nowadays there are values between $100 to $25,000, back then there were "DOLLARS", "HUNDRED", and "THOUSAND."  Thus the "suspense beta" is pretty high if someone were to pick the number 10 from among PUNCHBOARD, then pick "DOLLARS" (and as such, would be a huge disappointment!), but less so if the contestant picks the number 1 and then goes on to pick THOUSAND.

The other thing that makes the original Punch-A-Bunch rules wildly different is you don't know how many punches you will get ahead of time!  Since you picked prizes one at a time, you pulled your punch after each prize.  If you wanted to give your cash winnings back to try for another punch, you ran the risk of losing the other prizes and not winning another punch, thus walking away with nothing.  That was a nifty element of the gameplay, but ultimately slows it down and adds to tedium while watching it.  However, it does introduce a bit of strategy most contestants probably didn't pick up on: you would want to pick the prize you are least confident about first so you'd be less likely to give up your cash winnings and then not get another chance. Punch-A-Bunch started 1979 with the format we all know and love today, which definitely plays through quicker yet still provides plenty of entertainment and suspense.  The rule change was wise, as the game has been a favorite among fans ever since.