This recent post on CrunchGear shows off a new tablet app for Wired Magazines that is under development. This started the gears working in my head about a real problem emerging in the app ecosystem that has developed around Apple, Android and to a lesser extent RIM, Windows CE Mobile Phone, WebOS, et. al.

If I pick up a newspaper or magazine I know exactly how it’s going to work. I can skim through pages, read any article that grabs my fancy and curse at the subscription cards regardless of what type of periodical it is or who has published it. But in a world where we get our publications via app it’s possible that no two will ever be alike. Not only is it likely that no two interfaces will be the same but it’s also a near certainty that some will just plain suck. Many engineers are not designers and even those that are may not have any sense of good layout or UI. There shouldn’t be a learning curve when it comes to reading an article.

I envision this playing out:

• First, you hear about a great article around the water cooler at work.
• The next time you’re around your tablet you go and look it up.
• Curses, it’s not available online, but if you download the company’s app you can get it.
• Whoops, the app costs $4.99… well you heard it was a good story so you’ll pay it. After all there must be other articles you’d find interesting.
• After spending your lunch hour downloading the app and figuring out how it works you discover that the article you’re after isn’t in the current issue.
• You spend your next coffee break on Google trying to figure out how to get previous issues.
• Ah ha, there’s a separate app for previous issues… it’s also $4.99.
• After getting home you spend an hour on the phone navigating a phone tree until you can explain to the level one tech support person that you downloaded the wrong app.
• Level one manages to shrug his/her shoulders so over dramatically you can actually hear it over the phone. You’re then transferred to level two.
• Level two tells that there’s no way transfer the purchase price to the other app, but they’ll send you a refund card in 4 to 6 weeks so that you can get your five bucks back in 8 to 12 weeks. By the way you really should be calling the phone company with these types of issues.
• You go head an purchase the second app knowing you’ll get the the refund before you retire.
• After downloading and figuring out how this different UI works you’re exhausted and go to bed.
• You wake up the next morning enthusiastic that you’ll finally get to read this, surely, amazing article.
• You open the app which crashes, but it works fine the second time you try.
• You wade through 4 screens worth of advertisements and at long last you make it to the article.
• The article quotes two freely available AP stories and has one other paragraphs that amounts to “well, I guess we’ll have to wait and see.”
• …
• The jury turns in a guilty verdict and you’re introduced to your new cellmate Big Tony.

If the day comes where the majority of people are consuming their print media on portable devices like the Kindle or iPad I’m sure these issues will work themselves out. Until then this seems to be one of those issues where technology only serves to make our lives more complicated. Granted most of us won’t end up getting our salad tossed by Big Tony and his crew but the frustration is inevitable. On the bright side, if it does end up going to the extreme maybe the publisher will issue your refund in cigarettes.

Thanks to Stephen Poff for the CC image

I an effort to expand my knowedge base I’ve decided to learn a new language and undertake working with a full fledged framework for the first time. My webiste has suffered one of the worst cases or bit rot that I’ve seen. I’ve decided to redesign it and build it all using Python and Django. It will also give me a chance to work with JQuery and to experiment with some database design ideas.

I’ve also decided to kill two birds with one stone and work on being a better blogger at the same time. As such I’ll be doing my best to document everything I go through with this project. Look for that late this month or early in December.

I am firmly addicted to RSS feeds. This despite my assertion when they first came out that they weren’t useful to the end users of a website. They say the ability to change your mind is a mark of intelligence… right?

I currently subscribe to 66 feeds in google reader. On an average day this amounts to about 150 to 200 items that come through GR. I skim over quite a few and take my time reading the ones I like. This is a very manageable arrangement that take occupies about an hour of my day.

Then there are days like today in which just about every blog on the internet with any leaning toward technology has posted basically the same stories about iTunes, iPods, TC50, and/or Demo. As a result I’ll probably have sifted through about 500 feed items by the time the day is over.

What I want it some basic filtering built into to Google Reader. I am a big fan of the app in all other ways but the lack of filtering may actually end up being a deal breaker. I blame Apple… or at least the echo chamber that surrounds it. My desire for a GR filter started with the iPhone, grew to a serious annoyance with iPhone2, and has come to a head with iTunes8.

I know there are options out there. There’s a Greasemonkey scripts to add filters to Google Reader, but I have no desire to have Grease Monkey installed. There’s also yahoo pipes, but Yahoo’s been shutting a lot of stuff down lately and it would mean running every feed through Yahoo before I add it to GR which is less than ideal.

Now taking bets on when my desire for a filter starts to out-weigh my lazyness to the point where I switch to another RSS reader.

That was what, 4 characters more than 140?

I made it 55 hours without using the internet for anything personal. Then I got bored. Then I discovered some things. First I discovered that we’re going to Portland this weekend and a long drive means needing podcasts. My “Buzz Out Loud only” rule thusly had to be abolished. Secondly I learned that Spore came out yesterday and not Sunday as I had thought. This means my preorder should arrive at some point today and I’ll be damned if I’m going to wait.

Therefore I must fall back on the theorem of 2 + 2 = 5 for extremely large values of 2. Two+ days without internet plus 2 very good reasons to use the internet equals 5 days without internet. Success!

I actual have more reasons, but that makes my theorem more complicated. They include the season premier of Bones being available on Hulu, Friday company lunches with usually involve a fair amount of non-work related youtube, a need to brush up on my Perl skills, a largely empty DVR, and boredom. Oh the boredom.

Things I learned while not online: Aside from TV being far more insipid than I remember the last time I spent a fair amount of time devoting all my attention to it… not much. I played with the cat more than usual, which is good, and I finished (mostly) sorting all our CDs away space saving wallets.

Why, you might ask, am I doing this? A more cromulent question might revolve around why I’m telling you this. Never the less I shall only attempt to answer the first question.

1. I want to see if I can get more done if I’m not chained to my laptop. Twitter and Google Reader seem to occupy all my free time lately.
2. I want to prove to myself that I can do it. I don’t think I’m over dependent on the web, but it would be nice to know for sure.
3. I’m tired of it. The internet, at least in the circles I’ve been traveling lately, seems to be a deafening cacophony that frankly is giving me a headache.
4. I’m hoping my desire to read, dead since college, will return to me. I’ve got a stack of books that have gone unread thanks, at least in part, to my time online.
5. By starting now this will be over by the time my preorder of Spore arrives.
6. I couldn’t sleep tonight and in my daze this seems like a good idea.

For various reasons I am going to have to make three exceptions that will wholey invalidate this as an experiment at least from a scientific standpoint. Two of these are unavoidable and the 3rd is just… because.

1. Work
   There’s no way to escape this one. I work building a web application, I have to be able to use the internet to do that. I will, however, not be using the internet for anything recreational while working. I figure the timing for this is good since all of the shows I watch on hulu are currently on hiatus.
2. Email
   Over the years email has become my primary form of remote communication. Me giving up email would cut me off from most of my friends and family.
3. Buzz Out Loud
   This is that just because I mentioned previously. Firstly, this podcast is probably my current favorite thing in all of media. Secondly, I would go a little mad without any access to tech news. A half hour a day should be enough to keep me from going completely nuts.

I thought about keeping a journal like a civil war solder. Sadly I very much doubt that this experiment will produce any results that could be called interesting no matter how bored historians get in the future. I predict I’ll get some more chores done than usual and I’ll finish reading Watchman. Other than that it will probably just be business as usual. I will post the results here on Monday at the experiment’s conclusion… or shortly there after.

After thought: There are going to be over 1,000 unread items in my Google Reader by Monday morning. That is a little freightening.

And this is a photo from my 2007 Halloween cosume.

These are a creap pair of welding goggles that Bonnie found on eBay and I modified for my cyberpunk costume. I painted mine differently than they did and I added to optics from a very old digital camera. I also added the LEDs which blinked. Still if you look closely they are very similar.

I have verified this works with visa, mastercard, amex and it should work with discover and jcp too.

I did some searching for a fake credit card numbers that validate which probably put on some goverment watch lists somewhere. Via a blog post by the Rimm-Kaufman Group I found this Amex number which should demonstrate this method.

371449635398431

Step 1. Reverse the number
3 7 1 4 4 9 6 3 5 3 9 8 4 3 1 becomes
1 3 4 8 9 3 5 3 6 9 4 4 1 7 3

Step 2. Double every second number
1 3 4 8 9 3 5 3 6 9 4 4 1 7 3 becomes
1 6 4 16 9 6 5 6 6 18 4 8 1 14 3

Step 3. Any 2 digit sums from step 2 have their component number added together. In other words 14 is 1 + 4 = 5
1 6 4 16 9 6 5 6 6 18 4 8 1 14 3 becomes
1 6 4 7 9 6 5 6 6 9 4 8 1 5 3

Step 4. Add all the numbers together
1 6 4 7 9 6 5 6 6 9 4 8 1 5 3 becomes
1+6+4+7+9+6+5+6+6+9+4+8+1+5+3 which is
80

Step 5. Is the sum evenly divisible by 10? If so then you can be 1 step closer to sure that the credit card number is valid. There are also other things you have to check that are card provider specific. For example, all visa number start with 4 and are either 13 or 16 digits long.

Thanks to shawnzlea for the image

I installed the MiniMe version of the distro, the Live CD of which weighs in at just over 200MB. It’s meant to be just enough to get get everything up and running so you can then, through ATP or Synaptic, install just the software you want. This was one of the major appeals of this Distro to me. The reason I gave up on Window, while I doubt I’ll ever run OSX, and why I’m moving away from Ubuntu is that I don’t so much extra crap built into the OS from the start. I want my operating system to operate my system, I’ll take care of the rest.

First Impressions

The first thing I noticed what that PCLINUXOS was able to start using the broadcom wifi in my laptop right out out of the gate. I opened up the network configuration tool and it asked me if I wanted to use the windows driver for via ndiswrapper for the card since there is no linux driver. I said yes and was done. In Ubuntu I had to use the wired ethernet to search Ubuntu’s forums for the right driver to use and instructions on how to set up ndiswapper.
PSLINUXOS time: 30 seconds
Ubuntu time: 45 minutes

My graphics card installed and worked correctly right from the get go. In Ubuntu I had to install the driver from Intel which was a hassle. Though the hassle was more Intel’s fault that Ubuntu’s.
PSLINUXOS time: 0
Ubuntu time: 20 minutes

Either MiniMe doesn’t come with klaptop or the install didn’t figure out that I was running a laptop because the utility wasn’t installed. But since I already had a working eithernet connection all I needed to do was install the utility via Synaptic and I was able to get all my power managment working just fine. I had to tweek the sleep settings a bit to get the machine to return from being suspended properly, but I’ve had to do this with every linux distro I’ve installed on it.
PCLINUXOS time: 2 minutes
Ubuntu time: 0

In the file /etc/acpi/events/sleep I changed:
action=/usr/sbin/pmsuspend memory
to
action=klaptop_acpi_helper --suspend

I then installed firefox3, thunderbird, openoffice, cd burning software, vlc, flash, and ksudoku. These packages represent about 99% of what I do on my laptop. Getting all the files downloaded and installed from Synaptic took about half an hour. I wouldn’t have to have installed these packages in Ubuntu, but I would have spent a lot of time uninstalling others.
PCLINUXOS time: 30 minutes
Ubuntu time: 1 hour

I lost all ability to connect to my Windows desktop via samba when I upgraded to Hardy Heron. From what I have gathered this is a problem with the Nautilus file manager. I tried changing the Thundar, but I never could get things working. Ubuntu would hang and hang while trying to connect then finally say it didn’t have an application to open a smb connection. In PCLINUXOS it connect right away. No hangs, no problems.
PCLINUXOS time: 0
Ubuntu time: weeks without success

Still to Do

I need to get Photoshop running, but I’m confident that the instructions I posted previously on the subject will work fine in PCLINUXOS. I also need to get the media buttons on the front of my laptop to work. I haven’t even attempted that yet so I’m not sure what it will take. It’s a minor thing, but they come in handy for changing volume.

So far I’m very happy with this Distro.

I’ve been playing with Ubuntu since the Warty Warthog days and this is the first upgrade that has not been a significant improvement. That makes me want to not give up on Ubuntu completely and instead downgrade back to Gutsy until the issues get worked out. Then there’s the bandwagon part of me that wants to jump on the the PCLINUXOS bandwagon.

Tomorrow after work I’m going to install PCLINUXOS on my home laptop and if things go well I’ll consider putting it on my work machine too. PCLOS has it’s .roots in Mandrake… aka Redhat, but it’s a full fledge distro in its own right now and it uses APT. I can work with anything so long as I have APT.

Thanks to Hayden Simons for the photo

A week ago tomorrow IDX decommissioned the last of its image servers. Over the last 2 and a half months I migrated a little over 20 million images, about 480 gigabytes, from our severs to Amazon’s S3 service. Most of that time was spend just occasionally checking in on the migration scripts that I had written or rewriting our image acquisition scripts to work with S3. We download images from about 190 sources every night as we gather MLS data on behalf of our clients.

The best part of the whole image migration and overhaul is that image acquisition is now tied into into our data balancer system. Each MLS in our system has a time stored in the database that is the earliest we can reliably download data from that source. Once we reach that time in the day the MLS goes through a series of steps triggered by a cronjob that runs once a minute. First the data is downloaded from what ever source makes it available. This can be ftp, http, soap, rets, or even direct sql connections. Next the data is parsed and made ready for insertion into our database. Once processing is done the data is geocoded so that we can easily map all the properties.

This was where the process stopped. When the app was first written image scripting was rushed as we were trying to meat our launch deadline. The image scripts were on different servers, so the data balancer couldn’t act on them directly. Instead each was launched as its own cronjob on one of the image servers. Every MLS is unique in the way we acquire images and is constantly changing, as such each must have its own acquisition script. Now each of those scripts is defined in our database.

Once per minute a script runs on our EC2 server that looks for image ready flags in our data balancing system. When it finds one it checks the database for the specific file that should be run to gather images. The script runs and then resets the image ready flag. As with data our image sources are varied. In some cases we generate URLs based on a know syntax, in some cases we’re given URLs by the MLS. In this cases we don’t need to store anything. Often we get images from some FTP source, via RETS, or in one case we download binary stored as BLOBs on a remote SQL server. Needless to say it’s complex to get all these images from 190 disparate sources, so anything we can do to automate things better is good.

My next project is building WSDL web services using NuSoap. This is uncharted territory for me, so I’m sure I’ll have more to say on this subject later.

I knew it was risky to make a self congratulatory post about a feature that had just launched. All and all it was pretty successful and took a deent load off the server, but two bugs cropped up that were sever enough to force me to revert the code.

The first issue will be easy enough to fix. Our app has several features, including a property slideshow, that are called remotely via javascript includes which also rely on the results class. Because the caching mechanism needs a PHP session id to avoid having one user contaminating another’s search results these tools stopped returning any properties. Luckely I wrote the constructor of the results class to have an all purpose override array as one of it’s parameters. So all I need to do to fix this issue is to generate a sessionID for the javascript includes to pass through to the results class and they should work again.

The second issue is going to take more work and creativity. When I worte the results class I built out the featured properties function to be generic. My thinking was that any property lising that is owned by one or our clients is a featured property and thus belongs on the featured properties page. Our clients, however, seem to have disagreed. They’ve figured out that they can append search variables to their featured property URLs and do things like make featured properties pages that are only for million+ dollar homes, or just commercial listings, or… whatever they want. This is all fine and good when the featured property search is repoerformed each time the page is called and is user agnostic, but not so much when my caching mechaism was in place. The caching mecanism treats all featured properties searches the same effectively ignoring any search terms added on.

I reverted the the previous version of the results class from our SVN repository and all went back to normal. Once I tie up these last couple loose ends I’ll be able to push the caching mechanism back out as part of Wednesday’s doubledot release. Here’s hoping it goes better this time. The caching mechanism did seem to take a noticeable load off the server, so it seems like a worthy endeavor to retry.

About nine months ago I completed a reworking (aka complete rewrite from the ground up) of the application’s results class. This is the code that assembles all the properties that meet the criteria of the search that has been performed and makes them available for what every they need to do. Once all the various data tables had been queried the matching results were placed in a temporary heap table so that they could be sorted, filtered (based on client preferences and/or MLS rules), and truncated if need be. I decided to use temporary heap tables because they’re fast and since they’re session specific I knew that I wouldn’t have to worry about one user contaminating another’s results.

The system has been working beautifully for these last nine months but as our traffic has grown (now upwards of 44,000 hits a day) mySQL was having trouble keeping up. All the heap tables we using a lot of the server’s RAM and since the heap tables were being destroyed as soon as the page was delivered searches had to be rerun completely just to move from page to page.

Todays patched changed things. The heap tables are gone in favor of a searchCache table (one for each client in our system) where all search results end up. When the same search is run again (like when switching pages) the results can be pulled from the cache instead of all the data tables needing to be queried again. All results are tagged with the users PHP session ID to prevent result contamination and every 4 hours the cache is cleaned to prevent the tables from getting too large. Featured property searches are also cached in our system and, because they are the slowest queries we perform^*, they are cached for 24 hours until we get new data.

I’m pleased so far. The patch was uploaded to our server 8 hours ago and thus far there are no reports of problems.

Thanks to bob the lomond for the photo.

^{*Featured results are the slowest because of the number of tables that have to be queried. Normal results only have to query 1 table per MLS being searched because they are property type specific. Featured properties are property type independent and thusly upwards of nine tables per MLS may need to be queried.}

The book above arrived at my apartment via Amazon the middle of last week. I made the decision to learn stored procedures as a way to expand my skill base and hopefully reduce some of the load we place on our SQL server at IDX by writing more efficient queries. I was expecting a book that was a couple hundred pages long describing some method of chaining statements or doing caching or… I don’t really know. What I got was a 600 page book that describes a fully realized ANSI standard coding language that loosely resembles Pascal.

My surprise does not equate to disappointment. The new possibilities that stored procedures open up to me are huge and I’m eager to explore them. I started using MySQL back when it was somewhere in its version 3 life span. Back then all I was after was something easier to work with than files for storing data. I had just recently started coding in PHP and was frustrated by how much more difficult it was to work with files in PHP 3 than it was in Perl 5. It’s only been in the last 18 months that I’ve really gotten in to MySQL and all it has to offer. As such I didn’t really know much of anything about procedures despite the fact that other DB systems have had them for years.

As it turns out there are actually 3 types of programs that make up MySQL procedures. The first it the “procedure” which is a full procedural program including control statements and loops. The second type is the “function” which, once written, can be used in any SQL statement to increase mysql’s functionality. The third type is the trigger which is code that will automatically fire when every a specific event occurs.

If you have any interest in learning procedures I’d recommend it. The time spent learning will be valuable. I am coming to realize that SQL is the most interesting part of application development for me. Relational databases are more powerful than I imagined back when I first started using them. Good SQL is the key to robust applications.

My company is looking to fill 2 positions, Palo Alto has at least one open, EngineWorks recently moved to Portland in part for a better employee base, and I know at least one other company is looking too. I never would have thought that PHP would be the big in demand skill I would have. Glad I have that biology degree to fall back on.

2 baritone saxophones and a bass saxophone. *Glee*

I beat Amazon!

Okay… not so much beat as figured out, and not so much Amazon as my own scripting. A mentioned in my last post I was having issues with corrupted images ongetting stored on S3 during my migration process. I determined that errors were being introduced during the transfers between our old image server via ftp AND during the REST upload to AWS.

I implemented the MD5 check I mentioned in my last post and added a step to the S3 upload. After transferring the file to S3 I perform a HEAD request on the object which sends my back a header containing, among other things, content-type and content length. I can then make sure that the content-length matches the size of the image I downloaded and that the content type is some type of image (useful since all errors are delivered as application/xml).

Little by little I’m developing a rock solid PHP class for S3 file handling.

Thanks to Lumaxart for the image

I’ve been having S3/EC2 problems over the last couple of days. I was migrating photos for an MLS in Washington and after 2 days of moving images in the background I found that a third of them were errored in some fashion. What’s worse is that I haven’t been able to track down the issue yet. S3 gives an AccessDenied error despite my script setting the access control at public-read… I hope.

My S3 class isn’t fully formed yet. And the holes are now staring back at me from the ether. The biggest issue is that I can’t currently read or edit the ACL for any object already on S3. The AccessDenied error make me think that my script improperly set the control level to start during upload. There also the possibility that errors were introduced into the image while it was being moved from the old image server and that S3 is erroring because the headers and content-type say it’s a jpg, but it’s actually gibberish.

In the mean time migration is halted because I can’t risk losing images. When I resume migration I’ll put a simple hash checker in place. On the old image server I’ll have a script that will accept a file name from _GET and run an MD5 on the request file file like so:
$file = $_GET['file'];
echo md5_file('my/base/path/'.$file);

Then I can read that output via curl and compare it to the MD5 of the file I just got via FTP. Images are our bread and butter. I have to be sure that I have an error free system before I continue.

Image from the Fail Blog

First I recommend that you uninstall any existing installations of wine you might have. This may be problematic if you have existing applications running through wine, but I was working on a clean system. Once you’re wine free install the latest version of wine by adding the their repositories thusly:
wget -q http://wine.budgetdedicated.com/apt/387EE263.gpg -O- | sudo apt-key add -
sudo wget http://wine.budgetdedicated.com/apt/sources.list.d/gutsy.list -O /etc/apt/sources.list.d/winehq.list
sudo apt-get update
sudo apt-get install wine

As of the time of this writing that will install wine version 0.9.57

Next you’ll need to make sure that the times32 font is installed in wine for Photoshop to be able to run properly:
wget http://heanet.dl.sourceforge.net/sourceforge/corefonts/times32.exe
wine times32.exe

Now you’ll need your Photoshop CS2 CD. Locate the Setup.exe file on the CD and run it using:
wine Setup.exe

That’s it. Once setup has run through you should have a working copy of CS on your system. You can now run photoshop using the command:
wine ~/.wine/drive_c/Program\ Files/Adobe/Adobe\ Photoshop\ CS2/Photoshop.exe

In order for the clone tool to work properly you’ll need to set the Movement Key in Ubuntu to “Super” aka the windows key. To do this open the systems window preferences by going to System->Preferences->Window

You also may want to increase Photoshops UI font size by going to (in CS2) Edit->Preferences->General, and change UI Font Size from Small to Medium.

When I first attempted to use Linux as a desktop operating system back in `96 or early `97  I remember it being a giant hassle to even get X-Windows to recognize my serial mouse. Flash forward a few years and everything is getting so easy that I almost miss the old days. At least back then you had to be a proper geek to run MS-Free.