Chimpunkan  -  D'Jacamo

A few nights ago my girlfriend called in a tizzy. She couldn’t get online. I had told her once before what to try, but in her tizzy she didn’t remember. I walked her through a few things on the phone, then went over to her apartment, did them again, and she was back on.

The Internet has become so intrinsic to our lives that when we’re without it we panic. People react like as though it were a rapidly spreading grease fire. All of a sudden they can’t remember how to put it out (baking soda or baking power?!) and repeatedly fumble finger 911. It would seem a sewer line backflow is less alarming.

If you know computing you’re acutely aware of the cliché that you become your friend's and family’s tech support, because it’s a truism. I wonder if the other technical fields suffer from this as well. I think probably not, because in the scale of technicians, computer people rank at the bottom, below plumbers and baristas.

So we have an incongruity. Critical systems, now necessary for daily life, supported by a cadre of nephews. Painful for everyone involved.

Grandmothers, grandfathers, mother, fathers, sisters, brothers, friends, girl friends and wives, it’s now upon you to take responsibility for your own internet. Don’t be reluctant, it’s not difficult. You will have to learn something, though, and if that’s a problem, you’ve accidently happened into the wrong century.

In an effort to empower you, here is a simple guide for you to follow if your internet is not working. Experience has proven that the following steps solve most everything that is truly not a hardware failure of some sort.


If you actually read all of this, took the time to understand it, and use it the next time you’re offline, you may experience one of our most enjoyable emotions: for the win! (http://en.wiktionary.org/wiki/for_the_win)

Over a year ago, I wrote:

"What I write here is true; purely non-fiction. My credentials aren't important. The significance in what I write comes from its veracity. If something is my opinion, I'll try and make sure it's doubly apparent."

While that has remained true, but I've posted much less than I would prefer, and not frequently enough to make the site interesting.

So I'm going to include more opinion, more current developments in the developent world, and a more excerpts from my career.

Essentially, I'm going to tell you what I think more often.

So why should you care what I think?  I've been in software development since 1995.  I'm what you'd probably call a "journeyman" programmer.  I've worked in most of the common technology, dabbled in the others, and am currently creating software for the web (ugh).

Again, the veracity is what matters.  I think you'll find that I'm sufficiently veraciful to warrant reading.

Perusing the tech news I often come across statements made by people or companies that are obvious lies.  You could argue for a less pointed synonym to "lie," but they're lies.

The recall is due to a "manufacturing error" (they made a last minute code change and pushed it past QA to meet the deadline); only  a few people's personal information were compromised (100,000 records were stolen, but hopefully only a few will notice being exploited); the database backup was "corrupt" (the configuration was changed and no one bothered to check to see if backups restored); the content was removed because it violated our "terms of service" (it's competition).  I could fill the page.

Because we work in a highly complex and specialized field we cover our mistakes with lies in the assumption that those less technical will not discover the subterfuge.  Those of us in the know turn a blind eye because we wish to use the same artifice to cover our mistakes.  It's a mutually agreed on level of disingenuousness that the entire industry feels entitled to use.

Perhaps those slippery statements cannot be easily disproved, but the effect is to create a pall of mistrust.

So no one believes us.  Do you trust the statements made by companies when it hits the fan?  We all know they're covering.

It's an ugly smear on the industry, being perpetuated by cowards.

When you do something wrong or make a mistake, be it as a company or an individual, 'fess up.  You'll have some pain in the near term, but in the long run you'll be building trust for both yourself and the industry as a whole.  And it might go a little way towards putting some respectability back into our profession.

Otherwise, we're just spineless lying nerds.

Not so long ago I enjoyed a love affair with computers.  Each morning I would wake up looking forward to what I might do on my computer that day.  I would get a happy feeling seeing a boot screen, enough to make me do a few little leaps of joy.  I marveled daily at the wonder of just hitting a key on the keyboard and it making something happen on the screen, the way a child does when they first encounter a computer.  I would sit an code for hours, the way a man paints and polishes his beautiful sail boat.

I let it get taken away from me.

It got taken away by switch ads and browsers as UI and infighting and DRM and ignorance and corporations and greed and stupid demands and money and fear and people who wanted a piece of me and the beer I drank to take it all.

*&%# all of that.  *&%# it all to Hell.

I'm getting my love affair back.

I'm not much for web browsers as user interfaces for a lot of reasons, but one of the primary causes of my consternation is JavaScript.

Poor JavaScript, the bastard child of Netscape, ECMA, and Microsoft, tied to brittle browser DOMs and saddled with truly weak typing, a clumsy object model, and arbitrarily implemented built in functions. Its name is even misnomer, as JavaScript has nothing to do with Java, being given that name by marketroids to make it sound trendy (imagine, Java trendy!). But despite all this JavaScript excels in many ways, and that's what can make it so infuriating. It can be clever, but more frequently, too clever.

For example, the following code:

var x = '';
var y = 0;
if (x == y)
{
    alert('wtf?');
}

If you run that it's going to put a 'wtf?' on your screen. Why, you may rightfully ask, is '' equal to 0?

Well, it goes like this. In Javascript an empty string and zero are both considered 'false' values, as opposed to a non-empty string and 1, respectively, which are 'true' values. So really you're asking if false == false, and of course that's true. Javascript is doing the right thing, in its own little wacky world of Booleans. For those of us who actually have to program in the language it'd be nice if true was true and false was false and an empty string was an empty string and 0 was just plain zero. I know Javascript is weakly typed, but 0 being equal to an empty string is positively feeble .

I guess I should know better than to compare two different data types and expect something good to come of it, but then, what good is the dynamic typing if not for such occasions?

Turns out the problem is we're not using the right equals. We need to use the 'strict' equality operator of ===, instead of the 'non-strict' version which just can't quite bring itself to impose discipline on those unruly comparisons. The strict equality operator compares both value and type, bringing the unadulterated logical hurt.

So:

if (x === y)
{
    alert('I see dead code...');
}

I think JavaScript didn't go far enough. I think there should be a 'super-strict' equality operator of ====, which compares value, type, and whether I should be using a more sensible language for my browser UI scripting. Unfortunately, in my case, it'd always return true.

Company management has discovered that sometimes if you take a chance on someone green and unproven, but obviously bright, they often turn out to be excellent hires. Some of our best developers came to the company in this way. Of course, some don't pass muster, and sometimes you get someone who's more than a little peculiar.

Patrick was hired because he had a bit of web development work on his resume, seemed sane, and we needed people. We knew it'd be a bit of an uphill climb for him to transition into the technologies we use, so we were all prepared to help him out the best we could. His orientation explicitly emphasized that he was to ask questions and seek direction from his peers. He took a desk next to mine and I made it abundantly clear I was available for any and all inquiries he might have at most any time. Others did the same. He seemed a bit quiet, and we wanted to reassure him, make him feel welcome. I suspect our coders can be unintentionally intimidating. We're a tight-knit group, all sending and receiving on the same wave-length, both technologically and personally.

Everyone takes some time to settle in and become personable, but Patrick wasn't speaking unless spoken to. He wasn't asking questions, and a lot of us couldn't figure out what he was doing most of the time. The Lead Developer took him under his wing for a few days and tried to get him up to speed on C# and have him contribute to a project, but as soon as Patrick was left alone nothing would happen. He'd come back with questions showing he had virtually no understanding of what he was being taught. We wanted to help, but we couldn't figure out where to start, and he didn't give us a clue by asking. He never asked for help on anything.

As the weeks went by and we discovered Patrick had some interesting abilities, though. First, he could sit all day at a desk and not say anything to anyone. We assumed he was getting tasks handed down from management, tasks he could work on with his limited skills, but peeking over his shoulder I couldn't figure out what that might be. Second, he had a ninja like ability to leave at 5PM without anyone noticing. As the end of the day approached I would remind myself to say good-bye to Patrick, to maybe include him in a bit of office camaraderie. But when I would remember and turn to say something he was always gone. Not once it many weeks did I see him leave at the end of the day. No one did.

Often I forgot he was next to me. He had a Zen-like ability to just evaporate in a room of developers. He didn't clear his throat, sniffle, or sneeze, and even his typing was silent. I considered giving him a squeaky chair just to tether him to the physcial world.

It eventually became obvious that Patrick wasn't working out. He was nice enough, but he'd never asked a single unprovoked question, no matter how many times you encouraged him. He wasn't connecting with his peers, we couldn't find any technology with which was proficient, and he wasn't progressing. He came to just one lunch with us, where I learned he moved from another town 150 miles to the north, had got a house and moved his family down, all for this job. It made his lack of progress all the more painful.

We kept dumbing down the tasks we gave him, trying to find something useful for him to do. One day he was given the task of simply putting a form on an existing page to collect survey information. We detailed what needed to be done and set him to the task.

A little after noon the Lead Developer was asking around for Patrick so as to check on his progress. Had anyone seen him go to lunch? No, he'd slipped out like he does at the end of the day. Hours went by. He'd always been punctual, so we were a bit perplexed. I pictured him materializing in his chair when no one was looking.

I looked at his desk and noticed something peculiar. The sparse collection of personal effects he had were missing, and all the company owned books he'd borrowed were neatly stacked on the corner of his desk. I informed management, and they tried his home phone - disconnected. They tried his cell - no longer in service. We used an admin login to get access to his machine and found it had been cleaned. Nothing remained except his installed software, his SVN projects, a handful of company email on the Exchange server, and the machine wiping utility he's used to erase all his personal information. It resembled our pristine developer setup.

Looking at his last project, he had gone to some free survey service and taken their generated HTML source and pasted it whole cloth into the existing page. He still had it posting to the survey service's servers instead of our own. It blew up as soon as you tried to run it.

A few days later someone drove by his house and found he no longer lived there.

Patrick was gone. Unsurprisingly, he'd said nothing to anyone. The last day was identical to the first and like most all the others. His vacated desk gave me the creeps, as he sometimes did when he sat behind me, so quiet and unapproachable.

We never did find out what happened to him. I wonder about him sometimes. I hope he found a job where he can do well. But mostly I hope he doesn't come back and use those ninja skills to sneak up and gut me like a fish.

A few hours after posting the proceeding story on Feb 21st, "CAT 5 Underfoot", I searched Google to see if by some outrageous fortune it had already been found and indexed. I searched on "CAT-5 Underfoot" and, surprisingly enough, there it was, the top search result. Color me impressed. Google is on the ball.

But I then I wondered, is "CAT-5" correct? I did another search, and found that it's usually written just "CAT 5", without the dash. So I logged back in, changed the title from "CAT-5 Underfoot" to "CAT 5 Underfoot," and made a few other edits to the post. I found that even if searched for "CAT 5 Underfoot" it still listed my post as the top search result, even thought the title didn't exactly match.

Out of curiosity I checked the IIS logs and sure enough there were some Googlebot entries, one being:

2008-02-21 00:09:10 66.249.73.242 - 192.168.254.25 80 GET /CAT5Underfoot.aspx - 200 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)

I figured I just got lucky and Google crawled my site soon after I posted. I can't imagine that they'd be so frequently crawling my pathetically unknown and seldom visited blog.

Then today, Feb 22nd, I decided to check and see if Google updated the title of the post. I search on "CAT 5 Underfoot". Nothing. I search on "CAT-5 Underfoot". More nothing. "CAT5 Underfoot". Big nothing. I search on "CAT 5 Underfoot Chimpunkan". Still nothing. WTF? Some of my old posts are still indexed, such as "middle name field required", but others are not that were certainly indexed before.

So I go to the Google Webmaster Tools for my site, and find this:

"Googlebot last successfully accessed your home page on Feb 12, 2008"

Really? Even though my own logs indicate that the Googlebot hit my homepage, and the exact page I'm searching for, on Feb 21st?

This can mean a few things:

• Google's Webmaster Tools don't report accurate crawl dates.
• By changing the title from "CAT-5" to "CAT 5" the post somehow became less findable. That doesn't account for my other posts no longer showing up, though.
• That by changing the title from "CAT-5" to "CAT 5" I ran afoul of some Google algorithm no-no and caused some of my posts to fall off the search results. Maybe I revise too much for Google.
• I broke some other Google (undocumented?) rule.
• Google's got issues.

The upshot of all this is that Google is dishonest at worst, capricious and inscrutable at best. Not that we didn't already know that. While this is not necessarily evil, it's sure as Hell annoying. And if you have a business that depends on Google search results and PageRank, you're in bed with a fickle lover.

Update - 2/22/08 10:15AM:

Less than two hours after posting this blog entry, Google has too picked it up and indexed it. Now searching on "CAT 5 underfoot" returns this post. I really can't imagine how Google is able to so quickly crawl and index my blog. Their Googlebot is beyond uber. But the Google Webmaster Tools page still says Feb. 12th as the last date Googlebot crawled this site. Not so uber.

For now, I'll give them a solid 'E' for effort, but not yet for evil.

Update - 2/24/08 10:15AM:

Will the weirndess never cease? As of 2/24/08 10:15AM, searching Google for "CAT 5 Underfoot" or "CAT-5 Underfoot" yields nothing in at least the first five page of results, not even this story, which was the top result for the same search on the 22nd. Amazingly, absolutetly nothing is returned by "CAT 5 Underfoot Chimpunkan". That makes this a two "WTF?!" post.

It must be sweet to be so massively influential yet so completely unaccountable.

Update - 2/25/08 7:40AM:

This morning searching Google for "CAT 5 underfoot" turns up this post, as it did on the 22nd. It's as if the activity of updating the post caused it to bubble up to the top of the search results again. We'll see how long it stays the top result. And for that extra dose of inscrutability, the Google Webmaster Tools pages still says the last time Google crawled my site was 2/12/08. That's after jumping through their "verification" hoops yesterday, too.

Update - 2/25/08 8:32PM: 

And, the post is gone again.  Seems like a good strategy to keep your Google ranking high is post a lot. 

Our company had grown quickly and our server room was less than organized. It consisted primarily of two gorilla racks packed with dozens of custom built servers, and at the far end switches, hubs, routers, and a fiber drop. Getting to the machines was tricky, involving careful navigation of all the cables, UPSs, and other miscellaneous network hardware that cluttered the approximately three foot clearance behind the gorilla racks. Anyone who had anything to do with the server room was exceptionally aware of our two main servers, Solo and Chewy, which housed our client's most important sites and databases. These two machine were sacrosanct, and were monitored by the second.

Our main networking ace had recently left the company, and we were coping with his replacement - his previous apprentice. She might have known the technology, but in other areas she was a bit lacking. I was trying to cut her some slack since she was inexperienced, young, and the rare woman in technology (my personal version of Affirmative Action that I've since abandoned). Eventually it became obvious she was in the wrong line of work.

One afternoon I get a call from a client: their site's down. It's a site on Solo. I do the required panic, and with a quick check, sure enough their site is down. Wait, no, every site I try on Solo is down.

I IM networking:

"Do you know that Solo is down?"

"No, I was just in the server room and everything seemed fine."

"I'm pretty sure it's down."

"Maybe the outgoing DNS is messed up again and we can't get to our own sites."

"No, a client called. They noticed it was down. I just verified it."

"Just a sec."

I get up and walk over to networking. On her screen, next to our IM conversation, is "Sid Meier's Pirates!" She's piloting a ship around.

"I found this on Underdogs. I love this game. I haven't played it in so long. I'm pretty sure Solo is fine. It's something wrong with our internal network."

"Yeah, I like that game, too. Could you let me in the server room?"

We head over to the server room, she unlocks it, and we go in. It's freezing cold and loud. Solo and Chewy are on, but with little disk IO, and Solo's networking light isn't flashing.

I switch the KVM to Solo and check all the obvious things. Nothing seems amiss, but it's off the network. I check the back of the machine, everything is plugged in, but I reseat the connections anyway. Still nothing.

"What did you do in here just a bit ago?"

"I plugged a new machine into the switch."

"It's voodoo, but let's disconnect it and see if that changes anything."

I stretch through to the back of the switch rack, and with some serious contortions, peek behind to find the right cable to disconnect. Thank God she labeled it. More nothing. I'm looking at the serious snarl of cables and hardware and begin to realize I'm in trouble. I start grasping at straws.

"Okay, could you show me exactly what you did when you were in here?"

"Sure. I brought in the machine and put it up here. I plugged it into the UPS, and got a long network cable for it. It's a long way from the switch. It's headless, so that's all I did. It's not even on yet."

"How did you choose the port to connect the plug into?"

"I just chose an open one. I'll plug it back in and show you."

She squeezes behind the gorilla rack and blithely proceeds to tread on the cables lying on the floor, sometimes yanking a machine's connections so hard that the case wiggles. She notices my whole-body cringe and quickly backs out.

I then squeeze behind the gorilla rack and start pushing in every single network connection I can see, be it machine, hub, switch, or router. She starts pushing them in from the front. I do my best to step in the little areas of exposed concrete amongst the tangles. A few minutes later Solo was back online.

I never knew which connection it was that was loose. Later we cleaned up the cabling and it wasn't so much a concern. Our Peter Principal network admin didn't stay around much longer. She had other telling mishaps: database backups that failed due to the drive being filled with her MP3 collection; setting up IIS and leaving the default SMTP relay open getting us blacklisted; and sometimes assigning the same address to two different machines or an address from the DHCP pool, sending us on a company wide game of IP address Whack-a-Mole. Worst of all was coming in late, being unreachable in emergencies, or completely out of it because she had a good night partying. I think she too eventually realized she was in the wrong line of work. Last I heard she was managing a bar.

At the height of the dotcom boom the company I worked for had hiring practices that bordered on philanthropy. The owner of the company has a kind heart and truly wanted to use his success to help others along. Often people with little or no technical skill were hired and then moved about the company to try and find a fit. Often it worked and we would get someone willing, astute, and eventually useful. Others would move about until they settled into a position where they could do the least damage until the bubble burst and they were laid off. I was one of the beneficiaries of this largess, being allowed to grow into a senior programmer from humble self-taught beginnings, so I never looked askance at these hires.

One such hire was brought on under the company's unspoken nepotism policy. We were literally a Mom & Pop software company, where cousins and in-laws minded the store. He was some sort of distant cousin of the owners, nearly technically illiterate, and had spent most of his life as a surf bum. He was good natured and well liked, but useless for developing software. He was infamous for coming into the developer's room and asking inane questions, but one particular instance has become legend.

"Hey, Dudes, sorry to bother you, but I'm totally stuck. I'm working on this project for the boss and they used Roman numerals in the notes. You guys always tell me to try and figure things out myself first, so I've been looking all morning in Word, and in my fonts, but I can't find the Roman numeral font. It must not be on my machine or something."

All you could hear were case fans.

Then someone spoke up.

"We don't have time to help you with that. Just use upper case I's, V's and X's for now."

"Oh, yeah, that's a good idea. Thanks man! Later."

My name isn't D'Jacamo. That's my nom de clavier because I want to hopefully remain anonymous. You could probably figure out who I am with some proficient Googling and network analysis, but that wouldn't get you much. I'm not a luminary, just another programmer writing software the best I can under the circumstances. It's not easy.

What I write here is true; purely non-fiction. My credentials aren't important. The significance in what I write comes from its veracity. If something is my opinion, I'll try and make sure it's doubly apparent.

So read on if you like. I hope to make you think, laugh, and maybe worry a bit about the runaway train of technology we're all aboard.

We frequently hear about credit card data that has been stolen from some ecommerce site. Usually the thief cracks their way into the records, or the information was inadvertently exposed due to ignorant coding and lax quality assurance. This has led to an emphasis on hardening servers and software. SSL has become mandatory. Certification firms provide vulnerability detection services allowing ecommerce sites to display a "hacker safe" badge touting their security. Password have become less crackable, with the ubiquitous "at least one uppercase, one number, and one special character" requirement. We wrap our database queries in code to thwart SQL injection. Our software constantly warns us not to give out personal or financial information over insecure channels.

Those are all well and good, but truthfully, they are rendered futile by the development practices of ecommerce web site owners and operators.

Development of websites has become globally distributed and is often performed by the "virtual corporations" we expected to arise from our networked businesses. Often an ecommerce site is created by many developers, many of which don't even know the others exist. A typical case could be the design done in San Francisco, the graphics in Seattle, the database in Houston, the payment gateway in the Philippines, the customizations in India, the integration in New York, and the hosting at the cheapest possible place that the owner can find.

To pull all these disparate resources together the website owner needs to give these far flung developers assets and security information. They need database connection strings, admin logins, SSL keys, FTP logins, hosting provider access, domain logins for terminal services, source code, and database backups.

This is where it all breaks down.

As a business operator bent on making money they handle these resources themselves, or delegate them carelessly. They are interested in having their ecommerce site developed for the least cost possible, not in the security of their customer's information, at least not unless it affects their profit margin. They often don't even know the nature of what they are providing to developers. Consequently they potentially expose their customer's information at every stage of the development process.

I have seen entire database backups, with thousands of credit card numbers, sent through email. I've been forwarded all the security information available because the owner did not know, and can't be bothered to know, what particular credential I needed. I have been provided with logins that were meant to be temporary but still work two years later. I have submitted code that was never reviewed by anyone other than me.

Often these exchanges are in the form of, "I need X done on my current site, here's all the information I have. I don't know what most of it means, but I'm sure you will." Enclosed is every possible login and password they have created or been given.

Often this information is provided before, as a hopelessly conscientious developer, I can stop them. Asking them to adopt more secure practices is asking them to incur cost. Cost they cannot afford because most of them are hanging by the thread of their current Google ranking.

So realize that every time you submit your credit card number to a website you're risking it being exposed to some random developer in some random country. They give away the keys to their kingdom to whoever can get them what they want the cheapest and fastest.

It's disconcerting how little this vector for potential security breaches is discussed. Is this a development community dirty little secret? Are we so thoughtless that we just take care of our little chunk of the project, take our money, and the greater good be damned?

I write ecommerce sites for a living. I wouldn't submit my VISA number to 99.9% of the sites on the internet.

On site at a large government agency to investigate reported problems with our software:

"I'd show you the errors I'm getting but I changed my password yesterday and forgot it and now I can't get into my computer."

"It's 10:30?! How long does it take for the networking people to respond?"

"Well, the request form is on the network, and I couldn't get into my computer, so I used Marcie's computer to report the problem, but the form doesn't have a place to say who has the problem so they went by Marcie's desk at 8:30 and I wasn't in yet. She said she told them it was me."

"Aren't they just down the hall? We could go ask them."

"They don't like it when we do that."

"Hmmm. What did you think you might have changed it to?"

"I can't remember at all. I usually put it on a post it on my monitor, but networking told us we couldn't do that anymore."

The wall behind her desk has about 25 pictures of a dog.

"Does it have something to do with your dog?"

"No, it's not that. I would remember."

"Do you mind if I try?"

"Sure, but you're going to get us in trouble. Networking don't like it when you try to guess your password over and over. It locks you out and then they have to do something."

"What's the name of your dog?"

"Howie."

I type in 'Howie'. The desktop appears.

...

It's a picture of Howie.

On the phone with the client:

"I see in the written spec you want the middle name field required when filling out the form. I'm not sure that's a good idea since many people don't have middle names. When they submit the form it's going to prompt for the middle name when they've left it blank."

"We need the form to be complete. We have to have the middle names. Program it to be required."

"And it'll be okay if they're prompted to input a middle name when they don't have one and leave the field blank? I suppose the validation dialog could ask them if they have a middle name if that's acceptable."

"No, not really."

"I'm not sure how to handle this then."

"Make the middle name field required only if they have one."

"Hey, do you have any self deleting code?"

"I used to."