What a difference a couple of lines of code can make.

A student asked me for advice for what to focus on getting into the compsci field. Here’s what I said.

Marco struck a nerve with his latest post lamenting the declining quality of Apple software. The post was picked up by “analysts” and debated on television by a panel of “experts”. While I understand the frustrations of those affected by more serious bugs than I’ve seen, I can’t help but wonder if they really understand what the alternatives are like.

It’s good to think of a computer as something like a cake with several layers. If the hardware is the first, and foundational layer, then the operating system is the second, and applications are the third. Today, we are going to look at that second layer, and leave with a basic understanding of what an operating system is, what it does, and what the differences are between the major operating systems available today.

GigaOm is running an article written by Ralph Dangelmaier, the CEO of BlueSnap, claiming “We’ve reached the end of ‘build it yourself’ software.” It’s a nice thought, along the same lines as “We’ve reached the end of ‘host it yourself hardware’,” and “We’ve reached the end of you needing anything other than what someone else has already developed.” In the past fourteen years I’ve been in the industry though, the systems I’ve seen run the best are the ones hosted on our own hardware running our own code. Off-the-shelf software can be great for certain situations, but if you are outsourcing a core function of your business, what kind of value are you really providing?

In the course of load testing a new system, we gathered the output from iostat from a group of servers. In addition to parsing through the device statistics, we thought it would be handy to graph the CPU stats as well. We set iostat to run every five seconds and captured the output in a text file, one per server. This gave me a sizable pool of data, but with everything I needed on separate lines.

I’ve been a Mac user since returning to the states in 2003, but I’ve been a Unix user for a few years longer. Originally, I was drawn to the Mac because it was a fantastic interface on top of a solid BSD Unix core, but as I grew more familiar with the platform, I became more drawn to the level of attention to detail. The Mac is a quality machine, but today I’m writing this on a Dell running FreeBSD.

Introduction

High Availability, HA for short, refers to an applications ability to continue operating after a hardware failure. HA comes in many different shapes and sizes, but two methods in production today are the presence of multiple machines performing the same task, and pairs of machines in a master-slave setup. Sometimes the master-slave setup is extended to include several slaves, but the main idea is that if the master should go away, the slave will pick up where the master left off, with no interruption in service.

We charged money for a good or service

I know this one is controversial, but there are enormous benefits and you can immediately reinvest a whole bunch of it in your project sips daiquiri. Your customers will appreciate that you have a long-term plan that doesn’t involve repackaging them as a product.

We’ve been having a months long discussion at work around which Linux OS to use. It’s all come to a head recently, and it looks like the winner is going to be Red Hat. The decision leaves a slightly sour taste in my mouth, but over the course of the past year I’ve gotten used to having it around. While trying to understand why I’ve got such a dislike for this particular flavor of Linux, I thought it might help to take another look at OpenBSD.

To enable an interface on a web server to be part of an IBM load balanced cluster, we need to be able to share an ip address between multiple machines. This breaks the IP protocol however, because you could never be sure which machine will answer for a request for that IP address. To fix this problem, we need to get down into the IP protocol and investigate how the Address Resolution Protocol or ARP, works.

Comparing two server operating systems, like SuSE Linux Enterprise Server (SLES) and RedHat Enterprise Linux (RHEL), needs to answer one question, “what do we want to do with the overall system”? The version of Linux running underneath the application is immaterial, as long as the application supports that version. It is my opinion that we should choose the OS that supports all of our applications, and gives us the best value for our money.

There’s been times when I’ve walked away from the command line, times when I’ve thought about doing something else for a living. There’s even been brief periods of time when I’ve flirted with Windows servers. However, I’ve always come back to Unix, in one form or another. Starting with Solaris, then OpenBSD, then every flavor of Linux under the sun, to AIX, and back to Linux. Unix is something that I understand, something that makes sense.

Back in ‘96 when I started in the tech field, I discovered that I have a knack for understanding technology. Back then it was HF receivers and transmitters, circuit flow and 9600 baud circuits. Now I’m binding dual gigabit NICs together for additional bandwidth and failover in Red Hat. The process, the flow of logic, and the basics of troubleshooting still remain the same.

To troubleshoot a system effectively, you need to do more than just follow a list of pre-defined steps. You need to understand the system, you need to know the deep internals of not only how it works, but why. In the past 13 years of working in technology, I’ve found that learning the why is vastly more valuable.

Which brings me back to why I love working with Unix systems again. I understand why they act the way that they do, I understand the nature of the behavior. I find the layout of the filesystem to be elegant, and a minimally configured system to be best. I know that there are a lot of problems with the FSH, and I know that it’s been mangled more than once, but still. In Unix, everything is configured with a text file somewhere, normally in /etc, but from time to time somewhere else. Everything is a file, which is why tools like lsof work so well.

Yes, Unix can be frustrating, and yes, there are things that other operating systems do better. It is far from perfect, and has many faults. But, in the end, there is so much more to love about Unix then there is to hate.

Today I found myself in the unfortunate situation of defending Linux to a man I think I can honestly describe as a Windows zealot. I hate doing this, as it leads to religious wars that are ultimately of no use, but it’s really my own fault for letting myself be sucked into it. It started when we were attempting to increase the size of a disk image in vmware, while Red Hat guest was running. It didn’t work, and we couldn’t find any tools to rescan the scsi bus, or anything else to get Linux to recognize that the disk was bigger. I was getting frustrated, and the zealot began to laugh, saying how easy this task was in Windows. Obviously, I felt slighted since I’m one of the Unix admins at $work, and decided I needed to defend the operating system and set of skills that pays the bills here at home. And so, we started trading snide remarks back and forth about Linux and Windows.

After spending a week with Linux as my sole computer, I find it very refreshing to come back home to my Mac. FedEx says my wife’s PC should be here tomorrow, so she can go back to Word 2007 and I can have Mactimus Prime back. It’s not that I didn’t enjoy working with Linux, I did, but I’ve found that once the geeky pleasure of discovering something new wears off, there are problems.

I’ve been using Linux for nearly a decade now. I first had a guy I met in a Navy school install it on my old IBM desktop. Back then, it was very hard to find the right drivers, and just about impossible to get on the Internet, seeing how almost every modem that shipped with a PC was a WinModem. X11 configuration was error prone to say the least, drivers for the sound card were hard to come by, and out in the rural English countryside where we lived, broadband was almost unheard of. Installing software could be a nightmare. Say I wanted to install a music player to listen to my CDs. The music player would have dependencies, certain libraries that needed to be installed, so I’d go and download the dependencies and try to install them, only to find out that the dependencies had dependencies! So, I’d download further dependencies, and eventually I’d be able to listen to my music. And then I’d try to launch something else, only to find out that in fulfilling the dependencies of the music player, I’d broken the dependencies of the other application that used to be installed and working.

My co-worker and I spent the better part of yesterday afternoon going through a former employee’s shell scripts to try to determine what they were and what he was trying to do. The script worked, for a while, but there were several mistakes. The mistakes were not in strict syntax, they were in style. Here are a few simple rules to follow to write great scripts:

1. Always, always, always start off each and every script with a shbang line: #!/bin/sh. Starting off your script with this line tells your shell where to find the interpreter for the commands in the script. Without this line, the script is using your user’s existing shell, the one you are typing in at the moment. This is bad because you are sharing environmental variables, and maybe changing environmental variables outside of your script, and not keeping it self contained and portable.

2. Keep your script self contained: If at all possible, try to avoid writing files in different directories. Or, even better, try to avoid writing files at all. Use variables when you can, write files when you have to.

3. Avoid sourcing other scripts or files containing functions: I read about this method in Wicked Cool Shell Scripts, but I disagree that it is as useful as they say. Writing a custom function to send an email is a great idea. separating it out of the script you are working on at the time is not. Again, keep the script self contained. There are obvious exceptions to this rule. If your function is over 50 lines of code, and reused in multiple other scripts, then by all means, source it. If your function is 10 lines, create a vi shortcut for it and add it to the top of the script.

4. Comment tasks: Each block of code in your shell script is meant for a specific task. Add a comment for this block. Make it easy to read, and simple to understand. Assume that you will not work there forever, and someone else will need to read your code and make sense out of it. Also assume that in a year, you will forget everything you did and why you did it and need a reminder.

5. Keep it simple: Scripts should flow logically from top to bottom. If you are creating functions, make it obvious using a comment. Reading a script should be as easy as reading a book, if it’s not, then you are intentionally making things overly complicated and difficult to read.

6. End each script with #EOF: This is purely a matter of taste, but I find it adds a nice closure to the script.

The easiest thing to do is to create another script who’s purpose in life is to create new scripts. Couple this script with a vi shortcut (mine is ,t) to create the skeleton of the script and you can quickly create powerful, well formatted, easy to read scripts. Here’s an example of mine:

#!/bin/sh
# 
# scripty.sh: This script creates other scripts
# Created: 25 Feb. 2009 -- inbound@jonathanbuys.com 
#
##############################################################

# A place for variables
VAR1="Set any variables at the top"
DATE=`date`
ANDTHEN="Whatever"

# A place for functions
some_func(){
    echo `date`
    echo "Whatever's Clever!"
}

# Get down to writing the script

echo $VAR1
echo $DATE
echo $ANDTHEN
some_func
# etc...

##############################################################
#EOF

This article doesn’t talk about syntax, only style. There’s plenty of help with syntax available on the Intertubes. Also, this is my style, as you progress as a sysadmin or scripter of some sort or another, you are bound to come up with your own style that suits you. My style is based on the documentation at grox.net. My style has evolved over time, as will yours, but this is a good place to start.

For better or worse, we are starting to put Ubuntu JeOS images into production in our network. Starting off, we will only put these systems in for our non-IBM services, no WebSphere or DB2, as IBM doesn’t officially support this configuration yet, but for everything else, JeOS looks like a perfect fit.

I’ve had this coffee cup on my desk at work for the past year or so now. It’s just a plain white cup, with the Ubuntu logo on it. I got it from CafePress. I loved it, for one, because the Ubuntu logo is great. Best Linux logo out there. I also loved it because as I was thinking about how to solve one problem or another, the cup was normally there with hot coffee waiting to be sipped as I pondered the solutions. Today I picked up the cup, walked towards the coffee pot, and dropped the cup. My wonderful Ubuntu coffee cup shattered as it hit the floor.

I loved that cup, so I didn’t want to break it. However, it seems appropriate, as today I also switched back to Windows at work. I’ve been running Ubuntu as my primary desktop at work for several months, and running XP in VirtualBox when needed. Lately, I’ve been needing the VM more and more, as I do more diagramming and planning in VMWare Infrastructure Client and Visio, both Microsoft centric applications. Also, rumor has it that in the next couple of months we will be replacing our aging Lotus Notes servers with Microsoft’s Exchange 2007. IBM released a Linux native Notes client which supports Ubuntu, and really works great. When we made the switch to Exchange, I was hoping to use the Evolution client that comes with Ubuntu. Unfortunately, Microsoft changed the MAPI standard for communicating with the server in Exchange 2007, and there is no supported Linux client. Which left me with two choices. Run Outlook in my VM, or moved everything back to Windows and conform to company standards. I debated this in my head for a couple of weeks, but in the past three days I’ve had X crash on me three times in Ubuntu. When X crashes, it takes all of my X applications with it, along with the data… it’s like Windows ‘95 all over again.

X crashing for no apparent reason was the nail in the coffin for me. I moved all my data over with a USB drive, and Monday I’ll format the Linux partition and fdisk /mbr from the XP recovery console.

I’ve really enjoyed using Linux, but honestly, it’s kind of relieving to be back in a supported environment again. There are still quite a few desktop tools missing from Ubuntu that are available on Macs and Windows. My current favorite so far is Evernote, with the aforementioned Visio running a close second. Launchy is nice… not as nice as Quicksilver or Gnome-Do, but nice.

Mentioning Gnome-Do brings up another point. Gnome-Do has been acting up lately, catching on something or other and eating up 99% CPU. The developers are aware of the problem, and are working on a solution. However, using Gnome-Do as an example, the very idea of what they are doing with “Release Early, Release Often”, completely goes against the grain of a business desktop. Any Linux desktop will contain beta-quality code, and when I’m relying on a computer to do my job, I can’t have it acting as a beta tester. Ubuntu is doing lots of cool stuff with 3D desktops and cutting edge software, but I don’t need it to be cool, I need it to work. Reliably.

One last note about why I’m not using Ubuntu at work any more. My computer is a Dell laptop, mostly used in a docking station, attached to a 22 inch monitor. I noticed after a while that my laptop was getting really hot in the docking station, and I couldn’t tell if Ubuntu was reading the docking station correctly or if it was displaying on both the internal monitor and the external monitor. When I popped the lid on the laptop, the monitor either came on suddenly or was on the entire time, and the keyboard was hot to the touch. In the Gnome “Screen Resolution” preferences I found that I could turn the monitor off from there, and I think that solved that issue, but I’m not sure. I’d hate to think that I was actually causing the hardware harm by running linux on it. I don’t want to spread FUD, but if its true, its true. When I’m running Windows, I don’t have that problem at all.

So, now I’m looking for a new coffee cup… something to inspire me, and be my companion in my little beige box. Whatever the new design is, it needs to be something that will last, something reliable, and something that’s in it for the long haul. Ubuntu has been good to me, both the OS, and the coffee cup, but in the end, they both broke, and I’ve got to move on.

It’s been nine years since I first installed Linux on a computer of mine. It didn’t last long back then, since I actually wanted to use the computer for surfing the web, sending email, and playing games. Linux has come a long way since then, and now it’s a reliable desktop system at work. However, my system is reliable (and enjoyable) because I am a geek, and I know exactly what it needs to make it run smooth.

I wrote this last year and never posted it. I’m glad I found it and can post it now.

Or, maybe a better title for this would be “They rebooted the server, why didn’t I get a page?” I’ve had that question asked of me a few times, and I’ve never had a good answer, so I thought I’d take a closer look at Nagios and see what is going on.

I mentioned in my last post that I was giving up my MacBook so my wife could go to college, and (until I can justify the cost of another Mac) that leaves me with her Linux PC. After the first few days of using Linux at home,as opposed to managing Linux servers at work, here are my initial impressions.

My wife’s PC is a gigantic Acer laptop, with a 1.6 Ghz centrino processor and a one gig of RAM. Hooked up to my external monitor, keyboard, and mouse, the PC runs Ubuntu surprisingly well. I can’t enable the desktop effects, but since Gnome has built in compositing support now that’s not a problem. The install went smooth, all the hardware was properly detected, wireless, sound, monitors… everything worked.

In OS X, there is a small collection of very high quality software. This is a fact, and there is really no debating it. Mac software that was developed specifically for the Mac is generally well crafted and performs as advertised, every time. In comparison, in Linux, there is a very large collection of freely available software of varying quality. Some of it is outstanding, check my last post on F-Spot for an example, and some of it leaves much to be desired.

My wife has recently gone back to college, and, seeing as she’ll need it much more than I do, I gave her my shiny, beloved MacBook. As compensation, I’ve got her old PC, running Ubuntu, which looks great on my 22 inch monitor. How long this will last, I don’t know. I’ve just recently started writing seriously again, and moving all of my “really important stuff” into Linux should give me lots to comment about as I note the differences between OS X and Ubuntu. However, I’ve heard rumor of MacBook Pro product refreshes… hummm….

As a general rule, I really don’t like to go outside of the box when it comes to Linux. And by that, I mean that I don’t like going outside of what is provided by what ever distribution you are using, be that SLES, Red Hat, or Ubuntu. A lot of people put a lot of work into making sure that the packages that are available for the distribution actually work in the distribution and do not interfere with any other apps. Linux will let you do what ever you want, but just because you can do something, doesn’t mean that you should.

Going outside the box can have disastrous results with Linux. Back in early 2000 and 2001 when I was installing SuSE and Mandrake on my old IBM box, I wound up in dependency hell more than once. If you’ve never been there, it goes something like this:

OK, I want to upgrade my music player to the latest version, so I’ll download the latest RPM. Wait, that failed, because it depends on a newer version of some library file that I don’t have, so I’ll go search the Internet and try to find that. OK, found it, downloaded the rpm, and it failed to install because it depends on a newer version of some other library file that I don’t have. Looks like there’s no RPM for that library, so I’ll download the source code and compile it. OK, ./configure; make; make install; Nope, that failed because of a gigantic list of dependencies that are not available! At this point, you have to make a decision: Do you go ahead and find the dependencies, or do you give up and have a drink instead. If you choose to go ahead, you download the source to a dozen different packages and install them, then compile your library, then compile your other library, then go to install the rpm to find that it fails because one of the applications you upgraded along the way is, get this, too new to support your music player, and the install still fails. Oh, and by the way, half of your other apps that used to work, don’t work anymore.

Where I spend my days ($WORK), we have multiple monitoring systems for just about every service on every server that we have. Many of these are Nagios, some are built in, and others are SiteScope. All of the systems generate email alerts that either go to our pagers, our email, or both. From time to time, management would ask a question like “How many pages do you get in a week on average”, which up till a couple of months ago, our answer was always “It just depends”.

I recently gave Linux my second, and final, serious shot at running it full time as my primary operating system on my MacBook. This time, it lasted all of three days before I dug out my Leopard install disk and began the long migration back to OS X. To preempt any questions on the subject, no I didn’t dual boot, and no, I didn’t have a good time machine backup. I was going to force myself to learn to do things the Linux way on my laptop.

Today I was out in the data center and decided to boot into Linux to get some work done on my Dell laptop. I was busy populating our internal wiki with hardware and OS data from our servers (how many dimms, what size, kernel level, etc…), which is a lot of work, lots of copy and paste, formating, grepping, going back and forth between the terminal and firefox. Lots of moving around, but not a lot of cpu or memory intensive tasks, just basic office tasks. I’m using Ubuntu, with Gnome and the desktop effects turned on, and I got so frustrated that I booted back into Windows. I hate to say it, but I was able to get more done in Windows today than I could in Linux.

A couple of months ago Linux Magazine published an article written by Scott Granneman comparing Mac OS X Leopard and Ubuntu Gutsy Gibbon. I’ve been torn between these two systems, and their respective predecessors for years, so the article was of great interest to me. As a matter of fact, I was wondering how I missed it to begin with. The article is well written, but can be boiled down to this one paragraph:

Linux has come a long way when it comes to ease of use, and it’s definitely getting better all the time, but overall Leopard is still ahead of Ubuntu (and both are way ahead of Vista). Apple makes mistakes, but overall its system is more logical, simple, consistent, and unified than Ubuntu, which still has too many elements that are overly complex, inconsistent, and fractured.