What a diff a tool makes

The CRM that my company uses is SugarCRM. We use the professional version as we wanted quoting, but there is an open source version available. We have also elected to have the on-site option, meaning we host it ourselves. There is an on-demand option that gets the task of running a server out of your hands and puts it in Sugar’s hands.

We have extensively modified our install of SugarCRM, which is written largely in php, with a little javascript thrown in. This means whenever Sugar updates their code, we start a laborious process of making sure our existing customization doesn’t get borked during the update.

The first few times this happened, we went through each file we had updated line by line. We had noted what files had been modified, but we still had review each updated file with a fine-tooth comb to make sure nothing broke. Invariably something broke. We have been able to fix everything, but not without at least a small impact on our sales staff.

I realized that some tool must have been written for developers and kernel updaters to compare files. My search quickly ended when I found the old *nix standby diff. I still have not mastered diff, but I know enough now so the update time has dropped drastically, and the errors that occur at an update have also been greatly cut.

Diff has many options and ways you can see the changes, from ignoring white space differences, to only showing what has changed, to side-by-side views. Here are a few that I have found extremely helpful.

First, this is how you use diff.

$ diff <options> <file1> <file2>

You can run diff without any options, but I don’t find it particularly useful.

Having an side-by-side option is great, though it has its limitations. I first started getting better with diff when, by trial and error, I figured out what worked for me. Sugar files have some very long lines, so I made my terminal as wide as the screen. I then ran the following command.

$ diff -y -W 120 <file1> <file2>

The “-y” flag puts each file side-by-side. The second flag (-W) tells diff not to use the default width and the number following the flag is the width you want. So in this case I set it for 120 characters. This shows you the entire file, however, the point of diff often is to find the difference, hence the name. We can take diff a step further to really narrow down our focus by adding the –suppress-common-lines flag.

$diff -y -W 120 –suppress-common-lines <file1> <file2>

Now you have all the lines diff has marked as different. In the column between the two files diff helps you find the differences. There are three common markers that are displayed here. They are >, < and | (that’s a pipe). If you see > in the middle column, the file on the right–which is <file2> has a line that the first file does not have. The arrow pointing like < means the opposite. If you see a pipe in the middle “|”, then each file has the line, but the line is different in each file. Here is a brief example of what that may look like:

if(!empty($_REQU | if(!empty($_
if (file_exists( <
$sugar_smart <
$sugar_smart <
} <
else { <
$sugar_smart | $sugar_smarty->a
} <
You can see most of the lines in this example are in the first file. There are two lines that have code, but something is different in them.

There are many other ways to use diff. Not all are side-by-side options. Many prefer just having the number of the lines that are changed. I, so far, prefer side-by-side. I find my tastes and ways of doing things are constantly changing as my experience level grows, so what may be easy for me today, is slow and cumbersome a month from now.

Have fun and keep exploring what hidden tools your Linux install has for you.

configure, make, make install

I am not sure if any one area of using a *nix system got to me more than ./configure, make, make install. I finally understand it, and yeah, pretty damn easy. For whatever reason I had a mental block on being able to remember those commands–and even if I did what order they went in. I have to credit the all-knowing The Linux Documentation Project (TLDP) for finally saving me on this one. An article written in 2003 on the subject made things click.

As luck would have it, I happened to need to install a new program, extract_chmLib. This program uses the chmLib library to extract html files from chm files. I had a few books in my personal collection that I wanted to have available online but they were in chm format. While I could boot into my XP install on VMWare and do the extraction, I’d really rather stay on the GNU/Linux side of things .

Not much info was out there on Google, but I did manage to find The Mad Philosopher’s blog that discussed his difficulties with getting extract_chmLib to work. He was able to install 0.35, but the most recent version is 0.39. I followed his steps, which immediately clicked with my reading of the TLDP article. I finally felt like I may be getting somewhere with ./configure, make, make install.

The philosopher’s site had an extra step, though, that got me stumped but made me feel quite victorious by the end of the evening. After “make install” was run, “make examples” was supposed to be run. The “make examples” command was supposed to add a few other programs to give chmLib more functionality. Now, I admit I was quite poor at the configure, make, make install sequence before tonight, but I had never recalled adding additional makes to the list.

At some point I decided to install 0.35 and see how that worked. Well, it didn’t. I got “make examples” to work just fine, but the program didn’t actually do anything. The man page is very straight-forward with no options to give, and no errors were showing up. The program acted like the chm was being converted to html files, but no such files appeared.

I decided a break was in order, so I got a Guinness and started to read about Apache configurations. I really know how to take a break, don’t I? I stop installing programs from source and relax with web server configurations. As I’m reading through a part on installing the mod_ssl library in Apache, I see a part about passing commands to ./configure. The light bulb immediately pops on and I jump back to my terminal.

Enabling examples in the configure step as opposed to adding another make step worked perfectly. So I test the program–which automatically was placed in /usr/local/bin as opposed to the “make examples” version which I had to symlink there–and all things are go. I had html files in seconds and scp’d them to my web directory.

I have made many of those types of connections when using Linux over the past two years. One minute something can feel completely foreign, and the next it is a basic part of your tool-belt that feels like it has been with you for years. Now I’ve got the basics of ./configure, make, make install down, and even added some intermediate usage, too.

Here is the usage to get extract_chmLib working (at least in Fedora Core 5). First download the program from the extract_chmLib link above. Next move the file to the directory of your choosing. Now just type:

$ tar xvf chmLib-0.39.tar.gz

$ cd chmLib-0.39/

$ ./configure –enable-examples

$ make

$ make install

That’s it. The extract_chmLib program is available in the /usr/local/bin directory. It works like this:

$ extract_chmLib <chmfile> <newdirectory>

Enjoy the program and keep making new GNU/Linux discoveries of your own.

Disappearing icons

So the hardware the sales reps use at my company had grown long in the tooth. We were using 1999 Compaqs set up as thin clients. The server held all programs, as well as acted as a file server–the thing was sluggish at best. We decided to go the route of program run locally and files on the server. Great, no problem.

I spec and price out some cheap but good computer parts and order them to build the sales reps boxes. We have a plan in mind to edit the /etc/exports file on the server and export each home directory to a unique computer. On the local computer the /etc/fstab file is edited to mount both the home directory and the company’s shared directory on the machine. This is pretty standard, straight-forward nfs stuff here.

On the local box we loaded up Fedora Core 3 and chose the required packages. Then once one hard drive was setup we used dd to make duplicates. That was much faster than reconfiguring them. Yeah, network installs would have been MUCH faster, but remember, I’m not Linux guru. A network install is how I’d do it today. A year ago with five months of sysadmin experience under my belt–and one year of Linux total–dd sounded like a great option.

I continue configuring home directories and config files and get a few computers set up. At some point I realize I set up the same person twice. Let’s call her Patty because, well, that’s her name. We were not allowing universal logins so I was going to delete Patty’s login and home directory from the local computer. At this point I had a few terminals open and thought I was being careful by using sudo for everything. OK, I was being careful by using sudo for everything, but you can still screw up royally by using sudo.

My mistake in this case was not paying attention to what terminal I was on. I had been editing the /etc/exports file on the server when I realized my mistake. Easy fix, I thought and proceeded to userdel Patty from that computer. Then my nearly fatal mistake came with an ‘rm -rf /home/patty’ command. For those that don’t know that handy command it means remove everything in this directory and do it without confirmation. Bye bye all three years of Patty’s data. I move on setting up the next rep on the local machine. I’m thinking everything is just fine.

Now, there is something you have to understand about Patty. She is not the most computer literate person (not the least either), but she is extremely colorful. She has developed an impressive knack for explaining what she sees going on in this Texas drawl. We are not talking in technical terms, here. We are talking visually. Her explanations are vivid and sprinkled with anachronistic sayings from Texas language that largely disappeared in the city environments. Even though born and raised in San Antonio, she speaks as though she is from an outlying cow town.

Patty walks into my office, head cocked to the side and a confused look on her face.

“Gabe, sweetie,” she says. “I think I did something to my computer.”

Already a slow but steady growing sense of dread encapsulates me. I know I was just working on her files. My mind starts racing through possible scenarios.

“You see,” she continues. “I was working and all of the sudden my icons start disappearing. They start popping away like popcorn kernels. Pop. Pop. Pop.”

At this point she waves her arms around, opening and closing each hand to symbolize the popcorn action of her icons.

“Your icons started disappearing?” I ask hesitantly.

Sean, the former IT manager and current GM starts laughing. He has not yet realized what is happening, but the thought of icons disappearing like some sort of lame Hollywood movie special effect (OK, not as bad as Disclosure, but maybe Hackers bad) has got to him.

“Yup, and now it is just blank and I can’t do nothing,” she says. “I even rebooted the darn thing and can’t do anything.”

“Alright, I’ll take a look in here,” I reply.

Sean and I immediately take a look at my command history. Sure enough, I was on the server and not the local machine when I typed the rm -rf command. Bad things. To make matters worse I had just changed the rsync schedule to run daily at 11 a.m. and 8 p.m. instead of just 8 p.m. The time was just before 11 when I ran the command, a little after 11 by the time my damage had been discovered. Three years of data gone from the primary server and backup server in a matter of minutes.

Luckily we have a dual-backup process in place. All data is also backed up to an external hard drive physically attached to the server each night. That had no change in backup times. I pull the data out of the external hard drive and rebuild her home directory and login. She lost all data from whenever she showed up that day to about 11 a.m. Way less than three years worth of files, contacts, quotes and other crucial information.

The fantastic thing is I can sit here a year later and crack up while typing out the story thinking about what it must have looked like to see icons disappear one-by-one off the desktop. Double-backups FTW.

The Accidental SysAdmin

I’m not kidding myself. I am no Linux guru. I get by. I get by way better than anyone else at my company–no offense all. It is a small business and that often means the person who knows more about computers is the defacto techie. Well, I’m much more than that, but in all likelihood less than whoever is running your network and servers.

I work at a small medical equipment and supply company in Texas, Aria Medical. We have 20-30 employees. I manage about 15 Linux boxes, a Linux server that handles e-mail (postfix with SpamAssassin, MailScanner, Pyzor and ClamAV), SugarCRM which uses Apache, PHP and MySQL, file serving duties, backup automation, disaster recovery, about 10 Windows boxes including one QuickBooks server and one Mac OS X. There is also a phone cabinet. I’d love Asterisk PBX to be in there, but I don’t have the time to handle phones, too. Our Web site is off site and I don’t want to take that on either.

I didn’t start out here trying to be the sysadmin. I was working at USAA (a financial services company) working my way up the ranks of their property and casualty insurance department. I got an opportunity to join Aria Medical–where my wife works as the webmaster and SEO/advertising whiz–as a techie lackey. My motivation was it was a casual work environment, I’d work with my wife, and I’d get to tinker with computers all day long, instead of just at night. Long story short, eight months later the IT manager becomes the GM leaving a hole. I was the most qualified at the company and there I was–barely able to use vim and running a system. Grepping was still a string of code I didn’t understand and made me think of Heinlein’s Stranger in a Strange Land and grokking. Sed? Awk? Not even blips on my radar.

I had the former IT guy around for a while, but then he moved on to Hawai’i. He set up the server, and at some point toward the end of his tenure with the company got his RHCE, so he knows what he is doing. I could lean on him, even though he really wanted me to figure stuff out on my own. At some point I had no choice.

I’ve made my share of mistakes. I’ve made some that could have been really bad if a decent backup system with multiple layers was not in place. No, nothing like trying rm -rf / because someone posted it on Digg or /.. That is what this blog is for–the tales of my trials, tribulations and triumphs as a newbie sysadmin.