Disappearing icons
So the hardware the sales reps use at my company had grown long in the tooth. We were using 1999 Compaqs set up as thin clients. The server held all programs, as well as acted as a file server–the thing was sluggish at best. We decided to go the route of program run locally and files on the server. Great, no problem.
I spec and price out some cheap but good computer parts and order them to build the sales reps boxes. We have a plan in mind to edit the /etc/exports file on the server and export each home directory to a unique computer. On the local computer the /etc/fstab file is edited to mount both the home directory and the company’s shared directory on the machine. This is pretty standard, straight-forward nfs stuff here.
On the local box we loaded up Fedora Core 3 and chose the required packages. Then once one hard drive was setup we used dd to make duplicates. That was much faster than reconfiguring them. Yeah, network installs would have been MUCH faster, but remember, I’m not Linux guru. A network install is how I’d do it today. A year ago with five months of sysadmin experience under my belt–and one year of Linux total–dd sounded like a great option.
I continue configuring home directories and config files and get a few computers set up. At some point I realize I set up the same person twice. Let’s call her Patty because, well, that’s her name. We were not allowing universal logins so I was going to delete Patty’s login and home directory from the local computer. At this point I had a few terminals open and thought I was being careful by using sudo for everything. OK, I was being careful by using sudo for everything, but you can still screw up royally by using sudo.
My mistake in this case was not paying attention to what terminal I was on. I had been editing the /etc/exports file on the server when I realized my mistake. Easy fix, I thought and proceeded to userdel Patty from that computer. Then my nearly fatal mistake came with an ‘rm -rf /home/patty’ command. For those that don’t know that handy command it means remove everything in this directory and do it without confirmation. Bye bye all three years of Patty’s data. I move on setting up the next rep on the local machine. I’m thinking everything is just fine.
Now, there is something you have to understand about Patty. She is not the most computer literate person (not the least either), but she is extremely colorful. She has developed an impressive knack for explaining what she sees going on in this Texas drawl. We are not talking in technical terms, here. We are talking visually. Her explanations are vivid and sprinkled with anachronistic sayings from Texas language that largely disappeared in the city environments. Even though born and raised in San Antonio, she speaks as though she is from an outlying cow town.
Patty walks into my office, head cocked to the side and a confused look on her face.
“Gabe, sweetie,” she says. “I think I did something to my computer.”
Already a slow but steady growing sense of dread encapsulates me. I know I was just working on her files. My mind starts racing through possible scenarios.
“You see,” she continues. “I was working and all of the sudden my icons start disappearing. They start popping away like popcorn kernels. Pop. Pop. Pop.”
At this point she waves her arms around, opening and closing each hand to symbolize the popcorn action of her icons.
“Your icons started disappearing?” I ask hesitantly.
Sean, the former IT manager and current GM starts laughing. He has not yet realized what is happening, but the thought of icons disappearing like some sort of lame Hollywood movie special effect (OK, not as bad as Disclosure, but maybe Hackers bad) has got to him.
“Yup, and now it is just blank and I can’t do nothing,” she says. “I even rebooted the darn thing and can’t do anything.”
“Alright, I’ll take a look in here,” I reply.
Sean and I immediately take a look at my command history. Sure enough, I was on the server and not the local machine when I typed the rm -rf command. Bad things. To make matters worse I had just changed the rsync schedule to run daily at 11 a.m. and 8 p.m. instead of just 8 p.m. The time was just before 11 when I ran the command, a little after 11 by the time my damage had been discovered. Three years of data gone from the primary server and backup server in a matter of minutes.
Luckily we have a dual-backup process in place. All data is also backed up to an external hard drive physically attached to the server each night. That had no change in backup times. I pull the data out of the external hard drive and rebuild her home directory and login. She lost all data from whenever she showed up that day to about 11 a.m. Way less than three years worth of files, contacts, quotes and other crucial information.
The fantastic thing is I can sit here a year later and crack up while typing out the story thinking about what it must have looked like to see icons disappear one-by-one off the desktop. Double-backups FTW.
April 10, 2007 at 8:54 pm
A classic tale indeed. I didn’t actually see the icons melting off of Patty’s screen but found it extremely funny to imagine. You’d think the terminal would just lock up or lose it’s connection to the server, but no, Patty got goofy hacker movie special effects, literally showing her files disappearing from the screen.
The reason we originally went to Linux thin-client systems and later NFS mounted home directories is because a sales rep once had his computer stolen. Nearly a year’s worth of his data, price quotes, etc. was lost. Losing data is so painful, hence the dual backup system was put in place.