Ok, so we have a Linux box sitting at our colo doing, amongst other things, some ASP stuff. Thing is, this server is a little broken. Someone, at some point in time prior to me working here, decided in their infinite knowledge and wisdom to ‘transfer’ one machine to another by rsyncing random bits of the filesystem over.
Naturally, as /etc and /var were included in this madness, it’s a very broken machine with an even more broken package management system.
So there I was today. I decided that, as I was implementing a new backup procedure on the machine after moving it out to the datacenter, that it would be helpful to fix the horribly broken mail system so I can actually get the results of the backup job mailed to me. No problemo – ripped out the horribly broken exim system and replaced it with a simple and elegant localhost-only sendmail set up. Hooray! Mail is fixed.
Except what’s this! I’m receiving loads of mails of failed cronjobs. Quickly fire up crontab to have a look. Nope, nothing in the crontab – must be the system crontab. Had a look in there. Removed a couple of synced-over crontabs for applications which weren’t even installed, but still no sign of the one generating the verbose errors.
To cut a long story short – crontab was cheerfully showing me the cron files stored where it expected them. Crond was happily running the cronjobs stored where it expected them. Sadly, the crontab application was not only from a different version of the cron package from the actual daemon, but it was from a completely seperate distro of linux which expected the cron files to be kept in a different place.