Oh God, Please Kill Me

Dear $workplace Friends,

We have had, for some considerable time, an unspoken policy that a machine is ‘deployed’ on the network when it has been plugged in. This has resulted in there being many servers, vms, and other random bits and pieces which have been plugged in, switched on, and denied the nicities of things like DHCP entries, DNS entries and so on.

As a result, bits of the network keep falling off. The DHCP work of the past few days goes some way towards rectifying this but there will be the occasional service which may be noticed as having vanished. This doesn’t mean the network is broken – it just means the bit of the network you’re on is fixed.

If you happen to notice something isn’t where it’s expected to be, please let me know and I’ll look at it. Please don’t attempt to fix it yourself as this will result in a violent slapping I have a very specific set of configuration guidelines in place. We are gradually managing to wrestle things into some semblance of order.

Thanks

John

“The 55 machine is dead!”

“The 229 machine is dead!”

These words mean nothing to me. If a machine is given a hard-coded IP address in the middle of a DHCP zone, is given no DNS entry, and the Sysadmin doesn’t know about it, OF COURSE IT WILL FUCKING BREAK!

I’ve just been asked to “ping” someone when I have a moment. I’m considering getting a stainless steel rj45 socket to hammer into his fucking skull. That way I can get him on the network and assign him an ip address. Following that, his request *might* actually make some kind of logical sense. Fucking weenies.

The point is, the point is, the point is, the point is, why do we always do this thing that we’ve always done instead of this thing that I’ve only just thought of? It’s imperative that we never do the thing we’ve always done ever again and I’m frankly amazed that we’ve even done it at all!

Oh dear god. So, it turns out that our flagship web application is invisible to Google. So, one of our sales guys (significantly more clued than the usual sales type) sticks /robots.txt into a browser to work out why.

FAIL ONE

robots.txt contains:

User-agent: *
Disallow: /

FAIL TWO

“Can you change this?” he asks. Well, of course I can – I assume there’s some reason they’ve done this – perhaps bits of the application aren’t supposed to be indexed. Anyway, I open robots.txt on the filesystem and discover, to my utter confusion that it contains this:

#!/bin/sh
echo Content-Type: text/html
echo
cat robots.t

To my even more increasing horror and confusion, robots.t contains:

User-agent: *
Disallow: /

Obviously, the engineers who did this part of the application were of the opinion that simply putting instructions in a text file was too simple and therefore not professional. Clearly, the addition of a second layer of pointless complexity was vital.

One of the pages of our flagship web application is aimed specifically at prospective candidates who have passed the interview stage and been offered a position.

The page asks them to either upload a scan of their passport or bring the document on their first day.

Which makes it quite clear why one such candidate has uploaded a word document containing the sentence “I don’t have a scanner.”

“I need accurate timings for the platform migration,” cried Orange Stupid.

“Certainly,” I said, “We’re realistically looking at 4 man-days for the dev database, 4 for the test database, a further four to hook them up to the front end interfaces. We’ll then need somewhere in the region of 8 man days for the search database as we’re not entirely sure how difficult that will be to integrate. At the end, I’d image we’ll need somewhere in the area of ten days testing before decommissioning the old one.”

“I see I see I see,” bumbled Orange Stupid. “So it’s imperative that it’s finished by Wednesday at the latest.”

We have a server built out of an HP ML110 with three network interfaces. It acts as a VPN server, gateway between two internal networks, provides the default gateway for one of the internal networks, and acts as a fileshare for one of the internal networks.

This server has a problem.

If you copy a large amount of data between the two networks – i.e. between the two internal interfaces – it occasionally kernel panics the machine. This would suggest to me that one of the internal network cards is hosed.

Here is a discussion which took place between myself and Orange Stupid. Note, this server is running Fedora Core 5 and always has been.

Me: I’ve added one of the machines on network B to the network A backup system. This will involve copying a large amount of data through the gateway on a daily basis. There’s a problem with one of the network cards, but the first time it dies, we’ll be able to see from the panic screen which card it is and replace it.

OS: Why don’t we swap over one of the internal cards with the external card and see if it still fails.

Me: Because the problem manifests when copying between the two internal interfaces. When it fails it’ll tell us on the actual display which card is dead.

OS: But can’t we swap over the two internal cards and see if it still fails?

Me: But if we just let the backup run, it’ll tell us which card failed.

OS: But we can move the cards to different PCIX slots and see if it still fails.

Me: Or we could just let the backup run and that will identify the specific card, and therefore the slot, allowing further investigation to take place without a lot of trial and error.

OS: What we could do is leave the physical hardware alone but switch the interfaces in software and see if it still happens.

Me: Or we could just let the backup run and it will actually tell us which card is broken.

OS: Why can’t we swap the cards over?

Me: Why can’t we just let it run and let it tell us which is faulty?

OS: I’ll never trust Ubuntu again.

Me: ……

Honestly. There’s two ways I can describe today.

Firstly, imagine I am separated from the rest of the office by a giant fishtank that fills my whole field of view. Now imagine that the fishtank is full of huge, twitching, orange octopi which are juggling hand grenades. Next, imagine there’s a slow-motion car crash involving a big purple car and a wailing ambulance happening on the other side of the fishtank. Now, imagine I’m trying to work on a large amount of really important, really complicated things while trying to see if I’m going to be blown up by the grenade juggling octopi and morbidly drawn to the slow motion car crash.

That’s my day.

Alternately, it can be expressed in perl as follows:

require bleeps;

require stupid::purple;

require stupid::orange;

sub fail() {

SUMMON_RAPTORS();

}

while(1) fail;

That is also my day.