Port Forwarding FTL

NoodlesOK, so we had a bit of a situation. We have a full rack of co-located production equipment, running a critical web application which can’t go down. The trouble is, the router and firewall belong to someone else and it’s getting turned off at the end of the week.

Not a problem, you cry, just get your own connectivity and firewall/router and hook em up. That’s fine. Until you get to the point where you have to change DNS to put it live.

So here’s the layout. Old production firewall is box A. Whole production system is a big virtual box B (with loads of machines in it). New production firewall, on a different external address, is box C.

Currently, traffic comes into box A, port forwards to box B, which has box A as its default route. All fine and dandy.  Trouble is, we want to change DNS so that traffic is routed to box C, which port forwards to box B, which….. sends its response out through its default route on box A, never to be seen again.

“You thick muppet,” you cry, “why not just change the default route on box B?” Well, remember that bit about no downtime? Just as connections to box C don’t work with the default route set as A, connections to A won’t work with the default route set to C – making things a little tricky during the DNS broadcast when the system effectively needs to answer to two addresses at once.

There is, naturally, a solution – a filthy FILTHY hack which makes the firewall look like the plate of noodles above. The core problem is this – traffic sent to an internal machine in box B leaves by the wrong route. Traffic on box C, however, leaves by the correct route. So, the plan is to make the traffic on box B “happen” on box C instead.

What?

Ok, here’s how it’s done. SSH TUNNELS! LOTS OF THEM! ONE FOR EACH SERVICE!

Look, if I forward port 80 to the internal machine, the traffic gets lost, right?

But the internal network is fine because it’s a flat class C – it’s only connections which require the default route which have issues. So, we do the port forward in two stages. First of all, we set up an SSH tunnel so that port 80 on the web server is forwarded to, say, port 12345 on the firewall’s internal interface. Then we follow up by forwarding port 80 on the firewall to port 12345 on the internal interface of the firewall, which transparently tunnels it to port 80 on the internal web server. Hooray!

Of course, this is a hack on a kludge of almost biblical proportions, but it only needs to be in place until the DNS has propagated, at which point we can change the default routes to point to box C, drop the tunnels and do the port forwarding properly.

So, as such, it’s not REALLY a filthy hack – it’s a piece of creative networking designed to avoid downtime while re-homing an entire system.

I want a job in academia so I can impose periods of arbitrary downtime and save having to turn my networking configs into chop suey.

Related posts

Creating a Bramble

So, my current Raspberry Pi cluster isn't really a true cluster - it's really...

OUTAAAAGE!

So, nelefa.org fell off the internet last night, for the first time since 1997....

A conversation with JD

j.daniels: 15 minute job to move disks and filesystems? That's impressive. What's your...

Latest posts

Oh my.

So, it looks like I'm averaging a post a year at the moment. That's...

State of the cluster

Here we see, in all its glory, the little mini-cluster of Computery Goodness which...

Creating a Bramble

So, my current Raspberry Pi cluster isn't really a true cluster - it's really...

Abe Yospe’s Wife.

There is a scurrilous rumour circulating that Abe Yospe's wife googles things really quickly...

Free Music

Here ye go. Three collections of slightly odd Berlin School synth music for your...

Leave a Comment

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.