Hitlessly change a Cisco switchport from an Access to a Trunk Port in Production

Monday, 27 Apr 2020

So you've got an Access Port on a Cisco Switch going down to a Host/Server of some sort, and for whatever reason you then swap out that Server for maybe a Virtualisation Host (ESXi, oVirt, Docker, k3s....), or in my case a Cisco WAVE594 with a built-in Tyan AST2050 BMC (where the iLO shares the same physical NIC as the Server, no seperate Mgmt Port) - and you want to do it hitlessly. Hmm, sounds like a good challenge to me!

Draw it up then!

What we have is:

  • Left-hand side (before)
    • Baremetal Server SERVER Eth0 connected to Cisco Switch SWITCH Gig 1/0/9
    • Eth0 set as 10.0.0.9/24 without VLAN tagging/membership
    • Cisco Switch SWITCH Gig 1/0/9 set as Member (Access) of VLAN 99
    • VLAN 99 associated with SVI99 10.0.0.1/24 for inter-VLAN Routing
  • Right-hand side (after)
    • Baremetal Server SERVER Eth0 connected to Cisco Switch SWITCH Gig 1/0/9
    • Eth0 associated with Software vSwitch (or similar)
    • Internal Mgmt Loopback/Interface associated with Physical NIC Eth0, set as 10.0.0.9/24 without VLAN tagging/membership
    • Virtual Machine VIRTUALMACHINE vNIC0 associated ultimately with SERVER Eth0 pNIC, but set as 10.2.0.28/24 and to tag VLAN 86
    • Cisco Switch SWITCH Gig 1/0/9 set as Trunk and Native (Access) for VLAN 99
    • VLAN 99 associated with SVI99 10.0.0.1/24 for inter-VLAN Routing
    • VLAN 86 associated with SVI86 10.2.0.1/24 for inter-VLAN Routing

Topology showing Cisco Switches and Access-connected Hosts

Tag vs Untag / Cisco vs Others

First, some terminology that always caught me out as a Cisco Guy - the difference between Tag and Untag, and Access and Trunk. In my simple Cisco Head, the world is just two things:

  • Access Ports
    • Member of one VLAN
    • 1:1 Port to VLAN mapping
  • Trunk Ports
    • Member of as many VLANs as you like
    • 1:many Port to VLAN mapping

I'd then learnt about a "Native" VLAN, but it was mostly a superficial reference to something that didn't make sense, so I moved on with my life. Then I encountered the following that change my views, and made it all "click" for me:

  • Tag Ports
  • Untag Ports
  • Voice VLANs

Voice VLANs never sat right with me, as they break the Simple Cisco Paradigm of "A (Access) or B (Trunk); there is no C (Other)" - as Cisco sell a Voice VLAN as if it's an Access Port:

interface Gig1/0/2
 description "Connection to an Overpriced Cisco Phone"
 switchport mode access
 switchport access vlan 100
 switchport voice vlan 101
end

Yet I know that's two VLANs down one port, so it's a friggin' 802.1q Trunk dammit - nothing access'y about that at all, but don't let that stop you using the "access" keyword! My mental model, you done broke my mental model argh!

Access is Trunk, Trunk is Voice, Voice is Access?

After a few coffees whiskeys though, it clicks - Cisco are once again, lying; the true answer is there are three states a port can be in:

  • Untagged for one VLAN (or "Access" as Cisco call it)

Except when it's voice, because who loves consistency? Not Cisco BU's clearly - "Oh hi, welcome to Model1! Yes, it's 'show mac-address-table' here... Oh hi, welcome to Model2! Yes, it 'show mac address-table' here. Mess up your scripting did that? LOLZ, sucks to be you, fool!"

  • Tagged for many VLANs (or "Trunk" as Cisco call it)
  • Tagged for many VLANs and Untagged for one special-little-snowflake VLAN (or "Trunk" with a "Native VLAN")

The third is what we're going to use, where one VLAN (of your choosing) doesn't get a VLAN tag appended to it when thrown onto the wire; meaning the reverse is true. What the above hints at is a Switch-centric view of the world, but if we take that from a Server-centric view of the world:

Port Type Switch-centric View
(Port Egress)
Server-centric View
(Port Ingress)
Action
Access No VLAN tag present No VLAN tag present Switch appends VLAN on ingress/Server no clue it's in a VLAN
Trunk VLAN tag present VLAN tag present Switch sees VLAN on ingress/Server knows it's in a VLAN

...which is why the "Tagged" and "Untagged" definitions make infinitely more sense than normal Cisco nomenclature "Access" versus "Trunk", as the Cisco speak hides the fact that a port can have three states (Access; Trunk with no Native & Trunk with Native), whereas a given logical Server port's membership is one of two states (Explicit Member - Tagged, Server knows the VLAN exists or Silent Member - Untagged, Server has no clue it is within a VLAN [Native]).

What's a BU? A Business Unit, obviously dummy - everyone in Cisco knows that, so we don't bother explaining that acronym, duh. What, you mean you didn't know that? Best hand that CCNP back, boyzone...

The solution

So back to the problem at hand; I want the Server's Mgmt IP to remain at 10.0.0.9/24, but not be Tagged (not know it's within a VLAN), and I want to hitlessly (or as near as possible) change the configuration on the fly from Access to Trunk. Here's my way forward:

  • Current configuration
    • interface GigabitEthernet1/0/9
      description "To SERVER Eth0"
      switchport mode access
      switchport access vlan 99
      no shutdown
      end
  • Future configuration
    • interface GigabitEthernet1/0/9
      description "To SERVER Eth0"
      switchport mode trunk
      switchport trunk encapsulation dot1q
      switchport trunk native vlan 99
      no shutdown
      end

Which effectively tells the ingress-behaviour (Server-to-Switch) of the Trunk to bang all untagged Server traffic (in our case, that's to or from Mgmt virtual/logical Interface, 10.0.0.9/24) into VLAN99 - without the Server ever knowing it is a member of VLAN99, but while allowing the Trunk to pass explicitly-tagged Virtual Machine, or other, traffic (such as VLAN86).

This is incredibly useful when working with inline iDRAC/BMC configurations, as you can do this without burning another cabling run/Server port (or in my case with the Cisco WAVE594, because it doesn't have any other option/no in-built Dedicated iDRAC Mgmt port):

  • Operating System/ISO Eth0 Port
    • Untag (hits the "Native" VLAN)
  • iDRAC/BMC Eth0 Port
    • Tag (hits the explicit VLAN)

Or the other way around - whatever works for you. Which is exactly the same that Cisco ended up bodging their "Voice VLAN" configuration to do, only it's a Trunk not an Access... let's not go there again.

Get bodging!

SSH Proxy Local Port Chaining to connect to an Isolated Server

Saturday, 25 Apr 2020

There you are with Internet access to an AWS EC2 instance, Azure Virtual Machine or VPS (Virtual Private Server) as we called them in the good old days, but beyond that - on a Private LAN of some sort (could be an AWS VPC, Azure VNET or Physical LAN) - there is another Server, which only has a Private IP Address. This pesky Server is what you'd like to access SQL Developer on, from your own PC. Hmm...

Diagram time

Here's a more descriptive Network Diagram, showing more specifics around the issue, namely:

  • Your PC/Mac/Laptop
    • Private IP doesn't matter
    • Public IP is 99.99.99.1
  • Internet-accessible VM
    • Public IP is 1.1.1.1
    • This is allowed through an Internal Firewall to SSH to 172.30.12.99 (sourced from it's LAN Private IP of 172.30.12.6)
    • This is not allowed to connect on SQL Database (TCP/3306) to your 172.30.12.99 Database Server
  • Database VM
    • No Public IP (not directly accessible from the Internet)
    • Private IP is 172.30.12.99
    • Has SSH Daemon/Server and SQL Server installed on it

SSH Proxy Local Port Chaining Topology

The ultimate goal is to enable SQL Developer on your PC to somehow speak to SQL Server (TCP/3306) on 172.30.12.99.

SSH Tunnels - Local Port Proxy

Which is where a handy feature of SSH comes in, where it has the ability to Tunnel.

Dynamic Port Tunneling (SOCKS Proxy)

You may have used this prior with Dynamic Port Tunneling, where you do something like this:

ssh -D 1080 user@1.1.1.1

Then configure 127.0.0.1:1080 as a SOCKS4 or SOCKS5 Proxy in your Firefox Browser like this:

Firefox SSH SOCKS Proxy Options Screen

And then you can browse some http://internalserver in your Firefox Browser as if you're controlling/appear to be the Internet-connected VM to upstream Servers/Systems. That won't help you here, mind.

Local Port Tunneling

Instead, we're going to use Local Port tunneling, which looks a bit like this:

ssh user@1.1.1.1 -L 999:127.0.0.1:3306

It's worth breaking down what this does, with some notes:

  • 999
    • This is the port that will be listening on your PC (i.e. 127.0.0.1 or localhost, from your perspective)
    • Any traffic sent to this port is chucked through the Tunnel, for the other side (SSH Server on 1.1.1.1) to deal with
  • 127.0.0.1
    • This is where you want the other side (1.1.1.1) to send your Tunneled traffic to (Destination IP Address)
    • Note in this case, it is itself; the 127.0.0.1 here refers to 1.1.1.1 going to itself, not your PC
    • Think of stuff after the first -L colon as "remote-side stuff"
  • 3306
    • This is where you want the other side (1.1.1.1) to send your Tunneled traffic to (Destination TCP Port)
    • This is the TCP-using process (in our case, it'll be another SSH not SQL, as you might think) on the other side (1.1.1.1)`

In our case, this wouldn't do much - as there is no Database Server (TCP/3306) running on our first Jump Box (1.1.1.1) - so the next bit is where the chaining comes in.

Local Port Chaining

Leaving this SSH session (above) open, we open a fresh Terminal/Command Prompt/SSH session from our PC to 1.1.1.1, with no Tunnels requested (bog standard SSH):

ssh user@1.1.1.1

Using this, we now tell 1.1.1.1 to SSH to our target Database (172.30.12.99), but this time we want the traffic to pop out at the actual Database Port (3306):

ssh user@172.30.12.99 -L 3306:127.0.0.1:3306

Finally, we can now connect to 127.0.0.1:999 on the PC, and it will traverse the two SSH Tunnels and pop out as being 172.30.12.99:3306 - even though that destination doesn't have Internet access.

Note in the above that the choice of using -L 3306 on the 1.1.1.1 PC is irrelevant; this could have been 998; but then the SSH command run against 172.30.12.99 would have had to read -L 998 instead of -L 3306. All you are doing here is setting up listeners, and matching that number up between the two SSH Commands.

What it achieves

SSH Proxy Local Port Chaining Resulting Topology

Which gets you out of a hole, and able to access a "Bastion" Server on a Private LAN or VNET somewhere, as long as you've got SSH access throughout the chain.

Have fun tinkering!