By Bryant Bisnar and Rafael Orta
A VLAN is a group of ports designated by the switch as belonging to the same broadcast domain. (That is, all ports carrying traffic for a particular subnet address would normally belong to the same VLAN.)
When using a VLAN you can group users by logical function and this helps to control bandwidth usage by allowing you to group high bandwidth users on low-traffic segments and to organize users from different LAN segments according to their need for common resources.
VLANs are typically used to reduce broadcast traffic and to increase security.
A group of network users assigned to a VLAN forms a broadcast domain that is separate from other VLANs that may be configured on a switch.
On a given switch, packets are forwarded only between ports that belong to the same VLAN.
Thus, all ports carrying traffic for a particular subnet address should be configured to the same VLAN. Cross-domain broadcast traffic in the switch is eliminated and bandwidth is saved by not allowing packets to flood out all ports.
Here are some guidelines and caveats you need to be aware of when using VLANs for interconnect.
If deploying the interconnect on a VLAN, there should be a 1:1 mapping of VLAN to non-routable subnet and the VLAN should not span multiple VLANs (tagged) or multiple switches. Deployment concerns in this environment include Spanning Tree loops when the larger IP network topology changes, Asymmetric routing that may cause packet flooding, and lack of fine grained monitoring of the VLAN/port.
Caveats you need to be aware of when deploying the interconnect on a shared switch:
- The interconnect has to be on a non-routable dedicated subnet mapped to a single dedicated VLAN.
- If VLANs are 'trunked' between switches for redundancy, the interconnect VLAN can be in that VLAN trunk. However, this trunk should have enough capacity to guarantee adequate bandwidth for all trunked VLANs. Example, avoid a 1GbE link between the switches when you have 8 1GbE VLANs expected to go across that 1GbE inter-switch link.
- If the interconnect participates in a VLAN trunk between redundant switches, the interconnect VLAN should be 'pruned' so that the interconnect VLAN traffic does not propagate into the corporate network. Interconnect traffic should remain on the switches to which the links are attached.
- The network engineer needs to protect the interconnect VLAN from Spanning Tree re-convergence events. Also if the switch supports it, turn off STP for the interconnect VLAN. Or, mitigate the impact of STP re-convergence events by defining a root bridge the minimizes the number of network device in a loop. Any STP re-convergence event that affects the interconnect switch must complete within 60 seconds.
- The interconnect VLAN should not be negatively impacted by Quality of Service (QoS) definitions for any other network traffic on that shared switch. If QoS is defined at the port level (OSI layer 2) and modifies the network packet. This may introduce latency in interconnect packet processing. Any QoS advantages for the interconnect need to be well understood and tested.
- If you are configuring the interconnect in a dedicated VLAN on a shared switch, that switch should never experience congestion or 'over subscription' that might negatively impact the interconnect traffic.
- The network engineer should label and document the use of the VLAN and label the ports on the switch accordingly.
- The network engineer should monitor load on our VLAN to ensure that there is no packet loss, buffer overflow, inexplicable latency and asymmetrical traffic patterns.
Unlike other non-Oracle RAC clustered environments, our use of a cluster interconnect is non-traditional. We use the cluster interconnect, not only for network heartbeats (c.f. traditional active/standby clustering) but also for block sharing between a global cache. It is, for all intents and purposes, a latency sensitive i/o channel that requires high bandwidth, high availability and it should be secure (non-shared).
