Иван Лысогор Senior System Engineer · 2015-07-30 · METAFABRICARCHITECTURE Иван...

METAFABRIC ARCHITECTURE

Иван Лысогор

Senior System Engineer

INTRODUCING THE METAFABRIC ARCHITECTURE

2 Copyright © 2013 Juniper Networks, Inc.

VM

VM

VM

VirtualPhysical

VM

VM

VM

VirtualPhysical

VM

VM

VM

VM

VM

VM

Virtual Virtual

My on-premises

data centerMy hosted

service provider

My managed

service provider

My cloud

service provider

VM

VM

VM

VirtualPhysical

VM

VM

VM

VirtualPhysical

SIMPLE. OPEN. SMART.

METAFABRIC ARCHITECTURE PILLARS

Simple SmartOpen


Easy to

deploy & use

Save time,

improve

performance

Maximize

flexibility

METAFABRIC ARCHITECTURE PORTFOLIO

Flexible building blocks; simple switching fabricsSwitching

Universal data center gatewaysRouting

Smart automation and orchestration toolsManagement


Simple and flexible SDN capabilitiesSDN

Adaptive security to counter data center threatsData Center Security

Reference architectures and professional servicesSolutions & Services

METAFABRIC REFERENCE ARCHITECTURE

Validated and tested designs

Reduce risk – accelerate


Version 1.0 – virtualized (VMware)

Enterprise data center with key

partners (IBM, EMC, F5)

Reduce risk – accelerate

customer adoption

Virtual Chassis Fabric

Up to 20 members

QFX5100 DEPLOYMENT OPTIONS

Virtual Chassis

Up to 10 members

QFabric

Managed as a Single Switch

4 Up to 128 members


Spine-Leaf

4

Layer 3 Fabric

L3 Fabric

QFX5100

QFX5100 PLATFORMQ4 2013 Q1 2014

� L2/L3 line rate forwarding

� 10GbE/40GbE and FCoE

� Feature-rich Junos, full L2/L3

48 X 1/10GbE 6 x 40GbE 24 X 40GbE Slot 1 Slot 2

96 X 1/10GbE 8x40GbE


� 1.5GHz Dual Core Intel Sandy Bridge X86 CPU

� 8GB Memory, 2x16GB SSD

� Innovated Junos software architecture

� Redundant, hot-swappable AC or DC power supply

� Redundant, hot-swappable fan tray

� AFI (FRU to port side) or AFO (Port to FRU side) airflow

� Beacon LED, no LCD panel

� Feature-rich Junos, full L2/L3

protocol, MPLS4 x 40GbE QSFP module

ADVANCED JUNOS SOFTWARE ARCHITECTURE

Provides the foundation for advanced functions

• ISSU (In-Service Software Upgrade)

• Other Juniper applications for additional service in a single switch

• Third-party application

• Can bring up the system much faster


Linux Kernel (Centos)Host NW Bridge KVM

JunOS

VM

(Active)

JunOS

VM

(Standby)

3rd Party

ApplicationJuniper Apps

ISSU (IN-SERVICE-SOFTWARE-UPGRADE)

• Master Junos VM controls the

hardware–PFE and FRU on the

system

• Master issues upgrade command

• System launches a new Junos VM

with new image as backup

• All states are synchronized to the

PFE

Contro

l

Master/

Backup

Election

Other

JUNOS

process

MASTER VM

PFE

Contro

l

Other

JUNOS

process

Master/

Backup

Election

Backup VM


• All states are synchronized to the

new backup Junos

• Detach PFE from current master,

then attach to backup Junos (hot

move)

• The PFE control component in new

master will control the forwarding

• Stop the new backup VM

l l

HOST OS

OTHER HARDWAREPFE hardware

Partition

warm boot

Partition

for PFE

warm boot

Software Bridge

INSIGHT TECHNOLOGY

Hotspot & microburst impacts application

performance

� Not visible with traditional counters

� Network operation is blind folded

Captures microburst events which exceed

defined thresholds

Adjustable sampling intervals

Qu

eu

e D

ep

th o

r Q

ue

ue

La

ten

cy

High Threshold

Microburst


Adjustable sampling intervals

Reports the microburst events

instantaneously via

� CLI

� Syslog

� Log file (human readable format)

� Streaming (Java Script Object Notification, CSV, TSV

formats)

Time

Qu

eu

e D

ep

th o

r Q

ue

ue

La

ten

cy

Buffer Utilization Monitoring

And Reporting

Low Threshold

UNIFIED FORWARDING TABLE

• Flexibly allocate L2 MAC, L3 host and LPM (Longest

Prefix Match) resources from a single pool• L3 host holds /32 IPv4 or /128 IPv6 routes

• LPM table holds any routes not handled by L3 host table

• Optimized forwarding table size based on deployment

scenarios

• Use system resource efficientlyUFT (Unified Forwarding Table)


• Use system resource efficientlyUFT (Unified Forwarding Table)

L2 MAC + L3 Host + LPM

UFT (Unified Forwarding Table)

L2 MAC + L3 Host + LPML2 MAC LPML3 Host


L2 MAC + L3 Host + LPML2 MAC LPML3 Host

UNIFIED FORWARDING TABLE


L2 MAC + L3 Host + LPM288K (L2 MAC)

16K

(LPM)

16K

(L3

Host)



16K

(LPM)80K (L3 Host)

Profile 1: l2-heavy-one

Profile 3: l2-heavy-three (Default)

Profile 2: l2-heavy-two




16K

(LPM)144K (L3 Host)



16K

(LPM)208K (L3 Host)


L2 MAC + L3 Host + LPM

32K (L2

MAC)128K (LPM)

16K

(L3

Host)

Profile 3: l2-heavy-three (Default)

Profile 4: l3-heavy

Profile 5: LPM-heavy*

*under test, may come after FRS

� Zero-touch provisioning

� Ops/event scripts

� Python

� Network Director API

Network Automation

AUTOMATION*


Simple Network

Architecture

� VMware

� Puppet, Chef

� OpenStack

� CloudStack

Data Center

Automation

*Not all features will be available at FRS

JUNOS ENHANCED AUTOMATION IMAGE

� Junos Enhanced Automation image provides increased flexibility to our

large Data Center customers

� VeriExec disabled on Junos Flex enables customers to run unsigned

binaries on QFX 5100

� Ability to run Python/Ruby with custom Libraries like

Collectd/Ganglia/Monit/etc


� Puppet and Chef packaged with Junos Flex to help MSDCs automate

configuration


VIRTUAL CHASSIS FABRIC

VCF ESSENTIALS

Active Backup

LogicalPhysical


1 RU, 48 SFP+ & 1 QIC

Node #1 Node #16Node #3 Node #2 Node #4

� Single device to manage

� Accessible from any member of fabric

� In band Virtual Backplane to enable Junos LC-RE communications

� Multi-path forwarding

VCF BUILDING BLOCKS

QFX5100-24Q(40GE)QFX5100-48S(10GE)

VCF 10/40GE spine nodes

VCF 1/10/40GE leaf nodes


EX4300 (1GE)

QFX5100-48S(10GE)

QFX3500(10GE) QFX3600(40GE)

QFX5100-24Q(40GE)

VCF BUILDING BLOCKS - COMPATIBILITY MATRIX

Platform VCF spine node VCF leaf node

QFX5100-24Q ✓✓✓✓ ✓✓✓✓

QFX5100-48S ✓✓✓✓ ✓✓✓✓

QFX5100-96S ✓✓✓✓ ✓✓✓✓


Scales to 20 members

QFX5100-96S

QFX3500 ✗✗✗✗ ✓✓✓✓

QFX3600 ✗✗✗✗ ✓✓✓✓

EX4300 ✗✗✗✗ ✓✓✓✓

VCF SCALE

All QFX5100 Mixed

Spine QFX5100-24Q QFX5100-24Q QFX5100-48S

Leaf QFX5100-48S

QFX5100-24Q

QFX5100-96S

QFX5100-48S

QFX5100-24Q

QFX5100-96S

QFX3500 & QFX3600

EX4300

EX4300

Scale QFX5100 Lowest Common Scale


Scale QFX5100 Lowest Common Scale

root@opus# set chassis forwarding-options ?

Possible completions:

l2-profile-one MAC: 288K L3-host: 16K LPM: 16K

l2-profile-three MAC: 160K L3-host: 88K LPM: 16K

l2-profile-two MAC: 224K L3-host: 56K LPM: 16K

l3-profile MAC: 96K L3-host: 120K LPM: 16K

lpm-profile MAC: 32K L3-host: 16K LPM: 128K

L2 MAC 128K

L3 Host 8k

L3LPM 16K

L3 Multicast4K

IPv6 scale= IPv4 LPM/4

QFX3500/3600 Scale

L2 MAC 64K

L3 Host 32k

L3LPM 16K

L3 Multicast16K

EX4300 Scale

DEPLOYMENT FLEXIBILITY

10G 1/10/40G 1G

10G40G

QFX5100-24Q QFX5100-24Q QFX5100-48S

1GE, 10GE & 40GE all in one fabric


10/40G spine nodes & 1/10/40G leaf nodes

10G POD 1/10/40G POD 1G POD

Spine Node QFX5100-24Q QFX5100-24Q QFX5100-48S

Leaf Node QFX5100-48S

QFX5100-24Q

QFX5100-96S

QFX3500 & QFX3600

QFX5100-48S

QFX5100-24Q

QFX5100-96S

QFX3500 & QFX3600

EX4300

EX4300

OPERATIONAL SIMPLICITY - PLUG ‘N’ PLAY

member 1 {

role routing-engine;

serial-number SER1ALNUM1;

}

member 2 {


serial-number SER1ALNUM2;

}

member 3 {


serial-number SERIALNUM3;

}


member 4 {


serial-number SERIALNUM4;

}

1 RU, 48 SFP+ & 1 QIC

Non-

Factory

Default or

3rd Party Spine nodes & leaf nodes are auto provisioned

Factory-default device will join the fabric

Non factory-default device will not join the

fabric

Configuration and image synchronization

HA - RESILIENT CONTROL & DATA PLANE

Active Hot- Backup Backup Control Plane Redundancy

Quaternary RE (routing engine) redundancy

Resilient In-Band Control plane

GRES ,NSR, NSBuplink redundancy

Redundant Routing engines

Backup


GRES ,NSR, NSB

1 RU, 48 SFP+ & 1 QIC

Data Plane Redundancy

OVM VM VM

vSwitch

Virtual Server

OVM VM VM

vSwitch

Virtual Server

Server multi-homing

Active-active uplink forwarding

server multi-homing

uplink redundancy

FORWARDING PLANE (SMART TRUNKS)

SW 1 SW 2 SW 4SW 3

L1 L2 L3 L4 L16


Automatic fabric trunks

• Fabric links automatically aggregated into trunks (LAGs)

Fabric trunk types

• Next Hop (NH)-trunks: from local to direct neighbors

• Remote Destination (RD)-trunks: from local to a remote destination PFE

Weights based path (instead of NH link) bandwidth ratio to avoid fabric congestion

1 RU, 48 SFP+ & 1 QIC

SW 5 SW 16

HA - HITLESS UPGRADE WITH ISSU

Upgrade one rack/node at a time

Applications run on half bandwidth

Long maintenance window

Upgrade multiple racks at a time

Application run on full bandwidth

Shorter maintenance window

Does not require hardware

Hitless upgrade using single switch


Today

Does not require hardware

redundancy

VCF

Services GWWAN/Core

VCF ARCHITECTURE PROVIDES

� Predictable application performance

� Deterministic latency

� Mixed 1/10/40G fabric

� Integrated control plane

Integrated RE


OVM VM VM

vSwitch

Virtual Server

OVM VM VM

vSwitch

Virtual ServerBare Metal

1 RU, 48 SFP+ & 1 QIC

Deterministic latency

� Resilient multi-path

� High bi-sectional bandwidth

� Smart leafs (local switching)

� Network ports on spine switches

Integrated control plane

� Integrated RE

� GRES/NSR/NSB

� Plug-and-play fabric

� Analytics on fabric ports


NG DC INTERCONNECT- EVPN

Scenario with VMTO

enabledScenario without VMTO

VM MOBILITY TRAFFIC OPTIMIZATION


PRIVATE MPLS WAN PRIVATE MPLS WAN

VLAN 10 VLAN 10 VLAN 10VLAN 10

DC1 DC2 DC1 DC2

VPLS DEPLOYMENT OPTIONS WITH MX – TODAY

MX Series

MX Series

MC-LAG

SRX

MX Series

LAG

VC

LAG LAG

IP, MPLSIP, MPLS IP, MPLS

LAG LAG

A A A ASS

LAG


SRX

NAT

FW

LB

IPSec

SRX

Switch

NAT

FW

LB

IPSecSwitch

NAT

FW

LB

IPSec

SRX

Switch

>1 VPLS devices

VPLS controlled Active-

Standby

Per VLAN

>1 VPLS devices

MC-LAG controlled Active-

Standby on LAN

Per VLAN

One VPLS device

Active forwarding through

all links of LAG

Server 1

DC 1

20.20.20.100/24

DCI WITH VPLS AND VRRP

VLAN 20


DC 2VLAN 10

10.10.10.100/24

DC 3

10.10.10.200/24

VLAN 10

Server 2 Server 3

PRIVATE MPLS WAN

Active

VRRP

DG:

10.10.10.1

Standby

VRRP

DG:

10.10.10.1

Standby

VRRP

DG:

10.10.10.1

Standby

VRRP

DG:

10.10.10.1

Task:

Server 3 in Data Center 3 needs to send packets

to Server 1 in Data Center 1.

Problem:

Server 3’s active Default Gateway for VLAN 10

is in Data Center 2.

Effect:

1. Traffic must travel via Layer 2 from Data

Center 3 to Data Center 2 to reach VLAN 10’s

active Default Gateway.

2. The packet must reach the Default Gateway

in order to be routed towards Data Center 1.

This results in duplicate traffic on WAN links

and suboptimal routing – hence the “Egress

Trombone Effect.”

EVPN provides standard-based VLAN Extension over a

shared IP/MPLS network.

http://datatracker.ietf.org/doc/draft-ietf-l2vpn-

evpn/?include_text=1

EVPN REQUIREMENTS (ON TOP OF VPLS)

All-Active Multi-Homing

Better Control Over

All available paths should be used (CE-PE, PE-PE)

MAC learning happens in control plane


Better Control Over

MAC Learning

ARP/ND Flooding

Minimization

L3 Egress Traffic

Forwarding Optimization

L3 Ingress Traffic

Forwarding Optimization

MAC learning happens in control plane

Proxy ARP support

Usage of Default Gateway Extended Community

Automatic advertisement of host routes into L3 VPN

Server 1

DC 1

20.20.20.100/24

EVPN: NO EGRESS TROMBONE EFFECT

VLAN 20


DC 2VLAN 10

10.10.10.100/24

DC 3

10.10.10.200/24

VLAN 10

Server 2 Server 3

PRIVATE MPLS WAN

Active RVI

DG:

10.10.10.1

Active RVI

DG:

10.10.10.1

Active RVI

DG:

10.10.10.1

Active RVI

DG:

10.10.10.1

Task:

Server 3 in Datacenter 3 needs to send packets

to Server 1 in Datacenter 1.

Solution:

Virtualize and distribute the Default Gateway

so it is active on every router that participates

in the VLAN.Effect:

1. Egress packets can be sent to any router on

VLAN 10 allowing the routing to be done in

the local datacenter. This eliminates the

“Egress Trombone Effect” and creates the

most optimal forwarding path for the Inter-DC

traffic.

EVPN TEST TOPOLOGY


EVPN

SUPPORTED CE-PE TOPOLOGY

Do not try to configure MC-LAG on PEs

Do not try to configure single LAG towards two PEs

CE (qfabric)

PE1 (MX240-3)

Supported CE-PE configPE1/PE2 config CE config


MPLS

PE2 (MX240-4)

HOW TO PREVENT DUPLICATE COPIES ON MULTI-HOMED SEGMENTS?

Designated Forwarder (DF) is elected

for each EVI or entire Ethernet

Segment.

DF is responsible for forwarding of

BUM traffic


CE1

PE1

PE2

MPLS

PE3 CE2

LAG

EVI LOAD BALANCING

Per default ALL CE links will be actively used for traffic

forwarding. Half of EVIs will have PE1 as DF and another half

PE2 as DF.

PE2

PE1


VM EGRESS TRAFFIC OPTIMIZATION

EVPN advantages over VPLS:

- No need for VRRP, Multi-homing VPLS, MC-LAG (less machinery and

protocol dependencies)

- IRB within EVPN VRF is configured on all PEs with a same IP address

(copy&paste IRB config on all PEs)

- Each PE has a mapping between Default GW IP and all PEs MACs

- If VM moves from DC1 to DC2 it continue to use “old” MAC address

from PE located in DC1. However, both PEs in DC2 forward traffic


from PE located in DC1. However, both PEs in DC2 forward traffic

destined to this MAC locally.

IRB MAC on MX240-4

IRB MAC on MX480-3

IRB MAC on MX480-4

EVPN ROUTE TYPE 2: MAC ADVERTISEMENT ROUTE

If you need to decode pcaps with EVPN NLRIs then you could use dissector I

put into Wireshark GIT repository: https://code.wireshark.org/review/#/c/296/


Server 1

DC 1

20.20.20.100/24

WITHOUT VMTO: INGRESS TROMBONE EFFECT

Route Mas

k

Cost Next Hop

10.10.10.0 24 5 Datacenter 2

10.10.10.0 24 10 Datacenter 3

DC 1’s Edge Router Table Without

VMTO

VLAN 20


DC 2VLAN 10

10.10.10.100/24

DC 3

10.10.10.200/24

VLAN 10

Server 2 Server 3

PRIVATE MPLS WAN

Task:



Problem:

Datacenter 1’s edge router prefers the path to

Datacenter 2 for the 10.10.10.0/24 subnet. It has

no knowledge of individual host IPs.

Effect:

1. Traffic from Server 1 is first routed across

the WAN to Datacenter 2 due to a lower cost

route for the 10.10.10.0/24 subnet.

2. Then the edge router in Datacenter 2 will

send the packet via Layer 2 to Datacenter 3.

10.10.10.0/24

Cost 5

10.10.10.0/24

Cost 10

VLAN 20

Server 1

DC 1

20.20.20.100/24

WITH VMTO: NO INGRESS TROMBONE EFFECT

Route Mas

k

Cost Next Hop

10.10.10.0 24 5 Datacenter 2

10.10.10.0 24 10 Datacenter 3

10.10.10.10

0

32 5 Datacenter 2

10.10.10.20

0

32 5 Datacenter 3DC 1’s Edge Router Table WITH VMTO

10.10.10.100/32 Cost 10.10.10.200/32 Cost


DC 2VLAN 10

10.10.10.100/24

DC 3

10.10.10.200/24

VLAN 10

Server 2 Server 3

PRIVATE MPLS WAN

Effect:

1. Ingress traffic destined for Server 3 is sent

directly across the WAN from Datacenter 1 to

Datacenter 3. This eliminates the “Ingress

Trombone Effect” and creates the most

optimal forwarding path for the Inter-DC

traffic.

Task:



Solution:

In addition to sending a summary route of

10.10.10.0/24 the datacenter edge routers also

send host routes which represent the location

of local servers.

10.10.10.0/24

Cost 5

10.10.10.0/24

Cost 10

10.10.10.100/32 Cost

5

10.10.10.200/32 Cost

5

REFERENCES

MetaFabric Solution Brief:

http://www.juniper.net/us/en/local/pdf/solutionbriefs/3510

495-en.pdf

MetaFabric 1.0 Reference Architecture:

http://www.juniper.net/us/en/local/pdf/reference-

architectures/8030012-en.pdf


architectures/8030012-en.pdf

MetaFabric 1.0 Design and Implementation Guide:

http://www.juniper.net/us/en/local/pdf/design-

guides/8020020-en.pdf

Иван Лысогор Senior System Engineer · 2015-07-30 · METAFABRICARCHITECTURE Иван...

Documents

Transcript of Иван Лысогор Senior System Engineer · 2015-07-30 · METAFABRICARCHITECTURE Иван...