Download - Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Transcript
Page 1: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Extending KVM Models Toward High-Performance NFV

Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang 14 October 2014

Page 2: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Legal Disclaimer �  INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO

LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.

�  Intel may make changes to specifications and product descriptions at any time, without notice.

�  All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.

�  Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.

�  Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

�  *Other names and brands may be claimed as the property of others.

�  Copyright © 2014 Intel Corporation.

Page 3: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  The Challenge

•  Architecture Proposals for NFV for KVM

•  Current Status and Summary

3

Agenda

Page 4: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

4

NFV Vision from ETSI Source: http://portal.etsi.org/nfv/nfv_white_paper2.pdf

Page 5: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  High performance across all packet sizes, including small packets (e.g. 64B)

•  Real-time processing, including low latency and jitter

•  RAS

•  Security

•  ...

5

New/Different Requirements for NFV Compared with Conventional Virtualization

Focus on Performance Topics Today

Page 6: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

The Challenge

0

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

160,000,000

64 128 192 256 320 384 448 512 576 640 704 768 832 896 960 1024 1088 1152 1216 1280 1344 1408 1472

10GbE Packets Per Second 40GbE Packets Per Second 100GbE Packets Per Second

6.72ns

16.8ns

67.2ns

Satu

ratio

n Li

ne R

ate

(MPP

S)

Source: DPDK Summit, Venky Venkatesan, “Application Performance Tuning and Future Optimizations in DPDK”, September 8, 2014 https://www.youtube.com/watch?v=qpfwDySweUA

Disclaimer: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Page 7: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

42 55

93

164.9

255

0

50

100

150

200

250

300

2009 2010 2011 2012 2013 Syst

em L

evel

L3

Perf

orm

ance

(M

PPS)

* Other names and brands may be claimed as the property of others.

Disclaimer: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Intel® DPDK Performance A snapshot of on different architectures

Integrated Memory Controller PCI-E Gen2

Data Direct I/O Integrated PCI-E Gen3 AVX (integer, 128-bit)

4x10 GbE NICs

Plat

form

Fe

atur

es

Source: DPDK Summit, Venky Venkatesan, “Application Performance Tuning and Future Optimizations in DPDK”, September 8, 2014 https://www.youtube.com/watch?v=qpfwDySweUA

Page 8: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Linux Kernel

VM or User Process

Kernel (virt. I/O)

VM1

VT-d, SR-IOV

Middle Box (e.g. virtual switch)

KVM

Kernel (virt. I/O)

VM2

...

8

Focus Areas for NFV Performance on KVM Recall 67.2ns, 16.8ns, …

Fast and Efficient Inter-VM Communication

Generic: Network I/O, NUMA, NUMA-I/O, Caching, Affinity, …

Page 9: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  More cores

•  More middle boxes per socket, per server

•  Service chaining on server

•  Lower latency

•  Inter-VM (i.e. intra-node) vs. Inter-node

•  Higher Bandwidth

•  Memory (or cache) vs. PCIe bus

9

Why Inter-VM Communication?

Figure 1. The Intel® Xeon® processor E5-2600 V2 product family Microarchitecture

Source (Figure 1.): https://software.intel.com/en-us/articles/intel-xeon-processor-e5-2600-v2-product-family-technical-overview

Page 10: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  Notifications for queue control

•  Kick, Door Bell

•  Virtual Switch

•  Packet Transmission

•  Copy, etc.

•  Transitions

•  User-Kernel

•  Guest-Host

10

Inter-VM Communication on KVM

Host OS

Rx VM

tap

Tx VM

tap switch

app app

hypervisor

Stack

drv

dev

dev

Stack

drv

Switching path can be a big performance bottleneck

X

Y

X 0.712 Mpps*

Y 0.717 Mpps*

*Intel internal measurements

64B packets, virtio-net + vhost-net

Page 11: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

TSC Cycles (Haswell 3.2GHz), Round Trip*: •  User<->Kernel (System Call) in VM (on KVM)

•  E.g. getppid(): 1300 (≈ 400ns) •  Guest<->Host (Hyper Call)

•  E.g. Null Hypercall: 1500-1600 (≈ 500ns)

To reach Saturation Line Rate (10GbE): •  If system call/Hyper call is used for each 64B packet transmission, we

would need: •  > 6-7 Cores**

•  40GbE:

•  > 24-28 Cores?

11

Cost of Transitions/Isolation Perspective of CPU Cycles

**:400/67.2 = 5.9, 500/67.2 = 7.4 *Intel internal measurements Disclaimer: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and

MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Practically, those are rather lower bounds because batching is limited and actual packet processing in hypercalls overturns gain of batching.

Page 12: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  The Challenge

•  Architecture Proposals for NFV for KVM

•  Current Status and Summary

12

Agenda

Page 13: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

1.  Move knowledge and control for inter-VM communication to VMs

2.  Allow VMs to access other VMs to share or access memory in a safe way

•  Provide VMs with “Protected Memory View” •  Mapping itself is provided by the hypervisor

3.  Allow VMs to use low-latency notification mechanisms w/o VM exits or interrupts •  E.g. MONITOR/MWAIT, Posted Interrupt

13

Solutions: Empower Guests in a Safe Way Avoid hypervisor interventions

Page 14: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Motivation: •  Why does a kernel module need to know about data structures

for PV drivers in guests?

•  Because we trust kernel or kernel modules only.

•  What if we trust specific (part of) guests…

•  Vhost-net in guest can avoid hypercalls if it can directly access destination guests (virtqueue, etc.)

14

Example: vhost-net Functionality in Guests vhost-user is already there

Page 15: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

15

High-Level Architecture for Fast Inter-VM Communication (w/o VT-d, SR-IOV)

Linux Kernel

KVM

VM1

Kernel

virto-net

VM2

Kernel

virto-net

Shared memory for synchronization

Fast Path Fast Path

Protected Memory View

Low-Latency Notification

1. Data Transmission

2. Notification Direct Access to Guests In Protected Memory View

Vhost-net API

Fast Path can work with virtio-net or independently

virtio-net virtio-net

Page 16: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Linux Kernel

KVM

VM1

Kernel

virtio-net

VM2

Kernel

virtio-net

Shared memory for synchronization

Fast Path Fast Path

Middle Box (e.g. virtual switch)

VM0

VT-d, SR-IOV

Fast Packet Transmission

Shared memory for synchronization

16

High-Level Architecture for Fast Inter-VM Communication (with VT-d, SR-IOV)

Fast Packet Transmission can be in user-level

Page 17: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  VMFUNC instruction with EAX = 0

•  Value in ECX selects an entry from the EPTP (Extended-Page-Table Pointer) list

•  Available in Ring 0-3, executed in guest •  No VM exit

•  Can be virtualized if not available

17

Introducing VM Function 0: EPTP* Switching

… EPTP …

ECX (index)

EPTP list (4KB)

VMCS (per VCPU) *:Extended-Page-Table Pointer

Page 18: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

EPTP Switching and Trampoline Code

Guest Physical Pages

Protected View (code, data)

- - -

Default EPT

EPT: Host Physical Pages

XWR

18

No Access Trampoline code for

VMFUNC

XWR -WR

X-R X-R

•  VMFUNC executed outside Trampoline Code will cause EPT violation at next instruction

•  Hypervisor needs to restore Default EPT to deliver virtual interrupts

EPTP switching

Default View Alternate View

Page 19: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

start_xmit(*skb, *dev) {

... send(packets);}

send(*packet) {... VMFUNC #0, EPTP; Tx(packets); VMFUNC #0, 0}

Page Boundary

Tx(*packet) { move_data(); notifify();}

XWR

start_xmit(*skb, *dev) {

... send(packets);}

send(*packet) {... VMFUNC #0, EPTP; Tx(packets); VMFUNC #0, 0}

Tx(*packet) { move_data(); notifify();}

---

-WR

-WRXWR

XWR

X-R X-R

EPT Perm.

Modify queue descriptors

Modify queue descriptors

Move Data by Tx()

19

More Details: Transmitting Packets Destination VM Source VM

Trampoline Code

Protected View

2

4 3

1

5

Default View Alternate View

Page 20: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  Posted Interrupt

•  Deliver virtual interrupts on destination guests w/o VM exits.

•  Already supported by KVM •  Still requires VM exit on source guest

•  MONITOR/MWAIT (Energy-Efficient Polling) between guests

•  The feature is not advertised on KVM today

•  Use variables on shared memory between source and destination

•  PAUSE Loop (Polling) between guests

•  Lowest latency, but not energy efficient

In practice, combine Interrupt and Polling (like NAPI)

20

Low-Latency Notification Known methods

Page 21: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Minimize impact of TLB misses, cache misses: •  Large pages (both guest, EPT, VT-d), NUMA, IO-NUMA, Data

Direct I/O

•  E.g. LIFO memory pool

•  Zero-copy

•  E.g. Add source buffers mapping to EPT of destination •  If EPT PTEs were not valid, no INVEPT is required

21

Practices for Performance General

Disclaimer: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Page 22: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Frequency of VMFUNC operation: •  Cost of VMFUNC is about 150 TSC cycles (Haswell, 3.2 GHz)*

•  Around 50ns, and sensitive to TLB, caches •  Recall 67.2ns, 16.8ns, …

To reach Saturation Line Rate (10GbE): •  If VMFUNC is called for each 64B packet transmission, we

•  > 1-2 Cores (100ns for round-trip)

•  40GbE: •  > 4-8 Cores?

•  The cost of VMFUNC would be relatively small, and it would provide scalable performance

22

Practice for Performance EPTP Switching

*Intel internal measurements Disclaimer: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

getppid() in VM: 1300 (≈ 400ns)

Null Hypercall: 1500-1600 (≈ 500ns)

Practically, those are rather lower bounds because batching is limited and actual packet processing overturns gain of batching.

Page 23: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  Trampoline Code is loaded by the guest, but the EPT permission (X-R) is set by KVM

•  Should be signed together with the code in the Protected View in advance

•  The set of pages (in Destination VM) accessed by code in Protected View need to be checked and added by KVM

•  In a way, code in Protected View is an extension of the KVM/hypervisor running in controlled environment (still in VXM non-root mode)

23

Security Consideration

Page 24: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  The Challenge

•  Architecture Proposals for NFV for KVM

•  Current Status and Summary

24

Agenda

Page 25: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

PoC in progress: •  Measured cost of

VMFUNC, memory bandwidth

•  Enabled and measured latency of MONITOR/MWAIT in guests

•  Measuring path A

•  Working on path B

25

Current Status PoC

Linux Kernel

KVM

VM1

Kernel

virto-net

VM2

Kernel

virto-net

Shared memory for synchronization

Fast Path Fast PathA

B virtio-net virtio-net

Page 26: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

Benefits of the Architecture: •  Contain knowledge and control for Inter-VM communication in

guests

•  Allow KVM to enable more optimization and customization for guests to handle high network loads efficiently

•  More efficient and scalable than existing ones

•  Work with direct I/O assignment as well

Next Step:

•  Complete PoC and get more data

26

Summary

Page 27: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

27

Backup

Page 28: Extending KVM Models Toward High-Performance NFV · 2016-02-07 · Extending KVM Models Toward High-Performance NFV Jun Nakajima, James Tsai, Mesut Ergin, Yang Zhang, and Wei Wang

•  Can occur only in guest (vector 20)

•  Some EPT violations can generate #VE instead of VM exits (controlled by hypervisor)

•  Can virtualized if not available

#VE: Virtualization Exception