东莞网站制作方案定制,免费建站系统wordpress,西安复工无需核酸检测,微商怎么开通1. 背景 没有使用DVR的场景#xff1a; 从图中可以明显看到东西向和南北向的流量会集中到网络节点#xff0c;这会使网络节点成为瓶颈。 如果启用DVR#xff0c;如下图#xff1a; 对于东西向的流量#xff0c; 流量会直接在计算节点之间传递。 对于南北向的流量#xff…1. 背景 没有使用DVR的场景 从图中可以明显看到东西向和南北向的流量会集中到网络节点这会使网络节点成为瓶颈。 如果启用DVR如下图 对于东西向的流量 流量会直接在计算节点之间传递。 对于南北向的流量如果有floating ip流量就直接走计算节点。如果没有floating ip则会走网络节点。 2.部署以及流量走向 2.1东西向流量 VM1 (10.0.1.5 Net1) ping VM2 (10.0.2.5 Net2) 1) VM1 (10.0.1.5) - qr (10.0.1.1) VM1 根据默认路由发送arp广播请求qr网关的地址请求到网关地址后icmp报文走向qr口。 关于报文格式的一点解释当VM1 ping VM2时报文的源/目的IP始终不变报文的源/目的MAC则会根据不同的路段而变化。 同时br-tun网桥会丢弃目的地址是interface_distributed接口的arp广播不至于让不必要的流量流向外面 # ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid0x4):
...
cookie0x0, duration64720.432s, table1, n_packets4, n_bytes168, idle_age64607, priority3,arp,dl_vlan1,arp_tpa10.0.1.1 actionsdrop
... 2qr (10.0.1.1) - qr (10.0.2.1) 进入qrouter namespace后利用linux内核的高级路由功能查看路由规则。 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
32768: from 10.0.1.5 lookup 16
32769: from 10.0.2.3 lookup 16
167772417: from 10.0.1.1/24 lookup 167772417
167772417: from 10.0.1.1/24 lookup 167772417
167772673: from 10.0.2.1/24 lookup 167772673 先查看main表 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table main
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28 在main表中满足以上路由因此会从另一个qr口出去。Q1不同计算节点的同一子网下qr口ip是相同的吗 3qr - br-int 之后需要去查询10.0.2.5的MAC地址 MAC是由neutron使用静态ARP的方式设定的由于Neutron知道所有VM的信息因此他可以事先设定好静态ARP # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip nei
10.0.1.5 dev qr-ddbdc784-d7 lladdr fa:16:3e:da:75:6d PERMANENT
10.0.2.3 dev qr-001d0ed9-01 lladdr fa:16:3e:a4:fc:98 PERMANENT
10.0.1.6 dev qr-ddbdc784-d7 lladdr fa:16:3e:9f:55:67 PERMANENT
10.0.2.2 dev qr-001d0ed9-01 lladdr fa:16:3e:13:55:66 PERMANENT
10.0.2.5 dev qr-001d0ed9-01 lladdr fa:16:3e:51:99:b8 PERMANENT
10.0.1.4 dev qr-ddbdc784-d7 lladdr fa:16:3e:da:e3:6e PERMANENT
10.0.1.7 dev qr-ddbdc784-d7 lladdr fa:16:3e:14:b8:ec PERMANENT
169.254.31.29 dev rfp-0fbb351e-a lladdr 42:0d:9f:49:63:c6 STALE 此时报文进入br-int根据table 0 进行normal转发 cookie0x0, duration16440.644s, table0, n_packets1074, n_bytes104318, idle_age8917, priority1 actionsNORMAL normal动作则表示根据OVS fdb表项匹配目的MAC地址从而决定该报文要往哪个端口发送。如果没有该MAC的fdb表项记录则进行泛洪对除了报文进来的端口以外的所有同属于一个vlan的端口发送该报文。例如 # ovs-appctl fdb/show br-intport VLAN MAC Age
LOCAL 0 da:91:42:cd:fb:44 1818 0 52:54:00:a9:b8:b0 019 0 52:54:00:a9:b8:b1 0 因此如果此时VM2也在该compute node上则VM2也会直接收到该报文不需要走br-tun有了VM2的MAC fdb表项记录后。否则继续往br-tun走。 4br-int - br-tun - 出compute node 1 然后报文从br-int进入br-tun匹配流表 cookie0x0, duration66172.51s, table0, n_packets58, n_bytes5731, idle_age20810, hard_age65534, priority1,in_port3 actionsresubmit(,4)cookie0x0, duration67599.526s, table0, n_packets273, n_bytes24999, idle_age1741, hard_age65534, priority1,in_port1 actionsresubmit(,1)cookie0x0, duration64437.052s, table0, n_packets28, n_bytes2980, idle_age20799, priority1,in_port4 actionsresubmit(,4)cookie0x0, duration67601.704s, table0, n_packets5, n_bytes390, idle_age65534, hard_age65534, priority0 actionsdropcookie0x0, duration66135.811s, table1, n_packets140, n_bytes13720, idle_age65534, hard_age65534, priority1,dl_vlan1,dl_srcfa:16:3e:66:13:af actionsmod_dl_src:fa:16:3f:fe:49:e9,resubmit(,2)cookie0x0, duration64082.141s, table1, n_packets2, n_bytes200, idle_age64081, priority1,dl_vlan2,dl_srcfa:16:3e:69:b4:05 actionsmod_dl_src:fa:16:3f:fe:49:e9,resubmit(,2)cookie0x0, duration66135.962s, table1, n_packets1, n_bytes98, idle_age65301, hard_age65534, priority2,dl_vlan1,dl_dstfa:16:3e:66:13:af actionsdrop cookie0x0, duration64082.297s, table1, n_packets0, n_bytes0, idle_age64082, priority2,dl_vlan2,dl_dstfa:16:3e:69:b4:05 actionsdropcookie0x0, duration66136.115s, table1, n_packets4, n_bytes168, idle_age65534, hard_age65534, priority3,arp,dl_vlan1,arp_tpa10.0.1.1 actionsdropcookie0x0, duration64082.449s, table1, n_packets2, n_bytes84, idle_age63991, priority3,arp,dl_vlan2,arp_tpa10.0.2.1 actionsdropcookie0x0, duration67599.22s, table1, n_packets123, n_bytes10687, idle_age1741, hard_age65534, priority0 actionsresubmit(,2) 先匹配table 0然后匹配table 1它会把源MAC地址另一个qr口改为全局唯一与计算节点绑定的MAC。 这个全局唯一和计算节点绑定的MAC地址是由neutron全局分配的数据库中可以看到这个MAC是每个host一个 它的base MAC是可以在neutron.conf中配置的 同时后面的两条table1会丢弃目标ip是interface_distributed接口的ARP和目的MAC是interface_distributed的包以防止虚机发送给本地IP的包不会被转发到网络中。 然后继续查询table 2table 2是vxlan表如果是广播包就会查询表22如果是单播包就查询table 20 cookie0x0, duration67601.554s, table2, n_packets176, n_bytes16981, idle_age20810, hard_age65534, priority0,dl_dst00:00:00:00:00:00/01:00:00:00:00:00 actionsresubmit(,20)cookie0x0, duration67601.406s, table2, n_packets92, n_bytes7876, idle_age1741, hard_age65534, priority0,dl_dst01:00:00:00:00:00/01:00:00:00:00:00 actionsresubmit(,22) 广播MAC地址是FF:FF:FF:FF:FF:FF组播MAC地址以01-00-5E开头具体可查看http://book.51cto.com/art/200904/120471.htm匹配规则满足CIDR。 ICMP包是单播包因此会查询表20由于开启了L2 pop功能在表20中会事先学习到应该转发到哪个VTEP cookie0x0, duration64015.308s, table20, n_packets0, n_bytes0, idle_age64015, priority2,dl_vlan2,dl_dstfa:16:3e:51:99:b8 actionsstrip_vlan,set_tunnel:0x3eb,output:4 (Q2社区br-tun下面的隧道口是如何与物理口建立联系的?) 5进compute node 2 - br-tun 在br-tun中从外面进入的报文将首先匹配以下table0表 cookie0x0, duration66293.658s, table0, n_packets31, n_bytes3936, idle_age22651, hard_age65534, priority1,in_port3 actionsresubmit(,4)cookie0x0, duration69453.368s, table0, n_packets103, n_bytes9360, idle_age22651, hard_age65534, priority1,in_port1 actionsresubmit(,1)cookie0x0, duration66292.808s, table0, n_packets20, n_bytes1742, idle_age3598, hard_age65534, priority1,in_port4 actionsresubmit(,4)cookie0x0, duration69455.675s, table0, n_packets5, n_bytes390, idle_age65534, hard_age65534, priority0 actionsdrop 在table 4中会将对应的vni改为本地vlan id之后查询表9: cookie0x0, duration65937.871s, table4, n_packets32, n_bytes3653, idle_age22651, hard_age65534, priority1,tun_id0x3eb actionsmod_vlan_vid:3,resubmit(,9)cookie0x0, duration66294.732s, table4, n_packets19, n_bytes2025, idle_age3598, hard_age65534, priority1,tun_id0x3e9 actionsmod_vlan_vid:2,resubmit(,9)cookie0x0, duration69455.115s, table4, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority0 actionsdrop 在表9中如果发现包的源地址是全局唯一并与计算节点绑定的MAC地址就将其转发到br-int: cookie0x0, duration69453.507s, table9, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority1,dl_srcfa:16:3f:fe:49:e9 actionsoutput:1cookie0x0, duration69453.782s, table9, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority1,dl_srcfa:16:3f:72:3f:a7 actionsoutput:1cookie0x0, duration69453.23s, table9, n_packets56, n_bytes6028, idle_age3598, hard_age65534, priority0 actionsresubmit(,10) 6br-tun - br-int 进入br-int后在table 0中如果是全局唯一并与计算节点绑定的MAC地址就查询table 1否则就正常转发 在table 1中事先设定好了flow如果目的MAC是发送给VM2就将源MAC改为Net2的网关MAC地址qr口Q3修改源MAC的原因为了报文能返回。 cookie0x0, duration70039.903s, table0, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority2,in_port6,dl_srcfa:16:3f:72:3f:a7 actionsresubmit(,1)cookie0x0, duration70039.627s, table0, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority2,in_port6,dl_srcfa:16:3f:fe:49:e9 actionsresubmit(,1)cookie0x0, duration70040.053s, table0, n_packets166, n_bytes15954, idle_age4184, hard_age65534, priority1 actionsNORMALcookie0x0, duration66458.695s, table1, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority4,dl_vlan3,dl_dstfa:16:3e:51:99:b8 actionsstrip_vlan,mod_dl_src:fa:16:3e:69:b4:05,output:12cookie0x0, duration66877.515s, table1, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority4,dl_vlan2,dl_dstfa:16:3e:14:b8:ec actionsstrip_vlan,mod_dl_src:fa:16:3e:66:13:af,output:9cookie0x0, duration66877.369s, table1, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority2,ip,dl_vlan2,nw_dst10.0.1.0/24 actionsstrip_vlan,mod_dl_src:fa:16:3e:66:13:af,output:9cookie0x0, duration66458.559s, table1, n_packets0, n_bytes0, idle_age65534, hard_age65534, priority2,ip,dl_vlan3,nw_dst10.0.2.0/24 actionsstrip_vlan,mod_dl_src:fa:16:3e:69:b4:05,output:12 7br-int - VM2 至此VM2就会收到VM1的包了。从通信的过程可以看到跨网段的东西向流量没有经过网络节点。 2.2 南北向流量VM有floating ip VM1 local ip10.0.1.5 , floating ip: 172.24.4.5ping 8.8.8.8 1VM1 (10.0.1.5) - qr (10.0.1.1) 与上面一致 2) qr (10.0.1.1) - rfp (169.254.31.28) - fpr (169.254.31.29) 进入qrouter namespace后 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
32768: from 10.0.1.5 lookup 16
32769: from 10.0.2.3 lookup 16
167772417: from 10.0.1.1/24 lookup 167772417
167772417: from 10.0.1.1/24 lookup 167772417
167772673: from 10.0.2.1/24 lookup 167772673 在main表中没有合适的路由 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table main
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28 由于包是从10.0.1.5发来的之后会查看table 16包会命中这条路由。 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table 16
default via 169.254.31.29 dev rfp-0fbb351e-a 路由之后会通过netfilter的POSTROUTING链中进行SNAT # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-float-snat (1 references)pkts bytes target prot opt in out source destination0 0 SNAT all -- * * 10.0.2.3 0.0.0.0/0 to:172.24.4.70 0 SNAT all -- * * 10.0.1.5 0.0.0.0/0 to:172.24.4.5
... 之后就可以看到包会通过rfp-0fbb351e-a发送给169.254.31.29。 端口rfp-0fbb351e-a和fpr-0fbb351e-a是一对veth pair。在fip namespace中你可以看到这个接口 3) fpr (169.254.31.29) - fg (172.24.4.6) 到了fip的namespace之后会查询路由 在main表里有通往公网的默认路由 # ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 ip route
default via 172.24.4.1 dev fg-081d537b-06
169.254.31.28/31 dev fpr-0fbb351e-a proto kernel scope link src 169.254.31.29
172.24.4.0/24 dev fg-081d537b-06 proto kernel scope link src 172.24.4.6
172.24.4.5 via 169.254.31.28 dev fpr-0fbb351e-a
172.24.4.7 via 169.254.31.28 dev fpr-0fbb351e-a 通过fg-081d537b-06发送到br-ex。这是从虚机发送到公网的过程。(Q4br-ex上的流表是什么样的如果没有br-ex直接走br-int流表会有什么变化) 外网 ping VM1 ( floating ip: 172.24.4.5) 1fip namespace 此时fip的namespace会做arp代理 Q5arp代理的作用外部arp广播报文进入fip ns查询172.24.4.5的mac地址由于arp报文无法跨路由器传播而且该ip在qrouter ns里。 # ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 sysctl net.ipv4.conf.fg-081d537b-06.proxy_arp
net.ipv4.conf.fg-081d537b-06.proxy_arp 1 可以看到接口的arp代理是打开的对于floating ip 有以下路由 # ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 ip route
...
172.24.4.5 via 169.254.31.28 dev fpr-0fbb351e-a
172.24.4.7 via 169.254.31.28 dev fpr-0fbb351e-a
... ARP会去通过VETH Pair到IR(Inter Router)的namespace中去查询在IR中可以看到接口rfp-0fbb351e-a配置了floating ip: # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: rfp-0fbb351e-a: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether ea:5c:56:9a:36:9c brd ff:ff:ff:ff:ff:ffinet 169.254.31.28/31 scope global rfp-0fbb351e-avalid_lft forever preferred_lft foreverinet 172.24.4.5/32 brd 172.24.4.5 scope global rfp-0fbb351e-avalid_lft forever preferred_lft foreverinet 172.24.4.7/32 brd 172.24.4.7 scope global rfp-0fbb351e-avalid_lft forever preferred_lft foreverinet6 fe80::e85c:56ff:fe9a:369c/64 scope link valid_lft forever preferred_lft forever
17: qr-ddbdc784-d7: BROADCAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UNKNOWN group default link/ether fa:16:3e:66:13:af brd ff:ff:ff:ff:ff:ff inet 10.0.1.1/24 brd 10.0.1.255 scope global qr-ddbdc784-d7 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe66:13af/64 scope link valid_lft forever preferred_lft forever
19: qr-001d0ed9-01: BROADCAST,UP,LOWER_UP mtu 1500 qdisc noqueue state UNKNOWN group default link/ether fa:16:3e:69:b4:05 brd ff:ff:ff:ff:ff:ff inet 10.0.2.1/24 brd 10.0.2.255 scope global qr-001d0ed9-01 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe69:b405/64 scope link valid_lft forever preferred_lft forever 因此fip的namespace会对这个floating ip进行ARP回应。 外部发起目标地址为floating ip的请求后fip会将其转发到IR中IR的RPOROUTING链中规则如下 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-PREROUTING (1 references)pkts bytes target prot opt in out source destination0 0 REDIRECT tcp -- * * 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 96970 0 DNAT all -- * * 0.0.0.0/0 172.24.4.7 to:10.0.2.30 0 DNAT all -- * * 0.0.0.0/0 172.24.4.5 to:10.0.1.5
... 这条DNAT规则会将floating ip地址转换为内部地址之后进行路由查询 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28 目的地址是10.0.1.0/24网段的因此会从qr-ddbdc784-d7转发出去。之后就会转发到br-int再到虚机。 2.3 南北向流量VM没有floating ip 在虚机没有floating ip的情况下从虚机发出的包会首先到IRIR中查询路由 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
32768: from 10.0.1.5 lookup 16
32769: from 10.0.2.3 lookup 16
167772417: from 10.0.1.1/24 lookup 167772417
167772673: from 10.0.2.1/24 lookup 167772673 会先查询main表之后查询167772417表。Q7不会匹配table 16 # ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table 167772417
default via 10.0.1.6 dev qr-ddbdc784-d7 这个表会将其转发给10.0.1.6,而这个IP就是在network node上的router_centralized_snat接口。 在network node的snat namespace中我们可以看到这个接口。 $ sudo ip netns exec snat-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-snat (1 references)pkts bytes target prot opt in out source destination0 0 SNAT all -- * * 10.0.1.0/24 0.0.0.0/0 to:172.24.4.40 0 SNAT all -- * * 10.0.2.0/24 0.0.0.0/0 to:172.24.4.4
... 这里就和以前的L3类似会将没有floating ip的包SNAT成一个172.24.4.4(DVR的网关臂)。这个过程是和以前L3类似的不再累述。 参考http://www.sxt.cn/u/756/blog/3168 3. QA (未完) 转载于:https://www.cnblogs.com/xingyun/p/4876083.html
