K8s集群pod信息
```bash
$ kubectl get pod -o wide
```
```bash
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox-deployment-6576988595-dbpq7 1/1 Running 4 33h 10.244.1.12 k8s-node1 <none> <none>
busybox-deployment-6576988595-l5w7r 1/1 Running 4 33h 10.244.2.14 k8s-node2 <none> <none>
busybox-deployment-6576988595-wfvn2 1/1 Running 4 33h 10.244.2.13 k8s-node2 <none> <none>
```
#### _**实验 `"pod-10.244.1.12(k8s-node1)"` ping `"pod-10.244.2.14(k8s-node2)"`，跟踪数据包的传输过程。**_
_**1. "10.244.1.12" ping "10.244.2.14" 匹配默认路由0.0.0.0走容器eth0，到达veth pair的另一端veth8001ebf4**_
kubectl连接到pod-10.244.1.12 ping 10.244.2.14
```sh
$ kubectl exec -ti busybox-deployment-6576988595-dbpq7 sh
```
```html
/ # ping 10.244.2.14
PING 10.244.2.14 (10.244.2.14): 56 data bytes
64 bytes from 10.244.2.14: seq=0 ttl=62 time=0.828 ms
```
kubectl连接到pod-10.244.1.12查看路由信息
```sh
$ kubectl exec -ti busybox-deployment-6576988595-dbpq7 sh
```
```html
/ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.1.1 0.0.0.0 UG 0 0 0 eth0 #数据包会匹配这条路由
10.244.0.0 10.244.1.1 255.255.0.0 UG 0 0 0 eth0
10.244.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
```
查看k8s-node1的pod-10.244.1.12 eth0对应veth pair另一端为veth8001ebf4 [(如何查看容器对应的veth网卡)](https://ynotes.cn/blog/article_detail/260)
k8s-node1抓取veth8001ebf4网卡的数据包
```sh
tcpdump -i veth8001ebf4
```
```sh
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth8001ebf4, link-type EN10MB (Ethernet), capture size 262144 bytes
00:30:00.124500 IP 10.244.1.12 > 10.244.2.14: ICMP echo request, id 1336, seq 495, length 64
```
_**2.veth8001ebf4桥接到cni0,数据包发送到cni0**_
```sh
$ tcpdump -i cni0 -e -nnn -vvv
```
```sh
tcpdump: listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:32:29.522019 d6:10:b7:91:f0:ac > 0a:58:0a:f4:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16442, offset 0, flags [DF], proto ICMP (1), length 84)
10.244.1.12 > 10.244.2.14: ICMP echo request, id 1862, seq 89, length 64
```
_**3.cni0查看路由表route -n,会路由匹配10.244.2.0-flannel.1**_
```sh
[root@k8s-node1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.100.254 0.0.0.0 UG 100 0 0 ens192
10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.1.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1 #会匹配到这条路由
172.16.100.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
```
_**4.flannel.1收到cni0的数据帧。a.修改内部数据帧的地址(MAC source:FE:90:C4:18:69:A7[k8s-node1 flannel.1的MAC],MAC dest:B6:72:AE:36:7B:6C[k8s-node2 flannel.1的MAC])，b.封装vxlan头(VNI:1)，c.再封装UDP头部(UDP dest port:8472)，d.封装节点ip头。**_
a.修改内部数据帧，源MAC地址为k8s-node1 flannel.1的MAC地址。目的MAC地址为10.244.2.0网络的网关10.244.2.0(k8s-node2 flannel.1)的MAC地址
```sh
arp -n|grep flannel.1
```
```sh
10.244.2.0 ether b6:72:Ae:36:7b:6c CM flannel.1 #内部网络网关10.244.2.0的MAC地址
10.244.0.0 ether 6e:f8:85:d7:09:17 CM flannel.1
```
b.封装vxlan头,VNI为vetp设备的vxlan id
c.封装UDP头部，dest port为vetp设备的dstport
查看flannel.1 vetp信息
```sh
$ ip -d link show flannel.1
```
```sh
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether fe:90:c4:18:69:a7 brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 1 local 172.16.100.101 dev ens192 srcport 0 0 dstport 8472 nolearning ageing 300 noudpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 #vxlan id 为1,vetp UDP dstport 8472
```
d.封装节点ip头，源ip为k8s-node1的ip，目的ip k8s-node2的ip。目的ip通过查看bridge fdb 对应的vetp MAC获取节点ip
```sh
$ bridge fdb|grep flannel.1
```
```sh
e2:3f:07:99:Cf:6f dev flannel.1 dst 172.16.100.102 self permanent
b6:72:Ae:36:7b:6c dev flannel.1 dst 172.16.100.102 self permanent #通过vetp MAC地址找到节点ip:172.16.100.102
6e:f8:85:d7:09:17 dev flannel.1 dst 172.16.100.100 self permanent
```
查看ens192的数据包(vetp已封包完成),分析数据包内容
```sh
tcpdump -l -nnnvvveXX -i ens192 'port 8472 and udp[8:2] = 0x0800 & 0x0800'
```
```sh
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
02:09:45.801867 00:50:56:93:6a:3a > 00:50:56:93:63:3b, ethertype IPv4 (0x0800), length 148: (tos 0x0, ttl 64, id 30086, offset 0, flags [none], proto UDP (17), length 134)
172.16.100.101.40592 > 172.16.100.102.8472: [no cksum] OTV, flags [I] (0x08), overlay 0, instance 1
fe:90:c4:18:69:a7 > b6:72:Ae:36:7b:6c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 8965, offset 0, flags [DF], proto ICMP (1), length 84)
10.244.1.12 > 10.244.2.14: ICMP echo request, id 3143, seq 1102, length 64
0x0000: 0050 5693 633b 0050 5693 6a3a 0800 4500 #帧头:0050 5693 633b^DEST_MAC,0050 5693 6a3a^SRC_MAC,0800^IPV4。IP头:4^ipv4,5^四字节个数,00^TOS
0x0010: 0086 7586 0000 4011 e3f4 ac10 6465 ac10 #0086^总长度,7586^标识符,0000^偏移量,40^生存周期,11^上层协议,e3f4^校验和,ac10 6465^SRC_IP,
0x0020: 6466 9e90 2118 0072 0000 0800 0000 0000 #ac10 6466^DST_IP。UDP头:9e90^SRC_PORT,2118^DST_PORT,0072^UDP长度,0000^校验和。VXLAN头:08^标志位,00 0000^保留字段
0x0030: 0100 b672 ae36 7b6c fe90 c418 69a7 0800 #0000 01^VNID,00^保留字段。内部帧头:b672 ae36 7b6c^DST_MAC,fe90 c418 69a7^SRC_MAC,0800^IPV4。
0x0040: 4500 0054 2305 4000 3f01 ffa2 0af4 010c #内部IP头:IP头:4^ipv4,5^四字节个,00^TOS,#0054^总长度,2305^标识符,4000^偏移量,3f^生存周期,01^上层协议,ffa2^校验和,0af4 010c^SRC_IP
0x0050: 0af4 020e 0800 847d 0c47 044e 0d1e 55cf #0af4 020e^DST_IP。ICMP协议:08^请求报文,00^代码,内部数据帧:847d^校验和,0c47 044e^内部CFS,0d1e 55cf^外部CFS。
0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0090: 0000 0000 ....
```
