Calico网络
```sh
Calico主要由三个部分组成：
Felix：以DaemonSet方式部署，运行在每一个Node节点上，主要负责维护宿主机上路由规则以及ACL规则。
BGP Client（BIRD）：主要负责把 Felix 写入 Kernel 的路由信息分发到集群 Calico 网络。
Etcd：分布式键值存储，保存Calico的策略和网络配置状态。
calicoctl：允许您从简单的命令行界面实现高级策略和网络。
```
#### 一、卸载flannel
1.1 k8s删除flannel pod
```sh
kubectl delete -f kube-flanneld.yaml
```
1.2 删除flannel网卡
```sh
$ ip link delete cni0
$ ip link delete flannel.1 #删除flannel网卡,如果是udp模式,则网卡为:flannel.0
```
1.3 查看路由
```sh
ip route
```
```sh
default via 172.16.13.254 dev ens192 proto static metric 100
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 172.16.13.81 dev ens192
10.244.2.0/24 via 172.16.13.82 dev ens192
172.16.13.0/24 dev ens192 proto kernel scope link src 172.16.13.80 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
```
1.4 删除路由
```sh
ip route delete 10.244.1.0/24 via 172.16.13.81 dev ens192
ip route delete 10.244.2.0/24 via 172.16.13.82 dev ens192
```
`注意：不要清除防火墙规则`
#### 二、安装calico
##### 2.1 下载calico安装文件
```sh
wget https://docs.projectcalico.org/manifests/calico-etcd.yaml
```
##### 2.2 修改etcd证书
获取证书和秘钥的base64编码结果
```sh
$ cat /etc/kubernetes/pki/etcd/ca.crt |base64 -w 0
$ cat /etc/kubernetes/pki/etcd/server.crt |base64 -w 0
$ cat /etc/kubernetes/pki/etcd/server.key |base64 -w 0
```
修改calico-etcd.yaml
```sh
$ vim calico-etcd.yaml
```
```yaml
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: calico-etcd-secrets
namespace: kube-system
data:
etcd-ca: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0VENDQWNtZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFTTVJBd0RnWURWUVFERXdkbGRHTmsKTFdOaE1CNFhEVEl4TURrd05EQXl..." #cat /etc/kubernetes/pki/ca.crt |base64 -w 0 输出的结果
etcd-cert: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQVENDQWlXZ0F3SUJBZ0lJUmsrNkR4Szdrb0V3RFFZSktvWklodmNOQVFFTEJRQXdFakVRTUE0R0ExVUUKQXhNSFpYUmpaQzFqWVRBZUZ3M" #cat /etc/kubernetes/pki/server.crt |base64 -w 0 输出的结果
etcd-key: "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBdFNTTHBDMUxyWjdTcTBCTmh5UjlTYi83OThXTHJxNHNoZUUzc2RKQVA2UzJpR0VxCnBtUVh" #cat /etc/kubernetes/pki/server.crt |base64 -w 0 输出的结果
---
kind: ConfigMap
apiVersion: v1
metadata:
name: calico-config
namespace: kube-system
data:
etcd_endpoints: "https://172.16.13.80:2379" #etc的地址
etcd_ca: "/calico-secrets/etcd-ca"
etcd_cert: "/calico-secrets/etcd-cert"
etcd_key: "/calico-secrets/etcd-key"
...
- name: IP_AUTODETECTION_METHOD
value: "interface=ens.*" #修改查找节点网卡名的匹配规则
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Never" #Always表示IPIP模式，修改为Never,启用BGP
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16" #修改CIDR的子网
...
```
部署calico
```sh
$ kubectl apply -f calico-etcd.yaml
```
查看calico的pod状态
```sh
$ kubectl get pods -n kube-system
```
```sh
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5499fb6db5-w4b4z 1/1 Running 0 15h
calico-node-569mh 1/1 Running 0 15h
calico-node-g6m6j 1/1 Running 0 15h
calico-node-g7p7w 1/1 Running 0 15h
coredns-7f89b7bc75-gdzxn 0/1 pending 13 4d3h
coredns-7f89b7bc75-s5shx 0/1 pending 13 4d3h
etcd-shudoon101 1/1 Running 1 4d3h
kube-apiserver-shudoon101 1/1 Running 1 4d3h
kube-controller-manager-shudoon101 1/1 Running 1 4d3h
kube-proxy-dpvzs 1/1 Running 0 4d2h
kube-proxy-svckb 1/1 Running 1 4d3h
kube-proxy-xlqvh 1/1 Running 0 4d2h
kube-scheduler-shudoon101 1/1 Running 2 4d3h
```
重建coredns(网络不通)
```sh
$ kubectl get deployment -n kube-system -o yaml >coredns.yaml
$ kubectl delete -f coredns.yaml
$ kubectl apply -f coredns.yaml
```
查看coredns网络信息
```sh
$ kubectl get pods -n kube-system -o wide
```
```sh
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-5499fb6db5-79g4k 1/1 Running 0 3m6s 172.16.13.80 shudoon101 <none> <none>
calico-node-hr46s 1/1 Running 0 90m 172.16.13.81 shudoon102 <none> <none>
calico-node-n5h78 1/1 Running 0 90m 172.16.13.82 shudoon103 <none> <none>
calico-node-vmrbq 1/1 Running 0 90m 172.16.13.80 shudoon101 <none> <none>
coredns-7f89b7bc75-c874x 1/1 Running 0 3m6s 10.244.236.192 shudoon101 <none> <none>
coredns-7f89b7bc75-ssv86 1/1 Running 0 3m6s 10.244.236.193 shudoon101 <none> <none>
etcd-shudoon101 1/1 Running 0 125m 172.16.13.80 shudoon101 <none> <none>
kube-apiserver-shudoon101 1/1 Running 0 125m 172.16.13.80 shudoon101 <none> <none>
kube-controller-manager-shudoon101 1/1 Running 0 125m 172.16.13.80 shudoon101 <none> <none>
kube-proxy-fbbkw 1/1 Running 0 124m 172.16.13.81 shudoon102 <none> <none>
kube-proxy-mrghg 1/1 Running 0 125m 172.16.13.80 shudoon101 <none> <none>
kube-proxy-t7555 1/1 Running 0 124m 172.16.13.82 shudoon103 <none> <none>
kube-scheduler-shudoon101 1/1 Running 0 125m 172.16.13.80 shudoon101 <none> <none>
```
```sh
$ ip route
```
```sh
default via 172.16.13.254 dev ens192 proto static metric 100
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 172.16.13.81 dev ens192 proto bird
10.244.2.0/24 via 172.16.13.82 dev ens192 proto bird
10.244.193.128/26 via 172.16.13.82 dev ens192 proto bird
10.244.202.0/26 via 172.16.13.81 dev ens192 proto bird
10.244.236.192 dev calif9ec1619c50 scope link
blackhole 10.244.236.192/26 proto bird
10.244.236.193 dev califa55073ed4c scope link
172.16.13.0/24 dev ens192 proto kernel scope link src 172.16.13.80 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
```
#### 三、安装calicoctl
```sh
$ wget -O /usr/local/bin/calicoctl https://github.com/projectcalico/calicoctl/releases/download/v3.11.1/calicoctl
$ chmod +x /usr/local/bin/calicoctl
```
创建配置文件
```sh
$ mkdir /etc/calico
$ vim /etc/calico/calicoctl.cfg
```
```yaml
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "etcdv3"
etcdEndpoints: "https://172.16.13.80:2379"
etcdKeyFile: "/etc/kubernetes/pki/etcd/server.key"
etcdCertFile: "/etc/kubernetes/pki/etcd/server.crt"
etcdCACertFile: "/etc/kubernetes/pki/etcd/ca.crt"
```
```sh
$ calicoctl node status
```
```sh
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 172.16.13.81 | node-to-node mesh | up | 14:33:24 | Established |
| 172.16.13.82 | node-to-node mesh | up | 14:33:25 | Established |
+--------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
```
#### 四、配置Route Reflector模式
```sh
Calico 维护的网络在默认是（Node-to-Node Mesh）全互联模式，Calico集群中的节点之间都会相互建立连接，用于路由交换。但是随着集群规模的扩大，mesh模式将形成一个巨大服务网格，连接数成倍增加。
这时就需要使用 Route Reflector（路由器反射）模式解决这个问题。
确定一个或多个Calico节点充当路由反射器，让其他节点从这个RR节点获取路由信息。
在BGP中可以通过calicoctl node status看到启动是node-to-node mesh网格的形式，这种形式是一个全互联的模式，默认的BGP在k8s的每个节点担任了一个BGP的一个喇叭，一直吆喝着扩散到其他节点，随着集群节点的数量的增加，那么上百台节点就要构建上百台链接，就是全互联的方式，都要来回建立连接来保证网络的互通性，那么增加一个节点就要成倍的增加这种链接保证网络的互通性，这样的话就会使用大量的网络消耗，所以这时就需要使用Route reflector,也就是找几个大的节点，让他们去这个大的节点建立连接，也叫RR，也就是公司的员工没有微信群的时候，找每个人沟通都很麻烦，那么建个群，里面的人都能收到，所以要找节点或着多个节点充当路由反射器，建议至少是2到3个，一个做备用，一个在维护的时候不影响其他的使用。
```
##### 4.1 关闭 node-to-node BGP网格
添加default BGP配置,调整nodeToNodeMeshEnabled和asNumber：
```sh
$ cat bgp.yaml
```
```yaml
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false #禁用node-to-node mesh
asNumber: 64512 #calicoctl get nodes --output=wide 获取
```
##### 4.2 查看bgp配置，MESHENABLED为false
```sh
$ calicoctl get bgpconfig
```
```sh
NAME LOGSEVERITY MESHENABLED ASNUMBER
default Info false 64512
```
##### 4.3 配置指定节点充当路由反射器
为方便让BGPPeer轻松选择节点，通过标签选择器匹配，也就是可以去调用k8s里面的标签进行关联，我们可以给哪个节点作为路由发射器打个标签
给路由器反射器节点打标签，我这将shudoon102打上标签
```sh
$ kubectl label node shudoon102 route-reflector=true
```
##### 4.4 配置路由器反射器节点,配置集群ID
```sh
$ calicoctl get node shudoon102 -o yaml > node.yaml
```
```yml
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"shudoon102","kubernetes.io/os":"linux","route-reflector":"true"}'
creationTimestamp: "2021-09-09T03:54:27Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: shudoon102
kubernetes.io/os: linux
route-reflector: "true"
name: shudoon102
resourceVersion: "27093"
uid: 3c1f56e8-4c35-46d7-9f46-ef20984fec41
spec:
bgp:
ipv4Address: 172.16.13.81/24
routeReflectorClusterID: 244.0.0.1 #集群ID
orchRefs:
- nodeName: shudoon102
orchestrator: k8s
```
应用配置
```sh
$ calicoctl apply -f node.yaml
```
##### 4.5 配置其他节点连接反射器
```sh
$ cat bgp1.yaml
```
```yml
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: peer-with-route-reflectors
spec:
nodeSelector: all() #所有节点
peerSelector: route-reflector == 'true'
```
```sh
$ calicoctl apply -f bgp1.yaml
Successfully applied 1 'BGPPeer' resource(s)
```
```sh
$ calicoctl get bgppeer
NAME PEERIP NODE ASN
peer-with-route-reflectors all() 0
```
```sh
$ calicoctl node status
```
```sh
Calico process is running.
IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+---------------+-------+----------+-------------+
| 172.16.13.81 | node specific | up | 08:50:34 | Established |
+--------------+---------------+-------+----------+-------------+
```
##### 4.6 添加多个路由反射器
现在进行对路由反射器添加多个，100个节点以内建议2-3个路由反射器
```sh
$ kubectl label node shudoon103 route-reflector=true
```
```sh
$ calicoctl get node shudoon103 -o yaml > node2.yaml
```
```yml
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"shudoon103","kubernetes.io/os":"linux","route-reflector":"true"}'
creationTimestamp: "2021-09-09T03:54:28Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: shudoon103
kubernetes.io/os: linux
route-reflector: "true"
name: shudoon103
resourceVersion: "29289"
uid: da510109-75bf-4e92-9074-409f1de496b9
spec:
bgp:
ipv4Address: 172.16.13.82/24
routeReflectorClusterID: 244.0.0.1 #添加集群ID
orchRefs:
- nodeName: shudoon103
orchestrator: k8s
```
应用配置
```sh
$ calicoctl apply -f node.yaml
```
查看calico节点状态
```sh
$calicoctl node status
```
```sh
Calico process is running.
IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+---------------+-------+----------+-------------+
| 172.16.13.81 | node specific | up | 08:50:34 | Established |
| 172.16.13.82 | node specific | up | 08:54:47 | Established |
+--------------+---------------+-------+----------+-------------+
```
#### 问题:
`a.如何卸载calico网络?`
```sh
$ kubectl delete -f calico-etcd.yaml
```
参考:
https://blog.51cto.com/u_14143894/2463392
