兜兜    2021-09-15 15:56:38    2021-10-27 15:14:16   

Keepalived LVS ingress
```sh 防火墙端口映射80/443到LVS的VIP对应80/443,LVS负载均衡K8S节点IP的80/443端口。ingress-nginx-controller服务暴露方式通过(HostNetwork:80/443)。实现的效果119.x.x.x:80/443-->172.16.100.99:80/443(LVS VIP)--> 172.16.100.100:80/443,172.16.100.101:80/443,172.16.100.102:80/443 ``` 配置规划 ```sh +----------------+----------------+--------+--------------------------+ | Host | IP | Port | SoftWare | +----------------+----------------+--------+--------------------------+ | LVS01 | 172.16.100.27 | 80/443 | LVS,Keepalived | | LVS02 | 172.16.100.28 | 80/443 | LVS,Keepalived | | RS/k8s-master1 | 172.16.100.100 | 80/443 | ingress-nginx-controller | | RS/k8s-master2 | 172.16.100.101 | 80/443 | ingress-nginx-controller | | RS/k8s-node1 | 172.16.100.102 | 80/443 | ingress-nginx-controller | | VIP | 172.16.100.99 | 80/443 | / | +----------------+----------------+--------+--------------------------+ ``` 安装lvs和keepalived(`172.16.100.27/172.16.100.28`) ```sh $ yum install ipvsadm keepalived -y $ systemctl enable keepavlied ``` 配置keepalived(`172.16.100.27`) ```sh $ cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id LVS_27 #route id } vrrp_instance VI_1 { state MASTER #主节点 interface ens192 #网卡 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.16.100.99 } } virtual_server 172.16.100.99 443 { delay_loop 3 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 172.16.100.100 443 { weight 1 TCP_CHECK { connect_port 443 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.100.101 443 { weight 1 TCP_CHECK { connect_port 443 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.100.102 443 { weight 1 TCP_CHECK { connect_port 443 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } virtual_server 172.16.100.99 80 { delay_loop 3 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 172.16.100.100 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.100.101 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 172.16.100.102 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } EOF ``` 配置keepalived(`172.16.100.28`) ```sh $ cat /etc/keepalived/keepalived.conf ... global_defs { router_id LVS_28 #route id,两台机器配置不一样 } vrrp_instance VI_1 { state BACKUP #备份节点 interface ens192 #网卡名 virtual_router_id 51 priority 99 #优先级 ... ``` 配置RS节点(`172.16.100.100/172.16.100.101/172.16.100.102`) ```sh $ cat >/etc/init.d/lvs_rs.sh <<EOF vip=172.16.100.99 mask='255.255.255.255' dev=lo:1 case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $dev $vip netmask $mask #broadcast $vip up echo "The RS Server is Ready!" ;; stop) ifconfig $dev down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce echo "The RS Server is Canceled!" ;; *) echo "Usage: $(basename $0) start|stop" exit 1 ;; esac EOF ``` 启动脚本 ```sh $ chmod +x /etc/init.d/lvs_rs.sh $ /etc/init.d/lvs_rs.sh start ``` ```sh $ ip a #查看lo:1接口VIP是否绑定 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 172.16.100.99/32 scope global lo:1 valid_lft forever preferred_lft forever ``` 启动keepalived(`172.16.100.27/172.16.100.28`) ```sh $ systemctl start keepavlied ``` 查看VIP是否绑定 ```sh $ ip a 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:50:56:93:ed:a4 brd ff:ff:ff:ff:ff:ff inet 172.16.100.27/24 brd 172.16.100.255 scope global noprefixroute ens192 valid_lft forever preferred_lft forever inet 172.16.100.99/32 scope global ens192 valid_lft forever preferred_lft forever ``` 查看LVS信息 ```sh $ ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.100.99:80 rr persistent 50 -> 172.16.100.100:80 Route 1 0 0 -> 172.16.100.101:80 Route 1 0 0 -> 172.16.100.102:80 Route 1 0 0 TCP 172.16.100.99:443 rr persistent 50 -> 172.16.100.100:443 Route 1 0 0 -> 172.16.100.101:443 Route 1 0 0 -> 172.16.100.102:443 Route 1 0 0 ```
阅读 128 评论 0 收藏 0
阅读 128
评论 0
收藏 0

兜兜    2018-07-10 11:41:52    2019-11-14 14:34:32   

Keepalived 高可用 LVS 负载均衡
### LVS+NAT **`通过网络地址转换,调度器重写请求报文的目标地址,根据预设的调度算法,将请求分派给后端的RS;RS的响应报文通过调度器时,报文的源地址被重写,再返回给客户,完成整个负载调度过程。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 Real Server: 192.168.50.251/192.168.50.252 网络模式:NAT ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:80 -s rr ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.251:80 -m ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.252:80 -m ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # rr 轮训调度 # -m 网络地址转换NAT ``` #### **RS配置** 安装web ```bash yum install nginx -y ``` 修改网关 ```bash vim /etc/sysconfig/network-scripts/ifcfg-enp0s3 ``` ```ini GATEWAY0=192.168.50.253 ``` #### 测试(外网机器) `注意:外网测试,同网段直接访问192.168.50.253,LVS仅修改目的地址成RS,当RS应答给客户端,发现为同网段,不会经过LVS去做SNAT,客户端发送的DEST地址和收到的应答包的SRC地址不一致丢弃` 路由器上面做个端口映射 113.119.xx.xx:8999->192.168.50.253:80 ```bash curl http://113.119.xx.xx:8999/ ``` &emsp; ### LVS+DR(`无VIP`) **`VS/DR通过改写请求报文的MAC地址,将请求发送到RS,而RS将响应直接返回给客户。同VS/TUN技术一样,VS/DR技术可极大地 提高集群系统的伸缩性。这种方法没有IP隧道的开销,对集群中的RS也没有必须支持IP隧道协议的要求,但是要求调度器与RS都有一块网卡连 在同一物理网段上。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` 配置RS MAC静态绑定 `因为负载均衡服务器使用的是真实IP 192.168.50.253,当查询ARP,因为RS回环接口配置192.168.50.253,所以不会应答ARP,相反,如果配置VIP,那么回环地址配置VIP,192.168.50.253请求ARP的时候,RS会应答ARP` ```bash arp -s 192.168.50.251 08:00:27:8a:58:c1 arp -s 192.168.50.252 08:00:27:29:31:d8 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 逻辑网卡添加ip地址192.168.50.253 ```bash ifconfig lo:0 192.168.50.253 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253,出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.253 dev lo:0 ``` &emsp; ### LVS+DR(`有VIP`) #### 准备环境 ```bash VIP: 192.168.50.240 LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 配置VIP ```bash ifconfig eth0:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.240:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 添加VIP ```bash ifconfig lo:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253,出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.240 dev lo:0 ``` &emsp; ### LVS+NAT **`通过网络地址转换,调度器重写请求报文的目标地址,根据预设的调度算法,将请求分派给后端的RS;RS的响应报文通过调度器时,报文的源地址被重写,再返回给客户,完成整个负载调度过程。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 Real Server: 192.168.50.251/192.168.50.252 网络模式:NAT ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:80 -s rr ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.251:80 -m ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.252:80 -m ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # rr 轮训调度 # -m 网络地址转换NAT ``` #### **RS配置** 安装web ```bash yum install nginx -y ``` 修改网关 ```bash vim /etc/sysconfig/network-scripts/ifcfg-enp0s3 ``` ```ini GATEWAY0=192.168.50.253 ``` #### 测试(外网机器) `注意:外网测试,同网段直接访问192.168.50.253,LVS仅修改目的地址成RS,当RS应答给客户端,发现为同网段,不会经过LVS去做SNAT,客户端发送的DEST地址和收到的应答包的SRC地址不一致丢弃` 路由器上面做个端口映射 113.119.xx.xx:8999->192.168.50.253:80 ```bash curl http://113.119.xx.xx:8999/ ``` &emsp; ### LVS+DR(`无VIP`) **`VS/DR通过改写请求报文的MAC地址,将请求发送到RS,而RS将响应直接返回给客户。同VS/TUN技术一样,VS/DR技术可极大地 提高集群系统的伸缩性。这种方法没有IP隧道的开销,对集群中的RS也没有必须支持IP隧道协议的要求,但是要求调度器与RS都有一块网卡连 在同一物理网段上。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` 配置RS MAC静态绑定 `因为负载均衡服务器使用的是真实IP 192.168.50.253,当查询ARP,因为RS回环接口配置192.168.50.253,所以不会应答ARP,相反,如果配置VIP,那么回环地址配置VIP,192.168.50.253请求ARP的时候,RS会应答ARP` ```bash arp -s 192.168.50.251 08:00:27:8a:58:c1 arp -s 192.168.50.252 08:00:27:29:31:d8 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 逻辑网卡添加ip地址192.168.50.253 ```bash ifconfig lo:0 192.168.50.253 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253,出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.253 dev lo:0 ``` &emsp; ### LVS+DR(`有VIP`) #### 准备环境 ```bash VIP: 192.168.50.240 LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 配置VIP ```bash ifconfig eth0:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.240:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 添加VIP ```bash ifconfig lo:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253,出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.240 dev lo:0 ``` &emsp; ### LVS+TUNNEL(内网跨网段) #### 准备环境 ```bash Client: 192.168.10.3 路由器(LINUX) 192.168.10.4 192.168.20.4 VIP: 192.168.10.100 LVS主机: 192.168.10.5 Real Server: 192.168.20.3 网络模式:TUNNEL ``` &emsp; #### **DR配置** 配置VIP ```bash ifconfig eth1:1 192.168.10.100 netmask 255.255.255.255 broadcast 192.168.10.100 up route add -host 192.168.10.100 dev eth1:1 ``` 修改内核参数 ```bash echo "0" >/proc/sys/net/ipv4/ip_forward echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects echo "1" >/proc/sys/net/ipv4/conf/eth1/send_redirects ``` 配置lvs ```bash ipvsadm -C ipvsadm -A -t 192.168.10.100:8080 -s rr ipvsadm -a -t 192.168.10.100:8080 -r 192.168.20.3 -i ``` &emsp; #### **RS配置** 配置tunnel隧道 ```bash modprobe ipip ifconfig tunl0 192.168.10.100 netmask 255.255.255.255 broadcast 192.168.10.100 up route add -host 192.168.10.100 dev tunl0 ``` 修改内核参数 ```bash echo 0 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/tunl0/arp_announce echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter ``` 启动服务 ```bash echo "RS SERVER" >index.html python -m SimpleHTTPServer 8080 ``` &emsp; #### 客户端测试 ```bash curl http://192.168.10.100:8080 ``` ``` RS SERVER ``` `测试成功` &emsp; #### RS主机抓包 监听隧道接口 ```bash tcpdump -i tunl0 -nnn -vvv ``` ```bash 08:32:34.269392 IP (tos 0x0, ttl 64, id 51443, offset 0, flags [DF], proto TCP (6), length 60) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [S], cksum 0xdbc2 (correct), seq 2270636359, win 28200, options [mss 1410,sackOK,TS val 1958324 ecr 0,nop,wscale 7], length 0 08:32:34.271186 IP (tos 0x0, ttl 64, id 51444, offset 0, flags [DF], proto TCP (6), length 52) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [.], cksum 0xa968 (correct), seq 2270636360, ack 1942327447, win 221, options [nop,nop,TS val 1958326 ecr 1821109], length 0 08:32:34.271345 IP (tos 0x0, ttl 64, id 51445, offset 0, flags [DF], proto TCP (6), length 135) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [P.], cksum 0xa230 (correct), seq 0:83, ack 1, win 221, options [nop,nop,TS val 1958327 ecr 1821109], length 83: HTTP, length: 83 GET / HTTP/1.1 User-Agent: curl/7.29.0 Host: 192.168.10.100:8080 Accept: */* ``` `通过上面的抓包日志可以分析DR已经将SYN包发送到RS主机` 监听eth1接口8080端口的数据包 ```bash tcpdump -i eth1 and port 8080 -nnn -vvv ``` ```bash 08:31:57.116678 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [S.], cksum 0x95e6 (incorrect -> 0x6552), seq 2019760809, ack 3127325422, win 27960, options [mss 1410,sackOK,TS val 1783956 ecr 1921172,nop,wscale 7], length 0 08:31:57.118567 IP (tos 0x0, ttl 64, id 8764, offset 0, flags [DF], proto TCP (6), length 52) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [.], cksum 0x95de (incorrect -> 0xfff2), seq 1, ack 84, win 219, options [nop,nop,TS val 1783958 ecr 1921174], length 0 08:31:57.118811 IP (tos 0x0, ttl 64, id 8765, offset 0, flags [DF], proto TCP (6), length 69) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [P.], cksum 0x95ef (incorrect -> 0x4015), seq 1:18, ack 84, win 219, options [nop,nop,TS val 1783958 ecr 1921174], length 17: HTTP, length: 17 HTTP/1.0 200 OK ``` `通过上面的抓包日志可以发现,RS也将应答包SYN+ACK直接发送给客户端` **`总结:LVS tunnel 跨网段转发是成功的。`** &emsp; ### LVS+TUNNEL(公网) `Ip Tunnel模式最大的优点就在于它可以跨网段转发,没有DR和NAT模式的组网限制。这在部署上带来的很大的灵活性,甚至还可以跨机房转发,不过不建议这样使用,一是会带来跨机房间的流量,提高了成本;二是跨机房转发必然会要在RS机房上绑定LVS机房的VIP,这有可能会被运营商的防火墙认为是IP伪造请求而拦截` #### 准备环境(Vultr VPS) ```bash Client: 183.54.238.66 VIP: 104.238.150.254 LVS主机: 167.179.115.37 Real Server: 202.182.125.31 139.180.202.67 网络模式:TUNNEL ``` &emsp; #### **DR配置** DR主机所在的VPS申请添加一个公网ip,配置到eth0:1接口上 ```bash ifconfig eth0:1 104.238.150.254 netmask 255.255.255.255 broadcast 104.238.150.254 up route add -host 104.238.150.254 dev eth0:1 ``` 修改内核参数 ```bash echo "0" >/proc/sys/net/ipv4/ip_forward echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects ``` 配置lvs ```bash ipvsadm -C ipvsadm -A -t 104.238.150.254:8080 -s rr ipvsadm -a -t 104.238.150.254:8080 -r 202.182.125.31 -i ipvsadm -a -t 104.238.150.254:8080 -r 139.180.202.67 -i ``` &emsp; #### **RS配置** 配置tunnel隧道 ```bash modprobe ipip ifconfig tunl0 104.238.150.254 netmask 255.255.255.255 broadcast 104.238.150.254 up route add -host 104.238.150.254 dev tunl0 ``` 修改内核参数 ```bash echo 0 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/tunl0/arp_announce echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter ``` 启动服务 ```bash python -m SimpleHTTPServer 8080 ``` &emsp; #### 客户端测试 浏览器访问 `http://104.238.150.254:8080` `浏览器显示访问不了!!!` &emsp; #### RS主机抓包 监听隧道接口 ```bash tcpdump -i tunl0 -nnn -vvv ``` ```bash tcpdump: listening on tunl0, link-type RAW (Raw IP), capture size 262144 bytes 09:20:12.713031 IP (tos 0x0, ttl 116, id 29938, offset 0, flags [DF], proto TCP (6), length 48) 183.54.238.66.63376 > 104.238.150.254.8080: Flags [S], cksum 0x5a81 (correct), seq 3523639844, win 8192, options [mss 1440,nop,nop,sackOK], length 0 09:20:54.742880 IP (tos 0x0, ttl 116, id 30339, offset 0, flags [DF], proto TCP (6), length 52) 183.54.238.66.63400 > 104.238.150.254.8080: Flags [S], cksum 0xb006 (correct), seq 2103403813, win 8192, options [mss 1440,nop,wscale 2,nop,nop,sackOK], length 0 09:20:54.993380 IP (tos 0x0, ttl 116, id 30342, offset 0, flags [DF], proto TCP (6), length 52) 183.54.238.66.63402 > 104.238.150.254.8080: Flags [S], cksum 0x8306 (correct), seq 484897436, win 8192, options [mss 1440,nop,wscale 2,nop,nop,sackOK], length 0 09:20:57.744463 IP (tos 0x0, ttl 116, id 30376, offset 0, flags [DF], proto TCP (6), length 52) ``` `通过上面的抓包日志可以分析DR已经将SYN包发送到RS主机` 监听客户端访问服务器8080端口的数据包 ```bash tcpdump -i eth0 host 183.54.238.66 and port 8080 -vvv -nnn ``` ```bash tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 09:19:13.659753 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:14.861060 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:16.660745 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:19.061063 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) ``` `通过上面的抓包日志可以发现,RS也将应答包SYN+ACK直接发送给客户端` **`总结:LVS tunnel DR转发,RS应答是正常的。RS应答的时候被Vultr的防火墙认为是IP伪造请求而拦截,所以导致实验失败!`** &emsp; ### LVS+FULLNAT `LVS 当前应用主要采用 DR 和 NAT 模式,但这 2 种模式要求 RealServer 和 LVS 在同一个 vlan中,导致部署成本过高;TUNNEL 模式虽然可以跨 vlan,但 RealServer上需要部署 ipip 模块等,网络拓扑上需要连通外网,较复杂,不易运维。` `为了解决上述问题,我们在 LVS 上添加了一种新的转发模式:FULLNAT,该 模式和 NAT 模式的区别是:Packet IN 时,除了做 DNAT,还做 SNAT(用户 ip->内 网 ip),从而实现 LVS-RealServer 间可以跨 vlan 通讯,RealServer 只需要连接到内 网;` LVS FULLNAT 实战:https://www.haxi.cc/archives/LVS-FULLNAT实战.html 相关链接: LVS-ospf集群:http://noops.me/?p=974
阅读 2108 评论 0 收藏 0
阅读 2108
评论 0
收藏 0