YNOTES笔记

![](https://files.ynotes.cn/18-7-23/70377481.jpg) #### **项目目录结构** ```bash blog |-- docker-compose.yml #docker-compose编排文件 |-- mysql | |-- conf | | `-- mysqld.cnf #mysql配置文件 | |-- data #mysql数据存放目录 | |-- db_init_sql | | `-- blog.sql #blog的sql数据表结构 | `-- log #mysql日志目录 |-- nginx | |-- conf | | |-- mysite.template #生成blog.itisme.co.conf配置的样例文件 | | `-- nginx.conf #nginx配置文件 | |-- log #nginx日志目录 | | | | | `-- ssl | |-- fullchain.pem #ssl证书链 | `-- privkey.pem #ssl证书私钥 `-- uwsgi-django |-- build | |-- Dockerfile #构建uwsgi-django镜像的文件 | `-- requirements.txt #django项目需要的安装包 |-- conf | `-- config.ini #uwsgi启动配置参数 `-- my_project #django项目 |-- blog #blog应用 |-- manage.py `-- my_project ``` #### **新建docker项目数据配置存放目录** ```bash $ mkdir /data/docker_project/blog -p $ cd /data/docker_project/blog ``` #### **创建mysql容器使用的目录** ```bash $ mkdir mysql/{conf,data,db_init_sql,log} -p $ chmod 777 mysql/log ``` conf：存放mysql配置文件 data：存放mysql数据的目录 db_init_sql：存放的是mysql容器初始化的sql(存放django项目的建表语句) log：存放mysql日志,修改权限为777 　 #### **编辑mysql配置文件mysql/conf/mysqld.cnf（项目使用emoji表情，编码使用的是utf8mb4）** ```bash [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid default-time-zone = '+08:00' character-set-server=utf8 character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci character-set-client-handshake = FALSE innodb_buffer_pool_size = 128M sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES [client] default-character-set = utf8mb4 [mysql] default-character-set = utf8mb4 ``` #### **创建uwsgi-django容器使用的目录** ```bash $ mkdir uwsgi-django/{build,my_project,conf} -p ``` my_project：存放django项目 build:存放构建镜像uwsgi-django需要的文件 conf：存放uwsgi启动的配置文件 #### **创建uwsgi-django的Dockerfile文件(用于构建uwsgi-django镜像)** ```bash $ vim uwsgi-django/build/Dockerfile ``` ``` FROM python:2.7-slim RUN apt-get update && apt-get install -y \ gcc \ gettext \ mysql-client default-libmysqlclient-dev \ libpq-dev \ --no-install-recommends && rm -rf /var/lib/apt/lists/* ENV DJANGO_VERSION 1.9.5 RUN pip install mysqlclient psycopg2 uwsgi django=="$DJANGO_VERSION" WORKDIR /usr/src/app COPY requirements.txt ./ RUN pip install -r requirements.txt WORKDIR /usr/src/app/my_project ``` #### **项目的requirements.txt上传到django目录(项目所在的环境执行)** ```bash $ pip freeze >/root/requirements.txt ``` #### **拷贝requirements.txt到docker-compose编排所在机器uwsgi-django目录** ```bash $ scp /root/requirements.txt root@docker-host:/data/docker_project/blog/uwsgi-django/build ``` ### **增加uwsgi程序的配置文件uwsgi-django/conf/config.ini** ``` [uwsgi] socket = 0.0.0.0:8888 chdir = /usr/src/app/my_project module = my_project.wsgi master = true processes = 10 socket = /tmp/my_project.sock vacuum = true uid = 498 ``` #### **创建nginx容器使用的目录** ```bash $ mkdir nginx/{conf,ssl,log} $ chmod 777 nginx/log ``` conf：存放nginx的配置文件 ssl:存放SSL证书目录 log：存放日志目录 #### **编辑nginx/conf/nginx.conf** ```nginx user nginx; worker_processes 2; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { use epoll; worker_connections 10240; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; include /etc/nginx/conf.d/blog.itisme.co.conf; } ``` #### **编辑nginx/conf/mysite.template(生成blog.itisme.co.conf的模板文件，通过envsubst替换环境变量)** ```nginx upstream uwsgi-django { server uwsgi-django:$UWSGI_PORT; # for a web port socket (we'll use this first) } server { listen $NGINX_PORT; server_name $NGINX_HOST; charset utf-8; rewrite ^(.*)$ https://${server_name}$1 permanent; } server { listen $NGINX_SSL_PORT ssl http2 default_server; listen [::]:443 ssl http2 default_server; server_name $NGINX_HOST; #证书配置 ssl_certificate "/etc/nginx/ssl/blog.itisme.co/fullchain.pem"; ssl_certificate_key "/etc/nginx/ssl/blog.itisme.co/privkey.pem"; ssl_session_cache shared:SSL:1m; ssl_session_timeout 10m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE; ssl_prefer_server_ciphers on; # max upload size client_max_body_size 75M; # adjust to taste #django项目的上传文件的目录 location /upload { alias /data/app/my_project/my_project/upload; # your Django project's media files - amend as required } #django项目的静态文件目录 location /static { alias /data/app/my_project/my_project/static_all; } #django项目uwsgi配置 location / { uwsgi_pass uwsgi-django; include /data/app/my_project/my_project/uwsgi_params; # the uwsgi_params file you installed } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } ``` #### **拷贝证书到nginx/ssl目录** ```bash $ scp fullchain.pem root@docker-host:/data/docker_project/blog/nginx/ssl $ scp privkey.pem root@docker-host:/data/docker_project/blog/nginx/ssl ``` #### **编辑docker-compose.yml (客户端->nginx->uwsgi->django->db)** ```bash $ vim docker-compose.yml ``` ```yaml version: '3' services: db: image: mysql:5.7 restart: always container_name: blog-db environment: MYSQL_ROOT_PASSWORD: 123456 MYSQL_DATABASE: blog MYSQL_USER: blog MYSQL_PASSWORD: 123456 volumes: #挂载mysql配置文件 - ./mysql/conf/mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf #挂载数据库初始化脚本 - ./mysql/db_init_sql:/docker-entrypoint-initdb.d #挂载mysql数据目录 - ./mysql/data:/var/lib/mysql #挂载mysql日志目录 - ./mysql/log:/var/log uwsgi-django: #使用./uwsgi-django/build/目录下的Dockerfile构建镜像 build: ./uwsgi-django/build/ #当存在build时，image参数表示的是构建后的镜像名 image: uwsgi-django:1.9.5 restart: always depends_on: - db container_name: blog-uwsgi-django environment: DB_NAME: blog DB_USER: blog DB_PASS: 123456 DB_PORT: 3306 WEB_URL: blog.itisme.co volumes: #挂载django项目 - ./uwsgi-django/my_project:/usr/src/app/my_project #挂载uwsgi配置文件 - ./uwsgi-django/conf:/usr/src/app/uwsgi/conf command: uwsgi /usr/src/app/uwsgi/conf/config.ini nginx: #使用nginx官方稳定版镜像 image: nginx:stable restart: always depends_on: - uwsgi-django container_name: blog-nginx environment: NGINX_HOST: blog.itisme.co NGINX_PORT: 80 NGINX_SSL_PORT: 443 UWSGI_PORT: 8888 ports: - 80:80 - 443:443 volumes: #挂载nginx配置 - ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf #挂载站点模板文件 - ./nginx/conf/mysite.template:/etc/nginx/conf.d/mysite.template #挂载ssl证书 - ./nginx/ssl/fullchain.pem:/etc/nginx/ssl/blog.itisme.co/fullchain.pem - ./nginx/ssl/privkey.pem:/etc/nginx/ssl/blog.itisme.co/privkey.pem #挂载项目静态文件 - ./uwsgi-django/my_project/my_project/upload:/data/app/my_project/my_project/upload - ./uwsgi-django/my_project/my_project/static_all:/data/app/my_project/my_project/static_all #挂载uwsgi参数文件 - ./uwsgi-django/my_project/my_project/uwsgi_params:/data/app/my_project/my_project/uwsgi_params - ./nginx/log/:/var/log/nginx/ #envsubst替换/etc/nginx/conf.d/mysite.template变量 command: /bin/bash -c "envsubst < /etc/nginx/conf.d/mysite.template > /etc/nginx/conf.d/blog.itisme.co.conf && nginx -g 'daemon off;'" ``` #### **修改django项目的mysql配置(修改项目里的mysql配置成上面的environment中指定的环境变量)** ```bash import os _env = os.environ #django指定可以访问的域 DOMAIN = _env['WEB_URL'] DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': _env['DB_NAME'], 'HOST': 'db', 'PORT': _env['DB_PORT'], 'USER': _env['DB_USER'], 'PASSWORD': _env['DB_PASS'], 'OPTIONS': {'charset':'utf8mb4'}, } } ``` #### **启动项目** ```bash $ docker-compose up ``` #### **启动项目后台运行** ```bash $ docker-compose up -d ``` #### **查看docker进程** ```bash $ docker-compose ps ``` ``` Name Command State Ports ---------------------------------------------------------------------- blog-db docker-entrypoint.sh mysqld Up 3306/tcp blog-nginx /bin/bash -c envsubst < /e ... Up 0.0.0.0:443->443/tcp, 0.0.0.0:8880->80/tcp blog-uwsgi-django uwsgi /usr/src/app/uwsgi/c ... Up ``` #### **浏览器访问https://ynotes.cn/blog/**

![](https://files.ynotes.cn/18-7-23/70377481.jpg) #### **项目目录结构** ```bash test/ |-- docker-compose.yml |-- env | |-- db.env #db容器环境变量 | |-- project.env #toomcat容器环境变量 | `-- tomcat.env #tomcat容器环境变量 |-- mysql | |-- conf | | `-- mysqld.cnf #mysql配置文件 | |-- data #mysql数据目录 | |-- db_init_sql | | `-- competitionShare.sql #tomcat项目的数据库表结构 | `-- log #mysql日志文件 `-- tomcat |-- log #tomcat日志 `-- webapps #tomcat项目存放目录 `-- tomcat_project ``` #### **新建docker项目数据配置存放目录** ```bash $ mkdir /data/docker_project/test -p $ cd /data/docker_project/test ``` #### **创建mysql容器使用的目录** ```bash $ mkdir mysql/{conf,data,db_init_sql,log} -p $ chmod 777 mysql/log ``` conf：存放mysql配置文件 data：存放mysql数据的目录 db_init_sql：存放的是mysql容器初始化的sql(如建表语句) log：存放mysql日志　 #### **编辑mysql配置文件mysql/conf/mysqld.cnf** ```bash [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock symbolic-links=0 log-error=/var/log/mysqld.log collation-server=utf8_general_ci pid-file=/var/run/mysqld/mysqld.pid default-time-zone = '+08:00' character-set-server=utf8 innodb_buffer_pool_size = 1024M sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES [client] default-character-set=utf8 [mysql] default-character-set=utf8 ``` #### **创建tomcat容器使用的目录** ```bash $ mkdir tomcat/{log,webapps} -p ``` log：存放tomcat日志 webapps：存放tomcat项目的目录 #### **编辑docker-compose.yml** ```yaml version: '3' services: db: image: mysql:5.7 restart: always environment: MYSQL_ROOT_PASSWORD: 123456 MYSQL_DATABASE: db_name MYSQL_USER: test MYSQL_PASSWORD: 123456 ports: - 3306:3306 volumes: - ./mysql/conf/mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf - ./mysql/db_init_sql:/docker-entrypoint-initdb.d - ./mysql/data:/var/lib/mysql - ./mysql/log:/var/log tomcat: image: tomcat:8.0.53-jre8 restart: always depends_on: - db container_name: test-tomcat env_file: - ./env/project.env environment: JDBC_URL: jdbc:mysql://db:3306/db_name??characterEncoding=UTF-8 JDBC_USER: test JDBC_PASS: 123456 ports: - 8787:8080 volumes: - ./tomcat/webapps:/usr/local/tomcat/webapps - ./tomcat/log:/log ``` #### **修改tomcat项目的mysql配置(修改项目里的mysql配置成上面的environment中指定的环境变量)** ```bash jdbc_url=${JDBC_URL} jdbc_username=${JDBC_USER} jdbc_password=${JDBC_PASS} ``` #### **增加环境变量配置文件project.env（该配置为tomcat启动参数）** ```bash JAVA_OPTS="-Dsupplements.host=supplements" CATALINA_OPTS=-server -Xms256M -Xmx1024M -XX:MaxNewSize=256m -XX:PermSize=64M -XX:MaxPermSize=256m ``` #### **启动项目** ```bash $ docker-compose up ``` #### **启动项目后台运行** ```bash $ docker-compose up -d ```

### LVS+NAT **`通过网络地址转换，调度器重写请求报文的目标地址，根据预设的调度算法，将请求分派给后端的RS；RS的响应报文通过调度器时，报文的源地址被重写，再返回给客户，完成整个负载调度过程。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 Real Server: 192.168.50.251/192.168.50.252 网络模式:NAT ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:80 -s rr ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.251:80 -m ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.252:80 -m ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # rr 轮训调度 # -m 网络地址转换NAT ``` #### **RS配置** 安装web ```bash yum install nginx -y ``` 修改网关 ```bash vim /etc/sysconfig/network-scripts/ifcfg-enp0s3 ``` ```ini GATEWAY0=192.168.50.253 ``` #### 测试(外网机器) `注意:外网测试,同网段直接访问192.168.50.253，LVS仅修改目的地址成RS，当RS应答给客户端，发现为同网段，不会经过LVS去做SNAT，客户端发送的DEST地址和收到的应答包的SRC地址不一致丢弃` 路由器上面做个端口映射 113.119.xx.xx:8999->192.168.50.253:80 ```bash curl http://113.119.xx.xx:8999/ ``` &emsp; ### LVS+DR(`无VIP`) **`VS/DR通过改写请求报文的MAC地址，将请求发送到RS，而RS将响应直接返回给客户。同VS/TUN技术一样，VS/DR技术可极大地提高集群系统的伸缩性。这种方法没有IP隧道的开销，对集群中的RS也没有必须支持IP隧道协议的要求，但是要求调度器与RS都有一块网卡连在同一物理网段上。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` 配置RS MAC静态绑定 `因为负载均衡服务器使用的是真实IP 192.168.50.253，当查询ARP，因为RS回环接口配置192.168.50.253，所以不会应答ARP，相反，如果配置VIP，那么回环地址配置VIP，192.168.50.253请求ARP的时候，RS会应答ARP` ```bash arp -s 192.168.50.251 08:00:27:8a:58:c1 arp -s 192.168.50.252 08:00:27:29:31:d8 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 逻辑网卡添加ip地址192.168.50.253 ```bash ifconfig lo:0 192.168.50.253 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253，出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.253 dev lo:0 ``` &emsp; ### LVS+DR(`有VIP`) #### 准备环境 ```bash VIP: 192.168.50.240 LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 配置VIP ```bash ifconfig eth0:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.240:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 添加VIP ```bash ifconfig lo:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253，出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.240 dev lo:0 ``` &emsp; ### LVS+NAT **`通过网络地址转换，调度器重写请求报文的目标地址，根据预设的调度算法，将请求分派给后端的RS；RS的响应报文通过调度器时，报文的源地址被重写，再返回给客户，完成整个负载调度过程。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 Real Server: 192.168.50.251/192.168.50.252 网络模式:NAT ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:80 -s rr ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.251:80 -m ipvsadm -a -t 192.168.50.253:80 -r 192.168.50.252:80 -m ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # rr 轮训调度 # -m 网络地址转换NAT ``` #### **RS配置** 安装web ```bash yum install nginx -y ``` 修改网关 ```bash vim /etc/sysconfig/network-scripts/ifcfg-enp0s3 ``` ```ini GATEWAY0=192.168.50.253 ``` #### 测试(外网机器) `注意:外网测试,同网段直接访问192.168.50.253，LVS仅修改目的地址成RS，当RS应答给客户端，发现为同网段，不会经过LVS去做SNAT，客户端发送的DEST地址和收到的应答包的SRC地址不一致丢弃` 路由器上面做个端口映射 113.119.xx.xx:8999->192.168.50.253:80 ```bash curl http://113.119.xx.xx:8999/ ``` &emsp; ### LVS+DR(`无VIP`) **`VS/DR通过改写请求报文的MAC地址，将请求发送到RS，而RS将响应直接返回给客户。同VS/TUN技术一样，VS/DR技术可极大地提高集群系统的伸缩性。这种方法没有IP隧道的开销，对集群中的RS也没有必须支持IP隧道协议的要求，但是要求调度器与RS都有一块网卡连在同一物理网段上。`** #### 准备环境 ```bash LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.253:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.253:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` 配置RS MAC静态绑定 `因为负载均衡服务器使用的是真实IP 192.168.50.253，当查询ARP，因为RS回环接口配置192.168.50.253，所以不会应答ARP，相反，如果配置VIP，那么回环地址配置VIP，192.168.50.253请求ARP的时候，RS会应答ARP` ```bash arp -s 192.168.50.251 08:00:27:8a:58:c1 arp -s 192.168.50.252 08:00:27:29:31:d8 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 逻辑网卡添加ip地址192.168.50.253 ```bash ifconfig lo:0 192.168.50.253 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253，出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.253 dev lo:0 ``` &emsp; ### LVS+DR(`有VIP`) #### 准备环境 ```bash VIP: 192.168.50.240 LVS主机: 192.168.50.253 08:00:27:e6:f4:0a Real Server: 192.168.50.251 08:00:27:8a:58:c1 192.168.50.252 08:00:27:29:31:d8 网络模式:DR ``` #### **DR配置** #### 安装ipvsadm ```bash yum install ipvsadm -y ``` #### 设置ipv4转发 ```bash sysctl -w net.ipv4.ip_forward=1 ``` #### 关selinux,firewall,iptables ```bash setenforce 0 systemctl stop firewall iptables -F ``` #### 配置VIP ```bash ifconfig eth0:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` #### 设置ipvsadm ```bash ipvsadm -A -t 192.168.50.240:8080 -s rr #虚拟服务端口需要和真实服务端口一致 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.251:8080 -g -w 1 ipvsadm -a -t 192.168.50.240:8080 -r 192.168.50.252:8080 -g -w 1 ipvsadm -S # -A 添加虚拟服务 # -a 添加一个真是的主机到虚拟服务 # -S 保存 # -s 选择调度方法 # -g DR模式 # rr 轮训调度 ``` #### **RS配置** 启动web服务 ```bash python -m SimpleHTTPServer 8080 ``` 修改内核参数 ```bash echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce ``` 添加VIP ```bash ifconfig lo:0 192.168.50.240 netmask 255.255.255.255 broadcast 192.168.50.255 ``` 添加路由(确保请求的IP是192.168.50.253，出去的数据包也为192.168.50.253) ```bash route add -host 192.168.50.240 dev lo:0 ``` &emsp; ### LVS+TUNNEL(内网跨网段) #### 准备环境 ```bash Client: 192.168.10.3 路由器(LINUX) 192.168.10.4 192.168.20.4 VIP: 192.168.10.100 LVS主机: 192.168.10.5 Real Server: 192.168.20.3 网络模式:TUNNEL ``` &emsp; #### **DR配置** 配置VIP ```bash ifconfig eth1:1 192.168.10.100 netmask 255.255.255.255 broadcast 192.168.10.100 up route add -host 192.168.10.100 dev eth1:1 ``` 修改内核参数 ```bash echo "0" >/proc/sys/net/ipv4/ip_forward echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects echo "1" >/proc/sys/net/ipv4/conf/eth1/send_redirects ``` 配置lvs ```bash ipvsadm -C ipvsadm -A -t 192.168.10.100:8080 -s rr ipvsadm -a -t 192.168.10.100:8080 -r 192.168.20.3 -i ``` &emsp; #### **RS配置** 配置tunnel隧道 ```bash modprobe ipip ifconfig tunl0 192.168.10.100 netmask 255.255.255.255 broadcast 192.168.10.100 up route add -host 192.168.10.100 dev tunl0 ``` 修改内核参数 ```bash echo 0 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/tunl0/arp_announce echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter ``` 启动服务 ```bash echo "RS SERVER" >index.html python -m SimpleHTTPServer 8080 ``` &emsp; #### 客户端测试 ```bash curl http://192.168.10.100:8080 ``` ``` RS SERVER ``` `测试成功` &emsp; #### RS主机抓包监听隧道接口 ```bash tcpdump -i tunl0 -nnn -vvv ``` ```bash 08:32:34.269392 IP (tos 0x0, ttl 64, id 51443, offset 0, flags [DF], proto TCP (6), length 60) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [S], cksum 0xdbc2 (correct), seq 2270636359, win 28200, options [mss 1410,sackOK,TS val 1958324 ecr 0,nop,wscale 7], length 0 08:32:34.271186 IP (tos 0x0, ttl 64, id 51444, offset 0, flags [DF], proto TCP (6), length 52) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [.], cksum 0xa968 (correct), seq 2270636360, ack 1942327447, win 221, options [nop,nop,TS val 1958326 ecr 1821109], length 0 08:32:34.271345 IP (tos 0x0, ttl 64, id 51445, offset 0, flags [DF], proto TCP (6), length 135) 192.168.10.3.46734 > 192.168.10.100.8080: Flags [P.], cksum 0xa230 (correct), seq 0:83, ack 1, win 221, options [nop,nop,TS val 1958327 ecr 1821109], length 83: HTTP, length: 83 GET / HTTP/1.1 User-Agent: curl/7.29.0 Host: 192.168.10.100:8080 Accept: */* ``` `通过上面的抓包日志可以分析DR已经将SYN包发送到RS主机` 监听eth1接口8080端口的数据包 ```bash tcpdump -i eth1 and port 8080 -nnn -vvv ``` ```bash 08:31:57.116678 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [S.], cksum 0x95e6 (incorrect -> 0x6552), seq 2019760809, ack 3127325422, win 27960, options [mss 1410,sackOK,TS val 1783956 ecr 1921172,nop,wscale 7], length 0 08:31:57.118567 IP (tos 0x0, ttl 64, id 8764, offset 0, flags [DF], proto TCP (6), length 52) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [.], cksum 0x95de (incorrect -> 0xfff2), seq 1, ack 84, win 219, options [nop,nop,TS val 1783958 ecr 1921174], length 0 08:31:57.118811 IP (tos 0x0, ttl 64, id 8765, offset 0, flags [DF], proto TCP (6), length 69) 192.168.10.100.8080 > 192.168.10.3.46732: Flags [P.], cksum 0x95ef (incorrect -> 0x4015), seq 1:18, ack 84, win 219, options [nop,nop,TS val 1783958 ecr 1921174], length 17: HTTP, length: 17 HTTP/1.0 200 OK ``` `通过上面的抓包日志可以发现，RS也将应答包SYN+ACK直接发送给客户端` **`总结:LVS tunnel 跨网段转发是成功的。`** &emsp; ### LVS+TUNNEL(公网) `Ip Tunnel模式最大的优点就在于它可以跨网段转发，没有DR和NAT模式的组网限制。这在部署上带来的很大的灵活性，甚至还可以跨机房转发，不过不建议这样使用，一是会带来跨机房间的流量，提高了成本；二是跨机房转发必然会要在RS机房上绑定LVS机房的VIP，这有可能会被运营商的防火墙认为是IP伪造请求而拦截` #### 准备环境(Vultr VPS) ```bash Client: 183.54.238.66 VIP: 104.238.150.254 LVS主机: 167.179.115.37 Real Server: 202.182.125.31 139.180.202.67 网络模式:TUNNEL ``` &emsp; #### **DR配置** DR主机所在的VPS申请添加一个公网ip,配置到eth0:1接口上 ```bash ifconfig eth0:1 104.238.150.254 netmask 255.255.255.255 broadcast 104.238.150.254 up route add -host 104.238.150.254 dev eth0:1 ``` 修改内核参数 ```bash echo "0" >/proc/sys/net/ipv4/ip_forward echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects ``` 配置lvs ```bash ipvsadm -C ipvsadm -A -t 104.238.150.254:8080 -s rr ipvsadm -a -t 104.238.150.254:8080 -r 202.182.125.31 -i ipvsadm -a -t 104.238.150.254:8080 -r 139.180.202.67 -i ``` &emsp; #### **RS配置** 配置tunnel隧道 ```bash modprobe ipip ifconfig tunl0 104.238.150.254 netmask 255.255.255.255 broadcast 104.238.150.254 up route add -host 104.238.150.254 dev tunl0 ``` 修改内核参数 ```bash echo 0 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/conf/tunl0/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/tunl0/arp_announce echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter ``` 启动服务 ```bash python -m SimpleHTTPServer 8080 ``` &emsp; #### 客户端测试浏览器访问 `http://104.238.150.254:8080` `浏览器显示访问不了！！！` &emsp; #### RS主机抓包监听隧道接口 ```bash tcpdump -i tunl0 -nnn -vvv ``` ```bash tcpdump: listening on tunl0, link-type RAW (Raw IP), capture size 262144 bytes 09:20:12.713031 IP (tos 0x0, ttl 116, id 29938, offset 0, flags [DF], proto TCP (6), length 48) 183.54.238.66.63376 > 104.238.150.254.8080: Flags [S], cksum 0x5a81 (correct), seq 3523639844, win 8192, options [mss 1440,nop,nop,sackOK], length 0 09:20:54.742880 IP (tos 0x0, ttl 116, id 30339, offset 0, flags [DF], proto TCP (6), length 52) 183.54.238.66.63400 > 104.238.150.254.8080: Flags [S], cksum 0xb006 (correct), seq 2103403813, win 8192, options [mss 1440,nop,wscale 2,nop,nop,sackOK], length 0 09:20:54.993380 IP (tos 0x0, ttl 116, id 30342, offset 0, flags [DF], proto TCP (6), length 52) 183.54.238.66.63402 > 104.238.150.254.8080: Flags [S], cksum 0x8306 (correct), seq 484897436, win 8192, options [mss 1440,nop,wscale 2,nop,nop,sackOK], length 0 09:20:57.744463 IP (tos 0x0, ttl 116, id 30376, offset 0, flags [DF], proto TCP (6), length 52) ``` `通过上面的抓包日志可以分析DR已经将SYN包发送到RS主机` 监听客户端访问服务器8080端口的数据包 ```bash tcpdump -i eth0 host 183.54.238.66 and port 8080 -vvv -nnn ``` ```bash tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 09:19:13.659753 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:14.861060 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:16.660745 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 104.238.150.254.8080 > 183.54.238.66.63354: Flags [S.], cksum 0xa58c (incorrect -> 0x6d88), seq 2281292955, ack 1846351957, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 09:19:19.061063 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) ``` `通过上面的抓包日志可以发现，RS也将应答包SYN+ACK直接发送给客户端` **`总结:LVS tunnel DR转发,RS应答是正常的。RS应答的时候被Vultr的防火墙认为是IP伪造请求而拦截，所以导致实验失败！`** &emsp; ### LVS+FULLNAT `LVS 当前应用主要采用 DR 和 NAT 模式，但这 2 种模式要求 RealServer 和 LVS 在同一个 vlan中，导致部署成本过高；TUNNEL 模式虽然可以跨 vlan，但 RealServer上需要部署 ipip 模块等，网络拓扑上需要连通外网，较复杂，不易运维。` `为了解决上述问题，我们在 LVS 上添加了一种新的转发模式：FULLNAT，该模式和 NAT 模式的区别是：Packet IN 时，除了做 DNAT，还做 SNAT（用户 ip->内网 ip），从而实现 LVS-RealServer 间可以跨 vlan 通讯，RealServer 只需要连接到内网；` LVS FULLNAT 实战：https://www.haxi.cc/archives/LVS-FULLNAT实战.html 相关链接: LVS-ospf集群：http://noops.me/?p=974