WEB集群部署
服務(wù)器資源分配
| 服務(wù)器角色 | 服務(wù)器 A | 服務(wù)器 B |
|---|---|---|
| Nginx 負(fù)載均衡 | ?（單點(diǎn)部署于 B） | ?（主負(fù)載均衡） |
| Spring Boot 應(yīng)用 | ?（Fastbee，含 Ignite 節(jié)點(diǎn)） | ?（Fastbee，含 Ignite 節(jié)點(diǎn)） |
| 數(shù)據(jù)庫 | ?（MySQL + Tdengine） | 主從模式待實(shí)現(xiàn).... |
| 緩存 | ?（Redis 單點(diǎn)，帶持久化） | 主從架構(gòu)待實(shí)現(xiàn)... |
1. 環(huán)境準(zhǔn)備（兩臺(tái)服務(wù)器均執(zhí)行）
安裝 Docker 和 Docker Compose
# 安裝Docker curl -fsSL https://get.docker.com | sh systemctl start docker && systemctl enable docker # 安裝Docker Compose curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
2. 服務(wù)器 A 部署：數(shù)據(jù)庫、Redis、Fastbee服務(wù)
? 假設(shè)兩臺(tái)服務(wù)器A和B，公網(wǎng)IP：123.0.0.1 和 124.0.0.2 內(nèi)網(wǎng): 10.1.1.10 和 20.2.2.20
? 打包后端jar包
fastbee-admin.jar? 打包前端
dist包配置docker-compose文件
按照文檔提示進(jìn)行修改
version: '3.8' services: redis: image: redis:7.0.0 container_name: redis ports: - 6379:6379 privileged: true network_mode: host volumes: - /var/data/redis:/usr/local/etc/redis - /var/data/redis/data:/data command: [ '--requirepass', 'admin123', '--appendonly', 'yes' ] #密碼是admin123,自行修改 mysql: image: mysql:5.7 container_name: mysql environment: MYSQL_ROOT_PASSWORD: admin123 #密碼是admin123,自行修改 MYSQL_DATABASE: fastbee #數(shù)據(jù)庫名，自行更改 ports: - 3306:3306 privileged: true network_mode: host volumes: - /var/data/mysql/data:/var/lib/mysql - /var/data/mysql/initdb:/docker-entrypoint-initdb.d command: [ 'mysqld', '--character-set-server=utf8', '--collation-server=utf8_unicode_ci', '--default-time-zone=+8:00', '--lower-case-table-names=1', '--bind-address=0.0.0.0' ] # 假設(shè)兩臺(tái)服務(wù)器A和B，公網(wǎng)IP：123.0.0.1 和 124.0.0.2 內(nèi)網(wǎng): 10.1.1.10 和 20.2.2.20 java: image: openjdk:8-jre container_name: java ports: - 8080:8080 - 47500-47509:47500-47509 - 47101:47101 - 11211:11211 - 10800:10800 - 48500:48500 - 48880:48880 environment: WORKER_HOST: 123.0.0.1 # 服務(wù)器A公網(wǎng)IP 或內(nèi)網(wǎng)IP IGNITE_LOCAL_IP: 10.1.1.10 # 服務(wù)器A內(nèi)網(wǎng)IP IGNITE_CLUSTER_NODES: 10.1.1.10,20.2.2.20 # 服務(wù)器A和B內(nèi)網(wǎng)IP network_mode: host volumes: - /var/data/java/fastbee-admin.jar:/server.jar - /var/data/java/license:/license - /var/data/java/libtaos.so:/usr/lib/libtaos.so - /var/data/java/uploadPath:/uploadPath - /var/data/java/logs:/logs - /etc/localtime:/etc/localtime depends_on: - redis - mysql - tdengine entrypoint: - sh - -c - | exec java \ -Xmx4g -Xms4g \ #4核8G配置,根據(jù)情況調(diào)優(yōu) -DIGNITE_QUIET=false \ -DIGNITE_NO_DISCO_ORDER=true \ -Djava.net.preferIPv4Stack=true \ -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m \ -XX:+UseG1GC -XX:G1HeapRegionSize=32m \ -XX:ParallelGCThreads=4 -XX:ConcGCThreads=2 \ -Dfile.encoding=UTF-8 \ -Dignite.heap.offheap.size=4g \ #4核8G配置,根據(jù)情況調(diào)優(yōu) -jar /server.jar tdengine: image: tdengine/tdengine:3.3.2.0 hostname: tdengine container_name: tdengine ports: - 6030:6030 - 6041:6041 - 6043-6049:6043-6049 - 6043-6049:6043-6049/udp network_mode: host volumes: - /var/data/tdengine/log:/var/log/taos - /var/data/tdengine/data:/var/lib/taos - /var/data/tdengine/conf:/etc/taos - /etc/localtime:/etc/localtime
3. 服務(wù)器 B 部署：Nginx 負(fù)載均衡、Spring Boot 應(yīng)用
nginx.conf負(fù)載均衡配置
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
keepalive_requests 1000;
client_max_body_size 100m;
gzip on;
gzip_min_length 1k;
gzip_buffers 16 64k;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_types text/plain application/javascript application/x-javascript text/javascript text/css application/xml;
gzip_vary on;
gzip_proxied expired no-cache no-store private auth;
gzip_disable "MSIE [1-6]\.";
# 服務(wù)器A和B的內(nèi)網(wǎng)IP 自行更改
upstream backend {
server 10.1.1.10:8080 max_fails=1 fail_timeout=10s; # 服務(wù)A內(nèi)網(wǎng)IP,自行修改
server 20.2.2.20:8080 max_fails=1 fail_timeout=10s; # 服務(wù)B內(nèi)網(wǎng)IP,自行修改
keepalive 60;
least_conn;
}
server {
listen 80;
server_name localhost;
charset utf-8;
# SSL 默認(rèn)訪問端口號(hào)為443
listen 443 ssl;
server_name localhost;
charset utf-8;
# 證書文件的路徑
ssl_certificate /usr/share/nginx/ssl/fastbee.crt;
# 私鑰文件的路徑
ssl_certificate_key /usr/share/nginx/ssl/fastbee.key;
ssl_session_timeout 10m;
# 請(qǐng)按照以下協(xié)議配置
ssl_protocols TLSv1.2 TLSv1.3;
# 請(qǐng)按照以下套件配置，配置加密套件，寫法遵循openssl 標(biāo)準(zhǔn)
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
ssl_prefer_server_ciphers on;
#................其他配置
location /prod-api/ {
proxy_pass http://backend/;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "";
proxy_redirect off;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
send_timeout 300s;
}
}
}docker-compose配置修改
version: '3.8'
services:
nginx:
image: nginx:stable
container_name: nginx
ports:
- "80:80"
- "443:443"
volumes:
- /var/data/nginx/vue:/usr/share/nginx/html
- /var/data/nginx/view:/usr/share/nginx/view
- /var/data/nginx/h5:/usr/share/nginx/h5
# - /var/data/nginx/ssl:/usr/share/nginx/ssl #SSL證書有則放開
- /var/data/nginx/logs:/var/log/nginx
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
network_mode: host
# 假設(shè)兩臺(tái)服務(wù)器A和B，公網(wǎng)IP：123.0.0.1 和 124.0.0.2 內(nèi)網(wǎng): 10.1.1.10 和 20.2.2.20
java:
image: openjdk:8-jre
container_name: java
ports:
- 8080:8080
- 47500-47509:47500-47509
- 47101:47101
- 11211:11211
- 10800:10800
- 48500:48500
- 48880:48880
environment:
WORKER_HOST: 124.0.0.2 # 服務(wù)器A公網(wǎng)IP 或內(nèi)網(wǎng)IP
IGNITE_LOCAL_IP: 20.2.2.20 # 服務(wù)器A內(nèi)網(wǎng)IP
IGNITE_CLUSTER_NODES: 10.1.1.10,20.2.2.20 # 服務(wù)器A和B內(nèi)網(wǎng)IP
volumes:
- /var/data/java/fastbee-admin.jar:/server.jar
- /var/data/java/license:/license
- /var/data/java/libtaos.so:/usr/lib/libtaos.so
- /var/data/java/uploadPath:/uploadPath
- /var/data/java/logs:/logs
- /etc/localtime:/etc/localtime
network_mode: host
entrypoint:
- sh
- -c
- |
exec java \
-Xmx4g -Xms4g \
-DIGNITE_QUIET=false \
-DIGNITE_NO_DISCO_ORDER=true \
-Djava.net.preferIPv4Stack=true \
-XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m \
-XX:+UseG1GC -XX:G1HeapRegionSize=32m \
-XX:ParallelGCThreads=4 -XX:ConcGCThreads=2 \
-Dfile.encoding=UTF-8 \
-Dignite.heap.offheap.size=4g \
-jar /server.jar4. 服務(wù)目錄
按照上面修改好后
- 將項(xiàng)目中 docker目錄下面，ServerA中 data復(fù)制至 服務(wù)器A
/var目錄下 - 將項(xiàng)目中 docker目錄下面，ServerB中 data復(fù)制至 服務(wù)器B
/var目錄下
執(zhí)行目錄授權(quán)
cd /var/data
setenforce 0
chmod 777 -R /var/data5.部署
在服務(wù)器A /var/data 執(zhí)行
docker-compose up -d
# 查看部署情況
#[root@VM-xxx data]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
429020770a4d openjdk:8-jre "sh -c 'exec java \\\n…" 16 minutes ago Up 16 minutes java
eebb0c7c40e9 tdengine/tdengine:3.3.2.0 "/tini -- /usr/bin/e…" 16 minutes ago Up 16 minutes tdengine
501697f9bfa7 redis:7.0.0 "docker-entrypoint.s…" 16 minutes ago Up 16 minutes redis
0b31966146f5 mysql:5.7 "docker-entrypoint.s…" 16 minutes ago Up 16 minutes mysql在服務(wù)器B /var/data 執(zhí)行
docker-compose up -d
# 查看部署情況
#[root@VM-xxx data]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9e7622669ec5 openjdk:8-jre "sh -c 'exec java \\\n…" About a minute ago Up About a minute java
4ed9ca408461 nginx:stable "/docker-entrypoint.…" About a minute ago Up About a minute nginx6.kafka負(fù)載均衡
1.分區(qū)與副本機(jī)制
數(shù)據(jù)分片：每個(gè)topic被劃分到多個(gè)分區(qū)，部分在不同Broker上，生產(chǎn)者發(fā)送消息，通過分區(qū)策略將消息均勻分配到各分區(qū)，實(shí)現(xiàn)寫入負(fù)載均衡
副本備份: 每個(gè)分區(qū)有多個(gè)副本，Leader處理讀寫請(qǐng)求， Follower同步數(shù)據(jù)，Leader故障，F(xiàn)ollower替換
2. 消費(fèi)者組
同一消費(fèi)者組內(nèi)的消費(fèi)者通過分配策略（Round Robin，Range），分?jǐn)偡謪^(qū)消費(fèi)任務(wù)，例如通過輪訓(xùn)分配，確保同一消費(fèi)者組中分區(qū)數(shù)據(jù)平衡
3.實(shí)施(重點(diǎn))
kafka的分區(qū)無法在代碼實(shí)現(xiàn)，需要借助外部工具，例如之前提到的 kafka-console-ui
下面舉例實(shí)現(xiàn)數(shù)據(jù)上報(bào)的消費(fèi)負(fù)載均衡：
目前數(shù)據(jù)上報(bào)主題 fmq_property_post 消費(fèi)實(shí)例是 10個(gè)，假設(shè)現(xiàn)在集群部署兩個(gè)web服務(wù)節(jié)點(diǎn)，該主題的消費(fèi)實(shí)例總數(shù)是 20，那么要達(dá)到數(shù)據(jù)上報(bào)消費(fèi)負(fù)載均衡需要滿足 分區(qū)數(shù) >= 所有節(jié)點(diǎn)主題消費(fèi)實(shí)例總數(shù)
更改分區(qū)數(shù)
驗(yàn)證kafka負(fù)載均衡
生產(chǎn)者發(fā)送數(shù)據(jù)，查看 消費(fèi)者組 - 消費(fèi)情況 -每個(gè)分區(qū)的消費(fèi)是否有并相對(duì)均衡
7.驗(yàn)證Ignite集群
方式1：查看java容器 sys-info.log
Cluster [hosts=2, CPUs=8, servers=2, clients=0, topVer=4, minorTopVer=1] hosts =2 表示集群節(jié)點(diǎn)為2
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=592e782c, name=fastbee-server, uptime=03:37:01.004]
^-- Cluster [hosts=2, CPUs=8, servers=2, clients=0, topVer=4, minorTopVer=1]
^-- Network [addrs=[10.0.24.6], localHost=10.0.24.6, discoPort=47501, commPort=47101]
^-- CPU [CPUs=4, curLoad=0.83%, avgLoad=1.04%, GC=0%]
^-- Heap [used=612MB, free=85.05%, comm=4096MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=4, qSize=0]
^-- Striped thread pool [active=0, idle=8, qSize=0]- 方式2：可視化界面
? 待開發(fā)....
8.后端負(fù)載均衡驗(yàn)證
方式1：在 Spring Boot 中添接口打印請(qǐng)求來源 IP：
定位ToolController，后端全路徑為
iot/tool/test
@GetMapping("/test")
public String test(HttpServletRequest request) {
return "接收到請(qǐng)求，來源 IP: " + request.getRemoteAddr();
}- 方式2：停止某節(jié)點(diǎn) JAVA容器服務(wù)
docker stop java前端訪問無出現(xiàn) 正常，即驗(yàn)證成功