326 lines
8.5 KiB
Markdown
326 lines
8.5 KiB
Markdown
|
# Docker 部署 Keepalived 2.3.4 实现三节点高可用 VIP
|
|||
|
|
本文档详细说明如何在三台机器上通过 Docker 部署 Keepalived 2.3.4，实现 VIP（${VIP_IP}）的高可用，并通过仲裁节点防止脑裂问题。
|
|||
|
|
|
|||
|
|
## 1. 节点信息
|
|||
|
|
| 节点名称 | IP 地址 | 角色 | 基础优先级 | 备注 |
|
|||
|
|
| --- | --- | --- | --- | --- |
|
|||
|
|
| Node A | ${NODE_A_IP} | 主节点 (Master) | 110 | MySQL 故障优先级扣 50 → 60 |
|
|||
|
|
| Node B | ${NODE_B_IP} | 备节点 (Backup) | 100 | MySQL 故障优先级扣 50 → 50 |
|
|||
|
|
| Node C | ${NODE_C_IP} | 仲裁节点 (Arbiter) | 40 | 参与选举但不持有 VIP |
|
|||
|
|
|
|||
|
|
|
|||
|
|
+ **VIP 地址**：${VIP_IP}
|
|||
|
|
+ **仲裁节点作用**：Node C 仅参与选举流程，防止 Node A 和 Node B 之间发生脑裂，正常场景下不持有 VIP。
|
|||
|
|
|
|||
|
|
## 2. 构建 Keepalived 镜像（解决 libip4tc 报错）
|
|||
|
|
采用多阶段构建方式，确保镜像最小化且包含所有运行时依赖。
|
|||
|
|
|
|||
|
|
### 2.1 Dockerfile 内容
|
|||
|
|
```dockerfile
|
|||
|
|
# Stage 1: 构建 Keepalived 二进制文件
|
|||
|
|
FROM alpine:3.23.2 AS builder
|
|||
|
|
|
|||
|
|
# 安装构建依赖
|
|||
|
|
RUN apk add --no-cache \
|
|||
|
|
gcc \
|
|||
|
|
make \
|
|||
|
|
libc-dev \
|
|||
|
|
openssl-dev \
|
|||
|
|
libnl3-dev \
|
|||
|
|
ipset-dev \
|
|||
|
|
iptables-dev \
|
|||
|
|
libnfnetlink-dev \
|
|||
|
|
libnftnl-dev \
|
|||
|
|
libmnl-dev \
|
|||
|
|
net-snmp-dev \
|
|||
|
|
libssh2-dev \
|
|||
|
|
pcre2-dev \
|
|||
|
|
autoconf \
|
|||
|
|
automake \
|
|||
|
|
linux-headers \
|
|||
|
|
curl \
|
|||
|
|
tar
|
|||
|
|
|
|||
|
|
# 下载并编译 Keepalived 2.3.4
|
|||
|
|
ENV KEEPALIVED_VERSION=2.3.4
|
|||
|
|
RUN curl -sSL https://www.keepalived.org/software/keepalived-${KEEPALIVED_VERSION}.tar.gz -o keepalived.tar.gz && \
|
|||
|
|
tar -zxf keepalived.tar.gz && \
|
|||
|
|
cd keepalived-${KEEPALIVED_VERSION} && \
|
|||
|
|
./configure --prefix=/usr --sysconfdir=/etc && \
|
|||
|
|
make && \
|
|||
|
|
make install DESTDIR=/install
|
|||
|
|
|
|||
|
|
# Stage 2: 构建最终运行镜像
|
|||
|
|
FROM alpine:3.23.2
|
|||
|
|
|
|||
|
|
# 安装运行时依赖
|
|||
|
|
RUN apk add --no-cache \
|
|||
|
|
libnl3 \
|
|||
|
|
ipset \
|
|||
|
|
iptables \
|
|||
|
|
libnftnl \
|
|||
|
|
libmnl \
|
|||
|
|
libnfnetlink \
|
|||
|
|
net-snmp-libs \
|
|||
|
|
libssh2 \
|
|||
|
|
pcre2 \
|
|||
|
|
openssl \
|
|||
|
|
ca-certificates \
|
|||
|
|
tzdata \
|
|||
|
|
libip4tc \
|
|||
|
|
libip6tc
|
|||
|
|
|
|||
|
|
# 从构建阶段复制编译好的二进制文件
|
|||
|
|
COPY --from=builder /install /
|
|||
|
|
|
|||
|
|
# 创建默认配置目录
|
|||
|
|
RUN mkdir -p /etc/keepalived
|
|||
|
|
|
|||
|
|
# 暴露 VRRP 协议端口 (112)
|
|||
|
|
EXPOSE 112
|
|||
|
|
|
|||
|
|
# 前台运行 Keepalived（适配 Docker 容器运行模式）
|
|||
|
|
# --dont-fork: 不后台运行
|
|||
|
|
# --log-console: 日志输出到控制台（便于 Docker 日志查看）
|
|||
|
|
ENTRYPOINT ["keepalived", "--dont-fork", "--log-console"]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 2.2 构建镜像命令
|
|||
|
|
```bash
|
|||
|
|
docker build -t keepalived:2.3.4 .
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 3. 配置文件部署
|
|||
|
|
### 3.1 前置准备
|
|||
|
|
在三台机器上统一创建配置目录：
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
mkdir -p /data/keepalived
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.2 Docker Compose 配置（三台机器通用）
|
|||
|
|
创建 `/data/keepalived/docker-compose.yml`：
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
services:
|
|||
|
|
keepalived:
|
|||
|
|
image: keepalived:2.3.4
|
|||
|
|
container_name: keepalived
|
|||
|
|
restart: always
|
|||
|
|
network_mode: host # 主机网络模式，确保 VRRP 协议正常通信
|
|||
|
|
cap_add: # 添加网络管理权限
|
|||
|
|
- NET_ADMIN
|
|||
|
|
- NET_BROADCAST
|
|||
|
|
- NET_RAW
|
|||
|
|
volumes:
|
|||
|
|
- /data/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf:ro # 只读挂载配置文件
|
|||
|
|
- /data/keepalived/check-mysql.sh:/etc/keepalived/check-mysql.sh:ro # 挂载 MySQL 检测脚本（仅 Node A/B 需要）
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.3 各节点 Keepalived 配置
|
|||
|
|
#### 3.3.1 Node A（${NODE_A_IP}）配置
|
|||
|
|
路径：`/data/keepalived/keepalived.conf`
|
|||
|
|
|
|||
|
|
```nginx
|
|||
|
|
global_defs {
|
|||
|
|
router_id ${ROUTER_ID_A}
|
|||
|
|
vrrp_skip_check_adv_addr
|
|||
|
|
script_user root # 指定脚本执行用户
|
|||
|
|
enable_script_security # 启用脚本安全校验，防止执行未授权脚本
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# MySQL 存活检测脚本定义
|
|||
|
|
vrrp_script check_mysql {
|
|||
|
|
script "/etc/keepalived/check-mysql.sh"
|
|||
|
|
interval 3 # 检测间隔：3 秒
|
|||
|
|
weight -50 # 检测失败时，优先级扣 50
|
|||
|
|
fall 3 # 连续 3 次失败判定为故障
|
|||
|
|
rise 2 # 连续 2 次成功判定为恢复
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
vrrp_instance VI_1 {
|
|||
|
|
state BACKUP # 所有节点统一设为 BACKUP，靠优先级决定 Master
|
|||
|
|
interface ${INTERFACE} # 替换为宿主机实际网卡名（如 ens33/br0 等）
|
|||
|
|
virtual_router_id ${VIRTUAL_ROUTER_ID} # 虚拟路由 ID，集群内必须一致（1-255）
|
|||
|
|
priority 110 # 基础优先级
|
|||
|
|
preempt_delay 30 # 优先级恢复后，延迟 30 秒抢占 VIP（减少抖动）
|
|||
|
|
advert_int 1 # VRRP 通告间隔：1 秒
|
|||
|
|
|
|||
|
|
# 认证配置（集群内必须一致）
|
|||
|
|
authentication {
|
|||
|
|
auth_type PASS
|
|||
|
|
auth_pass ${AUTH_PASS}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# 单播配置（替代组播，适配部分网络环境）
|
|||
|
|
unicast_src_ip ${NODE_A_IP} # 本机 IP
|
|||
|
|
unicast_peer { # 集群其他节点 IP
|
|||
|
|
${NODE_B_IP}
|
|||
|
|
${NODE_C_IP}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# 虚拟 IP 配置
|
|||
|
|
virtual_ipaddress {
|
|||
|
|
${VIP_IP}/32 # VIP 地址（32 位掩码）
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# 绑定检测脚本
|
|||
|
|
track_script {
|
|||
|
|
check_mysql
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
#### 3.3.2 Node B（${NODE_B_IP}）配置
|
|||
|
|
路径：`/data/keepalived/keepalived.conf`
|
|||
|
|
|
|||
|
|
```nginx
|
|||
|
|
global_defs {
|
|||
|
|
router_id ${ROUTER_ID_B}
|
|||
|
|
vrrp_skip_check_adv_addr
|
|||
|
|
script_user root
|
|||
|
|
enable_script_security
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# MySQL 存活检测脚本定义
|
|||
|
|
vrrp_script check_mysql {
|
|||
|
|
script "/etc/keepalived/check-mysql.sh"
|
|||
|
|
interval 3
|
|||
|
|
weight -50
|
|||
|
|
fall 3
|
|||
|
|
rise 2
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
vrrp_instance VI_1 {
|
|||
|
|
state BACKUP
|
|||
|
|
interface ${INTERFACE} # 替换为宿主机实际网卡名
|
|||
|
|
virtual_router_id ${VIRTUAL_ROUTER_ID}
|
|||
|
|
priority 100 # 基础优先级（低于 Node A）
|
|||
|
|
advert_int 1
|
|||
|
|
|
|||
|
|
authentication {
|
|||
|
|
auth_type PASS
|
|||
|
|
auth_pass ${AUTH_PASS}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
unicast_src_ip ${NODE_B_IP}
|
|||
|
|
unicast_peer {
|
|||
|
|
${NODE_A_IP}
|
|||
|
|
${NODE_C_IP}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
virtual_ipaddress {
|
|||
|
|
${VIP_IP}/32
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
track_script {
|
|||
|
|
check_mysql
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
#### 3.3.3 Node C（${NODE_C_IP}，仲裁节点）配置
|
|||
|
|
路径：`/data/keepalived/keepalived.conf`
|
|||
|
|
|
|||
|
|
```nginx
|
|||
|
|
global_defs {
|
|||
|
|
router_id ${ROUTER_ID_C}
|
|||
|
|
vrrp_skip_check_adv_addr
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
vrrp_instance VI_1 {
|
|||
|
|
state BACKUP
|
|||
|
|
interface ${INTERFACE} # 替换为宿主机实际网卡名
|
|||
|
|
virtual_router_id ${VIRTUAL_ROUTER_ID}
|
|||
|
|
priority 40 # 最低优先级（仅参与选举，不持有 VIP）
|
|||
|
|
advert_int 1
|
|||
|
|
|
|||
|
|
authentication {
|
|||
|
|
auth_type PASS
|
|||
|
|
auth_pass ${AUTH_PASS}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
unicast_src_ip ${NODE_C_IP}
|
|||
|
|
unicast_peer {
|
|||
|
|
${NODE_A_IP}
|
|||
|
|
${NODE_B_IP}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
# 仲裁节点不配置 virtual_ipaddress，不持有 VIP
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.4 MySQL 检测脚本（仅 Node A/B 需要）
|
|||
|
|
路径：`/data/keepalived/check-mysql.sh`
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
#!/bin/sh
|
|||
|
|
# 检测逻辑：
|
|||
|
|
# 1. timeout 2：限制命令 2 秒内完成，避免脚本阻塞
|
|||
|
|
# 2. nc -z：仅检测端口连通性（无数据传输）
|
|||
|
|
# 3. 屏蔽所有输出，仅返回退出码
|
|||
|
|
|
|||
|
|
if timeout 2 nc -z 127.0.0.1 3306 > /dev/null 2>&1; then
|
|||
|
|
# 端口连通，返回 0（Keepalived 判定正常）
|
|||
|
|
exit 0
|
|||
|
|
else
|
|||
|
|
# 端口不通/超时，返回 1（Keepalived 触发优先级扣减）
|
|||
|
|
exit 1
|
|||
|
|
fi
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
添加执行权限：
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
chmod +x /data/keepalived/check-mysql.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 4. 启动与验证
|
|||
|
|
### 4.1 启动容器
|
|||
|
|
三台机器执行相同命令：
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
cd /data/keepalived
|
|||
|
|
docker-compose up -d
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 4.2 日志查看
|
|||
|
|
```bash
|
|||
|
|
docker logs -f keepalived
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> 正常日志会显示 VRRP 选举过程、VIP 绑定/释放、脚本检测结果等信息。
|
|||
|
|
>
|
|||
|
|
|
|||
|
|
### 4.3 VIP 验证
|
|||
|
|
在任意节点执行，查看 VIP 是否绑定：
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
ip addr show ${INTERFACE} # 替换为实际网卡名
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
+ 正常情况下，VIP（${VIP_IP}）会绑定在优先级最高的节点（初始为 Node A）。
|
|||
|
|
+ 输出示例：`inet ${VIP_IP}/32 scope global ${INTERFACE}`
|
|||
|
|
|
|||
|
|
## 5. 故障场景分析
|
|||
|
|
详细的故障场景模拟与 VIP 切换逻辑分析，请参阅：[Keepalived 故障场景分析](./fault-analysis.md)
|
|||
|
|
|
|||
|
|
## 6. 核心总结
|
|||
|
|
### 6.1 优先级规则
|
|||
|
|
+ VIP 归属由 Keepalived **有效优先级** 决定：基础优先级 A(110) > B(100) > C(40)；
|
|||
|
|
+ MySQL 故障会让 A/B 优先级扣 50，Keepalived 离线则失去选举资格。
|
|||
|
|
|
|||
|
|
### 6.2 仲裁节点作用
|
|||
|
|
+ Node C 仅参与选举流程，不持有 VIP；
|
|||
|
|
+ 核心作用是防止 A、B 脑裂，正常/单节点故障场景下不影响 A/B 选举逻辑。
|
|||
|
|
|
|||
|
|
### 6.3 风险与优化
|
|||
|
|
+ **风险**：A/B MySQL 均故障但 Keepalived 正常时，VIP 仍绑定到 A，此时无可用 MySQL；需在业务层增加数据库存活检测，避免连接失效的 VIP。
|
|||
|
|
+ **抢占优化**：A 恢复后会在 `preempt_delay`（30 秒）后抢占 VIP，若需避免业务抖动，可在 Node A 配置中添加 `nopreempt` 关闭抢占。
|
|||
|
|
|
|||
|
|
### 6.4 关键配置注意事项
|
|||
|
|
+ 所有节点 `virtual_router_id` 和认证信息必须一致；
|
|||
|
|
+ 网卡名需替换为宿主机实际名称；
|
|||
|
|
+ 仲裁节点 C 不配置 `virtual_ipaddress`，不挂载 MySQL 检测脚本。
|
|||
|
|
|