告别手动拷贝用Ansible自动化部署Spark 3.x集群基于CentOS 7在分布式计算领域Spark集群的部署往往伴随着大量重复性劳动。传统的手动操作不仅效率低下还容易因人为疏忽导致配置不一致。本文将介绍如何利用Ansible这一自动化运维利器实现Spark 3.x集群的一键式部署让运维人员从繁琐的配置工作中彻底解放。1. 环境准备与Ansible基础配置部署Spark集群前需要确保所有节点满足基本运行条件。以下是一个典型的三节点集群架构控制节点1台运行Ansible负责执行部署任务工作节点2台运行Spark的Worker进程1.1 系统基础配置所有节点需要预先完成以下配置# 关闭SELinux需要重启生效 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config # 配置主机名解析所有节点 echo 192.168.1.10 spark-master /etc/hosts echo 192.168.1.11 spark-worker1 /etc/hosts echo 192.168.1.12 spark-worker2 /etc/hosts1.2 Ansible控制节点安装在控制节点上安装Ansible# CentOS 7安装EPEL源 yum install -y epel-release # 安装Ansible yum install -y ansible # 验证安装 ansible --version配置SSH免密登录到所有工作节点# 生成密钥对 ssh-keygen -t rsa # 分发公钥到所有节点 ssh-copy-id rootspark-master ssh-copy-id rootspark-worker1 ssh-copy-id rootspark-worker22. Ansible Inventory与Playbook设计2.1 定义主机清单创建/etc/ansible/hosts文件定义Spark集群节点[spark_master] spark-master [spark_workers] spark-worker1 spark-worker2 [spark_cluster:children] spark_master spark_workers2.2 基础Playbook结构创建spark-cluster.yml作为主Playbook--- - name: Deploy Spark 3.x Cluster hosts: spark_cluster become: yes vars: spark_version: 3.3.1 spark_install_dir: /opt/spark java_version: 11 tasks: # 后续任务将在这里添加3. 自动化部署Spark集群3.1 安装Java环境Spark运行依赖Java环境添加以下任务到Playbook- name: Install OpenJDK yum: name: java-{{ java_version }}-openjdk-devel state: present - name: Set JAVA_HOME lineinfile: path: /etc/environment line: JAVA_HOME/usr/lib/jvm/java-{{ java_version }}-openjdk state: present3.2 下载并安装Spark使用Ansible的get_url模块下载Spark二进制包- name: Create installation directory file: path: {{ spark_install_dir }} state: directory mode: 0755 - name: Download Spark get_url: url: https://archive.apache.org/dist/spark/spark-{{ spark_version }}/spark-{{ spark_version }}-bin-hadoop3.tgz dest: /tmp/spark-{{ spark_version }}-bin-hadoop3.tgz checksum: sha256:https://archive.apache.org/dist/spark/spark-{{ spark_version }}/spark-{{ spark_version }}-bin-hadoop3.tgz.sha256 - name: Extract Spark unarchive: src: /tmp/spark-{{ spark_version }}-bin-hadoop3.tgz dest: {{ spark_install_dir }} remote_src: yes extra_opts: [--strip-components1]3.3 配置环境变量添加Spark环境变量配置- name: Configure Spark environment blockinfile: path: /etc/profile.d/spark.sh block: | export SPARK_HOME{{ spark_install_dir }} export PATH$PATH:$SPARK_HOME/bin create: yes4. 集群配置与优化4.1 主节点配置在spark-master节点上配置- name: Configure master node hosts: spark_master tasks: - name: Create spark-env.sh template: src: templates/spark-env.sh.j2 dest: {{ spark_install_dir }}/conf/spark-env.sh mode: 0644对应的模板文件templates/spark-env.sh.j2内容#!/usr/bin/env bash export SPARK_MASTER_HOST{{ inventory_hostname }} export SPARK_MASTER_PORT7077 export SPARK_MASTER_WEBUI_PORT8080 export SPARK_WORKER_CORES4 export SPARK_WORKER_MEMORY8g4.2 工作节点配置工作节点需要知道主节点地址- name: Configure worker nodes hosts: spark_workers tasks: - name: Create spark-env.sh template: src: templates/spark-env.sh.j2 dest: {{ spark_install_dir }}/conf/spark-env.sh mode: 0644 - name: Configure workers template: src: templates/workers.j2 dest: {{ spark_install_dir }}/conf/workers mode: 0644templates/workers.j2文件内容{% for host in groups[spark_workers] %} {{ hostvars[host][ansible_hostname] }} {% endfor %}5. 服务管理与集群启动5.1 创建Systemd服务为Spark主节点和工作节点创建服务- name: Configure master service hosts: spark_master tasks: - name: Create spark-master.service template: src: templates/spark-master.service.j2 dest: /etc/systemd/system/spark-master.service mode: 0644 - name: Reload systemd systemd: daemon_reload: yes - name: Enable and start spark-master systemd: name: spark-master state: started enabled: yes对应的spark-master.service.j2模板[Unit] DescriptionApache Spark Master Afternetwork.target [Service] Typesimple Userroot ExecStart{{ spark_install_dir }}/sbin/start-master.sh ExecStop{{ spark_install_dir }}/sbin/stop-master.sh Restarton-failure [Install] WantedBymulti-user.target5.2 验证集群状态部署完成后可以通过以下命令验证集群状态# 在主节点查看Spark UI curl http://spark-master:8080 # 查看工作节点日志 journalctl -u spark-worker -f6. 高级配置与调优6.1 配置参数优化根据集群硬件配置调整Spark参数参数名默认值推荐值说明spark.executor.memory1g4g每个Executor的内存分配spark.executor.cores12每个Executor的CPU核心数spark.driver.memory1g2gDriver进程内存大小spark.default.parallelism832默认并行度6.2 安全配置为生产环境添加基本安全措施- name: Configure Spark security hosts: spark_cluster tasks: - name: Create spark-defaults.conf template: src: templates/spark-defaults.conf.j2 dest: {{ spark_install_dir }}/conf/spark-defaults.conf mode: 0644对应的安全配置模板示例spark.authenticatetrue spark.authenticate.secretyour-secret-key spark.ui.filtersorg.apache.spark.deploy.master.ui.MasterUIFilter7. 日常运维与扩展7.1 集群扩展添加新节点到现有集群将新节点添加到Ansible inventory运行Playbook时使用--limit参数只对新节点执行更新主节点的workers文件ansible-playbook spark-cluster.yml --limitnew-worker-node7.2 版本升级Spark版本升级流程graph TD A[备份现有配置] -- B[停止集群服务] B -- C[下载新版本] C -- D[安装新版本] D -- E[恢复配置] E -- F[启动集群]注意升级前务必测试新版本与现有应用的兼容性