[INFRA] EMR集群内节点自动配置系统深入理解

张

张建站

2026/8/9 19:22:46

10分钟阅读

请注意本文部分内容经过AI辅助生成虽然经过笔者检查但是并不保证内容的正确性请自行判断准确性本文对相关后果不承担责任EMR 集群的配置系统负责将用户通过 Configurations API 定义的参数如hdfs-site、yarn-site等 classification应用到每个节点的实际配置文件中。该系统在两个场景下工作首次启动配置Initial Provisioning集群创建时节点首次启动并应用配置运行时重配置Reconfiguration集群运行中通过ModifyInstanceGroupsAPI 修改配置两个场景共享同一套底层机制provision-node Puppet区别仅在于触发方式和状态转换路径。核心组件provision-node项目说明入口脚本/var/lib/aws/emr/provision-node→/usr/share/aws/emr/node-provisioner/bin/provision-node实际程序Java 程序com.amazonaws.emr.node.provisioner.Programclasspath/usr/share/aws/emr/node-provisioner/lib/*日志路径/mnt/var/log/provision-node/apps-phase/configurationVersion/uuid/stderr入口脚本是一个简单的 bash wrapper#!/usr/bin/bashset-exsudo/usr/share/aws/emr/node-provisioner/bin/provision-node$实际的 provision-node 脚本解析参数后以nohup方式启动 Java 进程nohup--java-classpath$CLASSPATHcom.amazonaws.emr.node.provisioner.Program\--yum-config-optionskip_missing_names_on_installFalse\--no-repo-provision$args$STDOUT_LOG2$STDERR_LOG0/dev/null关键参数参数说明--install安装组件 RPM 包--provision执行 Puppet 配置并启动服务--component-names指定组件列表逗号分隔不指定则从控制面获取-d日志子目录格式为configurationVersion/uuidinstance-controller (IC)IC 是 provision-node 的调用方负责首次启动时通过InstanceConfiguratorDoConfigureThread线程调用 provision-node运行时通过Poller线程的doRollingReconfiguration方法检测配置版本变化并触发重配Puppet / Bigtopprovision-node 内部使用 Puppet通过 Bigtop 框架来实际生成配置文件和管理服务。Puppet 的 Hiera 数据层级决定了配置的优先级和合并方式。首次启动配置流程IC 启动后InstanceConfiguratorDoConfigureThread线程在 bootstrap actions 完成BA_COMPLETED后调用 provision-node# /emr/instance-controller/log/instance-controller.log 16:00:25,527 INFO InstanceConfiguratorDoConfigureThread: Script 221aa460-... - Fetching file /var/lib/aws/emr/provision-node 16:00:25,528 INFO InstanceConfiguratorDoConfigureThread: startExec /var/lib/aws/emr/provision-node --install --provision -d0/221aa460-c789-4cb5-aad9-c053425c27ec 16:00:27,542 INFO InstanceConfiguratorDoConfigureThread: waitProcessCompletion ended with exit code 0注意参数-d0/221aa460-...其中0是configurationVersion。从控制面拉取配置provision-node 启动后PlatformConfigApplierFactory向 EMR 控制面请求当前节点的配置#/mnt/var/log/provision-node/apps-phase/0/221aa460-c789-4cb5-aad9-c053425c27ec/stderr16:00:28,662INFOPlatformConfigApplierFactory:platform configuration response:{configurations:[{classification:hdfs-site,properties:{dfs.data.dir:/mnt/hdfs,/mnt1/hdfs,dfs.replication:1,dfs.namenode.handler.count:64,...}},{classification:yarn-site,properties:{yarn.nodemanager.resource.memory-mb:6144,yarn.nodemanager.resource.cpu-vcores:4,...}},{classification:spark-defaults,properties:{spark.executor.memory:4743M,spark.executor.cores:2,...}},...],componentNames:[hadoop-client,hadoop-hdfs-namenode,hive-server2,spark-on-yarn,hbase-hmaster,...],clusterId:j-NIRY0CET7ORZ,instanceId:i-0c55d20ec0a458ddf,instanceType:m5.xlarge,nodeType:master,releaseLabel:emr-7.12.0,mountedDirs:[/mnt,/mnt1],configurationVersion:0,namenodeFormatted:false,isReprovision:false,isReversion:false,reconfigurationType:OVERWRITE}响应中包含configurations所有 classification 的配置参数EMR 默认值用户自定义值的合并结果componentNames该节点需要安装的组件列表configurationVersion配置版本号首次为0nodeType节点类型master/core/taskmountedDirs已挂载的磁盘目录由 setup-devices 阶段创建isReprovision是否为重配置首次为false安装组件 RPM开始安装具体的components16:00:28,692 INFO NodeProvisionerWorkflow: No componentNames passed. Fetching componentNames from platform 16:00:28,706 INFO NodeProvisionerWorkflow: Installing components: [hadoop-client, mariadb-server, delta, tez-on-yarn, hbase-thrift-server, emr-goodies, ...]大部分 RPM 已预装在 AMI 中is already installed只有少数需要额外安装# /mnt/var/log/provision-node/apps-phase/0/221aa460-c789-4cb5-aad9-c053425c27ec/stdout Package hadoop-client-3.4.1.amzn.4-1.amzn2023.x86_64 is already installed. Package spark-core-3.5.6.amzn.1-1.amzn2023.x86_64 is already installed. ... Installing: aws-emr-notebook-agent noarch 1.23.0-1 emr_apps 14 M emr-notebook-env x86_64 1.0.0-1 emr_apps 497 M hbase-master noarch 2.6.2.amzn.3-1.amzn2023 emr_apps 8.0 k ... 16:03:22,026 INFO ComponentInstaller: Took 2 minutes, 53 seconds and 67 milliseconds to install packages生成 Puppet Hiera 数据Emr7ConfigApplier将控制面返回的 JSON 配置转换为 Puppet Hiera YAML 格式16:03:22,304 INFO Emr7ConfigApplier: Applying Emr7 configuration overrides 16:03:22,351 INFO PuppetConfigurator: contents of core_set_hiera.yaml: --- defaults: datadir: /etc/puppetlabs/puppet/hieradata data_hash: yaml_data hierarchy: - name: bigtop paths: - site.yaml # 最高优先级用户配置 EMR 计算值 - bigtop/%{hadoop_hiera_ha_path}.yaml # HA/非HA 配置 - bigtop/cluster.yaml # Bigtop 框架默认值 - generated.yaml # 自动生成的密码等 - core_set_context.yaml # 执行集上下文标识 version: 5site.yaml由 provision-node 在内存中动态生成运行时注入 Puppet不落盘为独立文件包含了所有配置参数。以hdfs-site为例用户设置的dfs.replication: 1会出现在# site.yaml 中的 hdfs-site 配置hadoop::common_hdfs::hdfs_site_overrides:dfs.replication:1dfs.namenode.handler.count:64dfs.data.dir:/mnt/hdfs,/mnt1/hdfs...Hiera 查找优先级site.yamlbigtop/noha.yamlbigtop/cluster.yamlgenerated.yamlsite.yaml中的值会覆盖cluster.yaml中的默认值这就是用户自定义配置生效的机制。Puppet 分批应用配置Puppet 分三批执行core_set先执行完成后aux_set_1和aux_set_2并行执行16:03:22,489 INFO BigtopPuppeteer: setConfig reportdir: /var/log/provision-node/reports/0/221aa460-... 16:03:24,902 INFO BigtopPuppeteer: Set 1 puppet configs 16:04:46,748 INFO BigtopPuppeteer: Applied puppet core_set : Changes were successfully applied. # ~82s 16:05:02,846 INFO BigtopPuppeteer: Applied puppet aux_set_2 : Changes were successfully applied. # ~16s ┐ 并行 16:05:13,319 INFO BigtopPuppeteer: Applied puppet aux_set_1 : Changes were successfully applied. # ~27s ┘Puppet 执行的具体操作根据 Hiera 数据生成 XML 配置文件如/etc/hadoop/conf/hdfs-site.xml创建必要的目录结构并设置权限启动对应的服务进程NameNode、ResourceManager、HiveServer2 等Puppet 报告保存在/mnt/var/log/provision-node/reports/0/uuid/ ├── core_set/hostname/timestamp.yaml ├── core_set/hostname/timestamp.puppetreport.json ├── aux_set_1/hostname/timestamp.yaml └── aux_set_2/hostname/timestamp.yamlIC 状态转换首次启动时 IC 的 provision 状态转换链路# /emr/instance-controller/log/instance-controller.log 16:00:23,749 provision: PENDING # IC 启动等待 provision-node 完成 ... (provision-node 执行中持续 PENDING 约 5 分钟) 16:05:13,613 provision: SUCCESSFUL # provision-node 完成节点上线运行时重配置流程触发方式IC 的Poller线程每 ~30 秒执行一次doRollingReconfiguration检查每个 Instance Group 的配置版本# 正常轮询无配置变更 16:48:27,984 INFO Poller: No instances require reconfiguration for IG: ig-98Z5LSYM6ADI with requested configuration version 0. 16:48:27,984 INFO Poller: No instances require reconfiguration for IG: ig-1Y2TWODR3B7IU with requested configuration version 0. 16:48:27,984 INFO Poller: No instances require reconfiguration for IG: ig-1XQ3QT6FP7PPX with requested configuration version 0. 16:48:27,984 INFO Poller: long poll - doRollingReconfiguration took 0ms当用户通过ModifyInstanceGroupsAPI 提交新配置后控制面的configurationVersion递增。Poller 检测到版本不匹配触发重配# 检测到版本变化触发重配 16:49:25,366 INFO InstanceConfiguratorDoConfigureThread: Reconfigure instance from version 0 to version 1. 16:49:25,380 INFO InstanceConfiguratorDoConfigureThread: startExec /var/lib/aws/emr/provision-node --install --provision -d1/fb1f804e-5605-473e-976c-1b58d9aee049 ↑ configurationVersion 1provision-node 执行过程重配时 provision-node 的执行过程与首次启动基本一致但有几个关键差异。控制面响应中的差异字段# /mnt/var/log/provision-node/apps-phase/1/fb1f804e-.../stderr 16:49:26,408 INFO PlatformConfigApplierFactory: platform configuration response: { configurations : [ { classification : hdfs-site, properties : { dfs.replication : 2, ← 从 1 变为 2 ... } }, ... ], configurationVersion : 1, ← 从 0 变为 1 isReprovision : true, ← 从 false 变为 true isReversion : false, reconfigurationType : OVERWRITE }字段首次启动运行时重配configurationVersion01递增isReprovisionfalsetruedfs.replication12本次修改RPM 安装阶段重配时所有 RPM 已安装此阶段几乎瞬间完成对比首次启动的 2 分 53 秒差异显著。16:49:27,584 INFO ComponentInstaller: Took 1 second and 14 milliseconds to install packagesPuppet 应用阶段16:49:27,827 INFO Emr7ConfigApplier: Applying Emr7 configuration overrides 16:49:28,007 INFO PuppetConfigurator: contents of site.yaml: ... hadoop::common_hdfs::hdfs_site_overrides: dfs.replication: 2 ← 新值 ... emr::is_reprovision: !!bool true ← 标记为重配置 hadoop::namenode::is_reprovision: !!bool false ← NameNode 不重新格式化 ...Puppet 执行结果16:50:11,233 INFO BigtopPuppeteer: Applied puppet core_set : Changes were successfully applied. # ~43s 16:50:16,595 INFO BigtopPuppeteer: Applied puppet aux_set_2 : There were no changes to apply. # 无变更 16:50:21,842 INFO BigtopPuppeteer: Applied puppet aux_set_1 : Changes were successfully applied. # ~11s注意aux_set_2报告There were no changes to apply——因为本次只修改了hdfs-site与 aux_set_2 管理的组件Notebook、httpd 等无关Puppet 智能跳过了无变更的部分。IC 状态转换运行时重配的状态转换链路16:49:23,357 provision: SUCCESSFUL # 重配前节点正常运行 16:49:27,989 provision: RECONFIGURING # 检测到版本变化开始重配 16:49:58,002 provision: RECONFIGURING # provision-node 执行中 16:50:24,471 provision: SUCCESSFUL # 重配完成节点恢复正常滚动重配Master 节点的 IC 负责协调所有节点的重配按 Instance Group 滚动执行# Master 自身先完成重配 16:49:25,366 INFO InstanceConfiguratorDoConfigureThread: Reconfigure instance from version 0 to version 1. 16:49:27,382 INFO InstanceConfiguratorDoConfigureThread: waitProcessCompletion ended with exit code 0 # 然后通知其他 IG 的节点重配 16:49:27,992 INFO Poller: Reconfiguring instances: [i-03dc982658b9b4309] for IG: ig-1Y2TWODR3B7IU with requested configuration version 1. 16:49:27,992 INFO Poller: Reconfiguring instances: [i-0b4648b9ef7704727] for IG: ig-1XQ3QT6FP7PPX with requested configuration version 1. 16:49:27,992 INFO Poller: No instances require reconfiguration for IG: ig-98Z5LSYM6ADI with requested configuration version 1. ↑ Master 自身已完成无需再配首次启动 vs 运行时重配对比对比项首次启动 (Initial Provisioning)运行时重配 (Reconfiguration)触发方式IC 的InstanceConfiguratorDoConfigureThread在BA_COMPLETED后自动执行IC 的Poller.doRollingReconfiguration每 ~30s 轮询检测版本变化调用命令provision-node --install --provision -d0/uuidprovision-node --install --provision -d1/uuidconfigurationVersion0递增1,2, …isReprovisionfalsetrueRPM 安装耗时~2 分 53 秒部分包需要下载安装~1 秒所有包已安装Puppet 执行所有 set 都有变更需要应用仅受影响的 set 有变更其他报告no changesPuppet core_set 耗时~82 秒~43 秒IC 状态转换PENDING→SUCCESSFULSUCCESSFUL→RECONFIGURING→SUCCESSFUL日志目录apps-phase/0/uuid/apps-phase/1/uuid/滚动策略无所有节点并行首次配置有按 IG 滚动避免服务中断配置来源创建集群时的 Configurations 参数ModifyInstanceGroupsAPI 提交的新配置配置转换链路总览用户 API (CreateCluster / ModifyInstanceGroups) ↓ EMR 控制面存储配置 (configurationVersion: N) ↓ 每个节点的 IC 检测到版本变化 ↓ IC 调用 provision-node --install --provision -dN/uuid ↓ provision-node (Java) 从控制面拉取配置 (PlatformConfigApplierFactory) ↓ 返回 JSON: { configurations: [...], configurationVersion: N, ... } ↓ Emr7ConfigApplier 转换为 Puppet Hiera YAML (site.yaml) ↓ Puppet Hiera 层级合并: site.yaml (最高优先级用户配置 EMR 计算值) bigtop/noha.yaml 或 bigtop/ha.yaml bigtop/cluster.yaml (Bigtop 框架默认值) generated.yaml (自动生成的密码等) ↓ Puppet 分批执行: core_set → HDFS, YARN, ZooKeeper, HBase 等核心组件 aux_set_1 → Spark, Livy, Hudi 等 (与 core_set 完成后并行) aux_set_2 → Notebook, httpd 等 (与 aux_set_1 并行) ↓ 生成配置文件 (如 /etc/hadoop/conf/hdfs-site.xml) 重启受影响的服务 ↓ IC 状态变为 SUCCESSFUL节点配置完成日志路径汇总日志路径说明IC 主日志/emr/instance-controller/log/instance-controller.log配置触发、状态转换、滚动重配协调provision-node stderr/mnt/var/log/provision-node/apps-phase/version/uuid/stderr配置拉取、Hiera 生成、Puppet 执行provision-node stdout/mnt/var/log/provision-node/apps-phase/version/uuid/stdoutRPM 安装输出Puppet core_set 日志/mnt/var/log/provision-node/apps-phase/version/uuid/puppet_core_set.log核心组件 Puppet 详细日志Puppet aux_set_1 日志/mnt/var/log/provision-node/apps-phase/version/uuid/puppet_aux_set_1.log辅助组件集 1 Puppet 详细日志Puppet aux_set_2 日志/mnt/var/log/provision-node/apps-phase/version/uuid/puppet_aux_set_2.log辅助组件集 2 Puppet 详细日志Puppet 报告/mnt/var/log/provision-node/reports/version/uuid/set/hostname/YAML 和 JSON 格式的 Puppet 执行报告IC 触发记录/mnt/var/log/provision-node/uuid/stdout和stderrIC 调用 provision-node 的输出apps-phase/下的数字目录即configurationVersion每次重配会新增一个目录可通过目录数量判断集群经历了多少次配置变更# 查看配置变更历史ls/mnt/var/log/provision-node/apps-phase/# 输出: 0 1# 说明: 首次启动(0) 一次重配(1)关于 install-components 的说明install-components.service会读取/var/aws/emr/componentNames.csv并提前调用provision-node --install --component-names预安装 RPM。在 EMR 7.12.0 中该优化步骤已被禁用cloud-init 的write_files配置中不包含componentNames.csv该文件不存在install-components脚本中start函数的调用被注释掉# /usr/bin/install-components 末尾$PROVISION_NODE_SCRIPT--install--component-names$component_names}# start ← 被注释函数不会执行exit0install-components.service虽然存在并正常启动但实际不执行任何操作# journalctl -u install-components.service install-components[3944]: COMPONENT_NAMES_FILE_PATH/var/aws/emr/componentNames.csv install-components[3944]: PROVISION_NODE_SCRIPT/var/lib/aws/emr/provision-node install-components[3944]: exit 0 ← 直接退出所有组件安装由 IC 触发的provision-node --install --provision一步完成不再分为预安装和配置两个阶段。Puppet Hiera 数据目录结构/etc/puppetlabs/puppet/hieradata/ ├── bigtop/ │ ├── cluster.yaml # Bigtop 框架默认配置模板 │ ├── ha.yaml # HA 模式配置 │ └── noha.yaml # 非 HA 模式配置 ├── core_set_context.yaml # 内容: context::execution_set: core_set ├── aux_set_1_context.yaml # 内容: context::execution_set: aux_set_1 ├── aux_set_2_context.yaml # 内容: context::execution_set: aux_set_2 └── generated.yaml # 自动生成的数据库密码等site.yaml不在磁盘上由 provision-node 在运行时动态注入 Puppet 进程。配置拉取模型EMR 采用拉模型Pull Model进行配置分发每个节点的 IC 独立轮询控制面检测配置版本变化每个节点的 provision-node 独立从控制面拉取配置不存在 Master 向 Core/Task 推送配置的过程这种设计的优势无论集群规模多大都能并行完成配置不存在单点瓶颈Master 不是配置分发的瓶颈节点故障不影响其他节点的配置过程新加入的节点自动获取最新版本的配置