k8s部署EFK日志管理系统一、拉取镜像搭建harbor私服docker安装docker-compose 安装Harbor安装准备镜像二、搭建Elasticsearch kibana三、搭建filebeat查看是否配置成功一、拉取镜像为了避免docker外网仓库拉取不了镜像导致pod一直运行不了这里我们搭建一个harbor私服搭建harbor私服docker安装//安装工具包sudoyuminstall-yyum-utils //设置远程仓库sudoyum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.reposudoyum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo //docker 安装sudoyuminstalldocker-ce //启动docker systemctl startdockersystemctl restartdocker//docker 版本dockerversion //开机自启动chkconfigdockeron /sudosystemctlenabledocker.servicedocker-compose 安装//下载 Docker Compose 至 /usr/local/bin 目录并赋予执行权限sudocurl-Lhttps://github.com/docker/compose/releases/download/v2.23.3/docker-compose-uname-s-uname-m-o/usr/local/bin/docker-composesudochmodx /usr/local/bin/docker-compose //docker compose 版本dockercompose versionHarbor安装下载版本压缩包https://github.com/goharbor/harbor/tagswgethttps://github.com/goharbor/harbor/releases/download/v2.9.4/harbor-offline-installer-v2.9.4.tgz解压安装包tarzxf harbor-offline-installer-v2.9.4.tgzcdharbor/ //cp yml配置cpharbor.yml.tmpl harbor.ymlharbor.yml 文件配置内容修改hostname为本机地址本地测试https配置暂时注释。hostname:192.168.224.130# https related config配置/etc/docker 下的config.json/damon.json添加配置{insecure-registries:[192.168.224.130]}5.重新加载配置后重启dockersystemctl restartdockerservicedockerrestart启动成功后进入harbor安装目录执行启动命令执行./ install.sh命令成功后会在harbor安装目录下生成docker-compos.yml文件后续启动停止使用docker compose进行控制。通过ip进行访问结果如下。准备镜像# 在本机拉取镜像dockerpull docker.elastic.co/elasticsearch/elasticsearch:7.17.2dockerpull docker.elastic.co/kibana/kibana:7.17.2dockerpull docker.elastic.co/beats/filebeat:7.17.2# 对镜像重打标签 将${harbor_url}和${harbor_project}换成自己的harbor私服地址和目录dockertag docker.elastic.co/elasticsearch/elasticsearch:7.17.2${harbor_url}/${harbor_project}/elasticsearch:7.17.2dockertag docker.elastic.co/kibana/kibana:7.17.2${harbor_url}/${harbor_project}/kibana:7.17.2dockertag docker.elastic.co/beats/filebeat:7.17.2${harbor_url}/${harbor_project}/filebeat:7.17.2# 登陆自己的harbor服务器dockerlogin-uadmin-p${password}${harbor_url}# 上传至harbor仓库dockerpush${harbor_url}/${harbor_project}/elasticsearch:7.17.2dockerpush${harbor_url}/${harbor_project}/kibana:7.17.2dockerpush${harbor_url}/${harbor_project}/filebeat:7.17.2如果此处的Harbor目录是私有的则需要在k8s集群中创建docker拉取harbor私库的密钥# -n 后是指定的空间根据自己的情况去更改,不加-n默认是default空间因为本次EFK安装在kube-system命名空间下所以-n为kube-system。kubectl create secret docker-registry harbor-pull-secret --docker-server${harbor_url}--docker-usernameadmin --docker-password${password}-nkube-system#检查密钥是否创建成功kubectl get secrets-nkube-system二、搭建Elasticsearch kibana创建配置文件elasticsearch.ymlcluster.name: my-es node.name:node-1path.data: /usr/share/elasticsearch/data#path.logs: /var/log/elasticsearchbootstrap.memory_lock:falsenetwork.host:0.0.0.0 http.port:9200discovery.seed_hosts:[127.0.0.1,[::1]]cluster.initial_master_nodes:[node-1]#增加参数使head插件可以访问eshttp.cors.enabled:truehttp.cors.allow-origin:*http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type xpack.monitoring.collection.enabled:true创建kibana配置文件kibana.ymlserver.port:5601server.host:0.0.0.0elasticsearch.hosts:http://es-kibana-0.es-kibana.kube-system:9200kibana.index:.kibanai18n.locale:zh-CNmonitoring.ui.elasticsearch.hosts:[http://es-kibana-0.es-kibana.kube-system:9200]monitoring.ui.enabled:true其中elasticsearch.hosts的地址构成为pod名:es-kibana-0,service名:es-kibana,命名空间:kube-system,以及service中配置的端口号9200创建elasticsearch和kibana的配置文件configmap#在编写yml配置文件的目录执行该命令kubectl create configmap es-config-nkube-system --from-fileelasticsearch.yml kubectl create configmap kibana-config-nkube-system --from-filekibana.yml配置NFS服务器1).安装NFS服务Ubuntusudoaptupdatesudoaptinstallnfs-kernel-servercentosyum update yum-yinstallnfs-utils# 创建或使用用已有的文件夹作为nfs文件存储点mkdir-p/home/data/nfs/sharevi/etc/exports写入如下内容/home/data/nfs/share *(rw,no_root_squash,sync,no_subtree_check)# 配置生效并查看是否生效exportfs-rexportfs# 启动rpcbind、nfs服务#Centossystemctl restart rpcbindsystemctlenablerpcbind systemctl restart nfssystemctlenablenfs#Ubuntusystemctl restart rpcbindsystemctlenablerpcbind systemctl start nfs-kernel-serversystemctlenablenfs-kernel-server# 查看 RPC 服务的注册状况rpcinfo-plocalhost# showmount测试showmount-elocalhost以上都没有问题则说明安装成功2).k8s注册nfs服务新建storageclass-nfs.yaml文件粘贴如下内容:## 创建了一个存储类apiVersion: storage.k8s.io/v1 kind: StorageClass#存储类的资源名称metadata: name: nfs-storage#存储类的名称自定义annotations: storageclass.kubernetes.io/is-default-class:true#注解是否是默认的存储注意KubeSphere默认就需要个默认存储因此这里注解要设置为“默认”的存储系统表示为true代表默认。provisioner: k8s-sigs.io/nfs-subdir-external-provisioner#存储分配器的名字自定义parameters: archiveOnDelete:true## 删除pv的时候pv的内容是否要备份--- apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default spec: replicas:1#只运行一个副本应用strategy:#描述了如何用新的POD替换现有的PODtype: Recreate#Recreate表示重新创建Podselector:#选择后端PodmatchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner#创建账户containers: - name: nfs-client-provisioner image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2#使用NFS存储分配器的镜像volumeMounts: - name: nfs-client-root#定义个存储卷mountPath: /persistentvolumes#表示挂载容器内部的路径env: - name: PROVISIONER_NAME#定义存储分配器的名称value: k8s-sigs.io/nfs-subdir-external-provisioner#需要和上面定义的保持名称一致- name: NFS_SERVER#指定NFS服务器的地址你需要改成你的NFS服务器的IP地址value:192.168.0.0## 指定自己nfs服务器地址- name: NFS_PATH value: /home/data/nfs/share## nfs服务器共享的目录 #指定NFS服务器共享的目录volumes: - name: nfs-client-root#存储卷的名称和前面定义的保持一致nfs: server:192.168.0.0#NFS服务器的地址和上面保持一致这里需要改为你的IP地址path: /home/data/nfs/share#NFS共享的存储目录和上面保持一致--- apiVersion: v1 kind: ServiceAccount#创建个SA账号metadata: name: nfs-client-provisioner#和上面的SA账号保持一致# replace with namespace where provisioner is deployednamespace: default ---#以下就是ClusterRoleClusterRoleBindingRoleRoleBinding都是权限绑定配置不在解释。直接复制即可。kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups:[]resources:[nodes]verbs:[get,list,watch]- apiGroups:[]resources:[persistentvolumes]verbs:[get,list,watch,create,delete]- apiGroups:[]resources:[persistentvolumeclaims]verbs:[get,list,watch,update]- apiGroups:[storage.k8s.io]resources:[storageclasses]verbs:[get,list,watch]- apiGroups:[]resources:[events]verbs:[create,update,patch]--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default rules: - apiGroups:[]resources:[endpoints]verbs:[get,list,watch,create,update,patch]--- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io服务器地址和共享的目录volumeMounts: - name: nfs-client-root#定义个存储卷mountPath: /persistentvolumes#表示挂载容器内部的路径env: - name: PROVISIONER_NAME#定义存储分配器的名称value: k8s-sigs.io/nfs-subdir-external-provisioner#需要和上面定义的保持名称一致- name: NFS_SERVER#指定NFS服务器的地址你需要改成你的NFS服务器的IP地址value:192.168.224.130## 指定自己nfs服务器地址- name: NFS_PATH value: /home/data/nfs/share## nfs服务器共享的目录 #指定NFS服务器共享的目录volumes: - name: nfs-client-root#存储卷的名称和前面定义的保持一致nfs: server:192.168.224.130#NFS服务器的地址和上面保持一致这里需要改为你的IP地址path: /home/data/nfs/share#NFS共享的存储目录和上面保持一致创建StorageClasskubectl apply-fstorageclass-nfs.yaml# 查看是否存在kubectl get sc创建es存储pvcpv配置文件es-pvc.yamlapiVersion: v1 kind: PersistentVolumeClaim metadata: name: es-pv-claim namespace: kube-system labels: app: es spec: accessModes: - ReadWriteMany storageClassName:nfs-storageresources: requests: storage: 20Gikubectl apply-fes-pvc.yaml创建es-kibana的yaml配置文件: es-statefulset.yamlapiVersion: apps/v1 kind: StatefulSet metadata: labels: app: es-kibana name: es-kibana namespace: kube-system spec: replicas:1selector: matchLabels: app: es-kibana serviceName:es-kibanatemplate: metadata: labels: app: es-kibana spec: containers: - image:[Harbor私库地址]/elasticsearch:7.17.2 imagePullPolicy: IfNotPresent name: elasticsearch resources: requests: memory:800Micpu:800mlimits: memory:1Gicpu:1000mvolumeMounts: - name: es-config mountPath: /usr/share/elasticsearch/config/elasticsearch.yml subPath: elasticsearch.yml - name: es-persistent-storage mountPath: /usr/share/elasticsearch/data env: - name: TZ value: Asia/Shanghai - image:[Harbor私库地址]/kibana:7.17.2 imagePullPolicy: IfNotPresent name: kibana env: - name: TZ value: Asia/Shanghai volumeMounts: - name: kibana-config mountPath: /usr/share/kibana/config/kibana.yml subPath: kibana.yml volumes: - name: es-config configMap: name: es-config - name: kibana-config configMap: name: kibana-config - name: es-persistent-storage persistentVolumeClaim: claimName: es-pv-claim#创建podkubectl create-fes-statefulset.yaml# 查看kubectl get pod-owide-nkube-system|grepes# 使用curl命令测试elasticsearch是否正常kubectl get pod-owide-nkube-system|grepes然后在集群内部服务器上测试是否能通信curl10.244.185.198:9200创建es-kibana cluserip的svces-cluster-none-svc.yamlapiVersion: v1 kind: Service metadata: labels: app: es-kibana name: es-kibana namespace: kube-system spec: ports: - name: es9200 port:9200protocol: TCP targetPort:9200- name: es9300 port:9300protocol: TCP targetPort:9300clusterIP: None selector: app: es-kibana type: ClusterIPkubectl apply-fes-cluster-none-svc.yaml创建es-kibana的nodeport类型的svces-nodeport-svc.yamlapiVersion: v1 kind: Service metadata: labels: app: es-kibana name: es-kibana-nodeport-svc namespace: kube-system spec: ports: - name:9200-9200 port:9200protocol: TCP targetPort:9200#nodePort: 9200- name:5601-5601 port:5601protocol: TCP targetPort:5601#nodePort: 5601selector: app: es-kibana type: NodePortkubectl apply-fes-nodeport-svc.yaml外网暴露的端口是k8s随机分配的使用下面命令可以查看#在服务器使用命令查看kubectl get svc-nkube-system|grepes-kibana可以看到Kibana的端口为31058然后就能使用nodeipport访问检查es是否注册上Kibana点击侧边栏找到堆栈检测然后点Nodes至此Elasticsearch kibana已经搭建完成接下来搭建filebeat。三、搭建filebeat创建filebeat主配置文件filebeat.settings.configmap.ymlapiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: filebeat-config labels: app: filebeat data: filebeat.yml:|- filebeat.inputs: - type: container enabled:truepaths: - /var/log/containers/*.log include_lines:[ERROR,WARN]multiline: pattern: ^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}negate:truematch: after processors: - add_kubernetes_metadata: in_cluster:truehost:${NODE_NAME}matchers: - logs_path: logs_path:/var/log/containers/- add_cloud_metadata: - add_kubernetes_metadata: matchers: - logs_path: logs_path:/var/log/containers/- add_docker_metadata: output.elasticsearch: hosts:[http://[k8s节点ip]:32766]index:filebeat-demo-%{[agent.version]}-%{yyyy.MM.dd}setup.template.name:filebeat-demosetup.template.pattern:filebeat-demo-*setup.ilm.rollover_alias:filebeat-demosetup.ilm: policy_file: /etc/indice-lifecycle.jsonhosts: [“http://[k8s节点ip]:32766”]指定 Elasticsearch 节点的地址和端口。端口号为第二步安装es时nodeport暴露的端口号。#执行kubectl apply-ffilebeat.settings.configmap.yml创建Filebeat索引生命周期策略配置文件为了防止大量的数据存储可以利用 indice 的生命周期来配置数据保留。 如下所示的文件中配置成每天或每次超过5GB的时候就对 indice 进行轮转并删除所有超过30天的 indice 文件。vifilebeat.indice-lifecycle.configmap.yml--- apiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: filebeat-indice-lifecycle labels: app: filebeat data: indice-lifecycle.json:|-{policy:{phases:{hot:{actions:{rollover:{max_size:5GB,max_age:1d}}},delete:{min_age:30d,actions:{delete:{}}}}}}#执行kubectl apply-ffilebeat.indice-lifecycle.configmap.ymlFilebeat操作权限filebeat.permission.ymlapiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: filebeat rules: - apiGroups:[]resources: - namespaces - pods - nodes verbs: - get -watch- list --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ServiceAccount metadata: namespace: kube-system name: filebeat#执行kubectl apply-ffilebeat.permission.ymlFilebeat Daemonset配置文件filebeat.daemonset.yml--- apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: filebeat labels: app: filebeat spec: selector: matchLabels: app: filebeat template: metadata: labels: app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds:30containers: - name: filebeat image:[Harbor私服地址]/filebeat:7.17.2 args:[-c,/etc/filebeat.yml,-e,]env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: runAsUser:0resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly:truesubPath: filebeat.yml - name: filebeat-indice-lifecycle mountPath: /etc/indice-lifecycle.json readOnly:truesubPath: indice-lifecycle.json - name: data mountPath: /usr/share/filebeat/data - name: varlog mountPath: /var/log readOnly:true- name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly:true- name: dockersock mountPath: /var/run/docker.sock volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: filebeat-indice-lifecycle configMap: defaultMode: 0600 name: filebeat-indice-lifecycle - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: dockersock hostPath: path: /var/run/docker.sock - name: data hostPath: path: /var/lib/filebeat-data type: DirectoryOrCreate检查是否执行成功kubectl get pod-owide-nkube-system|grepfilebeat#如下图全为Running则表示运行成功查看是否配置成功首先在侧边栏找到Stack Management选择索引管理查看是否有以filebeat-demo加时间戳为名的索引创建索引模式查看日志点击侧边栏选择discover就能看到Filebeat收集到的容器日志可以按照自己的需求进行日志筛选