The documentation you are viewing is for Dapr v1.2 which is an older version of Dapr. For up-to-date documentation, see the latest version.

操作方法:在 Kubernetes 中搭建 Fluentd、Elastic search 和 Kibana

如何在Kubernetes安装Fluentd、Elastic Search和Kibana来搜索日志

先决条件

安装 Elasticsearch 和 Kibana

  1. Create namespace for monitoring tool and add Helm repo for Elastic Search

    kubectl create namespace dapr-monitoring
    
  2. Add Elastic helm repo

    helm repo add elastic https://helm.elastic.co
    helm repo update
    
  3. 使用 Helm 安装 Elastic Search

By default the chart creates 3 replicas which must be on different nodes. If your cluster has less than 3 nodes, specify a lower number of replicas. For example, this sets it to 1:

helm install elasticsearch elastic/elasticsearch -n dapr-monitoring --set replicas=1

Otherwise:

helm install elasticsearch elastic/elasticsearch -n dapr-monitoring

If you are using minikube or want to disable persistent volumes for development purposes, you can disable it by using the following command:

helm install elasticsearch elastic/elasticsearch -n dapr-monitoring --set persistence.enabled=false,replicas=1
  1. Install Kibana

    helm install kibana elastic/kibana -n dapr-monitoring
    
  2. 校验

    Ensure Elastic Search and Kibana are running in your Kubernetes cluster.

    kubectl get pods -n dapr-monitoring
    NAME                            READY   STATUS    RESTARTS   AGE
    elasticsearch-master-0          1/1     Running   0          6m58s
    kibana-kibana-95bc54b89-zqdrk   1/1     Running   0          4m21s
    

安装 Fluentd

  1. Install config map and Fluentd as a daemonset

Download these config files:

Note: If you already have Fluentd running in your cluster, please enable the nested json parser to parse JSON formatted log from Dapr.

Apply the configurations to your cluster:

kubectl apply -f ./fluentd-config-map.yaml
kubectl apply -f ./fluentd-dapr-with-rbac.yaml
  1. Ensure that Fluentd is running as a daemonset; the number of instances should be the same as the number of cluster nodes. In the example below we only have 1 node.
kubectl get pods -n kube-system -w
NAME                          READY   STATUS    RESTARTS   AGE
coredns-6955765f44-cxjxk      1/1     Running   0          4m41s
coredns-6955765f44-jlskv      1/1     Running   0          4m41s
etcd-m01                      1/1     Running   0          4m48s
fluentd-sdrld                 1/1     Running   0          14s

使用 JSON 格式化日志安装 Dapr

  1. 使用 JSON 格式化日志启用 Dapr

    helm repo add dapr https://dapr.github.io/helm-charts/
    helm repo update
    helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
    
  2. Enable JSON formatted log in Dapr sidecar

添加 dapr.io/log-as-json: "true" annotation 到你的部署yaml.

示例:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pythonapp
  namespace: default
  labels:
    app: python
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python
  template:
    metadata:
      labels:
        app: python
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "pythonapp"
        dapr.io/log-as-json: "true"
...

搜索日志

Note: Elastic Search takes a time to index the logs that Fluentd sends.

  1. Port-forward to svc/kibana-kibana
$ kubectl port-forward svc/kibana-kibana 5601 -n dapr-monitoring
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
Handling connection for 5601
Handling connection for 5601
  1. Browse http://localhost:5601

  2. Click Management -> Index Management

kibana management

  1. Wait until dapr-* is indexed.

index log

  1. Once dapr-* indexed, click Kibana->Index Patterns and Create Index Pattern

create index pattern

  1. Define index pattern - type dapr* in index pattern

define index pattern

  1. Select time stamp filed: @timestamp

timestamp

  1. Confirm that scope, type, app_id, level, etc are being indexed.

Note: if you cannot find the indexed field, please wait. it depends on the volume of data and resource size where elastic search is running.

indexing

  1. Click discover icon and search scope:*

Note: it would take some time to make log searchable based on the data volume and resource.

discover

参考资料