DAY-5: Logging with EFK Stack - (ElasticSearch,FluentBit,Kibana)

When working with microservices and distributed systems, logs become the primary source of information to troubleshoot issues. Logs can help us:

• Diagnose errors: Quickly identify issues in different microservices across multiple namespaces.

• Improve performance: Monitor system performance, track latency and identify bottlenecks.

• Ensure security: Identify suspicious activities or failed requests.

Without centralized logging, managing logs from multiple sources across different services and namespaces becomes inefficient. By aggregating logs in one place, we can easily query and analyze them and reducing incident response time.

Components of EFK Stack

The EFK Stack consists of three main components:

1. Elasticsearch (ES): A search and analytics engine that stores the aggregated log data.
   ES uses PV and backed by EBS volumes for data storage/backups

2. Fluentbit: A lightweight log forwarder that collects logs from Kubernetes nodes and forwards them to Elasticsearch.

3. Kibana: A visualization tool that allows us to query and create dashboards for logs stored in Elasticsearch.

FluentBit: DaemonSet vs Sidecar

• DaemonSet (DS): Fluentbit is deployed as a DaemonSet across all nodes in the cluster. This setup is ideal for complex log aggregation as it collects logs both at the node and pod level and forwards them to Elasticsearch. It is more resource-efficient since only one Fluentbit pod per node is needed, reducing overhead, operational complexity, and cost.

• Sidecar Approach: Fluentbit is deployed as a sidecar container alongside each application pod. While this setup provides more isolation, it is less resource-efficient because it consumes more resources by running alongside every application container. Additionally, managing a sidecar deployment can lead to more configuration complexity and overhead.

  Recommendation: For most use cases, especially for log aggregation across the entire cluster, using Fluentbit as a DaemonSet is recommended unless there are specific security requirements or a need for service isolation.

EFK VS ELK stack

• EFK Stack: Elasticsearch, Fluentbit, and Kibana.

• ELK Stack: Elasticsearch, Logstash, and Kibana.

Fluentbit vs Logstash

• Fluentbit: A lightweight and resource-efficient log forwarder that gathers logs from multiple sources and forwards them to Elasticsearch (or other destinations like Splunk).

• Logstash: A more feature-rich log aggregator that offers advanced processing features like filtering, labelling, and transforming logs before forwarding. It is better suited for environments where logs need significant processing before storage.

  Recommendation: For simple log forwarding and aggregation in Kubernetes, Fluentbit is ideal. If you need more advanced log processing, consider using Logstash.

Implementation :

1. Ensure your EKS cluster is running and accessible with managed node group(2 Nodes),OIDC configured.

   

2. Create a Service account for Elasticsearch and attach IAM Role for Service Account (IRSA)

   

   Elasticsearch needs permissions to access AWS resources like EBS volumes for persistent storage. Create an IAM role with necessary permissions and associate it with a Kubernetes service account using IRSA (IAM Roles for Service Accounts).

3. Deploy EBS CSI DRIVER

   

   CSI driver enables creation of PV dynamically to meet the needs of the claim using SC whenever Elasticsearch pod requests storage via PVC

4. Create a logging Namespace

   

5. Deploy ElasticSearch and kibana and expose the kibana service of type LoadBalancer to access via UI

   

   • Installed ElasticSearch, It sets the number of replicas, specifies the storage class, and enables persistence labels to ensure data is stored on persistent volumes.

6. Retrieve Elastisearch username and password to allow FluentBit to forward logs to ES it needs to connect to ES.

   

7. Deploy FluentBit via helm charts by using custom values yaml file with updated ElasticSearch credentials.

   Here is the Repository: https://github.com/iam-veeramalla/observability-zero-to-hero/blob/main/day-5/readme.md

   Fluentbit Configurations:

   • INPUT: Define where Fluentbit collects logs from (e.g., /var/log/containers/*.log).

   • OUTPUT: Define the destination for log forwarding (Elasticsearch).

   • FILTER: Set filters to control which logs to forward (e.g., exclude logs from certain namespaces).

   • SERVICE: Define the service properties of Fluentbit.

Updated the HTTP_Passwd field in the fluentbit-values.yml file with the password retrieved earlier and installed fluentbit in the cluster,

8. Verify everything running in logging Namespace, FluentBit runs as DS and kibana pod running

   

9. Expose kibana svc of type LoadBalancer, Access using AWS ELB DNS on port 5601 in browser and Use the username and password retrieved earlier

   

10. Deploy application stack in cluster

    

11. Create a new app data view to visualize logs from different namespaces or services.

    

    We can also filter out the logs, In the FluentBit configuration file we said to ignore any logs from logging namespace

    

12. Further, Use KQL (Kibana Query Language) to interact with the logs and create custom dashboards for monitoring.

13. CleanUp all the resources and Cluster

    eksctl delete cluster --name observability
    

By implementing the EFK Stack in Kubernetes, we can achieve centralized log aggregation that simplifies troubleshooting, enhances observability, and helps with performance monitoring. Fluentbit, as a lightweight and efficient log forwarder plays an important role in forwarding logs from both nodes and pods to Elasticsearch, where they are stored and visualized in Kibana.