DAY-6: Distributed Tracing using Jaeger

What is Jaeger?

Jaeger is an open-source, end-to-end distributed tracing system used for monitoring and troubleshooting microservices-based architectures. It helps developers understand how requests flow through a complex system, by tracing the path a request takes and measuring how long each step in that path takes.

Why Use Jaeger?

In modern applications, especially microservices architectures, a single user request can touch multiple services. When something goes wrong, it’s challenging to pinpoint the source of the problem. Jaeger helps by:
Identifying bottlenecks: See where your application spends most of its time.
Finding root causes of errors: Trace errors back to their source.
Optimizing performance: Understand and improve the latency of services.

Setting Up Jaeger

Step 1: Instrumenting Your Code

To start tracing, you need to instrument your services. This means adding tracing capabilities to your code. Most popular programming languages and frameworks have libraries or middleware that make this easy.
We have already instrumented our code using OpenTelemetry libraries/packages. For more details, refer to https://github.com/iam-veeramalla/observability-zero-to-hero/blob/main/day-4/application/service-a/tracing.js

Step 2: Components of Jaeger

Jaeger consists of several components:
Agent: Collects traces from your application.
Collector: Receives traces from the agent and processes them.
Query: Provides a UI to view traces.
Storage: Stores traces for later retrieval (often a database like Elasticsearch).

Step 3: Implementation

Make sure the EKS cluster is running with managed node group(2 nodes), OIDC configured.
Create IAM role for Service Account

IAM role allows EBS CSI controller to interact with AWS resources, specifically for managing EBS volumes in the Kubernetes cluster.
Deploy EBS CSI driver using previously created IAM service account role to allow the driver to manage EBS volumes securely.
Install ElasticSearch and retrieve username and password
Export ElasticSearch CA certificate
```
 kubectl get secret elasticsearch-master-certs -n logging -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca-cert.pem
```
This command retrieves the CA certificate from the Elasticsearch master certificate secret and decodes it, saving it to a ca-cert.pem file. We use this same file to create configmap and secret for Jaeger to use to connect to Elasticsearch(mTLS).
Create Tracing Namespace, ConfigMap for Jaeger’s TLS certificate

Create Secret for Elasticsearch TLS

 kubectl create secret generic es-tls-secret --from-file=ca-cert.pem -n tracing

Add Jaeger Helm Repository and install using custom values.yaml with the ElasticSearch password
Port Forward Jaeger Query Service

This allows port forwarding 8080 on your local machine to the Jaeger Query service, allowing you to access the Jaeger UI locally.
Deploy Application stack which were instrumented using opentelemetry in dev namespace
Access the service using AWS ELB DNS in browser and identify traces coming in Jaeger UI
Cleanup all resources and cluster
```
eksctl delete cluster --name observability
```
Why do Jaeger and Fluent Bit handle authentication and TLS differently when connecting to databases like Elasticsearch or Cassandra?

Jaeger implements HTTPS authentication with TLS by default when communicating with its storage backend.Implements mTLS handshake while Jaeger needs a secret to authenticate with the db, database needs a ConfigMap or secret to connect back securely.

Fluent Bit runs as a DaemonSet (DS) and supports both HTTP and HTTPS for sending logs. By default, it uses HTTP and relies on username/password authentication to connect securely to Elasticsearch. Unlike Jaeger, Fluent Bit is designed to authenticate using credentials (username/password) rather than enforcing TLS by default.If you enable TLS encryption, Fluent Bit can use HTTPS instead of HTTP