This project demonstrates a Go application integrated with a comprehensive observability stack, including distributed tracing, metrics collection, and load testing.
- Go Application: A simple HTTP service with Swagger documentation
- OpenTelemetry: For distributed tracing
- Jaeger: For trace visualization
- Prometheus: For metrics collection
- Grafana: For metrics and trace visualization
- Locust: For load testing
- Docker and Docker Compose
- Go 1.16 or later (for local development)
-
Clone this repository:
git clone [email protected]:CoryBarney/grafana-otel-go-example.git cd grafana-otel-go-example -
Build and start the services:
docker-compose up -d -
Access the services:
- Go Application: http://localhost:8080
- Swagger UI: http://localhost:8080/swagger/index.html
- Grafana: http://localhost:3000
- Jaeger UI: http://localhost:16686
- Prometheus: http://localhost:9090
- Locust: http://localhost:8089
docker-compose.yml: Defines all services and their configurationsDockerfile: Builds the Go application containerotel-collector-config.yaml: Configures the OpenTelemetry Collectorprometheus.yml: Configures Prometheus scrape targetsgrafana-datasources.yaml: Configures Grafana data sourcesgrafana-dashboards.yaml: Configures Grafana dashboard provisioningdashboards/go-app-dashboard.json: Grafana dashboard for the applicationlocustfile.py: Defines the load testing scenarios
To work on the Go application locally:
-
Change to application directory:
cd app -
Install dependencies:
go mod download -
Run the application:
go run main.go -
To update Swagger documentation:
swag init -
To run tests:
go test -v
Note: The tests use a no-op tracer to handle tracing in the test environment. This allows the tests to run without a full OpenTelemetry setup.
Locust is configured to start automatically and run a load test against the application. To modify the load test:
- Edit
locustfile.py - Restart the Locust service:
docker-compose restart locust
Note: The Locust service is configured with an increased open file limit (65536) to support high-volume load testing. If you're running this setup on a system with lower limits, you may need to adjust the ulimits configuration in the docker-compose.yml file.
- Metrics: View in Grafana or query directly in Prometheus
- Traces: View in Jaeger UI or in Grafana (if configured)
- Logs: View docker logs:
docker-compose logs -f
- To add new metrics, update the Go application and Prometheus configuration
- To modify tracing, update the OpenTelemetry configuration in the Go application
- To add or modify Grafana dashboards, edit the JSON files in the
dashboardsdirectory
- If services fail to start, check the logs:
docker-compose logs <service-name> - Ensure all required ports are available on your host machine
- Verify that all configuration files are present and correctly formatted
-
Connection refused errors: Ensure that the service names in your Docker Compose file match the hostnames used in your configurations.
-
Locust file limit warning: If you see a warning about the system open file limit being too low, check that the
ulimitsconfiguration in the Docker Compose file is applied correctly. -
No data in Grafana: Verify that Prometheus and Jaeger are successfully scraping data from your application. Check the Prometheus targets page and Jaeger search page to confirm data is being collected.
-
Permission Denied on Mounted Files: Vertify Context and Setting of Selinux or AppArmour (
sudo setenforce 0)
- The current setup is designed for local development and testing. For production use, you would need to adjust configurations for security and scalability.
- Monitor the resource usage of your containers, especially during load testing, to ensure your host system can handle the load.
- Consider adjusting the retention periods and sample rates in Prometheus and Jaeger for longer-term data storage in a production environment.
Below is a high-level overview of the system architecture:
This diagram illustrates the flow of data between the different components of the system:
- Locust sends HTTP requests to the Go Application for load testing.
- The Go Application processes requests and generates observability data.
- Metrics are sent to Prometheus for storage and querying.
- Traces are sent to the OpenTelemetry Collector, which forwards them to Jaeger.
- Grafana collects data from both Prometheus and Jaeger for visualization and monitoring.
This project uses GitHub Actions for continuous integration. On every push to the main branch and for every pull request, the following steps are automatically run:
- Swagger documentation is generated
- Go tests are executed
go vetis run to check for potential issuesstaticcheckis used for additional static analysis
You can see the detailed configuration in the .github/workflows/go-tests.yml file.
To run these checks locally:
cd app
swag init
go test -v ./...
go vet ./...
staticcheck ./... # You need to install staticcheck first: go install honnef.co/go/tools/cmd/staticcheck@latestNote: Make sure you have swag installed locally to generate Swagger docs: go install github.com/swaggo/swag/cmd/swag@latest
Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.
This project is licensed under the MIT License - see the LICENSE.md file for details.
- OpenTelemetry community for providing excellent documentation and tools
- Grafana Labs for their powerful visualization platform
- Jaeger team for their distributed tracing system
- Locust team for their user-friendly load testing tool
- OpenTelemetry Documentation
- Jaeger Documentation
- Prometheus Documentation
- Grafana Documentation
- Locust Documentation
For any additional questions or support, please open an issue in the project repository.