How OpenTelemetry Facilitated the Instrumentation of Hardware Sentry
Starting from version 3.0.00, Hardware Sentry is observable. Emitting traces and metrics would not have been possible without instrumenting the solution. Find out how OpenTelemetry made it possible.
Maxence Crouzy is a 4th year student at EPITA, the first engineering school in computer science in Paris. He joined Sentry Software as an intern in September 2022. Assisted by Nassim Boutekedjiret, Software Architect, he has instrumented Hardware Sentry with OpenTelemetry. He agreed to share his experience in this post.
Instrumenting Hardware Sentry with OpenTelemetry is an ambitious project. Can you tell us how this project started?
Hardware Sentry is a powerful monitoring solution capable of reporting the health status of any internal component that composes a server, a storage system, or a network switch. Oddly though, Hardware Sentry was not that good at reporting its own health status. When a problem occurred with the Hardware Sentry Agent, we could not get a big picture of what was happening. This was inadmissible for an observability solution of such renown. Hardware Sentry needed to be observable, and therefore instrumented, to provide final users with traces, metrics, and logs.
How did you proceed?
I first did some research to find the best method to instrument Hardware Sentry; a method that would not be too intrusive. Nassim and I first considered Aspect Oriented Programming using Spring in Java as it was already used in Hardware Sentry. We made our first proof of concept. While promising, this method has not been approved by the team.
Why has Aspect Oriented Programming been rejected?
Well, it was mostly because Aspect Oriented Programming was not a standardized method and did not fit in with the ambition of Hardware Sentry: comply as much as possible with OpenTelemetry. We therefore turned to OpenTelemetry.
Does OpenTelemetry allow code instrumentation?
Yes, it does. Two methods exist in OpenTelemetry: the manual instrumentation and automatic instrumentation. We tested both and finally opted for the automatic instrumentation since it is the simplest one.
What was required for Hardware Sentry to be observable?
To be observable, Hardware Sentry needed an instrumentation agent. OpenTelemetry provides a Java agent JAR, opentelemetry-javaagent.jar, which can be attached to any Java application. This agent is not provided by default and needs to be downloaded from the opentelemetry-java-instrumentation repository before being included into Hardware Sentry. This is transparent for our users who just have to follow a few configuration steps to enable OpenTelemetry traces.
How to enable OpenTelemetry traces for Hardware Sentry?
To enable OpenTelemetry traces, our users just comment out the traces pipeline inside the otel/otel-config.yaml file, configure the exporter, add Java options to properly link the OpenTelemetry Java Agent to the Hardware Sentry Agent, configure the service and the exporter and that is! All these steps are detailed in the Hardware Sentry user documentation.
How did the project go?
Instrumenting Hardware Sentry with OpenTelemetry turned out to be pretty easy. After a few months of research and work we have been able to make Hardware Sentry emit its own traces and metrics. It did not require significant changes in the source code. We only had to annotate the code on the functions and parameters we wanted to trace. We did not face major problems.
What traces does Hardware Sentry emit?
Hardware Sentry emits several traces which track every protocol request and describe the internal “path” that Hardware Sentry takes to execute its internal tasks. Those traces offer a clear view of what happens when a problem occurs and allow users to easily track down the arising problem. Those traces can be sent to any observability backend that natively supports OpenTelemetry such as Datadog or Grafana Tempo.
What about metrics?
Hardware Sentry emits several java metrics such as the memory used, requested, and committed, the number of loaded and unloaded classes, the number of executing threads, the CPU utilization, etc.
Does Hardware Sentry emit logs?
We considered emitting logs, but this feature was not provided by the SDK. Emitting logs is currently only possible with manual instrumentation. This would have required a significant amount of work. We thus chose not to implement it in v3.0.00. Logging should soon be available through automatic instrumentation.
It was your first time working with OpenTelemetry. What did you think about it?
I must admit that I was pleasantly surprised. OpenTelemetry is extremely powerful. It is the first standard in the observability world. I found that instrumentation was easy to integrate. Since few changes are brought to the code and they are easy to read, it is easy for anyone to resume the project and get up to speed quickly. Additionally, the feature is well documented. OpenTelemetry is the future!
Now that Hardware Sentry is observable, what's coming next?
The next step is displaying traces and metrics into the Hardware Sentry Dashboards for Datadog and for Grafana. I am working on it and hope to complete this by the end of my internship.
Thanks to Maxence for sharing his experience in instrumenting Hardware Sentry with OpenTelemetry. To learn more about this new feature introduced in Hardware Sentry v3.0.00, book a session with one of our expert.