Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11065][Sort] Provides a method to add openTelemetryAppender for the sort connector #11066

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

qy-liuhuo
Copy link
Contributor

In order to more conveniently view the log output, using OpenTelemetry to achieve centralized log reporting

Motivation

In order to more conveniently view the sort task log, we need to achieve a centralized log reporting method.

Modifications

  • Added some Maven dependencies related to openTelemetry at inlong-sort/sort-flink/base/pom.xml and pom.xml
  • Designed and implemented an OpenTelemetryLogger class to add log reporting functionality for sort-connectors

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

The way to integrate OpenTelemetryAppender for connectors that comply with the FLIP-27 standard is as follows:

  1. Create a new OpenTelemetryLogger object in the corresponding SourceReader construction method and configure relevant parameters(log-collector endpoint,logLevel,layout and so on)
  2. Call the install method of the openTelemetryLogger object in the start method of SourceReader
  3. Call the uninstall method of the openTelemetryLogger object in the close method of SourceReader

@qy-liuhuo
Copy link
Contributor Author

Just use it like this:

import org.apache.inlong.sort.base.util.OpenTelemetryLogger;

public class XXXSourceReader<T>
{

    private static final Logger LOG = LoggerFactory.getLogger(MySqlSourceReader.class);

    private final OpenTelemetryLogger openTelemetryLogger;

    public XXXSourceReader() {
        ...
        this.openTelemetryLogger = new OpenTelemetryLogger(); // initialize OpenTelemetryLogger
    }

    @Override
    public void start() {
        openTelemetryLogger.install(); // install OpenTelemetryLogger
        ...
    }

    @Override
    public void close() throws Exception {
        openTelemetryLogger.uninstall(); // uninstall OpenTelemetryLogger
        super.close();
    }
    
    ...
}

And you can configure the opentelemetryAppender by providing some parameters:

OpenTelemetryLogger(String serviceName, String endpoint, Layout<?> layout, Level logLevel)

@qy-liuhuo
Copy link
Contributor Author

A Demo at #11067

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] Provides a method to add openTelemetryAppender for the sort connector
2 participants