Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IOError during tracing when running PG::Connection#exec in a thread #3891

Open
tdeo opened this issue Sep 5, 2024 · 0 comments
Open

IOError during tracing when running PG::Connection#exec in a thread #3891

tdeo opened this issue Sep 5, 2024 · 0 comments
Labels
bug Involves a bug community Was opened by a community member

Comments

@tdeo
Copy link

tdeo commented Sep 5, 2024

Current behaviour

Hello,

We're getting the following IOError when using the parallel gem in the following way:

IOError: stream closed in another thread (IOError)
  from ddtrace (1.23.3) lib/datadog/tracing/contrib/pg/instrumentation.rb:27:in `exec'
  from ddtrace (1.23.3) lib/datadog/tracing/contrib/pg/instrumentation.rb:27:in `block in exec'
  from ddtrace (1.23.3) lib/datadog/tracing/contrib/pg/instrumentation.rb:142:in `block in trace'
  from ddtrace (1.23.3) lib/datadog/tracing/trace_operation.rb:198:in `block in measure'
  from ddtrace (1.23.3) lib/datadog/tracing/span_operation.rb:150:in `measure'
  from ddtrace (1.23.3) lib/datadog/tracing/trace_operation.rb:198:in `measure'
  from ddtrace (1.23.3) lib/datadog/tracing/tracer.rb:385:in `start_span'
  from ddtrace (1.23.3) lib/datadog/tracing/tracer.rb:159:in `block in trace'
  from ddtrace (1.23.3) lib/datadog/tracing/context.rb:45:in `activate!'
  from ddtrace (1.23.3) lib/datadog/tracing/tracer.rb:158:in `trace'
  from ddtrace (1.23.3) lib/datadog/tracing.rb:18:in `trace'
  from ddtrace (1.23.3) lib/datadog/tracing/contrib/pg/instrumentation.rb:105:in `trace'
  from ddtrace (1.23.3) lib/datadog/tracing/contrib/pg/instrumentation.rb:26:in `exec'
  from packs/devtools/db/app/services/devtools/fork_database_helper.rb:217:in `block in analyze'

My usage:

# packs/devtools/db/app/services/devtools/fork_database_helper.rb

def analyze(tables_to_analyze)
  Parallel.each(tables_to_analyze, in_threads: 4) do |table|
    conn = PG::Connection.new(connection_string)
    conn.exec("ANALYZE #{table}") # This is line 217
  ensure
    conn.finish if conn && !conn.finished?
  end
end

Expected behaviour

No error is thrown

Steps to reproduce

Running the code above

Environment

  • datadog version: ddtrace (1.23.3), libdatadog (7.0.0.1.0)
  • Configuration block (Datadog.configure ...):
  Datadog.configure do |c|
    c.diagnostics.debug if ENV['DD_DEBUG'].present?
    c.tracing.partial_flush.enabled = %w[t true 1].include?(ENV['DD_PARTIAL_FLUSH'])

    c.profiling.enabled = true
    c.profiling.advanced.force_enable_gc_profiling = true

    c.env = Rails.env.to_s
    c.service = Rails.application.class.module_parent.name.downcase
    c.version = ENV['SOURCE_VERSION']

    c.tracing.instrument :rails, request_queuing: :exclude_request
    c.tracing.instrument :sidekiq, tag_args: true, service_name: 'sidekiq'
    if ENV['REDIS_URL'].present?
      c.tracing.instrument :redis, describes: { url: ENV['REDIS_URL'] }, service_name: 'redis'
    end
    if ENV['REDIS_CACHE_URL'].present?
      c.tracing.instrument :redis, describes: { url: ENV['REDIS_CACHE_URL'] }, service_name: 'redis-cache'
    end
    c.tracing.instrument :http, split_by_domain: true
    c.tracing.instrument :httpclient, split_by_domain: true
    c.tracing.instrument :stripe
    c.tracing.instrument :aws
    c.tracing.instrument :rake
    c.tracing.instrument :pg, comment_propagation: 'full'

    # Track read-replica separately from primary
    c.tracing.instrument :active_record, describes: :primary_replica, service_name: 'jeancaisse-postgres-replica'

    # Enable DataDog runtime metrics (currently in beta phase)
    c.runtime_metrics.enabled = true

    # Add tags to all traces (documentation: https://datadoghq.dev/dd-trace-rb/#environment-and-tags)
    # The repository URL is hardcoded because it's not expected to change frequently.
    # If it does change, it will break the source code integration for DataDog, which can be easily fixed.
    c.tags = {
      'git.repository_url' => 'https://github.com/pennylane-hq/jeancaisse',
      'git.commit.sha' => ENV['SOURCE_VERSION'],
    }
  end
  • Ruby version: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [aarch64-linux]
  • Operating system:: aarch64 GNU/Linux running in a Docker image originating from ruby:3.3.4-slim-bookworm
  • Relevant library versions: parallel (1.26.3)
@tdeo tdeo added bug Involves a bug community Was opened by a community member labels Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Involves a bug community Was opened by a community member
Projects
None yet
Development

No branches or pull requests

1 participant