not binary compatible

[zhongshan365]

I want to integrate opentracing into kong. But i got follow error :
/usr/local/share/lua/5.1/kong/cmd/start.lua:24: nginx configuration is invalid (exit code 1):
nginx: [emerg] module "/usr/local/kong/nginx/modules/ngx_http_opentracing_module.so" is not binary compatible in /usr/local/kong/nginx.conf:10
nginx: configuration file /usr/local/kong/nginx.conf test failed

I use follow versions:
kong version: 2.6.0
nginx version: openresty/1.19.9.1
ngx_http_module: linux-amd64-nginx-1.19.9-ngx_http_module.so.tgz

Nginx version is same to ngx_http_module, but it's do not working.

Please help me. Thanks

[chrisforrette]

I'm currently struggling with this exact issue as well. I've attempted a few approaches...

Binary install

I was trying to install the binary in a Docker image like so (checking nginx -V showed me nginx version: openresty/1.19.3.2 so I'm using the 1.19.3 binary):

FROM kong:2.5.1-alpine

RUN wget -P /tmp https://github.com/opentracing-contrib/nginx-opentracing/releases/download/v0.21.0/linux-amd64-nginx-1.19.3-ot16-ngx_http_module.so.tgz
RUN tar -zxvf /tmp/linux-amd64-nginx-1.19.3-ot16-ngx_http_module.so.tgz -C /usr/local/openresty/nginx/modules/
RUN chown kong:root /usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so

ENV KONG_NGINX_MAIN_LOAD_MODULE=/usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so

And got this:

nginx: [emerg] dlopen() "/usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so" failed (Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so)) in /usr/local/kong/nginx.conf:6

And I gathered from this issue: #72 that this is because the alpine is lacking some shared libs. I tried to install a few additional dependencies but it didn't seem resolvable without compiling from source, which I'd like to avoid.

Docker multi-stage builds

From the same GitHub issue, I saw that there is now an alpine Docker image (as of ~1 month ago) so I attempted to use that via a multi-stage build approach:

FROM opentracing/nginx-opentracing:edge-alpine AS opentracing-builder

FROM kong:2.5.1-alpine
COPY --from=opentracing-builder /usr/local/lib/libopentracing.so.1.6.0 /usr/local/lib/
RUN \
  ln -s /usr/local/lib/libopentracing.so.1.6.0 /usr/local/lib/libopentracing.so.1  && \
  ln -s /usr/local/lib/libopentracing.so.1 /usr/local/lib/libopentracing.so
COPY --from=opentracing-builder /usr/local/lib/libopentracing.a /usr/local/lib/
COPY --from=opentracing-builder /usr/local/lib/libopentracing.so /usr/local/openresty/nginx/modules/
COPY --from=opentracing-builder /usr/local/lib/libopentracing.a /usr/local/openresty/nginx/modules/
COPY --from=opentracing-builder /etc/nginx/modules/ngx_http_opentracing_module.so /usr/local/openresty/nginx/modules/

ENV KONG_NGINX_MAIN_LOAD_MODULE=/usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so

But that resulted in this error:

nginx: [emerg] module "/usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so" version 1021004 instead of 1019003 in /usr/local/kong/nginx.conf:6

I believe this is because the Nginx versions aren't the same between what ships with Kong and what's baked into the opentracing/nginx-opentracing:edge-alpine. Since there are so many Nginx-version-specific releases of the binary, I'm assuming Nginx is very finicky about version. I noticed an NGINX_VERSION environment variable in the Dockerfile here but it's not called out as an ARG so I don't know how it gets set. But I tried building an image locally with this:

export NGINX_VERSION=1.19.3
docker build -t opentracing-alpine-local . -f --build-arg BUILD_OS=alpine --build-arg NGINX_VERSION=1.19.3 --target=final

...and tried the multi-stage build again but got the same error.

---

I'm a bit stuck, but I've seen mention of loading modules dynamically for openresty, and I see there's an openresty-specific Dockerfile/image but it looks complicated and I don't know the openresty or Lua landscape well so I'm not sure what the best/simplest approach is.

I'm hoping to find a minimally custom way to install this and ship APM to Datadog and would love some guidance, suggestions, or eyeballs on my examples here to point out if I'm missing something.

[lucacome]

@chrisforrette
NGINX_VERSION in the Dockerfile here https://github.com/opentracing-contrib/nginx-opentracing/blob/master/Dockerfile#L226 is an env variable that is already inside the container, it's not read from the build arguments.

You would have to manually change https://github.com/opentracing-contrib/nginx-opentracing/blob/master/Dockerfile#L213 and https://github.com/opentracing-contrib/nginx-opentracing/blob/master/Dockerfile#L241 to FROM nginx:1.19.3-alpine as ... because right now there's not a way to customize the version of nginx with build arguments.

If you change those two lines using the version you need, it should work (I haven't tested it)

[miry]

You can check how ingress-nginx is building: https://github.com/kubernetes/ingress-nginx/blob/main/images/nginx/rootfs/build.sh

[chrisforrette]

@lucacome Thanks for the info! I tried hard-coding the Nginx version to 1.19.3 in the Dockerfile in this repo, building it, then referencing it in my Dockerfile but ended up with the same is not binary compatible error I got when installing the binary:

Error: 
/usr/local/share/lua/5.1/kong/cmd/prepare.lua:12: could not prepare Kong prefix at /usr/local/kong: nginx configuration is invalid (exit code 1):
nginx: [emerg] module "/usr/local/openresty/nginx/modules/ngx_http_opentracing_module.so" is not binary compatible in /usr/local/kong/nginx.conf:6
nginx: configuration file /usr/local/kong/nginx.conf test failed

stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/cmd/prepare.lua:12: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
	/usr/local/bin/kong:9: in function 'file_gen'
	init_worker_by_lua:51: in function <init_worker_by_lua:49>
	[C]: in function 'xpcall'
	init_worker_by_lua:58: in function <init_worker_by_lua:56>

A cursory google search tells me I need to remove --with-compat from the nginx setup but that's easier said than done as that's buried in Kong somewhere.

I'm also wondering if it needs to be compiled and/or loaded differently with Lua...

[chrisforrette]

@miry Thanks for sharing that! I'm hoping to avoid building from scratch but I might end up there...

[lucacome]

@chrisforrette I tried building the module with this Dockerfile for Alpine from Openresty https://github.com/openresty/docker-openresty/blob/1.19.9.1-3/alpine/Dockerfile but still says that is not binary compatible. I'm not really familiar with openresty or kong, so I think I'm out of ideas for now 😞

[chrisforrette]

So since Kong = OpenResty + Lua + Nginx, and nginx-opentracing is also on top of Nginx, we've taken a variety of approaches to try to compile everything in the right way and can't seem to get them to play nicely so we're resorting to writing a custom Kong plugin for tracing 😕

Thanks anyway for your help @lucacome!

[edify42]

I did notice when we tried using the debian docker images they weren't built with the --with-compat option - simple search of the openresty docker github

At the time of writing, only the alpine, focal and bionic (alpine + ubuntu) images had the nginx build option attached. I went through the commit history for the debian buster image but I don't think it ever had the option applied.

The fix for us was simply to switch over to ubuntu builds and I suspect the same could be done for a kong build @zhongshan365 and @chrisforrette