Help debugging mcrypt issue - $@ 150,000

[kevkevinpal]

[Description]

Our team is having trouble getting debugging an mcrypt dependency issue when running npm install

[Complexity] (1,2,3,5,7)

2

[To Reproduce]

Download the repo

run npm install

it should fail and you should see the error below

[Acceptance Criteria]

Given We have the repo downloaded clean

When we run the code npm install

Then it should install properly with no mcrypt error

[antonilol]

when running npm install on a fresh git clone
i get with node v17.7.1 and npm 8.5.4 a grpc compile error
with node v16.13.2 and npm 8.1.2 it installs all dependencies

with both node versions mcrypt leaves a message:

npm WARN deprecated [email protected]: Please use cryptian package instead of this package

in your log i see a permission denied, so try chmod +xing that script (i think it wants to run that script) or if it tries to write there you probably need sudo

[kevkevinpal]

when running npm install on a fresh git clone i get with node v17.7.1 and npm 8.5.4 a grpc compile error with node v16.13.2 and npm 8.1.2 it installs all dependencies

with both node versions mcrypt leaves a message:

npm WARN deprecated [email protected]: Please use cryptian package instead of this package

in your log i see a permission denied, so try chmod +xing that script (i think it wants to run that script) or if it tries to write there you probably need sudo

So I think instead we're trying to remove mcrypt and use cryptian instead

w1ntermute is taking on the testing I'm not sure if you're in the developing on sphinx tribe that's where the discussion is happening

latest changes are made to this lib https://github.com/evanfeenstra/JSCryptor to use cryptian

[antonilol]

I'm not sure if you're in the developing on sphinx tribe

yes im in there, i see its mac specific so that is why i didnt get the error (i use linux)

[kevkevinpal]

Any idea what maybe causing this not sure if the versioning of node/npm is the issue this started occurring when I was trying to replicate the issue when the test suite was failing

[antonilol]

(screenshot of your screenshot)

looks like an old npm (and probably also old nodejs)

[kevkevinpal]

I think I figured it out, npm .npmrc file was giving an explicit path for node-gyp which was wrong.

I don't remember ever editing that file tho ¯_(ツ)_/¯

[kevkevinpal]

Trying to upgrade a node cluster from nodejs 12 -> 16 and I'm getting the mcrypt error again

npm ERR! code 1
npm ERR! path /relay/node_modules/mcrypt
npm ERR! command failed
npm ERR! command sh -c node-gyp rebuild
npm ERR! make: Entering directory '/relay/node_modules/mcrypt/build'
npm ERR! make: Leaving directory '/relay/node_modules/mcrypt/build'
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | arm64
npm ERR! gyp info find Python using Python version 3.8.2 found at "/usr/bin/python3"
npm ERR! gyp info spawn /usr/bin/python3
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/relay/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/relay/node_modules/mcrypt/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/relay/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/root/.cache/node-gyp/16.17.1/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/root/.cache/node-gyp/16.17.1',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/relay/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/root/.cache/node-gyp/16.17.1/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/relay/node_modules/mcrypt',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info spawn make
npm ERR! gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
npm ERR! make: printf: Operation not permitted
npm ERR! make: *** [lib/libmcrypt/libmcrypt.target.mk:175: Release/obj.target/libmcrypt/lib/libmcrypt/modules/algorithms/3-way.o] Error 127
npm ERR! gyp ERR! build error
npm ERR! gyp ERR! stack Error: `make` failed with exit code: 2
npm ERR! gyp ERR! stack     at ChildProcess.onExit (/relay/node_modules/node-gyp/lib/build.js:194:23)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
npm ERR! gyp ERR! System Linux 4.14.209-160.339.amzn2.aarch64
npm ERR! gyp ERR! command "/usr/bin/node" "/relay/node_modules/.bin/node-gyp" "rebuild"
npm ERR! gyp ERR! cwd /relay/node_modules/mcrypt
npm ERR! gyp ERR! node -v v16.17.1
npm ERR! gyp ERR! node-gyp -v v8.4.1
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-01-31T17_58_14_234Z-debug-0.log

[antonilol]

with this exact node version (v16.17.1, and npm 8.15.0 that comes with it) i dont get this error when running npm install. almost all things i can find on the internet say that there are missing dependencies (i must have them on my system because the build does not fail)
i see you use Linux 4.14, that probably works but other dependencies that old could cause problems. search results say this error can occur with an outdated docker engine (i assume this runs in docker)
also, everywhere Alpine linux (a distro) is mentioned. do you use it in here?

[kevkevinpal]

yea I think the cluster is using this docker file https://github.com/stakwork/sphinx-relay/blob/master/Dockerfile.lndrelay which means its using golang:1.13-alpine

I'm trying to update the containers in place because taking them down and bringing them back up changes the container id which we need.

I'm using this script to update each container

for container in `docker ps -q`; do
  containername=$(docker inspect --format='{{.Name}}' $container);
  if [[ $containername != *"traefik"* ]]; then
    if [[ $containername != *"ecs-agent"* ]]; then
      if [[ $container != "eacb55362b87" || $container != "d92f419f77a0" ]]; then
        echo "=====Starting node update for container: $container======"
        docker exec -it $container bash -c 'rm core.*' 
        docker exec -it $container bash -c 'supervisorctl stop relay' 
        docker exec -it $container bash -c 'rm -rf node_modules' 
        docker exec -it $container bash -c 'supervisorctl stop lnd' 
        docker exec -it $container bash -c 'echo "db.bolt.auto-compact=true" >> /relay/.lnd/lnd.conf' 
        docker exec -it $container bash -c 'supervisorctl start lnd' 
       #Sleep for 30 seconds to let db compact
        sleep 30
        docker exec -it $container bash -c 'echo "http://dl-cdn.alpinelinux.org/alpine/v3.16/main" > /etc/apk/repositories' 
        docker exec -it $container bash -c 'echo "http://dl-cdn.alpinelinux.org/alpine/v3.16/community" >> /etc/apk/repositories' 
        docker exec -it $container bash -c 'apk del nodejs-npm' 
        docker exec -it $container bash -c 'apk del nodejs' 
        docker exec -it $container bash -c 'apk add -X https://dl-cdn.alpinelinux.org/alpine/v3.16/main -u alpine-keys --allow-untrusted' 
        docker exec -it $container bash -c 'apk update && apk upgrade' 
        docker exec -it $container bash -c 'ln -s /usr/bin/node /usr/local/bin/node -f' 
        docker exec -it $container bash -c 'apk add nodejs' 
        docker exec -it $container bash -c 'apk add npm' 
        docker exec -it $container bash -c 'node --version' 
        docker exec -it $container bash -c 'python3 -m pip install supervisor' 
        docker exec -it $container bash -c 'git fetch -a origin' 
        docker exec -it $container bash -c 'git fetch origin pull/423/head:PR423' 
        docker exec -it $container bash -c 'git stash' 
        docker exec -it $container bash -c 'git checkout PR423' 

        docker exec -it $container bash -c 'npm install' 
        docker exec -it $container bash -c 'supervisorctl start relay'
        echo "=====Finished node update for container: $container======"
	fi
    fi
  fi
done

let me know if there's anything you recommend me trying

[antonilol]

this answer i found on SO looks similar (i think bycrypt is similar to mcrypt) with the same error make: printf: Operation not permitted

in this answer the problem would be solved by updating/installing dependencies with the following commands:

apk add --update alpine-sdk
apk add libffi-dev openssl-dev
apk add python-dev python3-dev

could you try running these in the update script?

[antonilol]

taking them down and bringing them back up changes the container id which we need

just curious, what depends on the container id staying the same?

[kevkevinpal]

I'm getting this error when trying to add those dependencies

ERROR: unable to select packages:
  libffi-3.4.2-r1:
    conflicts: libffi-3.2.1-r6
    satisfies: libffi-dev-3.4.2-r1[libffi=3.4.2-r1]
               python3-3.10.9-r0[so:libffi.so.8]
  libffi-3.2.1-r6:
    conflicts: libffi-3.4.2-r1
    breaks: libffi-dev-3.4.2-r1[libffi=3.4.2-r1]
    satisfies: python2-2.7.18-r0[so:libffi.so.6]
ERROR: unable to select packages:
  python-dev (no such package):
    required by: world[python-dev]
  libffi-3.2.1-r6:
    conflicts: libffi-3.4.2-r1
    satisfies: python2-2.7.18-r0[so:libffi.so.6]
  libffi-3.4.2-r1:
    conflicts: libffi-3.2.1-r6
    satisfies: python3-3.10.9-r0[so:libffi.so.8]

taking them down and bringing them back up changes the container id which we need

just curious, what depends on the container id staying the same?

We have them in a spreadsheet linking to who's using which node for customer support and etc, if we lost whos node is connected to which container it would be hard to debug if we needed to know which node is mapped to which user

[kevkevinpal]

Fyi the clusters I've been updating previously didn't have any issue just when I got to these two it started to give me this mcrypt error

[kevkevinpal]

I'm not sure if this might be the issue
npm ERR! gyp info using [email protected]

in the docker file it seems we're installing [email protected]

is the build when doing npm install trying to use a version of node-gyp that wont work?

[Evanfeenstra]

We really should not have mcrypt as a dependency at all.... its only needed by jscryptor, but i refactored that away from mcrypt a long time ago https://github.com/Evanfeenstra/JSCryptor

[antonilol]

We really should not have mcrypt as a dependency at all.... its only needed by jscryptor, but i refactored that away from mcrypt a long time ago https://github.com/Evanfeenstra/JSCryptor

why is mcrypt installed then? is it also a dependency for a different package? i dont see it directly required in package.json

[kevkevinpal]

@Evanfeenstra I see that you have mcrypt taken out on your own fork but are we still using the default JSCryptor I don't see it pointing to your fork in the package.json

*nvm it is your fork didn't realize you're fognet*

[kevkevinpal]

So I was able to make progress by removing mcrypt dependency from this PR chesstrian/JSCryptor#23 and then publishing to npm

Able to run npm install and I'm going to test that images are still working as intended to but will post what exactly I'm doing when I test

[kevkevinpal]

@Evanfeenstra you said jscryptor was only being used for paid meme images, from what I tested so far I was able to send a paid meme image and have it paid and decrypted. Both on different versions of jscryptor

I can try the other way around too

[kevkevinpal]

hmm seems like the new changes on jscryptor is breaking the image tests which is not good

kevkevinpal#23

[kevkevinpal]

closing this issue as I think its resolved now, haven't seen in a while