Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom build python package and missing environment variable in reticulate-python #1666

Open
matthewgson opened this issue Sep 12, 2024 · 10 comments

Comments

@matthewgson
Copy link

matthewgson commented Sep 12, 2024

I’m not sure if this should be classified as a bug or a misunderstanding on my part, so feel free to close this issue if it’s not a bug.

Since it’s hard to create a reproducible example, let me describe the scenario: I built a custom C++ library, QuantLib, (not using the pip version since I need intraday calculations) on my server in ~/local/(as I don’t have sudo privileges). I installed the Python bindings (via SWIG) for this package in my reticulate environment, which was created using install_miniconda(), and is located at ~/.local/share/r-miniconda/envs/r-reticulate.

I set up the package building and binding in bash using the following environment variables.
The package and everything works just as expected when running the python from the bash terminal:

export PATH=$HOME/local/bin:$PATH
export CPPFLAGS="-I$HOME/local/include"
export LDFLAGS="-L$HOME/local/lib"
export LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH
export PATH=~/.local/share/r-miniconda/envs/r-reticulate/bin:$PATH

However, when I try to run Python from R using reticulate repl_python(), I get an ImportError, which seems related to the PATH or LD_LIBRARY_PATH environment variable:

import QuantLib as ql
# ImportError: libQuantLib.so.1: cannot open shared object file: No such file or directory

libQuantLib.so.1 file is located at ~/local/lib as above LD_LIBRARY_PATH specifies.

I’ve tried setting these environment variables in R via .Renviron, and also directly from Python (from R-reticulate) using:

os.environ["LD_LIBRARY_PATH"] = $HOME/local/lib:$LD_LIBRARY_PATH

But the error persists.

What could be causing this issue? Below is the full error trace from R:

>>> import QuantLib
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/home/ms/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/QuantLib-1.35-py3.10-linux-x86_64.egg/QuantLib/__init__.py", line 19, in <module>
    from .QuantLib import *
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
  File "/home/ms/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/QuantLib-1.35-py3.10-linux-x86_64.egg/QuantLib/QuantLib.py", line 10, in <module>
    from . import _QuantLib
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook
    module = hook()
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 120, in _hook
    return _find_and_load(name, import_)
ImportError: libQuantLib.so.1: cannot open shared object file: No such file or directory
@t-kalinowski
Copy link
Member

Are you able to load the custom built Python package outside of R/reticulate, in a vanilla Python session?

@matthewgson
Copy link
Author

Yes, it works when run in vanilla python- on condition that environment variables has to be defined (~/.bashrc above) in prior. Vanilla python produces the same error if not defined - and I suspect reticulate python somehow doesn't recognize those env variables defined above ways.

@t-kalinowski
Copy link
Member

t-kalinowski commented Sep 12, 2024

Conda environments requires some workarounds unfortunately (things are simpler with virtual environments...)

Can you try re-setting LD_LIBRARY_PATH in Python and R, after reticulate is finished initializing the Python session?

E.g.:

library(reticulate)
py_available(TRUE) # initialize Python

local({
  path_to_my_sos <- "~/local/lib"

  py_env <- import("os")$environ
  py_env$update(list("LD_LIBRARY_PATH" = sprintf(
    "%s:%s", path_to_my_sos, py_env$get("LD_LIBRARY_PATH", "")
  )))
  Sys.setenv(LD_LIBRARY_PATH = sprintf(
    "%s:%s", path_to_my_sos, Sys.getenv("LD_LIBRARY_PATH", "")
  ))
})

@matthewgson
Copy link
Author

matthewgson commented Sep 12, 2024 via email

@t-kalinowski
Copy link
Member

Yes, try it either way. I suspect that the LD_LIBRARY_PATH that the R session started with is not the one that's presented to the Python session.

@matthewgson
Copy link
Author

matthewgson commented Sep 12, 2024

Thank you. I reinstalled miniconda and tried, but still seeing those errors unfortunately. Please see:

> library(reticulate)
> py_config()
python:         /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python
libpython:      /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.10.so
pythonhome:     /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate:/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate
version:        3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
numpy:          /home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/numpy
numpy_version:  2.1.1
> py_available(TRUE)
[1] TRUE
> local({
  path_to_my_sos <- "~/local/lib"

  py_env <- import("os")$environ
  py_env$update(list("LD_LIBRARY_PATH" = sprintf(
    "%s:%s", path_to_my_sos, py_env$get("LD_LIBRARY_PATH", "")
  )))
  Sys.setenv(LD_LIBRARY_PATH = sprintf(
    "%s:%s", path_to_my_sos, Sys.getenv("LD_LIBRARY_PATH", "")
  ))
})
> py_env
Error: object 'py_env' not found
> repl_python()
Python 3.10.14 (/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python)
Reticulate 1.38.0 REPL -- A Python interpreter in R.
Enter 'exit' or 'quit' to exit the REPL and return to R.
>>> import QuantLib as ql
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)
  File "/usr/local/lib/R/site-library/reticulate/python/rpytools/loader.py", line 96, in _run_hook

When vanilla python:

bash-4.4$ echo $LD_LIBRARY_PATH
/home/gunsu.son/local/lib:/apps/squashfuse/0.5.2/lib:/opt/slurm/lib64:/home/gunsu.son/local/lib::
bash-4.4$ which python
~/.local/share/r-miniconda/envs/r-reticulate/bin/python
bash-4.4$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.executable)
/home/gunsu.son/.local/share/r-miniconda/envs/r-reticulate/bin/python
>>> import QuantLib as ql 
>>> # runs

Please let me know if you'd like to see any relevant information I can provide. Thank you.

@t-kalinowski
Copy link
Member

Is the conda environment activated in the 2nd code chunk? I don't see the typical (r-reticulate) or (base) conda prefix. If you activate the conda env, does your custom built module still import successfully?

Unless you have a strong motivator, I would encourage switching to virtualenvs.

# reticulate::install_python("3.11") # or use /usr/bin/python or https://github.com/rstudio/python-builds

reticulate::virtualenv_create("r-reticulate", force = TRUE)

@matthewgson
Copy link
Author

matthewgson commented Sep 12, 2024

Yes.

(r-reticulate) bash-4.4$ which python
~/.local/share/r-miniconda/envs/r-reticulate/bin/python
(r-reticulate) bash-4.4$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import QuantLib as ql
>>> 

The Python installation through pyenv didn’t work as expected, so I decided not to use that approach. Instead, I created a virtual environment (venv) using the system Python on the server. However, my concern was that the libpath configured by py_config() points to the system Python path, rather than the venv folder.

@t-kalinowski
Copy link
Member

t-kalinowski commented Sep 18, 2024

Would you be able to provide a minimal example I could run locally to reproduce the error?

@matthewgson
Copy link
Author

Sure- though it may not seem "minimal" but reproducible. I was able to reproduce the same result on my local mac.

Setup r-reticulate venv, from python installed from brew

virtualenv_create("r-reticulate", python = "/opt/homebrew/bin/python3")

On terminal

source ~/.virtualenvs/r-reticulate/bin/activate
pip install setuptools
brew install boost cmake

Install QuantLib

cd ~
curl -L -O https://github.com/lballabio/QuantLib/releases/download/v1.35/QuantLib-1.35.tar.gz
tar -xvzf QuantLib-1.35.tar.gz
cd QuantLib-1.35
mkdir build
cd build
cmake .. -DQL_HIGH_RESOLUTION_DATE=ON -DBoost_INCLUDE_DIR=/opt/homebrew/opt/boost/include -DCMAKE_INSTALL_PREFIX=$HOME/local
make -j$(sysctl -n hw.ncpu)
make install

echo 'export PATH=$HOME/local/bin:$PATH' >> ~/.zshrc
source ~/.zshrc
which quantlib-config 

Build python package binding

cd ..
curl -L -O https://github.com/lballabio/QuantLib-SWIG/releases/download/v1.35/QuantLib-SWIG-1.35.tar.gz
tar -xvzf QuantLib-SWIG-1.35.tar.gz
cd QuantLib-SWIG-1.35

autoreconf --install

./configure --disable-csharp --disable-r --disable-java --disable-java-autoload --disable-java-finalizer --disable-scala CXXFLAGS="-I/opt/homebrew/opt/boost/include -std=c++17" CPPFLAGS="-I/opt/homebrew/opt/boost/include" LDFLAGS="-L/opt/homebrew/opt/boost/lib" 
make -C Python 
make install

Package is installed:

pip list

Package    Version
---------- -------
numpy      2.1.1
pip        24.2
QuantLib   1.35
QuantLib   1.35
setuptools 75.1.0
wheel      0.44.0

Testing on vanilla python

cd ~
python
import QuantLib as ql

Testing from R

library(reticulate)
repl_python()
import QuantLib as ql

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/matthewson/Library/R/arm64/4.4/library/reticulate/python/rpytools/loader.py", line 122, in _find_and_load_hook
    return _run_hook(name, _hook)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants