Python install - Misleading exception raised in Setup.py on GPU install

[joseortiz3]

Environment info

Operating System: Windows 10 Pro
CPU: i7-7700K
C++/Python/R version: Python 3 Anaconda 5.0.0

Error Message:

For GPU installs, the error message that says "Please install Visual Studio or MS Build first" is given when Visual Studio is installed but boost is not installed.

Hence, the python line 123 in setup.py that raises this exception should either branch into more cases perhaps checking to see whether or not boost or other packages are installed, should print more information to command line, or should say this error can be caused by not having the required dependencies other than VS/MSBuild, such as boost.

Steps to reproduce

1. Do not have boost on your system
2. Have VS 2017 with all required (?) packages installed, also have CUDA installed
3. Try pip install lightgbm --install-option=--gpu
4. Notice the error saying "Please install Visual Studio or MS Build first"
5. Install Boost according to this guide
6. Notice no error anymore.

Attached Images

Note: The only thing that made the install work was installing Boost, contrary to the exception raised.

[StrikerRUS]

@joseortiz3 Thank you for sharing your case. We'll redesign error messages to more informative.

[StrikerRUS]

Fixed in #1034.

[Untoha2]

@joseortiz3, @StrikerRUS, @huanzhang12
Hello, I understand that the issue is quite outdated, but I've faced it. I've followed the guide and managed to build boost with cmake and Visual Studio 2019 (I've tried various versions of boost: at least those that are in the attached guide), but I always get the same error. I would be so thankful if somebody explains to me what is the problem. I attach screenshots with cmake configurations and with the error. I have windows 10 x64.

[StrikerRUS]

@Untoha2 Hi!

I'm not sure, but try to remove spaces from arguments.

Also, please attach log file C:\Users\ANTON\LightGBM_compilation.log here, it should help a lot.

[StrikerRUS]

@Untoha2 Also, never execute commands from System32 folder: #1961 (comment).

[Untoha2]

@StrikerRUS, Thank you for answering
I removed spaces and changed working directory from System32 but nothing has changed. I get the same error.

It is also strange but log file C:\Users\ANTON\LightGBM_compilation.log is empty.

[Untoha2]

@StrikerRUS
Maybe it matters but previously I tried to compile LightGBM with Mingw and got the same result.

[StrikerRUS]

@Untoha2 Have you downloaded another version of Boost? Previously you targeted 1.70, but in the latest comment I see 1.69. Also, for VS 2019 you need .../lib64-msvc-14.2, not 14.1.

Also, I see you have CMake GUI. What about compiling LightGBM dll there and then use --precompile flag?

[StrikerRUS]

@Untoha2 Oh, I just noticed that #2083 isn't in 2.2.3 version! So, it seems that manual compilation with CMake (GUI or CLI, whatever you prefer) and later using --precompile flag is the only way to compile with VS 2019.

[Untoha2]

@StrikerRUS
I'm sorry, but I didn't understand how I should use --precompile flag . I will describe my installation:

1. I downloaded and installed boost_1_70_0-unsupported-msvc-14.2-64.exe
   
2. In the Powershell I cloned LightGBM

cd C:\mvs_lightgbm
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build

3. Compiled LightGBM with Cmake GUI using Visual Studio 2019
   
   So at this point I understand that LightGBM with gpu option is already compiled.
4. Then i type this in Powershell (add --precompile option).

pip install lightgbm --install-option=--gpu --install-option="--opencl-include-dir=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/include" --install-option="--opencl-library=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/lib/x64/OpenCL.lib" --install-option="--boost-librarydir=C:/local/boost_1_70_0/lib64-msvc-14.2" --install-option="--boost-root=C:/local/boost_1_70_0" --install-option="--precompile"

And get this

C:\Users\ANTON\Anaconda3\lib\site-packages\pip\_internal\commands\install.py:207: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting lightgbm
  Using cached https://files.pythonhosted.org/packages/c9/ce/3aff55e25e282383c19c5a5fb7387fd400e64b1a1036671aefa63ceeaaf4/lightgbm-2.2.3.tar.gz
Requirement already satisfied: numpy in c:\users\anton\anaconda3\lib\site-packages (from lightgbm) (1.17.0)
Requirement already satisfied: scipy in c:\users\anton\anaconda3\lib\site-packages (from lightgbm) (1.1.0)
Requirement already satisfied: scikit-learn in c:\users\anton\anaconda3\lib\site-packages (from lightgbm) (0.21.2)
Requirement already satisfied: joblib>=0.11 in c:\users\anton\anaconda3\lib\site-packages (from scikit-learn->lightgbm) (0.13.2)
Skipping bdist_wheel for lightgbm, due to binaries being disabled for it.
Installing collected packages: lightgbm
  Running setup.py install for lightgbm ... error
    Complete output from command C:\Users\ANTON\Anaconda3\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\ANTON\\AppData\\Local\\Temp\\pip-install-zxky34ua\\lightgbm\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\ANTON\AppData\Local\Temp\pip-record-2bj712vo\install-record.txt --single-version-externally-managed --compile --gpu "--opencl-include-dir=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/include" "--opencl-library=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/lib/x64/OpenCL.lib" --boost-librarydir=C:/local/boost_1_70_0/lib64-msvc-14.2 --boost-root=C:/local/boost_1_70_0 --precompile:
    running install
    running build
    running build_py
    creating build
    creating build\lib
    creating build\lib\lightgbm
    copying lightgbm\basic.py -> build\lib\lightgbm
    copying lightgbm\callback.py -> build\lib\lightgbm
    copying lightgbm\compat.py -> build\lib\lightgbm
    copying lightgbm\engine.py -> build\lib\lightgbm
    copying lightgbm\libpath.py -> build\lib\lightgbm
    copying lightgbm\plotting.py -> build\lib\lightgbm
    copying lightgbm\sklearn.py -> build\lib\lightgbm
    copying lightgbm\__init__.py -> build\lib\lightgbm
    running egg_info
    writing lightgbm.egg-info\PKG-INFO
    writing dependency_links to lightgbm.egg-info\dependency_links.txt
    writing requirements to lightgbm.egg-info\requires.txt
    writing top-level names to lightgbm.egg-info\top_level.txt
    reading manifest file 'lightgbm.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'build'
    warning: no files found matching '*.so' under directory 'lightgbm'
    warning: no files found matching '*.so' under directory 'compile'
    warning: no files found matching '*.dll' under directory 'compile\Release'
    warning: no files found matching '*.dll' under directory 'compile\windows\x64\DLL'
    warning: no previously-included files matching '*.py[co]' found anywhere in distribution
    writing manifest file 'lightgbm.egg-info\SOURCES.txt'
    copying lightgbm\VERSION.txt -> build\lib\lightgbm
    running install_lib
    creating C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\basic.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\callback.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\compat.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\engine.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\libpath.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\plotting.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\sklearn.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\VERSION.txt -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    copying build\lib\lightgbm\__init__.py -> C:\Users\ANTON\Anaconda3\Lib\site-packages\lightgbm
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\setup.py", line 338, in <module>
        'Topic :: Scientific/Engineering :: Artificial Intelligence'])
      File "C:\Users\ANTON\Anaconda3\lib\site-packages\setuptools\__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\ANTON\Anaconda3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\ANTON\Anaconda3\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Users\ANTON\Anaconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\setup.py", line 262, in run
        install.run(self)
      File "C:\Users\ANTON\Anaconda3\lib\site-packages\setuptools\command\install.py", line 61, in run
        return orig.install.run(self)
      File "C:\Users\ANTON\Anaconda3\lib\distutils\command\install.py", line 557, in run
        self.run_command(cmd_name)
      File "C:\Users\ANTON\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\ANTON\Anaconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\ANTON\Anaconda3\lib\site-packages\setuptools\command\install_lib.py", line 12, in run
        outfiles = self.install()
      File "C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\setup.py", line 210, in install
        src = find_lib()[0]
      File "C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\setup.py", line 27, in find_lib
        LIB_PATH = [os.path.relpath(path, CURRENT_DIR) for path in libpath['find_lib_path']()]
      File "C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\lightgbm\libpath.py", line 37, in find_lib_path
        raise Exception('Cannot find lightgbm library file in following paths:\n' + '\n'.join(dll_path))
    Exception: Cannot find lightgbm library file in following paths:
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\lightgbm\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\lightgbm\compile\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\compile\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lib\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\compile\Release\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\compile\windows\x64\DLL\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\Release\lib_lightgbm.dll
    C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\windows\x64\DLL\lib_lightgbm.dll

    ----------------------------------------
Command "C:\Users\ANTON\Anaconda3\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\ANTON\\AppData\\Local\\Temp\\pip-install-zxky34ua\\lightgbm\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\ANTON\AppData\Local\Temp\pip-record-2bj712vo\install-record.txt --single-version-externally-managed --compile --gpu "--opencl-include-dir=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/include" "--opencl-library=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/lib/x64/OpenCL.lib" --boost-librarydir=C:/local/boost_1_70_0/lib64-msvc-14.2 --boost-root=C:/local/boost_1_70_0 --precompile" failed with error code 1 in C:\Users\ANTON\AppData\Local\Temp\pip-install-zxky34ua\lightgbm\

Could you please tell me where I am wrong. I suppose I misunderstand the use of --precompile option or maybe something wrong with paths. Thank you!

[StrikerRUS]

So at this point I understand that LightGBM with gpu option is already compiled.

Unfortunately, no. CMake only generated configuration. Now you should compile it with VS.

cd build
cmake --build . --target ALL_BUILD --config Release

Now you have compiled dll LightGBM library, so, you can use --precompile flag.

cd ../python-package
python setup.py install --precompile

[Untoha2]

@StrikerRUS
When I call

cd C:\mvs_lightgbm\LightGBM\build
cmake --build . --target ALL_BUILD --config Release

I get the error that name cmake is not recognized as command, function or scenario.

Because there is no file with this name in the directory. How can I call cmake from the given directory?

I'm sorry, I am not a big pro.

[StrikerRUS]

@Untoha2 Add CMake to your PATH variable.

[Untoha2]

@StrikerRUS
Thank you so much! Finally I've managed to install it.
Maybe it's another topic but when I ran LGBMRegressor on GPU I didn't get much speed up. GPU is actually working during learning but it is only 10% activated - the rest of computation is conducted on CPU.

Maybe you can show me thread on GitHub where I can find possible ways to speed up my GPU computations.
Thank you once again!

[StrikerRUS]

@Untoha2 Glad that you've managed to build LightGBM GPU version!

For GPU-version performance, please consider starting reading from this issue #768 and follow some links in it. Then, you may want to get familiar with some benchmarks: Laurae2/ml-perf#8, Laurae2/ml-perf#6, szilard/GBM-perf#11.