How to compile code into a cli installable?

[simkimsia]

For context, I'm trying to fork kedro to be another CLI but for generating Dockerized Django projects from cookiecutter templates. See #1190 for details.

So I manage to make the def new function work for my context. However, the way i test it currently is

go to my dockerized-django-cli project (codenamed greendeploy) and type

python -m greendeploy new --verbose

Then it's generated inside the cli project codebase.

How do I compile the source code to be a python cli executable? So i can go anywhere in the directory and type greendeploy new like kedro new?

I am running on macOS 12.1 Apple Silicon and using py3.10 in my venv if it matters.

[antonymilne]

This is actually super easy: you just need to make an entry point for console_scripts. Here's the line where it's done in kedro: https://github.com/kedro-org/kedro/blob/main/setup.py#L170.

I recommend this tutorial if you want to learn more about how it works - it's where I first learnt about it.

I should also warn you that the route from kedro.framework.cli:main to actually executing the def new function when you do kedro new is a little complicated, since we need to take account of plugins and a local cli.py file, which could in theory overwrite the default in-built definition of a kedro xyz command. We also split the definition of CLI commands into multiple files. We wrote our own KedroCLI to extend click's system to achieve all this. However, if you're just looking to make a greendeploy new and have no such complications you could make it a lot simpler and might not even need to use click at all.

Just to give you an idea, this is what happens under the hood (from these docs):

[simkimsia]

I took what you said about reading the tutorial

So I did this:

python setup.py develop

but i got an error about Entry points must be listed in groups.

I am using setup.cfg so not entirely similar to how currently kedro uses purely setup.py

THis is my commandline

(greendeploy-cli-py3101) kimsia@kimsia-MBP2021 greendeploy-cli % pip-sync   
Everything up-to-date
(greendeploy-cli-py3101) kimsia@kimsia-MBP2021 greendeploy-cli % python setup.py develop
Traceback (most recent call last):
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2851, in get_entry_map
    ep_map = self._ep_map
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2813, in __getattr__
    raise AttributeError(attr)
AttributeError: _ep_map. Did you mean: '_dep_map'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kimsia/Projects/greendeploy-cli/setup.py", line 21, in <module>
    setup(use_scm_version=True)
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/setuptools/__init__.py", line 152, in setup
    _install_setup_requires(attrs)
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/setuptools/__init__.py", line 142, in _install_setup_requires
    dist = MinimalDistribution(attrs)
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/setuptools/__init__.py", line 134, in __init__
    distutils.core.Distribution.__init__(self, filtered)
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/setuptools/dist.py", line 451, in __init__
    for ep in pkg_resources.iter_entry_points('distutils.setup_keywords'):
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 641, in <genexpr>
    for entry in dist.get_entry_map(group).values()
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2853, in get_entry_map
    ep_map = self._ep_map = EntryPoint.parse_map(
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2538, in parse_map
    raise ValueError("Entry points must be listed in groups")
ValueError: Entry points must be listed in groups

this is my setup.py

import re
from codecs import open
from glob import glob
from itertools import chain
from os import path

from setuptools import find_packages, setup

name = "greendeploy"
here = path.abspath(path.dirname(__file__))

# get package version
with open(path.join(here, name, "__init__.py"), encoding="utf-8") as f:
    result = re.search(r'__version__ = ["\']([^"\']+)', f.read())

    if not result:
        raise ValueError("Can't find the version in greendeploy/__init__.py")

    version = result.group(1)

setup(use_scm_version=True)

this is my setup.cfg

[metadata]
name = greendeploy-cli
url = https://github.com/greendeploy-org/greendeploy-cli/
license = Apache License 2.0
author = GreenDeploy.io
author_email = [email protected]
description = greendeploy-cli is a CLI for creating and using Dockerized Django projects.
long_description = file: README.rst
classifiers =
    Development Status :: 5 - Production/Stable
    Intended Audience :: Developers
    Intended Audience :: System Administrators
    License :: OSI Approved :: Apache Software License
    Operating System :: OS Independent
    Programming Language :: Python
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3 :: Only
    Programming Language :: Python :: 3.10
    Programming Language :: Python :: Implementation :: CPython
    Programming Language :: Python :: Implementation :: PyPy
    Topic :: System :: Systems Administration

[options]
python_requires = ~=3.10
setup_requires = setuptools_scm
packages = find:
zip_safe = false
install_requires =
    # direct dependencies
    pep517
    pip < 22.0
    pip-tools ~= 6.5; python_version > '3.6'
    pyfiglet == 0.7.2
    cookiecutter ~= 1.7.0
    gitpython ~= 3.0
    python-json-logger ~= 2.0
    PyYAML >=4.2, <6.0
    # indirect dependencies
    setuptools >= 38.0  # typically needed when pip-tools invokes setup.py
    wheel  # pip plugin needed by pip-tools

# this is for entry point for the cli
# see https://amir.rachum.com/blog/2017/07/28/python-entry-points/ to learn more abt entry points
[options.entry_points]
console_scripts =
    greendeploy = greendeploy.framework.cli:main

I had followed this https://stackoverflow.com/a/50078487/80353 on how to write the entry_points inside setup.cfg

Also strangely now when i run python -m greendeploy new --verbose i now have similar errors

(greendeploy-cli-py3101) kimsia@kimsia-MBP2021 greendeploy-cli % python -m greendeploy new --verbose
Traceback (most recent call last):
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2851, in get_entry_map
    ep_map = self._ep_map
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2813, in __getattr__
    raise AttributeError(attr)
AttributeError: _ep_map. Did you mean: '_dep_map'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kimsia/.pyenv/versions/3.10.1/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/kimsia/.pyenv/versions/3.10.1/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/kimsia/Projects/greendeploy-cli/greendeploy/__main__.py", line 10, in <module>
    main()
  File "/Users/kimsia/Projects/greendeploy-cli/greendeploy/framework/cli/cli.py", line 98, in main
    cli_collection = GreenDeployCLI(project_path=Path.cwd())
  File "/Users/kimsia/Projects/greendeploy-cli/greendeploy/framework/cli/cli.py", line 53, in __init__
    ("Global commands", self.global_groups),
  File "/Users/kimsia/Projects/greendeploy-cli/greendeploy/framework/cli/cli.py", line 91, in global_groups
    return [cli, create_cli, *load_entry_points("global")]
  File "/Users/kimsia/Projects/greendeploy-cli/greendeploy/framework/cli/utils.py", line 43, in load_entry_points
    for entry_point in entry_points:
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 641, in <genexpr>
    for entry in dist.get_entry_map(group).values()
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2853, in get_entry_map
    ep_map = self._ep_map = EntryPoint.parse_map(
  File "/Users/kimsia/.venv/greendeploy-cli-py3101/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2538, in parse_map
    raise ValueError("Entry points must be listed in groups")
ValueError: Entry points must be listed in groups

[simkimsia]

i feel bad abt asking for free help here. so if i have to pay for support, i am happy to do so. really embarrassed about asking for free help. just let me know abt the costs or payment methods. happy to sign any NDA if necessary. really keen to fork kedro (it's very well written) into the usecase of template builder for dockerized django projects

[antonymilne]

Hmm, this is getting a bit far from kedro support but having played around with these things before I should be able to give some pointers. Since it's the python setup.py develop that fails in the first place, it's not to do with the kedro CLI or click - it's more a question of entry points and setuptools. Figuring stuff out with setuptools often just comes down to trial and error and a lot of perseverance though...

In no particular order, I'd try some of the following to pin down the error further:

• different version of Python
• different version of setuptools
• pip install rather than python setup.py
• python setup.py install rather than develop
• use setup.py rather than setup.cfg (though from what I can see you did this right)

I'd also try to get a very minimal version of this working first, without using the kedro CLI stuff at all. Just to try and get a working setup.cfg that points to a super simple script. After that you should be able to get it working with greendeploy exactly the same way, since this is a setuptools problem rather than anything in your codebase specifically. The way to tell that it's working is that which greendeploy should find something once the entrypoint has been registered successfully.

[antonymilne]

Also your offer of paying for help is very thoughtful, but really not necessary! 😀 Thank you though

[simkimsia]

The solution was to remove all the .egg files and folders and fix the setup.cfg and retry

[options.entry_points]
    console_scripts =
        greendeploy = greendeploy.framework.cli:main