Allowing TypeVar to include any user defined Literal[<strings>]

[coretl]

I would like to define a holder of a user defined type like so:

class Box(Generic[T]):
    def __init__(self, datatype: type[T]):
        self.datatype = datatype
        self.value: Optional[T] = None
    
    def set(self, value: T):
        self.value = value

but I would like to constrain the datatype to be int, str, a user defined Enum, or a Literal["of", "user", "defined", "strings"].

I thought that defining T to be:

T = TypeVar("T", int, str, Enum, LiteralString)

might do the trick, but it fails on Box(Literal["a", "b"]) with:

Argument of type "UnionType" cannot be assigned to parameter "datatype" of type "type[T@Box]" in function "__init__"
  Type "UnionType" is not assignable to constrained type variable "T"  (reportArgumentType)

Is there a way to achieve this?

Code sample in pyright playground

[erictraut]

You're attempting to use a value-constrained TypeVar here. I generally recommend avoiding these if possible. They are poorly specified in the typing spec and have many unusual and surprising behaviors and limitations. This concept is unique to Python and doesn't appear in other programming languages that support generics. There's a good reason for this.

In your code sample, you're defining a TypeVar that includes a value constraint of str and LiteralString, but LiteralString is a subtype of str. All LiteralString values are already legal values of the type str, so that's an immediate red flag.

Another issue here is that you're attempting to pass the runtime value for a type expression (Literal["a", "b"]) to a parameter that is annotated as type[T]. That's not allowed according to the typing spec because type[T] is compatible only with instances of the type class (or subclasses thereof). The runtime type of the expression Literal["a", "b"] is either _LiteralGenericAlias or UnionType depending on how you spell it. Either way, it's not compatible with type[T].

There is a draft PEP 747 that provides a way to annotate a parameter that accepts runtime values evaluated from type expressions, but it has not yet made its way through the spec'ing and approval process.

I'm not able to offer you any specific advice about how to write your proposed Box class because I don't have enough understanding of how you intend to use it or what constraints you would like to impose on the solution. For example, how is your implementation going to make use of the datatype passed to it? Is it going to crack open the _LiteralGenericAlias or UnionType object and decode the contents of the type expression? For what purpose?

[coretl]

Thanks for the quick reply!

I'm not able to offer you any specific advice about how to write your proposed Box class because I don't have enough understanding of how you intend to use it or what constraints you would like to impose on the solution. For example, how is your implementation going to make use of the datatype passed to it? Is it going to crack open the _LiteralGenericAlias or UnionType object and decode the contents of the type expression? For what purpose?

My application is making a type safe interface to a control system. The control system has defined datatypes like INT, STRING, FLOAT64_ARRAY that I map one-to-one with python types int, str and npt.NDArray[np.float64]. The example Box class is like the Python interface for a single control system variable, where the user defines in the constructor what type they expect, and their usage of that variable is checked at static analysis time. At runtime it is then checked against the real control system, and if it matches then pre-defined functions convert between the control system and python types. If it doesn't match an error is raised.

The issue I have is with control system ENUMs. I have to support:

• An ENUM where all the possible choices are known in advance, I chose to map this to an Enum subclass as in the example and runtime check that the choices on both sides match
• An ENUM where only a subset of the possible choices are known in advance. I chose to map this to Literal["known", "choices"] and runtime check that the Literal choices are a subset of the control system choices

For the enum checks, I do indeed crack open the types and work out how to do the conversion, but I wasn't quite able to provide a constrained TypeVar to use in the Box class so that the users of the library would know what sort of types they could pass to Box.__init__ and have it do the right conversions.

If I can't find a solution to this then I could just drop to using str for the subset enum case, but it would be nice to give the user some type checking of the set method of their subset enum.

You're attempting to use a value-constrained TypeVar here. I generally recommend avoiding these if possible. They are poorly specified in the typing spec and have many unusual and surprising behaviors and limitations. This concept is unique to Python and doesn't appear in other programming languages that support generics. There's a good reason for this.

That is interesting, I would be very interested to know of any alternative approaches that I could use instead...

[erictraut]

Have you considered using a generic base class with specialized (concrete) derived classes?

class ControlValue[T]:
    def __init__(self):
        self.value: T | None = None

    def set(self, value: T):
        self.value = value

class IntControlValue(ControlValue[int]): ...
class StrControlValue(ControlValue[str]): ...
# Etc.

[coretl]

I simplified my example too much, the complexity is that I have multiple backends for different control systems, and to keep the backend classes for each control system small I choose to make it a pure abstract class that is implemented for each control system. The wrapper class is the same for each control system, and provides the interface the user wants rather than what is simple to implement for the control system.

Here is a more realistic example (also in an example repo):

import json
import time
from abc import abstractmethod
from collections.abc import Iterator
from enum import Enum
from typing import Any, Literal, TypeVar
import numpy as np
import numpy.typing as npt
import requests

class CSBackend[T]:
    """An abstract backend allowing a CS value to be sent or received"""
    @abstractmethod
    def send(self, value: T): ...
    @abstractmethod
    def recv(self) -> T: ...


class CSValue[T]:
    """A wrapper class that presents the interface the user would like to a CS value"""

    def __init__(self, backend: CSBackend[T]):
        self.backend = backend

    def set(self, value: T):
        # Do logging, caching, other common things
        self.backend.send(value)

    def get(self) -> T:
        # Do logging, caching, other common things
        return self.backend.recv()

    def poll(self, interval: int) -> Iterator[T]:
        while True:
            time.sleep(interval)
            yield self.get()
    # And many other methods...

# The sort of things we can send over HTTP
HTTPType = TypeVar("HTTPType", bound=str | int | Enum | npt.NDArray[np.float64])

def convert_to(datatype: type[HTTPType], value: Any) -> HTTPType: ...

class HTTPBackend(CSBackend[HTTPType]):
    def __init__(self, datatype: type[HTTPType], uri: str) -> None:
        self.datatype = datatype
        self.uri = uri

    def send(self, value: HTTPType):
        requests.put(self.uri, json.dumps(value))

    def recv(self) -> HTTPType:
        return convert_to(self.datatype, requests.get(self.uri).json())


# A helper function to create these wrapper instances
def http_value(datatype: type[HTTPType], uri: str) -> CSValue[HTTPType]:
    # We would return a different backend if the service is in simulation mode...
    backend = HTTPBackend(datatype, uri)
    return CSValue(backend)


# Another different backend would support different types
MQTTType = TypeVar("MQTTType", bound=str | int | Enum)

def mqtt_value(datatype: type[MQTTType], host: str, port: int) -> CSValue[MQTTType]: ...

# A way to define enums
class MyEnum(str, Enum):
    a = "A"
    b = "B"
    # Check that CS enum is precisely {"A", "B"}

# The use wants to form these values into devices, of which there will be many
class MyGoodDevice:
    def __init__(self, http_uri: str = "http://example.com"):
        # These all work
        self.v1 = http_value(str, http_uri)
        self.v2 = mqtt_value(str, "localhost", 1883)
        self.v3 = http_value(MyEnum, http_uri)
        self.v4 = http_value(npt.NDArray[np.float64], http_uri)
        self.v4.set(np.array([1]))

# These legitimately fail
http_value(dict, "http://example.com")
http_value(npt.NDArray[np.float64], "http://example.com").set(
    np.array([1], dtype=np.int32)
)
mqtt_value(npt.NDArray[np.float64], "localhost", 1883)

# My original question was about making this work
http_value(Literal["a", "b"], "http://example.com")

But given what you said about needing to be compatible with type[T], I could switch to using:

class SubsetEnum(Enum): ...

class MySubsetEnum(str, SubsetEnum):
    a = "A"
    b = "B"
    # Check that {"A", "B"} is a subset of CS enum

http_value(MySubsetEnum, "http://example.com")

I'm happy with this approach, and so is pyright, so this solves my question, but I'd be interested to know if there are any other red flags in my example