Serialize & Deserialize Kotlin Delegates

[wakaztahir]

What is your use-case and why do you need this feature?
Right now there is no way to serialize kotlin delegates and even if there is , its not easy !
I use Jetpack Compose , If I use type like MutableState and specify a custom serializer I have to type .value for each property everywhere in my code where I used it , which is just very annoying and code doesn't look good

Describe the solution you'd like
I would like an easy annotation / way to serialize delegated properties , easy way to enforce serialization for delegated properties

[sandwwraith]

There's no built-in way to do so, but you can write a custom serializer that uses getters and setters of the delegates.

Maybe we can implement @SerializeByGetterAndSetter annotation

[OliverO2]

@sandwwraith

There's no built-in way to do so, but you can write a custom serializer that uses getters and setters of the delegates.

It seems that the compiler is actually completely ignoring properties with delegates for serialization, even those with a custom serializer annotation @Serializable(with = ...).

My use case is the serialization of a graph with nodes intended to look like this (simplified):

class TreeNode private constructor(override val id: ReferenceableID) : Referenceable() {
    @Serializable(with = LazyReference.Serializer::class)
    var parent: TreeNode? by LazyReference()  // <- stores the node's ID internally
    var children: List<TreeNode> = mutableListOf()

    constructor(id: ReferenceableID, initialParent: TreeNode? = null) : this(id) {
        parent = initialParent
    }
}

When the children of a parent node are deserialized, each of their parent properties refers to the immediate parent node, which has not been constructed yet. To deal with this cycle during deserialization, I delay parent resolving: I use a context which remembers deserialized nodes via a HashMap<ReferenceableID, Referenceable>. LazyReference then uses this context later to find the node by its ID.

The closest I could get with the current implementation is via a separate backing property:

class TreeNode private constructor(override val id: ReferenceableID) : Referenceable() {
    private var _parentID: ReferenceableID? = null  // <-- will be serialized
    var parent: TreeNode? by LazyReference(this::_parentID)  // <-- will not be serialized
    var children: List<TreeNode> = mutableListOf()

    constructor(id: ReferenceableID, initialParent: TreeNode? = null) : this(id) {
        parent = initialParent
    }
}

Could you consider making the compiler honor @Serializable(with = ...) for properties with delegates and not ignore those?

[wakaztahir]

You would need to write a surrogate , that's how I did it , I had nodes inside which were delegates by mutable state , since I am using jetpack compose to render a big map so nodes contained a self reference

So I had to recursively convert the nodes to surrogates and register surrogate classes in serializersModule and I serialized and deserialized surrogate classes instead of actual nodes

@OliverO2

I wrote the surrogate for parent class instead of the class that was being delegated.

[OliverO2]

Good to hear that surrogates worked for you. The downside of both approaches is boilerplate which we are all trying to avoid. If I'm not overlooking something, this could all be avoided by letting the serialization compiler plugin accept getters and setters instead of insisting on backing fields.

[OliverO2]

Some thoughts on a possible implementation:

As explained in the section Delegated properties – Translation rules, the compiler generates an auxiliary property for each delegated property like this:

private val parent$delegate: LazyReference
var parent: TreeNode? by parent$delegate

To serialize the delegated property in the above example:

• The corresponding auxiliary property (parent$delegate) would be serialized.
• The delegated property's deserializer (the one for parent) would return the delegate (a LazyReference object).
• The delegate would be assigned to the auxiliary property (parent$delegate).
• The delegated property (parent) would be left as is.

As long as the delegate itself is serializable, all of this could work without any additional annotation.

[wakaztahir]

I guess they don't want delegated properties to be serializable by default

[OliverO2]

They don't have to be. That could depend on the delegate class being annotated with @Serializable.

[wakaztahir]

A lot of classes from other libs and jetpack compose won't have serializable annotation over them

So how would this be fixed , would we have to implement a serializer for the delegate class ourselves ?

[OliverO2]

You might want to look at Deriving external serializer for another Kotlin class (experimental). However, it will often be unfeasible to serialize classes that were not designed with serialization in mind.

[sandwwraith]

@wakaztahir Can you please provide an example of serializable class with delegates that can be used with Jetpack Compose? I'm investigating the issue and so far it seems that delegates should only be used to obtain State<T> instance

[OliverO2]

I think the original discussion about this started here on Slack: https://kotlinlang.slack.com/archives/C7A1U5PTM/p1628060200011800

[wakaztahir]

I am using mutable state inside classes by delegating it , MutableState

Yes , In Jetpack Compose MutableState<T> / State<T> are the only delegating classes , I am also obtaining State<T> instances only with delegates but in general delegating with classes that use generics to return same type as the parameter is common , I probably exaggerated

[pdvrieze]

There's no built-in way to do so, but you can write a custom serializer that uses getters and setters of the delegates.

Maybe we can implement @SerializeByGetterAndSetter annotation

I find that for more complex serializations I need a "delegate" (which can be a "private" member class) that is just a simple class with properties (and the ability to be constructed from the actual type - toDelegate(), as well as the reverse - Delegate.toActual(). It needs a fairly trival custom serializer. It would be good to have an annotation to allow this custom serializer to be automatically generated.

[wakaztahir]

Any progress on this ?

[wakaztahir]

Jetpack compose also includes mutable state list , which is the snapshot state list inheriting from list of course
And mutable state map

Thought I'd mention these classes because they are used a lot and sould be easily serializable like a normal list yet I have to cast it or if I use them as properties then provide a serializer either surrogate / custom

[sandwwraith]

@wakaztahir We have this in plans, but no particular timeframe

[pdvrieze]

I would see this working best in combination with the explicit backing field keep (when applied to delegates).

[wakaztahir]

@sandwwraith Is it done ? or when will this feature be available

[sandwwraith]

No, it's not being developed at the moment

[mgroth0]

I've developed most of my classes to use property delegates for the sake of autosaving and listenable properties and such. I'm really excited about the potential of kotlinx.serialization but this issue is somewhat killing it for me.

I seem to have two choices:

1. Keep using my property delegates but use custom serializers. This roughly doubles the amount of code I have to write to define a class.
2. Design my classes to have internal data models that don't use property delegates, make sure everything is bound correctly, then build and serialize my classes based on those internal models.

A @SerializeByGetterAndSetter annotation would be a lifesaver.

[OliverO2]

Serializing delegate properties is actually much simpler. It is not even necessary to introduce an extra annotation. Almost everything pretty much works out of the box right now:

The Kotlin compiler transforms a delegate property declared like this:

    var parent: TreeNode? by ReferenceDelegate()

into these two properties:

    var `parent$delegate` = ReferenceDelegate()  // (1) auxiliary property
    var parent: TreeNode? by `parent$delegate`   // (2) accessor property

Actually, serialization for the two transformed properties works as intended: (1) serializes correctly, (2) is ignored for serialization.

The only thing that is missing currently:

• Have the serialization compiler plugin recognize the auxiliary property (1) as serializable if its delegate class (ReferenceDelegate above) is serializable.

Here is a completely working code example demonstrating the above:
Gist: Serializing delegate properties with kotlinx.serialization

Note: Names enclosed in backticks are not supported by the Android runtime.

Is there a chance of having this seemingly simple solution implemented, or getting a PR accepted?

EDIT: Inconsistent delegate name corrected.

[mgroth0]

@OliverO2 I 100% agree with you. It would be logical for the kotlinx.serialization compiler to recognize if an auxiliary property is serializable. The current behavior can continue to be the default for non-serializable delegates.

Serialization here seems highly straightforward. It is possible to get a reference to a property delegate itself with KProperty0.getDelegate at runtime already.

What I'm curious about is how the delegation process (the "by" keyword) actually works and if that can be done at runtime or it requires compiler magic

@Serializable
  class NameDelegate {
	val name = "rex"
	operator fun getValue(thisRef: Any?, property: KProperty<*>) = name
  }

  @Serializable
  class Dog {
	val name by NameDelegate()
  }

  val dog = Dog()

  // Serializing a delegate is easy 
  // (this is symbolic of what the kotlinx.serialization compiler might setup)
  val json = buildJsonObject {
	val wasAccessible = dog::name.isAccessible
	dog::name.isAccessible = true
	put(dog::name.name, JsonPrimitive((dog::name.getDelegate() as NameDelegate).name))
	dog::name.isAccessible = wasAccessible
  }

  val newdog = Dog()
  // deserializing a delegate is hard
  // this causes an error because val cannot be reassigned. 
  // Also, this doesnt even set up the delegate using the "by" mechanism
  newdog.name = json["name"]!!.jsonPrimitive.content

So my question is how deeply embedded into the kotlin compilation process does this feature addition have to be?

[pdvrieze]

@mgroth0 To make this work correctly (transparently) it would need to be implemented into the compiler plugin. It would be worth to note that it would also need to deal with operator provideDelegate on the deserialization side (especially if it is a read-only delegate).

[OliverO2]

I've looked into the serialization compiler plugin. The point where it decides which properties to serialize seems to be the compiler's frontend (resolving) phase. At that time, the plugin sees the accessor property, which it correctly decides not to serialize. But it does not see the synthetic auxiliary property (...$delegate), which seems to be created in the compiler's backend phase (though I could not spot where exactly).

To find out where to pick up the auxiliary property for serialization (backend IR) code generation, one would need a proper understanding of the interactions between various compiler components (frontend, extension points, backend), in particular the sequencing of those. And there is basically no documentation, so it is quite time-consuming to figure this out. The required change might still be straightforward.

In the meantime, we can still have serializable delegates by adding the auxiliary property manually, as shown above. It's one line of extra boilerplate per delegated property.

[pseusys]

There's no built-in way to do so, but you can write a custom serializer that uses getters and setters of the delegates.

Maybe we can implement @SerializeByGetterAndSetter annotation

This would be also handy in case of serializing and deserializing inline properties in case if their set call with parameters passed from serialized input in constructor is required.

[elizarov]

I'll just leave a note here that this issue is connected to the upcoming "explicit backing fields" feature (see Kotlin/KEEP#278). It will provide a more explicit mechanism than property delegation. That is, when "explicit backing fields" is implemented, one can consider a delegated property like this:

var property: Type by createDelegate()  // CASE (1)

simply to be a shorthand notation for a more verbose explicit backing field declaration like this:

var property: Type // CASE (2)
  field = createDelegate()
  get() = field.getValue(this, ::property)
  set(value) { field.setValue(this, ::property, value) } 

The serialization design will have to take into account those two cases and what happens when the case (1) declaration is expanded into the case (2) declaration.

[wakaztahir]

missing this feature very much

here's something I tried , thinking derived serializer might set the stateful property

    @Serializable
    abstract class Something(open val prop: String)

    @Serializable
    class StatefulSomething : Something("") {
        override var prop: String by mutableStateOf("")
    }
    
    @OptIn(ExperimentalSerializationApi::class)
    @Serializer(forClass = StatefulSomething::class)
    object StatefulSomethingSerializer

    val json = Json {
        serializersModule = SerializersModule {
            polymorphic(Something::class) {
                subclass(StatefulSomething::class)
            }
        }
    }

    @Test
    fun testStatefulSerialization() {
        val state = StatefulSomething()
        state.prop = "hello-world1"
        assertEquals("hello-world1",state.prop)

        val encoded = json.encodeToString(state)
        assertEquals("{\"prop\":\"hello-world1\"}", encoded)

        state.prop = "something-else"
        val encoded2 = json.encodeToString(state)
        assertEquals("{\"prop\":\"something-else\"}", encoded2)

        // This test fails
        val decoded = json.decodeFromString<StatefulSomething>(StatefulSomethingSerializer, encoded)
        assertEquals("hello-world", decoded.prop)
    }

[wakaztahir]

Hi, any updates ?

[sandwwraith]

No, delegates are completely transient right now.

[mdsadiqueinam]

I think this feature will come in k2 compiler

[sandwwraith]

@sadiqueWiseboxs There's no special support for delegates in kotlinx.serialization in K2.

[pdvrieze]

@sandwwraith Perhaps if/when support for access to backing fields is added, this might also make sense to allow it to expose the backing value for a delegate property. At that point the backing value could be annotated/marked as serializable (or not) and handled by the plugin. There are a lot of ifs and buts though including whether to serialize the delegate or the value, and the current approach to having an explicitly named private property as delegate would still make a lot of sense.