ModelSerializer performance

[salomvary]

Hi folks,

I was working on optimizing the performance of a small(ish) project we have in production and found out that most of our computing time is spent in ModelSerializers. I've made some isolated benchmarks and found out that ModelSerializers are pretty pretty slow when compared to Serializers, roughly 3 to 6 times slower, depending on the method of measuring.

I've looked through issues and discussions to see if someone has done research and optimization work before but have not found much. I understand that ModelSerializers involve more runtime magic, but is this 3 to 6 fold difference in speed realistic?

I've had a brief look using a profiler but found no obvious low hanging fruits (other than sometimes using except clause on the happy path depending on the use case).

Given the maturity of DRF I'd be surprised to be the first one looking into this, but I have some time available for having a closer look. Any advice or pointers would be appreciated.

(I am aware that caching might be a way to optimize for performance, but in our case responses are dynamically generated for pretty much all requests. And of course rewriting to plain Serializers is still an option, but hoping that we don't have to go down that path.)

Thanks!

[TGoddessana]

I'm not sure if this is what you're looking for, but tomchristie, the creator of the django rest framework, wrote this article about performance.

https://www.dabapps.com/insights/api-performance-profiling-django-rest-framework/

Just out of curiosity, is the latency of the ModelSerializer significantly slower than other operations like database connections, sql queries, etc?

[salomvary]

Thanks for the article, will read through it. I wish database operations were our bottleneck though, but no, it seems to be serialization with ModelSerializers.

I've conducted a simple "end-to-end" benchmark against a toy DRF app that looks up data from a database and serializes using Serializers or ModelSerializers, and the difference is striking. I got 130 ms average response times with ModelSerializer and 40ms with Serializer on an Apple M1 Max, serializing 1000 objects with ~10 text fields in each request. The experience with a production app is similar.

[onegreyonewhite]

@salomvary I completely agree with you. The main issue with ModelSerializer is that it builds fields on the fly during the serializer's initialization, rather than during class initialization. This is particularly noticeable with lists. The impact on performance can be mitigated by declaring all serializer fields manually, but then the benefits of using ModelSerializer are greatly reduced.

Additionally, we can't modify the field list of such a serializer without encountering a desynchronization error. This is because the framework checks that the field list in Meta matches the declared fields. I believe this is the error they were trying to bypass.

I think it would be correct to have a pool where undeclared fields are stored during the first initialization of the serializer or class. All subsequent initializations would pull ready-made fields from this pool if they were not declared. In this case, the pool would consist of the model class as the key, and the value would be a dictionary of fields and their names.

If the developers approve this proposal, I could easily implement it.

[salomvary]

The impact on performance can be mitigated by declaring all serializer fields manually, but then the benefits of using ModelSerializer are greatly reduced.

@onegreyonewhite Yep, I definitely like the declaration convenience benefits of ModelSerializer :)

I even played with the idea of hacking together a blend of Serializer and ModelSerializer that takes a ModelSerializer-like configuration and generates a true plain Serializer that can not (easily) be modified run-time, but which likely covers 99% of the use cases.

[tomchristie]

If the developers approve this proposal, I could easily implement it.

You don't need anyone else's permission. Implement it and share either as a gist or a third party package. Job done.