Thread pool should support CPU affinity

[oleksiyskononenko]

It seems that currently our thread pool doesn't set CPU affinity, that may have serious performance implications.

[samukweku]

@oleksiyskononenko if I may ask, how did U discover this? What's d implication/relevance? Is it sth you are working on (so I can see how U resolved it)? Is there a way to compare before and after the fix? Thanks

[oleksiyskononenko]

There is nothing to be discovered here, because we are using our own thread pool and we never implemented CPU affinity in that pool. I do not work on this for the moment, but this could be an important next task. Yeah, there are a lot of ways to compare, just benchmark parallel performance before/after implementation on various systems.

[sh1ng]

@oleksiyskononenko if I may ask, how did U discover this? What's d implication/relevance? Is it sth you are working on (so I can see how U resolved it)? Is there a way to compare before and after the fix? Thanks

We use part of datatable's c++ code in our in-house project(s) and we noticed that when ther're a lot cpu consumption from another running processes it may affect dt's thread pool performance. Or when there were a lot of tasks scheduled and dt's thread pool was misconfigured. As @oleksiyskononenko mentioned we need precise benchmarks mimicking those scenarios to see the difference.

[oleksiyskononenko]

I guess we saw the effect even earlier, in particular, #3329 could be related.

[samukweku]

@sh1ng not sure who I should reach out to, since @oleksiyskononenko is not the lead on this project. who can I reach out to , in terms of PRs and more?
plans for the datatable project, etc. thanks

[sh1ng]

I'll discuss this with @st-pasha and some other folks.

[sh1ng]

@samukweku thank you for your contribution. As you see there's no active development for the project. I've started an internal discussion about possible options.

[samukweku]

@sh1ng glad to hear. I do hope activity resumes soon. It is a lovely project