artificial workflow limit

[oliver-sanders]

There's an artificial limit to the number of workflows that will update in the GUI based on the number of cores on the system.

Instructions to reproduce:

1. Install and start a workflow.
2. Navigate to it in the tree view.
3. Leave this workflow running and goto (1).

Repeat the above instructions enough times and you will eventually end up with a workflow that does not update. The tree view will show the initial tasks in the workflow, but the workflow itself could be many cycles ahead of this state.

The number of repetitions of the above instructions required to replicate the bug is the number of virtual CUP cores on your system plus 5.

Cause

We are using a TheadPoolExecutor to manage workflow subscriptions in threads:

cylc-uiserver/cylc/uiserver/data_store_mgr.py

Line 90 in 49fa3ef

self.executor = ThreadPoolExecutor()

cylc-uiserver/cylc/uiserver/data_store_mgr.py

Lines 150 to 156 in 49fa3ef

	self.executor.submit(
	self._start_subscription,
	w_id,
	contact_data['name'],
	contact_data[CFF.HOST],
	contact_data[CFF.PUBLISH_PORT]
	)

But, we don't specify the maximum number of threads so we get the default which is the number of virtual cores on the machine plus 4:

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor

So on a dual core machine (2 physical cores = 4 virtual cores) the maximum number of workflows that will update is 8! Ouch!

Solutions

1. Configure a larger number.

• What's the overheads of lots of threads?
• Is this a problem?

2. Stop using threads.

• See Revisit use of multiple subscription sockets/threads #194

[wxtim]

Replicated on my ol' home laptop. (Ancient processor)

[oliver-sanders]

Closed by #570