No entities displayed in /config/entities, and severe lag

[Mariusthvdb]

Checklist

• I have updated to the latest available Home Assistant version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in safe mode to rule out problems with unsupported custom resources.

Describe the issue you are experiencing

no entities are displayed, and the filter box really lags, in any Desktop browser, iOS app shows fine.

navigating away after trying to load the entities is also suffering the lag, and I need to reload the browser view, to be able to browse the instance again on other views

Describe the behavior you expected

show all entities when nothing is filtered, and show filtered entities when a filter is actually entered

Steps to reproduce the issue

1. browse to /config/entities

...

What version of Home Assistant Core has the issue?

2024.6.0b8

What was the last working version of Home Assistant Core?

2024.5

went super fast and smooth

could it be https://www.home-assistant.io/blog/2024/06/05/release-20246/#data-tables-improvements is acting up?

In which browser are you experiencing the issue with?

Any: Safari, Edge, Chrome, Firefox dont show

iOS companion app shows fine (which also suggests this is not about the number of entities?)

Which operating system are you using to run this browser?

Sonoma 14.5

State of relevant entities

No response

Problem-relevant frontend configuration

No response

Javascript errors shown in your browser console/inspector

No response

Additional information

Seems to be mainly an issue on entities, not on devices or helpers. Although on those 2, the lag is also very noticeable compared to 2024.5

Inspector on local storage shows:

[Mariusthvdb]

if I click on an integration in the integration dashboard, I can see its entities after a very long wait:

going back to the ui, and revisiting the config/entities dashboard again after that 'remembers' those integration entities.... (which is very clumsy, because one doesnt go there for the preselection, but for the overview)

We need to clear the filter for Integrations, but, even then the preselected entities remain in sight:

the entities count shows the filters work, the entities list however remains as is

Jun-05-2024.15-07-23.mp4

[Mariusthvdb]

Ive just noticed that Collapse all on the Helpers dashboard does not collapse at all, I can only expand all

Jun-08-2024.17-38-37.mp4

[Mariusthvdb]

ok so current.ly running HA 2024.6.3, and I did discover something I hadn't tested before:

I can see all of my entities in the iOS companion app.

But, ONLY there. Any browser on the Desktop remains empty. Including the Mac Desktop app.
I tested Chrome, Firefox, Edge and Safari, all in their current up to date versions.

[RMMTSLLP]

Hi I'm having exactly the same issues. Only seems to be with 1 out of the 2 installations I have. Has this been sorted yet? It's quite frustrating.
Regards

[Mariusthvdb]

can you check which integrations/custom cards etc differ between the 2 installs?

and what size installs (number of entities) are you talking about?

[RMMTSLLP]

The two setups are really different the one that is working running the latest version 2024.6.3 has 592 entities and 15 integrations. The broken setup was working on version 2024.5.5 entities 2637 integrations 34. I'm just going through some more backups to see when the issue first occurred. I assume you still have this same problem, looking on the web not many of us have this issue.

[Mariusthvdb]

yep, currently investigating.....
for me it was the beta of ha 2024.6 that directly introduced it

They did change something in the new filter engines of the entities dashboard, the data tables, which was widely announced, and to me, that is the only possible reason that stands out.

I havent changed anything in my custom cards during that period.

I could roll back to 2024.5 but fear the issues the setup (production) would suffer ....

[Mariusthvdb]

extra info:

I just restarted in Safe mode, and, after an initial improvement of seeing the entities populated (took more than a few seconds, so much slower than before), I am back to seeing nothing:

So, still in safe mode, after navigating to some other views and back again

[RMMTSLLP]

I can confirm all was well with;

Core
2024.5.5
Supervisor
2024.06.0
Operating System
12.3
Frontend
20240501.1

With this restored from backup file I see no issue with config/entities

I ugraded to ;

Core
2024.6.3
Supervisor
2024.06.0
Operating System
12.3
Frontend
20240610.1

The non display of config/entities returns. Sometimes the page does populate (Very rarely) also sometimes the page seems to lock up stopping me from navigating away to another page (Quite Often).

Is there anything you would like me to try before I install my latest backup so I can get back to a functioning system without "config/entities"

[Mariusthvdb]

glad I am not alone, that's all I can say...
I reported during beta, but it was ignored unfortunately.

It would be useful if you could state wether or not you added custom cards/integrations between 2024.5 and 2024.6, to rule out any custom problem.

I believe to have proven with the Safe mode restart above that cant be the source of the issue at hand, but always better to positively identify.

The non display of config/entities returns. Sometimes the page does populate (Very rarely) also sometimes the page seems to lock up stopping me from navigating away to another page (Quite Often).

100% my experience.
I briefly saw the entities list being populated in my Safe mode test, but quickly thereafter all were gone again, and the UI lag was all over.

Hope this will get some attention of the core Frontend devs soon.

[RMMTSLLP]

OK I tried running in safemode really no joy on my PC ....... however when running in safe mode if I use my S21 (Android) + Home Assistant App the config/entities seems to perform ok much more reliable on the PC very little delay less than 2 seconds

[RMMTSLLP]

Screen_Recording_20240617_105833_Edge.mp4

Hope this is it

[Mariusthvdb]

just downgraded to 2024.5.5 and see all entities like before... so definitely a HA 2024.6 thing

i had to update to 2024.6 though, because of the File integration migration....

I use my S21 (Android) + Home Assistant App the config/entities seems to perform ok

yes, as reported in the OP, my iOS companion app is showing the entities correctly. Only device though..

back on 2024.6 entities are gone again

[RMMTSLLP]

Sorry to eger and missed that bit.

[Mariusthvdb]

some new discovery:

with the Inspector local storage tab open, I noticed the earlier reported ha-filter-floor-areas being empty, and that made me think if it simply didnt cause a Non selection of all entities.....

so I tried to + a few areas, floor areas to be precise, and they were ticked and immediately un-ticked. I did notice the line in the right panel being edited with some of the entities belonging to the areas I selected.

what's more, I noticed entities being listed in the entities list! Finally!

Apparently we need to select floor areas for any entity to be listed. This can not be can it? I have a lot of areas in the house, which are grouped in floors (several floors per actual physical floor).

Wouldnt it be 'normal behavior' if one does not select a floor, that all floors are selected and the entities of those floors would be listed? ofc it would, just like in the Status filter. Select nothing, show all.

The Helpers view does work like that, why the different behavior?

another anomaly here is, that when adding a top level floor, none of the areas are auto ticked:

which makes it even more difficult to understand what is selected and what not.

besides that, the lag is now very visible, and understandable, if it actually adds all of these entities to the list in the local storage, and does so each and every time it changes.

causing this:

and making that non-manipulable. need to reload there to be able to get to the tick boxes again.

the fact this is a per browser setting, meaning if I have different panes open on the same machine (1 for editing, 1 for browsing, 1 for finding entities) and accidentally navigate in the wrong pane, I am in trouble all over....

@RMMTSLLP please check if you can confirm this?

[RMMTSLLP]

OK I've done some more testing and I think I can confirm what you are saying is correct. However I have to be in "safe mode to really make sense. I agree in config/helpers it defaults to display ALL helpers. You can then go on to search as required. In config/entities it defaults to show NO entity data. If you select an area (the tickbox briefly shows and dissappears) after a period entities for that area are displayed. When HA is NOT in "safe mode" I get no sense whatsoever. The screen freezes sometimes on a refresh it displays entity data, more often not. It certainly is not usable unlike config/helpers.
bedroom.pdf

[karwosts]

I can make a guess of what is happening here, my guess is that your system may be just overloading the data table.

In frontend most components are updated every time the hass object updates, and that update is triggered anytime any entity on your system changes its state object.

With the entities data table, it seems that anytime that happens (any state changed), we attempt to re-sort and re-filter all the entities in the table. As the amount of entities in the system grows, the sorting time takes longer and longer. As the frequency of updates increases, the rate at which we are asked to sort and re-sort grows more and more frequent.

I suspect eventually this will hit some critical threshold where we are repeatedly asked to reupdate the sort order before we're able to finish sorting. I don't know what happens exactly in this case, but it seems like when this happen the system breaks down and it just appears to hang and become completely unresponsive. That may be why it seems to behave erratically, sometimes appearing to work for only a moment, and when it does work it is exhibiting very slow performance.

I can observe such by just creating an automation that toggles a boolean every 5ms. Anytime this automation is enabled, I cannot get my entities table to do anything, it just says "no data" until I disable the automation.

Maybe you can try running the performance profiler in your browser devtools, and see if its spending 100% time in the render/update process.

[Mariusthvdb]

Thanks for chiming in! we're desperate... ;-)

With the entities data table, it seems that anytime that happens (any state changed), we attempt to re-sort and re-filter all the entities in the table. As the amount of entities in the system grows, the sorting time takes longer and longer. As the frequency of updates increases, the rate at which we are asked to sort and re-sort grows more and more frequent.

could this be happening in 2024.6, and Not in 2024.5? Because thats when this started happening, 100% reproducible.

I can observe such by just creating an automation that toggles a boolean every 5ms. Anytime this automation is enabled, I cannot get my entities table to do anything, it just says "no data" until I disable the automation.

dont think there is such a heavy updating going on in my system, but I do have a few that are quite active... most of those are related to the power metering of the smart meter, and some templates based off of those.

that however has never bitten the Frontend in the least.

which makes me suggest this change to the data_tables is simply not up to par with modern requirements. and should be reverted really.

I mean, the fact that 2 posters here are suffering this, does not mean nobody else is. Seems highly unlikely given the exact same experience we notice.

Most users will probably never go to that view, so might never notice.

Maybe you can try running the performance profiler in your browser devtools, and see if its spending 100% time in the render/update process.

I'll do anything you want me to, but please be clear what exactly you need me to do. I can run profiler without issue, it's my main correspondence with BDraco ;-)

or are you looking for some other tab in Inspector?

btw, the fact this does Not happen on iOS companion app would seem to indicate this is not a system backend issue, but a Desktop Frontend issue? How else can my mobile phone show the entities without a sigh

[karwosts]

As far as I know what I'm describing is not a recent change, it's always been that way. If something changed in 2024.6 I have no idea what that could be.

Maybe something subtle changed that just pushed it over the edge.

[Mariusthvdb]

It was the announced change to the data tables probably https://www.home-assistant.io/blog/2024/06/05/release-20246/#data-tables-improvements

without those, in 2024.5 all was super smooth

unless ofc there would have been other backend changes that could cause heavy lifting.

[Mariusthvdb]

let me just keep this alive in 2024.7.1, still not getting a solid response on the entities pages

[RMMTSLLP]

I can confirm the issue still remains the same in 2024.7.1 is there anything we can do to bring this higher up the priority list?

[Mariusthvdb]

Could this be related? I can't recall seeing this before but the message seems related ?

Bron: components/homeassistant/exposed_entities.py:388
Eerst voorgekomen: 17:33:01 (1 gebeurtenissen)
Laatst gelogd: 17:33:01

Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 515, in _async_callback_delayed_write
    await self._async_handle_write_data()
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 540, in _async_handle_write_data
    await self._async_write_data(self.path, data)
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 545, in _async_write_data
    await self.hass.async_add_executor_job(self._write_data, self.path, data)
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 552, in _write_data
    data["data"] = data.pop("data_func")()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/homeassistant/exposed_entities.py", line 388, in _data_to_save
    "exposed_entities": {
                        ^
RuntimeError: dictionary changed size during iteration

then again, maybe not, this is 'exposed' to the Assistant..

[Mariusthvdb]

where I said this in the Opening post:

iOS companion app shows fine (which also suggests this is not about the number of entities?)

this is now no longer the case, and mobile stopped being reliably responsive on entities view also in beta 2024.8.0. Here too now do I have to re-navigate to the dashboard a few times to get the list populated. Not yet as bad as on desktop, but starting to worsen for sure

it seems clear that the new options in the search filters keep making these pages heavier for the more fleshed out instances.
actually breaking the entities page
adding more will only make things worse, as happened now in 2024.8, where the selection of 2 new properties can be done, and saved....

As mentioned in the issue now also for helpers #21530

[Mariusthvdb]

separate post here too: see #21530 (comment)

main difference in the Helpers page in 2024.8 is the addition of the Yaml entities. In my case, only template entities (and 1 group...)
Since these are also shown in the Entities dashboard, could it be these are the culprit of the issue?

Given the fact these templates are constantly being evaluated, as that is what templates do, maybe there is something written in local cache/storage all the time, and blowing up because of that??

could it be this after all then #21001 (comment)

[Mariusthvdb]

I can make a guess of what is happening here, my guess is that your system may be just overloading the data table.

In frontend most components are updated every time the hass object updates, and that update is triggered anytime any entity on your system changes its state object.

With the entities data table, it seems that anytime that happens (any state changed), we attempt to re-sort and re-filter all the entities in the table. As the amount of entities in the system grows, the sorting time takes longer and longer. As the frequency of updates increases, the rate at which we are asked to sort and re-sort grows more and more frequent.

I suspect eventually this will hit some critical threshold where we are repeatedly asked to reupdate the sort order before we're able to finish sorting. I don't know what happens exactly in this case, but it seems like when this happen the system breaks down and it just appears to hang and become completely unresponsive. That may be why it seems to behave erratically, sometimes appearing to work for only a moment, and when it does work it is exhibiting very slow performance.

I can observe such by just creating an automation that toggles a boolean every 5ms. Anytime this automation is enabled, I cannot get my entities table to do anything, it just says "no data" until I disable the automation.

been debugging the Helpers page for the same reason, so I figured to return here with some results: I can see this in the debug log

and this list goes all the way down to the last entities, which in my config are zones

and yet, after having received this complete list of entities (which took more than a few seconds btw) there is still nothing displayed in the view itself

reloading the view, makes it wait a very long time, I then see the connection lost Toast, and after that see

and finally the list is populated. Not complete, most icons aren't there....

Maybe you can try running the performance profiler in your browser devtools, and see if its spending 100% time in the render/update process.

I did the profiler before, but maybe you mean something else here? I did turn on the debug tools if that is what you mean?

[karwosts]

I think at one point you were going to try disabling some of your most noisy entities, to see if that helps? Did anything ever come of that? If you want help profiling which entities are updating most heavily, I can try give you a javascript to count updates.

I'll mention that I setup an instance with 15,000 random sensors (roughly the max expected entities), and an automation that toggles once a second, and while sluggish a bit I didn't have any problem getting the table to functionally display.

So I still feel like maybe you've got just too many entities doing too many updates for the table to keep up, or some kind of bad infinite circular loop in templating or something, causing a very high rate of entity update.

[RMMTSLLP]

Hi Guys, just to confirm that I still have the original issue which was introduced by HA 2024.6 and is still present in 2024.7.4 from what I see and do not understand if we had systems with high device and entity count why was this not taken into account with subsequent updates. The system was capable of displaying the high number of entities then alas not now . This seems to be a breaking change and should be given some importance. It seems we/you are chasing a symptom and not the cause. If the system cannot handle high entity/device counts surely this should be restricted/flagged when devices are added or when upgrades are implemented or fixed so the system behaves as per before 2024.6 . But more importantly the changes made at HA 2024.6 should be sorted.

To my mind trying to reduce devices/entities and their subsequent reporting is totally the wrong approach.

If there is any testing you require I am happy to help I must add my in depth knowledge of HA is limited.

This is in no way intending to upset anyone and the work you are doing I just think maybe we're missing the point here.

Regards,

Roger Morris

[Mariusthvdb]

I think at one point you were going to try disabling some of your most noisy entities, to see if that helps? Did anything ever come of that?

No nothing. Ive done it all: test in safe mode, factually disabling all of the custom integrations entities, and frontend stuff. No change.
I changed my most intense trigger based template from updating based on the smart meter, to once a minute. No change

I dont see any abnormal percentages in any of the sensors that indicate either the backend or the frontend is doing overtime.

So, I fear there is just nothing to be done on my part in the config.

I did try to disable the template entities in the Helper dashboard, and not select the integration in Entities, but that is practically impossible given the tools we have.
The effort I did, had no result

If you want help profiling which entities are updating most heavily, I can try give you a javascript to count updates.

I would appreciate anything that could help me solve the Home Assistant issue is creating here, please yes.

I'll mention that I setup an instance with 15,000 random sensors (roughly the max expected entities), and an automation that toggles once a second, and while sluggish a bit I didn't have any problem getting the table to functionally display.

As said, I stooped believing it has anything to do with the number of entities. Why else would 2024.5 have had no issue at all, at either of these pages.

So I still feel like maybe you've got just too many entities doing too many updates for the table to keep up, or some kind of bad infinite circular loop in templating or something, causing a very high rate of entity update.

I could still be the way the frontend is now programmed indeed, Its the only thing at this time I can figure it to be.

It was not until 2024.6 introduced the new filter options, and, maybe even more importantly, started writing the preferences and the states to the cache, that these troubles started.

And again, after new features were introduced in 2024.8 beta right now, the Helpers page becoming completely unreliable.

seems a 1-1 relationship, and anything I can doing to reduce the number or the frequency has not helped.
Ive been debugging the Helpers issue with @silamon who Ill tag here once, so the 2 of you are matched .

[parautenbach]

In frontend most components are updated every time the hass object updates, and that update is triggered anytime any entity on your system changes its state object.

With the entities data table, it seems that anytime that happens (any state changed), we attempt to re-sort and re-filter all the entities in the table. As the amount of entities in the system grows, the sorting time takes longer and longer. As the frequency of updates increases, the rate at which we are asked to sort and re-sort grows more and more frequent.

In my case, I have over 1000 entities. Of these, about 50 update every second, with then a number of template sensors and automations triggering off that. I can definitely see this being a possible cause for the issues.

I'll mention that I setup an instance with 15,000 random sensors (roughly the max expected entities), and an automation that toggles once a second, and while sluggish a bit I didn't have any problem getting the table to functionally display.

Could you up the frequency for 50-100 of those to once per second?

[RMMTSLLP]

Hi Guys,

Correct me if I'm wrong does not "/developer-tools/state" display the same table as "/config/entities" if it is why does developer-tools display correctly and not config?

Also why was it working before 2024.6 and not after, my entity count has changed much, I bet if I downgraded to 2024.5.5 all would be well again.

Looking to upgrade to 2024.8.0 ..........maybe :-)

[Mariusthvdb]

well, maybe the same table.

however, dev tools states has been a hog since forever, and it can take up to 30 seconds in my instance to finally show.
once it does however, we can simply filter anything and it is really very quick. I just need to take care the search string doesn't become cleared again or otherwise its starts becoming very slow once more. I prevent that from happening by adding the new search item in the filter box, and then delete the former string

so first search for sensor.xyz, then if I need switch.abc, I dont clear sensor.xyx, but type sensor.xyzswitch.abc and then delete the sensor.xyz.... I know it's hacky but it gets the job done (btw, same in the attributes column (which is the real star of this page), to find the exact attributes if we need them for templating)

this is also exactly why we can never do without the dev tools states, and the newer config/entities is absolutely no replacement.

Getting back to the matter at hand:
exactly because the dev tools states has been slow for years up to the unacceptable and was never fixed (other than not landing on the States view, but on the Yaml view when navigating there for the left side menu), it was with pleasure I visited the /config/entities up to 2024.6. And then this started to be broken....

downgrading is no option I am afraid, and upgrading doesnt fix it either.....

the only way I can get entities to show at all, is when navigating from an Integration in the Integrations view and click its entities. Apparently at that time the filters work ok, and show me the selected Integration entities

even when navigating from Zwave, which alone has over 2000 entities, clicking that and they stand immediately

[ildar170975]

I have a rather small setup with ~3000 entities.
Most sensors are updated with a frequency >= 60 sec; device_trackers (~30..40 entities) are updated more frequently.
Using no labels.
Client = Chrome + Win10, server = i5-2500K.
See no lags in Settings -> Entities.

It was speculated earlier that may be recently added "modified_at" property may affect some lags.
But according to my tests, this "modified_at" is not updated at state/attributes change.

[karwosts]

Ok @Mariusthvdb lets try a little experiment (or anyone else who's having same issue who wants to participate)

Open your entities page in Chrome browser, and open the web developer tools (Ctrl-Shift-I).

In the developer tools console, paste the following:

function patchMyFunc (funcName) {
    var context = document.querySelector("body > home-assistant").shadowRoot.querySelector("home-assistant-main").shadowRoot.querySelector("ha-drawer > partial-panel-resolver > ha-panel-config > ha-config-entities").shadowRoot.querySelector("#entity_id").shadowRoot.querySelector("hass-tabs-subpage > ha-data-table");
    var oldFunc = context[funcName];
    context[funcName] = function() {
        ++context[funcName].count;
        const now = new Date();
        const dt = now - context[funcName].time;
        context[funcName].time = now;
        console.log(`running ${funcName} (elapsed ${dt}ms) `, arguments[0].keys().toArray())
        return oldFunc.apply(context, arguments)
    }
    context[funcName].count = 0;
    context[funcName].time = new Date();
}

After you paste that (and hit enter), then type: patchMyFunc("willUpdate") and hit enter.

You should start to see a flood of console logs of data table being updated. I put a timestamp counter in there to track how fast it is updating. On my table it's fairly sleepy, often going 5-10 seconds between updates.

I'm guessing that you're going to get a firehose of events every few milliseconds, maybe even so much it can't keep up rendering them. But would be interesting to see what happens.

Should look something like this:

Let me know what you see as the output (screenshots good)

[PlayFaster]

Hi. Thanks for this. My results below. Seeing updates every 100ms to 200ms on average. No entities display (as per this issue).

My setup: HA 2024.7.4, with 4.7K entities. I have a power monitor that updates every 3 to 5 seconds, but its been doing that for over two years. No other integrations or templates that I am aware of that update super-fast (although it seems there must be something?).

[karwosts]

Updated patch function for debugging that will print the entities that changed on each hass update:

function patchMyFunc (funcName) {
    var context = document.querySelector("body > home-assistant").shadowRoot.querySelector("home-assistant-main").shadowRoot.querySelector("ha-drawer > partial-panel-resolver > ha-panel-config > ha-config-entities").shadowRoot.querySelector("#entity_id").shadowRoot.querySelector("hass-tabs-subpage > ha-data-table");
    var oldFunc = context[funcName];
    context[funcName] = function() {
        ++context[funcName].count;
        const now = new Date();
        const dt = now - context[funcName].time;
        context[funcName].time = now;
        console.log(`running ${funcName} (elapsed ${dt}ms) `, arguments[0].keys().toArray())
	
	if(arguments[0].get("hass") && context.hass?.states) {
		console.log(Object.entries(context.hass.states).filter(([id,state]) => state !== arguments[0].get("hass").states[id]).map(([id, state]) => id))
	}

        return oldFunc.apply(context, arguments)
    }
    context[funcName].count = 0;
    context[funcName].time = new Date();
}

[Mariusthvdb]

thanks Karwosts, that looks very impressive!

here is my output with the latest script:

and a screen video

very interesting to see what is actually happening in the system for once ;-)

I do hope the changes that were brought to the Frontend can be reverted, so the backend goings on wont keep the Frontend from behaving nicely once again.

Nothing we can do about our sensors being what they are of course.

[Mariusthvdb]

btw, as were now also seeing the degradation on the helpers page, I edited it to run in helper:

function patchMyFunc (funcName) {
    var context = document.querySelector("body > home-assistant").shadowRoot.querySelector("home-assistant-main").shadowRoot.querySelector("ha-drawer > partial-panel-resolver > ha-panel-config > ha-config-helper").shadowRoot.querySelector("#entity_id").shadowRoot.querySelector("hass-tabs-subpage > ha-data-table");
    var oldFunc = context[funcName];
    context[funcName] = function() {
        ++context[funcName].count;
        const now = new Date();
        const dt = now - context[funcName].time;
        context[funcName].time = now;
        console.log(`running ${funcName} (elapsed ${dt}ms) `, arguments[0].keys().toArray())
        return oldFunc.apply(context, arguments)
    }
    context[funcName].count = 0;
    context[funcName].time = new Date();
}

but apparently not good..... getting a type error on the path.. Could you please check that ?

[karwosts]

Thanks for the captures.

I do hope the changes that were brought to the Frontend can be reverted, so the backend goings on wont keep the Frontend from behaving nicely once again.

I don't think anything is going to be reverted, (I wouldn't even know what to revert if I wanted to), but I'm somewhat confident this is just a small bug that maybe needs a slight change in how the asynchronous sort worker interacts with the data changes.

But for now this is still in a data gathering stage to really understand the problem. I have suspicions, but I can't say I understand it 100% yet.

[Mariusthvdb]

I don't think anything is going to be reverted,

Ofc I didnt mean the new functionality! I like the way that is going. I meant the troubles were seeing ;-)

Please have look at the script for Helpers too?

[karwosts]

Helpers is mainly the same deal but the context path is just different. You can replace with this line if you want:

    var context = document.querySelector("body > home-assistant").shadowRoot.querySelector("home-assistant-main").shadowRoot.querySelector("ha-drawer > partial-panel-resolver > ha-panel-config > ha-config-helpers").shadowRoot.querySelector("hass-tabs-subpage-data-table").shadowRoot.querySelector("hass-tabs-subpage > ha-data-table");

[Mariusthvdb]

hmmm seems worse than entities!?
And where I figured this would only update on actual Helpers, that appears not to be the case.. How can it be this is seeing way more updates than the entities page?

[silamon]

Most calls to websocket are throttled or debounced, seems like the states aren't making it render pages multiple times per second. I think a throttled needs to be added.

[teskanoo]

This definitely needs to be throttled. The increase in traffic has hung several of my pages and views in home assistant.

The increase in data transfer between home assistant and my phone has actually been so huge that my very small 1 gig of monthly transfer has now been consumed on the 8th of the month.

[Mariusthvdb]

Wow, I hadn’t even checked that. Normally I won’t get past the 2 gb per month.

right now this has already been surpassed by HA alone in the first week …

ofc I can’t prove this or be exact about the cause, but it is remarkable, and new

[karwosts]

I don't believe the throttling proposed here has anything to do with actual data transfer between client and server, but just the redraw rate of the lit components.

If you want to discuss client data usage that's probably something for different thread.

There is no known "increase in traffic" as far as I'm aware. The server has always sent a firehose of all state updates to any active clients.

[Mariusthvdb]

I don't believe the throttling proposed here has anything to do with actual data transfer between client and server, but just the redraw rate of the lit components.

If you want to discuss client data usage that's probably something for different thread.

There is no known "increase in traffic" as far as I'm aware. The server has always sent a firehose of all state updates to any active clients.

@teskanoo would you open that separate Issue/discussion please, as you brought it up here?

[silamon]

If it doesn't increase or slightly increase anymore since 2024.8.0 it should be fixed. During beta, the service worker was malfunctioning which may resulted in higher downloads during that period.

Also, please remain on topic in this thread.

[Mariusthvdb]

now that I think of it, Ive noticed the same regression in performance using custom: auto-entities when the label filter was introduced.

Aware this is custom, but, could this now identified issue also have affected auto-entities, and, more importantly, could the throttler PR also fix it there?

Or, should we point Thomas Loven to that PR, to maybe include that into his custom card?

[silamon]

I've locked this conversation since you keep going off-topic and pulling things in here that don't belong to it.