New [Experimental] Release: 0.7.5-beta

[freakent]

Dear All,

I have made a new release of the dbus-mqtt-driver that completely removes the YAML dependency from this project. This should address the problems that have plagued this project for months caused by the need to install the PyYaml library on Venus. It should also pave the way to make the driver installable by setuphelper in a subsequent release.

I have switched to using ini files which are natively supported by Python. This feels like quite a substantial change to the internals of the driver, so I am releasing this first version without YAML as a BETA. I have performed a lot of testing myself but I would like to collect feedback from users before making a final release.

To upgrade just follow the install instructions in the v0.7.5-beta readme as before.
https://github.com/freakent/dbus-mqtt-devices/tree/v0.7.5-beta

If you try this release, please comment below whether it works successfully for you or not. All feedback is appreciated.

-- updated release number to v0.7.5-beta

[freakent]

This release addresses issues #55, #56 and #57.

FYI @stundenblume, @Hofyyy, @GallusMax, @pos-ei-don, @stromibaer

[SoulRaven]

Version: 0.7.4-beta

something is wrong i guess, i have this payload on "device/serverRoom/Status ":

{
"clientId": "serverRoom",
"connected": 1,
"version": "v0.7.4-Beta",
"services": {
"t1": "temperature"
}
}

but i get:

@4000000064eb42d006c17054 INFO:device_manager:Received device status message {'clientId': 'serverRoom', 'connected': 1, 'version': 'v0.7.4-Beta', 'services': {'t1': 'temperature'}}
@4000000064eb42d006f8ef5c WARNING:device_manager:status.service type temperature is not supported, please check services.yml

[freakent]

@SoulRaven Thanks for reporting that. New beta version 0.7.5-beta available.

[SoulRaven]

thx, now is working, but some things are not setup,
version, and device name using instance ID.

also, is it possible to setup the "Name" directly from device topic?

L.E.

@4000000064eb73613055a264 INFO:device_service:Registering service temperature for client ha_sonoff_th at path com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room
@4000000064eb736131562274 this is what we found <configparser.ConfigParser object at 0xb5f0a220>
@4000000064eb73613156459c INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted
@4000000064eb736136a3ccf4 INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/TemperatureType does not exist yet or must be adjusted
@4000000064eb73620056ebf4 INFO:settingsdevice:Setting /Settings/Devices/mqtt_ha_sonoff_th_server_room/ClassAndVrmInstance does not exist yet or must be adjusted
@4000000064eb7362063ede8c INFO:vedbus:registered ourselves on D-Bus as com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room
@4000000064eb736207892554 INFO:device_service:Registered Service com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room under DeviceInstance 3
@4000000064eb736207ca0e84 INFO:device_manager:publish {'portalId': 'c0619ab2f11e', 'deviceInstance': {'server_room': '3'}} to device/ha_sonoff_th/DBus, status is 0
@4000000064eb7362082430bc INFO:device_service:setting changed, setting: CustomName, old: My Temperature Sensor, new: My Temperature Sensor
@4000000064eb7362086eb6dc INFO:device_service:setting changed, setting: TemperatureType, old: 2, new: 2
@4000000064eb737823e10d0c INFO:device_manager:Received device status message {'clientId': 'ha_sonoff_th', 'connected': 0}
@4000000064eb737828f6e67c INFO:device_service:Unregistered mqtt_ha_sonoff_th_server_room from dbus
@4000000064eb73782903b3d4 INFO:device:Removed Service server_room from client ha_sonoff_th
@4000000064eb737829102754 INFO:device_manager:**** Unregistering device ha_sonoff_th ****

[freakent]

Sorry, can you clarify

thx, now is working, but some things are not setup,version, and device name using instance ID.

[SoulRaven]

Sorry, can you clarify

thx, now is working, but some things are not setup, version, and device name using instance ID.

the Device name is not set also the Firmware version. What i have see in code Firmware Versions contains the value from payload "version" and the device name is the "client id"
In log seams that

INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted
somehow is not setup the costumName

Also will be great to have the possibility to setup directly from topic the "Name" of the sensor

[freakent]

This has me puzzled, it's not an area in the code that was changed. I checked in MQTT explorer what is being returned by Venus and it does include these values:

although they do not show in the UI

I am testing on pre-release Venus 3.1-32, so could be a venus bug. Which version of Venus OS are you testing on?

[freakent]

INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted somehow is not setup the costumName

This is just a debug message output by the Venus dbus library, it is always output when initialised.

Also will be great to have the possibility to setup directly from topic the "Name" of the sensor

I don't want to complicate the registration process. Once registered and you have a deviceid you can send a message to W/<portal id>/temperatire/<deviceid>/CustomName to change the name programmatically. You can do this for any of the parameters you wish to set not just CustomName.

[SoulRaven]

i have attached the printscreen with VenusOS version

[Becker884]

Hi, i tried today with 3.01 on RPi2, setup takes only few seconds, my Gridmeter & PV runs:

only the " " weren't there before

do you have that too?

[freakent]

@Becker884 Thanks for trying it out and spotting this error. I'll update the grid.ini config file to remove the quotes in the next release.

# Total bought energy over all phases
description: "kWh"
format: "{} kWh"

[Ac/Energy/Reverse]
# Total sold energy over all phases
description: "kWh"
format: "{} kWh"
  
[Ac/Power]
# Total power of all phases, preferably real power
description: "W"
format: "{} W"

[Ac/Current]
description: "A AC"
format: "{} A"

[Becker884]

Can I correct this on my own?
if yes -> where to find the grid.ini?

[freakent]

Yes, you can. It's services/grid.ini

See https://github.com/freakent/dbus-mqtt-devices/blob/Config-Files/services/grid.ini

[Becker884]

ok, but where can I find this file on my RPi2 ?

[freakent]

/data/drivers/dbus-mqtt-devices-0.7.5-beta/services/grid.ini

[gharshaw]

@freakent Does the latest release remove the YAML dependency? I thought the 0.7.x versions is where you are removing that dependency

[freakent]

That's correct, 0.6.x uses pyyaml.

[DJZorro]

Thank you for your driver,
With this I was able write to the dbus.
I picked the weatherinfo from Openweather.API.
Imported it in OpenRed.
Pulled out the information I wanted to display
Modified some date, Wind direction, wind speed, Sunrise and Sunset.
Pubished those data with MQTT and your driver to dbus.
Picked them up from dbus by with modify gps.ini
Pubished them on the screen with modifying OverviewHubEnhanced.qml

Long way but succesfull in the end:

[gurkc006]

Hi!
Just a comment from me: I used 0.7.5-beta for the last weeks without any problems on VenusOS 3.20_10. I use it to emulate one 3-phase PV Inverter from an 3-phase-AC-meter using a ESP32 and also for two temperature sensors. Worked very stable the last weeks.
Today I tried upgrading to VenusOS 3.20_33. After rerunning the install.sh script everything seems to run again. BUT: the despite the fact, that the different devices seem to be created correctly, there is no data available. this might be due to the fact that 3.20_33 switched now completley to FlashMQ. So the basic device creation seems to work, but there is no data available. I checked also with dbus-spy: only the name available, all other values are not visible.


Here are the topics for device creation in MQTT Explorer:

Here is a screenshot of the remote console:

Devices are visiblen, but empty...
So maybe I was a little bit naive thinking it might acutally work out of the box?!
Can I provide further information? Or is there a plan to update the driver because the FlashMQ-driver will be the future for VenusOS and it would be great, if the dbus-mqtt-devices would evolve to a dbus-flashMQ-devices :-)
Thanks a lot for the great work! I really appreciate it!
Christian

[freakent]

Thanks for providing the results of your research. Have you tried running the current release version of this driver (not the beta) on VenusOS 3.20_33?

Whilst VenusOS is being switched from Mosquitto to FlashMQ (both are MQTT brokers), I would avoid testing the Beta driver in a beta version of VenusOS, there are too many moving parts. The beta driver is very experimental at this time and not ready for production use.

[gurkc006]

Hi freakent,
yes I know, installing the latest beta of driver and OS was maybe a little be too much :-)
I tried installing 0.6.3 release, but had no luck with installing the YAML-dependency. I followed the steps from the installation instructions (ModuleNotFoundError: No module named 'dataclasses' -> opkg install python3-modules) but this doesn't seem to help. Always getting this dataclass error. Here you can find the console output AFTER installing the python3-modules and rerunning setup.sh: console.txt
So with this verion I don't get the driver to run properly. I had this behaviour earlier also with older VenusOS versions (that's the reason I switched to 0.7.x at all).
So for the moment I suppose switching back to VenusOS 3.20_10 and dbus-mqtt-devices 0.7.5_beta (that was working properly) and hoping, that the error will be fixed soon.
Thanks

[gurkc006]

Just for debugging purpose: VenusOS 3.20_33, dbus-mqtt-devices config-files branch head. Just restarted Venus and waited for the dbus-mqtt-devices logfiles (/var/log/dbus-mqtt-devices/current) to populate: creation.txt
For me, it looks not so bad for the first request (meter1). But no reaction on the second request (ds3g).
I tried to find some other entries in different logfiles, but nothing interesting...
Maybe that helps finding the problem? Is there anything I can test? I'm a little bit careful "playing" around with creating new devices using mqtt, as I still have some leftovers from such tests from a year ago, which are still visible in my VRM portal sidebar... But if necessary I would test of course. Thanks

[freakent]

I would prefer to figure out why python3-modules is not installing for you. What is the output from opkg install python3-modules? What does the output from opkg list-installed tell you?

[gurkc006]

Sorry for the delayed answer. Had some test this morning but had to get my system running again, as it's kind of productive... Here are some files from this evening (no solar at the moment, therefore I can test a little...)
I switched to VenusOS 3.20_33 and try using dbus-mqtt-devices 0.6.3 main branch from github. Symptoms are, that the driver seems to crash around every minute, probably when it receives new "device creation messages" via mqtt.
The logfile shows:
vOS_3.20.33_dbus-mqtt-devices0.6.3main.txt
Here are the outputs from list-installed:
vOS_3.20.33_list-installed.txt
and python -m pip list:
vOS_3.20.33_pip-list.txt
Don't know, if that helpes? There seem to be a YAML parse error somewhere.
I will switch back now to VenusOS 3.20_10 with running dbus-mqtt-devices 0.7.5_beta as this is running stable for me at the moment.
Good night

[ronstekk]

Hi everybody!

Maybe someone of you can help me with this issue related to Firmware 3.20:

The service is starting and running, devices can be initialized, but they don't get any messages from MQTT, I guess dbus-mqtt has been already replaced by dbus-flashmq in 3.20

Can someone confirm, that the current driver : dbus-mqtt-devices-0.6.3
does not work with Victron Firmware 3.20?

(I rolled back to 3.14 and everything works like a charm again)

Thx.
Ron

[gurkc006]

I tried the same with the new stable venus OS 3.20 and 0.7.5_beta. Same result: service starting and running, I see the devices in dbus-spy and in console, but no values. Same results as on 08.12.2023. Didn't try using 0.6.3... But maybe because both versions have problems with dbus-flashmq it's a more deeper problem.
Will switch back to 3.20~10 for the moment.
Thanks
Christian

[freakent]

Please can we put an end to this myth about there being a problem with FlashMQ. There is no problem caused by FlashMQ.

[freakent]

I tested 0.63 today with the latest VenusOS 3.20. I did find another unrelated issue that I have a fix for so will put out a new release tonight. Apart from this no issues caused by FlashMQ.

[gurkc006]

Hi freaknet,
sorry to blame this to FlashMQ. Maybe I did assume this because for me everytime I switch to a VenusOS with flashMQ the driver didn't work as before. But of course it might be a) a problem in my setup or b) another problem unrelated to FlashMQ.
I'm very thankful for you fantastic work and would like to help to debug my problems. So maybe I will wait for your new release and test this. If this also doesn't work maybe I open a new issu then with a detailed description of my setup and my problems. Maybe you can guide me then to better debug my problem.
Thanks again, Christian

[freakent]

@ronstekk,
Just to clarify, Victron have replaced the underlying mqtt broker from Mosquitto to FlashMQ. The original dbus-mqtt interface has been re-written as a FlashMQ plugin. However, the mqtt functionality remains pretty much the same. This driver works the same on FlashMQ with the dbus plugin interface as it did on mosquitto with dbus-mqtt interface.

I am sorry you had a problem with version 0.6.3 of the driver. In my testing of the offical 3.20 release today I managed to reproduce an issue originally reported by @garshaw. This has now been fixed and I am creating a new 0.6.4 release to include this fix. It is possible you were also affected by this. The issue was to do with the way the driver was registering devices on the dbus and had nothing to do with FlashMQ.

For now 0.7.x-beta is an unstable release and should not be used for live systems at all. With all the changes in VenusOS3.20 I decided to let the dust settle and verify this driver and the continue work on the beta against a stable release. I hope this clarifies situation for you.

[freakent]

I am closing this thread and will start a new discussion when I have a new Beta version.

[disaster123]

I had 0.7.5 running for a few month without any problems. Today i upgraded venus to 3.20 and 0.7.5-beta wasn't working anymore. So switched to 0.6.4 as suggested but 0.6.4 does not run and only gives the yaml error:

@4000000065cc97d9004c223c *** starting dbus-mqtt-devices ***
@4000000065cc97d92e21595c Traceback (most recent call last):
@4000000065cc97d92e2170cc   File "/data/drivers/dbus-mqtt-devices-0.6.4/dbus_mqtt_devices.py", line 34, in <module>
@4000000065cc97d92e21806c     from device_manager import MQTTDeviceManager
@4000000065cc97d92e218c24   File "/data/drivers/dbus-mqtt-devices-0.6.4/device_manager.py", line 17, in <module>
@4000000065cc97d92e219bc4     import yaml
@4000000065cc97d92e21a394 ModuleNotFoundError: No module named 'yaml'
@4000000065cc97da01f526ec *** starting dbus-mqtt-devices ***

I'm a little bit lost...

[freakent]

This Beta version has known issues and should NOT be used in a live system. Whenever you install a new version of the driver it is impoertant to run the setup.sh script as per the instructions.

[disaster123]

Running setup.sh again results in:

root@raspberrypi4:/data/drivers# ./dbus-mqtt-devices-0.6.4/bin/setup.sh
dbus-mqtt-devices: Setup in /data/drivers/dbus-mqtt-devices-0.6.4 started
dbus-mqtt-devices: Checking to see if Python's Pip is installed
pip 20.0.2 from /usr/lib/python3.8/site-packages/pip (python 3.8)
dbus-mqtt-devices: Checking to see if python3-misc library is installed
dbus-mqtt-devices: Pip install module dependencies
Requirement already satisfied: PyYAML~=6.0 in /data/home/root/.local/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (6.0)
dbus-mqtt-devices: Set up Victron module libraries
dbus-mqtt-devices: Set up device service to autorun on restart
dbus-mqtt-devices: Adding device service to /data/rc.local
dbus-mqtt-devices: Setup complete

[Miro-2000]

driver 0.6.04 with Venus OS 3.14
the GridPower Value is now with quotation marks => example => "78W"
PV-Power without quotation marks !!!

by the way .... 0.64 and 0.7.5 driver fails with Venus OS 3.20
PV-Meter and Grid-meter are without values only ---

[freakent]

This is most likely something to do with the services.yml file. Please check to see if the GridPower attribute has quotes around it.

[disaster123]

OK was able to fix this by adding:

sys.path.append('/data/home/root/.local/lib/python3.8/site-packages')

into device_manager.py

[freakent]

Which problem does this fix? If you have run the setup script no other modules should be needed.

[freakent]

This Beta program is now closed. Please use the generally available version 0.6.4.