How does MAVLink versioning work in v1, v2

[hamishwillee]

This is attempting to get clarification on versioning question I raised here: mavlink/mavlink#795 (comment)

Versioning in MAVLink is quite unclear. It is not clear:

1. What versioning mechanisms we have
2. What each versioning mechanism is supposed to achieve.
3. What are the shortcomings
4. How we might improve.

Ultimately the point of versioning is to ensure that systems are able to recognise and handle compatible messages - that they don't inadvertently attempt to handle messages they don't understand.

I think MAVLink handles the compatibility/version problem in the following way.

• All messages include a protocol-specific start byte. If a receiver doesn't understand the protocol (start byte) it will simply not see the message. This means old MAVLink 1 engines will ignore all v2 messages.
• MAVLink 2 messages with ids > 256 are not send when using v1 protocol.
• MAVLink 2 adds new fields to some MAVLink 1 messages (these are marked with the <extension> tag before new message fields).
  • These fields are not send (or received) in v1 messages.
  • These fields ignored if received by an implementation that doesn't understand them
  • These fields are zero filled when received by an implementation that understands them, but sent by a system that does not.
• All MAVLink messages include a CRC code, which captures the structure of the message definition (field names and order - presumably type?). If any of these things differ between sender and receiver then the CRC calculated for the message at each end will be different and the receiver will know the message is incompatible.
• MAVLink 2 also adds (in)compatibility flags, that contain flags about changes in the handling for the packet - for example, a different packet format that has been used. This can, for example, be used to prevent MAVLink 2 attempting to handle signing, if the implementation does not understand the signing format used.

So as above MAVLink 1 receivers will only handle MAVLink 1 messages. MAVLink 2 messages will get additional messages (id>256) and fields for low id messages. As far as I can tell, the only versioning check available is the CRC - you can tell if a message is compatible.

1. Is that correct - i.e. is "compatible version of message" the only check we have?
2. What if a message is incompatible - does the reciever (always/sometimes) send a response (what is it?) or is the message dropped?
3. How can the sender/receiver work out what version they each have to check compatibility.

The implication is that if you iterate a dialect you can add new messages, but you can't modify, remove or replace a message without breaking compatibility. If you break this compatibility you can't tell what version it happened in - all you can really do is make both ends update to the latest version of the dialect.

There are two other "versioning" artifacts in the message definitions.

      <mavlink>
      <version>3</version>
      <dialect>0</dialect>

4. What are these actually used for?
   • ie how are they used in the toolchain
   • what value should they have
   • when should the value iterate.
   • If a dialect includes a dialect (that has a version and dialect number), what does that mean?

[julianoes]

• All messages include a protocol-specific start byte. If a receiver doesn't understand the protocol (start byte) it will simply not see the message. This means old MAVLink 1 engines will ignore all v2 messages.

Correct.

• MAVLink 2 messages with ids > 256 are not send when using v1 protocol.

Yes, they can't be used, so I guess they are not sent?

[HW Edited] Yes, definitely not.

• MAVLink 2 adds new fields to some MAVLink 1 messages (these are marked with the <extension> tag before new message fields).
  • These fields are not send (or received) in v1 messages.

Correct.

• These fields ignored if received by an implementation that doesn't understand them

I think so, yes.

[HW Edited] Yes, according to existing docs from Lorenz

• These fields are zero filled when received by an implementation that understands them, but sent by a system that does not.

Presumably, not sure.

[HW Edited] Yes, according to existing docs from Lorenz

• All MAVLink messages include a CRC code, which captures the structure of the message definition (field order, field names, field type, array length for array fields). If any of these things differ between sender and receiver then the CRC calculated for the message at each end will be different and the receiver will know the message is incompatible.

Sounds right, I'm not 100% sure.

[HW Edited]
OK, so I am pretty sure this is correct based on the information and code sample in serialisation and the implementation: https://github.com/mavlink/pymavlink/blob/master/generator/mavparse.py#L385

BUT, the concern here is that this does not appear to calculate CRC on extension fields: ArduPilot/pymavlink#196

• MAVLink 2 also adds (in)compatibility flags, that contain flags about changes in the handling for the packet - for example, a different packet format that has been used. This can, for example, be used to prevent MAVLink 2 attempting to handle signing, if the implementation does not understand the signing format used.
  If an incompatible flag is encountered the packet is dropped.

I have never understood these flags, and I don't think they are implemented or tested.

[HW Edited] This is correct. They may not be tested, but they are implemented in the C library - i.e. there is a test against the incompatible flags supported by the parser. At this time the only incompatible flag is support for signing: MAVLINK_IFLAG_MASK

FYI, these are documented in MAVLink 2 > Packet Compatibility Flags. In essence they allow you to specify that the frame has to be handled specially - the implementation could then encode/decode a different frame format.

So as above MAVLink 1 receivers will only handle MAVLink 1 messages. MAVLink 2 messages will get additional messages (id>256) and fields for low id messages. As far as I can tell, the only versioning check available is the CRC_EXTRA - you can tell if a message is compatible.

1. Is that correct - i.e. is "compatible version of message" the only check we have?

Not sure.

[HW Edited] Fairly sure this is true for message definitions.

2. What if a message is incompatible - does the reciever (always/sometimes) send a response (what is it?) or is the message dropped?

I think it is silently dropped.

[HW Edited] I think on further inspection that the helper decoder function mavlink_frame_char_buffer() reports a status of MAVLINK_FRAMING_BAD_CRC. There is no requirement to send anything back to the sender.

3. How can the sender/receiver work out what version they each have to check compatibility.

There is also the [https://mavlink.io/en/messages/common.html#PROTOCOL_VERSION) check that would presumably resolve some of the problems listed here if it was implemented everywhere.

[HW Edited] THanks. IMO probably not because

• there is a lot of change that is untracked between version numbers
• git hashes are useful for telling that something is identical, but AFAIK they are not ordered, so you can tell you're not identical, but not whether you're compatible.

This would only be of some use if it included current/min/max version compatibility for the spec AND the spec version was iterated every time it was updated in any way.

Even so this would just tell you that the mavlink version could process the messages, not whether the system would handle them.

The implication is that if you iterate a dialect you can add new messages, but you can't modify, remove or replace a message without breaking compatibility. If you break this compatibility you can't tell what version it happened in - all you can really do is make both ends update to the latest version of the dialect.

Right, but dialects are a whole other ballpark different from just the versioning problems.

There are two other "versioning" artifacts in the message definitions.

      <mavlink>
      <version>3</version>
      <dialect>0</dialect>

I think this version tag refers to the xml definition version but not the wire protocol version. So, this number only matters for code generators that parse the xml. (It's always been 3 even though things like </extensions> have been added.

[HW Edited] THanks. I think you mean the other way round - it is the wire protocol version, not the XML version. That makes sense but is very "not useful".

I'm not sure if the dialect tag is ever used, or how.

[HW Edited] I don't think either of them are used. Looking at mavparse.py the version is collected but not used, and the dialect is completely ignored.

[julianoes]

[HW Edited] THanks. I think you mean the other way round - it is the wire protocol version, not the XML version. That makes sense but is very "not useful".

No, I meant it as I said it. The wire protocol is v1.0 and v2.0. The xml version was always 3.

[HW Edited] I don't think either of them are used. Looking at mavparse.py the version is collected but not used, and the dialect is completely ignored.

Sad, too much stuff that isn't used.