Define goals for the style expression language

[ZeLonewolf]

I recommend that maplibre-gl includes a complete and robust expression API that enables style designers to manipulate tile attributes programmatically, in accordance with the original language vision. This feature would give style designers the flexibility to determine which processing should occur on the tile generation side and which should take place in the user's web browser. A powerful expression language would be a significant capability differentiator that would influence a style implementer to select maplibre-gl over mapbox-gl.

A strong expression language is vital for the sustainable growth of vector tile-based maps, as it enables new use cases. For instance, a community organization can host a vector tile server with a centralized schema, with OpenMapTiles being a leading candidate. Hosting a vector server has been a long-standing vision within the OpenStreetMap project. More recently, the Operations Working Group of the OpenStreetMap Foundation has been investigating the hosting of a vector tile server (openstreetmap/operations#565). The aim is to facilitate community-driven style development providing a tile server with a standard schema that anyone can use. If maplibre-gl incorporates a complete and robust expression language, it will empower such centrally-hosted tile servers to support multiple styles with different requirements without needing to customize the delivered tiles.

Maintainability is a concern when expanding the style specification. However, testing individual style expressions is straightforward since they usually map to single functions in the implementing language. Thus, expressions are easier to maintain compared to other parts of the codebase.

A robust expression language has been a priority for mapbox-gl well before the maplibre fork. Unless there is a compelling reason to change course, we should follow the consensus established by the community when mapbox-gl was still an open project. The following tickets provide evidence of the pre-fork community support for a robust expression language, as well as the ongoing customer demand for it:

• Master ticket tracking expression API completeness mapbox/mapbox-gl-js#6484
• Many NSPredicate and NSExpression operators have no equivalent in style JSON mapbox/mapbox-gl-native#11786
• Add some missing math expressions mapbox/mapbox-gl-js#5853
• Allow expressions as elements in array output values mapbox/mapbox-gl-js#6155
• Add color modifying expressions mapbox/mapbox-gl-js#5676
• Add find-and-replace expression operator mapbox/mapbox-gl-js#4100
• Distance expression operator mapbox/mapbox-gl-js#6062

While we should have a constructive debate about including individual expressions, it's crucial that maplibre-gl's broadly-stated strategic goal is to enable expression logic on the client side. This will provide style developers with a wider range of options when deciding where to implement different aspects of cartographic decision-making in their technology stack. Let's engage in a healthy discussion on this matter.

[HarelM]

I know the following might not be a favorable stand, but I think it's worth voicing it anyway. Feel free to 👎...
This is my personal and professional opinion and does not represent neither the board or maplibre-gl-js project.

While I understand the requirements and the appeal, I generally don't agree with this direction.
I don't think you can generate tiles that will work for everyone, they will either be too big or too complicated, or both which means longer load time and complexity when trying to style them.
When one of the main repeating concerns is the performance, either the time it takes to load the map, the time it takes to load the data or the fps I don't think expanding the language to facilitate more ways to get it wrong is the solution.
I think tile generation tools that allow selecting the right data is where we should look for the solution and not by extending the language.
Another concern regarding "creating a complete and robust API" is that I don't think this project should focus on creating yet another coding language (based on a json schema) that almost no one writes in, doesn't have good documentation and doesn't have good examples. that a great recipe for getting it wrong.
While documentation can be improved, and examples can be created, I think it derails the project from the actual goal of creating beautiful maps that are simple to build.
While there are cases that should be considered, I think they should be picked with cautious.
And obviously what you said about maintainance.

[1ec5]

It’s fair to expect a new issue to summarize the prior art and reflect more recent developments. But for what it’s worth, I just hope the maintainers aren’t actively avoiding looking at historical discussions that don’t have any licensing concerns. That wouldn’t be sustainable, since the code isn’t documented well enough and there was never a practice around committing design documents to the repository.

[HarelM]

In order to know that something doesn't have a licensing concerns you need to read it or know when was the last comment and that it was before the license change etc. This is problematic as you can imagine.
I'm trying to avoid digging into mapbox issues to keep the ability to be impartial, sorry to crash your hopes... :-/
In most cases, you can summarize the issue well enough in the new repo, it is the maintainers' role to help you find the right place in the code to solve your issue, while it may take some time to get there, this is how I see things, we should not relay on mapbox repos (at all).

[1ec5]

Just to reiterate, the mapbox-gl-native repository is still completely BSD-licensed. If there’s a problem with looking at the issues there, then there’s also a problem with forking that repository in the first place.

[HarelM]

Yup.

[1ec5]

It’s your prerogative to decide which repositories you’re willing to look at, of course. It isn’t for me to say what level of risk you should be willing to accept. But if a maintainer intends to avoid relevant discussion and places the onus on a contributor to characterize that discussion, then the contributor will expect an assumption of good faith, and the maintainers should not be surprised if the contributor gets frustrated when they are met with skepticism.

Anyways, this is pretty off-topic for the style specification. 😄

[HarelM]

As for defining goals, which I completely ignored in my last comment (and only said what not to do, which is not a great way to define goals), I think the main goals should be:

• pairity between the platforms
• extend the existing style definitions to be data driven as much as possible
• Facilitate for a single place to validate the style for any coding language
• Improve documentation and examples

These might be a more short-term goals, but I believe we should focus on those before trying to look ahead.

Edit:

• Ability to extend the expression or style spec with hooks or plugins

[1ec5]

I agree that improving platform parity and implementing data-driven styling for more properties would significantly improve the developer experience. The fewer gotchas, the better. But these are somewhat open-ended goals – should they block additions to the style specification indefinitely? Could the style specification accept new additions under the condition that they don’t make the situation any worse? For example, a new string-manipulating expression operator inherently supports data-driven styling and would be relatively easy to implement in GL JS and gl-native simultaneously.

[HarelM]

As written in my previous comment, I don't think there's a general rule we can apply here, I think every addition should be looked carefully and discussed.

[wipfli]

I'm quite happy with the expressions in general, but I would find it useful to make more things data-driven. I think there are some layout properties that are not data-driven yet like line cap and line dash array...

[1ec5]

Even for those of us who are pretty content with the language overall, an explicitly stated scope for the language would still be useful, even if it’s just “complete what Mapbox started”. That would give contributors the confidence to contribute features or fixes without unexpectedly wading into an existential debate.

[1ec5]

Maybe some criteria for new operators could be something like:

• The operator would enable a clearly articulated use case that’s practical (not contrived) and presentational (not a hack to work around a flaw in the tiles). We can debate what counts as contrived or hacky. 🙂
• It isn’t straightforward to implement the operator using the available operators. As I mentioned in Add some missing math expressions mapbox/mapbox-gl-js#5853 (comment), some of the obvious missing math operators are only needed for convenience. For example, you can already write your own ceiling operator based on available operators, but not everyone would get it right.
• The operator would have an intuitive translation in NSExpression on iOS/macOS or the expression DSL on Android. So for example, anything touching regular expressions would need additional scrutiny because of the different regex dialects that developers expect in each platform’s languages.

[wipfli]

Good points

[wipfli]

Another goal I have is to not break the style spec even if it would make it better.

For example I think we should keep support for the {} things which are called function I belief.

[ZeLonewolf]

The issue of breaking versus non-breaking changes should be satisfied by semantic versioning. I'm not aware of any proposed feature that's a breaking change on the table anyhow. Should one be identified, it can get added to a list of "breaking changes for the next major version increase." But I do agree that breaking changes should be strenuously avoided unless absolutely necessary.

[wipfli]

I think we should not introduce breaking changes in the style spec. Maybe this is too restrictive, I don't know, but I just like the idea a lot that the style spec does not break...

[ZeLonewolf]

It's an entirely philosophical discussion until someone brings a breaking change to the table that we can't live without. Until that day, not breaking the spec is clearly the right answer.

[1ec5]

For example I think we should keep support for the {} things which are called function I belief.

These are two separate legacy features that have both been kept around for backwards compatibility:

• Style functions are a more basic syntax for interpolation. Under the hood, some expressions still get compiled down into style functions as an optimization. This is the reason you can’t nest an interpolate expression inside a case or arithmetic expression and other very annoying limitations. It would be possible to remove these limitations without breaking the style specification, by implementing expressions more generically under the hood while preserving functions as syntactic sugar.
• As a special case, text-field and icon-image accept tokens in curly braces that refer to feature properties. Under the hood, these tokens get converted to get expressions: [core] Convert token strings to expressions mapbox/mapbox-gl-native#12402. The NSExpression support on iOS and the expression DSL on Android have never supported this syntax due to the ambiguity it creates. As far as I can tell, tokens are no longer documented anywhere explicitly, but they’ve persisted in GL JS examples and Stack Overflow answers that developers tend to copy.

The issue of breaking versus non-breaking changes should be satisfied by semantic versioning. I'm not aware of any proposed feature that's a breaking change on the table anyhow. Should one be identified, it can get added to a list of "breaking changes for the next major version increase." But I do agree that breaking changes should be strenuously avoided unless absolutely necessary.

Every stylesheet is required to declare a style specification version in its root, in case a new version is ever released. The last release was version 8, around the time of Mapbox iOS SDK v1.0. The first implementation of data-driven styling nearly forced a v9: mapbox/mapbox-gl-js#1406. Instead, it was implemented in a backwards-compatible way. The GL JS team decided to freeze the style specification on v8, limiting any changes to backwards-compatible changes, though they held out some hope for a future omnibus change in v9, with all the pent-up changes at once to minimize disruption: mapbox/mapbox-gl-js#6584.

[wipfli]

Other question: should we allow something like addProtocol for the expressions? I.e. a mechanism for people to inject custom logic?

[ZeLonewolf]

There are essentially three options:

1. Add sufficient expression logic such that data can be manipulated with the same flexibility that's possible with a general purpose programming language
2. Allow style developers to inject their own logic via some sort of plugin system
3. Accept that users will create their own forks to satisfy their needs for client-side expressions

[1ec5]

Cross-posting from #36 (reply in thread):

I see that maplibre/maplibre-gl-js#1295 is revisiting the decision in mapbox/mapbox-gl-native#7860. From a developer perspective, it would be really nice to be able to pass in a lambda instead of defining a declarative expression. Once Objective-C added the ability to make an NSExpression out of a block (lambda), no one ever looked back and used the traditional NSExpression operators. But if performance is the concern, then I don’t know how giving developers full, unoptimizable control would be the solution.

Tradeoffs include:

• The stylesheet is no longer self-contained when using this feature. Supporting multiple platforms requires writing code for each platform, just like in runtime styling.
• In Objective-C and Java, it’s way too tempting and easy for developers to capture state into the lambda, leading to a memory leak or concurrency issue: Data-driven styling should support arbitrary functions mapbox/mapbox-gl-native#7860 (comment). This may not be as much of an issue for GL JS or the iOS SDK when writing in Swift (where escaping closures must be annotated).
• The renderer must treat the lambda as a black box. There is no opportunity for optimization.

[HarelM]

I generally think this has its merits.
The options that @ZeLonewolf are generally correct, my 2 cents on them:
1 - I don't think the project should go into that direction and I don't think we'll manage to actually cover everything, for example the last thing I want to introduce is some for-loops inside the style.
3 - This means we are not hearing our user needs, while in some cases it would be better to create a fork, most of us don't really want to maintain a fork just for something silly, so in general this should be avoided as well
2 - Considering the above two, I don't think there's another option besides this one... yes, it might break the style, yes, it might not be performance optimized, but it can and should facilitate for proof of concepts and we can always decide which of the implementation we decide to bring into the code library, when we have a good extension code it should be fairly easy to decide what to adopt...

[1ec5]

for example the last thing I want to introduce is some for-loops inside the style.

I agree. At one point, there was a proposal to extend the expression language to be a general-purpose programming language, maybe even Turing-complete: mapbox/mapbox-gl-js#7010. But that would’ve been way too esoteric, too much indirection. Instead, we could satisfy most of the desire for loops using functional operators on arrays (filter, map), along with existing operators like concat and slice.

Functional-programming-style operators would even address much of the desire for injecting JavaScript functions (the “protocol” idea above), which sounds infeasible to me because paint expressions get compiled down to shader code. If you want to run arbitrary JavaScript on every frame, you might as well write a custom OpenGL/WebGL layer or less performant runtime styling code hooked up to a per-frame event – both approaches are already supported.

[wipfli]

Somebody told me once that they do not use any expressions in native at all. Instead the implement things with C++ directly. That seemed to be more performant than expressions. I have to ask them again. Just wanted to note here that apparently one also do things in JS and C++ which are done with expressions, I guess at the expense of having to write things in different languages...

[1ec5]

Of course, almost all of the Android, iOS, and macOS SDKs are redundant to gl-native and less performant. Good luck getting a typical Swift or Kotlin developer to write C++17. 😉

[louwers]

@wipfli Interesting...

[ovivoda]

I agree with @1ec5 on this, that we will need to have a OS specific way of doing this, and not expect that one will go deeply into C++ and implement a way of doing things. They would rather find another solution that fits their needs that has an specific OS level SDK/API

[ovivoda]

Hi

I just want to share my view on this, mostly an eagle eye view on proposals, generally speaking:

When we start with a proposal, a project proposal, I think we should state what the end goal is. Initially in the form of a simple statement then detail it until we refine it enough and agree on what we want to achieve.
Then, based on the above, we can find/research ways of achieving the goals in the proposal and move forward to break it in small chunks and just in the end implement it.

That being said, I think that, the discussions here are too much focused on what to change, how to achieve, before we actually identified/agreed on the clear/tangible goals of the proposal.

One thing that will help is, that all of us, in the discussion, are posting their view of the tangible clear goals.