Clarify the definition of "navigation scope", "applied", and off-scope theming

[mgiuca]

Closes #755.

This change (choose one):

• Breaks existing normative behavior (please add label "breaking")
• Adds new normative requirements
• Adds new normative recommendations or optional items
• Makes only editorial changes (only changes informative sections, or
  changes normative sections without changing behavior)
• Is a "chore" (metadata, formatting, fixing warnings, etc).

Commit message:

This further clarifies that a manifest is allowed to apply to URLs that
are not within its scope (that this was not clear was a left-over
mistake from #701), and that a document within scope is allowed to
override the theme color. Adds a new recommendation that an out of scope
document not be allowed to override the theme color.

This introduces new normative recommendations, only insofar as they were
already commonly understood, to my knowledge.

Navigation scope: No longer defined in terms of the URLs that a manifest
is allowed to apply to (since a manifest may apply to any URL). Rather,
it's just a set of URLs; the other normative requirements around
navigation scope define what it means.

Applied: Clarified that it is the user agent's discretion whether or not
to apply a manifest when a top-level browsing context is created, with
examples of when you would and would not apply it (previously, there was
simply no text around when a manifest should be applied).

Theme color: Fixes a "MAY" requirement imposed on the document
(requirements can only be imposed on user agents). Clarifies this
recommendation as only applying to in-scope documents, and adds a
recommendation against applying document theme for out-of-scope
documents.

---

Preview | Diff

[mgiuca]

(why is there no way for me to reply to this comment inline?)

... or could even apply a user configured one...

Not sure what you're suggesting here or where this fits into the sentence.

Edit: Oh, I see, you're replying to my comment. I saw this comment only as a reply to the code change:

GitHub...

[benfrancis]

I proposed that since it's been 2 years, we collapse this text to not mention the history of it, just say that user agents are not allowed to block or transfer the navigation to another TLBC.

Are you sure you want to forbid user agents from transferring any navigation to another top level browsing context? Is this not a product decision?

[benfrancis]

I thought it might be useful to provide some examples of implementations I've worked on outside of Android and iOS, in order to not constrain the thinking to contemporary mobile operating systems and be more open minded about how this specification might be used in the future.

Firefox OS 2.6

In Firefox OS 2.6 the web browser was merged with the core system UI such that users could seamlessly transition between "pinned" web apps, and the rest of the web. This is in contrast with other mobile operating systems in which there are much firmer boundaries between installed applications.

When browsing the web, web applications could be "pinned" (installed) using their web app manifest. Whenever the user navigated to a URL within the scope of a pinned app, the manifest metadata was applied to the browsing context and whenever they navigated away from the scope of the app, the manifest would be un-applied.

When navigating to a URL within the scope of a pinned app with a standalone display mode, the URL bar would shrink down into the status bar and turn into a title bar to look more like an app. When navigating out of the scope of the pinned app, the URL bar would expand out into a full URL bar to look more like a browser. (The user could always access the full URL bar by tapping to expand it).

There's a video here which shows part of that behaviour.

Krellian Kiosk

I'm currently working on an implementation of a web application runtime where a single web application may be installed to an interactive kiosk for use by the public. In this use case, the administrator wants to ensure that only the installed web application can be navigated to, and any attempted navigations outside of the scope of the web application are blocked. I believe Chrome OS has a kiosk mode which does something similar.

Another related use case is in deploying a single web application to a digital signage display, which has completely passive content and no user interaction on the display itself. Web app installation and navigation are controlled over the network from a separate controller application.

The reason I'm providing these examples is to try not to constrain future innovation in user agents by simply describing the behaviour of existing smartphone web browsers in the specification. I suggest it would better to allow some flexibility in the specification for user agents to decide when to apply a manifest and how navigations outside of the manifest's scope should be handled.

How about defining scope as "the set of URLs to which the manifest applies." This could be qualified by saying that it doesn't mean that user agents can't also apply some aspects of the manifest's metadata to other URLs if they are navigated to from an application context, but that the metadata is primarily intended to apply to that set of URLs, not the whole web.

[mgiuca]

@benfrancis it seems like your practical objection to this change comes from a desire to have additional restricted modes (such as kiosk modes) that do restrict PWAs to their scope or a restrictive set of URLs.

I would suggest that such a mode falls outside of the manifest spec, and into the realm of (legitimate) user agent interventions, by way of offering their users (administrators) additional control over what URLs can be browsed to. It's very similar to a parental control feature that a web browser may offer, to let parents restrict the set of browsable URLs to a specific allow-list. You don't need to the permission of the HTML spec to prevent navigation to a URL that's been banned by the parent, it's essentially a deviation from the spec requested by the user.

I would argue it makes sense to keep these two things (the scope of a PWA, and a kiosk-like prevention of navigating outside the PWA) separate. There's no reason kiosk mode needs to necessarily be limited to the scope of the PWA. For example, you might let the administrator configure an allow-list, so that you can navigate to certain other origins within the PWA, but not the whole internet.

Are you sure you want to forbid user agents from transferring any navigation to another top level browsing context? Is this not a product decision?

It isn't a product decision, because it violates the basic assumption that sites make about how HTML navigation works. (As we discovered two years ago in #646.) This isn't obvious, to me anyway, it took months of analysis, discussion, and experimentation (as you can see by reading over #646, the corresponding Chrome bug https://crbug.com/861618, and the corresponding manifest PR #701), to come to the realisation that this wasn't a mere product decision; allowing UAs to open target=_self links in a new top-level browsing context was breaking websites because it violated the normal navigation flow of HTML. So, we changed the Manifest spec to explicitly forbid UAs from doing that. This PR was not intended to re-open the debate; it's just fixing up a piece of text mistakenly left over from #701 .

As I said above, UAs are free to make a product decision to restrict the set of navigable URLs, at the request of the user, but this should be opted in by the user or administrator (e.g., by using a special kiosk mode). The normal operating mode of a compliant browser should not break websites.

How about defining scope as "the set of URLs to which the manifest applies." This could be qualified by saying that it doesn't mean that user agents can't also apply some aspects of the manifest's metadata to other URLs if they are navigated to from an application context, but that the metadata is primarily intended to apply to that set of URLs, not the whole web.

That seems to be more or less what the spec (erroneously) says today: scope is "the set of URLs to which the manifest applies", but elsewhere in the spec it says that the manifest can apply to any URL, not just one in the navigation scope. Those are contradictory statements. I'm looking for clear normative text here that can be implemented in a web browser. "primarily intended to apply" is not a clear statement.

I think the change suggested in this PR is entirely compatible with your kiosk feature.

[benfrancis]

@benfrancis it seems like your practical objection to this change comes from a desire to have additional restricted modes (such as kiosk modes) that do restrict PWAs to their scope or a restrictive set of URLs.

Actually my main concern is that the specification does not prevent future user agents from coming up with better solutions to the off-scope navigation problem than the current in-app browser trend. The examples I gave were just to illustrate that the way Android and iOS currently work are not the only ways user agents can work.

I would suggest that such a mode falls outside of the manifest spec, and into the realm of (legitimate) user agent interventions, by way of offering their users (administrators) additional control over what URLs can be browsed to.

As long as the specification makes clear that such user agent interventions are in fact legitimate, then I agree that's fine.

It isn't a product decision, because it violates the basic assumption that sites make about how HTML navigation works. (As we discovered two years ago in #646.)

Thank you for the additional context, I hadn't read that thread before. Having read through all of the rationale I can see why current browsers have converged on option b1 from #646 (comment). In testing the latest version of Firefox Preview (Fenix) it appears that is the latest behaviour there too.

However, now that I better understand the rationale I have another concern. Does this change not break the concept of deep linking as described in section 3.2 and the Abstract section?

If all off-scope navigations must stay inside the current application context (but with added chrome), it is no longer possible to deep link between installed web applications. At least not within the constraints of the Android and iOS application frameworks.

The Firefox OS 2.6 example I gave above actually most closely resembled option b1, but deep linking was possible because it essentially removed the distinction between an app window and browser window. It was therefore possible to seamlessly transition between the two states. A single browsing context could transition between having different manifests applied and the same window could effectively transition from one app to another. As far as I know that is not possible on Android or iOS.

Would you agree that with this change, deep linking could only ever happen if the source of a navigation explicitly requests a new window via target=_blank or window.open()? And even then some of the same technical constraints may cause problems. That seems like a real shame.

In the interests of getting to Candidate Recommendation I don't particularly object to changing the definition of scope to match what current browsers do, but I do want to highlight the potential implications for future user agents which may not be operating within the same constraints.

[mgiuca]

However, now that I better understand the rationale I have another concern. Does this change not break the concept of deep linking as described in section 3.2 and the Abstract section?

If all off-scope navigations must stay inside the current application context (but with added chrome), it is no longer possible to deep link between installed web applications. At least not within the constraints of the Android and iOS application frameworks.

The Firefox OS 2.6 example I gave above actually most closely resembled option b1, but deep linking was possible because it essentially removed the distinction between an app window and browser window. It was therefore possible to seamlessly transition between the two states. A single browsing context could transition between having different manifests applied and the same window could effectively transition from one app to another. As far as I know that is not possible on Android or iOS.

I think I understand that concern. There's some degree of incompatibility between the following two designs:

1. Navigating from an in-scope URL in an app context to an out-of-scope URL remains in the app context, but with a navigation bar UI. [That's the behaviour I'm trying to capture.]
2. Navigating from a normal browsing context to an in-scope URL transitions into an app context. [That's what you're talking about above as deep links.]

The sort-of-incompatibility comes from the inconsistency that when moving outwards, the app context is fixed, but when moving inwards, the app context changes. To your point, when moving from an in-scope URL in an app context to a URL in another app's scope, there's direct conflict between the above two.

The way this resolves in my mind is that a browsing context never changes its application context. Number 2 above isn't true. If you ever navigate to a URL inside another app's context that is explicitly captured by that app (see #764), then it should not simply transition the current browsing context into an application context, it should create a new top-level browsing context in that new app context. That's how native apps typically work; you don't somehow "transform" one app window or activity into that of another application. You construct a new activity in that new application.

I think the whole model is simpler, and consistent, if we don't allow a top-level browsing context to change which manifest is applied to it (if any) once it has been created.

[benfrancis]

Hi @mgiuca, thank you for responding.

The way this resolves in my mind is that a browsing context never changes its application context. Number 2 above isn't true. If you ever navigate to a URL inside another app's context that is explicitly captured by that app (see #764), then it should not simply transition the current browsing context into an application context, it should create a new top-level browsing context in that new app context.

OK, that makes sense. It was not clear to me from the specification and proposed text that app-to-app deep linking would be possible though. Do you think this could be made more explicit?

The discussion in #764 looks like a promising potential way forward for deep linking into installed web applications.

That's how native apps typically work; you don't somehow "transform" one app window or activity into that of another application. You construct a new activity in that new application.

I agree that is how Android and iOS work. But I would argue this is the result of trying to shoehorn the web into the native app model of existing mobile operating systems, not necessarily the optimal design for web applications which would work to the strengths of the web.

I think we may have to agree to disagree that the optimal behaviour is for every installed web application to act as a mini browser that can be navigated to any URL on the web, and that by default a link only ever goes from browser to app or app to app, and never app to browser. But I concede that in order to move towards a Candidate Recommendation, describing the current behaviour of implementations is the most pragmatic and low-friction way forward.

[mgiuca]

Hey @benfrancis

OK, that makes sense. It was not clear to me from the specification and proposed text that app-to-app deep linking would be possible though. Do you think this could be made more explicit?

Sure. Maybe we can tackle it as part of the #764 work. (Or, if you feel like it, feel free to put up a PR suggesting stronger wording for deep linking. FYI that section is very old and I don't think it's been touched since I've been an editor on the Manifest.)

I agree that is how Android and iOS work. But I would argue this is the result of trying to shoehorn the web into the native app model of existing mobile operating systems, not necessarily the optimal design for web applications which would work to the strengths of the web.

I think we may have to agree to disagree that the optimal behaviour is for every installed web application to act as a mini browser that can be navigated to any URL on the web, and that by default a link only ever goes from browser to app or app to app, and never app to browser. But I concede that in order to move towards a Candidate Recommendation, describing the current behaviour of implementations is the most pragmatic and low-friction way forward.

OK. As I've said above, at a personal level, I agree with you wholeheartedly. I do not like it when apps keep you inside their frame when you're off browsing the web, and I turn it off for every app that I can. There's a reason we have full-featured web browser apps! However, I must note the reality that the web platform is in competition to native apps platforms, and that gives developers a certain amount of sway over us... they overwhelmingly want to keep users inside their app frames by default (so that the user can get back to their app with one tap), and so it behooves us as platform designers to accomodate that, in the way that best respects the user (i.e., giving the user controls to grab the URL and re-open the page in a web browser, for instance). If we don't do that, we'll find sites building their own browser UI in an iframe instead, which makes it much harder for us to intervene on the part of the user. So that's why I'm cautiously supportive of this "navigation off scope from within the app" behaviour. (And there are other, much more legitimate use cases, such as the authentication-on-another-origin case.)