Can we reduce or eliminate the swift_deps_index.json?

[cgrindel]

Some members of the community have expressed a desire to reduce or eliminate the index JSON file that is currently checked into repositories that use rules_swift_package_manager. The following is an overview of the information that is currently stored in the file and an explanation of why it was created.

Structure

1. direct_dep_identities: This is a list of the Swift packages that are direct dependencies of the client's workspace.
2. modules: This is a list of all the Swift modules declared by the direct and transitive dependencies of the client's workspace. The important pieces of information are the name, the type, and the Bazel target label.
3. products: This is a list of all the Swift products declared by the direct and transitive dependencies of the client's workspace. The important pieces of information are the name, the type and the Swift module targets that are required to depend on the product.
4. packages: This is a list of all of the Swift packages that are required by the direct and transitive dependencies of the client's workspace. This tells rules_swift_package_manager how to construct the swift_package declarations.

How is it generated?

It is generated by running the //:swift_update_pkgs target. Behind the scenes, this is a call to
Gazelle's update-repos command. This command fully resolves the transitive dependencies using SPM
commands (e.g. swift package resolve). This, in turn, downloads all of the transitive dependencies
resulting in the annoying .build directory in the workspace.

How is the index JSON file used?

The index JSON file is used to:

1. Construct the repository rule declarations (e.g. swift_package) generated by the bzlmod logic.
2. Generate Bazel build files in the repository rules (e.g. swift_package).
3. Generate Bazel build files in the client's workspace via the Gazelle plugin.

Why do we need an index file when Go and other languages do not need one?

This has to do with how imports work in different languages. In short, Swift imports are not unique
and they do not map one-to-one to a Bazel target.

Background

Let's look at Go imports. They are unique. Outside of the standard packages, the imports are string
values derived from a package URL (e.g. github.com/stretchr/testify/assert). It is easy to
differentiate between two different but similarly named packages (e.g.
github.com/stretchr/testify/assert vs gotest.tools/v3/assert). Also, Go packages conveniently
map to a single Bazel target via go_library declarations. Hence, the Go Gazelle plugin can read
the import string value and know exactly which Bazel target should be associated.

Unfortunately, Swift is not nearly as convenient. First, Swift imports are non-unique string values
(e.g. Logging). The value alone does not tell us which Swift package provides the module. To fully
resolve to the correct module, we need information from the package manifest. Second and more
insidiously, a Swift module import can result in one or more Bazel targets needing to be depended
upon to compile properly. This is mostly an issue with Swift products that include multiple Swift
targets. Notice that the product declaration can accept multiple Swift targets.

Construct the repository rule declarations (e.g. swift_package) generated by the bzlmod logic.

To generate the swift_package declarations, we need the remote and commit information for all of
the Swift packages (direct and transitive) and we need information to resolve Swift target
dependencies (see next section). The remote and commit information exists in the Package.resolved.
Unfortunately, the information to properly resolve the Swift target dependencies is not present in
the Package.resolved.

Generate Bazel build files in the repository rules (e.g. swift_package).

In SPM, Swift targets can depend on Swift targets or Swift products. As mentioned previously, Swift
products can consist of one or more Swift targets. So, Swift products can map to one or more Bazel
targets (Swift target = Bazel target). There is no construct in Bazel that represents a group of
targets. So, to properly resolve a dependency on a Swift product, we need a mapping of products to
Bazel targets.

NOTE: Without fully resolving the transitive dependencies for a package, there is no way to know
what targets are associated with a Swift product.

Generate Bazel build files in the client's workspace via the Gazelle plugin.

To properly map Swift import statements to Bazel targets, we need to know

1. The packages that the client project directly depends upon. (We do not want to resolve to
   similarly named modules in the transitive dependencies.)
2. The Swift products that are provided by each Swift package.
3. The modules/Swift targets that are provided by each Swift product.

Short of parsing and evaluating the Package.swift, there is no way to know which of the Swift
packages are direct dependencies for the client workspace. Hence, we store this in the Swift index
JSON file.

Could we eliminate the Swift index file?

Potentially. However, there are some trade-offs.

Construct the repository rule declarations (e.g. swift_package) generated by the bzlmod logic.

The information for downloading the Swift packages is in the Package.resolved. If we are getting
rid of the Swift index file, then there is no need to pass that along. So, we should be all set.

Generate Bazel build files in the repository rules (e.g. swift_package).

Repository rules execute independently. There is no way to tell Bazel that one repository rule
depends on another repository rule. So, to generate build files, the repository rules must have all
of the information that they need or be able to derive it. If we do not provide the Swift index
file, each Swift package must fully resolve their own transitive dependencies.

As was mentioned previously, this is done using SPM commands. So, each Swift package would need to
do the following:

1. Repository rule downloads the Swift package.
2. Repository rule executes swift package update. This downloads the transitive dependencies for
   the Swift package.
3. Generate an index mapping Swift modules to Bazel targets.
4. Generate the Bazel build file.

Let's look at an example project that depends on two Swift packages: A and B. Package B
depends on two other Swift packages: C and D.

       Client workspace
             /\
            /  \
           A    B
                 \
                  C
                   \
                    D

To generate build files for A, B, C, and D, we would need to

• Download A once (repository rule).
• Download B once (repository rule).
• Download C two times (repository rule, B resolution)
• Download D three times (repository rule, B resolution, C resolution)
• Generate five Swift module to Bazel target maps (client workspace, one for each Swift package).

In this example with four Swift packages, we would have performed seven downloads and generated five
maps. As you can see, deeper dependency trees result in costlier, duplicative work.

Generate Bazel build files in the client's workspace via the Gazelle plugin.

To generate build files for the client, we need to know the Swift packages that the client depends
upon and we need to generate a map of the Swift modules to Bazel targets. The list of direct
dependencies can be derived by running swift package dump on the client workspace and the Swift
packages could provide a file that maps the Swift modules/targets to Bazel targets.

Conclusion

Ultimately, I opted to include the Swift index JSON file to solve a couple of needs. However, the
primary win is that we cache information about the dependencies. This avoids extra work when
performing builds and reduces some complexity when debugging resolution issues.

[AttilaTheFun]

Thanks for writing this up @cgrindel !

To be clear, my goal is to remove the swift_deps_index.json from the repository, but I don't mind if an index like this exists inside of the repository rule(s). Also hopefully we could find a way to keep the .build directory out of the repository as well. In my ideal world only Package.swift and Package.resolved would be checked into the repository (with the latter being an optional lock file, similar to the optional SHAs for the http_file / http_archive rules.)

I also understand the complexities around swift package resolution and the product / target dilemma make things more difficult.
That said, I have a couple of ideas of how we could tackle this issue.

Firstly, while swift package dump-package and Package.resolved only contain the urls and revisions from the direct package dependencies, swift package update does recursively resolve and fetch the entire package graph. If you look at the contents of the .build directory after running it, you will see:

artifacts/
checkouts/
repositories/
workspace-state.json

I believe this JSON file contains most of what we need. Its format is:

{
  "object" : {
    "artifacts": [
      {
        "kind" : "xcframework",
        "packageRef" : {
          "identity" : "agoraaudio_ios",
          "kind" : "remoteSourceControl",
          "location" : "https://github.com/AgoraIO/AgoraAudio_iOS",
          "name" : "AgoraRtcKit"
        },
        "path" : "/Users/logan/Developer/Bazel/app_engine/.build/artifacts/agoraaudio_ios/AgoraAiEchoCancellationExtension/AgoraAiEchoCancellationExtension.xcframework",
        "source" : {
          "checksum" : "840779bf446bbe6e271ce4036fc24e52c6404b1288044205633b5961bf63f554",
          "type" : "remote",
          "url" : "https://download.agora.io/swiftpm/AgoraAudio_iOS/4.2.2/AgoraAiEchoCancellationExtension.xcframework.zip"
        },
        "targetName" : "AgoraAiEchoCancellationExtension"
      },
      ...
    ],
    "dependencies": [
      {
        "basedOn" : null,
        "packageRef" : {
          "identity" : "agoraaudio_ios",
          "kind" : "remoteSourceControl",
          "location" : "https://github.com/AgoraIO/AgoraAudio_iOS",
          "name" : "AgoraRtcKit"
        },
        "state" : {
          "checkoutState" : {
            "revision" : "62ea5dda567e6c394a4a14c40c4738d5604924b8",
            "version" : "4.2.2"
          },
          "name" : "sourceControlCheckout"
        },
        "subpath" : "AgoraAudio_iOS"
      },
      ...
    ],
  },
  "version" : 6
}

Note that the dependencies here contain not only direct dependencies but transitive dependencies as well. The one thing that this file is missing is the dependencies between packages and targets / products within those targets. However, with the checkouts, this is easily solved. We can have an action change directory into the .build/checkouts/[subpath] directory and run swift package dump-package to parse its Package.swift file and gather this information.

To avoid re-running swift package resolve for each top-level package dependency, we could have a swiftpkg "root" repository rule which runs the swift package manager commands and builds an index from them which it exposes as an exported file, e.g. @swiftpkg//:index.json. Then the other repository rules like @swiftpkg_firebase//:Sources_FirebaseAnalytics would depend on @swiftpkg//:index.json and it would just be responsible for downloading its own repository and providing its own Bazel targets. I have verified locally that repository-rules can depend on each-other, they just can't form a cycle.

To avoid checking the .build directory into the repository, I'm thinking we could could copy / symlink the Package.swift (and potentially Package.resolved) files into a temporary directory created by an action of the the root repository rule and run the swift package commands inside that directory. The workspace-state.json file could be declared to Bazel and the Package.swift (+Package.resolved) would be file target attributes so it would be hermetic and cache-able. The action would build the full index (potentially with the same format as the existing swift-deps-index.json file) which would be consumed by the other repository rules.

To summarize, my proposal is as follows:

1. The swiftpkg repository rule action copies or symlinks Package.swift (and potentially Package.resolved) into a temporary directory and runs swift package update.
2. The action then parses the workspace-state.json file together with the Package.swift files of the direct and transitive dependencies under checkouts using swift package dump-package to build index.json at a path declared in the Bazel workspace. It then deletes the temporary directory.
3. The index.json file is exported by the swiftpkg repository rule and consumed to update the WORKSPACE / MODULE and by the swiftpkg_dependency repository rules.

[cgrindel]

I may be missing something, but I don't believe this will work:

• The build files for the Swift package repositories cannot be generated until we have the Swift module to Bazel targets mapping.
• I don't believe there is a way for a repository rule to depend on another repository rule.

To make something like this work, I think that the swiftpkg repository would need to declare the Swift package repositories.

[AttilaTheFun]

Following up here, I created an example repository demonstrating how my proposed approach can work with a single module extension, no .build in the repository, and no additional files checked into the repository:
https://github.com/AttilaTheFun/swiftpkg_example

[brentleyjones]

This has been done in recent releases.