[vcpkg scripts] Simplify `vcpkg_download_distfile` output handling for Asset Caching #40945

JavierMatosD · 2024-09-12T20:47:16Z

This PR updates the vcpkg_download_distfile script to remove direct messaging and rely on output from the vcpkg executable. This change ensures that messages consider asset caching.

Before:
(Example log with redundant messaging)

~/dev/vcpkg % vcpkg install fmt
Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...
The following packages will be built and installed:
    fmt:[email protected]#2
  * vcpkg-cmake:arm64-osx@2024-04-23
  * vcpkg-cmake-config:arm64-osx@2024-05-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet arm64-osx...
Compiler found: /Library/Developer/CommandLineTools/usr/bin/c++
Restored 0 package(s) from /Users/javiermatos/.cache/vcpkg/archives in 7.58 us. Use --debug to see more details.
Installing 1/3 vcpkg-cmake:arm64-osx@2024-04-23...
Building vcpkg-cmake:arm64-osx@2024-04-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 16 ms.
Elapsed time to handle vcpkg-cmake:arm64-osx: 57.2 ms
vcpkg-cmake:arm64-osx package ABI: ab01a101fdcc710bfb6d8a1b6cc2824f8a5d1ea5fcc7d0f54f1b52df68c7da50
Installing 2/3 vcpkg-cmake-config:arm64-osx@2024-05-23...
Building vcpkg-cmake-config:arm64-osx@2024-05-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/copyright
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Stored binaries in 1 destinations in 14.2 ms.
Elapsed time to handle vcpkg-cmake-config:arm64-osx: 196 ms
vcpkg-cmake-config:arm64-osx package ABI: 2e73acf69b779ce7be5136d726e4307ae589be20e90cf4a06baaac8633fd82a7
Installing 3/3 fmt:[email protected]#2...
Building fmt:[email protected]#2...
-- Downloading https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz -> fmtlib-fmt-10.2.1.tar.gz...
[DEBUG] To include the environment variables in debug output, pass --debug-env
[DEBUG] Trying to load bundleconfig from /Users/javiermatos/dev/vcpkg/vcpkg-bundle.json
[DEBUG] Failed to open: /Users/javiermatos/dev/vcpkg/vcpkg-bundle.json
[DEBUG] Bundle config: readonly=false, usegitregistry=false, embeddedsha=nullopt, deployment=Git, vsversion=nullopt
[DEBUG] Metrics enabled.
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Feature flag 'dependencygraph' unset
[DEBUG] 1000: execute_process(curl --fail -L https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz --create-dirs --output /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part)
[DEBUG] 1000: cmd_execute_and_stream_data() returned 0 after   566531 us
[DEBUG] Trying to hash /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part
[DEBUG] /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part has hash 27df90c681ec37e55625062a79e3b83589b6d7e94eff37a3b412bb8c1473f757a8adb727603acc9185c3490628269216843b7d7bd5a3cb37f0029da5d1495ffa
Downloading https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz
[DEBUG] /Users/runner/work/1/s/src/vcpkg/commands.download.cpp(148): 
[DEBUG] Time in subprocesses: 566531us
[DEBUG] Time in parsing JSON: 4us
[DEBUG] Time in JSON reader: 0us
[DEBUG] Time in filesystem: 1362us
[DEBUG] Time in loading ports: 0us
[DEBUG] Exiting after 578 ms (575746us)

-- Extracting source /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz
-- Applying patch fix-visibility.patch
-- Applying patch fix-write-batch.patch
-- Using source at /Users/javiermatos/dev/vcpkg/buildtrees/fmt/src/10.2.1-a991065f88.clean
-- Found external ninja('1.11.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/lib/pkgconfig/fmt.pc
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/debug/lib/pkgconfig/fmt.pc
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/usage
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 70.5 ms.
Elapsed time to handle fmt:arm64-osx: 3.8 s
fmt:arm64-osx package ABI: a6d6451d7953939a412dcafadbd5f80ab424b3c39a37dd8f6fa0d13530682c71
Total install time: 4.1 s
The package fmt provides CMake targets:

    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt)

    # Or use the header-only version
    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt-header-only)

~/dev/vcpkg %

After:
(Example log with streamlined output that reflects asset caching)

~/dev/vcpkg % vcpkg install fmt --no-binarycaching                 
Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache hit for cmake-3.29.2-macos-universal.tar.gz; downloaded from: file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533
Extracting cmake...
The following packages will be built and installed:
    fmt:[email protected]
  * vcpkg-cmake:arm64-osx@2024-04-23
  * vcpkg-cmake-config:arm64-osx@2024-05-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet arm64-osx...
Compiler found: /Library/Developer/CommandLineTools/usr/bin/c++
Installing 1/3 vcpkg-cmake:arm64-osx@2024-04-23...
Building vcpkg-cmake:arm64-osx@2024-04-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/copyright
-- Performing post-build validation
Elapsed time to handle vcpkg-cmake:arm64-osx: 61.1 ms
vcpkg-cmake:arm64-osx package ABI: 44ec445be8ff2dfe41221664683af0e4d54a584531320d608d72d533f9403db8
Installing 2/3 vcpkg-cmake-config:arm64-osx@2024-05-23...
Building vcpkg-cmake-config:arm64-osx@2024-05-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/copyright
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Elapsed time to handle vcpkg-cmake-config:arm64-osx: 126 ms
vcpkg-cmake-config:arm64-osx package ABI: 66faabcff87a98b7f4eee82713f982c1d16b7890aa2e128d751dddeab46afd0e
Installing 3/3 fmt:[email protected]...
Building fmt:[email protected]...
Asset cache hit for fmtlib-fmt-11.0.2.tar.gz; downloaded from: file:///Users/javiermatos/dev/asset-cache/47ff6d289dcc22681eea6da465b0348172921e7cafff8fd57a1540d3232cc6b53250a4625c954ee0944c87963b17680ecbc3ea123e43c2c822efe0dc6fa6cef3
-- Extracting source /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-11.0.2.tar.gz
-- Applying patch fix-write-batch.patch
-- Using source at /Users/javiermatos/dev/vcpkg/buildtrees/fmt/src/11.0.2-463dc5ff21.clean
-- Found external ninja('1.11.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/lib/pkgconfig/fmt.pc
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/debug/lib/pkgconfig/fmt.pc
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/usage
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/copyright
-- Performing post-build validation
Elapsed time to handle fmt:arm64-osx: 3.1 s
fmt:arm64-osx package ABI: 3fcdbaa4539f8a1e2e1e4f103c34f523364ab26bc81f599dd06eb1268a870191
Total install time: 3.3 s
The package fmt provides CMake targets:

    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt)

    # Or use the header-only version
    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt-header-only)

~/dev/vcpkg %

BillyONeal · 2024-09-13T23:44:21Z

scripts/cmake/vcpkg_download_distfile.cmake

            RESULT_VARIABLE error_code
            WORKING_DIRECTORY "${DOWNLOADS}"
        )

        if(NOT "${error_code}" EQUAL "0")
-            message("${output}")
            z_vcpkg_download_distfile_show_proxy_and_fail("${error_code}")


Can you confirm that this doesn't repeat error messages?

Confirmed it does not repeat the error message. That said, I did find a bug with our asset caching output.

To repro ensure that x-block-origin is not enabled, trigger an error by giving vcpkg_download_distfile a bad URL, and try to install the package.

It will spit out:

error: Missing <pkg> and downloads are blocked by x-block-origin. which is false.

scripts/cmake/vcpkg_download_distfile.cmake

JavierMatosD · 2024-09-16T20:41:25Z

CI failure is related to a known issue with sentry-native and not related to this PR.

use vcpkg output in download_distfile

4e4d602

JavierMatosD mentioned this pull request Sep 12, 2024

[vcpkg_download_distfile] always display output for downloading #40804

Closed

jimwang118 assigned JonLiu1993 Sep 13, 2024

JonLiu1993 changed the title ~~Simplify vcpkg_download_distfile output handling for Asset Caching~~ [vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching Sep 13, 2024

JonLiu1993 added category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:internal This PR or Issue was filed by the vcpkg team. labels Sep 13, 2024

BillyONeal reviewed Sep 13, 2024

View reviewed changes

konjac reviewed Sep 14, 2024

View reviewed changes

scripts/cmake/vcpkg_download_distfile.cmake Show resolved Hide resolved

konjac reviewed Sep 14, 2024

View reviewed changes

scripts/cmake/vcpkg_download_distfile.cmake Show resolved Hide resolved

JavierMatosD requested a review from BillyONeal September 16, 2024 19:59

print message when using file from downloads

71e9fa3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg scripts] Simplify `vcpkg_download_distfile` output handling for Asset Caching #40945

[vcpkg scripts] Simplify `vcpkg_download_distfile` output handling for Asset Caching #40945

JavierMatosD commented Sep 12, 2024

BillyONeal Sep 13, 2024

JavierMatosD Sep 16, 2024 •

edited

Loading

JavierMatosD commented Sep 16, 2024

[vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching #40945

Are you sure you want to change the base?

[vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching #40945

Conversation

JavierMatosD commented Sep 12, 2024

BillyONeal Sep 13, 2024

Choose a reason for hiding this comment

JavierMatosD Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

JavierMatosD commented Sep 16, 2024

[vcpkg scripts] Simplify `vcpkg_download_distfile` output handling for Asset Caching #40945

[vcpkg scripts] Simplify `vcpkg_download_distfile` output handling for Asset Caching #40945

JavierMatosD Sep 16, 2024 •

edited

Loading