Improving build time and size of cilium-agent rock #7
Conversation
…, clear compile cache
| source: https://github.com/libbpf/bpftool.git | ||
| source-tag: v7.0.0 |
There was a problem hiding this comment.
We used to pull bpf-next with a specific source commit that pointed to v7.0.0 version of bpftool. The issue is bpf-next is huge and pulls a lot of kernel code just to build simple bpftool, the git server being slow there also does not help.
So instead we are switching to use the official bpftool mirror that only syncs bpftool related header files and code, which is way smaller and builds way faster.
This saves ~8GiB of disk space.
| llvm-10-bpf: | ||
| plugin: cmake | ||
| source-type: git | ||
| source: https://github.com/llvm/llvm-project.git | ||
| source-branch: "llvmorg-10.0.0" | ||
| source-subdir: "llvm" | ||
| cmake-generator: Ninja | ||
| override-pull: | | ||
| craftctl default | ||
| git cherry-pick 29bc5dd19407c4d7cad1c059dea26ee216ddc7ca | ||
| git cherry-pick 13f6c81c5d9a7a34a684363bcaad8eb7c65356fd | ||
| git cherry-pick ea72b0319d7b0f0c2fcf41d121afa5d031b319d5 | ||
| override-build: | | ||
| cmake $CRAFT_PART_SRC_WORK -G "Ninja" \ | ||
| -DLLVM_TARGETS_TO_BUILD="BPF" \ | ||
| -DLLVM_ENABLE_PROJECTS="clang" \ | ||
| -DBUILD_SHARED_LIBS="OFF" \ | ||
| -DCMAKE_BUILD_TYPE="Release" \ | ||
| -DLLVM_BUILD_RUNTIME="OFF" \ | ||
| -DCMAKE_INSTALL_PREFIX="/usr/local" | ||
|
|
||
| ninja clang llc llvm-objcopy | ||
|
|
||
| cp -r $CRAFT_PART_BUILD/bin $CRAFT_PART_INSTALL | ||
| stage: | ||
| - -bin | ||
| organize: | ||
| bin/clang-10: usr/bin/clang | ||
| bin/llc: usr/bin/llc | ||
| bin/llvm-objcopy: usr/bin/llvm-objcopy |
There was a problem hiding this comment.
We are dropping llvm-10-bpf part entirely. This was just statically built llvm tools with only the bpf target turned on. The cherry picked commits are there to enable the bpf targets / specific to that. Cilium also states that technically any llvm+clang >= 10 with bpf target turned on is good, check here for reference. So instead of pulling the whole llvm-10 source and building it, we pull it and stage it from the apt repository.
| - libiberty-dev | ||
| - llvm-15 | ||
| - clang-15 | ||
| - clang-tools-15 | ||
| - lldb-15 | ||
| - lld-15 | ||
| - clang-format-15 | ||
| - libc++-15-dev | ||
| - libc++abi-15-dev | ||
| build-environment: | ||
| - EXTRA_CFLAGS: --static | ||
| - LLVM_CONFIG: "/usr/bin/llvm-config-15" | ||
| - LLVM_STRIP: "/usr/bin/llvm-strip-15" |
There was a problem hiding this comment.
These are related to bpftool mirror's build requirements. We build it statically with LLVM+clang 15.
| @@ -41,6 +48,7 @@ parts: | |||
| source-type: git | |||
| source: https://github.com/cilium/proxy.git | |||
| source-tag: v1.27 | |||
| source-depth: 1 | |||
There was a problem hiding this comment.
These are all around the PR so we don't fetch the entire history of a git repo, saves us some disk space and clone time.
cilium/rockcraft.yaml
Outdated
| override-prime: | | ||
| craftctl default | ||
| rm -rf /root/.cache/bazel |
There was a problem hiding this comment.
cilium-envoy uses bazel to build, and bazel creates a huge cache that's ~6G. This doesn't work well with rockcraft especially in CI scenarios, where the rockcraft cache isn't persisted for long. We save a lot of space in the lxd container. I'll actually move this to override-build to save up on maximum disk space usage during runtime since override-prime will wait for everything to finish.
| cp -a $CRAFT_PART_INSTALL/usr/bin/clang-10 $CRAFT_PART_INSTALL/usr/bin/clang | ||
| cp -a $CRAFT_PART_INSTALL/usr/bin/llc-10 $CRAFT_PART_INSTALL/usr/bin/llc | ||
| cp -a $CRAFT_PART_INSTALL/usr/bin/llvm-objcopy-10 $CRAFT_PART_INSTALL/usr/bin/llvm-objcopy |
There was a problem hiding this comment.
Just duplicating symlinks for clang, llc and llvm-objcopy which are the only binaries needed by cilium afaik. Sadly it only looks for these direct names and doesn't scan for versioned tools.
| override-prime: | | ||
| craftctl default | ||
| rm -rf /root/.cache/go-build |
There was a problem hiding this comment.
Again some compile caches for go which doesn't help us very much, so removing this saves ~2G disk space.
There was a problem hiding this comment.
Brilliant work! @berkayoz Well done on finding all these little hacks to improve.
Changed bpftool mirror, pull lvm-10 from upstream instead of building, clear compile cache