bazel: foundation #20849

dotnwat · 2024-07-04T18:05:57Z

Bazel build foundation.

This builds all dependencies from source, along with a solid chunk of our foundational libraries (container, bytes, model, utils, net, rpc, etc...), but doesn't build the final redpanda binary.

The idea behind merging this now is that with CI running (see GHA checks) we can lock-in the progress here. If we were to not merge, then as build coverage increased in the development branch, we would be increasingly chasing and fixing regressions created in upstream.

Best to review all commits combined. The argument for not squashing all these commits is that they contain quite a lot of good information, including some work that was replaced as development proceeded (e.g. openssl replacing dependencies) but which is useful to have in the historical log.

Backports Required

Release Notes

none

Signed-off-by: Noah Watkins <[email protected]>

Useful for passing in BOOST_TEST_MODULE=foo to add a main(). Signed-off-by: Noah Watkins <[email protected]>

Signed-off-by: Noah Watkins <[email protected]>

The config option `--@seastar//:numa=True` didn't read very intuitively, so changed it to `numactl`. Signed-off-by: Noah Watkins <[email protected]>

Signed-off-by: Noah Watkins <[email protected]>

c4milo · 2024-07-09T19:48:23Z

bazel/install-deps.sh

+  exit 1
+fi
+
+deb_deps=(


For these is where rules-nixpkg/nixpkgs_package or devenv/devbox could be handy in terms of determinism.

ah right on. is anyone using these?

We use devenv in Cloud to stand up the entire stack in our laptops. DevProd is using devbox.

TBH it's better to move all tools into Bazel - I don't think that's a huge lift for this list. nixpkg is a fair amount of complexity, and I've had problems with clang toolchains and such not having a good development environment because of all the extra flags they inject to make things hermetic.

bazel/repositories.bzl

Co-authored-by: Mike Boquard <[email protected]>

The hash changed when a previous commit altered a comment. Signed-off-by: Noah Watkins <[email protected]>

Signed-off-by: Noah Watkins <[email protected]>

rockwotj

Looks really nice, first full round of review/feedback.

MODULE.bazel

rockwotj · 2024-07-10T16:11:58Z

MODULE.bazel

+use_repo(non_module_dependencies, "numactl")
+use_repo(non_module_dependencies, "openssl")
+use_repo(non_module_dependencies, "openssl-fips")
+use_repo(non_module_dependencies, "rapidjson")


Does this work? https://registry.bazel.build/modules/rapidjson

The published version appears to be too old. I added a note into MODULE.bazel.

SG - FWIW you can override the version much like we are doing for the compilation commands database generator

SG - FWIW you can override the version much like we are doing for the compilation commands database generator

i see this pattern, and it always seems super sketchy. like, IIUC, it effectively uses the BUILD files from BCR and source files from whatever version you swap in for the override? i mean, it works fine for a header-only library, assuming that the BUILD uses globbing. Generally I'd want the BUILD to be perfectly in sync with the source...

Yeah, maybe we should just be proposing updates upstream. Let me read about how dependency resolution works with non modules

MODULE.bazel

src/v/crypto/BUILD

src/v/ssx/BUILD

src/v/utils/BUILD

vbotbuildovich · 2024-07-10T17:24:34Z

new failures in https://buildkite.com/redpanda/redpanda/builds/51320#01909d77-59ec-4b8b-9a11-2f7f47141d4d:

"rptest.transactions.tx_admin_api_test.TxAdminTest.test_mark_transaction_expired"

vbotbuildovich · 2024-07-10T17:33:59Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51320#01909d77-59f1-43e8-87c2-e6822f584ed6

Signed-off-by: Noah Watkins <[email protected]>

Also removes an empty dependency list. Signed-off-by: Noah Watkins <[email protected]>

Signed-off-by: Noah Watkins <[email protected]>

rockwotj

LGTM

rockwotj · 2024-07-11T15:22:53Z

src/v/bytes/BUILD

+        "include/bytes/iobuf.h",
+    ],
+    strip_include_prefix = "include",
+    visibility = ["//visibility:public"],


FWIW instead of making everything public, you can also make public the default.

package( default_visibility = ["//visibility:public"], )

https://bazel.build/reference/be/functions#package

Also at somepoint we probably want to limit visibility, but it's probably fine for now.

ahh right. honestly after this PR merges I think it's gonna be open season on evolving the Bazel build. Hopefully we put down a solid foundation!

I think this is a good foundation - I am happy to merge and help tweak stuff as we go along.

dotnwat added 30 commits June 20, 2024 10:44

bytes: reference header in current directory

7eebf73

Signed-off-by: Noah Watkins <[email protected]>

bazel: add third party dependencies

281b8c9

Signed-off-by: Noah Watkins <[email protected]>

bazel: build wrapper

9895d33

Signed-off-by: Noah Watkins <[email protected]>

bazel: add baes library bazel build

39d3d9a

Signed-off-by: Noah Watkins <[email protected]>

bazel: add ssx library bazel build

61a0fe1

Signed-off-by: Noah Watkins <[email protected]>

bazel: add strings library bazel build

65ffd83

Signed-off-by: Noah Watkins <[email protected]>

bazel: add bytes library bazel build

80802ae

Signed-off-by: Noah Watkins <[email protected]>

bazel: add container library bazel build

e5aa44f

Signed-off-by: Noah Watkins <[email protected]>

bazel: add utils library bazel build

06ff960

Signed-off-by: Noah Watkins <[email protected]>

bazel: add crypto library bazel build

b90e0ca

Signed-off-by: Noah Watkins <[email protected]>

bazel: add random library bazel build

2b47fbd

Signed-off-by: Noah Watkins <[email protected]>

bazel: add serde library bazel build

b8d3ed2

Signed-off-by: Noah Watkins <[email protected]>

bazel: add test_utils library bazel build

e8efde5

Signed-off-by: Noah Watkins <[email protected]>

bazel: update seastar and add config options

6826627

Signed-off-by: Noah Watkins <[email protected]>

bazel: add missing hwloc depdendency

e2fcf0a

Signed-off-by: Noah Watkins <[email protected]>

bazel: use label conditions

be287e1

Signed-off-by: Noah Watkins <[email protected]>

bazel: use inherited c++ standard setting

e58bd47

Signed-off-by: Noah Watkins <[email protected]>

bazel: clarify seastar build comments

f9319a1

Signed-off-by: Noah Watkins <[email protected]>

bazel: make strings library fine grained

82e7954

Signed-off-by: Noah Watkins <[email protected]>

bazel: add no-seastar boost test macro

4e96dcc

Signed-off-by: Noah Watkins <[email protected]>

bazel: add strings library tests

bbfde7d

Signed-off-by: Noah Watkins <[email protected]>

bazel: add crypto tests

9a1f8e6

Signed-off-by: Noah Watkins <[email protected]>

bazel: make random library more fine grained

b52e2a7

Signed-off-by: Noah Watkins <[email protected]>

bazel: add random library tests

169f910

Signed-off-by: Noah Watkins <[email protected]>

bazel: allow defines to pass through for test macro

ddd4cbd

Useful for passing in BOOST_TEST_MODULE=foo to add a main(). Signed-off-by: Noah Watkins <[email protected]>

bazel: add tests to utils library

cc035b8

Signed-off-by: Noah Watkins <[email protected]>

bazel: add base outcome test

01abe7a

Signed-off-by: Noah Watkins <[email protected]>

Merge remote-tracking branch 'origin/dev' into bazel

0bbd40e

bazel: use a better name for seastar numactl option

075f5eb

The config option `--@seastar//:numa=True` didn't read very intuitively, so changed it to `numactl`. Signed-off-by: Noah Watkins <[email protected]>

bazel: add ssx tests

1fcd770

Signed-off-by: Noah Watkins <[email protected]>

dotnwat requested review from michael-redpanda and a team as code owners July 9, 2024 04:23

dotnwat requested review from gousteris, mmaslankaprv, StephanDollberg, rockwotj, andrwng and c4milo and removed request for a team and gousteris July 9, 2024 04:23

c4milo reviewed Jul 9, 2024

View reviewed changes

michael-redpanda reviewed Jul 10, 2024

View reviewed changes

bazel/repositories.bzl Outdated Show resolved Hide resolved

dotnwat and others added 5 commits July 10, 2024 07:00

bazel: clarify wording on openssl version requirements

24f2079

Co-authored-by: Mike Boquard <[email protected]>

bazel: update module lock file

4d952e0

The hash changed when a previous commit altered a comment. Signed-off-by: Noah Watkins <[email protected]>

Merge remote-tracking branch 'origin/dev' into bazel

c091280

bazel: update ubuntu:noble deps

69699d0

Signed-off-by: Noah Watkins <[email protected]>

bazel: run gha build with ubuntu:noble

b7ceb49

Signed-off-by: Noah Watkins <[email protected]>

rockwotj reviewed Jul 10, 2024

View reviewed changes

dotnwat added 7 commits July 10, 2024 14:37

bazel: move tests into separate package

be1d2d9

Signed-off-by: Noah Watkins <[email protected]>

bazel: add missing ssx library deps

0f2faf6

Signed-off-by: Noah Watkins <[email protected]>

bazel: use conventional underscore in naming

0bf26e2

Signed-off-by: Noah Watkins <[email protected]>

bazel: make it easier to find bazel TODOs

65c4600

Signed-off-by: Noah Watkins <[email protected]>

bazel: add missing dependencies

134f6b7

Also removes an empty dependency list. Signed-off-by: Noah Watkins <[email protected]>

bazel: add note about use of non-BCR dependencies

99abab1

Signed-off-by: Noah Watkins <[email protected]>

bazel: add utils:base64 library

6b3bc74

Signed-off-by: Noah Watkins <[email protected]>

rockwotj approved these changes Jul 11, 2024

View reviewed changes

dotnwat merged commit 42ebcfc into dev Jul 11, 2024
21 checks passed

dotnwat deleted the bazel branch July 11, 2024 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bazel: foundation #20849

bazel: foundation #20849

dotnwat commented Jul 4, 2024 •

edited

Loading

c4milo Jul 9, 2024 •

edited

Loading

dotnwat Jul 9, 2024

c4milo Jul 9, 2024

rockwotj Jul 10, 2024

rockwotj left a comment

rockwotj Jul 10, 2024

dotnwat Jul 11, 2024

rockwotj Jul 11, 2024

dotnwat Jul 11, 2024

rockwotj Jul 11, 2024

vbotbuildovich commented Jul 10, 2024

vbotbuildovich commented Jul 10, 2024

rockwotj left a comment

rockwotj Jul 11, 2024

dotnwat Jul 11, 2024

rockwotj Jul 11, 2024

bazel: foundation #20849

bazel: foundation #20849

Conversation

dotnwat commented Jul 4, 2024 • edited Loading

Backports Required

Release Notes

c4milo Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rockwotj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vbotbuildovich commented Jul 10, 2024

vbotbuildovich commented Jul 10, 2024

rockwotj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotnwat commented Jul 4, 2024 •

edited

Loading

c4milo Jul 9, 2024 •

edited

Loading