feat(ci)_: build on windows and macos #5840

mendelskiv93 · 2024-09-16T18:39:41Z

This PR:

Creates a common Jenkinsfile for Unix systems (Linux and macOS) and introduces changes to the Linux build path by utilizing the getAgentLabel() function.
Adds support for Windows builds, where we override the default shell to avoid using Nix Shell.

Key Changes:

Unified Jenkinsfile for Linux and macOS builds.
Adjusted Linux build path to use the getAgentLabel() function.
Windows build support added with overridden SHELL to bypass Nix Shell.

status-github-bot · 2024-09-16T18:39:48Z

Hey @mendelskiv93, and thank you so much for making your first pull request in status-go! ❤️ Please help us make your experience better by filling out this brief questionnaire https://goo.gl/forms/uWqNcVpVz7OIopXg2

github-actions · 2024-09-16T18:40:16Z

We require commits to follow the Conventional Commits, but with _ for non-breaking changes.
Please fix these commit messages:

Commit message is not well-formed: "windows/macos ci build"

status-im-auto · 2024-09-16T18:40:31Z

Jenkins Builds

Click to see older builds (109)

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
❌	`3e8b378`	#1	2024-09-16 18:40:30	~27 sec	`macos`	📄`log`
✔️	`3e8b378`	#1	2024-09-16 18:41:21	~1 min	`tests-rpc`	📄`log`
✖️	`3e8b378`	#1	2024-09-16 18:41:27	~1 min	`tests`	📄`log`
✔️	`3e8b378`	#1	2024-09-16 18:43:18	~3 min	`windows`	📦`zip`
✔️	`3e8b378`	#1	2024-09-16 18:43:46	~3 min	`linux`	📦`zip`
✔️	`3e8b378`	#1	2024-09-16 18:45:48	~5 min	`android`	📦`aar`
✔️	`3e8b378`	#1	2024-09-16 18:47:15	~7 min	`macos`	📦`zip`
✔️	`3e8b378`	#1	2024-09-16 18:49:41	~9 min	`ios`	📦`zip`

✖️	`88f369c`	#2	2024-09-23 07:45:54	~1 min	`tests`	📄`log`
✔️	`88f369c`	#2	2024-09-23 07:46:56	~2 min	`tests-rpc`	📄`log`
✔️	`88f369c`	#2	2024-09-23 07:48:47	~4 min	`android`	📦`aar`
✔️	`88f369c`	#2	2024-09-23 07:49:27	~5 min	`ios`	📦`zip`

✔️	`6dc7965`	#3	2024-09-24 18:11:45	~1 min	`tests-rpc`	📄`log`
✖️	`6dc7965`	#3	2024-09-24 18:11:53	~1 min	`tests`	📄`log`
✔️	`6dc7965`	#3	2024-09-24 18:12:13	~1 min	`android`	📦`aar`
✔️	`6dc7965`	#3	2024-09-24 18:15:09	~4 min	`ios`	📦`zip`

❌	`615c629`	#4	2024-09-24 18:18:08	~14 sec	`macos`	📄`log`
❌	`615c629`	#4	2024-09-24 18:18:08	~16 sec	`macos`	📄`log`
❌	`615c629`	#4	2024-09-24 18:18:17	~21 sec	`linux`	📄`log`
❌	`615c629`	#4	2024-09-24 18:18:25	~27 sec	`windows`	📄`log`
✖️	`615c629`	#4	2024-09-24 18:19:18	~1 min	`tests`	📄`log`
✔️	`615c629`	#4	2024-09-24 18:19:31	~1 min	`android`	📦`aar`
✔️	`615c629`	#4	2024-09-24 18:21:08	~3 min	`ios`	📦`zip`
✔️	`615c629`	#4	2024-09-24 18:21:10	~3 min	`tests-rpc`	📄`log`

❌	`b1ada8c`	#5	2024-09-24 18:27:52	~12 sec	`macos`	📄`log`
✖️	`b1ada8c`	#5	2024-09-24 18:28:35	~53 sec	`tests`	📄`log`
✔️	`b1ada8c`	#5	2024-09-24 18:29:17	~1 min	`android`	📦`aar`
✔️	`b1ada8c`	#5	2024-09-24 18:30:07	~2 min	`tests-rpc`	📄`log`
❌	`b1ada8c`	#5	2024-09-24 18:30:14	~2 min	`windows`	📄`log`
✔️	`b1ada8c`	#5	2024-09-24 18:30:26	~2 min	`ios`	📦`zip`
✔️	`b1ada8c`	#5	2024-09-24 18:31:22	~3 min	`linux`	📦`zip`
✔️	`b1ada8c`	#5	2024-09-24 18:33:20	~5 min	`macos`	📦`zip`

❌	`a15b5f2`	#6	2024-09-26 15:06:54	~16 sec	`macos`	📄`log`
❌	`a15b5f2`	#6	2024-09-26 15:07:40	~58 sec	`windows`	📄`log`
✖️	`a15b5f2`	#6	2024-09-26 15:07:48	~1 min	`tests`	📄`log`
✔️	`a15b5f2`	#6	2024-09-26 15:07:52	~1 min	`tests-rpc`	📄`log`
✔️	`a15b5f2`	#6	2024-09-26 15:09:58	~3 min	`ios`	📦`zip`
✔️	`a15b5f2`	#6	2024-09-26 15:10:29	~3 min	`linux`	📦`zip`
✔️	`a15b5f2`	#6	2024-09-26 15:11:58	~5 min	`android`	📦`aar`
✔️	`a15b5f2`	#6	2024-09-26 15:13:40	~7 min	`macos`	📦`zip`
❌	`a15b5f2`	#1	2024-09-26 17:55:33	~30 sec	`null`	📄`log`

❌	`1e8b012`	#7	2024-09-26 18:21:52	~11 sec	`macos`	📄`log`
✔️	`1e8b012`	#7	2024-09-26 18:22:44	~59 sec	`windows`	📦`zip`
✖️	`1e8b012`	#7	2024-09-26 18:23:05	~1 min	`tests`	📄`log`
✔️	`1e8b012`	#7	2024-09-26 18:23:24	~1 min	`android`	📦`aar`
✔️	`1e8b012`	#7	2024-09-26 18:23:43	~2 min	`linux`	📦`zip`
✔️	`1e8b012`	#7	2024-09-26 18:23:51	~2 min	`tests-rpc`	📄`log`
✔️	`1e8b012`	#7	2024-09-26 18:24:47	~3 min	`macos`	📦`zip`
✔️	`1e8b012`	#7	2024-09-26 18:24:58	~3 min	`ios`	📦`zip`
❌	`1e8b012`	#8	2024-09-27 06:07:42	~2 min	`macos`	📄`log`
✔️	`1e8b012`	#8	2024-09-27 18:25:11	~3 min	`ios`	📦`zip`
❌	`1e8b012`	#16	2024-09-29 20:02:41	~11 sec	`macos`	📄`log`
❌	`1e8b012`	#14	2024-09-29 20:02:41	~12 sec	`macos`	📄`log`
✔️	`1e8b012`	#14	2024-09-29 20:03:33	~1 min	`tests-rpc`	📄`log`
✔️	`1e8b012`	#14	2024-09-29 20:03:39	~1 min	`windows`	📦`zip`
✖️	`1e8b012`	#14	2024-09-29 20:03:45	~1 min	`tests`	📄`log`
✔️	`1e8b012`	#14	2024-09-29 20:03:50	~1 min	`android`	📦`aar`
✔️	`1e8b012`	#14	2024-09-29 20:04:21	~1 min	`linux`	📦`zip`
✔️	`1e8b012`	#15	2024-09-29 20:05:55	~3 min	`ios`	📦`zip`
❌	`1e8b012`	#17	2024-09-30 06:27:57	~50 sec	`macos`	📄`log`

❌	`6cae3e7`	#9	2024-09-29 18:45:41	~51 sec	`macos`	📄`log`
✖️	`6cae3e7`	#8	2024-09-29 18:45:51	~57 sec	`tests`	📄`log`
✔️	`6cae3e7`	#8	2024-09-29 18:46:10	~1 min	`windows`	📦`zip`
✔️	`6cae3e7`	#8	2024-09-29 18:46:29	~1 min	`android`	📦`aar`
✔️	`6cae3e7`	#8	2024-09-29 18:46:35	~1 min	`macos`	📦`zip`
✔️	`6cae3e7`	#8	2024-09-29 18:46:51	~2 min	`linux`	📦`zip`
✔️	`6cae3e7`	#8	2024-09-29 18:47:02	~2 min	`tests-rpc`	📄`log`
✔️	`6cae3e7`	#9	2024-09-29 18:47:55	~3 min	`ios`	📦`zip`
❌	`6cae3e7`	#10	2024-09-29 18:58:24	~46 sec	`macos`	📄`log`

❌	`664a743`	#9	2024-09-29 19:24:06	~14 sec	`android`	📄`log`
❌	`664a743`	#9	2024-09-29 19:24:07	~13 sec	`linux`	📄`log`
❌	`664a743`	#10	2024-09-29 19:24:08	~14 sec	`ios`	📄`log`
❌	`664a743`	#11	2024-09-29 19:24:17	~12 sec	`macos`	📄`log`
❌	`664a743`	#9	2024-09-29 19:24:17	~14 sec	`macos`	📄`log`
✖️	`664a743`	#9	2024-09-29 19:24:17	~13 sec	`tests`	📄`log`
✔️	`664a743`	#9	2024-09-29 19:25:01	~59 sec	`windows`	📦`zip`
✔️	`664a743`	#9	2024-09-29 19:26:04	~2 min	`tests-rpc`	📄`log`

❌	`bb80325`	#10	2024-09-29 19:34:56	~11 sec	`android`	📄`log`
❌	`bb80325`	#12	2024-09-29 19:35:08	~10 sec	`macos`	📄`log`
❌	`bb80325`	#10	2024-09-29 19:35:08	~12 sec	`linux`	📄`log`
❌	`bb80325`	#11	2024-09-29 19:35:08	~13 sec	`ios`	📄`log`
❌	`bb80325`	#10	2024-09-29 19:35:08	~12 sec	`macos`	📄`log`
✖️	`bb80325`	#10	2024-09-29 19:35:08	~13 sec	`tests`	📄`log`
✔️	`bb80325`	#10	2024-09-29 19:35:57	~1 min	`windows`	📦`zip`
✔️	`bb80325`	#10	2024-09-29 19:36:52	~2 min	`tests-rpc`	📄`log`

❌	`3d708d9`	#11	2024-09-29 19:39:55	~12 sec	`android`	📄`log`
❌	`3d708d9`	#11	2024-09-29 19:39:55	~10 sec	`linux`	📄`log`
❌	`3d708d9`	#13	2024-09-29 19:40:02	~10 sec	`macos`	📄`log`
❌	`3d708d9`	#12	2024-09-29 19:40:02	~13 sec	`ios`	📄`log`
❌	`3d708d9`	#11	2024-09-29 19:40:03	~10 sec	`macos`	📄`log`
✖️	`3d708d9`	#11	2024-09-29 19:40:05	~14 sec	`tests`	📄`log`
✔️	`3d708d9`	#11	2024-09-29 19:40:46	~58 sec	`tests-rpc`	📄`log`
✔️	`3d708d9`	#11	2024-09-29 19:40:51	~59 sec	`windows`	📦`zip`

❌	`ec52a6c`	#12	2024-09-29 19:44:17	~12 sec	`android`	📄`log`
❌	`ec52a6c`	#12	2024-09-29 19:44:18	~12 sec	`linux`	📄`log`
❌	`ec52a6c`	#13	2024-09-29 19:44:18	~13 sec	`ios`	📄`log`
❌	`ec52a6c`	#14	2024-09-29 19:44:26	~11 sec	`macos`	📄`log`
❌	`ec52a6c`	#12	2024-09-29 19:44:27	~14 sec	`macos`	📄`log`
✖️	`ec52a6c`	#12	2024-09-29 19:44:27	~12 sec	`tests`	📄`log`
❌	`ec52a6c`	#12	2024-09-29 19:44:27	~12 sec	`windows`	📄`log`
✔️	`ec52a6c`	#12	2024-09-29 19:45:12	~58 sec	`tests-rpc`	📄`log`

❌	`ee73bec`	#13	2024-09-29 19:53:49	~11 sec	`android`	📄`log`
❌	`ee73bec`	#14	2024-09-29 19:53:54	~13 sec	`ios`	📄`log`
✖️	`ee73bec`	#13	2024-09-29 19:53:55	~10 sec	`tests`	📄`log`
❌	`ee73bec`	#13	2024-09-29 19:54:05	~22 sec	`linux`	📄`log`
❌	`ee73bec`	#13	2024-09-29 19:54:05	~22 sec	`macos`	📄`log`
❌	`ee73bec`	#15	2024-09-29 19:54:06	~23 sec	`macos`	📄`log`
✔️	`ee73bec`	#13	2024-09-29 19:54:43	~59 sec	`tests-rpc`	📄`log`
✔️	`ee73bec`	#13	2024-09-29 19:55:02	~1 min	`windows`	📦`zip`

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
❌	`5ffdbe9`	#15	2024-09-30 07:16:47	~13 sec	`android`	📄`log`
❌	`5ffdbe9`	#15	2024-09-30 07:17:02	~13 sec	`linux`	📄`log`
✖️	`5ffdbe9`	#15	2024-09-30 07:17:04	~13 sec	`tests`	📄`log`
❌	`5ffdbe9`	#18	2024-09-30 07:17:26	~47 sec	`macos`	📄`log`
✔️	`5ffdbe9`	#15	2024-09-30 07:17:58	~1 min	`windows`	📦`zip`
✔️	`5ffdbe9`	#15	2024-09-30 07:18:28	~1 min	`macos`	📦`zip`
✔️	`5ffdbe9`	#15	2024-09-30 07:18:44	~2 min	`tests-rpc`	📄`log`
✔️	`5ffdbe9`	#16	2024-09-30 07:20:14	~3 min	`ios`	📦`zip`

❌	`e18c381`	#19	2024-09-30 09:20:11	~50 sec	`macos`	📄`log`
✖️	`e18c381`	#16	2024-09-30 09:20:26	~56 sec	`tests`	📄`log`
✔️	`e18c381`	#16	2024-09-30 09:20:28	~1 min	`windows`	📦`zip`
✔️	`e18c381`	#16	2024-09-30 09:20:53	~1 min	`android`	📦`aar`
✔️	`e18c381`	#16	2024-09-30 09:21:05	~1 min	`macos`	📦`zip`
✔️	`e18c381`	#16	2024-09-30 09:21:22	~2 min	`linux`	📦`zip`
✔️	`e18c381`	#16	2024-09-30 09:21:29	~2 min	`tests-rpc`	📄`log`
✔️	`e18c381`	#17	2024-09-30 09:22:30	~3 min	`ios`	📦`zip`

jakubgs

Seems like the only difference here is whether we use sh or nix.shell() right?

Why not just make a wrapper shell function that uses one or the other depending on platform?

_assets/ci/Jenkinsfile.windows

_assets/ci/Jenkinsfile.unix

jakubgs

Looking good.

_assets/ci/Jenkinsfile.desktop

jakubgs

Good stuff.

_assets/ci/Jenkinsfile.desktop

jakubgs · 2024-09-27T13:13:03Z

This build error on MacOS x86_64:

In file included from /nix/store/rc2mfzjdfnhfm8grc4zi0p85m4k7rj7k-apple-framework-CoreServices-11.0.0/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43:
In file included from /nix/store/ggzvbag576nnphvr1y5869yx26m0kr82-apple-framework-Security-11.0.0/Library/Frameworks/Security.framework/Headers/Security.h:87:
/nix/store/ggzvbag576nnphvr1y5869yx26m0kr82-apple-framework-Security-11.0.0/Library/Frameworks/Security.framework/Headers/SecCode.h:35:10: fatal error: 'xpc/xpc.h' file not found
#include <xpc/xpc.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [Makefile:215: statusgo-library] Error 1

Is most probably caused by missing component of Apple SDK, which actually does exist under darwin.apple_sdk.libs.xpc:
https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/os-specific/darwin/apple-sdk/default.nix

We can see an example of pulling an Apple SDK framework in nim-waku:
https://github.com/waku-org/nwaku/blob/dc9725b98a30db457028cef57da8c0691c6d3589/shell.nix#L16

jakubgs · 2024-09-27T13:16:01Z

We could pull in only darwin.apple_sdk.libs.xpc to limit size, but we could also pull in whole darwin.apple_sdk to avoid future issues with missing libraries like this.

jakubgs · 2024-09-27T13:19:33Z

One possible fix would be to override the Apple SDK version since it's different depending on architecture:

  # Pick an SDK
  apple_sdk = if stdenv.hostPlatform.isAarch64 then apple_sdk_11_0 else apple_sdk_10_12;

https://github.com/NixOS/nixpkgs/blob/e9ee548d90ff586a6471b4ae80ae9cfcbceb3420/pkgs/top-level/darwin-packages.nix#L50-L51

So we could just override that in the nix/overlay.nix:

  drawin.apple_sdk = prev.darwin.apple_sdk_11_0;

Since Apple SDK is probably already include in some form in the stdenv for MacOS.

nix/pkgs/codecov-cli/default.nix

_assets/ci/Jenkinsfile.desktop

Added appleLibXPC to buildInputs to pull in darwin.apple_sdk.libs.xpc.

mendelskiv93 requested a review from a team September 16, 2024 18:39

jakubgs approved these changes Sep 17, 2024

View reviewed changes

_assets/ci/Jenkinsfile.windows Outdated Show resolved Hide resolved

siddarthkay reviewed Sep 18, 2024

View reviewed changes

_assets/ci/Jenkinsfile.unix Outdated Show resolved Hide resolved

mendelskiv93 marked this pull request as draft September 19, 2024 10:47

mendelskiv93 force-pushed the ci/desktop-build branch from 3e8b378 to 88f369c Compare September 23, 2024 07:44

jakubgs approved these changes Sep 24, 2024

View reviewed changes

_assets/ci/Jenkinsfile.desktop Outdated Show resolved Hide resolved

mendelskiv93 force-pushed the ci/desktop-build branch 3 times, most recently from 615c629 to b1ada8c Compare September 24, 2024 18:27

jakubgs approved these changes Sep 26, 2024

View reviewed changes

jakubgs reviewed Sep 26, 2024

View reviewed changes

_assets/ci/Jenkinsfile.desktop Outdated Show resolved Hide resolved

jakubgs reviewed Sep 26, 2024

View reviewed changes

_assets/ci/Jenkinsfile.desktop Outdated Show resolved Hide resolved

windows/macos ci build

a15b5f2

mendelskiv93 force-pushed the ci/desktop-build branch from b1ada8c to a15b5f2 Compare September 26, 2024 15:06

fix_:add x86_64-darwin hash for codecov pkgs

1e8b012

mendelskiv93 force-pushed the ci/desktop-build branch 6 times, most recently from ee73bec to 1e8b012 Compare September 29, 2024 20:02

siddarthkay reviewed Sep 30, 2024

View reviewed changes

nix/pkgs/codecov-cli/default.nix Show resolved Hide resolved

siddarthkay reviewed Sep 30, 2024

View reviewed changes

_assets/ci/Jenkinsfile.desktop Show resolved Hide resolved

fix_:include appleLibXPC

e18c381

Added appleLibXPC to buildInputs to pull in darwin.apple_sdk.libs.xpc.

mendelskiv93 force-pushed the ci/desktop-build branch from 5ffdbe9 to e18c381 Compare September 30, 2024 09:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ci)_: build on windows and macos #5840

feat(ci)_: build on windows and macos #5840

mendelskiv93 commented Sep 16, 2024

status-github-bot bot commented Sep 16, 2024

github-actions bot commented Sep 16, 2024 •

edited

Loading

status-im-auto commented Sep 16, 2024 •

edited

Loading

jakubgs left a comment

jakubgs left a comment

jakubgs left a comment

jakubgs commented Sep 27, 2024

jakubgs commented Sep 27, 2024

jakubgs commented Sep 27, 2024

feat(ci)_: build on windows and macos #5840

Are you sure you want to change the base?

feat(ci)_: build on windows and macos #5840

Conversation

mendelskiv93 commented Sep 16, 2024

status-github-bot bot commented Sep 16, 2024

github-actions bot commented Sep 16, 2024 • edited Loading

status-im-auto commented Sep 16, 2024 • edited Loading

Jenkins Builds

jakubgs left a comment

Choose a reason for hiding this comment

jakubgs left a comment

Choose a reason for hiding this comment

jakubgs left a comment

Choose a reason for hiding this comment

jakubgs commented Sep 27, 2024

jakubgs commented Sep 27, 2024

jakubgs commented Sep 27, 2024

github-actions bot commented Sep 16, 2024 •

edited

Loading

status-im-auto commented Sep 16, 2024 •

edited

Loading